The purpose of a typical clinical laboratory evaluation for an assay is to determine if the assay has adequate performance. What is implied by adequate performance is that the errors are small enough so that clinicians will not make incorrect medical decisions based on assay error.
The typical protocol for a method comparison experiment to assess performance is the split sample protocol, whereby a sample is processed and then divided in two to be analyzed by both systems.
In a protocol to assess imprecision, sufficient sample is processed to provide a pool that is analyzed repeatedly by the system.
Both of these protocols are biased. The reason is that for either the method comparison or imprecision experiment, the same processed sample is being analyzed rather than collecting and processing each sample separately. The opportunity for error due to processing has been excluded in these experiments.
Why does this situation persist? Because another reason – often the main laboratory reason – for performing these evaluations is to answer the question is this new analyzer or method as good as the existing method. In this sense, the protocols are not biased. So I don’t advocate changing the way the protocols are done but one should realize that when one considers the other implied other goal of medical utility for these studies, the protocols are biased. In fact, the implied goal of medical utility is routinely stated as a conclusion of evaluations, such as the ABC assay is acceptable for use in monitoring patients with XYZ.