Excluding funny business, there are many other reasons why most evaluations are biased but this blog entry will focus on just one – reagent bias. Error due to different reagent lots is one of the biggest error sources for diagnostic products.
When a new product is being evaluated, there are usually just a few reagents that are available (and produced by the manufacturing process). This is also the time that the calibration process has been finalized, which minimizes average bias. The problem is that with time, the manufacturing process will produce reagents that exhibit bias since the raw materials will change, the process might change, and so on.
The bias occurs because the reagents lots that are sampled are bunched in time around product launch. An unbiased evaluation would sample reagents throughout the product life cycle, but without a time machine, this is not possible.
Companies can explore the extent of reagent lot error by doing Taguchi type experiments, whereby reagents are created and evaluated that include levels of components at the ends of expected concentrations according to a factorial design. But these reagents should not be used in a method comparison evaluation because each of these reagents is a rare and thus unlikely event and also not part of the manufacturing process. So the reagent lot bias that occurs in evaluations is unavoidable.