There has been a recent incident that has set the quality folks abuzz. As reported in The New York Times, a hospital treated a number of its workers for whooping cough, due to a positive test for that condition. It was later determined that no one had whooping cough – all of the test results were false positives. In a standards committee, I cited this article as an example of why it is important to perform a FMEA (Failure Mode and Effects Analysis) as there has been some resistance that FMEAs are too complicated for hospital laboratories.
Westgard cited the Times article as a reason to stress the need for method validation skills. I agree with most of what he says although I suggest that in addition to performing a method validation, one must also consider pre- and post-analytical issues – a reason to perform FMEA.
However, I disagree with one of Westgard’s points:
“Finally, there are those damned statistics that get in the way of a practical understanding of experimental results. As evidence of this problem, Clinical Chemistry (the International Journal of Molecular Diagnostics and Laboratory Medicine) recommends that authors utilize the Bland-Altman approach (difference plot with t-test statistics) for analyzing method comparison data, in spite of the fact that regression techniques are usually much more informative, particularly in identifying proportional analytical errors that invalidate the error estimates from t-test analysis. Evidently, laboratory scientists are not sophisticated enough to understand and utilize regression analysis correctly. That again speaks to the inadequacy of our education and training programs and the lack of proper guidance in the validation of molecular tests, even by a leading international journal.”
This advice is incorrect. Total error is more informative than regression and a better first step in assessing assay performance. Proportional error does not invalidate the t-test. Among the methods for assessing total error are:
|Model: combine systematic and random errors||Can discard outliers, model can be wrong, only accounts for 95% of results, specs are for components||Westgard, Peterson, others||None, but components based on EP5 and EP9|
|Model: GUM||Can discard outliers, model can be wrong, very complicated, only accounts for 95% of results||ISO||Clin. Chem. 51|
|Bland Altman||Can discard outliers?, normal data assumption can mislead, only accounts for 95% of results||Bland Altman||EP21|
|Mountain Plot||Need a lot of data||Krouwer, Monti, Lynch||EP21|
*Or champions **Clinical and laboratory standards institute
If total error is unacceptable, further analysis may be warranted, such as regression.