Whooping Cough and False Positives


There has been a recent incident that has set the quality folks abuzz. As reported in The New York Times, a hospital treated a number of its workers for whooping cough, due to a positive test for that condition. It was later determined that no one had whooping cough – all of the test results were false positives. In a standards committee, I cited this article as an example of why it is important to perform a FMEA (Failure Mode and Effects Analysis) as there has been some resistance that FMEAs are too complicated for hospital laboratories.

Westgard cited the Times article as a reason to stress the need for method validation skills. I agree with most of what he says although I suggest that in addition to performing a method validation, one must also consider pre- and post-analytical issues – a reason to perform FMEA.

However, I disagree with one of Westgard’s points:

“Finally, there are those damned statistics that get in the way of a practical understanding of experimental results. As evidence of this problem, Clinical Chemistry (the International Journal of Molecular Diagnostics and Laboratory Medicine) recommends that authors utilize the Bland-Altman approach (difference plot with t-test statistics) for analyzing method comparison data, in spite of the fact that regression techniques are usually much more informative, particularly in identifying proportional analytical errors that invalidate the error estimates from t-test analysis. Evidently, laboratory scientists are not sophisticated enough to understand and utilize regression analysis correctly. That again speaks to the inadequacy of our education and training programs and the lack of proper guidance in the validation of molecular tests, even by a leading international journal.”

This advice is incorrect. Total error is more informative than regression and a better first step in assessing assay performance. Proportional error does not invalidate the t-test. Among the methods for assessing total error are:

Technique Issues Origin* CLSI** Standard
Model:  combine systematic and random errors Can discard outliers, model can be wrong, only accounts for 95% of results, specs are for components Westgard, Peterson, others None, but components based on EP5 and EP9
Model: GUM Can discard outliers, model can be wrong, very complicated, only accounts for 95% of results ISO Clin. Chem. 51
Bland Altman Can discard outliers?, normal data assumption can mislead, only accounts for 95% of results Bland Altman EP21
Mountain Plot Need a lot of data Krouwer, Monti, Lynch EP21

*Or champions **Clinical and laboratory standards institute

If total error is unacceptable, further analysis may be warranted, such as regression.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: