About 20 years ago, I attended a seminar given by Dr. George Bowers of Hartford Hospital about his attempt to standardize enzyme assay results (1). What I remember most is a comment he made to a group of people after the talk, that his idea “would take 10 years or so to get accepted.” This struck me as strange. Why would a good idea take so long to be accepted. Yet, there are other examples, such as measures by the British navy (2) to prevent scurvy, but 200 years after a solution had been found!
This essay chronicles my attempts to champion the use of the right model to evaluate the total analytical error of diagnostic assays, something I have been working on for over 15 years. I should emphasize that I am not the originator of the right model, but have championed it.
1979 Lawton, et al. publish a paper that describes how to model the total analytical error for diagnostic assays.
1980 Brauer and Rand publish Lawton’s model in a form more accessible to clinical chemists.
1986 Bland and Altman publish a simple way of evaluating assays in The Lancet. Whereas clinical chemists don’t typically read The Lancet, this article is so frequently cited, that this paper is well known to clinical chemists. This approach, while not the same as the Lawton model, provides an alternative to estimate the total analytical error for diagnostic assays, when one has a reference method.
1991 Westgard et al. issue a model for cholesterol analytical performance, which ignores the Lawton model.
1992 Krouwer champions using the right model for diagnostic assays and critiques the NCEP (National Cholesterol Education Program) model of analytical cholesterol performance and goals.
1995 Krouwer and Monti publish a nonparametric way to assess total analytical error, to complement the Bland-Altman approach.
2001 Krouwer publishes a Letter critiquing Bruns and Boyd’s use of the wrong model to evaluate glucose analytical performance and goals. Bruns and Boyd respond and agree.
2002 Krouwer publishes a paper similar to the 1992 Archives paper, stimulated by a talk by Petersen who advocated an incorrect model and basically blew off a question about using the right (e.g., Lawton model).
2002 Miller et al. compare how the Lawton model and the NCEP model give different results.
2002 Dewitte, et al. show that adoption of Bland-Altman plots has been slow. From their Letter, “We observed increasing use of the Bland–Altman plot over the years, from 8% in 1995 to 14% in 1996, and 31–36% in more recent years.”
2003 Krouwer critiques an updated version of the NCEP program whereby the model for analytical cholesterol performance and goals is unchanged from previous models.
2003 Krouwer earlier had proposed a guideline for NCCLS (now CLSI) that became an approved project. He was the chairholder for this project, which was published as EP21A.
2005 Recommendations for Clinical Laboratory Improvement Amendments of 1988 (CLIA) Waiver Applications. This FDA guidance uses EP21A.
2007 Westgard, long a proponent of an incorrect model, reissues this model on his website.
2007 Krouwer critiques creatinine goals and analytical performance provided by a standards group – The National Kidney Disease Education Program (NKDEP) Laboratory Working Group. Although this group started with the right model in their paper, they didn’t use it.
So what does it take to get your idea accepted? I’m afraid I still don’t know the answer.
- Bowers GN and McComb RB.A unifying reference system for clinical enzymology: aspartate aminotransferase and the International Clinical Enzyme Scale Clin Chem 1984;30:1128-1136.
- Mosteller F. Innovation and evaluation. Science 1981;211:881-886.
1979 Lawton WH, Sylvester EA, Young-Ferraro BJ. Statistical comparison of multiple analytic procedures: application to clinical chemistry. Technometrics 1979;21:397-409.
1980 Brauer GA and Rand RN. Techniques for defining and measuring quality in clinical chemistry. In: Rand RN, Eilers RJ, Lawson, NS, Broughton A eds. Quality Assurance in Health Care: A Critical Appraisal of Clinical Chemistry, Washington DC: American Association of Clinical Chemistry and College of American Pathologists; 1980:207-224.
1986 Bland JM, Altman DG. Statistical methods for assessing agreement between 2 methods of clinical measurement. Lancet 1986 i, 307-310.
1991 Westgard JO, Petersen PH, and Wiebe DA Laboratory process specifications for assuring quality in the U.S. National Cholesterol Education Program. Clin Chem 1991;37:656-661
1992 Krouwer JS. Estimating Total Analytical Error and Its Sources: Techniques to Improve Method Evaluation. Arch Pathol Lab Med 1992;116:726-731.
1995 Krouwer JS and Monti, KL. A Simple Graphical Method to Evaluate Laboratory Assays, Eur J Clin Chem and Clin. Biochem 1995;33:525-527.
2001 Krouwer JS. How to Improve Total Error Modeling by Accounting for Error Sources Beyond Imprecision and Bias. Clin Chem 2001;47:1329-30.
2002 Krouwer JS. Setting Performance Goals and Evaluating Total Analytical Error for Diagnostic Assays. Clin Chem 2002;48:919-927.
2002 Miller WG, Waymack PP, Anderson FP, Ethridge SF, Jayne EC. Performance of four homogeneous direct methods for LDL-cholesterol. Clin Chem 2002;48:489-498.
2002 Dewitte K, Fierens C, Stöckl D, and Thienpont LM. Application of the Bland–Altman Plot for Interpretation of Method-Comparison Studies: A Critical Investigation of Its Practice Clin Chem 2002;48:799-801.
2003 Krouwer JS. Problems with the NCEP (National Cholesterol Education Program) Recommendations for Cholesterol Analytical Performance. Arch Pathol Lab Med 2003;127:1249.
2003 Estimation of Total Analytical Error for Clinical Laboratory Methods; Approved Guideline NCCLS EP21A, NCCLS, 940 West Valley Road Suite 1400 Wayne, PA., 2003
2005 Recommendations for Clinical Laboratory Improvement Amendments of 1988 (CLIA) Waiver Applications. See http://www.fda.gov/cdrh/oivd/guidance/1171.pdf
2007 The Meaning and Application of Total Error. See http://www.westgard.com/essay111.htm
2007 Krouwer JS. A recommended improvement for specifying and estimating serum creatinine performance. Clinical Chemistry 2007;53:1715-1716.