Why I’m still mad after four years

May 22, 2007

I don’t brood over things, but recently I had occasion to revisit something which still bothers me. The CLSI (formerly NCCLS) document EP11 which is about uniformity of performance claims was a controversial document, which I previously discussedhttps://jkrouwer.wordpress.com/2006/03/12/uniformity-of-product-performance-claims/. There was pressure from CLSI management to cancel this document, which resulted in a CLSI strategy conference for Evaluation Protocols in 2003.

This conference was facilitated by Ed Kimmelman, who in my opinion did a poor job. Everything had to go through him which was then reissued by him, often in a garbled form. I called this a filter (scroll down to: 5/14/06 – Beware of the filter) not a facilitator.   This was a lost opportunity but what made me mad was a report circulated to the attendees – 30 or so influential people in the field – in which Kimmelman made the following statement on page 1 under “Background”:

“NCCLS management has the belief that the process of developing evaluation protocols within the NCCLS consensus process can be improved.

In recent years it has become apparent that there has been difficulty moving certain evaluation protocols through the consensus process due to a number of reasons, including

– dissatisfaction with the content of such protocols, – dissatisfaction with the constituency representation on the Evaluation Protocols Area Committee, and   perceived failure of NCCLS and area committee management to meet their responsibilities under the NCCLS Administrative Procedures”

I have highlighted the item in yellow that most irritated me, although the rest of the statements are also not true. So basically, my management – I was the area committee chairholder – was being questioned. Therefore, I put together some facts about – and have now updated – document status when I was chairholder (1999-2004), before and after.

Evaluation Protocol Project Activity: 1999-2004

Document – Last pre 1999 Action Formal title Status 1999 Status 2004 After 2004 2002 Core: Ave. Sales / year
EP5T – 1992 Evaluation of Precision Performance of Clinical Chemistry Devices EP5A published EP5A2 published 2004 1:266
EP6P – 1986 – Evaluation of the Linearity of Quantitative Analytical Methods No action for 13 years! EP6A published 2003 2:96
EP7P – 1986 Interference Testing in Clinical Chemistry No action for 13 years! EP7A published 2002 1:219
EP9A – 1995 Method Comparison and Bias Estimation Using Patient Samples No action for 7 years EP9A2 published 2002 2:283
EP10A – 1998 Preliminary Evaluation of Quantitative Clinical Laboratory Methods EP10 A2 published 2002 EP10 A3 published 2006 2:237
EP11P – 1995 Uniform Description of Claims for in Vitro Diagnostic Tests No action for 8 years EP11 cancelled by Board 2003 2:45
EP12 Project approved – 1986 User Protocol for Evaluation of Qualitative Test Performance No action for 13 years! EP12A published 2002 2:27
EP13R – 1995 Laboratory Statistics—Standard Deviation ———— ———— 2:45
EP14 Project approved – ? Evaluation of Matrix Effects EP14P published EP14A published 2001 EP14A2 published 2005 2:51
EP15P – 1998 User Demonstration of Performance for Precision and Accuracy EP15A published 2001 EP15A2 published 2006 2:48
EP17 Project (see note) Protocols for Determination of Limits of Quantitation EP17A published 2004 —–
EP18 Project approved – ? Quality Management for Unit-Use Testing EP18P published EP18A published 2002 2:23
EP19 Project approved – 1997 A Framework for NCCLS Evaluation Protocols EP19R published 2002 2:25
EP20 Project approved – 1998 Quality Goals for Acceptable Performance and Threshold Criteria for Outliers EP20 cancelled 2003 —–
EP21 Project approved – 1998 Total Error for Clinical Laboratory Methods EP21A published 2003 No Data

Notes:

EP10 – I was the chairholder of EP10A, EP10A2 and EP10A3. EP17 – an earlier version had been cancelled in 2000, EP17 was re-approved 2001 EP19 – This was published as a “P” document in 2000, then as a “R” (Report) EP20 – The chairholder resigned in 2001, a new subcommittee was started 2002, the project was cancelled by the area committee. EP21 – I proposed and led EP21. The last column refers to a financial tracking system, I put in place for all CLSI documents. Core refers to sales – 1: most sales, 2: moderate sales, 3: least sales.

Of the 14 projects for which action was expected – a chairholder serves for six years – every document was either advanced or cancelled during my term. This included three documents for which no action had taken place each for 13 years! (e.g., more than two chairholder terms).

I called the head of NCCLS and requested an apology in writing to be distributed to correct the Kimmelman statement. After a heated exchange, I was later told that I might not serve my final year as chairholder and the next area committee meeting might be cancelled. Well, neither of those things happened. The area committee took place with the president and president-elect in attendance, who tried to get me again to cancel EP11. I didn’t – EP11 was advanced and the Board cancelled it.

Dan Tholen as the vice-chairholder and who supported me often in writing, was slated to replace me but was not invited by NCCLS to be chairholder. I think that NCCLS reasoned that they could put up with me for one more year to finish my term, but putting up with a new chairholder with similar views for an additional six years was to be avoided.

So in all, there are battles within organizations to get one’s way – that’s not the issue. What bothered me was that I was attacked publically for poor performance and this was not true. I never got my apology – the other people all won awards.

These days, I still contribute to CLSI. I gave a recent talk at their annual forum, am the chairholder of an existing project (EP18) and have just proposed a new project.

 

Advertisements

What it takes to get your idea accepted

May 13, 2007

About 20 years ago, I attended a seminar given by Dr. George Bowers of Hartford Hospital about his attempt to standardize enzyme assay results (1). What I remember most is a comment he made to a group of people after the talk, that his idea “would take 10 years or so to get accepted.” This struck me as strange. Why would a good idea take so long to be accepted. Yet, there are other examples, such as measures by the British navy (2) to prevent scurvy, but 200 years after a solution had been found!

This essay chronicles my attempts to champion the use of the right model to evaluate the total analytical error of diagnostic assays, something I have been working on for over 15 years. I should emphasize that I am not the originator of the right model, but have championed it.

1979 Lawton, et al. publish a paper that describes how to model the total analytical error for diagnostic assays.

1980 Brauer and Rand publish Lawton’s model in a form more accessible to clinical chemists.

1986 Bland and Altman publish a simple way of evaluating assays in The Lancet. Whereas clinical chemists don’t typically read The Lancet, this article is so frequently cited, that this paper is well known to clinical chemists. This approach, while not the same as the Lawton model, provides an alternative to estimate the total analytical error for diagnostic assays, when one has a reference method.

1991 Westgard et al. issue a model for cholesterol analytical performance, which ignores the Lawton model.

1992 Krouwer champions using the right model for diagnostic assays and critiques the NCEP (National Cholesterol Education Program) model of analytical cholesterol performance and goals.

1995 Krouwer and Monti publish a nonparametric way to assess total analytical error, to complement the Bland-Altman approach.

2001 Krouwer publishes a Letter critiquing Bruns and Boyd’s use of the wrong model to evaluate glucose analytical performance and goals. Bruns and Boyd respond and agree.

2002 Krouwer publishes a paper similar to the 1992 Archives paper, stimulated by a talk by Petersen who advocated an incorrect model and basically blew off a question about using the right (e.g., Lawton model).

2002 Miller et al. compare how the Lawton model and the NCEP model give different results.

2002 Dewitte, et al. show that adoption of Bland-Altman plots has been slow. From their Letter, “We observed increasing use of the Bland–Altman plot over the years, from 8% in 1995 to 14% in 1996, and 31–36% in more recent years.”

2003 Krouwer critiques an updated version of the NCEP program whereby the model for analytical cholesterol performance and goals is unchanged from previous models.

2003 Krouwer earlier had proposed a guideline for NCCLS (now CLSI) that became an approved project. He was the chairholder for this project, which was published as EP21A.

2005 Recommendations for Clinical Laboratory Improvement Amendments of 1988 (CLIA) Waiver Applications. This FDA guidance uses EP21A.

2007 Westgard, long a proponent of an incorrect model, reissues this model on his website.

2007 Krouwer critiques creatinine goals and analytical performance provided by a standards group – The National Kidney Disease Education Program (NKDEP) Laboratory Working Group. Although this group started with the right model in their paper, they didn’t use it.

So what does it take to get your idea accepted? I’m afraid I still don’t know the answer.

References

  1.     Bowers GN and McComb  RB.A unifying reference system for clinical enzymology: aspartate aminotransferase and the International Clinical Enzyme Scale Clin Chem 1984;30:1128-1136.
  2.     Mosteller  F. Innovation and evaluation. Science 1981;211:881-886.

1979 Lawton WH, Sylvester EA, Young-Ferraro BJ. Statistical comparison of multiple analytic procedures: application to clinical chemistry. Technometrics 1979;21:397-409.

1980 Brauer GA and Rand RN. Techniques for defining and measuring quality in clinical chemistry. In: Rand RN, Eilers RJ, Lawson, NS, Broughton A eds. Quality Assurance in Health Care: A Critical Appraisal of Clinical Chemistry, Washington DC: American Association of Clinical Chemistry and College of American Pathologists; 1980:207-224.

1986 Bland JM, Altman DG. Statistical methods for assessing agreement between 2 methods of clinical measurement. Lancet 1986 i, 307-310.

1991 Westgard JO, Petersen PH, and Wiebe DA Laboratory process specifications for assuring quality in the U.S. National Cholesterol Education Program. Clin Chem 1991;37:656-661

1992 Krouwer JS. Estimating Total Analytical Error and Its Sources: Techniques to Improve Method Evaluation. Arch Pathol Lab Med 1992;116:726-731.

1995 Krouwer JS and Monti, KL. A Simple Graphical Method to Evaluate Laboratory Assays, Eur J Clin Chem and Clin. Biochem 1995;33:525-527.

2001 Krouwer JS. How to Improve Total Error Modeling by Accounting for Error Sources Beyond Imprecision and Bias. Clin Chem 2001;47:1329-30.

2002 Krouwer JS. Setting Performance Goals and Evaluating Total Analytical Error for Diagnostic Assays. Clin Chem 2002;48:919-927.

2002 Miller WG, Waymack PP, Anderson FP, Ethridge SF, Jayne EC. Performance of four homogeneous direct methods for LDL-cholesterol. Clin Chem 2002;48:489-498.

2002 Dewitte K, Fierens C, Stöckl D, and Thienpont LM. Application of the Bland–Altman Plot for Interpretation of Method-Comparison Studies: A Critical Investigation of Its Practice Clin Chem 2002;48:799-801.

2003 Krouwer JS. Problems with the NCEP (National Cholesterol Education Program) Recommendations for Cholesterol Analytical Performance. Arch Pathol Lab Med 2003;127:1249.

2003 Estimation of Total Analytical Error for Clinical Laboratory Methods; Approved Guideline NCCLS EP21A, NCCLS, 940 West Valley Road Suite 1400 Wayne, PA., 2003

2005 Recommendations for Clinical Laboratory Improvement Amendments of 1988 (CLIA) Waiver Applications. See http://www.fda.gov/cdrh/oivd/guidance/1171.pdf

2007 The Meaning and Application of Total Error. See http://www.westgard.com/essay111.htm

2007 Krouwer JS. A recommended improvement for specifying and estimating serum creatinine performance. Clinical Chemistry 2007;53:1715-1716.