As a consultant, much of my work are projects that I suggest. As might be expected, I have to sell people on these ideas. Sometimes, it is clear that stated objections are not the real reason the client doesn’t want to do the project. This is known as resistance and takes on many forms such as “great idea, let’s do it on the next project” or “great idea, but our problem is different” or “I just don’t understand.”

There have been some reject votes for the revised CLSI total error standard, EP21. Based on several discussions, the two major objections look like resistance. Here’s why.

The first objection is that EP21 is too complicated. Now the people who voted to reject EP21 are all senior people in the field of clinical chemistry. EP21 involves graphing differences between a new and comparison method. In one plot the differences are manipulated with subtractions. This is probably the simplest of all the CLSI evaluation documents.

The second objection is that the revised document includes total error from any source that is present in the evaluation protocol (not just analytical error). So what does this mean? Well, if one were going to evaluate a POC device such as a glucose meter which had fingersticks as a valid sample mode, the evaluation should employ fingersticks as the sampling method for the POC glucose meter under test. It is likely that the comparison method would be a laboratory glucose instrument using a venous sample. This means that differences between test and comparison will include errors due to improper fingerstick technique, which is a user error, not an analytical error. This is appropriate because the goal of the evaluation is to estimate performance in routine use.

So what were the objections? We can’t be expected to evaluate all pre- and post-analytical errors such as problems with the LIS. Let’s develop a new standard for pre- and post-analytical errors.

But the revised EP21 does not suggest evaluating all pre- and post-analytical errors, it advises one not to exclude the opportunity for relevant errors in the protocol, such as the glucose example above.

The consultant’s task is to expose objections that have no merit.

I previously commented that the title of EP23 makes no sense.

The EP23 subcommittee’s response to my comment is that my use of “quality control” is not what is meant in EP23.

EP22 and EP23 were originally proposed to deal with CMS’s proposal for equivalent quality control. This use of quality control clearly meant running 2 liquid control samples per day for each assay. This is how I used the term quality control, how it is used in principle in virtually any text about quality control  (which is of course uses quality control for processes other than the clinical laboratory), and how most people will understand it.

EP23 now defines quality control as “part of quality management focused on fulfilling quality requirements (ISO 9000).  Curiously, their cited reference  contains the following:

“ISO. An action to eliminate the cause of a detected non-conformity or other undesirable situation. ISO 9000. Geneva, Switzerland: International Organization for Standardization; 2005  3.6.5.”

This is the definition of a corrective action.

The problem is that EP23 is calling all corrective actions and preventive actions “quality control.” Perhaps this is to conform to some ISO terminology – I don’t have ISO 9000:2005. This is unnecessarily confusing and almost everyone will misinterpret what EP23 calls quality control.

Virtually any evaluation protocol has a recommended sample size, or at least a procedure for calculating a sample size. This entry explores some problems with sample sizes.

Assume that it is desired to calculate a sample size for an error grid evaluation (1). See reference one for details, but in an error grid evaluation, one performs a method comparison, plots the results in the grid and calculates the percentage of results in each error grid zone. In an error grid, there are least two areas of interest – the innermost zone (called “A” here) which contains most of the differences and an outer zone (called “C” here) which should contain no results as these differences have a high potential for serious patient harm.

I will skip the discussion of calculating the sample size for zone A. To calculate the sample size for zone C, one needs a goal – assume that one wishes less than one result per million in zone C. It can be shown (2) that the required sample size to prove with 95% confidence that less than one result per million is in zone C is to run 371,000,000 samples and observe no results in zone C. Additionally, the candidate assay has to be run in a representative way (with respect to routine use) in the method comparison. Since this number of samples is a bit much, how can one be confident that the goal for zone C will be achieved? The answer is using risk management techniques. The clinical laboratory has to perform FMEA/fault tree analysis (3) to ensure that user errors don’t cause zone C results and the manufacturer has to perform FMEA/fault tree analysis to ensure that the system itself doesn’t cause zone C results.

References

1. CLSI/NCCLS. How to Construct and Interpret an Error Grid for Diagnostic Assays EP27 Proposed Guideline. CLSI/NCCLS document EP27-P. Wayne, PA: NCCLS; 2009.
2. Hahn GJ and Meeker WQ. Statistical intervals. A guide for practitioners. Wiley: New York, 1991, pp 103-105.
3. CLSI/NCCLS Risk Management Techniques to Identify and Control Laboratory Error Sources. Proposed Guideline –Third Edition CLSI/NCCLS document EP18-P3 Wayne, PA: NCCLS; 2009.

I had occasion to read about the suggestion that some fraction (often 50%) of biological variation should play a role in setting assay performance standards. This makes no sense to me. Here’s why.

The most fundamental measure of assay performance is diagnostic accuracy. That is – sensitivity the percentage of people tested whose assay value is above the cutoff and who have the disease and, and specificity the percentage of people tested whose assay value is below the cutoff and who do not have the disease.

Biological variation serves to decrease diagnostic accuracy. If a person who does not have the disease has a spike in the assay due to biological variation and this elevates the value beyond the cutoff, a false positive is the result. The more biological variation, the more the decrease in diagnostic accuracy. Analytical error does the same thing – the more error, the lower the observed diagnostic accuracy. From a diagnostic accuracy standpoint, there is no difference between biological variation and analytical error. Thus, it makes no sense that the performance of an assay should be allowed to reach 50% of the biological variation.

How should performance standards be set?

Use error grids to define limits where no results should occur (e.g., errors large enough to have high potential to cause patient harm). These limits are called limits of erroneous results (LER) by the FDA. These are the most important limits and are set using clinical judgment.

The area in an error grid to contain most of the results (often 95%) is less important and can be set using performance achieved by existing technology, with the caveat that considerations must be given to special circumstances such as cost, turn around time when it’s important, and so on.

Here are some comments (some paraphrased) and suggestions made on a blog about adapting the Aviation Safety Reporting System to reduce the rate of medical errors.

Incident reporting provides “no useful information about the true frequency of errors in an institution.” It’s too expensive, it takes too much time. There’s too much data. We should only report errors that cause temporary or serious harm.

These comments were made by Robert M. Wachter, MD. From his blog’s bio …

 “He has published 200 articles and 6 books in the fields of quality, safety, and health policy. He is also a national leader in the fields of patient safety and healthcare quality. He is editor of AHRQ WebM&M, a case-based patient safety journal on the Web, and AHRQ Patient Safety Network, the leading federal patient safety portal.”

There are two ways to reduce the rate of medical errors.

1. Lower the probability of errors that have not yet occurred
2. Lower the rate of errors that have occurred.

The Aviation Safety Reporting System is an example of tackling #2.

Many (most) errors do not directly cause harm – they have the potential to cause harm. This can be understood be mapping out each medical procedure. To suggest to not report all errors will shortchange the system.

The most important errors receive focus by using a Pareto chart or table. The fastest way to reduce error rates relies on a suitable ranking system.

All of this takes time, training, and commitment.

Some successful medical examples: anesthesiology improvements in the 70s 80s (1). The recent reduction of infections in placing central lines (2).

Dr. Wachter is going in the wrong direction.

References

1. Cooper JB, Newbower RS, Long CD, McPeek B: Preventable anesthesia mishaps: A study of human factors. ANESTHESIOLOGY 1978; 49:399-406. An online version of Paper 5 can be found at http://qshc.bmj.com/content/vol11/issue3/#CLASSIC_PAPERS
2. Pronovost P. et al. An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU. N Engl J Med 2006;355:2725-32

PS – A commentator who agrees with Dr. Wachter offers a standard bit of resistance to using the Aviation Safety Reporting System in medicine – We’re different – medicine is more complicated.

There is/was? a promising marker for prostate cancer called EPCA-2 (1). The person who discovered the test is now being sued by the company that licensed the marker (2). According to the lawsuit:

“Notwithstanding the spectacular (and false) results proclaimed by defendants, the Getzenberg assay was no more accurate in distinguishing cancerous tissue from normal tissue than flipping a coin,”

Among the coauthors in reference 1 are well known clinical chemists. They may have a lot of explaining to do.

Another well known clinical chemist questioned the validity of the results (3). He may be looking pretty good when all of this gets straightened out.

We shall see.

References

1. Leman ES, Cannon GW, Trock BJ, Sokoll LJ, Chan DW, Mangold Ln Partin AW, Getzenberg RH. EPCA-2: a highly specific serum marker for prostate cancer. Urology. 2007;69:714-20.
2. http://www.pittsburghlive.com/x/pittsburghtrib/news/pittsburgh/s_641304.html
3. Point:EPCA-2: A promising new serum biomarker for prostatic carcinoma? Diamandis EP. Clinical Biochemistry 2007;40:1437-1439.

Glucose has been in the news lately both in the New York Times and the medical literature.

A standard favoring tight glycemic control was dropped, possibly because the glucose meters used were inaccurate (1-3.)

Glucose meters that use glucose dehydrogenase can give very wrong answers in dialysis patients (4).

And finally, the FDA is considering revising glucose standards (5). This blog entry is about glucose standards revision. The article mentions that FDA is considering revising the performance standards in ISO 15197 which are: 95% of values must be with 20% of reference at 75 mg/dL or above and within 15 mg/dL below 75 mg/dL. The Boyd and Bruns modeling paper is referenced and Bruns is quoted in the article. I have previously critiqued the Boyd and Bruns paper (6).

Here is the main point which is not covered in the article. The main problem with the ISO standard is that it specifies performance for only 95% of the data. This of course leaves up to 5% of the data as unspecified and means that if up to 5% of glucose results had large enough error so that hyperglycemic patients were classified as hypoglycemic and vice versa, that assay would be acceptable according to ISO. This is equivalent to saying that up to a 5% wrong site surgery rate is acceptable! 100% of the data must be specified as is the case with glucose error grids, which predated the ISO guideline.

A second problem with the ISO guideline is that the performance limits ignore user error. But user error contributes to the final result and must be part of the performance specification.

The protocol must also be part of the guideline. In a short method comparison, it is possible to observe no large errors. To supplement this, specific analytical properties of the assay must be specified as well as risk management criteria. There are recent glucose recalls where software was faulty and allowed units to be changed from mg/dL to mmol/L or vice versa without customer knowledge.

I mention in passing that the Boyd and Bruns article referred to the article underestimate total user due to an inadequate model which fails to account for interferences. Reference 4 is an example of interferences and responsible for at least 13 deaths.

References

1. See: http://www.nytimes.com/2009/08/18/health/policy/18diabetes.html?_r=1&scp=1&sq=glucose&st=cse
2. Intensive versus Conventional Glucose Control in Critically Ill Patients. NEJM 2009;360:1283-1297
3. Scott MG, Bruns DE, Boyd JC, and Sacks DB. Tight glucose control in the intensive care unit: Are glucose meters up to the task? Clin Chem 2009;55:18-20.
4. See: http://www.fda.gov/MedicalDevices/Safety/AlertsandNotices/PatientAlerts/ucm177189.htm
5. See: http://www.aacc.org/publications/cln/2009/september/Pages/inside0909.aspx
6. Krouwer, JS. How to Improve Total Error Modeling by Accounting for Error Sources Beyond Imprecision and Bias, Clin Chem 2001;47:1329-30.

DB responded to my entry about “Just because it’s not easy to measure …”,  which gave me mixed feelings. I am thankful that he acknowledged that his statement was not what he meant but I must admit that I am in awe of DB – hence I was uncomfortable that he would apologize.

Since I follow his blog, my first instinct was to comment that I agreed with the comment made by “Curious” in the entry that had the quote. But DB didn’t respond to Curious’s comment so I left it alone.

In my field of laboratory medicine, I will continue – having started about 20 years ago – to advocate for measuring everything that’s important. This includes not just measuring the easy things like precision and bias but also the difficult things like rare interferences or user errors.

I added the quote from DB, because it’s important for guidelines. With healthcare reform, we can expect more rather than fewer guidelines in part fueled by the 1.1 billion for comparative effectiveness research.  Obama said:

“The point is we want to use science, we want doctors and medical experts to be making decisions that all too often right now are driven by skewed policies, by outdated means of reimbursement, or by insurance companies.“

I became aware of this from a Dr Rich entry. Comparative effectiveness research will be based on data, analyzed to yield measurements, which will turn into conclusions and recommendations. There is a famous statistical example about how difficult it is to measure things. Youden (1) compiled 15 different estimates of  the astronomical unit from scientists who estimated that quantity over the years 1895–1961. The confidence interval constructed by every scientist did not overlap the confidence interval of his predecessor. The difficulties are only greater in medicine. Just getting agreement on definitions is important as I cited an example for side effects of prostatectomy where urinary incontinence was defined as using greater than 3 pads per day implying that less than 3 pads per day = continence. Maybe urologists could agree with that definition, but I don’t think patients would.

References

 Youden WJ. Enduring values. Technometrics 1972;14:1–11.

The American Diabetes Association (ADA) has revised its recommendation for diagnosis of diabetes and now recommends using hemoglobin A1c to diagnose diabetes (1). They also say:

A1C tests to diagnose diabetes should be performed using clinical laboratory equipment. Point-of-care instruments have not yet been shown to be sufficiently accurate or precise for diagnosing diabetes.

Maybe this is a trend to slow down the adoption of point-of-care (POC) assays.

Scott, et. al. speculate that one of the reasons that tight glycemic control (TGC) in ICUs has been dropped as a guideline is that the use of POC glucose meters (meaning less accurate) as opposed to laboratory assays may have contributed to the adverse findings of TGC.

This will also mean that the CLSI standard POCT09-P Selection Criteria for Point-of-Care Testing Devices will need to be revised since although they suggest conducting performance evaluations, their examples of benefits of POC assays now include two examples where the accuracy of POC tests have either been rejected or questioned (1-2). POCT09-P also cites the benefit of a POC troponin assay, where performance was tested by surveying clinicians – hardly a rigorous test (3) and not conforming to their own recommendation for a real evaluation.

References

1. International Expert Committee Report on the Role of the A1C Assay in the Diagnosis of Diabetes. Diabetes Care 2009;32:1327-1334.
2. Scott MG, Bruns DE, Boyd JC, and Sacks DB. Tight glucose control in the intensive care unit: Are glucose meters up to the task? Clin Chem 2009;55:18-20.
3. Lee-Lewandrowski E, Corboy D, Lewandrowski, K, Sinclair J, McDermot S,  Benzer, TI. Implementation of a Point-of-Care Satellite Laboratory in the Emergency Department of an Academic Medical Center. Archives of Pathology and Laboratory Medicine 2003;127:456–460.

There have been several objections to measuring errors that are not as easy as calculating a standard deviation.

One comment was – pre-analytical error is important but can’t be measured in a method comparison protocol. It needs to be handled by risk management.

Similar arguments were made during a meeting for measurement uncertainty, where it was suggested that large but rare analytical errors be handled by risk management.

DB writes:

I favor limited guidelines, but not measurement.  Measurement has too many unintended consequences.

ISO 15197, a standard for home glucose meters has a specification for total analytical error which does not include errors due to pre-analytical error. User errors are to be evaluated separately and without stating any analysis procedure:

Results shall be documented in a report

Unfortunately, risk management as used by these people means sweeping these problems under the rug and is the same as DB’s advice about not measuring things or the ISO guidance.

It’s time to start measuring errors from all sources – it’s possible and necessary.