Below is a manufacturer specification for releasing a reagent for an assay.

It is an all or nothing approach, meaning that all reagents in the green are sold and have full value for the manufacturer, while all reagents out of spec (red line) are scrapped and have no value for the manufacturer. This is true for the two data points shown next to the arrow, one has full value, the other no value in spite of the fact that the data points have almost the same amount of error.

In the figure below is the consumer representation.

The blue line shows how patient harm is not an all or nothing approach but rather that patient harm increases with increasing error. For example, for a small glucose error in the hyperglycemic region, the insulin dose is slightly off. But a large glucose error (say a high hyperglycemic reading when truth is hypoglycemic) is life threatening.

This representation exists for many industries (say the fit of a car door, for example).

What’s the point?

In clinical chemistry, specifications from standard bodies (example glucose meter spec ISO 15197) use the manufacturer spec, not the consumer spec. But there are ways to express the consumer (laboratory) viewpoint, by using an error grid. For glucose meters, error grids exist but they are not part of ISO 15197.

Reminds me of the Bluto clip “Seven years of college down the drain” So what happened?

A previous president of AACC explained his reject vote of EP21 …

“Pre-and post analytic error is not measurable in a reproducible manner. It will vary by location (ED vs. clinic, ICU private office etc.), personnel obtaining samples, reporting results.  It will vary depending on the LIS/interface of the institution, distance from the lab that the blood is drawn and then transported to, etc. The lab and manufacturers have very little control over this.  Bias and imprecision can be measured quite well and then total analytic error can be estimated.”

This person either doesn’t understand EP21 or pretends not to understand EP21.

EP21 does not attempt to assess error sources.
EP21 does not combine error sources such as implied in the last sentence of the commentator.
EP21 simply measures differences from the comparative assay as total error.

Pre-analytical error is part of what is measured in an EP21 experiment in certain cases. Some examples:

Air bubbles allowed in a syringe of arterial blood contaminate the pO2 measurement with room air.
A tacrolimus assay involves a manual pretreatment step that if performed incorrectly can affect results.

Moreover, EP21 is an experiment. No results are reported to clinicians. Patient acquisition, use of the LIS, and reporting of results will not be conducted according to the routine methods of the laboratory. So any errors that are part of the experiment and not representative of routine use should be excluded. Also, many pre-analytical errors that are important in routine use will be nullified by the fact that this is an experiment. Example: pre-analytical error of sample preparation is likely to be nullified if the experiment is a split sample design.

So why mention pre-analytical error at all?

Because there is the opportunity to introduce bias in the experiment by eliminating the opportunity for pre-analytical error to occur.

Example: compare split sample venous blood instead of a finger stick sample to venous blood for assays that routinely use finger stick and whose comparative assays use venous blood. This would be biased because one has excluded the possibility for error to occur from a finger stick.

So the commentator’s argument against mentioning pre-analytical error doesn’t seem like a valid reason to reject the document.

I became aware of an article that discusses the crash of Air France 447. Since airline safety is always used as a reference for patient safety, one can ask what one can learn from this event. According to the article and comments to the Wachter blog, pilot error – specifically panic and decision paralysis in the face of stress were important factors (formally the Yerkes-Dodson law). Now, clinical chemists don’t face the acute stress that results from threats to personal survival – such as loss of an airspeed indicator at 37,000 feet. However, there is a less acute form of stress that affects clinical chemists – the pressure to release products. This can lead to products that harm patients. When bad decisions are made under the pressure of this stress, one can’t blame “management” (the source of the stress). It’s up to each individual to make the right decision.

Lack of management support – People sense this in spite of what management might say and the result is less participation.

Low interest level – My experience with engineers is that they like to design things. Meetings to challenge the design do not command their attention. Maybe things are different in labs.

Poor facilitator – FMEA meetings require a leader. If the leadership is poor, the results will suffer.

Wrong level of granularity – The level of detail in examining process steps could either be not detailed enough or too detailed.

Process is challenged – The person who developed parts of the process may be in the FMEA meeting, which can create an adversarial situation.

The existing process is simply documented – but it is never challenged.

Meeting is too adversarial – A legitimate challenge to the process can become adversarial.

The FMEA process is seen as being imposed from outside of the department (which often is true).

Good results are uninspiring – To say, “You know that event that could kill patients that has never happened? We’ve just changed our process to make that event even less likely.” Not a very compelling thing for management to hear.

Driving in my car, I happened to hear Donald Berwick interviewed on NPR. Dr. Berwick is well known for quality improvement in healthcare – my previous and somewhat heretical observations are recounted and amplified here.

In, 2003, Dr. Berwick had an editorial in the Washington Post (July 29, 2003) and New England Journal of Medicine (2003;348:2570-2572) which I disagreed with in part, because it was too dismissive of root cause analysis. My response was: There is nothing wrong with the concept of a root cause. Int J Qual Health Care 2004;16:263.

I also objected that Dr. Berwick referred to one of his coworkers as “Tom Nolan, one of the leading quality-improvement scholars ofour time.” Now as for leading quality improvement scholars, I’ve heard of Deming, Juran, Crosby, and Taguchi but until mentioned by Dr. Berwick, I hadn’t heard of Tom Nolan. Scholars should have a body of work. One can view Nolan’s. It seemed to me that Nolan had been anointed his status by Dr. Berwick.

I also disagreed with how FMEA (Failure Mode Effects Analysis) was (and is) conducted at the IHI (Institute for Healthcare Improvement). Besides the classification columns, IHI has three columns: failure mode, causes, and effects. These columns are nested within “steps.” But these steps are not individual process steps. In the IHI example case of chemotherapy medication, the step “administration” implies many separate process steps but none are spelled out. A real FMEA starts with a process map which leads to a series of individual process steps. The process steps are the starting point of a FMEA and each process step can fail in different ways. Process steps can be procedure steps (process blood), detection steps (check for hemolysis), and recovery steps (request new sample). Classification should be only for severity and probability of occurrence but IHI also includes detection. What does detection of a detection process step mean? I don’t understand why people are trying to oversimplify a traditional FMEA (occurs in CLSI EP23 too). This oversimplification makes things more complicated.

In addition, there is the following problem. In a traditional FMEA, classification scores are obtained by multiplying severity and probability of occurrence and the scores are totaled across all events. IHI wants to see the total score reduced – this is the stated aim of the IHI example. Now the only way to do this is to lower probability of occurrence. While this sounds reasonable, most severe events already have probability at the lowest level, meaning that focus in IHI FMEAs will be on less severe events. This is somewhat clouded when you throw in detection as a classification but it is nevertheless a serious problem.

So my extremely limited experience with the execution of quality improvements efforts championed by Berwick is not good, but he is a charismatic speaker.

One of the key questions asked for a diagnostic assay is “is the performance good enough?” This question takes on forms such as: what should the performance goals be? How should they be evaluated? From a regulator’s point of view, there are two decisions that can be made:

Accept the assay – with the risk that patients may be harmed due to assay error
Reject the assay – with the risk that patients may be harmed due to the lack of information that would have been obtained by the assay

Now for any good assay, patients harmed from assay error is always extremely low (e.g., self justifying because it’s a good assay).

And a good assay provides important information to a clinician (e.g., self justifying because it’s a good assay). This also means that the lack of information from a rejected assay would likely cause great harm.

Hence the rational decision is always to approve an assay. Why doesn’t the FDA always approve assays? Perhaps because assays that harm patients are like a plane crash and no one likes plane crashes.

Small planes are identified by their tail numbers which in the US are “N” numbers. IMHO, there are good and bad N numbers with the good N numbers being easy to say and the bad N numbers being tongue twisters. I fly two planes N13151 and N2081S (Cessna two zero eight one Sierra) or (Skyhawk two zero eight one Sierra). I have trouble with this N number and stumble over it all the time. Talking to others, they have trouble with certain N numbers as well. This included some air traffic controllers during a recent visit to a control tower – they told me the N numbers that give them trouble.

My publications come in bunches. These three were just published.

Jan S. Krouwer: A Widespread Myth About Point-of-Care Devices Point of Care: The Journal of Near-Patient Testing & Technology 2011;10:146-147.

Jan S. Krouwer: Analysis of the Performance of the OneTouch SelectSimple Blood Glucose Monitoring System: Why Ease of Use Studies Need to Be Part of Accuracy Studies. Journal of Diabetes Science and Technology, 2011;5:610-611.

Jan S. Krouwer: Evaluation of the Analytical Performance of the Coulometry-Based Optium Omega Blood Glucose Meter: What Do Such Evaluations Show? Journal of Diabetes Science and Technology, 2011;5:618-620.

Additionally, I was interviewed for an article about risk management in Clinical Laboraotry News: http://www.aacc.org/publications/cln/2011/november/Pages/ANewApproachtoQualityControl.aspx?

The USPT has concluded that men should not be screened for prostate cancer with a PSA assay. In a blog that I follow, DB concurred with this recommendation. In his post, he gave a bunch of opinions and then said, “when one examines the data…” he agreed with the USPT recommendation.

I contend that to examine the data, means to read every word of the ERSPC and PLCO studies, question every assumption, every data analysis method, every result, every conclusion. This is examining the data although it does not mean that one will disagree. I have not examined the data although I remember hearing that a significant percentage of the control group was contaminated because they were screened for PSA.

Here are some possibilities for men who die of prostate cancer with the assumption that incurable prostate cancer is associated with cells that have escaped the prostate.

1. Cells have escaped the prostate before any currently available detection method detects cancer, rendering current treatments (surgery, radiation) ineffective because the escaped cells are not treated.
2. The current treatments are occasionally ineffective (cells are contained within the prostate but not all cancer cells are killed).
3. The cancer is not detected until cells escape from the prostate, because no detection methods are used, especially no PSA test is performed.
4. The cancer is not detected until cells escape from the prostate, because the detection methods generate false negatives.

Note: prostate is often compared to a walnut which is the picture.

The FDA reported that over the years 1997-2009, 13 people died due to maltose interference in glucose meters. To recall, maltose can cause large increases in a glucose reading which could lead to hypoglycemic individuals being treated with insulin.

Based on an FDA meeting, Dr. Shuren reported that there are 24 million American who have diabetes and of these, 30% (7.2 million) take insulin. Assume that each person tests 3 times per day on average. This means that over the 12 year period, there were about 94.6 billion tests.

The failure rate due to the maltose problem which caused deaths is 13/94.6 billion or 0.00000001374%. Suffice it to say that this is a small number. Were the death rate underreported by 100 fold, the percentage would still be tiny and 0.000001374%. Of course, tiny would not be any consolation if the event happened to a loved one.

So when a laboratory performs a 100 sample method comparison and finds no values in the higher regions of an error grid, it doesn’t mean that the assay has been proven to be safe. What can be done is to apply risk management studies (Failure Mode Effects Analysis, fault trees) in addition to a method comparison.