Glucose meter QC – too much?

November 13, 2016

dsc_1445edp

A colleague has been corresponding with me about glucose meter QC and recently sent me this paper. Basically, the authors are concerned about the high cost of QC for glucose meters and have data in their paper to show that their glucose meters are very reliable.

Actually, they state their meters are better than highly reliable because to quote from their paper: “no meter failure is detected by current QC testing procedure”. Well, not so fast. In some cases a repeated QC failure was corrected by using a new vial of strips. To me, this indicates a basic misunderstanding of glucose meters. One can think of the glucose meter testing process as having three components:

 

  1. The user – who must correctly use the meter
  2. The reagent strip – this is where the chemistry occurs
  3. The meter – hardware and software with the outcome being a glucose result

 

It seems as if these authors consider the meter as the item to be controlled. Yet it is highly unlikely that the meter could provide incorrect results – certainly no results if a meter’s hardware failed. But the reagent strip is where the action occurs and a bad strip could cause wrong answers, so in the authors’ study, QC did detect bad strips and presumably prevented wrong results.

I will comment at a later date about QC and user error.

What if the authors had shown no failures due to QC. Does that justify reducing QC to perhaps a monthly (as suggested by CMS) or less frequency? Cost is an important issue. But the purpose of QC is to detect errors. QC is not useless if no errors are found.

The purpose of QC is to detect errors to prevent an incorrect result to be reported. This is to prevent a clinician from making the wrong medical decision – based on test error, which causes harm to the patient. Hence, an assumption is that a correct result is needed to prevent patient harm. (If this is not the case, then one can argue that no QC is needed nor is the test needed in the first place).

But the frequency of QC actually detecting errors is not important as long as it can been shown that QC can detect errors. If the system is reliable, the error rate will be low.

The message is one would never remove safeguards just because of low error rates. For example, in hospitals and nuclear power plants, monitoring for radiation is a QC like practice and costs money. The frequency of problems is not relevant.

Advertisements

Westgards Detection and IQCP

November 1, 2016

dsc02015edp

I received an email recently that alerted me to three seminars from the 2016 AACC meeting that are online. One is by the Westgards, so I had a look. This is quite an interesting presentation and shows the breadth of the contributions that the Westgards have made to quality in laboratory medicine.

Yet, one thing caught my eye and so here are my comments. Thus, the Westgards complain that in risk management as espoused by CLSI EP23, detectability has been omitted.

What they mean is that for each failure event, EP23 wants one to estimate the severity and probability of occurrence of that failure event. The Westgards suggest that the detectability of the failure event needs to be assessed as well and state that this is how industry does it.

Well maybe some industries, but I worked in industry and our company did not use detectability (we used severity and probability of occurrence).

Now in the context of EP23, I agree with the Westgards use of detectability. The problem is that EP23 itself is a poor adaptation of risk management. I commented on this before but here it is again.

As an example of a failure mode of a process step, assume that the failure is sample hemolysis which occurs during the process step to turn a whole blood sample into serum. As you go across the rows in an EP23 style risk analysis, you might see that a mitigation for this failure mode is to visually check whether the sample has been hemolyzed and how effective this check is. In this case – for this row item – you could add detectability to severity and probability of occurrence.

Here are the problems with this approach, whether you have added detectability or not.

For most labs, this (example) is already established laboratory practice. That is, labs already check to see whether samples are hemolyzed. All that has been done is to document it. Not much in the way of formal risk analysis has been done although there will be some benefit to this documentation.

The problem is that the row is “collapsed.” It really has two additional process steps embedded in it. Here it is uncollapsed:

Process step – process whole blood into serum
Detection step – examine serum for the possibility of hemolysis
Recovery step – if the serum has been hemolyzed, request a new sample

One can see that it makes no sense to ask for the detectability of a detection step.

I note in passing that one of the most important detection process steps for any assay is running quality control.

Note that each of these steps above are process steps and each can fail. Whereas the severity will be the same for the failure for each of these steps, the probability of occurrence may differ. Because each step can fail, one needs to assess whether a mitigation step is required.

BTW, one should not discount failures in the recovery step. In the Challenger accident, engineers warned about the potential problem (detection) but delaying the launch failed (recovery). And of course, recovery steps are only performed if detection steps detect something.

Disclaimer – I may not have the latest version of EP23, but another problem in developing the potential failure modes (EP23 call these hazards) is that the process is not fully delineated – it is too high level. In a more traditional FMEA, the list of process steps is long and reflects what is actually done, not some high level description.

And each process step can fail in multiple ways. EP23 is a hazard based list. A process based list is better since one can ask how else each process step can fail. Although EP23 does some of this, it’s embedded within a row and makes things too complicated. Here’s an example of a few ways the above detection step of examining the sample for hemolysis can fail:

  1. Technician simply misses seeing a hemolyzed sample (non-cognitive error – we all make them)
  2. Technician misses seeing a sample due to inadequate training (cognitive error)
  3. Technician misses seeing a sample due to distraction (phone call, or talking to colleague).
  4. Technician ignores hemolyzed sample due to pressure from management to process samples.

On a separate note, how does IQCP help one modify QC frequency?


IQCP – waste of time? No surprise

July 30, 2016

DSC01569edp

Having looked at a blog entry by the Westgards, which is always interesting, here are my thoughts.

Regarding IQCP, they say it’s mostly been a “waste of time”, an exercise of paperwork to justify current practices, with very little change occurring in QC practices.

This is no surprise to me – here’s why.

There are two ways to reduce errors.

FMEA (or similar programs) reduces the likelihood of rare but severe errors.

FRACAS (or similar programs) reduces the error rate of actual errors, some of which may be severe.

Here are the challenges with FMEA

  1. It takes time and personnel. There’s no way around this. If sufficient time is not provided with all of the relevant personnel present, the results will suffer. When the Joint Commission required every hospital to perform at least one FMEA per year, people complained that performing a FMEA took too much time.
  2. Management must be committed. (I was asked to facilitate a FMEA for a company – the meetings were scheduled during lunch. I asked why and was told they had more important things to do). Management wasn’t committed. The only reason this group was doing the FMEA was to satisfy a requirement.
  3. FMEA requires a facilitator. The purpose of FMEA is to challenge the ways things are done. Often, this means challenging people in the room (e.g., who have put systems in place or manage the ways things are done). This can create an adversarial situation where subordinates will not speak up. Without a good facilitator, results will suffer.
  4. The guidance to perform a FMEA (such as EP23) is not very good. Example: Failure mode is a short sample. The mitigation is to have someone examine each tube to ensure the sample volume is adequate. The group moves on to the next failure mode. The problem is that the mitigation is not new – it’s existing laboratory practice. Thus, as the Westgards say – all that has happened is the existing process has been documented. That is not FMEA. (A FMEA would enumerate the many ways that someone examining each sample could fail to detect the short sample).
  5. Pareto charts are absent in the guidance. But real FMEAs require Pareto charts.
  6. I have seen reports where people say their error rate has been reduced after they conducted a FMEA. But there are no error rates in a FMEA (errors rates are in a FRACAS). So this means no FMEA was carried out.
  7. And how anyone could say they have conducted a FMEA and conclude that it is ok to run QC monthly.

Here are the challenges with FRACAS

  1. FRACAS requires a process where errors are counted in a structured way (severity and frequency) and reports issued on a periodic basis. This requires knowledge and commitment.
  2. FRACAS also requires periodic meetings to review errors whereby problems are assigned to corrective action teams. Again, this requires knowledge and commitment.
  3. Absence of a Pareto chart is a flag that something is missing (no severity classification, for example).
  4. People don’t like to see their error rates.
  5. FRACAS requires a realistic (error rate) goal.

There are FRACAS success stories:

Dr. Peter Pronovost performed a FRACAS type approach on placing central lines and dropped the infection rate from 10% to 0 by the use of checklists.

In the 70s, the use of a FRACAS type approach reduced the error rate in anesthesiology instruments.

And FMEA failures

A Mexican teenager came to the US for a heart lung transplant. The donated organs were not checked to see if they were the right type. The patient died.


Unwarranted Conclusions

June 2, 2016

DSC_1509edp

Looking at a paper about QC procedures (subscription required), I admit I was intrigued by the title: “Selecting Statistical Procedures for Quality Control Planning Based on Risk Management.”

Just reading the abstract and the first few lines informs me that the conclusions are unwarranted because the authors claim, they can estimate the probability of patient harm based on which QC procedure is chosen.

A QC procedure helps to detect problems with the assay process. Patient harm can be caused by an assay process gone astray but it can also be caused by things with an assay process that has not gone astray. For example, a patient interference can cause patient harm and will not be detected by QC. Moreover, the authors assume that an out of control condition will occur in a constant fashion until it is detected by the next QC sample, but a shift in results that occurs for a limited number of samples can occur and is eliminated from consideration. So even QC considerations don’t include all possible errors.

Ok, I admit that I have stopped reading but it is clear that whatever the authors estimate (assuming their logic is correct) is an underestimate of the probability of patient harm.

That also makes me wonder, of all cases of patient harm caused by wrong medical decisions caused by assay error, what percentage are due to the assay process gone bad vs. other causes (e.g., interferences). For example searching for the word “interference” in the title of Clinical Chemistry over the last 10 years yielded 912 results.


IQCP – It’s about the money

April 22, 2016

44NedpT

There is an article in CAP Today about IQCP. I was struck by a quote in the beginning of the article:

“I didn’t stop to calculate what it would cost to do liquid quality control on all the i-Stat cartridge types every eight hours because the number would have been through the roof”

Now I understand that cost is a real issue, but so is harm to patients.

The original idea of EQC (equivalent quality control) was to reduce the frequency of QC if you did an experiment that showed good QC for 10 days. This was of course without merit with the potential to cause patient harm.

The current notion of IQCP is to perform risk analysis and reduce the frequency of QC. This also makes no sense. Risk analysis should always be performed and so should QC, at a frequency which allows the repeat of questionable results such that patients will not be harmed.


More on Antwerp, EFLM and pre-analytical error

April 14, 2016

DSC_1415edp

One of the talks in the Antwerp conference referred to an EFLM working group to come up with performance specifications for pre-analytical error. There is a talk on the EFLM website about this.

The recognition that pre-analytical error is a big part of ensuring the quality of laboratory tests is of course important; however, it’s hard to see how separate performance specifications for pre-analytical error (e.g., separate from analytical error) can be useful. [Actually, the presenter from the Antwerp conference agreed with my skepticism during a break.]

Pre-analytical error can be classified into three types

Type 1 – An error that is completely independent of the analytical process. Example: failure to wash the site that is sampled for a glucose meter test. If the site is contaminated with glucose, any glucose meter will report an elevated (and erroneous) result.

Type 2 – An error that is partly dependent on the analytical process. Example: a short sample for a glucose meter test that has an algorithm in the meter to detect short samples. If the algorithm is defective (an analytical error) and there is a short sample (a pre-analytical error), the glucose result may be erroneous.

Type 3 – A pre-analytical error that is indistinguishable from the analytical process. Example: air bubbles in a pO2 blood gas syringe. No matter who is performing the test, there is the possibility of having bubbles in the sample, a pre-analytical error which can cause an erroneous result.

 

One of the problems is that a typical evaluation will attempt to meet (analytical) performance specifications with type 1 and type 2 errors excluded from the evaluation. This is of course recognized by this EFLM group, hence their task. I note in passing that when type 3 errors occur, the performance evaluations include such pre-analytical errors even when trying not to (by excluding the possibility of type 1 and type 2 errors).

One fear is that the EFLM group will come up a bunch of separate performance specifications for pre-analytical error, independent of specifications for analytical error. I don’t see how this can work.

What would I do? I would use a reliability growth metric, which counts all errors (regardless of source) – see this paper.

Finally, where I wrote pre-analytical error, it should be pre- and post-analytical error.


High risk projects – a good thing in moderation

September 10, 2015

KLEWedp

I was cleaning out some old files and came across a strategy document. The strategy happened when our company had been bought and a consulting company was there to assess the various groups. The summary for our group was that we had lost focus and would be better served if we stuck to our core competence of evaluations and reliability. Evaluations meant product performance evaluations and statistical analyses related to getting a product approved by the FDA and reliability meant improving the reliability of instrument systems under development through the use of data analysis methods.

This was pretty standard stuff back then. When a company was bought, you brought in a consulting company started by Harvard Business School professors (this one was called Integral) to suggest a bunch of changes.

The problem was that our original core competence was just evaluations. The reliability component had been started by our group as a means of expanding our services where we thought we could provide value. In fact, our reliability services had become our most valuable service. Had this consulting company been brought in when our reliability program was in its infancy, undoubtedly, we would have been told to drop it and stick to evaluations.

The point is that most organizations look to grow and the attempts to grow involve risk. Some projects succeed – most fail. Our company back then tried to expand in other areas – pulmonary function, pulse oximetry, coagulation: all failed. One other project – the automation of an immunoassay analyzer – the ACS 180 – was a huge success.

So some efforts should be devoted to high risk high reward projects.