Blog Review

May 26, 2017

I started this blog 13 years ago in March 2004 – the first two articles are about six sigma, here and here. The blog entry being posted now is my 344th blog entry.

Although the blog has an eclectic range of topics, one unifying theme for many entries is specifications, how to set them and how to evaluate them.

A few years ago, I was working on a hematology analyzer, which has a multitude of reported parameters. The company was evaluating parameters with the usual means of precision studies and accuracy using regression. I asked them:

  1. a) what are the limits that, when differences from reference are contained within these limits, will ensure that no wrong medical decisions would be made based on the reported result (resulting in patient harm) and
  2. b) what are the (wider) limits that, when differences from reference are contained within these limits, will ensure that no wrong medical decisions would be made based on the reported result (resulting in severe patient harm)

This was a way of asking for an error grid for each parameter. I believe, then and now, that constructing an error grid is the best way to set specifications for any assay.

As an example about the importance of specifications there was a case for which I was an expert witness whereby the lab had produced an incorrect result that led to patent harm. The lab’s defense was that they had followed all procedures. Thus, as long as they as followed procedures, they were not to blame. But procedures, which contain specifications, are not always adequate. As an example, remember the CMS program “equivalent quality control”?


Glucose meter QC – too much?

November 13, 2016

dsc_1445edp

A colleague has been corresponding with me about glucose meter QC and recently sent me this paper. Basically, the authors are concerned about the high cost of QC for glucose meters and have data in their paper to show that their glucose meters are very reliable.

Actually, they state their meters are better than highly reliable because to quote from their paper: “no meter failure is detected by current QC testing procedure”. Well, not so fast. In some cases a repeated QC failure was corrected by using a new vial of strips. To me, this indicates a basic misunderstanding of glucose meters. One can think of the glucose meter testing process as having three components:

 

  1. The user – who must correctly use the meter
  2. The reagent strip – this is where the chemistry occurs
  3. The meter – hardware and software with the outcome being a glucose result

 

It seems as if these authors consider the meter as the item to be controlled. Yet it is highly unlikely that the meter could provide incorrect results – certainly no results if a meter’s hardware failed. But the reagent strip is where the action occurs and a bad strip could cause wrong answers, so in the authors’ study, QC did detect bad strips and presumably prevented wrong results.

I will comment at a later date about QC and user error.

What if the authors had shown no failures due to QC. Does that justify reducing QC to perhaps a monthly (as suggested by CMS) or less frequency? Cost is an important issue. But the purpose of QC is to detect errors. QC is not useless if no errors are found.

The purpose of QC is to detect errors to prevent an incorrect result to be reported. This is to prevent a clinician from making the wrong medical decision – based on test error, which causes harm to the patient. Hence, an assumption is that a correct result is needed to prevent patient harm. (If this is not the case, then one can argue that no QC is needed nor is the test needed in the first place).

But the frequency of QC actually detecting errors is not important as long as it can been shown that QC can detect errors. If the system is reliable, the error rate will be low.

The message is one would never remove safeguards just because of low error rates. For example, in hospitals and nuclear power plants, monitoring for radiation is a QC like practice and costs money. The frequency of problems is not relevant.


Westgards Detection and IQCP

November 1, 2016

dsc02015edp

I received an email recently that alerted me to three seminars from the 2016 AACC meeting that are online. One is by the Westgards, so I had a look. This is quite an interesting presentation and shows the breadth of the contributions that the Westgards have made to quality in laboratory medicine.

Yet, one thing caught my eye and so here are my comments. Thus, the Westgards complain that in risk management as espoused by CLSI EP23, detectability has been omitted.

What they mean is that for each failure event, EP23 wants one to estimate the severity and probability of occurrence of that failure event. The Westgards suggest that the detectability of the failure event needs to be assessed as well and state that this is how industry does it.

Well maybe some industries, but I worked in industry and our company did not use detectability (we used severity and probability of occurrence).

Now in the context of EP23, I agree with the Westgards use of detectability. The problem is that EP23 itself is a poor adaptation of risk management. I commented on this before but here it is again.

As an example of a failure mode of a process step, assume that the failure is sample hemolysis which occurs during the process step to turn a whole blood sample into serum. As you go across the rows in an EP23 style risk analysis, you might see that a mitigation for this failure mode is to visually check whether the sample has been hemolyzed and how effective this check is. In this case – for this row item – you could add detectability to severity and probability of occurrence.

Here are the problems with this approach, whether you have added detectability or not.

For most labs, this (example) is already established laboratory practice. That is, labs already check to see whether samples are hemolyzed. All that has been done is to document it. Not much in the way of formal risk analysis has been done although there will be some benefit to this documentation.

The problem is that the row is “collapsed.” It really has two additional process steps embedded in it. Here it is uncollapsed:

Process step – process whole blood into serum
Detection step – examine serum for the possibility of hemolysis
Recovery step – if the serum has been hemolyzed, request a new sample

One can see that it makes no sense to ask for the detectability of a detection step.

I note in passing that one of the most important detection process steps for any assay is running quality control.

Note that each of these steps above are process steps and each can fail. Whereas the severity will be the same for the failure for each of these steps, the probability of occurrence may differ. Because each step can fail, one needs to assess whether a mitigation step is required.

BTW, one should not discount failures in the recovery step. In the Challenger accident, engineers warned about the potential problem (detection) but delaying the launch failed (recovery). And of course, recovery steps are only performed if detection steps detect something.

Disclaimer – I may not have the latest version of EP23, but another problem in developing the potential failure modes (EP23 call these hazards) is that the process is not fully delineated – it is too high level. In a more traditional FMEA, the list of process steps is long and reflects what is actually done, not some high level description.

And each process step can fail in multiple ways. EP23 is a hazard based list. A process based list is better since one can ask how else each process step can fail. Although EP23 does some of this, it’s embedded within a row and makes things too complicated. Here’s an example of a few ways the above detection step of examining the sample for hemolysis can fail:

  1. Technician simply misses seeing a hemolyzed sample (non-cognitive error – we all make them)
  2. Technician misses seeing a sample due to inadequate training (cognitive error)
  3. Technician misses seeing a sample due to distraction (phone call, or talking to colleague).
  4. Technician ignores hemolyzed sample due to pressure from management to process samples.

On a separate note, how does IQCP help one modify QC frequency?


I’m impressed

August 26, 2016

DSC01813edpT

In my last blog entry, I noted that the problem with patent based QC is that publications talked about its benefits using simulations, but no one was actually using patient based QC. Well, the article that was upcoming has come out and these authors are using patient based QC and provide an example where a shift in sodium values was detected. So I’m impressed.


The problem with patient based QC

August 18, 2016

CPPP.edp

In an editorial (actually more of a mini review) in Clinical Chemistry, the various patient based QC methods are reviewed. The editorial is provided because of a companion article that has yet to appear.

One problem with patient based QC is that it is always compared to traditional QC, perhaps with the goal that it could replace traditional QC. But why not do both.

And perhaps a bigger problem is that people study patient based QC by performing simulations and/or providing examples showing that retrospective analysis of a (known) problem (perhaps a clinician complaint) would have been detected by patient based QC.

But I am not aware of anyone routinely using patient based QC (with or without traditional based QC), for all assays.


More AACC 2016 Philadelphia Notes

August 4, 2016

aacc

As for the AACC 2016 app, it deconstructs the three program books into something pretty useless. With the physical books, one can page through them rather quickly. But with the app, most of the titles are cut off, so it takes forever to find things.

The posters were so far away, they seemed to be in a different zip code.

Several posters were interesting and I was impressed by a poster presented by Linda AC De Grande about the use of patient medians. Maybe one day QC using patients will be mainstream.

Although I’m no longer part of CLSI, I have attended CLSI meetings at the AACC national meeting since the 80s. But these meetings are not held any more at the national AACC meeting. This makes CLSI less inclusive since people can no longer simply drop in on the meetings.

Anyone who stayed at the Marriott (like me) was very happy since the convention center was a block away.

At a Siemens presentation about IQCP, it was implied that conducting IQCP might allow one to run QC less than once a month as long as there was no conflict with manufacturer’s IFU. Not sure if the presenter was correct but a scary thought nevertheless.


IQCP – waste of time? No surprise

July 30, 2016

DSC01569edp

Having looked at a blog entry by the Westgards, which is always interesting, here are my thoughts.

Regarding IQCP, they say it’s mostly been a “waste of time”, an exercise of paperwork to justify current practices, with very little change occurring in QC practices.

This is no surprise to me – here’s why.

There are two ways to reduce errors.

FMEA (or similar programs) reduces the likelihood of rare but severe errors.

FRACAS (or similar programs) reduces the error rate of actual errors, some of which may be severe.

Here are the challenges with FMEA

  1. It takes time and personnel. There’s no way around this. If sufficient time is not provided with all of the relevant personnel present, the results will suffer. When the Joint Commission required every hospital to perform at least one FMEA per year, people complained that performing a FMEA took too much time.
  2. Management must be committed. (I was asked to facilitate a FMEA for a company – the meetings were scheduled during lunch. I asked why and was told they had more important things to do). Management wasn’t committed. The only reason this group was doing the FMEA was to satisfy a requirement.
  3. FMEA requires a facilitator. The purpose of FMEA is to challenge the ways things are done. Often, this means challenging people in the room (e.g., who have put systems in place or manage the ways things are done). This can create an adversarial situation where subordinates will not speak up. Without a good facilitator, results will suffer.
  4. The guidance to perform a FMEA (such as EP23) is not very good. Example: Failure mode is a short sample. The mitigation is to have someone examine each tube to ensure the sample volume is adequate. The group moves on to the next failure mode. The problem is that the mitigation is not new – it’s existing laboratory practice. Thus, as the Westgards say – all that has happened is the existing process has been documented. That is not FMEA. (A FMEA would enumerate the many ways that someone examining each sample could fail to detect the short sample).
  5. Pareto charts are absent in the guidance. But real FMEAs require Pareto charts.
  6. I have seen reports where people say their error rate has been reduced after they conducted a FMEA. But there are no error rates in a FMEA (errors rates are in a FRACAS). So this means no FMEA was carried out.
  7. And how anyone could say they have conducted a FMEA and conclude that it is ok to run QC monthly.

Here are the challenges with FRACAS

  1. FRACAS requires a process where errors are counted in a structured way (severity and frequency) and reports issued on a periodic basis. This requires knowledge and commitment.
  2. FRACAS also requires periodic meetings to review errors whereby problems are assigned to corrective action teams. Again, this requires knowledge and commitment.
  3. Absence of a Pareto chart is a flag that something is missing (no severity classification, for example).
  4. People don’t like to see their error rates.
  5. FRACAS requires a realistic (error rate) goal.

There are FRACAS success stories:

Dr. Peter Pronovost performed a FRACAS type approach on placing central lines and dropped the infection rate from 10% to 0 by the use of checklists.

In the 70s, the use of a FRACAS type approach reduced the error rate in anesthesiology instruments.

And FMEA failures

A Mexican teenager came to the US for a heart lung transplant. The donated organs were not checked to see if they were the right type. The patient died.