Risk based SQC – What does it really mean

December 4, 2017

Having just read a paper on risk based SQC, here are my thoughts…

CLSI has recently adopted a risk management theme for some of their standards. The fact that Westgard has jumped on the risk management bandwagon is as we say in Boston, wicked smaaht.

But what does this really mean and is it useful?

SQC as described in the Westgard paper is performed to prevent patient results from exceeding an allowable total error (TEa). To recall, TEa = |bias|/SD*1.65. I have previously commented that this model does not account for all error sources, especially for QC samples. But for the moment, let’s assume that the only error sources are average bias and imprecision. The remaining problem with TEa is that it is always given as a percentage of results, usually 95%. So if some SQC procedure were to just meet their quality requirement, up to 5% of patient results could exceed their TEa and potentially cause medical errors. This is 1 in every 20 results! I don’t see how this is a good thing even if one were to use a 99% TEa.

The problem is one of “spin.” SQC, while valuable, does not guarantee the quality of patient results. The laboratory testing process is like a factory process and with any such process, to be useful it must be in control (meaning in statistical quality control). Thus, SQC helps to guard against an out of control process. To be fair, if the process were out of control, patient sample results might exceed TEa.

The actual risk of medical errors due to lab error is a function not only of an out of control process but also due to all other error sources not accounted for by QC, such as user errors with patient samples (as opposed to QC samples), patient interferences, and so on. Hence, to say that risk based SQC can address the quality of patient results is “spin.” SQC is a process control tool – nothing more and nothing less.

And the best way of running SQC would be for a manufacturer to assess results from all laboratories.

Now some people might think, this is a nit-piking post but here is an additional point. One might be lulled into thinking that with this risk based SQC that labs don’t have to worry about bad results. But interferences can cause large errors that can cause medical errors. For example, in the maltose problem for glucose meters, 6 of 13 deaths occurred after an FDA warning. And recently, there have been concerns about biotin interference in immunoassays. So it’s not good to oversell SQC, since people might loose focus on other, important issues.

Advertisements

Blog Review

May 26, 2017

I started this blog 13 years ago in March 2004 – the first two articles are about six sigma, here and here. The blog entry being posted now is my 344th blog entry.

Although the blog has an eclectic range of topics, one unifying theme for many entries is specifications, how to set them and how to evaluate them.

A few years ago, I was working on a hematology analyzer, which has a multitude of reported parameters. The company was evaluating parameters with the usual means of precision studies and accuracy using regression. I asked them:

  1. a) what are the limits that, when differences from reference are contained within these limits, will ensure that no wrong medical decisions would be made based on the reported result (resulting in patient harm) and
  2. b) what are the (wider) limits that, when differences from reference are contained within these limits, will ensure that no wrong medical decisions would be made based on the reported result (resulting in severe patient harm)

This was a way of asking for an error grid for each parameter. I believe, then and now, that constructing an error grid is the best way to set specifications for any assay.

As an example about the importance of specifications there was a case for which I was an expert witness whereby the lab had produced an incorrect result that led to patent harm. The lab’s defense was that they had followed all procedures. Thus, as long as they as followed procedures, they were not to blame. But procedures, which contain specifications, are not always adequate. As an example, remember the CMS program “equivalent quality control”?


Glucose meter QC – too much?

November 13, 2016

dsc_1445edp

A colleague has been corresponding with me about glucose meter QC and recently sent me this paper. Basically, the authors are concerned about the high cost of QC for glucose meters and have data in their paper to show that their glucose meters are very reliable.

Actually, they state their meters are better than highly reliable because to quote from their paper: “no meter failure is detected by current QC testing procedure”. Well, not so fast. In some cases a repeated QC failure was corrected by using a new vial of strips. To me, this indicates a basic misunderstanding of glucose meters. One can think of the glucose meter testing process as having three components:

 

  1. The user – who must correctly use the meter
  2. The reagent strip – this is where the chemistry occurs
  3. The meter – hardware and software with the outcome being a glucose result

 

It seems as if these authors consider the meter as the item to be controlled. Yet it is highly unlikely that the meter could provide incorrect results – certainly no results if a meter’s hardware failed. But the reagent strip is where the action occurs and a bad strip could cause wrong answers, so in the authors’ study, QC did detect bad strips and presumably prevented wrong results.

I will comment at a later date about QC and user error.

What if the authors had shown no failures due to QC. Does that justify reducing QC to perhaps a monthly (as suggested by CMS) or less frequency? Cost is an important issue. But the purpose of QC is to detect errors. QC is not useless if no errors are found.

The purpose of QC is to detect errors to prevent an incorrect result to be reported. This is to prevent a clinician from making the wrong medical decision – based on test error, which causes harm to the patient. Hence, an assumption is that a correct result is needed to prevent patient harm. (If this is not the case, then one can argue that no QC is needed nor is the test needed in the first place).

But the frequency of QC actually detecting errors is not important as long as it can been shown that QC can detect errors. If the system is reliable, the error rate will be low.

The message is one would never remove safeguards just because of low error rates. For example, in hospitals and nuclear power plants, monitoring for radiation is a QC like practice and costs money. The frequency of problems is not relevant.


Westgards Detection and IQCP

November 1, 2016

dsc02015edp

I received an email recently that alerted me to three seminars from the 2016 AACC meeting that are online. One is by the Westgards, so I had a look. This is quite an interesting presentation and shows the breadth of the contributions that the Westgards have made to quality in laboratory medicine.

Yet, one thing caught my eye and so here are my comments. Thus, the Westgards complain that in risk management as espoused by CLSI EP23, detectability has been omitted.

What they mean is that for each failure event, EP23 wants one to estimate the severity and probability of occurrence of that failure event. The Westgards suggest that the detectability of the failure event needs to be assessed as well and state that this is how industry does it.

Well maybe some industries, but I worked in industry and our company did not use detectability (we used severity and probability of occurrence).

Now in the context of EP23, I agree with the Westgards use of detectability. The problem is that EP23 itself is a poor adaptation of risk management. I commented on this before but here it is again.

As an example of a failure mode of a process step, assume that the failure is sample hemolysis which occurs during the process step to turn a whole blood sample into serum. As you go across the rows in an EP23 style risk analysis, you might see that a mitigation for this failure mode is to visually check whether the sample has been hemolyzed and how effective this check is. In this case – for this row item – you could add detectability to severity and probability of occurrence.

Here are the problems with this approach, whether you have added detectability or not.

For most labs, this (example) is already established laboratory practice. That is, labs already check to see whether samples are hemolyzed. All that has been done is to document it. Not much in the way of formal risk analysis has been done although there will be some benefit to this documentation.

The problem is that the row is “collapsed.” It really has two additional process steps embedded in it. Here it is uncollapsed:

Process step – process whole blood into serum
Detection step – examine serum for the possibility of hemolysis
Recovery step – if the serum has been hemolyzed, request a new sample

One can see that it makes no sense to ask for the detectability of a detection step.

I note in passing that one of the most important detection process steps for any assay is running quality control.

Note that each of these steps above are process steps and each can fail. Whereas the severity will be the same for the failure for each of these steps, the probability of occurrence may differ. Because each step can fail, one needs to assess whether a mitigation step is required.

BTW, one should not discount failures in the recovery step. In the Challenger accident, engineers warned about the potential problem (detection) but delaying the launch failed (recovery). And of course, recovery steps are only performed if detection steps detect something.

Disclaimer – I may not have the latest version of EP23, but another problem in developing the potential failure modes (EP23 call these hazards) is that the process is not fully delineated – it is too high level. In a more traditional FMEA, the list of process steps is long and reflects what is actually done, not some high level description.

And each process step can fail in multiple ways. EP23 is a hazard based list. A process based list is better since one can ask how else each process step can fail. Although EP23 does some of this, it’s embedded within a row and makes things too complicated. Here’s an example of a few ways the above detection step of examining the sample for hemolysis can fail:

  1. Technician simply misses seeing a hemolyzed sample (non-cognitive error – we all make them)
  2. Technician misses seeing a sample due to inadequate training (cognitive error)
  3. Technician misses seeing a sample due to distraction (phone call, or talking to colleague).
  4. Technician ignores hemolyzed sample due to pressure from management to process samples.

On a separate note, how does IQCP help one modify QC frequency?


I’m impressed

August 26, 2016

DSC01813edpT

In my last blog entry, I noted that the problem with patent based QC is that publications talked about its benefits using simulations, but no one was actually using patient based QC. Well, the article that was upcoming has come out and these authors are using patient based QC and provide an example where a shift in sodium values was detected. So I’m impressed.


The problem with patient based QC

August 18, 2016

CPPP.edp

In an editorial (actually more of a mini review) in Clinical Chemistry, the various patient based QC methods are reviewed. The editorial is provided because of a companion article that has yet to appear.

One problem with patient based QC is that it is always compared to traditional QC, perhaps with the goal that it could replace traditional QC. But why not do both.

And perhaps a bigger problem is that people study patient based QC by performing simulations and/or providing examples showing that retrospective analysis of a (known) problem (perhaps a clinician complaint) would have been detected by patient based QC.

But I am not aware of anyone routinely using patient based QC (with or without traditional based QC), for all assays.


More AACC 2016 Philadelphia Notes

August 4, 2016

aacc

As for the AACC 2016 app, it deconstructs the three program books into something pretty useless. With the physical books, one can page through them rather quickly. But with the app, most of the titles are cut off, so it takes forever to find things.

The posters were so far away, they seemed to be in a different zip code.

Several posters were interesting and I was impressed by a poster presented by Linda AC De Grande about the use of patient medians. Maybe one day QC using patients will be mainstream.

Although I’m no longer part of CLSI, I have attended CLSI meetings at the AACC national meeting since the 80s. But these meetings are not held any more at the national AACC meeting. This makes CLSI less inclusive since people can no longer simply drop in on the meetings.

Anyone who stayed at the Marriott (like me) was very happy since the convention center was a block away.

At a Siemens presentation about IQCP, it was implied that conducting IQCP might allow one to run QC less than once a month as long as there was no conflict with manufacturer’s IFU. Not sure if the presenter was correct but a scary thought nevertheless.