The Diabetes Technology Society (DTS) surveillance protocol doesn’t seem right

January 16, 2017

CPPP.edp

The Diabetes Technology Society (DTS) has published a protocol that will allow a glucose meter to be tested to see if the meter meets the DTS seal of approval. This was instituted because for some FDA approved glucose meters, the performance of post release for sale meters from some companies did not meet ISO standards.

Before the DTS published their protocol, they published a new glucose meter error grid – the surveillance error grid.

But what I don’t understand is that the error grid is not part of the DTS acceptance criteria to gain the DTS seal of approval. (The error grid is plotted as supplemental material). Basically, to get DTS approval, one has to show that enough samples have differences from reference that fall within the ISO 15197:2013 standard. To be fair, the ISO standard and the “A” zone of the error grid have similar limits, but why not use the error grid, since the error grid was developed by clinicians whereas the ISO standard is weighted by industry members. And the error grid deals with results in higher zones.

Moreover, the DTS does not deal with outliers other than to categorize them – their presence does not disqualify a meter from getting DTS acceptance as long as the percentage of results within ISO limits is high enough.

So if a meter has a 1% rate of values that could kill a patient, it could still gain DTS seal of approval. This doesn’t seem right.

 


Book about noninvasive glucose meters

December 12, 2016

dsc_1614edp

Noninvasive glucose meters are the Holy Grail in glucose testing. To be able to get a glucose value without a finger stick would be a tremendous benefit to the millions of people who have to test themselves several times each day.

So there have scores of scientists who have worked on the problem, backed by diagnostic companies since the profit potential is huge.

I remember while at Ciba Corning, attending a lecture on near infrared spectroscopy given by a professor whom I think we were supporting to try to come up with a noninvasive glucose meter.

On a website devoted to diabetes, I became aware of a book which chronicles the quest for a noninvasive glucose meter. It is recent (2015 publication date), free, and written by a former chief scientific officer and VP of LifeScan who has been involved in this search for years.

I found it fascinating.


Test error and healthcare costs

December 7, 2016

dsc_1642edp

Conventional wisdom says that regulatory authorities approve assays that have the highest quality, meaning that the errors are small enough that no or little harm will arise because a clinician makes a wrong medical decision based on test error.

It is also true, although not talked about, that in most countries healthcare is rationed – the cost of treating everyone with every possible treatment is too high.

So here’s a hypothetical example using glucose meters.

First, we start out with the status quo for existing glucose meter quality and assume that on average, across all tests there will be some harm due to glucose meter error. The percentage of tests that harm people is unknown as is the range of harm but assume that these can be ascertained and do occur.

As for the hypothetical part…

There are 2 new glucose meters seeking approval

Meter A costs 100 times as much as current meters and is guaranteed to have zero error, as it is a breakthrough technology. Its use will reduce patient harm due to test error to zero.

Meter B costs 100 times less than current meters but isn’t quite as accurate or reliable. Patient harm will increase with the use of meter B.

If meter A is approved, because of healthcare rationing, costs will have to be transferred from other parts of healthcare to pay for meter A.

If meter B is approved, costs can be transferred from glucose meter testing to other parts of healthcare.

The point is not to try to answer whether meter A or meter B should be approved, but to illustrate that the cost issues associated with healthcare policy always exist but are rarely discussed.


Letter to be published

November 15, 2016

dsc_1420edp

Recently, I alerted readers to the fact that the updated FDA POCT glucose meter standard no longer specifies 100% of the results.

So I submitted a letter to the editor to the Journal of Diabetes Science and Technology.

This letter has been accepted – It seemed to take a long time for the editors to decide about my letter. I can think of several possible reasons:

  1. I was just impatient – the time to reach a decision was average
  2. The editors were exceptionally busy due to their annual conference which just took place.
  3. By waiting until the conference, the editors could ask the FDA if they wanted to respond to my letter.

I’m hoping that #3 is the reason so I can understand why the FDA changed things.


Glucose meter QC – too much?

November 13, 2016

dsc_1445edp

A colleague has been corresponding with me about glucose meter QC and recently sent me this paper. Basically, the authors are concerned about the high cost of QC for glucose meters and have data in their paper to show that their glucose meters are very reliable.

Actually, they state their meters are better than highly reliable because to quote from their paper: “no meter failure is detected by current QC testing procedure”. Well, not so fast. In some cases a repeated QC failure was corrected by using a new vial of strips. To me, this indicates a basic misunderstanding of glucose meters. One can think of the glucose meter testing process as having three components:

 

  1. The user – who must correctly use the meter
  2. The reagent strip – this is where the chemistry occurs
  3. The meter – hardware and software with the outcome being a glucose result

 

It seems as if these authors consider the meter as the item to be controlled. Yet it is highly unlikely that the meter could provide incorrect results – certainly no results if a meter’s hardware failed. But the reagent strip is where the action occurs and a bad strip could cause wrong answers, so in the authors’ study, QC did detect bad strips and presumably prevented wrong results.

I will comment at a later date about QC and user error.

What if the authors had shown no failures due to QC. Does that justify reducing QC to perhaps a monthly (as suggested by CMS) or less frequency? Cost is an important issue. But the purpose of QC is to detect errors. QC is not useless if no errors are found.

The purpose of QC is to detect errors to prevent an incorrect result to be reported. This is to prevent a clinician from making the wrong medical decision – based on test error, which causes harm to the patient. Hence, an assumption is that a correct result is needed to prevent patient harm. (If this is not the case, then one can argue that no QC is needed nor is the test needed in the first place).

But the frequency of QC actually detecting errors is not important as long as it can been shown that QC can detect errors. If the system is reliable, the error rate will be low.

The message is one would never remove safeguards just because of low error rates. For example, in hospitals and nuclear power plants, monitoring for radiation is a QC like practice and costs money. The frequency of problems is not relevant.


Comparison of Parkes glucose meter error grid with CLSI POCT 12-A3 Standard

November 8, 2016

dsc_1623edp

The graph below shows the Parkes error grid in blue. Each zone in the Parkes error grid shows increasing patient harm with the innermost zone A having no harm. The zones (unlabeled) start with A (innermost) and go to D or E.

The red lines are the POCT 12-A3 standard. The innermost line should contain 95% of the results. Since no more than 2% can be outside of the outermost red lines, these outermost red lines should contain 98% of the data.

comp

The red lines correspond roughly with the A zone of the Parkes error grid – the region of no patient harm.

Of course the problem is that in the CLSI guideline, 2% of the results are allowed to occur in the higher zones of the Parkes error grid corresponding to severe patent harm.


Westgards Detection and IQCP

November 1, 2016

dsc02015edp

I received an email recently that alerted me to three seminars from the 2016 AACC meeting that are online. One is by the Westgards, so I had a look. This is quite an interesting presentation and shows the breadth of the contributions that the Westgards have made to quality in laboratory medicine.

Yet, one thing caught my eye and so here are my comments. Thus, the Westgards complain that in risk management as espoused by CLSI EP23, detectability has been omitted.

What they mean is that for each failure event, EP23 wants one to estimate the severity and probability of occurrence of that failure event. The Westgards suggest that the detectability of the failure event needs to be assessed as well and state that this is how industry does it.

Well maybe some industries, but I worked in industry and our company did not use detectability (we used severity and probability of occurrence).

Now in the context of EP23, I agree with the Westgards use of detectability. The problem is that EP23 itself is a poor adaptation of risk management. I commented on this before but here it is again.

As an example of a failure mode of a process step, assume that the failure is sample hemolysis which occurs during the process step to turn a whole blood sample into serum. As you go across the rows in an EP23 style risk analysis, you might see that a mitigation for this failure mode is to visually check whether the sample has been hemolyzed and how effective this check is. In this case – for this row item – you could add detectability to severity and probability of occurrence.

Here are the problems with this approach, whether you have added detectability or not.

For most labs, this (example) is already established laboratory practice. That is, labs already check to see whether samples are hemolyzed. All that has been done is to document it. Not much in the way of formal risk analysis has been done although there will be some benefit to this documentation.

The problem is that the row is “collapsed.” It really has two additional process steps embedded in it. Here it is uncollapsed:

Process step – process whole blood into serum
Detection step – examine serum for the possibility of hemolysis
Recovery step – if the serum has been hemolyzed, request a new sample

One can see that it makes no sense to ask for the detectability of a detection step.

I note in passing that one of the most important detection process steps for any assay is running quality control.

Note that each of these steps above are process steps and each can fail. Whereas the severity will be the same for the failure for each of these steps, the probability of occurrence may differ. Because each step can fail, one needs to assess whether a mitigation step is required.

BTW, one should not discount failures in the recovery step. In the Challenger accident, engineers warned about the potential problem (detection) but delaying the launch failed (recovery). And of course, recovery steps are only performed if detection steps detect something.

Disclaimer – I may not have the latest version of EP23, but another problem in developing the potential failure modes (EP23 call these hazards) is that the process is not fully delineated – it is too high level. In a more traditional FMEA, the list of process steps is long and reflects what is actually done, not some high level description.

And each process step can fail in multiple ways. EP23 is a hazard based list. A process based list is better since one can ask how else each process step can fail. Although EP23 does some of this, it’s embedded within a row and makes things too complicated. Here’s an example of a few ways the above detection step of examining the sample for hemolysis can fail:

  1. Technician simply misses seeing a hemolyzed sample (non-cognitive error – we all make them)
  2. Technician misses seeing a sample due to inadequate training (cognitive error)
  3. Technician misses seeing a sample due to distraction (phone call, or talking to colleague).
  4. Technician ignores hemolyzed sample due to pressure from management to process samples.

On a separate note, how does IQCP help one modify QC frequency?