More Comments about IQCP

August 27, 2015

57813_image

The Westgard web has some comments about IQCP.

Here are mine.

  1. There is no distinction between potential errors and errors that have occurred. This is non-standard. In traditional risk management different methods are used for potential errors vs. errors that have occurred. For example on page 12 of the IQCP book which focuses on specimen risks, “Kim” reviewed log books and noted errors. Yet on the same page, Kim is instructed to ask “What could go wrong.” The problem is that there are clearly errors that have occurred yet there could be potential new errors that have never occurred.
  2. The mitigation steps to reduce errors look phony. For example, an error source is: “Kim noted some specimens remained unprocessed for more than 60 minutes without being properly stored.” The suggested mitigation is: Train testing personnel to verify and document: Collection time and time of receipt in laboratory and proper storage and processing of specimen. The reason the mitigation sounds phony is that most labs would already have this training in place. The whole point of risk management is to put in place mitigations that don’t already exist.
  3. There is no measurement of error rates. Because there is no distinction between potential errors vs. errors that have occurred, there is a missed opportunity to measure error rates. In the real world, when errors occur and mitigations are put in place, the error rate is measured to determine the effectiveness of the mitigations.
  4. The word “Pareto” cannot be found in IQCP. Here is why this is a problem. In IQCP, for each section, a few errors are mentioned. In the real world, for either potential errors or those that have occurred, the number of errors is much larger. So much larger that there are not enough resources to deal with all errors. That is why the errors are classified and ranked (the ranking is often displayed as a Pareto chart). The errors at the top of the chart are dealt with. In the naïve IQCP, there is no need to classify or rank errors because all are dealt with. The same problem occurs in CLSI EP23 and ISO 14197.

Conclusion: One might infer that no one who participated in the writing of IQCP has ever performed actual risk management using standard methods or perhaps any methods.


Comments about IQCP (Indvidualized Quality Control Plan)

August 16, 2015

photo 111

I had occasion recently to look at IQCP – here are some comments

It seems that if a lab performs IQCP, they can change their QC frequency (read reduce) from the CLIA minimum of twice per day. The lab can’t reduce it more than the manufacturer’s recommended frequency.

This is interesting as it seems that manufacturers will now have the ability to set QC frequency!

Say, the QC frequency turns out to be once a month (which was the case allowed for EQC).

To me, this makes no sense – IQCP or not – since this could mean that a month’s worth of patient samples are incorrect. How can this be good. Moreover, I remember talking to lab directors and asking them, does QC ever go out and the answer was yes, it does.

IQCP requires performing risk management. This should be performed anyway! The notion that performing risk management – which is not that easy to be done well – will somehow obviate the need to check performance by running QC is delusional and not in the best interests of clinicians and patients.


Biases in clinical trials performed for regulatory approval – update

June 12, 2015

basicedp

This article is now online (subscription required).


Biases in clinical trials performed for regulatory approval

May 31, 2015

review

The title of this post has been accepted for publication in the journal: Accreditation and Quality Assurance. The article describes common biases and how they might be avoided.


The revised total error standard EP21, an example of manufacturers dominating CLSI

May 18, 2015

10626368_10202759197363771_4993224402444483262_o

I had a chance to look at the revision of EP21 – the document about total error that I proposed and chaired. So after 12 years, here are the major changes.

In the original EP21, I realized that even if 95% of the results met goals, the remaining 5% might not, so there was a table which accounted for this. An acceptable assay had to have 100% of its results within goals. The revised EP21 – call it A2 – only talks about 95% of results (similar to the 2003 ISO glucose meter standard). There is no longer any mention of the remaining 5% – these remaining results are unspecified. This goes along with my thinking that manufacturers will refuse to talk about assay results that can cause severe injury or death. Thus, if 95% of the results just meet goals, a portion of the remaining 5% could cause severe injury or death and this portion even for a small percentage could be a big number (as one example, there are 8 billion glucose meter results each year in the US).

The mountain plot and all references to it are gone in A2. To recall, the mountain plot is ideal at visualizing outlier observations. In fact, there could be 10,000 observations but if there were 5 outliers, they would be clearly visible. In place of the mountain plot, there is a histogram with an example with normal looking results – the example that had outliers is gone. And the histogram has only 9 bins so if there were outliers, they would disappear. So again, this is a way to minimize talking about results which can cause major problems.

Somehow, sigma metrics have become part of A2. How this happens is a mystery. Perhaps someone can explain it to me, since whereas I understand the equation: Total error = |bias| + 2 x imprecision, the total error in EP21 is the difference between candidate and comparison assays and this difference can’t be separated into bias and imprecision.

And then there is the section on distinguishing between total error and total analytical error. This is part of the reason I was booted out of CLSI. A2 is constrained to include only analytical error.

Total error, including all sources of variation is the only thing that matters to clinicians. The total error experiment (e.g., EP21) will include errors from only those sources that are sampled. Practically speaking, the sources will be limited, even for analytical error. For example, even if more than one reagent is used, this is not the same as randomly sampling from the population of all reagents during the lifetime of the device – impossible since this involves future reagents that don’t yet exist. The same is true for pre- and post-analytical error but the point is one should not exclude pre- and post-analytical error sources from the experiment.

There is a section on various ways to establish goals. Examples shown are the ISO, CLSI, and NACB glucose meter standards, which have performance goals for glucose meters. A2 talks about the strengths and weaknesses of using expert bodies to create these standards. Now A2 has a reference from May of 2015, but somehow they missed the FDA draft guidance on glucose meters (January 2014) which unlike the examples cited in A2 wants evaluators to account for 100% of the data. And, FDA’s opinion about the ISO glucose meter standard is pretty clear:

Although many manufacturers design their BGMS validation studies based on the International Standards Organizations document 15197, FDA believes that the criteria set forth in the ISO 15197 standard do not adequately protect patients using BGMS devices in professional settings, and does not recommend using these criteria for BGMS devices.

I have published a critique of the CLSI glucose meter standard, which is available here.

When I was chair holder of the Evaluations Protocol Committee, there were battles between regulatory affairs people, who populated the manufacturing contingent and the rest of the committee. For example, I remember one such battle over EP6, the linearity document. The proposed new version finally had a sensible statistical method to evaluate nonlinearity but one regulatory affairs member insisted on having an optional procedure where one could just graph the data and look at it to declare whether it was linear. After many delays, this optional procedure was rejected.

By looking at the new version of EP21, my sense is that the regulatory affairs view now dominates these committees.


Total Error and Milan 3

May 11, 2015

DSC_1288edp

Having mentioned in my first blog entry “Total Error and Milan”, the fact that clinician surveys were dropped as a means of constructing performance specifications, I looked at the published paper on this topic. Many of the citations are from the 80s – there’s nothing wrong with that but I was surprised to see that a recent paper on glucose meter performance specifications, which is here and available before the Milan conference was not cited. In this glucose paper, 206 clinicians were surveyed using 4 scenarios and the range of glucose levels that would correspond to one of 5 types of actions: (A) emergency treatment for low BG; (B) take oral glucose; (C) no action needed; (D) take insulin; and (E) emergency treatment for high BG.

Maybe if the Milan conference were aware of this work, they would have added clinician surveys as a primary means to establish performance specifications.


Total Error and Milan 2

May 11, 2015

DSC_1292edp

Having looked further into this conference, I see that the original slides of the talks of the Milan conference are available as well as a list of articles (without needing a subscription).

So one of the articles of interest to me, was the one that describes using simulation to set performance goals. It is here.

And sure enough, this article refers to the glucose meter simulations originally published by Boyd and Bruns and continued by them and others which I have critiqued over the years.

An article that I wrote which shows why such a model can be misleading is now available without subscription and is here.

And another letter by me – published after the Milan conference – is here (subscription required). This makes three articles I published showing that the Boyd Bruns model is incomplete and misleading.


Follow

Get every new post delivered to your Inbox.