A simple improvement to total error and measurement uncertainty

January 15, 2018

There has been some recent discussion about the differences between total error and measurement uncertainty, regarding which is better and which should be used. Rather than rehash the differences, let’s examine some similarities:

1.       Both specifications are probability based.
2.       Both are models

Being probability based is the bigger problem. If you specify limits for a high percentage of results (say 95% or 99%), then either 5% or 1% of results are unspecified. If all of the unspecified results caused problems this would be a disaster, when one considers how many tests are performed in a lab. There are instances of medical errors due to lab test error but these are (probably?) rare (meaning much less than 5% or 1%). But the point is probability based specifications cannot account for 100% of the results because the limits would include minus infinity to plus infinity.

The fact that both total error and measurement uncertainty are models is only a problem because the models are incorrect. Rather than rehash why, here’s a simple solution to both problems.

Add to the specification (either total error or measurement uncertainty) the requirement that zero results are allowed beyond a set of limits. To clarify, there are two sets of limits, an inner set to contain 95% or 99% of results and an outer set of limits for which no results should exceed.

Without this addition, one cannot claim that meeting either a total error or measurement uncertainty specification will guarantee quality of results, where quality means that the lab result will not lead to a medical error.

Advertisements

Do it right the first time – not always the best strategy

December 14, 2017

Watching a remarkable video about wing suit flyers jumping into an open door of descending plane, it appears that they had tried to accomplish this feat 100 times before having success.

On page four of a document that summarizes the quality gurus: Crosby, Deming and Juran, Crosby’s “Do it right the first time” appears. Clearly, this would have been a problem for the wing suit flyers. Crosby’s suggestion is appropriate if the state of knowledge is high. For the wing suit flyers, there were many unknowns, hence the state of knowledge was low. When the state of knowledge is meager, as it was at Ciba Corning when we were designing in vitro diagnostic instruments, we used the test analyze and fix strategy (TAAF) as part of reliability growth management and FRACAS. This sounds like the opposite of a sane quality strategy but in fact was the fastest way to achieve reliability goals for our instruments.


Risk based SQC – What does it really mean

December 4, 2017

Having just read a paper on risk based SQC, here are my thoughts…

CLSI has recently adopted a risk management theme for some of their standards. The fact that Westgard has jumped on the risk management bandwagon is as we say in Boston, wicked smaaht.

But what does this really mean and is it useful?

SQC as described in the Westgard paper is performed to prevent patient results from exceeding an allowable total error (TEa). To recall, TEa = |bias|/SD*1.65. I have previously commented that this model does not account for all error sources, especially for QC samples. But for the moment, let’s assume that the only error sources are average bias and imprecision. The remaining problem with TEa is that it is always given as a percentage of results, usually 95%. So if some SQC procedure were to just meet their quality requirement, up to 5% of patient results could exceed their TEa and potentially cause medical errors. This is 1 in every 20 results! I don’t see how this is a good thing even if one were to use a 99% TEa.

The problem is one of “spin.” SQC, while valuable, does not guarantee the quality of patient results. The laboratory testing process is like a factory process and with any such process, to be useful it must be in control (meaning in statistical quality control). Thus, SQC helps to guard against an out of control process. To be fair, if the process were out of control, patient sample results might exceed TEa.

The actual risk of medical errors due to lab error is a function not only of an out of control process but also due to all other error sources not accounted for by QC, such as user errors with patient samples (as opposed to QC samples), patient interferences, and so on. Hence, to say that risk based SQC can address the quality of patient results is “spin.” SQC is a process control tool – nothing more and nothing less.

And the best way of running SQC would be for a manufacturer to assess results from all laboratories.

Now some people might think, this is a nit-piking post but here is an additional point. One might be lulled into thinking that with this risk based SQC that labs don’t have to worry about bad results. But interferences can cause large errors that can cause medical errors. For example, in the maltose problem for glucose meters, 6 of 13 deaths occurred after an FDA warning. And recently, there have been concerns about biotin interference in immunoassays. So it’s not good to oversell SQC, since people might loose focus on other, important issues.


More commitment needed from authors

November 5, 2017

I just read an interesting paper about irreproducibility in science. The authors suggest a remedy: namely; that “authors of such papers should be invited to provide a 5-year (and perhaps a 10-year) reflection on their papers”.

I suggested to Clinical Chemistry a few years ago that every paper should have a “recommendations” section. To recall, most papers have some or all of: an introduction, methods, results, discussion, and conclusion sections. But rarely if ever is there a recommendations section, although sometimes there is a recommendation in the conclusions section.

In my company, I established a reporting format that required a recommendations section. The recommendations required action words (e.g., verbs).

So a study to evaluate an assay might have as a conclusion: “Assay XYZ has met its performance specifications.” The corresponding recommendation might be: “Release assay XYZ for sale.”

Although the recommendation might seem to be a logical consequence of the conclusion, psychologically, the recommendation requires more commitment. Were there outliers? Did the study have enough samples? Was there possible bias?

In any case, Clinical Chemistry declined to accept my suggestion.

 


A measure of influence

October 30, 2017

When I worked at Corning Medical (later Ciba Corning), we worked in cubicles and everyone had a whiteboard. These whiteboards were used often and sometimes a section of the whiteboard would be marked so that the contents of that section would remain as in DO NOT ERASE! I was pleased to note that on a few whiteboards, something that I had said was in the do not erase section.

But there was one person who dominated the do not erase section, even after he had long left the company. This was David Simmons, who acted as in in-house psychologist. This was one measure of his influence.


Calculating measurement uncertainty and GUM

October 16, 2017

A recent article (subscription required) suggests how to estimate measurement uncertainty for an assay to satisfy the requirements of ISO 15189.

As readers may know, I am neither a fan of ISO nor measurement uncertainty. The formal document, GUM – The Guide to the Expression of Uncertainty in Measurement will make most clinical chemists heads spin. Let’s review how to estimate uncertainty according to GUM.

  1. Identify each item in an assay that can cause uncertainty and estimate its imprecision. For example a probe picks up some patient sample. The amount of sample taken varies due to imprecision of the sampling mechanism.
  2. Any bias found must be eliminated. There is imprecision in the elimination of the bias. Hence bias has been transformed into imprecision.
  3. Combine all sources of imprecision into a BHE (big hairy equation – my term, not GUMs).
  4. The final estimate of uncertainty is governed by a coverage factor. Thus, an uncertainty interval for 99% is wider than one for 95%. Remember that an uncertainty interval for 100% is minus infinity to plus infinity.

The above Clin Chem Lab Med article calculates uncertainty by mathematically summing imprecision of controls and bias from external surveys. This is of course light years away from GUM. The fact that the authors call this measurement uncertainty could confuse some to think that this is the same as GUM.

Remember that in the authors’ approach, there are no patient samples. Thus, the opportunity for errors due to interferences has been eliminated. Moreover, patient samples can have errors that controls do not. Measurement uncertainty must include errors from the entire measurement process, not just the analytical error.

Perhaps the biggest problem is that a clinician may look at such an uncertainty interval as truth, when the likely true interval will be wider and sometimes much wider.


Two examples of why interferences are important and a comment about a “novel approach” to interferences

September 29, 2017

I had occasion to read an open access paper “full method validation in clinical chemistry.” So with that title, one expects the big picture and this is what this paper has. But when it discusses analytical method validation, the concept of testing for interfering substances is missing. Precision, bias, and commutability are the topics covered. Now one can say that an interference will cause a bias and this is true but nowhere do these authors mention testing for interfering substances.

The problem is that eventually these papers are turned into guidelines, such as ISO 15197, which is the guideline for glucose meters. And this guideline allows 1% of the results to be unspecified (it used to be 5%). This means that an interfering substance could cause a large error resulting in serious harm in 1% of the results. Given the frequency of glucose meter testing, this translates to one potentially dangerous result per month for an acceptable (according to ISO 15197) glucose meter. If one paid more attention to interfering substances and the fact that they can be large and cause severe patient harm, the guideline may have not have allowed 1% of the results to remain unspecified.

I attended a local AACC talk given by Dr. Inker about GFR. The talk, which was very good had a slide about a paper about creatinine interferences. After the talk, I asked Dr. Inker how she dealt with creatinine interferences on a practical level. She said there was no way to deal with this issue, which was echoed by the lab people there.

Finally, there is a paper by Dr. Plebani, who cites the paper: Vogeser M, Seger C. Irregular analytical errors in diagnostic testing – a novel concept. (Clin Chem Lab Med 2017, ahead of print). Ok, since this is not an open access paper, I didn’t read it but what I can tell from Dr. Plebani comments, the cited authors have discovered the concept of interfering substances and think that people should devote attention to it. Duh! And particularly irksome is the suggestion by Vogeser and Seger of “we suggest the introduction of a new term called the irregular (individual) analytical error.” What’s wrong with interference?