A bone to pick with AACC

August 17, 2018

So I registered and was at the AACC meeting in Chicago, but I couldn’t make all of the scientific sessions of interest to me.

There was a link to download handouts for any session (great) but there were 2 problems:

  1. Some of the sessions were stated as “not yet available.” While perhaps understandable before the meeting, they are still listed as not yet available, 2 weeks after the meeting ended.
  2. Of the sessions that I downloaded, some of the material was unreadable (particularly graphs).

AACC needs to improve the quality of their meeting.


Who performed your test?

August 15, 2018

The conventional wisdom is that if you require some medical procedure based on the result of a medical test, before submitting to that procedure, you should have the test repeated.

Good advice, but more advice needs to be added. You should have the test repeated by a different method. In my book, I describe a case where due to suspected cancer from an elevated hCG result, the hCG assay was repeated 45 times while unnecessary treatment including surgery was performed. It wasn’t until the assay was repeated on a different method that in fact the hCG result was found to be normal – the woman never had cancer.

But my lab report that I view online, while having graphs of previous results and inclusion of expected normal ranges, does not provide any information as to what method or manufacturer was used to perform the test. I have seen a lab report from Europe where the manufacturer is listed. This information should be on lab reports.

Reliability and six sigma

August 4, 2018

I had occasion to read an article about six sigma and reliability – it is here.

The essence of this article is that by measuring a long term sigma value, one will know the reliability of results, where reliability is equated to the number of defects. Defects are defined as results that are outside of the performance goals.

To recall: six sigma = (allowable total error – bias)/CV. High six sigma numbers are good, low numbers not so good.

The problem with this article is that it bases everything on Normal distribution statistics. Now this may make sense if you are measuring rulers sold by Home Depot but it doesn’t work for blood tests.

Consider a glucose meter. Unlike the Home Depot ruler, there’s a lot more going on in a drop of blood. There are thousands of compounds that can interfere. Say that one does, and that the meter reads 340 and truth is 40 mg/dL. Assume that the CV at 40 is 3%. The value at 340 is 250 standard deviations away! I challenge anyone to try to calculate the probability of such an event. There’s not enough zeros on the planet. Thus, the 340 value, which can happen, is not part of the measuring error of the usual process.

So any attempt to judge the number of defects by a six sigma calculation will miss the really big errors. And these are the errors that cause harm to patients.

An additional problem is attaching significance to the numerical six sigma results. Now this may sound like heresy but here’s an example.

Say you were comparing the Roche glucose meter a few years ago (yes the one with the maltose interference problem) with some other meters. The Roche meter would have probably had a high six sigma value and thus looked good. Obviously, this would have been a bad choice.

But what about in general? Consider what a lower six sigma number means. Yes there will be more values beyond the performance limits but these values will be a few standard deviations away and close to the performance limits. Unfortunately, six sigma values provide no information about large errors.

Sorry, but to evaluate the possibility of large errors caused by interferences requires extensive interferences studies or alternatively huge patient correlations (the kind that no one does).

Theranos about to close?

August 4, 2018

There’s a rumor going around that Theranos will soon cease to exist. So before that happens, if it does, I wanted to see if the AACC people were still on the Theranos scientific board. The info about that, on the Theranos website is here.

So the following people, known in various present and former capacities at AACC are still on the board, with the last name perhaps a new addition.

Susan A. Evans
Jack Ladenson
Alan H.B. Wu


Theranos / AACC – You have to answer for Santino

July 31, 2018

The Theranos topic reappeared at the 2018 AACC meeting. This time there was an interview with John Carreyrou, the author who first published the problems of Theranos in The Wall Street Journal in October of 2015. The interview was well done in a question and answer format, much like a TV talk show. For people like me who read Carreyrou’s book Bad Blood, there was not much new. The AACC audience had a smug response as in, how could this have happened – I would have never bought this stuff.

But the two questions I wanted to hear were never asked!

  1. Why did AACC invite the founder of Theranos to present at AACC in 2016, after it was discovered that Theranos had committed fraud?
  2. Why did four former AACC presidents (Susan Evans, Ann Gronowski, Larry Kricka, and Jack Ladenson) join the Theranos scientific board after the 2016 AACC meeting?

Ok, maybe the Godfather reference doesn’t work, but I would have liked to hear answers to these questions.

How Should Glucose Meters be evaluated for Critical Care

July 24, 2018

There is a new, IFCC document with the title the same as this blog entry.  Ok, I know better than to try to publish a critique of an IFCC document, so I’ll keep my thoughts to this blog.

The glucose meter goals suggested by IFCC are the same as those contained in the CLSI document POCT12-A3. Now I do have a published critique of this CLSI standard – it is here. Not a surprise, but my critique of POCT12-A3 is not listed in the many IFCC references.

Upon skimming the IFCC document, it has the same accuracy goals as POCT12-A3, which basically leaves 2% of glucose meter results as unspecified (e.g., could be real bad). Since the IFCC document covers the possibility of interferences and user errors as a reason for errors, someone needs to tell me why 2% of glucose meter results are unspecified.

The problem is, say you did an evaluation with 100 samples and 1 of them had a large error (much greater than a 20% error). A problem with the POCT12-A3 spec is that allows one to say for the results of this evaluation, the spec has been met even though the bad result could cause patient harm. Hence, meeting the POCT12-A3 spec implies that one has achieved accuracy as suggested by a standards group and this could justify one to ignore the bad result.

A selected catalog of critiques

July 12, 2018

The highlighted articles can be viewed without a subscription.

Imprecision calculations – Evaluations commonly reported total imprecision as less than within-run imprecision. Correct calculations are explained.

How to Improve Estimates of Imprecision Clin. Chem., 30, 290-292 (1984)

Total error models – Modeling total error by adding imprecision to bias is popular but fails to account for several other error sources. These articles (and others) provide alternative models.

Estimating Total Analytical Error and Its Sources: Techniques to Improve Method Evaluation Arch Pathol Lab Med., 116, 726-731 (1992)

Setting Performance Goals and Evaluating Total Analytical Error for Diagnostic Assays Clin. Chem., 48: 919-927 (2002)

Too optimistic project completion schedules – Project managers would forecast completion dates that were never met. The article shows how to get better completion estimates using past data.

Beware the Percent Completion Metric Research Technology Management, 41, 13-15, (1998)

GUM – The guide to the expression of uncertainty in measurement was suggested to be performed by hospital labs. There’s no way a hospital lab could carry out this work.

A Critique of the GUM Method of Estimating and Reporting Uncertainty in Diagnostic Assays Clin. Chem., 49:1818-1821 (2003)

ISO 9001 – There have been many valuable quality initiatives. In the late 80s, ISO 9001 was a program to certify that companies that passed had high quality. But it was nothing more than documentation – it did nothing to improve quality. Maybe the lab equivalent ISO 15189 is the same.

ISO 9001 has had no effect on quality in the in-vitro medical diagnostics industry Accred. Qual. Assur., 9: 39-43 (2004)

Bland-Altman plots – Bland-Altman plots (difference plots) suggest plotting the difference of y-x vs. (y+x)/2 in order to prevent spurious correlations. But the article below shows that if x is a reference method, following Bland and Altman’s advice will produce a spurious correlation. The difference of y-x vs x should be plotted when x is a reference method.

Why Bland-Altman plots should use X, not (Y+X)/2 when X is a reference method Statistics in Medicine, 27 778-780 (2008)

Six Sigma – This metric is often presented as a sole quality measure but it basically measures only average bias and imprecision. As this article shows there can be severe problems with an assay even when it has a high sigma.

Six Sigma can be dangerous to your health Accred Qual Assur 14 49-52 (2009)

Glucose standards – The glucose meter standard ISO 15197 has flaws. This letter pointed out what the experts missed in a question and answer forum.

Wrong thinking about glucose standards Clin Chem, 56 874-875 (2010)

POCT12-A3 – The article explains flaws in this CLSI glucose standard

The new glucose standard POCT12-A3 misses the mark Journal of Diabetes Science and Technology, September 7 1400–1402 (2013)

Regulatory approval evaluations – The performance of assays during regulatory evaluations is often quite better than when the assays are in the field. The articles gives some reasons why.

Biases in clinical trials performed for regulatory approval Accred Qual Assur, 20:437-439 (2015)

MARD – This metric to classify glucose meter quality leaves a lot to be desired. The article below suggests an alternative

Improving the Glucose Meter Error Grid with the Taguchi Loss Function Journal of Diabetes Science and Technology, 10 967-970 (2016)

Interferences – Motivated by a recent paper where interferences were treated almost as a new discovery (and given a new name), this paper discusses how specifications and analyses methods can be improved by accounting for interferences. And I also mention how the CLSI EP7 standard reports interferences incorrectly and could cause problems for labs. 

Interferences, a neglected error source. Accred. Qual. Assur. 23(3):189-192 (2018).