November 21, 2014
At a recent AACC dinner meeting, I heard an interesting talk by Nader Rifai, the editor of Clinical Chemistry. About halfway through his talk, I remembered an event that took place a couple of years ago, so I asked him a question after his talk ended. My question and Nader’s responses went something like this:
Me: “A while ago, I read a commentary article that I didn’t agree with and submitted a Letter to the editor about it. The response from the journal was…”
Rifai: “It wouldn’t be reviewed because it wasn’t about an original article, right?”
Me: “Yes, that’s right, then I looked at a few issues and saw that the percentage of original articles is only about 50% of the journal. This means that one can’t comment about a large portion of the journal.”
Rifai: “Well, we were seeing Letters to the editor about other Letters to the editor and with commentary articles it is common that many people won’t have the same opinion as the author, so we don’t want to fill up the journal with such stuff.”
This is sort of what I remembered, not verbatim but that is the gist of it.
So basically, Rifai is putting Letters to the editor into a generic category similar to junk mail or the endless comments associated with Twitter or a blog and at the same time giving immunity to authors – other than those who write original articles – from any kind of comment.
But the problem is that commentary articles in Clinical Chemistry are about science and if the authors get the science wrong, it is a mistake to prevent people from pointing that out. That is unscientific / elitist. Perhaps contributing to this elitism was that Rifai mentioned that articles in Clinical Chemistry are of high quality due to the extensive review process. But this doesn’t guarantee correctness.
And Clinical Chemistry has changed its policy. I commented briefly on this topic before in this blog. My 2010 Letter to the editor about a “Question and Answer” type article was published. Moreover, I think my 2010 Letter had a role in shaping glucose meter standards but these days the Letter would not have been considered.
So now I have less interest in reading Clinical Chemistry.
November 20, 2014
A recent article in Clin Chem – available without subscription – purports to show the result reliability of different A1c assays (1).
The basic premise of this paper is that given:
- total error goals
- a QC program
- a study to estimate imprecision (CLSI EP5) and average bias (CLSI EP9)
one can determine the risk of reporting unreliable results.
This is simply not true. I have shown before – see ref 2 for the most recent – that the Westgard model of total error = (a multiple of imprecision + average bias) is incomplete and typically underestimates the true amount of error.
Thus, the authors’ risk of reporting unreliable results is itself unreliable and probably underestimates things because:
- There is no information about interfering substances, not even a list of the standard error of estimates from the regessions which would provide some information about this error source.
- One can assume that one reagent was used. Yet lot-to-lot reagent error is usually the largest component of error in an assay. Hence, this error source is inadequately measured
- One does not know if the people that ran the study are representative of people who routinely run the assay – important since user error is often a significant source of error.
And finally, the use of one set of total error goals is questionable. If some results fail the total error goal, one wants to know if they just fail or if they are way out because just as error can be small or large, so can the resulting patient harm. Studies of the type in the paper can’t really help here because they use one Normal distribution. But in the real world, errors tend to come from different sources (distributions) so the risk of large errors is completely unknown.
What should one do to get a better prediction of risk?
- Conduct risk analysis by performing a fault tree and FMEA (Failure Mode Effects Analysis) that includes
- The correct model (see reference 2)
- Account for the error sources missed in the paper (part of the fault tree / FMEA)
- Woodworth A Korpi-Steiner N, Miller JJ, et.al. Utilization of Assay Performance Characteristics to Estimate Hemoglobin A1c Result Reliability Clin Chem 2014 60 1073–1079 (2014).
- Krouwer JS The danger of using total error models to compare glucose meter performance. Journal of Diabetes Science and Technology, 2014;8:419-421.