A recent article in Clin Chem – available without subscription – purports to show the result reliability of different A1c assays (1).
The basic premise of this paper is that given:
- total error goals
- a QC program
- a study to estimate imprecision (CLSI EP5) and average bias (CLSI EP9)
one can determine the risk of reporting unreliable results.
This is simply not true. I have shown before – see ref 2 for the most recent – that the Westgard model of total error = (a multiple of imprecision + average bias) is incomplete and typically underestimates the true amount of error.
Thus, the authors’ risk of reporting unreliable results is itself unreliable and probably underestimates things because:
- There is no information about interfering substances, not even a list of the standard error of estimates from the regessions which would provide some information about this error source.
- One can assume that one reagent was used. Yet lot-to-lot reagent error is usually the largest component of error in an assay. Hence, this error source is inadequately measured
- One does not know if the people that ran the study are representative of people who routinely run the assay – important since user error is often a significant source of error.
And finally, the use of one set of total error goals is questionable. If some results fail the total error goal, one wants to know if they just fail or if they are way out because just as error can be small or large, so can the resulting patient harm. Studies of the type in the paper can’t really help here because they use one Normal distribution. But in the real world, errors tend to come from different sources (distributions) so the risk of large errors is completely unknown.
What should one do to get a better prediction of risk?
- Conduct risk analysis by performing a fault tree and FMEA (Failure Mode Effects Analysis) that includes
- The correct model (see reference 2)
- Account for the error sources missed in the paper (part of the fault tree / FMEA)
- Woodworth A Korpi-Steiner N, Miller JJ, et.al. Utilization of Assay Performance Characteristics to Estimate Hemoglobin A1c Result Reliability Clin Chem 2014 60 1073–1079 (2014).
- Krouwer JS The danger of using total error models to compare glucose meter performance. Journal of Diabetes Science and Technology, 2014;8:419-421.