Recently, I talked to someone who attended a conference on total error in Milan. Had I known about the conference or been invited, I would have attended. Searching the web, the Westgard web site has summaries and links about this conference. So here are my comments:
- The use of allowable performance specifications implies a set of limits that demarcate no harm from harm. This further implies that for many analytes, results that just exceed the limit will cause minor harm. But for many analytes, harm increases as the error increases (such as for glucose meter errors). Thus, small errors may result in minor harm and large errors can be fatal. This can be accounted for by using an error grid (such as a glucose meter error grid) which has separate zones for increasing error and harm.
- The allowable performance specifications are for analytical performance. Although pre- and post-analytical errors are mentioned, there is no attempt to present allowable performance specifications that include all sources of error. Thus, in the consensus statement, “The SPC encourages users to expand those specifications [referring to analytical performance specifications] to the total examination process.” This is not something that should be a user exercise.
- The primary method for establishing allowable analytical performance specifications is: “Based on the effect of analytical performance on clinical outcomes.” It is interesting to compare for this item, the unofficial summary from the Westgard site, to the official summary. Note that IMHO, the most important method, a clinician survey, has been dropped in the official version.
- Also problematic is the suggestion of #2 below of using simulations. In glucose meter modeling, I have published on how misleading these simulations have been.
In order to develop quality specifications using outcomes, you must complete one of the following:
- an Outcome study investigating the impact of analytical performance on clinical outcomes
- a Simulation study investigating the impact of analytical performance on the probability of clinical outcomes
- a Survey of clinicians’ and/or experts’ opinion investigating the impact of analytical performance on medical decisions
This can, in principle, be done using different types of studies:
- Direct outcome studies – investigating the impact of analytical performance of the test on clinical outcomes;
- Indirect outcome studies – investigating the impact of analytical performance of the test on clinical classifications or decisions and thereby on the probability of patient outcomes, e.g., by simulation or decision analysis.