Rare errors need to be evaluated differently than frequent errors

A paper about error grids has been rejected twice and one of the stumbling blocks seems to be that risk management and a method comparison are proposed to be used to evaluate error grid performance.

To recall, an error grid (see CLSI EP27P) is a way to specify and evaluate assay performance. One plots the values for a candidate and comparison method on an XY plot where zones describe the amount of patient harm possible. In a simple error grid, the closest zone to the identity line demarcates no harm from minor harm. The outermost zone demarcates minor harm from major harm. Error grids are well known in glucose monitoring and little used elsewhere.

To evaluate the performance of an assay with respect to the innermost zone, one performs a method comparison and counts the percentage of points that fall within the zone. One can also calculate confidence limits. This result tells one (for good assays) that most assay results will result in no patient harm while a few assays results may or will cause minor patient harm.

But this method won’t work for the outermost zone. The reason is that the number of expected results that will cause serious patient harm is extremely low – often less than 1 in a million. A method comparison experiment almost always uses a low number of samples (around 100 and often less) and while it is expected that no results fall in the outermost zone, this just does not prove much. To demonstrate by means of a method comparison experiment that one is very confident that no results will fall in the outermost zone, one would have to run millions of samples and this is impractical.

If one simply stops at the 100 sample method comparison, then one has demonstrated performance about no harm and minor harm and nothing about serious harm. What can one do? Perform risk management on the assay (FMEA – Failure Mode Effects Analysis – and fault trees) to demonstrate that the most serious potential errors have mitigations in place to reduce risk to an acceptable level (as low as possible within financial constraints).

So why is risk management so difficult for clinical chemists? Perhaps because most current performance requirements specify limits for only 95% of the results. This is unfortunate since it is the remaining 5% of results which can cause serious harm. Error grids address this problem by specifying limits for 100% of the data.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: