Pay for performance – the missing measure

Recently it has been suggested that to improve quality, hospitals will be paid based on their performance, where performance is taken to mean among other things a lower rate of medical errors. This of course implies that something will be measured. The number of measures suggested is huge. An idea of the extent of these measures is shown in the following table (1).

Number of measures for selected categories

Category Number of Measures
Process 343
Patient experience  83
Outcomes 129
Cardiovascular diseases 144
Pathological conditions, signs and symptoms 160

A global (total medical error) measure is needed

Specific measures are useful. However, what is also needed is a global or total medical error measure, e.g., the sum of all severity weighted individual medical error rates. Total medical error measures can exist for specific services and there can be one hospital wide global measure. For laboratory medicine, a list of specific measures has been suggested (2). Whereas all of these measures are useful, one would still like to know for the lab, what is the overall medical error rate.

A financial analogy

Investors look at many measures to decide whether to invest including the overall measure – profitability. It’s hard to imagine omitting this measure but that is what is being proposed in pay for performance. In financial reporting, if revenue is reported but not cost, there is no way to estimate profitability. The way profitability is reported is standardized so that companies can be compared. The same needs to happen for pay for performance, with a total medical error rate measure.

Some examples of problems when there is no global measure

In laboratory medicine, errors are often divided into the categories, pre-analytical, analytical, and post-analytical. Analytical errors are often given the most attention enough though their frequency is lower than the other categories. Part of the reason is that it is relative easy to quantify many analytical performance parameters. Even within the analytical error category, insufficient attention is given to some important errors such as interferences (3) because their estimation is more difficult. Moreover, in estimating certain parameters such as average bias, outlier data are often discarded. While this is legitimate with respect to average bias estimation, it is possible that the discarded data (and the origin of these errors) will disappear from consideration even though their effect will still be observed. There have been attempts to model total error (for analytical errors) which have their own problems. For example in a GUM (Guide for the Estimation of Uncertainty of Measurement) like approach (4), important errors were ignored if they were infrequent. In another case, an analytical total error model was shown to be incorrect (3). The possibility of incorrect models is remedied by a direct measure of total analytical error (5), which does not rely on a model.

In the preanalytical area, a patient sample mix-up is an important error but it is uncommon for this type of error to be compared to the analytical errors and it is also uncommon for attribute types of data to be compared to continuous variables such as bias. Without a Pareto like analysis of all observed errors, it is possible that resources will not be optimized to provide the quickest reduction for all medical errors according to their importance. The fact that different people may deal with different errors also complicates matters in the absence of a Pareto since the skill that people have in lobbying for funds may be out of whack with the results from a conceptual Pareto (e.g., one that could exist but doesn’t). Patient sample mix-ups would be investigated using FMEA, FRACAS, or root cause analysis while bias would be investigated with statistical analysis from a method comparison.

There is also the problem of goals, when one has a series of individual medical errors. How does one realistically set the error rate reduction for each error.

FRACAS (and FMEA) allow for a global measure

FRACAS – Failure Review And Corrective Action System FMEA – Failure Mode Effects Analysis

People in hospitals are familiar with the classification scheme used in FMEA to classify errors – the same one is used in FRACAS. That is, each error is classified (numerically) according to its severity and frequency of occurrence. The two numbers are multiplied together to get criticality. Once can add up this criticality and by means of a usage factor arrive at a global medical error rate. That is, one has not only a Pareto of individual medical errors, but also a measure of the total medical error rate which is simply the sum of all elements in the Pareto. Note that to reduce observed errors, FRACAS, and not FMEA will apply.

Given the Pareto based on a FRACAS, one can apply tools such as reliability growth management which allows one to track progress and predict when a total medical error rate goal will be achieved, as was shown for medical instrument reliability (6). For this case, analytical performance problems such as bias and hardware failures which affected availability (e.g., turn-around-time) were all classified and captured in the FRACAS.

Of course, people can argue that it is difficult in hospitals to have an error reporting program, since for a variety of reasons, there can be resistance to report medical errors. This is a problem that needs to be addressed. However, this problem exists whether one has a total medical error rate measure or a selection of individual medical error rate measures.

Specific measures are still needed

One can only reduce the total medical error rate goal by reducing individual medical error rates. Specific individual measures that make up the total can receive focus according to their ranking in a Pareto chart and this the fastest way to reduce the overall error rate. Note that the top individual measures from a Pareto chart may not correspond to a pre-designated list of measures. For example, one suggested pay for performance measure is the percent of patients receiving aspirin after undergoing coronary bypass surgery. But it is possible for a hospital that this measure meets it goal, but that other measures that lead to morbidity and mortality are high in the Pareto chart and are not on a pay for performance schedule.


  1. National Quality Measures Clearinghouse, see
  2. Hilborne, L. Developing a Core Set of Laboratory Based Quality Indicators.
  3. Krouwer, JS Setting Performance Goals and Evaluating Total Analytical Error for Diagnostic Assays. Clin Chem 2002;48:919-927.
  4. Krouwer, JS. A Critique of the GUM Method of Estimating and Reporting Uncertainty in Diagnostic Assays Clin Chem 2003;49:1218-1221.
  5. Krouwer, JS and Monti, KL. A Simple Graphical Method to Evaluate Laboratory Assays, Eur. J Clin Chem and Clin Biochem 1995;33, 525-527.
  6. Krouwer, JS. Using a Learning Curve Approach to Reduce Laboratory Error, Accred. Qual. Assur., 7: 461-467 (2002).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: