Simulating clinical outcomes to establish error limits for glucose meters

David Klonoff summarized (subscription required) the recent FDA glucose meetings. One of the talks and papers mentioned by David is a simulation by Breton and Kovatchev on how glucose errors affect clinical outcomes. This study is well worth reading.

There are two types of error limits used.

Regulatory limits are used to determine whether glucose meters can be sold. They are often said to be based on clinical grounds (such as ISO 15197), but in reality their main basis is existing technology and most meters will meet these limits.

Clinical limits are based on clinicians’ ideas of what’s good enough clinically. These limits may be beyond the existing technology. For example, in 1987, the ADA glucose limits were < 10% for analytical plus user error.

There are also several stakeholders.

Regulators are providers of regulations.

Manufacturers are consumers of regulations and want easy to meet regulations.

Clinical laboratories are consumers of regulations and also want easy to meet regulations (glucose meters are used off-label in hospitals and are managed by clinical laboratories).

Clinicians establish clinical error limits. The problem with clinician based error limits is that they are based on anecdotal data rather than designed studies. Designed studies are impossible to set up since as Breton and Kovatchev say, it would be unethical to allow glucose errors to occur.

Patients want the lowest error limits possible.

Simulations are important because they allow for glucose error to be varied while one observes the likely medical decisions made and their outcomes. The validity of the conclusions depend of course on the underlying models and assumptions in the simulations – something beyond what I can currently comment on.

However, I did critique an earlier simulation by Boyd and Bruns, mentioned by Breton and Kovatchev, which I will now briefly review.

Boyd and Bruns modeled glucose total error as average bias plus imprecision. I have argued many times that this underestimates the true total error.

The problem is that Boyd and Bruns implied that with average bias and imprecision at stated levels, one would get a specific total error and corresponding medical error. But since their method underestimates total error, the average bias and imprecision do not relate to the level of medical problems. Moreover, regulators or clinicians might specify limits for average bias and imprecision based on these studies. 

This is not an issue for Breton and Kovatchev because they are using imprecision with zero bias, not total error. However, it is possible that some will nevertheless misinterpret things and associate the Breton and Kovatchev results as being based on total error.

The hard part of these studies is modeling a diabetic patient. A much easier task is to model and simulate glucose error. One should start with the Lawton model – described here and reviewed in the Appendix. One has to also add user error to the Lawton model to simulate what routine glucose meter users will experience for glucose error. Alternatively, one could directly observe glucose meter total error by a suitable experiment as described in CLSI EP21A, soon to be replaced by EP21A2.

If this is not done, the simulation studies will not be faithful (if they claim they are simulating total error) and neither will the studies to evaluate meter performance.


The Westgard model is:

%TE = %Bias + 1.96(CVT) where,

%TE is percent total error
%Bias is percent average bias
CVT is total coefficient of variation due to imprecision.

The Lawton model is:

%TE = %Bias + 1.96(CVT) + 1.96(CVRI) where,

CVRI is the total coefficient of variation due to random interferences.

For glucose meters, CVRI is an important term. For example hematocrit interferes with glucose meters. Manufacturers state an allowable hematocrit range for glucose meters. But glucose error occurs within this hematocrit range although the error magnitude is considered “acceptable” by manufacturers. And hematocrit is just one interfering substance.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: