Proposed improvements to the Diabetes Technology Society surveillance protocol

I previously blogged about flaws in the Diabetes Technology Society surveillance protocol. I turned this entry into a commentary which has been accepted and should appear shortly in the J Diabetes Sci Technol.

Antwerp talk about total error

Looking at my blog stats, I see that a lot of people are reading the total analytical error vs. total error post. So, below are the slides from a talk that I gave at a conference in Antwerp in 2016 called The “total” in total error. The slides have been updated. Because it is a talk, the slides are not as effective as the talk.

 

 

TotalError

Do manufacturers always publish that their glucose meter is best?

After reading an evaluation article where the conclusion was that the manufacturer’s glucose meter was best, I went through some journals to see how often this happens. I searched 2 journals through 2012-2016.

To be included in the list below, the article had to meet the following criteria:

• The study was sponsored by a manufacturer
• There were 2 or more meters, not all by made by the sponsor

The results are shown below. Eight articles met the criteria. Some articles were clear whereby the article’s conclusion was that the manufacturer’s glucose meter was best. In other articles, I had to look through the data. If the manufacturer’s glucose meter was best, the score was 1, if some other manufacturer’s glucose meter was best, the score was 0, and in one case, it was a tie so the score was 0.5. The N refers to the number of meters in the article.

Reference	Company	Meter	Winner	N	Score
J Diabetes Sci Technol
2016 1316-1323	Sanofi	BGStar / iBGStar	BGStar / iBGStar	5	1
2015 1041-1050	Bayer	Contour	Contour Acc-Chek	4	0.5
2013 1294-1304	Bayer	Contour	Contour	5	1
2012 1060-1075	Roche	Accuchek	None declared best but Freestyle was best	43	0
2012 547-554	Abbot	Optimum Xceed	Optimum Xceed	6	1
Diabetes Technology and Therapeutics
2014 8-15	Bayer	Contour	Contour	5	1
2014 113-122	Ypsomed	Mylife Para / Mylife  Unio	None declared best	12	0
2012 330-337	Abbot	Freestyle	Freestyle	5	1

So 69% of the time the manufacturer’s glucose meter was best.

Help with sigma metric analysis

I’ve been interested in glucose meter specifications and evaluations. There are three glucose meter specifications sources:

FDA glucose meter guidance
ISO 15197:2013
glucose meter error grids

There are various ways to evaluate glucose meter performance. What I wished to look at was the combination of sigma metric analysis and the error grid. I found this article about the sigma metric analysis and glucose meters.

After looking at this, I understand how to construct these so-called method decision charts (MEDX). But here’s my problem. In these charts, the total allowable error TEa is a constant – this is not the case for TEa for error grids. The TEa changes with the glucose concentration. Moreover, it is not even the same at a specific glucose concentration because the “A” zone limits of an error grid (I’m using the Parkes error grid) are not symmetrical.

I have simulated data with a fixed bias and constant CV throughout the glucose meter range. But with a changing TEa, the estimated sigma also changes with glucose concentration.

So I’m not sure how to proceed.

The Diabetes Technology Society (DTS) surveillance protocol doesn’t seem right

The Diabetes Technology Society (DTS) has published a protocol that will allow a glucose meter to be tested to see if the meter meets the DTS seal of approval. This was instituted because for some FDA approved glucose meters, the performance of post release for sale meters from some companies did not meet ISO standards.

Before the DTS published their protocol, they published a new glucose meter error grid – the surveillance error grid.

But what I don’t understand is that the error grid is not part of the DTS acceptance criteria to gain the DTS seal of approval. (The error grid is plotted as supplemental material). Basically, to get DTS approval, one has to show that enough samples have differences from reference that fall within the ISO 15197:2013 standard. To be fair, the ISO standard and the “A” zone of the error grid have similar limits, but why not use the error grid, since the error grid was developed by clinicians whereas the ISO standard is weighted by industry members. And the error grid deals with results in higher zones.

Moreover, the DTS does not deal with outliers other than to categorize them – their presence does not disqualify a meter from getting DTS acceptance as long as the percentage of results within ISO limits is high enough.

So if a meter has a 1% rate of values that could kill a patient, it could still gain DTS seal of approval. This doesn’t seem right.

 

Book about noninvasive glucose meters

Noninvasive glucose meters are the Holy Grail in glucose testing. To be able to get a glucose value without a finger stick would be a tremendous benefit to the millions of people who have to test themselves several times each day.

So there have scores of scientists who have worked on the problem, backed by diagnostic companies since the profit potential is huge.

I remember while at Ciba Corning, attending a lecture on near infrared spectroscopy given by a professor whom I think we were supporting to try to come up with a noninvasive glucose meter.

On a website devoted to diabetes, I became aware of a book which chronicles the quest for a noninvasive glucose meter. It is recent (2015 publication date), free, and written by a former chief scientific officer and VP of LifeScan who has been involved in this search for years.

I found it fascinating.

Test error and healthcare costs

Conventional wisdom says that regulatory authorities approve assays that have the highest quality, meaning that the errors are small enough that no or little harm will arise because a clinician makes a wrong medical decision based on test error.

It is also true, although not talked about, that in most countries healthcare is rationed – the cost of treating everyone with every possible treatment is too high.

So here’s a hypothetical example using glucose meters.

First, we start out with the status quo for existing glucose meter quality and assume that on average, across all tests there will be some harm due to glucose meter error. The percentage of tests that harm people is unknown as is the range of harm but assume that these can be ascertained and do occur.

As for the hypothetical part…

There are 2 new glucose meters seeking approval

Meter A costs 100 times as much as current meters and is guaranteed to have zero error, as it is a breakthrough technology. Its use will reduce patient harm due to test error to zero.

Meter B costs 100 times less than current meters but isn’t quite as accurate or reliable. Patient harm will increase with the use of meter B.

If meter A is approved, because of healthcare rationing, costs will have to be transferred from other parts of healthcare to pay for meter A.

If meter B is approved, costs can be transferred from glucose meter testing to other parts of healthcare.

The point is not to try to answer whether meter A or meter B should be approved, but to illustrate that the cost issues associated with healthcare policy always exist but are rarely discussed.