On March 16-17, there was a public FDA/CDRH meeting on glucose meters. Here are some comments based on the transcripts of the March 16th meeting.
Jeffrey E. Shuren, M.D., J.D., Director, Center for Devices and Radiological Health – In the US, there are 24 million diabetics of which 30% take insulin and use glucose meters. FDA receives notice of about 12,000 adverse events about glucose meters each year.
Comment – This equals 7.2 million glucose meter users. If one tests once per day, the adverse event rate is 0.17%. If one tests four times per day, the adverse event rate is 0.04%. In either case, there will be an adverse event about once every two years.
Patricia Bernhardt, M.T.(ASCP), Office of in Vitro Diagnostic Device Evaluation and Safety – There is a evaluation of user performance required by the FDA that is distinct from the evaluation of accuracy. The comparative method for accuracy is typically the YSI meter whereas the comparative method for user performance is the candidate meter, with the evaluation being different users (health care professionals vs. lay users).
Comment – It makes no sense to separate these two studies, or putting things another way, there is no attempt to combine the information from the two studies to arrive at a total error estimate.
Mitchell Scott, Ph.D., Washington University School of Medicine, St. Louis, Missouri – They do 600,000 glucose results a year. If 95% are within limits that means 30,000 results can be anywhere. “So I think we really need to think about this 5 percent of the values just sort of being unclassified.”
Comment – Finally, we’re starting to see others point out the problem with only specifying 95% of the results. George Cembrowski and I mention this (1) and I mention this also in my Letter (2).
From August of 2002 in an email I wrote to the leader of the group working on ISO 15197:
“Along these lines, the section about total error is excellent. Yet, I have a concern. In EP21, there is mention that total error by itself is inadequate. One must have information about outliers. This seems to be missing in my (quick) reading of the ISO document – maybe I missed it. I mentioned this problem formally in a recent Clinical Chemistry paper (this year 919-927). Thus for glucose meters, if 4.9% of the values are worse then goals, the meter is ok (ISO) but 4.9% outliers is a huge problem. Of course, one has the Lifescan problem for a real example – you probably have a lot of information on this. I suspect their outlier rate was less than 1% but this was still a huge problem.”
I got a thank you for my email, but no action about the point I made. The reference above is (3).
Mitchell Scott, Ph.D., Washington University School of Medicine, St. Louis, Missouri – Dr. Scott has data in his hospital that show that 0.8% to 1.2% of glucose results are repeated within 15 minutes, indicating that the operator did not believe the results. The mean difference between the two results is 84 mg/dL.
Barry Ginsberg, M.D., Ph.D., Diabetes Consultants, Wyckoff, New Jersey – Virtually every small study shows great data (no values in zones C or higher in an error grid) but when you look at large studies from manufacturers, there is a 0.03 to 0.3% outlier rate with values 100 to 200 mg/dL different.
Comment – These outlier rates are close to the adverse event rate reported by the FDA.
David B. Sacks, M.D., M.B., Ch.B., Harvard Medical School and Brigham and Women’s Hospital, Boston, Massachusetts – “And as has been mentioned, the other recommendations, CLSI or ISO, 95 percent of results, and this has been discussed in some detail, but I’m going to discuss it again. So, as was suggested by Dr. Scott, I think we need an addendum to meet a performance criteria, because 5 percent are excluded from accuracy criteria. And these values can be essentially anything. So, if you do the calculation, if a patient does self-monitoring of blood glucose four times a day, you’d expect one result to be outside the analytical limit every five days. The problem is the patient won’t know which this one result is, which is outside. So that’s very, very frequent. So I think we need to define criteria that include these 5 percent of values.”
Comment – This sounds familiar, lol – from George and my reference 1 – “The problem with this [ISO] standard is simple: up to 5% of the results can be medically unacceptable. Consider what this means for SMBG. If a subject tests his (her) blood glucose four times daily, then on average there could be a medically unacceptable result every 5 days (once per 20 measurements).”
Steve Brotman, M.D., J.D., Advanced Medical Technology Association (AdvaMed) – “The standard currently governing blood glucose meters for self-testing, ISO 15197, itself recognizes the importance of usability improvement. Specifically, it notes that the goals for performance criteria should be weighed against the capabilities of current self- monitoring devices. Furthermore, the standard notes that care should be taken implementing performance requirements that cause manufacturers to focus design improvements on analytical performance at the expense of other important attributes, such as greater convenience and greater compliance. Thus, the standard acknowledges the careful balance of these factors and the minimum acceptable device performance for glucose meters for self-testing. The standard supports performance improvements beyond analytical performance, such as advances that reduce dependence on user technology, otherwise referred to as patient usability.”
Comment – Sounds like double talk. The ISO standard does not measure total error where error equals accuracy + user error.
Barry Ginsberg, M.D., Ph.D., Diabetes Consultants, Wyckoff, New Jersey – “But when you look at the ISO standard and ask, So if the actual blood glucose is 70, at the various ISO standards, what is my 95 percent confidence limits on what I’m going to see? And so at the ISO standard of 20 percent, that 70 is somewhere between 55 and 85.”
Comment –Dr. Ginsberg does not understand the standard. The 95 percent is not a confidence interval, it’s saying what percentage of the population should be contained within the stated limits.
Barry Ginsberg, M.D., Ph.D., Diabetes Consultants, Wyckoff, New Jersey – His experience is that four out of five do not wash their hands before testing and this can drastically affect results, especially considering the reduced sample volume in today’s meters.
Comment – Interesting point.
David C. Klonoff, M.D., F.A.C.P., Mills-Peninsula Health Services, San Mateo, California – A regulatory standard means this is something that’s achievable. When doctors make requests or standards, we tend to talk about what’s needed. Those are clinical standards.
Comment – Also in reference 1.
Diane Rutherford Ken Block Consulting Dallas, TX – She makes the point that high values are repeated but values we like may not be repeated, but also may not be accurate.
Comment – Interesting point. This is a type of selection bias and the overall effect is to underestimate the rate of outliers.
Richard Melker, University of Florida College of Medicine – “The last thing I want to say is what you said about hand washing, which I think is really interesting. Because if you wash your hands and you don’t dry them really well, you get low glucose readings, because you have water on your hands. So not washing your hands is a problem, and washing your hands and not drying — which takes a fair amount of time to do properly — is a problem. The other problem with not drying your hands completely is if you open the vial and you take a glucose test strip out while your hands are wet, you can ruin all the other test strips in that vial. Nobody teaches patients about any of these issues, so have at any of them.”
Comment – Interesting point. Mr. Melker is a type I diabetic.
Gary L. Myers, Ph.D., Division of Laboratory Sciences at the Centers for Disease Control and Prevention, Atlanta, Georgia – “There is no current consensus exists among manufacturers about the most appropriate way to publish guidelines on how a interfering substance is affecting a particular method.”
Comment – CLSI had a guideline to address this – EP11 (Uniformity of Claims) but (some) manufacturers with the help of AdvaMed killed it.
Note: All references were available before the FDA meeting.
- Krouwer JS and Cembrowski GS. A review of standards and statistics used to describe blood glucose monitor performance. Journal of Diabetes Science and Technology, 2010;4:75-83.
- Krouwer JS. Wrong thinking about glucose standards. Clin Chem, in press. Available online.
- Krouwer JS Setting Performance Goals and Evaluating Total Analytical Error for Diagnostic Assays. Clin Chem 2002;48:919-927.