FDA’s secret database

November 7, 2019

Ok, the title’s a little dramatic. I’ve been exploring the adverse event database (MAUDE) for glucose meters, which is a publicly available database. But some records were a summary of events. This shouldn’t be since individual events are supposed to be entered.

What happened was there was an alternative summary reporting (ASR) system in place, which was not available to the public.

This was exposed in March 2019 by some excellent reporting by Christina Jewett.

In June of 2019, the FDA described ASR and said it has ended with all data now in MAUDE (hence the summary data I found). They start out by saying “In the spirit of promoting public transparency...” Well to be transparent, don’t have secret databases in the first place! The problem is there was never any reasonable explanation for why this data was not available to the public. For example, one reason stated was “ASR reports were not made publicly available because they were not submitted in a format compatible with the public database.” Ok, so change the format!

The real basis for the ASR database is still a mystery.

Manufacturers classify events differently

October 20, 2019

In continuing to analyze the adverse event database – this time for continuous glucose monitors (CGM), I noticed something very strange. There are two major manufacturers each with their latest product. This is the result. To recall, events are classified as either M=malfunction, IN=injury, or D=death.

Manufacturer Percent events classified as injury
A 94%
B 0.2%

But the text for these events is similar. How can this be?

Analyzing the adverse event database

October 7, 2019

Looking at glucose meter adverse events from the FDA database, an SQL query showed that 73% of the time, a form of the verb allege was used by manufacturers to describe the event, as in “the user alleged that …” This is I guess one way of acknowledging that these events are unverified.

Flaws in the ISO 15197 standard (for glucose meters)

August 13, 2019

Having an occasion to read the ISO 15197 standard (for glucose meters) I notice the statements:

One of the reasons allowed to discard data is: “the blood-glucose monitoring system user recognizes that an error was made and documents the details”

This makes ISO a biased standard because in the real world there will be user error which generates outlier data.

And compounding things is this statement:

“Outlier data may not be eliminated from the data used in determining acceptable system accuracy, but may be excluded from the calculation of parametric statistics to avoid distorting estimates of central tendency and dispersion.”

The problem is outliers that are representative of what happens in the real world should not be thrown out to help statistics such as regression and precision from being distorted. Rather these statistics should not be used. An error grid is a perfectly adequate statistic to handle 100% of the data.

Confidence intervals not p values

July 31, 2019

The New England Journal has finally changed its mind and recommends confidence intervals instead of p values. The article is here.

My thoughts on QC

June 15, 2019

Having just read a Clin Lab News article on QC which IMHO is misleading, here are my thoughts.

The purpose of QC is to determine whether a process is in control or not. In clinical chemistry, the process is running an assay. An out of control process is undesirable because it can yield unpredictable results.

QC by itself cannot guarantee the quality of patient results, even when the process is in control. This is because QC does not detect all errors (example an interference).

The quality of the results of an in control process is called the process capability of the process (e.g., its inherent accuracy). QC cannot change this, regardless of the QC rules that are used.

QC is like insurance, hence cost should not be considered in designing a QC program. That is, regardless of how low risk a failure mode is, one should never abandon QC.

Although running more QC can detect an out of control process sooner, any QC program should always protect patient results from being reported when an out of control condition is detected. Risk is not involved.

Stakeholders that participate in performance standards

May 11, 2019

Performance standards are used in several ways: to gain FDA approval, to make marketing claims, and to test assays after release for sale that are in routine use.

Using glucose meters as an example…

Endocrinologists, who care for people with diabetes, would be highly suited to writing standards. They are in a position to know the magnitude of error that will cause an incorrect treatment decision.

FDA would also be suited with statisticians, biochemists, and physicians.

Companies through their regulatory affairs people know their systems better than anyone, although one can argue that their main goal is to create a standard that is as least burdensome as possible.

So in the case of glucose meters, at least for the 2003 ISO 15197 standard, regulatory affairs people ran the show.