Measuring things, again

August 29, 2009

star

DB responded to my entry about “Just because it’s not easy to measure …”,  which gave me mixed feelings. I am thankful that he acknowledged that his statement was not what he meant but I must admit that I am in awe of DB – hence I was uncomfortable that he would apologize.

Since I follow his blog, my first instinct was to comment that I agreed with the comment made by “Curious” in the entry that had the quote. But DB didn’t respond to Curious’s comment so I left it alone.

In my field of laboratory medicine, I will continue – having started about 20 years ago – to advocate for measuring everything that’s important. This includes not just measuring the easy things like precision and bias but also the difficult things like rare interferences or user errors.

I added the quote from DB, because it’s important for guidelines. With healthcare reform, we can expect more rather than fewer guidelines in part fueled by the 1.1 billion for comparative effectiveness research.  Obama said:

“The point is we want to use science, we want doctors and medical experts to be making decisions that all too often right now are driven by skewed policies, by outdated means of reimbursement, or by insurance companies.“

I became aware of this from a Dr Rich entry. Comparative effectiveness research will be based on data, analyzed to yield measurements, which will turn into conclusions and recommendations. There is a famous statistical example about how difficult it is to measure things. Youden (1) compiled 15 different estimates of  the astronomical unit from scientists who estimated that quantity over the years 1895–1961. The confidence interval constructed by every scientist did not overlap the confidence interval of his predecessor. The difficulties are only greater in medicine. Just getting agreement on definitions is important as I cited an example for side effects of prostatectomy where urinary incontinence was defined as using greater than 3 pads per day implying that less than 3 pads per day = continence. Maybe urologists could agree with that definition, but I don’t think patients would.

References

 Youden WJ. Enduring values. Technometrics 1972;14:1–11.


Not so fast, Point-of-Care

August 25, 2009

stop

The American Diabetes Association (ADA) has revised its recommendation for diagnosis of diabetes and now recommends using hemoglobin A1c to diagnose diabetes (1). They also say:

A1C tests to diagnose diabetes should be performed using clinical laboratory equipment. Point-of-care instruments have not yet been shown to be sufficiently accurate or precise for diagnosing diabetes.

Maybe this is a trend to slow down the adoption of point-of-care (POC) assays.

Scott, et. al. speculate that one of the reasons that tight glycemic control (TGC) in ICUs has been dropped as a guideline is that the use of POC glucose meters (meaning less accurate) as opposed to laboratory assays may have contributed to the adverse findings of TGC.

This will also mean that the CLSI standard POCT09-P Selection Criteria for Point-of-Care Testing Devices will need to be revised since although they suggest conducting performance evaluations, their examples of benefits of POC assays now include two examples where the accuracy of POC tests have either been rejected or questioned (1-2). POCT09-P also cites the benefit of a POC troponin assay, where performance was tested by surveying clinicians – hardly a rigorous test (3) and not conforming to their own recommendation for a real evaluation.

References

  1. International Expert Committee Report on the Role of the A1C Assay in the Diagnosis of Diabetes. Diabetes Care 2009;32:1327-1334.
  2. Scott MG, Bruns DE, Boyd JC, and Sacks DB. Tight glucose control in the intensive care unit: Are glucose meters up to the task? Clin Chem 2009;55:18-20.
  3. Lee-Lewandrowski E, Corboy D, Lewandrowski, K, Sinclair J, McDermot S,  Benzer, TI. Implementation of a Point-of-Care Satellite Laboratory in the Emergency Department of an Academic Medical Center. Archives of Pathology and Laboratory Medicine 2003;127:456–460.

Just because it’s not easy to measure …

August 24, 2009

measure

There have been several objections to measuring errors that are not as easy as calculating a standard deviation.

One comment was – pre-analytical error is important but can’t be measured in a method comparison protocol. It needs to be handled by risk management.

Similar arguments were made during a meeting for measurement uncertainty, where it was suggested that large but rare analytical errors be handled by risk management.

DB writes:

I favor limited guidelines, but not measurement.  Measurement has too many unintended consequences.

ISO 15197, a standard for home glucose meters has a specification for total analytical error which does not include errors due to pre-analytical error. User errors are to be evaluated separately and without stating any analysis procedure:

Results shall be documented in a report

Unfortunately, risk management as used by these people means sweeping these problems under the rug and is the same as DB’s advice about not measuring things or the ISO guidance.

It’s time to start measuring errors from all sources – it’s possible and necessary.


Interference reporting for diagnostic assays

August 17, 2009

barrier

Non specificity in diagnostic assays (interferences) is a problem. For example, using the search tool in Clinical Chemistry from 2006 to date, yielded 45 references. See reference 1 for one of the 45 (1).

Interferences can be thought of in two ways:

  1. a particularly bad interference will cause a huge error in a result
  2. a result can also exhibit smaller errors either by a single interference or by a combination of interferences, whose net effect is a smaller error.

Manufacturers study extensive lists of candidate interfering substances. Unfortunately, the way that many manufacturers report the results of interfering studies in package inserts is misleading with respect to case 2 above.

Manufacturers often cite a CLSI document – EP7A2 – which states that a claim can be that the following compounds: “were found not to interfere at the concentrations indicated.” But this only means that a bias of less than 10% was found. Later in the document, alternative bias claims are presented, which added to the first claim are:

  1. Substance did not interfere (< 10%)
  2. The observed amount of bias due to interference
  3. the maximum amount of bias due to interference that could occur

It’s not surprising that manufacturers tend to choose method 1 for claims. Besides being misleading, claim 1 is wrong statistically, since it amounts to stating that the null hypothesis has been proved, which is impossible.

There is an opportunity for CLSI to revisit its canceled standard – EP11 – Uniformity of Claims

References

  1. Dimeskia G, Jones B and Ungerer JPJ Interference from Rose Bengal with Total Bilirubin Measurement. Clin Chem 2009 55: 1040-1041.