EPCA-2 Update number 6

March 19, 2015

jail

For no particular reason, I searched for Dr. Getzenberg in Google. To recall about previous entries on this blog, search for EPCA-2 on this blog. (there is a search form on the top right of this blog). I found two rather different entries in Google.

One deals with the seventh retraction for articles written by Dr. Getzenberg

Another talks about awards distinction and how he is a senior leader in oncology and urology.


Hemoglobin A1c quality targets

March 16, 2015

KPSF3edp

There is a new article in Clinical Chemistry about a complicated (to me) analysis of quality targets for A1c when it would seem that a simple error grid – prepared by surveying clinicians would fit the bill.

Thus, this paper has problems. They are:

  1. The total error model is limited to average bias and imprecision. Error from interferences, user error, or other sources is not included. It is unfortunate to call this “total” error, since there is nothing total about it.
  2. A pass fail system is mentioned, which is dichotomous and unlike an error grid which allows for varying degrees of error with respect to severity of harm to patients.
  3. A hierarchy of possible goals are mentioned. This comes from a 1999 conference. But there is really only one way to set patient goals (listed near the top of the 1999 conference): namely; a survey of clinician opinions.
  4. Discussed in the Clinical Chemistry paper is the use of biological variation based goals for quality targets. Someone needs to explain to me how this could ever be useful.
  5. The analysis is based on proficiency survey materials, which due to the absence of patient interferences (see #1) is a subset of total error.
  6. From I could tell from their NICE reference (#11) in the paper, the authors have inferred that total allowable error should be 0.46% but this did not come from surveying clinicians.
  7. I’m on-board with six sigma in its original use at Motorola. But I don’t see its usefulness in laboratory medicine compared to an error grid.

User error matters!

March 12, 2015

Frozenedp copy

I’ve written before that total error means error from any source not just analytical error. Thus, if a clinician makes an incorrect treatment decision because the test result is wrong due to user error, it is little consolation to know that the analytical system was ok.

All of this applies to SMBG (self-monitoring blood glucose) where the treating “clinician” and user are the patient.

A Letter in Clinical Chemistry (subscription required) shows that whereas 9 out 10 glucose meters met performance standards when the tests were performed by expert users, only 6 out of 10 meters met standards when the tests were performed by routine users.

Of interest as well is that the authors cite as performance standards both the ISO 2013 standard and the suggested FDA draft performance standard from 2014.


The CLSI document EP19

February 6, 2015

preflightedp

I had occasion recently to see a final draft of CLSI EP19 – which is a framework for using CLSI evaluation documents. I may review this when it is officially released but here are three comments.

  1. There is a cause and effect diagram in EP19 listing assay attributes (precision, interferences, and so on) and the CLSI documents that are used to evaluate these attributes. I published (1) a diagram in 1992 (attributes only) and later adapted my diagram to include the associated CLSI documents and this diagram appeared in a 2002 publication (2). In 2005, I proposed to CLSI that the diagram appear in all CLSI evaluation standards – it is in EP10, EP18, and EP21 although it is not in more recent documents. Now I know that CLSI is a consensus organization whereby documents are a collaborative effort and my diagram has been modified further but there should be a citation to my prior work and there isn’t.
  2. In the clinical performance section, there is no mention of error grids (EP27). In fact, a search of EP19 shows that EP27 is never mentioned. This is most strange. After all, error grids are used to determine if an assay is good enough which is the whole point of an evaluation! Error grids are part of the FDA CLIA waiver recommended guideline and fundamental in glucose meter evaluations. I don’t understand how in years of document development of EP19, EP27 has received zero mention. I did check the list of CLSI publications on their website to make sure that EP27 is still for sale.
  3. There is mention that assay claims should clear – that’s it! no more details are given. Sadly, there was an entire document about uniformity of claims (EP11) that was killed by CLSI management after one manufacturer threatened to quit.

References

  1. Krouwer JS Estimating Total Analytical Error and Its Sources: Techniques to Improve Method Evaluation. 1192, Arch Pathol Lab Med., 116, 726-731.
  2. Krouwer JS Setting Performance Goals and Evaluating Total Analytical Error for Diagnostic Assays. Clin. Chem., 48: 919-927 (2002).

It’s up to the lab director – not really

February 4, 2015

post flight cirrusedp copy

I have previously commented that many CLSI evaluation standards at some point ask the question “is the assay performance good enough” and answer that question with “it’s up to the lab director.”

The problem is that lab directors are not clinicians and do not treat patients. Note that most lab directors are either PhD clinical chemists or pathologists and although pathologists are MDs, they are not clinicians because they do not treat patients.

Of course, lab directors do have a great deal of knowledge about assay performance but in my experience – especially in working on CLSI standards – lab directors tend to focus on analytical errors whereas only total error is of importance to clinicians and the source of errors that contribute to total error is a combination of analytical, pre- and post-analytical error.

So how should the “is it good enough” question be answered? An example appeared recently in the literature (1) where clinicians were surveyed as to what size glucose meter errors would start to cause problems for diabetics under several scenarios. The results provided limits for a glucose meter error grid. Note that there was no attempt to identify error limit sources – the limits simply reflect the observed error, regardless of its source.

Reference

  1. Klonoff DC, Lias C, Vigersky R, et al The surveillance error grid. J Diabetes Sci Technol. 2014;8:658-672.

Reply to Letter Published

January 30, 2015

KMVYsb

[The photo is Martha’s Vineyard snow removal operations taken during a flight two days after two feet of snow fell in the Boston area.]

I had mentioned that a Letter to the editor to Clinical Chemistry had been accepted. It is now online with a reply to my Letter (subscription required).

I had previously mentioned in this blog how the editor of Clinical Chemistry is not fond of letters and replies so any thought of me replying to the reply would be a lost cause. Not that I would anyway. The authors who replied were kind in their comments and I have only one comment which I make at the end of this entry.

One cynical comment about these glucose meter models that relate precision and bias to total error is that you can make beautiful contour graphs because there are three variables. If you add interferences, no more simple contour graphs.

But what does it take to add interferences to the glucose meter (simulation) model. First one needs to list all candidate interfering substances and test them. Manufacturers have already done this but unfortunately, don’t try to use the information in the package insert. You can thank CLSI EP7 for this which allows a manufacturer to say compound XYZ does not interfere – if the manufacturer finds that the interference is less than 10% and the goal was 10%. So there could be a bunch of compounds that interfere but at levels less than 10%. This means that unless one can access the original manufacturing data, one would have to do over all of the interference studies. Then one needs the patient distribution of the concentration of each interfering substance. With this information one can randomly select a concentration of each interfering substance and apply the appropriate equation to generate a bias.

Thus, simulations, while still models and subject to the possibility of being incorrect, can require a significant amount of work.

My comment to the authors who replied to my Letter deals with their statement: “This is exactly the reason we advised in our work to adopt accuracy requirements more stringent than those resulting from simulations.” A similar statement was made by Boyd and Bruns back when I similarly critiqued their model. Now for sure, if the required bias is reduced and interferences are small, this will work because the total error will meet goals. The problem is, one has no knowledge of the bias contributed by interferences. And perhaps more importantly, this strategy will not work to prevent errors in the D zone of an error grid. I mention in my last post that with a bias of zero and a CV of 5%, one could get a D zone error if the observation is 80 standard deviations away. This will not happen anytime soon, but a gross interference is possible.


Published and Bad Model

January 2, 2015

debate

I complained about two glucose modeling papers and an accompanying editorial in the December issue of Clinical Chemistry. My Letter to the editor about one on the papers has been accepted in Clinical Chemistry.

Although not in the Letter, here’s another example of why modeling glucose meter error using average bias plus multiples of the standard deviation (e.g., sampling from a Gaussian distribution) can be misleading. Say truth is 50 mg/dL, which is hypoglycemic and the meter reads 200mg/dL, which is hyperglycemic. This would be a serious error because the provider (or patient if self-monitoring) would administer insulin, when in fact sugar is needed.

But in terms of modeling, say the bias is zero and the glucose CV is 5%. This means the sd at 50 is 2.5 mg/dL. Now to get a value of 200 due to imprecision requires 80 standard deviations! Using a spreadsheet, I can’t get this probability – however, for 30 standard deviations, the probability has 200 zeros to the right of the decimal point followed by a one. In other words not going to happen.

But such errors do occur – albeit rarely – but much more frequently than an 80 standard deviation error.


Follow

Get every new post delivered to your Inbox.