What needs to be measured to ensure the clinical usefulness of an assay

July 19, 2016


I was happy to see an editorial which IMHO states the required error components that need to be understood to ensure the clinical usefulness of an assay. Of course bias and imprecision are mentioned. But in addition, the author mentions freedom from interferences and pre and post analytical errors.

One can ask don’t interferences and pre and post analytical errors cause bias? Since the answer is yes, then why do these terms need to be mentioned if it was already stated that bias is to be measured. The reason is the way bias is measured in many cases will fail to detect the biases from interferences and pre and post analytical errors.

For example, if regression is used, average bias will be estimated, not the individual biases that can occur from interferences.

If σ is estimated, this usually involves bias measured from either regression or from quality control samples so again interference biases don’t get counted.

Finally, most of the studies are done in ways in which pre and post analytical errors have been minimized – the studies are performed outside of the routine way of processing patient samples. Hence, to ensure the clinical usefulness of an assay, one must construct protocols that measure all of the error components mentioned in the first paragraph.

No surprise that Instructions For Use (package inserts) are weak

July 16, 2016



A recent letter in Clinical Chemistry (subscription required) talks about package inserts from manufacturers (also called instructions for use (IFU). The letter says that manufacturers’ IFUs often do not follow CLSI guidelines with respect to hemoglobin interference.

This should come as no surprise – here’s why.

The authors cite FDA regulations which state: “Limitation of the procedure: Include a statement of limitations of the procedure. State known extrinsic factors or interfering substances affecting results.”

This regulation leaves a lot of leeway as to what should appear in the IFU.

So the authors say that CLSI guidelines (C56 and EP7) are not followed. One should understand that CLSI guidelines are not regulations. No manufacturer has to follow them. Moreover, these guidelines are often manufacturer friendly as manufacturers dominate the committees who prepare the documents. For example, the authors cite C56 which has an example for how to report when there is no hemoglobin interference for glucose. The table contains the concentration of hemoglobin tested, two glucose levels, and bias < 10%.

This is messed up! If bias found were 9%, this CLSI guideline is suggesting that it is ok to say there was no bias!

So even if manufacturers followed CLSI guidelines, maybe this wouldn’t be so good.

To understand why a CLSI document would permit the claim “no bias” when 9% bias was found…

CLSI prides itself on equal influence of “professions” (e.g., clinical chemists in hospitals), “government” (e.g., FDA), and “manufacturers” (people in industry). But the industry people are largely from regulatory affairs and their role on committees has often been an obstructionist role. Basically, the industry – like industries in other fields – does not want to be regulated at all, so if there has to be a standard, the regulatory people try to make it as industry friendly as possible.

As an example of the obstructionist role, consider EP7. It was initially published as a “P” (proposed) version in 1986. Only “A” (accepted) versions are accepted by the FDA. So how long did it take for this standard to go from P to A: 16 years! (initially published in 2002.) It wasn’t until I was the chair of the Evaluation Protocol Committee that this project got moving faster than a snail’s pace and was finished.

And then there was the CLS standard EP11 – Uniformity of Claims. It was intended to be a guideline for IFUs. It’s hard to say if this standard would help since it could also be ignored. It was published as a “P” document in 1996. CLSI management (who was pressured by industry) pressured me to cancel it – I didn’t but they did and it was not advanced and is no longer available.

Finally, I can’t speak about other companies, but in the company that I worked for, IFUs were prepared by the marketing department.

Comments about bias

June 5, 2016


I was once again interested in a title I saw in Clinical Chemistry – this one was: An unbiased view of bias. Ok, I get it – this is a humorous piece. But bias is an important topic and Clinical Chemistry is not a very friendly journal to statistical issues.

So here, I describe bias in method evaluations, based on my article that was recently published.

First, some comments about bias.

Bias when present does not guarantee that the results will be different than the case with no bias. The problem is you just don’t know. For example, if you preselect samples for a method comparison instead of randomly selecting samples, the differences observed may not be affected due to the preselection.

Bias is not necessarily an evil strategy concocted by a manufacturer – many biases are unavoidable.

In my paper, I cover several sources of bias – one will be mentioned here.

Reagent bias – In many regulatory evaluations, several (usually 3) lots of reagents are used to demonstrate that lot-to-lot reagent bias is small. An unbiased way of selecting the lots would be to randomly select lots from the population of lots used during the lifetime of the assay. Of course, this is impossible since future lots don’t exist. Hence, the selection of the lots is biased. There are several reasons that variation due to reagent lots is likely to be underestimated.

  1. The lots selected – usually the only ones that exist – are similar to each other and potentially different from future lots because future lots could have components from different lots, and those components could be qualified using different procedures.
  2. The procedure to qualify reagents may be different in the future.
  3. Usually during a regulatory method comparison using different reagent lots, the observed bias is centered by adjusting the instrument algorithm. For example, if the bias with 3 lots is 0, 1, and 2%, the instrument algorithm will be changed so that the bias reported is -1, 0, and 1%. (Note that this centering does not affect the spread of the reagents.)
  4. This centering is not used in future lots.
  5. The lots used in the evaluation are under more scrutiny than future lots, which is another case of bias. For example, the lots, besides being qualified through the manufacturing process, are also examined by a method comparison using patent samples compared to a reference assay. Future lots don’t have this scrutiny. They are usually qualified without a method comparison.

A possible remedy to try to better estimate the reagent bias would be to prepare a set of reagents according to factorial principles using components concentrations at the range of their manufacturing tolerances.

Biases in these studies other than reagent bias include: conflict of interest, protocol, patient selection, and user.

Unwarranted Conclusions

June 2, 2016


Looking at a paper about QC procedures (subscription required), I admit I was intrigued by the title: “Selecting Statistical Procedures for Quality Control Planning Based on Risk Management.”

Just reading the abstract and the first few lines informs me that the conclusions are unwarranted because the authors claim, they can estimate the probability of patient harm based on which QC procedure is chosen.

A QC procedure helps to detect problems with the assay process. Patient harm can be caused by an assay process gone astray but it can also be caused by things with an assay process that has not gone astray. For example, a patient interference can cause patient harm and will not be detected by QC. Moreover, the authors assume that an out of control condition will occur in a constant fashion until it is detected by the next QC sample, but a shift in results that occurs for a limited number of samples can occur and is eliminated from consideration. So even QC considerations don’t include all possible errors.

Ok, I admit that I have stopped reading but it is clear that whatever the authors estimate (assuming their logic is correct) is an underestimate of the probability of patient harm.

That also makes me wonder, of all cases of patient harm caused by wrong medical decisions caused by assay error, what percentage are due to the assay process gone bad vs. other causes (e.g., interferences). For example searching for the word “interference” in the title of Clinical Chemistry over the last 10 years yielded 912 results.

Update on Blood Lead Goals

May 4, 2016


I have updated the section in the previous post on blood lead goals. They are also here.

Blood lead lowest allowable limit:

1960s 60ug/dL
1978   30ug/dL
1985   25ug/dL
1991   10 ug/dL
2012     5 ug/dL

Source Markowitz G, Rosner D. Lead Wars: The politics of science and the fate of America’s Children

Why do performance goals change – has human physiology changed?

May 3, 2016


[Photo is Cape Cod Canal] Ok, the title was a rhetorical question. Some examples of the changes:

Blood lead lowest allowable limit:

1960s 60ug/dL
1978   30ug/dL
1985   25ug/dL
1991   10 ug/dL
2012     5 ug/dL


Glucose meters:

2003 ISO 15197 standard is 20% above 75,
2013 ISO 15197 standard is 15% above 100,
2014 proposed FDA standard is 10% above 70.

The players:

Industry – Regulatory affairs professionals participate in standards committees and support each other through their trade organization, AdvaMed. The default position of industry is no standards – when standards are inevitable, their position is to make the standard as least burdensome as possible to industry.

Lab – Clinical chemists and pathologists are knowledgeable about assay performance. ALERTpathologists are not clinicians. Also, lab people are often beholden to industry since clinical trials are paid by industry, conducted in hospitals by clinical chemists or pathologists.

Clinicians – Sometime, clinicians are part of standards but less often than one might think.

Regulators – People from FDA, CDC, and other organizations have to decide to approve or reject assays and are often part of standards groups.

Patients – Patients have a voice sometimes – diabetes is an example.

Medical Knowledge – As the title implies, the medical knowledge related to performance goals is probably of little consequence. For example, the harm of lead exposure is not a recent discovery.

Technology – Improving assay performance due to technical improvements probably does play a role in standards. All of a sudden the performance standard is tighter and coincidently, assay performance has improved.

Cost – Healthcare is rationed in most countries so cost is always an issue, but it is rarely discussed.

Note that the earliest standard for these two assays is 100% or more lenient than the current standard.

IQCP – It’s about the money

April 22, 2016


There is an article in CAP Today about IQCP. I was struck by a quote in the beginning of the article:

“I didn’t stop to calculate what it would cost to do liquid quality control on all the i-Stat cartridge types every eight hours because the number would have been through the roof”

Now I understand that cost is a real issue, but so is harm to patients.

The original idea of EQC (equivalent quality control) was to reduce the frequency of QC if you did an experiment that showed good QC for 10 days. This was of course without merit with the potential to cause patient harm.

The current notion of IQCP is to perform risk analysis and reduce the frequency of QC. This also makes no sense. Risk analysis should always be performed and so should QC, at a frequency which allows the repeat of questionable results such that patients will not be harmed.


Get every new post delivered to your Inbox.