EP21 (Total Error) Update

EP21 is a CLSI standard about estimating total error for diagnostic assays. It was first published in 2002 and has been a core part of the FDA’s guidance for CLIA waived assays. I published a paper in 1992, which among other papers supports the standard (Krouwer JS. Estimating Total Analytical Error and Its Sources: Techniques to Improve Method Evaluation.  Arch Pathol Lab Med 1992;116:726-731).

I led a group to revise EP21A, as required by CLSI procedure and this was completed about a year ago. But although the revision to EP21A has been minor, EP21A2 has been stuck for a year!

So here are some of the objections to EP21A2.

One reject vote had a lengthy harangue which for the sake of simplicity could be summarized that EP21A2 was too complicated. This comment included:  

Finally, this document not only is a departure but an abandonment of the traditional formula for estimating total error as a quantity [TE= (z factor)(CV%) +Bias%) widely publicized by Westgard and others.

I pointed out that the owner of this comment is listed as an author of the original EP21A standard and never complained about the original standard being too complicated – the revision to the original standard is minor. But since the original standard was issued, the commentator’s company entered into a financial arrangement with Westgard. I should say that I tried to point out this conflict of interest but this part of my response to the comment was always deleted by CLSI.

Another comment objected to the inclusion of pre- and post-analytical error to EP21A2. I have since modified the text of the document but the basic idea remains – it is intended to estimate all (e.g. total) error in the protocol that would be expected to occur in routine use. So for example if a finger stick sample were used routinely, then the protocol should be conducted with finger stick rather than venous samples. Putting things another way, if venous blood were used, then the potential error source from finger sticks would have been excluded from the protocol, which is a source of bias.

The objection to this was:

Pre-and post analytic error is not measurable in a reproducible manner.  It will vary by location (ED vs. clinic, ICU private office etc.), personnel obtaining samples, reporting results. … We in the lab and manufacturers have very little control over this.  We can however measure bias and imprecision quite well and then estimate total analytic error.

This is a misconception – even if it weren’t – it wouldn’t matter. Whether one has control over an error is not the issue, the task is to measure the error that will be observed in routine use.

As an example, say there are nurses performing finger sticks on assays that are under control of the laboratory. One could perform an experiment to measure error from this source as an imprecision component. It would be as reproducible as any other imprecision component (e.g., calibration error). And as for control, reagent bias is an analytical error. Why does the commentator think that he has control over the amount of reagent bias in an assay.

The problem is that laboratorians have been comfortable in using the wrong model (e.g., Westgard) to measure total error and also in excluding pre-analytical error from evaluation experiments that could easily be included (an error that occurs in some standards such as the glucose meter standard ISO 15197).

EP21A2 attempts to make things right and provide more faithful estimates of the error that will be routinely observed.

Advertisements

6 Responses to EP21 (Total Error) Update

  1. W Gernand says:

    Jan,

    1. All Westgard’s products are based on classic TE conception, so it may be inconvenient to face your broader TE deffinition.

    2. Is your concept linked or similar to concept of uncertainty?

    • jkrouwer says:

      W. Gernand,

      Regarding #2 (don’t understand #1), EP21 is not based on uncertainty, assuming that you mean ISO’s GUM (Guide to the Expression of Uncertainty in Measurement). Both GUM and Westgard’s approach are models with Westgard’s model at a higher level than GUM since Westgard has only two terms, whereby these two terms are intended to capture all components of these two terms. GUM attempts to specify all components as individual imprecision terms and then combine them to get total error.

      EP21 simply looks at differences from the comparative method – there is no model to combine terms because there is only one term.

      Jan

  2. W Gernand says:

    How is it possible to assess pre-analytic errors using differences from the comparative method? (I understand that the comparative experiment is performed using one set of samples).

    • jkrouwer says:

      In EP21, one doesn’t assess the contribution of (any) error components (including pre-analytical errors). One assesses the total error. Other CLSI protocols are designed to estimate error components.

  3. W Gernand says:

    Thanks. I hope E21A2 will be published soon.

  4. […] had previously complained about a conflict of interest at CLSI. Bringing this up to CLSI didn’t help. I became aware from […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: