Discrepant Analysis in Every Day Situations

Abnormal patient sample results are often repeated, whereas non abnormal results are usually not repeated. The same is true for QC; namely, that values that are “out” are often repeated, whereas values that are “in”, are usually not repeated. This essay considers some attributes of this practice.

Discrepant analysis for evaluations

Before considering the above cases, consider discrepant analysis for evaluations (1-2). For example, consider an assay evaluation, where one compares an assay result to a gold standard diagnosis (3). Usually, most assay results will agree with the gold standard diagnosis, but there will be a few exceptions. In discrepant analysis, assay results that don’t agree are rerun. This results in either confirmation or non confirmation of the discrepancy. This is shown graphically below.

Candidate assay positive Candidate assay negative
Reference Method positive Result agrees Discrepant
Reference Method negative Discrepant Result agrees
Repeat Discrepants only  
Candidate assay positive Candidate assay negative
Reference Method positive Result agrees Discrepant
Reference Method negative Discrepant Result agrees

In the second table, the number of discrepant results are either the same or lower than the first table (with the number of “result agrees” either the same or higher).

Before continuing, one may ask what is the root cause for the discrepant result. Whereas there can be many root causes, consider two generic top level effects – the result upon rerun either gives essentially the same value (e.g., remains discrepant) or gives a different value (for the sake of argument, assume the new value is no longer discrepant). The first case is consistent with a fixed bias, whereas the second case is consistent with a random error.

So far, there is nothing wrong with the above practice and it is natural to try to explore discrepant samples. Where people get into trouble is estimating things with the results of the study (which of course is almost always the case, or else why do the study). In such an evaluation, one wants to estimate the analytical sensitivity and analytical specificity of the assay. The problem is, constructing these estimates using the discrepancy procedure above (e.g., results from the second table) is biased, and results in diagnostic accuracy estimates that are too optimistic (1-2). One can think of this bias intuitively. That is, consider a sample that agrees with the gold standard diagnosis. Were this sample rerun, it might yield a discrepant result upon rerun (due to statistical variation and especially if the initial result were close to being a discrepant result in the first case). But there is no chance for this to occur because in the above procedure, only samples that are initially discrepant are rerun. Hence, the estimates are too optimistic.

To summarize, one could of course run replicates for all samples, but this might add too much expense to the study. It is reasonable to try to resolve only the discrepant results by rerunning them as long as one calculates sensitivity and specificity correctly (see references 1-2).

General Comment about Discrepant Analysis in Every Day Situations

In every day situations, discrepant analysis is use to inform about an action to be taken. For a patient sample result, the result of discrepant analysis will inform between two treatment alternatives (typically corresponding to those associated with a “normal” or “abnormal” result). In QC, the result of discrepant analysis will inform whether or not to rerun a block of patient samples and to troubleshoot the assay.

Discrepant analysis for Patient Samples

In routine practice, patient sample results are often repeated according to rules set up by the clinical laboratory and in a typical case, only abnormal results are rerun.

This is not the same as the discrepant analysis discussed in the section on evaluations because there is no reference method. However, the same arguments apply, because only selected patient sample results are repeated. One could perhaps consider the result slated to be repeated as discrepant from a working hypothesis that the result was “normal”.

The practice of repeating only selected samples was questioned in another essay with respect to troponin I.

To recall, in that study a point of care assay result for troponin I was repeated only if the result was above the cutoff. The study was used to support a reduced length of stay in the emergency department.

The bottom line is what is the performance of the assay (e.g., its analytical sensitivity and analytical specificity) given the specific set of clinical laboratory’s practice of only repeating selected samples. This is likely to be different than the analytical sensitivity and analytical specificity of the assay as determined by an evaluation.

Discrepant analysis for QC

The same arguments apply to QC. That is, when QC is out, one of the first (troubleshooting) steps is to repeat the QC to see if it is repeatedly out (see cases 1 and 2 below). Yet, a QC that is “in” is not repeated. Note, that for multiple rule QC programs, nothing really changes, because if one needs three observations to fulfill a criterion before QC is considered out, then one can simply consider that set of observations as a case. So, again, the bottom line is what is the performance of the QC procedure (e.g., the equivalent of analytical sensitivity and analytical specificity) given the practice of only repeating discrepant samples.

QC is different than the patient case above, because the result of QC affects a block of patient samples. For example, if a patient sample result was initially abnormal, but normal upon several repeats, then it is likely that random error caused the initial result to be abnormal and that the result is normal and can be so reported. This type of argument does not apply to QC. If a QC sample that is “out” does not repeat as “out”, one has no way of knowing whether the cause of the initial “out” result affected one or more patient sample results.

Moreover, with QC one can distinguish between the following cases:

Case 1 – QC is out. Rerun the QC sample. If the rerun is in, declare the run OK. This is the discrepant analysis case under discussion.

Case 2 – QC is out. Declare the run out and rerun the patient samples. Rerun the QC sample as part of a procedure to troubleshoot the assay. This is not discrepant analysis since the decision to take action (rerun the patient samples and troubleshoot the assay) has already been made.

Acknowledgement Helpful comments were provided by Sten Westgard.

References

1.       Quantifying the bias associated with use of discrepant analysis. Lipman HB, and Astles JR Clinical Chemistry 1998;44:108-115.

2.       User protocol for evaluation of qualitative test performance: Approved Guideline EP12A 2002 CLSI 940 West Valley Road, Suite 1400, Wayne, PA 19087.

3.       There are several variations to this scheme with respect to the accuracy of the gold standard diagnosis and whether the repeat assay uses a different (e.g., better reference procedure). These are beyond the scope of this essay.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: