April 9, 2014
In the last 2 months, I’ve been asked to conduct 5 reviews, all for different journals, to determine if a manuscript should be accepted or not. I performed 4 of the reviews, and declined to review one manuscript because the title and abstract alerted to me the fact that I probably wouldn’t understand one word of the paper. Before that, there was a 4 month period with no review requests, so you never know when the requests will occur.
Performing these reviews is the other side of the coin – I’ve submitted many papers of my own and read reviews of my papers. I know how it feels to have my own paper severely criticized, so I try to be gentle in my reviews when I see something wrong, but on the other hand I never have a problem in pointing out problems.
The review request contains the title and abstract; if you agree to perform the review you get the full paper. Many papers require revision which often means another review, where I can see how the authors responded to my comments.
April 7, 2014
I occasionally come across articles that describe a method evaluation using GUM (Guide to the expression of Uncertainty in Measurement). These papers can be quite impressive with respect to the modeling that occurs. However, there is often a statement that relates the results to clinical acceptability. Here’s why there is a problem.
Clinically acceptability is usually not defined but often implied to be a method’s performance that will not cause patient harm due to assay error.
A GUM analysis usually specifies the location for 95% of the results. But if the analysis shows that the assay just meets limits, then 5% of the results will cause patient harm. Now according to GUM models, the 5% will be close to limits because the data are assumed to be Gaussian so this is a minor problem.
A bigger problem is that GUM analysis often ignores rare but large errors such as a rare interference or something more insidious such a user error that results in a large assay error. (Often GUM analyses don’t assess user error at all). These large errors, while rare, are associated with major harm or death.
The remedy is to conduct a FMEA or fault tree in addition to GUM to try to brainstorm how large errors could occur and whether mitigations are in place to reduce their likelihood. Unless risk analysis is added to GUM, talking about clinical acceptability is misleading.