My colleague Jim Westgard wrote a piece about risk management that deserves some comments. He dislikes the risk scoring schemes commonly in use because he says they “reflect subjective opinions and qualitative judgments.” He recommends that:
- Defects are scored using probability of occurrence from 0 to 1.0
- Severity is scored from 0 to 1.0
- Probability of detection is scored from 0 to 1.0
I mention in passing that two of these items are probabilities but severity is not a probability and arbitrarily ranked from 0 (no harm) to 1.0 (serious harm). Since the three items are multiplied together, I don’t know what this means.
But here are my two main points. Take probability of defect occurrence first. Say a defect is a very wrong result caused by electrical noise in a response, undetected by instrument algorithms. Westgard would like to change the probability of occurrence of this event from a scale such as extremely unlikely = 1, very unlikely = 2, and so on to a specific probability from 0 to 1.0. He wants to do this to prevent subjective opinions and qualitative judgments.
Now subjective opinions about this type of error from a person on the street would not make sense. But the opinion of a group of engineers who have developed the system would be of interest and yes the opinion is qualitative. But how does Westgard propose to get a quantitative probability? Who will provide this? It is possible through experiments to get an estimate for this defect but this could involve an enormous effort and this is only one potential defect. There could be thousands of potential defect causes, often depending on other causes and each requiring detailed experiments. Remember that a wrong result can be the cause of an operator error, pre or post analytical error and not just analytical error.
My other beef is about including probability of detection (also see reference below). The problem is detection is a process (QC is just one means of detection). For any incorrect result, there are many detection possibilities. For most analyzers, operators examine samples, a series of instrument algorithms are programmed to detect questionable results, QC is performed, serial results are queried using delta checks, and so on. And because detection is a process, there is the opportunity for failure of detection (often from multiple causes). So for example, QC may have some calculated probability of success, but there is the potential for failure because the control was not reconstituted properly, there was a bad vial, the control was expired, and so on.
Moreover, detection by itself will not prevent an error. One must also have a recovery. So with QC, one does not report results until troubleshooting has been completed. But troubleshooting (e.g., the recovery) is a process and it too can fail (again with multiple causes) and its potential for failure is ignored in the Westgard treatment.
So risk management using traditional FMEA isn’t so bad after all. But if you want to do something quantitative such as quantitative fault trees, it is unlikely to be within the financial constraints of your environment.
Schmidt MW. The Use and Misuse of FMEA in Risk Analysis. Medical Device and Diagnostic Industry 2004 p56 (March), available at http://www.devicelink.com/mddi/archive/04/03/001.html