Not easy to perform FMEA

I read an entry about risk analysis on the Westgard Web which prompts the following comments…

First, I basically tune out most of the official (CLIA, CLSI) regulations or standards about risk management. After a thorough analysis of EQC and EP23 and many comments – largely without effect – it’s time to move on. Until someone has an epiphany (maybe it will be me) there doesn’t seem to be much point.

But as to how well risk management will be done by laboratories, two anecdotes come to mind.

When I started to sell risk management software to hospitals (since retired and no longer available or supported) I started with a free beta version to those who would send me an email. The emails I got scared me. To paraphrase one – “I would love to evaluate your software. This FMEA is taking way too long.” Basically, what was desired was a menu of FEMA topics so that the user could click on one, such as for example “patient falls” and then get a completely filled in FMEA. They could then say, that looks right – done! [Note: There was a joint commission requirement for a hospital to perform at least one FMEA a year.]

At a medical device company, where FMEA was a required step, I was contracted to help. The problem was meetings were always scheduled during lunch. When I asked why, I was told, “We’re too busy, we can’t afford to slow down our real work.” Not the best start.

Some reasons that FMEA is so difficult:

• Many believe it is not needed, perhaps because the basic FMEA questions are asked throughout the design process
• It can be adversarial because the person who owns or developed the process is present as it is being discussed (challenged)
• It is inherently difficult to brainstorm to come up with failure modes and judge their likelihood and to come up with useful mitigations

This is why FRACAS is often more successful. But it is not a substitute for FMEA. Both should be done. And by the way, some people perform a FRACAS and call it a FMEA. How do I know? Because they describe an improvement to the observed failure rate. In FMEA, there is a potential, not an observed failure rate.

The other side of the coin in risk management

A flying blog that I read posed the question “go or no go” meaning would a pilot fly the flight with the potential for bad weather (in a general aviation airplane from San Francisco to Seattle). The intent of this was to get people who responded to discuss the weather reports that were presented in the blog and that’s just what people did. But in doing this, virtually all of the responses were about the risk side of the equation but every risk management decision should weigh the risk benefit tradeoff. Thus, the benefit of flying your own plane is the enjoyment of flying and also getting to a destination but in this case, the risk of bad weather (and its consequences) was greatly increased with no gain in benefit, and flying commercially was a viable alternative.

I suspect that for in-vitro diagnostics, the other side of the risk benefit equation is also neglected. My sense is that …

FDA focuses more on risk than benefit. That is, many diagnostic assays provide important information to the clinician and outweigh the harm of assay error. Put another way, more people are helped by the assay information provided to the clinician than the people that are harmed by assay error.

POC assays are often evaluated more on benefit than risk. Most POC assays can also be performed in the laboratory, albeit not as fast but the POC assays usually have more error. Does the rapid result outweigh harm due to increased error? In the case of A1C, some say no.

Protocols, Family Feud, and data analysis

If the audience in the TV game show Family Feud was comprised of scientists and was asked, “What is the most common question about protocols”, the number one answer would be “what should my sample size be?” This is usually what I am asked and my first response is, “what is your goal?” This is usually met with a blank stare because there are no goals so determining goals becomes the first task.

With goals and a proposed protocol, another key question is what is the best way to analyze the data. Often, this is asked because it is required to include a statistical analysis plan as part of the protocol document. The answer to this question is more difficult (in terms of protocol inclusion) because the real answer is the statistical methods to be used are whatever it takes to find out everything that’s going on in the data. This would require a book for inclusion in the protocol. For example, you perform a regression with data that look slightly non-linear so you try quadratic and cubic terms. There are also a few points that appear to be influential, so you repeat everything excluding all possible combinations of those points. And that’s just the start.

What goes in the report depends on where it’s going. If the report is internal to the company, then all of the warts in the data are discussed and possibly some speculation as to causes. It’s up to the company to decide if these “warts” are important (see last post). If the report is going to an agency, then it’s what is required by the agency and nothing more and certainly no speculation.

Once, at Ciba Corning, a customer informed us that one of the blood gas analytes was being influenced by another blood gas analyte. And sure enough, looking at previous evaluation data showed the effect to be present. To me, this was a disaster. Our group had missed pointing out something that was going on in the data. Much earlier, when my job was to run a reference lab for Technicon where we assigned values to master lots of calibrators, I was pointing out things missed by the statstical group.

Disagreeing with the stat books

The typical statistics book states that to evaluate something, you state a goal, perform the experiment, and determine if the goal has been met. I believe this is also what the FDA expects. Whereas this sounds reasonable, the problem is that for many companies, not much time is spent on goals during development and evaluation of methods. The result is that after the evaluation is done, people will now start to ask what the goal should really be. But changing the goal after the evaluation is done is frowned upon.

For example, an assay precision goal is set at 5% CV and the result of the evaluation is 5.1% Say the team meets and decides 5.1% is acceptable and changes the goal to 6%. Is there something wrong with this? I say no. In my experience at Ciba Corning, this type of situation occurred periodically which led to the team discussing what we should do. The decision was always unanimous and sometimes favored the product being withheld (or recalled) and sometimes favored product release. Ideally, we should have had such discussions for each goal before the evaluations but it never happened.

Sometimes during the discussions, a “limited product release” was suggested. I always thought this was funny because every product release was limited by our manufacturing capacity so a “limited product release” really meant limited by as fast as you could make and ship product.