Software Verification and Validation

 In spending two sessions with groups of people who verify and validate medical device software, I got the impression that most effort is spent on testing code (to the requirements that exist). In part, I based this assessment on the amount of questions (e.g., interest by the audience) when code testing was discussed vs. examining requirements. Yet, in reviewing recalls, and my experience in the IVD industry, I suspect that that most errors are caused by wrong requirements (see figure).

 

 

 This makes me recall some definitions.

Bug – A coding error that prevents the software from meeting its stated requirement. A divide by zero error is a bug, but if the denominator can never be zero, this bug will never be a failure. Never be zero means the value can never be zero without a code logic statement such as If X <> 0, then … If the code logic statement were present, there would be no divide by zero bug.

Failure – Any deviation from customer expectations. This rather liberal statement is similar to the general definition of quality by ASQ. Each failure must be evaluated by the software / product development team to decide whether they agree and of course deviations have non software causes.

Example – A home glucose meter produces a value over 500 mg/dL. The meter displays ERR1. This is a requirements error. It is known the value is too high ( it could be 501 or 1,000). The meter should say something like HIGH.

10/21/2007 FRACAS? – Never heard of it

I just got back from co-presenting a short course on medical device software verification and validation for AAMI. One of the topics that I discussed was the use of FRACAS to improve software reliability.

One of the first questions I asked was, has anyone heard of FRACAS? Only one person raised their hand.

I also asked – has anyone ever heard of software reliability growth management? Again, only one person raised their hand – in this case a different person. The rest of this entry is to try to explain these results.

Google returned 38,900 hits for the phrase “software reliability growth.” I assume that adding the word management did not make a difference. So people who have the responsibility to validate software for medical devices have not heard of (at least in this sample) a technique that is used by some and is written about. It’s not surprising that a similar result was found for FRACAS, which is really used to reduce errors from all sources – not just software. Here are my reasons regarding the lack of knowledge about FRACAS in the medical device industry.

1.       FRACAS is not required by the FDA. We live in a regulated world, where often, the prime quality goal of an organization is to stay out of trouble with regulators. This is an understandable goal and makes sense – the problem is that other important goals may be neglected and quality practices may be limited to those proscribed by the FDA. Product recalls, including those that have caused harm occur for approved products and not only at companies that get warning letters.

2.       Whereas reliability techniques associated  with preventing potential errors (FMEA) and preventing recurrence of observed errors (FRACAS) are both used in military programs, only FMEA seems to have made it into healthcare. In this course, most people raised their hand when asked – have you heard of FMEA? My take on this is that there is a bias towards FMEA, because it is associated with preventing potential errors. The notion that one can get anything useful by observing errors has been overlooked.

3.       This failure to recognize what has proved useful elsewhere (such as the defense industry) is perpetuated by various groups. For example, if one looks through the 2007 version of the ISO 14971 standard on risk management, there is not a single reference to FRACAS. The same results were found, using “Search” for websites for the Institute of Healthcare Improvement, National Quality Forum, and Leapfrog. Even using CAPA as a search term, yielded no results.

It’s time to realize that during product development, observing errors and implementing corrective actions all before product release is a form of risk management.