Westgard Quality Control Workshop – Part 2

I just returned from the Westgard quality Control Workshop, where I was a speaker and have a few blogs worth of comments – this is the second.

How does one determine acceptable risk

This was one of the questions asked by a participant – are there any guidelines? I also commented recently, that in spite of all of talk about risk management and putting in place control measures until one has acceptable risk, no one knows what acceptable risk means. Here’s some more thoughts on this.

There are different risks (1). These can be enumerated. These include:

perception – complaints from either hospital or non hospital staff

performance – traditional quality, including errors that can affect patient safety

financial – errors that threaten the financial health of the service including lawsuits

regulatory – errors that threaten the accreditation status of the service

So first, one must say which risk one has in mind. One can envision an acceptable regulatory risk (we always pass inspections) but an unacceptable patient safety risk.  Note also, that the risks are not necessarily unique. One can have a patient safety failure with or without a lawsuit.

Assume the risk in question is the performance risk and specifically about patient safety. The Cadillac version of assessing risk would be to perform a quantitative fault tree and arrive at a numerical probability of patient risk. This is unlikely and one would probably have a qualitative assessment. Whether the assessment is quantitative or qualitative, this still hasn’t answered the acceptability question.

The problem is there is no easy answer to this question. If one had unlimited funds, one could lower the risk to whatever level was desired but funds are limited by the economic healthcare policy of the laboratory’s country (2). So one answer of acceptable risk is how this economic policy is translated into regulations. (e.g., one follows existing regulations and passes inspections). Yet, this is only a quasi legal way of stating acceptable risk.

Recommendation

I suggest that risk be assessed by traditional means (FMEA, fault tree) which includes a Pareto chart or table to rank the risks. Then, if one optimizes the money that one has in implementing control measures (mitigations) by a portfolio type means, then one has an acceptable risk under the imposed financial constraints.

References

1.       Managing risk in hospitals using integrated Fault Trees / FMECAs. Jan S. Krouwer, AACC Press, Washington DC, 2004.

2.       See http://covertrationingblog.com/

Should one focus on a failure in a procedure or the outcome of such a failure?

Withholding payment for adverse events is a financial incentive to promote patient safety. Whether this incentive makes financial sense is something I will comment on later or perhaps not at all. For now, my comments are about the policy as it recently appeared (1).

 

 

The authors suggest the following criteria to withhold payment.

·         Evidence demonstrates that the bulk of the adverse events in question can be prevented by widespread adoption of achievable practices.

·         The events can be measured accurately, in a way that is auditable.

·         The events resulted in clinically significant patient harm.

·         It is possible, through chart review, to differentiate the adverse events that began in the hospital from those that were “present on admission” (POA).

The problem is with the third bullet and can perhaps be illustrated by the following figure.

In this figure FMEA events are shown by the dashed line.  The red dashed line is before FMEA. The green dashed line shows that after a successful FMEA, risk of failures has been reduced. FRACAS events are shown by the solid lines. The green line shows a reduction in the failure rate after FRACAS.

Keep in mind, for the dashed lines (FMEA), no failures have occurred, while for the solid lines, failures have occurred.

Now the policy defines a failure as an adverse patient outcome. One can view outcomes as the end of  an event cascade as in the next figure.

Assume that event C is an adverse patient outcome. According to the policy, payment is withheld only when event C is observed. In the first figure, the relevant concern area is shown by the ellipse as it is assumed that these are all high severity (severe patient harm) events.

This policy therefore excludes the following cases:

All FMEA events. That is, a procedure with a correctable high risk will be excluded from this policy because the event has not yet occurred. Considered the case of the Duke transplant error (2), before it happened. One can infer that this was a high risk procedure that would have benefited from a FMEA. In essence, this policy waits for disasters to happen.

All near miss events. Consider the case of the patient who had an MRI (3). Blood pressure monitor tubing had to be disconnected for the MRI. After the procedure, the tubing was incorrectly connected to an IV line. Before air was delivered from the automated blood pressure monitor, a family member noticed that things didn’t look right and contacted a nurse, who corrected the problem. Thus, there was no adverse event.

All defective procedures that don’t result in severe patient harm. Consider a healthcare worker who violates hospital policy (at risk behavior according to Marx (4)), which results in a patient fall. In this case, the fall results in a minor injury.  This is an important case because the policy fails to properly reflect risk management principles.

For a procedure that has a problem (e.g., a failed event), one has to classify the severity of the failed event and its probability (FMEA) or frequency of occurrence (FRACAS). The severity is classified not necessarily by the failed event but by the effect of the failed event. The effect is itself an event and can be a spectrum of severities. In the case of a patient fall, there is a distribution of harm associated with the fall event – some falls will result in severe harm, some will result in minor harm. Traditionally, in risk management, if severe harm is possible, then severity is associated with severe harm, even if the probability of severe harm is low. In this sense, severity is equated with potential outcome, regardless of whether that specific outcome has occurred.

One also has to classify the probability (FMEA) or frequency of occurrence of the event (FRACAS). Here, assuming FMEA, one could choose between the probability of the failed event or the probability of the effect of the event (the adverse outcome). It is recommended to use the probability of the failed event, not the probability of the effect of the event. This is because one usually has control over the failed event and does not have control over the effect of the event.

Example: If a clinical laboratory provides a clinician with an erroneous result and the effect of that could be patient harm, the event is classified as severe. The probability is the probability of erroneous result, not the probability of patient harm, because patient harm is outside of control of the clinical laboratory (the clinician might not act on the result, might suspect it is erroneous and request it to be repeated, and so on).

Summary

This policy will miss many quality issues and deviates from traditional risk management.

References

1. Wachter RM ,Foster NE and Dudley RA Medicare’s Decision to Withhold Payment for Hospital Errors: The Devil Is in the Details The Joint Commission Journal on Quality and Patient Safety 2008;34: 116-123, see http://psnet.ahrq.gov/resource.aspx?resourceID=6760
2. See http://www.cbsnews.com/stories/2003/03/16/60minutes/main544162.shtml
3. See http://www.ismp.org/newsletters/acutecare/articles/20030612.asp
4. Marx, D. Patient Safety and the “Just Culture”: A Primer for Health Care Executives http://www.mers-tm.net/support/Marx_Primer.pdf

---

Central lines and FRACAS

One hears of FRACAS success stories (like the one below) and FMEA failure stories (like the wrong blood type organs transplanted at Duke). A reason one doesn’t hear of FMEA success stories is that to say that something that has never happened is now even less likely to happen (due to FMEA) just isn’t too exciting. FMEA success stories are often not cases of FMEA, they are FRACAS, since rate improvements are discussed. FRACAS failures – we tried something, it didn’t work – are not very interesting.

A recent article in The New Yorker (1) provides an example of a FRACAS success story.

In the article, there is no mention of FRACAS but many of the steps were followed. The issue was a too frequent infection rate in central lines. It is important that one can measure this rate. One knows how many central lines are used, infections manifest themselves and their cause can be determined by culturing the lines. Some undercounting is possible but the rate seems fairly reliable.

The man behind the work, Dr. Peter Pronovost, first observed events for a month within the context of the process of placing central lines (e.g., process mapping). Errors in the process steps were identified. Since these steps were simple, such as washing hands, one could partly view these errors as non cognitive errors. This suggests a control measure such as a double check to prevent such “slips”. Actually, besides slips, there may have been some at-risk behavior (2). This is behavior that increases risk where risk is not recognized, or is mistakenly believed to be justified. The main control measure used was a checklist, with the addition of having nurses double check to see that the checklist steps were properly done. Then the rate was measured again and found to be considerably lower. All of this was published (3).

It was mentioned that an alternative control measure had been tried; namely, using central lines coated with antimicrobials. This expensive control measure failed to provide a substantial reduction in infection rates. This illustrates that one must be open minded when selecting control measures. There is sometimes a bias towards fixing the “system” (e.g., such as with coated lines) rather than fixing a people issue (e.g., which often implies blame). Dr. Pronovost implemented some system control measures by getting the manufacturer of central lines to include drapes and chlorhexidine – items that should have been available at the bedside but often were not.

Another big part of this story is ongoing resistance towards implementing this control measure more widely, even after it has been shown to be effective and low cost. Any control measure can be viewed as a standard and standards are not very popular. People will argue “but our situation is different”, “ICUs are too complicated for standards”, and so on. Financial incentives (or disincentives) for standards (e.g., P4P) loom. Dr. Gawande goes on to say how complicated things are in an ICU, yet there is precisely where standards helped. A similar situation happened in anesthesiology in the late 70s and early 80s. (Here, critical incident analysis was used and is basically the same as FRACAS.) The error rate was too high, effective control measures were developed, and widespread implementation of the control measures took considerable effort. You can read about that story here.

References

1.       Gawande A. Annals of Medicine. The checklist. The New Yorker, Dec. 7^th issue, 2007, see here (don’t know how long this link will work).

2.       Marx, D. Patient Safety and the “Just Culture”: A Primer for Health Care Executives http://www.mers-tm.net/support/Marx_Primer.pdf

3.       Pronovost P. et al. An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU. N Engl J Med 2006;355:2725-32.

ISO 14971 and Residual Risk

The last entry was about FMEA goals, yet, the word “goal” isn’t in ISO 14971. Maybe “goal” suffered the same fate as the word “mitigation” – banned from ISO. There is an implied goal in ISO 14971 - the residual risk must be acceptable. To recall, residual risk is the risk that remains after control measures have been taken. Here’s where things get a little tricky.

In cases where the residual risk is unacceptable, one is supposed to perform a risk benefit analysis to determine if benefits of the medical procedure performed by the device outweigh any possible residual risk.

To frame this discussion, consider two types of residual risk:

 

 

1.       A residual risk from a known issue, such as an interference, where eliminating this risk is not “practical “

2.       The overall residual risk from unknown issues. A certain amount of effort is used to search for risks (e.g., through FMEA, FTA, and FRACAS). At some point, more effort is considered not practical. Note: One can look at FDA recalls to see that unknown risks are often found in released products and lead to recalls (1).

Use of the word practical in ISO 14971 implies that in some cases, risk reduction is too expensive. This is not meant to be pejorative since everyone has limited resources.

In most cases in the standard, the cost benefit analysis is positioned as an analysis of the medical device’s clinical benefit to the patient vs. its risk. But ISO 14971 does point out an additional frame for the discussion.

“Those involved in making risk/benefit judgments have a responsibility to understand and take into account the technical, clinical, regulatory, economic, sociological and political context of their risk management decisions.”

To understand the issue, consider Type 1 diabetes as an example with the medical procedure being use of a home glucose meter. Because of risks 1 and 2 above, the glucose meter will fail and provide an erroneous result, albeit rarely. This is the current status and it is clear the benefit of the home glucose meter outweighs the risk (e.g., ADA recommendations to test for glucose). Yet, if one conducts a thought experiment and starts raising the frequency of (all) home glucose meter failures, simple decision analysis (2) still warrants use of the device. That is, measuring glucose, even if it occasionally (e.g., more often than rarely) gives an erroneous result, is better (clinically) than not measuring it.

If a company is working on a home glucose meter which provided an erroneous result too often (e.g., compared to existing meters), they will keep developing the meter until its failure rate is competitive. That is, there is a hierarchy of requirements for release for sale and often the competitive requirements (features needed to sell the product – including quality) are more stringent than any medical need or regulatory requirement (3).

Would you pay 2.5 million dollars to go to Cleveland?

Richard Fogoros suggests that there is a limit that we can spend for healthcare (4). To make this point, he says that if a plane could be built that could be survivable for most crashes, most people would not pay for an astronomical ticket price.

So regulators could require lower failure rates (less risk), causing companies to invest more, which would result in higher healthcare prices, but this is not done because it is unaffordable, hence the level of risk allowed is usually driven by competition. This is risk management but it is not the clinical benefit risk analysis described in ISO 14971– it is financial risk management.

References

1.       See http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfRES/res.cfm

2.       Krouwer JS. Assay Development and Evaluation: A Manufacturer’s Perspective, AACC Press, Washington DC, 2002, Chapter 3.

3.       Krouwer JS. Assay Development and Evaluation: A Manufacturer’s Perspective, AACC Press, Washington DC, 2002, pp 38-39.

4.       Fogoros RN. Fixing American Healthcare. Publish or Perish Press, Pittsburgh, 2007.

Fixing American Healthcare – A Review

My review of this book is from the perspective of a healthcare consumer and also as consultant to the medical device industry – I have no expertise in healthcare economics. In fact, the topic itself was initially of no interest for me – I figure we’re all going to get screwed and so someone talking about net present values of capitation expenditures would be a real snoozer. However, in this day and age of blogs, I came across the Covert Rationing Blog and found myself repeatedly coming back to this blog. Dr. Fogoros, aka DrRich, has a clear and entertaining writing style and made this topic interesting on his blog, so I bought the book. I was not disappointed.

The organization of this book is well thought out. The first 50 or so pages (out of slightly over 300) function as a summary of much of the analysis, after which people can either abandon ship or read on. I found Dr. Fogoros’s GUTH – grand unification theory about healthcare - to be quite compelling and also easy to understand. GUTH divides healthcare in four quadrants, all four combinations of centralized vs. the individual, and low quality and high quality. In this summary part, there is description of an investor session from 2000 which Dr. Fogoros attended. Here, Jim Clark (founder of Netscape) discussed his then latest venture – WebMD. I could have benefitted from Dr. Fogoros’s insight as to why WebMD would fail in its original concept, as I was one of the naive investors (fortunately only dabbling in this one). Simplifying insurers’  transaction costs and procedures was Jim Clark’s pitch, but the insurers did not want this simplification as their goal was to take money in but make it as complicated as possible to pay out for claims.

In the rest of the book, Dr. Fogoros supplies more details. What is so compelling to me is that when Dr. Fogoros exposes the forces at play, everything falls into place. There are no evil people, just people doing what they do best within the rules of society. So a football player that smashes his opponent on the field is cheered – off the field, the same behavior would land him in jail. In this book, the relevant players are like football players making hits on the field – they are not portrayed as evil.

Some of the discussions that were of interest: everything about money, the whole idea of covert (vs. open) healthcare rationing, the principle (that America refuses to abandon) that there can be no limits to healthcare, the destruction of the doctor patient relationship, the history and way HMOs work, why eliminating fraud won’t solve the healthcare cost problem, randomized clinical trials.

Two major groups are discussed as trying to control healthcare – the “Gekkonians” –who believe that market forces will reduce cost and the “Wonkonians” – who believe regulation can lower cost, largely by decreasing fraud.

Dr Fogoros has an engaging writing style. It is as if he is telling us a story, subtle humor is present  but the book is not a joke-a-thon.  One example - to illustrate the importance of cost in solutions, he says that one could do a lot more to make a plane crash survivable, but would you pay 2.5 million dollars for a ticket to Cleveland. Dr. Fogoros relays a chilling account of his own run-in with regulators, an experience that would make most people think of retirement. Thankfully for us, one reaction of his was to become an expert in the topic and write this book.

My somewhat cynical view of healthcare insurance has been that you pay expensive premiums for many years, at some time develop a serious illness, and then your policy is abruptly cancelled. Does Fixing American Healthcare simply play to my previous bias? Perhaps, but one should know that I complain about everything I encounter if I find a problem. Often, these complaints are published and thus, they are peer reviewed complaints about peer reviewed articles (the one that I am most proud of refers to the most cited publication in The Lancet). I do complain about a point made in Fixing American Healthcare. But it is a tiny point and does not detract from the main message of the book.

One of the values of this book is that it espouses the values of transparency, and just as importantly explains healthcare so that it is transparent. Transparency is the enemy of those with hidden agendas. I remember the resistance to unit pricing in food stores – some characterized it as too confusing, but its value was simplifying things. 

Of course, for Dr. Fogoros to point out problems is important, but what one also wants is proposed solutions. There is a preview of the solution in the section on clinical trials – openly ration healthcare and provide services to those who need it most. As one gets into these final sections about solutions, everything made sense to me, but I must admit, I need to reread these sections and since this will take some time, I thought it was important to provide this partial review, because this book is so important.

Overall, this book is fabulous and I learned a lot. It deserves to sell out of its first printing. For subsequent printings, ok, one final complaint - larger print would be nice.

10/21/2007 - Near Miss

William Marella writes about near misses in Patient Safety and Healthcare.  Much of what says makes sense but overall, the article itself is a near miss. Here’s why.

Mr. Marella reports that most hospitals follow regulators’ recommendations about reporting only about adverse events and not near misses. To understand the problem with this (beyond what Mr. Marella discusses), let’s look at FRACAS (Failure Reporting And Corrective Action System). With FRACAS, the steps are as follows (I’ve added emphasis as italics):

1.       Observe and report on all errors.

2.       Classify each error as to its severity and frequency of occurrence.

3.       Construct a Pareto chart.

4.       Implement corrective actions for the items at the top of the Pareto chart.

5.       Measure progress as an overall (e.g., combined) error rate.

6.       Continue steps 1-5 until the error rate goal is met.

So an immediate problem with what’s being done is that step #3 – constructing a Pareto chart is being handed down from regulators – and one can question the origin of this Pareto. Moreover, as Mr. Marella correctly points out, this Pareto chart is about adverse outcomes, not events in the process. To understand why this is a problem, consider the following chart about errors:

 

When errors occur, there is an opportunity for them to be detected. If detection (and recovery) are successful, a more serious error event has been prevented. So in this chart, error event A when either undetected or with successful detection and a failed recovery leads to error event B and if the same steps occur, error event B leads to error event C with each higher letter having a more severe consequence. As a real example of this, there was the national news story of the Mexican teenage girl who came to the US for a heart lungs transplant. Organs of the wrong blood type were selected (error event A) – this error was undetected and these unsuitable organs were transplanted (error event B). The correct reason that the patient’s health declined was detected but the recovery failed and the patient died (error event C).

Let’s consider detection in more detail. In planned detection, a (detection) step is part of the process. So, in a clinical laboratory, a specimen is examined to see if its adequate. For example, a serum sample that is red has been hemolyzed and will give an erroneous potassium result, so detection results in this sample not being analyzed – at least not for potassium. This causes a “delayed result” error rather than sending an erroneous result to clinician, which is more serious. Typically, detection steps are optimized so that it is more or less guaranteed so that they will be effective. In some cases, people have gone overboard – in one report, the average number of detection steps to assess if the surgery site is correct is 12 – this is too many.

However, a salient feature of a near miss is accidental detection. This unplanned detection signifies that there is a problem with the process that requires correction. There is of course no guarantee that accidental detection will occur the next time and it is likely that it won’t occur, so typically, when accidental detection occurs, severity is associated with the more serious event, as if the detection did not occur. The corrective action may be to create a planned detection step or to make other changes to the process. This also points out the problem with regulators constructing their own Pareto. By not collecting all errors and then classifying them, high severity errors (near misses) will be neglected. So basically, steps #1 and #2 in a FRACAS have been omitted.

Another problem, is the lack of constructing an overall metric and measuring it.

Some things to know about error rates

1. One should track only one (or in some cases a few) error rates.
2. The (overall) error rate goal should not be zero.
3. Resources are limited. One can only implement a limited number of mitigations.

The National Quality Forum (NQF) has identified 28 adverse events to be tracked, the so called “never events”. There is no way that one can establish allowable rates for each of these events and a “never event” implies an allowable rate of zero, which is meaningless. For those who have a problem with a zero error rate, one must understand, one is working with probabilities. For example, say one must have a blood gas result. Assume that one knows that the failure rate of a blood gas instrument is on average, once every 3 months, and when it fails, the blood gas system will be unavailable for one day. Say this failure rate is too frequent. One can address this by having 2, 3, or as many blood gas instruments as one wants – or can afford – with failure now occurring only when all blood gas instruments fail simultaneously. But no matter how many blood gas instruments one has, the estimated rate of failure is never zero, although it can be made low enough to be acceptable and perhaps so low that it can be assumed “never” to occur – although there is a big difference between the “never” used by the NQF and the estimated probability of failure. In fact, the difference between a calculated rate that is greater than zero but possible to occur in a one’s lifetime and a calculated rate that translates to “never” could be a substantial difference in cost. The blood gas example uses redundancy to prevent error. The wrong site surgery example above uses detection, which is of course much cheaper than buying additional instruments. Each mitigation has its own cost. Computer physician order entry is an expensive mitigation to prevent medication errors due to illegible handwriting. Financially, all of this reduces to a kind of portfolio analysis. One must select from a basket of mitigations an optimal set to achieve the lowest possible overall error rate at an affordable cost.

This (portfolio) analysis only makes sense if one is combining errors. If error A causes patient death or serious injury and error B does the same, and there are many more such events, one can combine these errors to arrive at a single error rate for all error events that cause patient death or serious injury. This is similar to financial analysis, whereby there is one “bottom line”, the profitability of the overall business – individual product lines are combined to arrive a one number.

The “Axiom of Industry” applied to healthcare

One of the most interesting blogs that I have come across is the Covert Rationing Blog. The author, DrRich (Richard N. Fogoros, MD) has written a book, “Fixing American Healthcare”, which I am in the process of reading. So far, it is a fabulous book, and I am learning a lot. I did take exception to a point that was made on DrRich’s blog and follow up on that here, based on getting to that section in his book.

His “axiom of industry” is that standardization of an industrial process reduces cost and improves outcomes. This industrial  idea is being applied to healthcare. DrRich gives a example where standardization applied to healthcare works (hip replacement) and where it doesn’t work (congestive heart failure - CHF). The reasons he provides – although not exactly so stated – are that for hip replacement, one has a high state of knowledge, and for CHF, one has an intermediate state of knowledge and when the state of knowledge is not high enough, standardization will not work.

This is where DrRich needs to continue with his industrial analogy. There are many processes in industry with a high state of knowledge as well as processes with an intermediate state of knowledge. Yes, in industry, one standardizes processes with a high state of knowledge, but this does not happen when the state of knowledge is inadequate.  Here, one uses a variety of approaches, including trial and error; that is, observing errors and then applying corrective actions. FRACAS (Failure Reporting And Corrective Action System) is a formal name for this method and believe it or not the acronym TAAF (Test Analyze And Fix) is also used. Whereas observing errors and then fixing them is not often admitted by quality managers as the method used, it is at times the best method to improve a process.

In healthcare, this method is often used as well. As patients, we are aware of the physician saying, let’s try treatment XYZ and see what happens, implying that if the treatment doesn’t work (an incorrect treatment decision) another treatment will be tried. If this actually happens and the second treatment works, one might not be happy but it is possible that the physician nevertheless followed a reasonable course of action. Moreover, for a disease condition one is not always in a “standardization” or “trial and error” situation. One often uses a mixture of the two. And, there is always the possibility that the state of knowledge for a disease may increase at some point to allow for standardization. I previously commented that standardization of a process that is not ready is likely to lock in unknown errors.

The other point that DrRich makes is that patients are not widgets. The implication is a little ominous here, namely; that morally deficient industrial managers given the chance, would discard patients as readily as widgets. I commented before, that one is optimizing a process – the correct analogy is to throw out an incorrect treatment – not a patient. Moreover, widgets are usually thought of as low cost items. No one considers a patient as low value. So here the analogy must be between patients and high cost widgets (of which there are many). In industry, as in medicine, loosing (discarding) a high cost item is not good.

One needs to ask, how many medical conditions are amenable to standardization (e.g., have a high state of knowledge). Covert rationing may well be responsible for patients being treated as widgets, including misapplying industrial processes, but these processes themselves can be applied to healthcare to benefit patients, although they will not solve healthcare costs.