Some technical comments about Six Sigma in the lab 7/2004

July 13, 2004

To recall, six sigma implies a process with 3.4 defects per million. Assume that one has a set of measurements that are normally distributed, for example lab glucose differences from reference. Many people remember that ± 2 standard deviations includes 95% of the values. The idea behind six sigma is that if one sets goals of ± 6 standard deviations, there will be very few defects (where defects are defined as being outside of the ± 6 standard deviation limits = 6 sigma. Of course, these limits must be clinically meaningful, one can’t just multiply the observed standard deviation by six to get the limits. So one starts with limits and sees where the measured standard deviation is and calculates how many “sigma” the process exhibits.

How do you get 3.4 defects – When I go through the math (1) to get the number of defects for a six sigma process, I get 0.00197, not 3.4 defects:

Defects per million = 1,000,000 * number of defects

where number of defects = (1 – probability good)

where probability good = F(6 sigma) – F(-6 sigma)

where F() = the cumulative distribution function of the normal distribution

What’s wrong? Six sigma assumes a 1.5 sigma bias, so if one repeats the calculations where probability good = F(1.5 + 6 sigma) – F(1.5 – 6 sigma), one gets 3.4.

Six sigma is defined for a normally distributed process. – Well, one might make the case that lab glucose differences from reference might be normally distributed but then again they might not, especially since when things go wrong there are outliers which tend to make things non normal. If the data are non normal, the six sigma numbers no longer apply.

But what does six sigma mean if one is talking about attribute data such as patient ID problems rather than continuous variables such as glucose differences. A patient ID mix up is clearly a defect but this does not come from a normally distributed process. Each attempt to match a patient ID and sample is either correct or incorrect. While one can count defects per million, this has nothing to do with “sigma” (e.g., a standard deviation).

A six sigma process has bias – Hard to believe but a six sigma process is defined to have a bias of 1.5 sigma. So what the big deal with this bias, if inclusion of a 1.5 sigma bias for a six sigma process still results in only 3.4 defects per million. The problem has been explained by George Klee, who shows that even a small bias can lead to increased patient misclassifications (2), which raises the risk for a wrong patient treatment. As an aside, a six sigma process with its 1.5 sigma bias, would not be suitable for the ISO standard GUM (the guide to the expression of uncertainty in measurement) since GUM requires all biases to be eliminated. So where did this bias come from. The originators of the six sigma concept were from Motorola, where it was observed that the average process tended to have a 1.5 sigma (drift) bias. Hence the inclusion of this bias was to be representative of real processes.

Does six sigma help labs, hospitals, or diagnostic companies – The above comments have nothing to do with the desirability of usefulness of six sigma. They are technical comments / answers for questions that I had. Of course no one will build in a bias to try to achieve six sigma! Six sigma is really a collection of quality improvement tools (3) and as such the name is somewhat misleading. A better name might be Total Quality Management – II (or a higher number). For example, FMEA (Failure Mode Effects Analysis) is part of Six Sigma and may involve an entire program without the measurement of a single standard deviation. Most “black belts” are managers and are neither statisticians nor people who have a career in data analysis and quality. The success of six sigma in organizations is a function of management commitment and training.

References

  1. Lucas JL. The essential six sigma Quality Progress 2002;35:27-31.
  2. Klee GG. Analytic performance goals based on direct effect of analytic bias on medical classification decisions. CDC 1995 Institute: Frontiers in Laboratory Practice Research. pp 219-226. Available online at : www.phppo.cdc.gov/dls/pdf/institute/klee.pdf
  3. Hahn GJ, Hill WJ, Hoerl RW, and Zinkgraf SA. The impact of six sigma improvement – A glimpse into the future of statistics. The American Statistician 1999;53:208-215

Six Sigma: An example, or a more serious technical issue 7/2004

July 12, 2004

In the first part of this series, some technical comments about six sigma were made – however these comments are largely curiosities as six sigma is really a collection of quality management tools.

Here the attempt is made to work through an example for a diagnostic assay.

The assay selected is a home glucose assay and the first step is to set a goal. Although there are a lot of ways to do this, there is an ISO document (15197) which contains goals. According to this standard, the “minimum acceptable accuracy” goal is:

“Ninety-five percent (95%) of the individual glucose results shall fall within ± 0,83 mmol/L (15 mg/dL) of the results of the manufacturer’s measurement procedure at glucose concentrations < 4,2 mmol/L (75 mg/dL) and within ±20 % at glucose concentrations >= 4,2 mmol/L (75 mg/dL).”

The ISO document says that “the minimum acceptable accuracy criteria are based on the medical requirements for glucose monitoring”.

First a word on terminology. In the ISO world, accuracy does not mean freedom from bias, rather it means freedom from all sources of error and is comparable to total error.

Now here is the problem. If one looks at the 75 mg/dL or less range, the goal states that one should not get values outside of ± 15 mg/dL, based on medical requirements. The problem is that this is stated for 95% of all values. So for an assay that just meets requirements this means that just under 5% of the values could exceed the goal which means that the number of defects one will see is 50,000 per million assays! In six sigma terms, this is close to a 3 sigma process (3.14487). This also means that this ISO document says that a 3 sigma process is acceptable.

 

Sigma Defects per Million
1 697,672.1
2 308,770.2
3 66,810.6
4 6,209.7
5 232.6
6 3.4

 

With 3 sigma goals, one should not expect six sigma processes.