Regulation is part of medical diagnostic assays. In the US, manufacturers are inspected by FDA, their products must be approved by FDA and manufacturers must (for all practical purposes) also obtain ISO 9001 certification. Labs are inspected by either COLA, CAP, or JCAHO and in some cases by state agencies. Both manufacturers and labs must follow existing laws relating to the development and manufacture of assays or the running of assays in labs.
Yet, in spite of these regulations there are many quality problems and there is no evidence that the rate of problems is declining (1-3).
Standards groups, while not actually involved in writing regulations, nevertheless play a role in shaping regulations. Two important standards groups for diagnostic assays are CLSI (Clinical and Laboratory Standards Institute – formerly NCCLS), and ISO / TC 212. CLSI serves as the secretariat for ISO / TC 212. FDA regularly adopts CLSI standards as voluntary consensus standards which causes most manufacturers to follow these standards, which makes them quasi regulations.
The CLSI standards group
CLSI provides standards for several aspects of diagnostics assays, including those related to quality (e.g., evaluation of assay performance). Much of what follows concerns the CLSI activities related to evaluations (e.g., via the Evaluation Protocols Area Committee).
CLSI provides its standards through a consensus process, which works by achieving membership balance among three groups (called constituencies), which are government, professions, and industry. That is, the content of the standards is balanced among the interests of the three constituencies. The membership for two of these groups is easy to describe:
government – Regulators, who either provide regulation or participate in regulatory activities such review of submissions or inspections.
professions – People who either manage or work in a lab.
One might think that industry (the third constituent) could be described by people in either R&D or manufacturing, which while not the only functions of industry, are often thought of as representative of what industry does. However, there are few of these people in CLSI. Instead, a large contingent of industry members are from regulatory affairs.
With this in mind, one can view CLSI in view of two regulatory related groups
- regulatory providers (e.g., government)
- regulatory consumers (e.g., manufacturers and labs, each of which consume different regulations) Formally, the regulatory consumers are two distinct groups (manufacturer consumers and lab consumers).
Viewed this way, the regulatory consumer group has a majority vote, for even though manufacturers and labs must follow different regulations, they still belong to the regulatory consumer group and the consumer group has a large contingent of regulatory affairs professionals.
Note: AdvaMed, the industry trade group, often takes positions on controversial CLSI documents and thus represents a powerful influence on CLSI standards.
Goals of each group
The regulatory providers have goals to protect the public (e.g., try to guarantee safe and effective assays). Unfortunately, these goals are rarely quantitative. The method of achieving these goals is through regulation.
The regulatory consumers goals are simple (and logical) – pass the (regulatory) inspection or achieve (certification / 510(k), etc.) approval with the minimal amount of effort. This goal is very similar to the student’s goal: how can I do well on the upcoming test with the least amount of studying.
This leads to conflicts in the preparation of standards:
horizontal vs. vertical standards – In horizontal standards, general principles and guidelines are laid out but details are left up to the user. In vertical standards, specific procedures are proscribed (e.g., if you follow the standard, these are the procedures that you will use). In fact, people who lobby against vertical standards often call them too prescriptive. Vertical guidelines are always more difficult to write because agreement must be reached about which among several procedures is selected. In horizontal procedures, virtually all procedures are listed as options for the user to follow.
quantitative vs. non quantitative goals – Quantitative goals are measurable and thus easier to verify than non quantitative goals. Looking at things another way, one usually does not have to worry about “meeting” non quantitative goals.
measurement vs. documentation – While not mutually exclusive, measurement is used to quantify things. Documentation can be used for anything but is often used to provide evidence that a procedure has been carried out.
Regulatory consumers often lobby for horizontal standards, with non quantitative goals (or goals to be supplied by the user), and favor documentation over measurement. If these standards become regulation, then passing regulation is easier to achieve with the downside that without goals or measurements, there is no way to know if quality is adequate to protect the public.
Some examples of less desirable standards outcomes
- ISO 15189 – a horizontal general quality standard for labs similar to ISO 9001 (see reference 4).
o As an aside, when I asserted that the general horizontal ISO quality standard 9001 was not useful in medical diagnostic companies, regulatory affairs people disagreed, whereas R&D people agreed.
- ISO 14971 – a horizontal risk management standard with an ISO TC212 appendix (annex in ISO terms)
- Quantitative quality goals – Separate ISO and CLSI projects that would have specified quantitative goals for assays were both cancelled.
- ISO 15197 – This ISO TC212 standard for home glucose meters is an exception with respect to quantitative goals, since it does have one. Unfortunately, in six sigma terms, a 3.1 sigma process is specified for glucose (see six sigma essay).
- Vertical standard on reporting assay performance – CLSI project (EP11) the uniformity of claims project was cancelled
o An earlier version of EP11P (P=proposed) was published. The approved version (EP11A) was cancelled by the CLSI Board of Directors although it had successfully gone through the consensus process.
- Horizontal standard on reporting assay performance – ISO TC212 project nearing completion
There are also examples of close calls. For the CLSI document on nonlinearity (EP6), there was an almost successful attempt to make this a horizontal rather than a vertical standard.
Some examples of desirable standards outcomes
- EP9 – A vertical standard on method comparison
- EP5 – A vertical standard on imprecision
- EP17 – A vertical standard on detection limits
- EP6 – A vertical standard on linearity
- EP7 – A vertical standard on interference testing
- EP14 A vertical standard on matrix effects
- EP15 – A vertical standard on verifying performance
- EP12 – A vertical standard on qualitative assays
- EP10 – A vertical standard on multifactor protocols
- EP21 – A vertical standard on total analytical error (accuracy in ISO terms)
- EP21 software – companion software for EP21
There are no easy answers to these issues. Recognizing the problem is a start. Additionally, there should be more people in CLSI from industry who represent R&D and manufacturing. This of course does not guarantee anything since regardless of the functional responsibility of industry people, they are still regulatory consumers.
Jan Krouwer has been an CLSI volunteer for over 20 years. He recently completed a term as chairholder of the area committee on evaluation protocols (1999-2004) and continues to participate on various committees.
Helpful comments were provided by Glenn Fine, Executive Vice President of CLSI and Dan Tholen of Dan Tholen Statistical Services.
Note: For another opinion on ISO documents related to diagnostic assays, see reference 5.
- Nevalainen D, Berte L, Kraft C, Leigh E, Morgan T. Evaluating laboratory performance on quality indicators with the six sigma scale. Arch Pathol Lab Med 2000;124:516-519
- Assay Development and Evaluation: A Manufacturer’s Perspective. Jan S. Krouwer, AACC Press, Washington DC, 2002 pp 1-6.
- For a chronology of the problem at Maryland General Hospital, see http://www.westgard.com/essay64.htm.
- Krouwer JS. ISO 9001 has had no effect on quality in the in-vitro medical diagnostics industry. Accred. Qual. Assur. 2004;9:39-43.