June 16, 2017
A recent article (subscription required) in Clinical Chemistry suggests that in many accuracy studies the results are overinterpreted. The authors go on to say that there is evidence of “spin” in the conclusions. All of this is a euphemistic way of saying the conclusions are not supported by the study that was conducted, which means the science is faulty.
As an aside, early in the article, the authors imply that overinterpretation can lead to false positives, which can cause potential overdiagnosis. I have commented that the word overdiagnosis makes no sense.
But otherwise, I can relate to what the authors are saying – I have many posts of a similar nature. For example…
I have commented that Westgard’s total error analysis while useful does not live up to his claims of being able to determine the quality of a measurement procedure.
I commented that a troponin assay was declared “a sensitive and precise assay for the measurement of cTnI” in spite of the fact that in the results section the assay failed the ESC- ACC (European Society of Cardiology – American College of Cardiology) guidelines for imprecision.
I published observations that most clinical trials conducted to gain regulatory approval for an assay are biased.
I suggested that a recommendation section should be part of Clinical Chemistry articles. There is something about the action verbs in a recommendation that make people think twice.
It would have been interesting if the authors determined how many of the studies were funded by industry, but on the other hand, you don’t have to be part of industry to state conclusions that are not supported by the results.
May 26, 2017
I started this blog 13 years ago in March 2004 – the first two articles are about six sigma, here and here. The blog entry being posted now is my 344th blog entry.
Although the blog has an eclectic range of topics, one unifying theme for many entries is specifications, how to set them and how to evaluate them.
A few years ago, I was working on a hematology analyzer, which has a multitude of reported parameters. The company was evaluating parameters with the usual means of precision studies and accuracy using regression. I asked them:
- a) what are the limits that, when differences from reference are contained within these limits, will ensure that no wrong medical decisions would be made based on the reported result (resulting in patient harm) and
- b) what are the (wider) limits that, when differences from reference are contained within these limits, will ensure that no wrong medical decisions would be made based on the reported result (resulting in severe patient harm)
This was a way of asking for an error grid for each parameter. I believe, then and now, that constructing an error grid is the best way to set specifications for any assay.
As an example about the importance of specifications there was a case for which I was an expert witness whereby the lab had produced an incorrect result that led to patent harm. The lab’s defense was that they had followed all procedures. Thus, as long as they as followed procedures, they were not to blame. But procedures, which contain specifications, are not always adequate. As an example, remember the CMS program “equivalent quality control”?
May 15, 2017
I have started the process of transferring older posts, which were in my now defunct website, to this blog. It will take a little while and additionally it will take some time for the search engines to catch up.
May 11, 2017
As of the end of 2016, I stopped consulting. I still follow lab medicine topics that interest me, and will still publish both papers in journals and entries on my blog.
I ended my website KrouwerConsulting.com – it doesn’t work anymore – and I realize that some of my earlier blog entries can’t be accessed because they were on KrouwerConsulting.com (written before I started my current blog). I will fix this and transfer all of these entries to my current blog, as soon as I figure it out!
I want to thank all of my clients. Their problems kept me thinking and I enjoyed both the technical tasks and meeting people to discuss a variety of issues.
April 29, 2017
Most people have seen the video of a passenger being forcibly pulled off a United flight. After some missteps, one would think that the CEO would say the right thing. But he said the following (as seen on CBS news on 4/27).
“Our policies were placed ahead of our shared values and procedures got in the way of what we know is right.”
To people like me, who have worked in industry, “shared values” sounds like a mission or vision statement. The problem is that a mission statement is supposed to dictate policies and procedures and of course, it is management that defines policies and procedures. Hence, the quote sounds hollow.
What a company does can be considered its mission. If it conflicts with its mission statement, then the mission statement is out of whack.
April 13, 2017
In the recent hubbub about the use of PSA to screen for prostate cancer, the word overdiagnosis has appeared several times. In an online dictionary, it is defined as “the diagnosis of a disease more often than it is actually present.”
But the diagnosis of prostate cancer is pretty straightforward, with a biopsy determining whether cancerous cells are present. The diagnosis can be either correct or incorrect. But, there can be overtreatment, when low-risk prostate cancer is treated aggressively. So somehow, overtreatment has spilled over into diagnosis to produce the word overdiagnosis. That is, if a person has a biopsy confirmed prostate cancer, which is low risk and receives aggressive treatment – he has been over treated, not over diagnosed.
April 11, 2017
For almost all of my career, I’ve been working to determine performance specifications for assays, including the protocol and data analysis methods to see if performance has been met. This work has been performed mainly for companies but occasionally also for standards groups. There are some big differences.
Within a company, the specifications are very important:
If the product is released too soon, before the required performance has been met, the product may be recalled, patients may suffer harm, and overall the company may suffer financially.
If the product is released too late, the company will definitely suffer financially as “time to market” has been shown in financial models to be a key success factor in achieving profit goals.
Company specifications are built around two main factors – what performance is competitive and how can the company be sure that no patients will be harmed. In my experience this has simply led to two goals – 95% of the differences between the company assay and reference should be within limits which guarantee a competitive assay and no differences should be large enough to cause patient harm (a clinical standard).
Standards groups seem to have a different outlook. Without being overly cynical, the standards adopted are often to guarantee that no company’s assay will fail the specification. Thus, 95% of differences between the assay and reference should be within these limits. There is almost never a mention about larger errors which may cause patient harm.
Thus, it is somewhat ironic that company specifications are usually more difficult to achieve then specifications published by the standards organizations.