According to a recent study, only one in 20 medical treatments have high-quality evidence to support their benefits. The study also found that the harms of treatments are measured much more rarely (one-third as much) as the benefits.
Patients and doctors – and whoever pays for them – should know that medical treatments are safe and effective, but it’s an open secret in the medical field that all treatments, including commonly used ones, are only not safe and effective. For example, anti-arrhythmic drugs were widely prescribed with the idea that they would reduce the number of deaths from heart attacks until a clinical trial revealed that they actually increased the risk of death.
In another example, it was recommended that infants be put to sleep on their stomachs based on expert opinion that babies would be less likely to choke on their vomiting until large studies revealed that sleeping on your stomach increases the risk of sudden infant death syndrome.
How big is this problem?
In the early 2000s, researchers estimated that between a quarter and half of treatments were supported by high-quality evidence. But these estimates are now outdated and used old methods (such as researcher opinion) to determine whether the evidence was of high quality or not. More recently, in 2020, a more rigorous estimate was published and found that only 10% of medical treatments were based on high quality evidence. However, this estimate was based on a small sample of 151 studies.
Meanwhile, some continue to insist that most treatments should work. How else can we explain that we lived ten years longer than our great-grandparents? Yet longer lifespans are at least partly explained by public health measures such as clean water, better nutrition and restrictions on smoking.
A sharper picture
To resolve the controversy over the proportion of treatments based on good evidence, an international team of researchers from the UK (University of Oxford), USA, Switzerland and Greece conducted a large study of 1,567 treatments health. The sample included all treatments tested in Cochrane reviews between 2008 and 2021. Cochrane reviews are rigorous studies that bring together all available relevant evidence on treatments. They are often referenced in national and international health guidelines.
The year 2008 was chosen as the deadline because that was when the Cochrane reviews incorporated a system called Quality of Evidence and Strength of Recommendations (Grade) grading to assess the reliability of evidence. Unlike previous ratings which were often based on opinion, Grade is more widely accepted and is used by over 100 organizations around the world. Using the rating results in a high, moderate, low, or very low quality rating.
The study found that 95% of treatments lack high quality evidence to support their benefits. Worse still, harm is only reported in about 33% of Cochrane reviews.
It is particularly worrying that the harms of health care interventions are rarely quantified. For a doctor or a patient to decide whether or not to use a treatment, he must know whether the advantages outweigh the disadvantages. If the damage is not properly measured, an “informed choice” is not possible.
A potential limitation of the study is that the rating might be too strict. Doctors and patients may be happy to use treatments whose benefits are not supported by high-quality evidence as long as they are supported by moderate-quality evidence. Even if this is true, the review found that less than half of the treatments are supported by high- or moderate-quality evidence.
Patients with conditions for which there are no effective treatments may be willing to try treatments that are not yet supported by even low-quality evidence. The study should not be used to constrain the choices of these patients.
Also, the sample may not be representative. In theory, treatments tested in recent Cochrane reviews may be less effective or based on lower-quality evidence than older treatments. However, given the rigor of the Cochrane reviews, this seems unlikely.
In practice, doctors may use ‘off-label’ treatments that are less likely to have been studied in Cochrane reviews and usually have lower quality evidence to back them up. Despite these potential limitations, the study still showed that most treatments are not supported by high-quality evidence.
Physicians, patients and those who pay them may wish to focus on treatments whose benefits and safety are established by high-quality evidence. Research funding should be allocated to generating high-quality evidence for treatments that are widely used but not yet supported by high-quality evidence on their benefits and harms.
Finally, the potential harms must be measured with the same rigor as the potential benefits. The evidence-based medicine community is right to continue to call for better quality research, and has also justified its skepticism that high-quality evidence for medical treatments is common or even improving.