• Thu. Aug 11th, 2022

The decline effect: why most medical treatments are disappointing


A prominent 17e The Englishman of the Century by the odd English name of Sir Kenelm Digby (1603-1665, pictured) was a founding member of the Royal Society, established in 1660 to support and promote science. Yet he wrote a book that went through 29 editions, recommending that wounds be treated with a “sympathy powder”: take six or eight ounces of Roman Vitriol. [copper sulphate], beat it very small in a mortar, sift it through a fine sieve when the sun enters the Lion, keep it in the heat of the sun and dry at night. The strangest thing about this potion is not that it should be made when the sun enters the Leo but that the balm should not be applied to the wound but to the weapon that caused the wound. Explicitly, the person who has been cut by a knife has to rub the powder on the knife.

When Sympathy Powder was applied to weapons, the wounds sometimes healed – not from the powder, but from the body’s natural recuperative abilities – which gave credit to the power of the Powder.

Unfortunately, much of the current medical research is also flawed. For example, some healthy patients are misdiagnosed as sick, and when they receive worthless treatment for a disease that does not exist, observers conclude that the treatment worked. A second problem is that some people who are genuinely sick improve as their bodies fight what ails them, with the result that worthless treatments can be credited again when no credit is due.

These misinterpretations are examples of the error known as post hoc ergo propter hoc: “after this, therefore because of this”. Just because one event occurs shortly before another does not mean that the first event caused the second event. For example, the ancient Egyptians noticed that the annual flood of the Nile was regularly preceded by the sight of Sirius – the brightest star visible on earth – appearing to rise on the eastern horizon just before sunrise. Sirius did not cause the flooding, but it was a useful predictor as there was an underlying temporal connection between the two events: each year Sirius would rise before dawn in mid-July and heavy rains. , starting in May in the Ethiopian highlands, also caused the Nile floods to begin in late July.

Has Donald Trump been “cured” of COVID-19 from remdesivir, dexamethasone, famotidine, a Regeneron antibody cocktail or a double cheeseburger? It would be a post-hoc mistake to credit any of these, or anything else Trump did or did to him, for the cure.

The gold standard for medical research is a randomized trial in which subjects are separated into a treatment group that receives the drug and a control group that does not. However, even the gold standard is not guaranteed to give the correct answer. Since the separation between the treatment and control groups is random, it may happen that the treatment group includes a disproportionate number of patients who are less seriously ill or who are more likely to recover on their own. , which leads to a misleading conclusion that a treatment under study works.

Such disappointments are magnified by the fact that the number of worthless treatments that could be tried is far greater than the number of actually useful treatments. To see what a difference this reality makes, assume a simple situation where 1) medical treatments work or don’t work and 2) a randomized trial will correctly identify 99% of all effective treatments as effective and 99% of all ineffective treatments as ineffective. . Bayes’ rule implies that if only 1 in 100 treatments tested are actually helpful, the probability that a treatment certified as effective will actually work is 50-50. For a more pessimistic result, suppose that only 1 in 1000 tested treatments is really effective. If so, 91% of all treatments certified to be effective are worthless.

Adding to the noise is the temptation to exploit the data, to ransack the data in search of encouraging correlations. This is how one study came to the mistaken conclusion that the risks of pancreatic cancer could be reduced by avoiding coffee, and another mistaken study concluded that people could be cured of certain illnesses by healing from a distance.

Given the prevalence of random variation, post-hoc errors, and data mining, it is not surprising that many “proven” treatments are disappointing when released to the general public. The model is so common in medical research that it has a name – “the decline effect”.

varios tubos para muestras de sangre sober informs medico

Another reason the benefits may be overstated is that medical trials focus on patients known to have a specific disease. Outside of trials and inside doctors’ offices, treatments are often prescribed for patients who have different illnesses, a combination of illnesses, or who have the symptoms but not the illness. For example, antibiotics are widely regarded as a wonder drug, and they are often very effective. However, some doctors seem to prescribe antibiotics reflexively despite possible side effects which include allergic reactions, vomiting, or diarrhea.

For childhood ear infections, the American Academy of Pediatrics now recommends that instead of prescribing antibiotics, parents and doctors wait to see if the body can fight off the infection on its own. More generally, The ICU Book, the best-selling and most respected guide to intensive care units, advises, “The first rule of antibiotics is to try not to use them, and the second rule is to try to do not use too much. “

The decline effect is likely to be even more prevalent when artificial intelligence (AI) algorithms are involved, as computers are so good at finding correlations and so bad at judging whether the patterns discovered make sense. Thus, algorithms are likely to give erroneous recommendations based on fleeting coincidences.

A team of physicians and biostatisticians from the Mayo Clinic and Harvard just reported the results of a survey of clinical staff who used AI-based clinical decision support (CDS) to improve blood sugar control in diabetic patients. When asked to rate the success of the AI-based CDS system in improving users’ ability to identify the correct intervention for individual patients on a scale of 0 (not at all helpful) to 100 (extremely useful), the median score was 11. Only 14% of users said they would recommend the AI-based CDS to another clinic.

The most common complaints were that the suggested interventions were too similar for different patients and therefore were not sufficiently tailored for each patient, the suggested interventions were inappropriate or unnecessary, and the system misclassified standard risk patients as high risk (a false positive rate).

IBM’s Watson has been a particularly notable disappointment in the medical AI arena in that, so far, he has promised too much and underestimated. Win at Danger and diagnosing cancer are very different tasks.

Walmart is now planning a major expansion in healthcare, based on Clover Health, which boasts that its Clover Assistant technology “gives your primary care physician a comprehensive view of your overall health and sends them personalized care recommendations. for you, just when you are ‘there’, using ‘data, clinical guidelines and machine learning to bring out the most pressing patient needs’. It sounds a lot more like ad copy written by a marketer than reliable advice written by doctors. If IBM’s Watson is struggling with diagnoses and treatments, Walmart shoppers may want to beware of the health care aisle.

There have been many great successes in medical science: the triple combination of HIV drugs that transformed HIV from a death sentence to a chronic disease, the benefits of statins, the effectiveness of antibiotics and treatment. diabetes with insulin. There have also been many successes in identifying the causes of disease: asbestos can lead to mesothelioma, benzene can cause cancer, and smoking is associated with cancer.

Even though medical research has had such wonderful success, blind faith is not justified. Researchers should keep the reasons for the decline effect in mind, and physicians and patients should anticipate the decline effect when deciding on treatments.


You may also find of interest:

Why has IBM Watson been a flop in medicine? Robert J. Marks and Gary Smith Explain How AI Could Not Identify What Information In The Tsunami From Medical Literature mattered.

and

Can AI Fight Deceptive Medical Research? No, because the AI ​​does not fix the “Texas sniper errors” that produce the bad data. (Gary Smith)


Leave a Reply

Your email address will not be published.