Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are doctors—to a striking extent—still drawing upon misinformation in their everyday practice? Dr. John Ioannidis has spent his career challenging his peers by exposing their bad science.
He’s what’s known as a meta-researcher, and he’s become one of the world’s foremost experts on the credibility of medical research. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed.
an obsession with winning funding has gone a long way toward weakening the reliability of medical research.
Ioannidis was shocked at the range and reach of the reversals he was seeing in everyday medical research. “Randomized controlled trials,” which compare how one group responds to a treatment against how an identical group fares without the treatment, had long been considered nearly unshakable evidence, but they, too, ended up being wrong some of the time.
“I realized even our gold-standard research had a lot of problems,” he says. Baffled, he started looking for the specific ways in which studies were going wrong.
And before long he discovered that the range of errors being committed was astonishing:
- from what questions researchers posed,
- to how they set up the studies,
- to which patients they recruited for the studies,
- to which measurements they took,
- to how they analyzed the data,
- to how they presented their results,
- to how particular studies came to be published in medical journals.
We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously.
“At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”
His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted:
- 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do
- 25 percent of supposedly gold-standard randomized trials, and as much as
- 10 percent of the platinum-standard large randomized trials.
The article spelled out his belief that researchers were frequently manipulating data analyses, chasing career-advancing findings rather than good science, and even using the peer-review process—in which journals ask researchers to help decide which studies to publish—to suppress opposing views.
even if a study managed to highlight a genuine health connection to some nutrient, you’re unlikely to benefit much from taking more of it, because we consume thousands of nutrients that act together as a sort of network, and changing intake of just one of them is bound to cause ripples throughout the network that are far too complex for these studies to detect, and that may be as likely to harm you as help you.
studies report average results that typically represent a vast range of individual outcomes. Should you be among the lucky minority that stands to benefit, don’t expect a noticeable improvement in your health, because studies usually detect only modest effects that merely tend to whittle your chances of succumbing to a particular disease from small to somewhat smaller.
Vioxx, Zelnorm, and Baycol were among the widely prescribed drugs found to be safe and effective in large randomized controlled trials before the drugs were yanked from the market as unsafe or not so effective, or both.
Even when the evidence shows that a particular research idea is wrong, if you have thousands of scientists who have invested their careers in it, they’ll continue to publish papers on it,
But even for medicine’s most influential studies, the evidence sometimes remains surprisingly narrow. Of those 45 super-cited studies that Ioannidis focused on, 11 had never been retested.
Perhaps worse, Ioannidis found that even when a research error is outed, it typically persists for years or even decades.
He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted, and discovered that researchers continued to cite the original results as correct more often than as flawed—in one case for at least 12 years after the results were discredited.
Doctors may notice that their patients don’t seem to fare as well with certain treatments as the literature would lead them to expect, but the field is appropriately conditioned to subjugate such anecdotal evidence to study findings.
The patient’s “anecdotal evidence” should come first, not data from averages. Who are doctors treating, the average patient or the patient sitting in front of them?
Evidence based medicine only works for the average person, but what if you’re not average?
the doctors have all been trained to order these tests, she notes, and doing so is a lot quicker than a long bedside chat. They’re also trained to ply the patient with whatever drugs might help whack any errant test numbers back into line. What they’re not trained to do is to go back and look at the research papers that helped make these drugs the standard of care.
“When you look the papers up, you often find the drugs didn’t even work better than a placebo. And no one tested how they worked in combination with the other drugs,” she says. “Just taking the patient off everything can improve their health right away.”
But not only is checking out the research another time-consuming task, patients often don’t even like it when they’re taken off their drugs, she explains; they find their prescriptions reassuring
In the case of pain patients, opioid drugs are far more than just reassuring. They are effective in relieving most pain; that’s why they are used during and after surgeries.
If the drugs don’t work and we’re not sure how to treat something, why should we claim differently?