Why Most Published Research Findings Are False

Why Most Published Research Findings Are False – PLoS Med. 2005 Aug – free full-text PMC article

There is increasing concern that most current published research findings are false.

Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.

Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.

In this essay, I discuss the implications of these problems for the conduct and interpretation of research. 

Bias

First, let us define bias as the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.

Let u be the proportion of probed analyses that would not have been “research findings,” but nevertheless end up presented and reported as such, because of bias.

Bias should not be confused with chance variability that causes some findings to be false by chance even though the study design, data, analysis, and presentation are perfect.

Bias can entail manipulation in the analysis or reporting of findings.

This kind of bias is obvious in studies on opioids when every single one of them shows how “”bad” they are, even though they’ve been used successfully to relieve pain for millennia.

Selective or distorted reporting is a typical form of such bias.

We may assume that u does not depend on whether a true relationship exists or not. This is not an unreasonable assumption, since typically it is impossible to know which relationships are indeed true.

Corollaries

Based on the above considerations, one may deduce several interesting corollaries about the probability that a research finding is indeed true.

Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.

Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.

Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.

Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true

Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

Most Research Findings Are False for Most Research Designs and for Most Fields

Below are listed types of studies and how likely they are to appear true: 

A finding from a well-conducted, adequately powered randomized controlled trial starting with a 50% pre-study chance that the intervention is effective is eventually true about 85% of the time.

Conversely, a meta-analytic finding from inconclusive studies where pooling is used to “correct” the low power of single studies, is probably false if R ≤ 1:3.

Research findings from underpowered, early-phase clinical trials would be true about one in four times, or even less frequently if bias is present.

Epidemiological studies of an exploratory nature perform even worse, especially when underpowered, but even well-powered epidemiological studies may have only a one in five chance being true, if R = 1:10.

PPV (Positive predictive value) for each claimed relationship is extremely low, even with considerable standardization of laboratory and statistical methods, outcomes, and reporting thereof to minimize bias.

PPV of Research Findings for Various Combinations of Power (1 – ß), Ratio of True to Not-True Relationships (R), and Bias (u).

Claimed Research Findings May Often Be Simply Accurate Measures of the Prevailing Bias

As shown, the majority of modern biomedical research is operating in areas with very low pre- and post-study probability for true findings.

Let us suppose that in a research field there are no true findings at all to be discovered. History of science teaches us that scientific endeavor has often in the past wasted effort in fields with absolutely no yield of true scientific information, at least based on our current understanding.

In such a “null field,” one would ideally expect all observed effect sizes to vary by chance around the null in the absence of bias. The extent that observed findings deviate from what is expected by chance alone would be simply a pure measure of the prevailing bias.

How Can We Improve the Situation?

A major problem is that it is impossible to know with 100% certainty what the truth is in any research question.

In this regard, the pure “gold” standard is unattainable. However, there are several approaches to improve the post-study probability

Better powered evidence, e.g., large studies or low-bias meta-analyses, may help, as it comes closer to the unknown “gold” standard.

However, large studies may still have biases and these should be acknowledged and avoided.

Moreover, large-scale evidence is impossible to obtain for all of the millions and trillions of research questions posed in current research

Moreover, one should be cautious that extremely large studies may be more likely to find a formally statistical significant difference for a trivial effect that is not really meaningfully different from the null

Second, most research questions are addressed by many teams, and it is misleading to emphasize the statistically significant findings of any single team.

What matters is the totality of the evidence.

Advertisements

2 thoughts on “Why Most Published Research Findings Are False

  1. Pingback: Benefits and harms of drugs for “neuropathic” pain | EDS and Chronic Pain News & Info

  2. Pingback: Embrace Uncertainty and Cut the Hype | EDS and Chronic Pain News & Info

Other thoughts?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s