The Trouble with Meta-Analyses

The trouble with Meta-Analyses – Thoughts About How They Should Contribute to Medical Science and Practice – by Milton Packer – Nov 2017

Many have critically examined the methodology of meta-analysis, and others have set standards for their execution. Despite such guidance, meta-analyses continue to proliferate, but we should ask: do they really contribute?

Esteemed organizations regard the conclusions of a well-executed meta-analysis as a higher level of evidence than a single well-done clinical trial.

This commentary explains why this cannot possibly be true.  

A Meta-Analysis is Only an Imperfect Observational Study

A meta-analysis is an observational study, but the author does no original work. Someone simply notices that several articles have data that pertain to a common topic and that they might show similar patterns.

In the past, the favored approach was to depict these in a narrative, but this task required insight into the details of each trial and a willingness to ask whether differences in design or execution might have contributed to differences in a study’s findings

The current approach to meta-analysis requires no such intellectual effort; little knowledge is needed about any trial, except that it possesses certain minimum features.

Advocates of meta-analyses claim that they select trials for inclusion or exclusion based solely on their methodological qualities without awareness of their results, but it is difficult to understand how that could happen.

Can the author of a meta-analysis claim to have read only the methods section of an article, but ignored the title, abstract, results, and discussion?

In reality, a meta-analysis is a mathematical method for combining data, which is weighted by the quantity but not the quality of the observations.

A trial that recorded many events, but was done imperfectly and was plagued by missing or confusing data, is given more weight than a small trial that was done impeccably but recorded few events

The methodology of meta-analysis increases the precision (but not the truthfulness) of any estimate.

Yet, to gain this additional measure of confidence, we must combine trials that

  • used different designs,
  • used different doses
  • for different durations, and
  • made observations with different degrees of care.

It is akin to thinking that one can evaluate a baseball team’s seasonal performance by summing the difference in scores at the end of the first inning of a few games, without paying attention to who participated and who was the competing team

Add the uncertainty that certain games were played under special rules, and the scores of some games are ignored

Because of their observational nature, meta-analyses are hypothesis-generating.

We do not intend for their findings to establish anything; instead, we expect them to be confirmed or refuted in a subsequent definitive trial.

What Types of Meta-Analyses Should Particularly Alarm Us?

  • Conclusions of Meta-Analyses Should Not Rely on Small Numbers of Events

For a meta-analysis to yield a stable and reliable estimate, the total number of events should exceed 200 to 300. How many meta-analyses are based on that much information?

  • Be Wary of Meta-Analyses That Rely on Indirect Comparisons

We should be wary of meta-analyses that predict the likely results of comparative experiments that have never been performed.

  • Meta-Analyses Should Not Tell Us What We Already Know or Obscure What We Should Remember
  • Meta-analyses should provide insights that are superior to those provided by a narrative summary of the data
  • Many meta-analyses only confirm existing knowledge and may conceal meaningful differences that can best be understood descriptively, rather than mathematically.

What Should We Expect From a Meta-Analysis?

Meta-analyses can be useful if they provide novel findings that reflect the design of their component trials and are based on a meaningful amount of evidence.

Unfortunately, the vast majority of meta-analyses do not approach such a standard.

Many exist only because they easily create a publication record for the authors.

I have little to add and agree with Dr. Accad’s thinking in this regard. I hope more medical professionals are listening to him.

Other thoughts?

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.