3 Reasons To Be Wary Of Meta-Analyses

on 3 Reasons To Be Wary Of Meta-Analyses | American Council on Science and Health – By Stan Young — January 26, 2016

…relevant to the increasing use of statistics in science and health issues, statistician Stephen John Senn said that …data are often tortured until they confess to exactly what a scholar wants the numbers to say.

In a meta-analysis,

  1. the scientific literature is searched,
  2. a subset of papers is selected and then
  3. a combined estimate is made.

It is essentially “conducting research about previous research.”

Thus, we face an even greater problem of “garbage in, garbage out” because the biomedical field is rife with awful reserach with poor design and methodologies. 

As an example, I am going to use a recent e-cigarette paper published in The Lancet, because it provocatively makes the claim that e-cigarettes makes it less likely, not more likely, that smokers will quit, and e-cigarettes are a popular and controversial topic.

Controversial subjects lend themselves to strange methodologies and use of statistics so it is good to use for a discussion on the problems in meta-analyses.

The paper is not about the health issues, obviously e-cigarettes are better for health than smoking so harm reduction is clear, but it instead tackles their efficacy in “smoking cessation,” which is the way nicotine gums and patches are used.

When tackling an issue that is controversial, there is always the chance that the data selected could be massaged into a result.

That is even more true in the current anti-opioid hysteria.

That leads to:

Problem #1 in a meta-analysis that occurs in this paper: Selection Bias

Using keywords to get their selection pool, they say their initial search yielded 577 papers, from which the authors then chose (selected) about 20 papers.

I did a simple Google Scholar search (“e-cigarette cessation”) and got over 300,000 results, so anyone who uncritically accepts their results has to implicitly trust their winnowing process: namely that 20 out of 577, much less out of 300,000, was a truly representative sample.

As you can imagine, by changing the selection rules even slightly, you can get a different result, because the basic premise of meta-analysis is to average out errors.

if selection bias instead causes papers that match confirmation bias to be selected, this “average” can be far from reality, due to outliers being included.

Problem #2 in a meta-analysis: Statistical Power

To have a legitimate meta-analysis, researchers must identify a common statistical measure the studies share — the effect size — and standard error that will allow computing a weighted average of that common statistical measure.

For that, researchers consider issues like sample sizes of the individual studies and study quality.

Given that consideration, it is obvious that any shift in selection rules, as I mentioned in selection bias (#1 above), can produce a much different result.

Results could have a great confidence interval and be completely wrong.

…the 20 papers were not uniform enough to be combined: they don’t vary slightly (due to chance); they vary a lot.

For statistical experts, including (one hopes) peer reviewers, that would have put a halt to the paper because combining such dissimilar results is like combining apples and steak. No answer can be legitimate.

Problem #3 with a meta-analysis: Using a too narrow and directed criterion in crafting the question.

In this paper the question was crafted so narrowly it is logical fallacy: Is there an association between e-cigarette use and cigarette smoking cessation ¦ irrespective of their motivation for using e-cigarettes?

there are obviously a number of reasons a person might start using e-cigarettes, one of which could be smoking reduction. It could be a fad. It could be harm reduction. It could be they like a flavor.

By using such a narrow question they are, intentionally or not, making the perfect result, an end to smoking, the enemy of the good, such as smoking harm reduction or a transition to quitting smoking by using a particular smoking cessation tool.


Their own analysis says that the studies they chose are too diverse to be combined.

But that doesn’t stop them from doing it anyway and basing their conclusion on corrupt data.

Obviously smoking exclusively is worse than smoking a little and having e-cigarettes, which is worse than having just e-cigarettes. The perfect result would be having no one inhale anything at all, but that approach does not work for most, which is why so many techniques are used in the broader goal of stopping smoking.

That is the broader problem with the article. It seems to be agenda-based against one product type, instead of objectively analyzing what works for smoking cessation.

Why doesn’t anyone complain about the blatantly anti-opioid agenda-based research studies that associate negative outcomes with “high opioid doses” instead of pain levels? (See Opioids Blamed for Side-Effects of Chronic Pain)

From a statistical validity point of view, this Lancet paper has to be ignored and instead there should be concern among the editorial staff as to why no reviewers caught these obvious problems.

Indeed, I wish the same would be done for all those studies on opioids.

Other thoughts?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.