Mindfulness Research’s Huge Problem

Mindfulness research’s huge problem with uninformative control groups – by James C Coyne – June 2016

Are enthusiasts protecting cherished beliefs about the power of mindfulness from disconfirmation?

Do any advantages of mindfulness training disappear in a fairly matched cage fight with a treatment of comparable frequency and intensity?

Finally, a more thorough look at the research indicates that mindfulness is more a placebo response than a treatment. It makes no sense that it’s recommended by medical experts in the place of effective medication. 

Very few of the 1000s of articles retrieved in a literature search with the keyword “mindfulness” represent advances in the limited evidence that mindfulness-based stress reduction (MBSR) is effective for physical health problems.

Only a few randomized controlled trials with appropriate control groups are available and they do not offer strong evidence for the efficacy of MBSR.

This blog post demonstrates how uninformative and misleading comparisons with no treatment or treatment as usual/routine care can be.

While the lack of adequately controlled studies could have initially reflected the naïveté of MBSR researchers, increasing acknowledgment of the problem suggests enthusiasts’ avoidance of confronting cherished beliefs with disconfirming evidence.

When cage fights are arranged between MBSR and appropriate active control groups, the alternative treatments are often shown to be superior and more cost-effective, even when MBSR enthusiasts are the referees.

A comprehensive systematic review and meta-analysis prepared for the US Agency for Healthcare Research and Quality (AHRQ)

Reviewed 18,753 citations, and found only 47 trials (3%) with 3515 participants that included an active control treatment.

• We found low evidence of no effect or insufficient evidence of any effect of meditation programs on positive mood, attention, substance use, eating habits, sleep, and weight.
• We found no evidence that meditation programs were better than any active treatment (ie, drugs, exercise, and other behavioral therapies).

An accompanying commentary on the review asked:

• • “The modest benefit found in the study by Goyal et al begs the question of why, in the absence of strong scientifically vetted evidence, meditation in particular and complementary measures in general have become so popular, especially among the influential and well educated…”
  • What role is being played by commercial interests?
  • Are they taking advantage of the public’s anxieties to promote use of complementary measures that lack a base of scientific evidence?
  • Do we need to require scientific evidence of efficacy and safety for these measures?

A reminder: treatments do not have effect sizes.

• MBSR does not have an effect size.
• Rather, comparisons of MBSR to other conditions have effect sizes, which will vary greatly with the comparison treatment and population being studied.

Not just any comparison/control condition will do.

A comparison/control condition must be suitably matched with MBSR in terms of

• frequency and intensity of contact,
• positive expectations, and
• overall levels of support and attention.

MBSR treatments typically involve weekly meetings, daylong workshops or retreats, and the expectations that patients will practice mindfulness daily.

Construction of an adequate control condition that matches these features can be challenging.

The domination of the MBSR literature by nonrandomized trials and randomized trials with inadequate control groups represents one contribution to an exaggeration of the efficacy of MBSR.

A study published in NEJM that did not evaluate MBSR nonetheless demonstrates how misleading poorly chosen control groups can be, especially for physical health outcomes.

Wechsler ME, Kelley JM, Boyd IO, Dutile S, Marigowda G, Kirsch I, Israel E, Kaptchuk TJ. Active albuterol or placebo, sham acupuncture, or no intervention in asthma. New England Journal of Medicine. 2011 Jul 14;365(2):119-26.

This randomized, double-blind, crossover pilot study involved screening 79 patients, of whom 46 with mild-to-moderate asthma met the entry criteria, and were randomly assigned to one of four study interventions. An inhaled albuterol bronchodilator was compared to one of three control conditions placebo inhaler, sham acupuncture, or no intervention.

Figure 4 from the article presents subjective outcomes for two self-report measures, perceived improvements in asthma symptoms on a visual-analogue scale and perceived credibility of treatment.

Patients reported substantial improvement not only with inhaled albuterol (50% improvement) but also with inhaled placebo (45%) and with sham acupuncture (46%). In contrast, the improvement reported with no intervention was only 21%.

The difference in the subjective drug effect between the active albuterol inhaler and the placebo inhaler was not significant (P=0.12), and the observed effect size was small (d=0.21)

The two double-blind conditions did not differ significantly from each other, but sham acupuncture was significantly more credible than both inhaler conditions (P<0.05).

Figure 3 from the article  presents the outcomes for an objective measure physiological responses – improvement in forced expiratory volume (FEV1), measured with spirometry   to each intervention (albuterol inhaler, placebo inhaler, sham acupuncture, and no intervention) across the three study visits.

Notice the remarkably consistent difference between “objective” and “subjective” outcomes. To me, this proves that acupuncture doesn’t work by changing any physiologic function, but merely makes people “believe they feel better”.

The authors concluded:

1. In this repeated-measures pilot study in which active-drug and placebo effects were assessed in patients with asthma, two different types of placebo had no objective bronchodilator effect beyond the improvement that occurred when patients received no intervention of any kind and simply underwent repeated spirometry (no-intervention control).
2. In contrast, the subjective improvement in asthma symptoms with both inhaled placebo and sham acupuncture was significantly greater than the subjective improvement with the no-intervention control and was similar to that with the active drug.

Relevance to Studies of MBSR.

Claims for the efficacy of MBSR depend heavily on RCTs comparing MBSR to waitlist.

I’m unaware of comparisons of the standard waitlist control condition to more appropriate comparison/control conditions.  However, this unusual pilot study provides some suggestive evidence that a waitlist is seriously deficient when compared to credible comparison/control conditions for which patients are likely to have positive expectations.

we can see that for subjective self-report measures, the large difference between placebo conditions with positive expectations and no treatment is certainly greater than the differences typically found between the MBSR and a waitlist.

The difference between a waitlist control group and a blinded control with blinding group with positive expectations is considerably greater than the difference between MBSR and a waitlist control group.

This spells trouble for anyone wanting to crow about MBSR.

I would welcome a direct test of this hypothesis by pitting MBSR against a placebo condition with positive expectations and another comparison control condition like waitlist or no treatment.

The contrast between results from subjective self-report and objective outcomes should be troubling to those needing to evaluate MBSR or other psychological interventions for clinical or health policy applications

If one relies on studies with subjective self-report as the primary outcome, the risk is that differences for objective health measures will be missed and ineffective treatments will be accepted as effective.

For a large proportion of studies of psychological interventions for chronic health conditions, the primary outcomes are indeed subjective self-report.

Cage fights between MBSR and active control conditions.

Comparisons between MBSR an active control conditions are the real test of whether MBSR is effective and distinctively so. Such “cage fights” become particularly important when MBSR enthusiasts are not the referee. Investigator allegiance is an important determinant of outcome. Yet even when cage fights are refereed by investigators rooting for MBSR, the results can be disappointing.

In a recent blog post, I examined a trial of MBSR for smoking cessation that was published too late to be included in the comprehensive systematic review and meta-analysis.

The well-designed study… Compared mindfulness-based abstinence therapy (MBAT) to cognitive behavior therapy, which was closely matched for frequency and intensity of contact and credibility.

The control/comparison group was four  5-10 minute individual counseling sessions. Although the comparison was lopsided in terms of frequency and intensity of meetings, there were no differences among the three groups.

The authors did not emphasize that  a reason for the finding that all three groups received a nicotine patch with instructions.

I would anticipate that comparisons between MBSR and appropriate active control conditions will be slow to accumulate.

But the results at this point are not encouraging of the notion that MBSR is distinctively more effective than other active control conditions when delivered with the same frequency of contact, intensity, and positive expectations.