Lies, Damn Lies and Drug Statistics: Treatment Version – by Maia Szalavitz
When Minnesota Teen Challenge (MNTC) responded to my recent blog entry about their anti-drug program, they cited a “study” to back their claims of being an effective treatment for addiction.
What this paper actually shows is how easy it is produce good looking numbers.
In that way, it’s actually quite instructive for anyone who wants to understand addiction research—or wants to avoid being taken in by exaggerated outcome data.
Their uncontrolled research includes the classic statistical ploys used by drug programs for decades to inflate success rates.
At first glance, a claim of 74% of graduates abstinent without relapse for six months sounds pretty good. OK, it’s only six months—but often, if you can make it six months at a time without relapsing, you are doing pretty well.
Let’s look a little closer at that number and how it was generated, however.
Start with the fact that MNTC participants are not just “addicts off the street.” They are seeking treatment—either because they have to in order to avoid prison or because they have decided they want to stop using.
Most have been through a detox program to help with withdrawal—and many will have dropped out before completing that. MNTC participants have also consented to attend a highly religious rehab—or had their parents consent for them. This suggests that we have already eliminated many of the addicts who aren’t motivated to recover before they even set foot in the door.
Because there is no control group, all of those facts already mean that any success in the program we see could be due to pre-existing motivation: not to anything special about the rehab. Only with a control group of similar people who get no treatment or attend a different rehab can we really tell what works and what doesn’t.
The second important clue to pick up is that word “graduates.” This means that no one who started but dropped out of the program was counted in the success rates.
When you do a clinical trial, you have to include drop-outs as failures to make your results as applicable as possible to the “real world.” Why?
Well, let’s say you have a drug that looks like a fabulous antidepressant in rats. You give it to 100 people and one person stays in the trial and is no longer depressed – but 99 drop out because the drug also causes an intense itching sensation all over the body. Not gonna be a blockbuster—not a useful drug, period. Your stock’s gonna tank!
The above is a great explanation of why “drop outs” must be counted as failures.
But by MNTC’s measure, there’s a 100% success with that one person who didn’t get the side effect!!!
so let’s use the low end of the typical drop out rate and say that 50% quit without graduating. (MNTC has also said that it uses different practices than the the national organization).
That cuts their 74% success rate in half to 37% if we look at this as what is called an “intention to treat,” study and count the dropouts as failures. That already puts them around average compared to other programs.
Your average drug program can get about 30-40% clean for six months
And with MNTC, there’s another wrinkle. When they surveyed the graduates, only 55% agreed to participate in the follow up study. This is another red flag.
If you think about it, who is most likely to be available to participate in a follow up on rehab success—the guy who is employed and off drugs or the one in jail or in the basement smoking meth?
Who is most likely to want to talk to a representative of a rehab—the woman who got clean or the one who is now depressed about failure, particularly if it’s a religious rehab where failure means return to sin?
We can’t assume that all of the 45% of the graduates who don’t respond failed—it’s just much more likely that the responders will be successes than failures.
So, if we again give them the benefit of the doubt and assume that only half of these have relapsed (again, a very generous assumption), we can say that roughly 31% of the total sample has stayed off drugs for six months.
If we count all of the nonresponders as failures, that yields a 20% abstinence rate at six months for graduates
Which is pretty close to the 15% who get clean, on average at any given time, with no treatment whatsoever!
The extra five percent, however, could well be a selection effect due to the motivated patients being the only ones who started treatment in the first place.
there is absolutely no basis for this program to make any claim of superiority to other treatments— some of which do demonstrate greater success without playing with data and without the risks carried by being outside of mainstream medicine.