The Problem with Surrogate Endpoints – Alex Gertner – Twitter stream from @setmoreoff – Feb 2019
Mr. Gertner shows why you cannot measure drug use by counting various “surrogate outcomes” like those listed below and expect to get a realistic number. Measurements can be dramatically skewed and lead to wildly inaccurate results when invalid surrogate endpoints are chosen.
1/ If you’d like to research the effect of a policy or program on drug use rates then you need to measure drug use.
- ED overdose visits,
- drug related arrests, and
- overdose deaths
are not good measures of drug use rates. Thoughts from >10 years working in drug policy & research
2/ Most drug poisonings likely do not reach ED.
ED overdose visits are determined in part by
- how visible PWUD are in a community,
- attitude of EMS/police towards PWUD,
- health provider preferences for documentation,
- availability of tx, and
- variability in composition of drug supply.
3/ Overdose deaths and arrests are similarly affected by many of these factors.
ED visits and overdose deaths have been moving in opposite directions in many places.
The underlying drug use rate may be increasing, decreasing or stable.
Don’t assume your policy affected drug use.
This is critically important!
It’s not difficult to find some numbers and manipulate them with statistics to show a reduction in drug use, but before you declare yourself victorious, you have to prove these numbers changed because of your policy.
Connecting policy and its results is far more complicated than most people assume.
4/ If the policy or program you’re studying appears to affect ED overdose visits consider if it’s a policy that makes PWUD more visible, easy to identify, or more connected to health system.
A change in ED visits is not a change in drug use. Same goes from drug related arrests.
5/ If the policy you’re studying appears to affect overdose deaths consider carefully whether there may be selection bias in your study.
The last few years have taught us that overdose death rates are hard to move and places that have done the most are often also most affected.
6/ If you’d like to make claims about drug use but don’t have a measure of drug use in your data then consider primary data collection.
If you don’t feel you have the training or resources for that consider partnerships and grants. It’s better than coming to wrong conclusions.
The scale of drug use dwarfs the scale of ED overdose visits and overdose deaths.
The vast majority of drug use does not lead to these outcomes.
If ED visits change it’s more likely a change in proximal factors that determine ED visits than a change in drug use rates. https://twitter.com/setmoreoff/status/1097512995876425729/photo/1
Bottom line: if you wouldn’t use
- drowning as a measure of swimming or vehicle
- deaths as a measure of driving or
- ED burn visits as a measure of candle use,
then don’t use overdose as a measure of drug use.
Overdoses are relatively rare events with complex contributing factors.
I’ve posted previously about the problems with surrogate endpoints in research: