Pain, Please: an Investigation of Sampling Bias in Pain Research – Kai Karos, Jessica M. Alleva, Madelon L. Peter – Science Direct – March 2018
I only had access to the abstract of this article in March, but now I have a PDF of the whole study – excerpts follow the abstract below.
The results are quite clear: thrill seekers – the ones who are more susceptible to addiction – are the ones who sign up for pain studies.
- Experimental pain research may be especially susceptible for sampling bias.
- Fear of pain was associated with perceived likelihood of participation.
- Sensation seeking was associated with participation in experimental pain research.
- Sampling bias can threaten the external validity and generalizability of pain research.
Experimental pain research frequently relies on the recruitment of volunteers. However, because experimental pain research often involves unpleasant and painful sensations, it may be especially susceptible to sampling bias.
That is, volunteers in experimental pain research might differ from nonvolunteers on several relevant variables that could affect the generalizability and external validity of the research.
This is even more blatant when lab animals are used: Lab animals studied in unrealistic conditions
We conducted two studies to investigate potential sampling bias in experimental pain research.
In Study 1 we assessed participants’ (N = 275; Age: 17 to 30 years) perceived likelihood of participating in pain research. Pain catastrophizing, fear of pain, illness and injury sensitivity, depression, anxiety, sensation seeking, gender identity, body appreciation, and social desirability were also assessed as potential predictors of the likelihood to participate
In Study 2, participants (N = 87; Age: 18 to 31 years) could sign up for two nearly identical studies, with only one involving painful sensations
36 participants signed up for the pain study and
51 participants signed up for the no-pain study.
Study 1 showed that lower levels of fear of pain, higher levels of sensation seeking, and older age predicted the perceived likelihood of participating in pain research
Study 2 demonstrated significantly higher levels of sensation seeking in participants who signed up for the pain study compared to those who signed up for the no-pain study.
The implications of these findings for future research, as well as the clinical conclusions based on experimental pain research, are discussed.
Below are excerpts from the full article:
A large body of psychological and medical research relies on the recruitment of volunteers, often students, making it susceptible to sampling bias. A review of sampling bias (sometimes referred to as volunteer, recruitment, or self-selection bias), argued that volunteers differ from nonvolunteers and that these differences can directly affect generalizability and external validity.
There are several areas of research in which the presence of sampling bias is well documented, such as
- general laboratory research,
- sexuality research, and
- body image research.
Studies concerned with unusual topics have been thought to attract volunteers who may particularly deviate from the norms of the general population.
Experimental pain research might be especially susceptible to sampling bias, because it commonly involves unpleasant and painful sensations. In fact, volunteers can often choose from a variety of different studies with comparable compensation, most of which do not involve any unpleasant or painful sensations.
Paradoxically, several traits that are relevant in the development and maintenance of chronic pain, such as pain-related fear, pain catastrophizing, illness and injury sensitivity, and depression and anxiety might be abnormally low in volunteers participating in pain research.
Further, individuals volunteering in research involving unpleasant, often unknown sensations might do so because they seek novel sensations more so than the average volunteer or because of social desirability.
Speculatively, gender identity might play a role as well. Individuals with a more masculine (and less feminine) gender identity might also feel more drawn to pain research, for example in an effort to affirm their “toughness” or “bravery.”
In addition, individuals who avoid participating in research that could cause unnecessary discomfort to their body might be more protective and appreciative of their body, demonstrated by increased body appreciation
The presence of sampling bias would pose a risk to the generalizability as well as internal and external validity of experimental pain research, especially when clinical populations are compared with “healthy controls.”
Differences in such studies are often interpreted as symptoms of the disorder, even though they could also be caused by abnormalities in the reference group. Despite these serious implications and its susceptibility for sampling bias, to the best of our knowledge, empirical research investigating sampling bias in pain research has not yet been conducted.
We investigated the presence of sampling bias in experimental pain research in 2 studies.
In study 1, the relationship between the perceived likelihood of participating in research involving painful sensations and several pain-specific (eg, fear of pain) and additional (eg, sensation-seeking) outcomes was investigated.
In study 2, using a more ecologically valid paradigm, we compared 2 samples of participants who either signed up for a study involving painful stimuli or a study without such stimuli that was otherwise identical. We hypothesized that individuals who are likely to participate in experimental pain research have lower levels of fear of pain, pain catastrophizing, and illness and injury sensitivity. In addition, we also explored possible differences in depression, anxiety, sensation-seeking, gender identity, body appreciation, social desirability, age, and sex.
Participants were 275 (63 male) undergraduates between 17 and 30 years old
Fear of Pain Questionnaire-3.
The Fear of Pain Questionnaire-3 (FPQ)25 assesses pain-related fear and consists of 30 items describing painful experiences (eg, “Being in an automobile accident”).
Participants are asked to “look at each item and think about how fearful you are of experiencing the pain associated with each item.” If they had never had a particular painful experience, they are told that they should “answer on the basis of how fearful you expect you would be if you had such an experience.” Items are rated from 1 = not at all to 5 = extreme. Scores on the 30 items are summed (range = 30–150), with higher scores reflecting greater fear of pain.
FPQ total and subscale scores have shown good internal consistency and construct validity in healthy U.S. female and male undergraduates
Pain Catastrophizing Scale.
The Pain Catastrophizing Scale (PCS)41 is a 13-item scale that assesses catastrophic pain-related cognitions. Participants are asked to reflect on past painful experiences and to indicate the extent (1 = not at all to 4 = all the time) to which they experienced the thoughts and feelings described (eg, “I feel I can’t go on”). Scores on these 13 items are summed, with higher scores indicating higher levels of pain catastrophizing (range = 0–52).
PCS scores have shown good internal consistency and construct validity in healthy Canadian female and male undergraduates.
Illness-Injury Sensitivity Inventory-Revised.
The IllnessInjury Sensitivity Inventory-Revised (IISI-R)6 comprises 9 items (eg, “I worry about my physical health”), rated from 0 = agree very little to 4 = agree very much. Item scores are summed (range = 0–36), with higher scores reflecting greater fear of illness and injury.
IISI-R total and subscale scores have shown good internal consistency and construct validity in U.S. female and male undergraduates
Patient-Reported Outcomes Measurement Information System Depression and Anxiety Short Forms.
The Patient-Reported Outcomes Measurement Information System (PROMIS) Depression (eg, “I felt worthless”) and PROMIS Anxiety (eg, “I felt fearful”) Short Forms1 each contain 4 items rated from 1 = never to 5 = always, and concern how participants have felt in the past week.
Items were designed to be applicable to clinical as well as general populations. Scores for each short form are averaged (range = 1–5), with higher scores reflecting higher levels of depression and anxiety, respectively.
Scores on the items of these subscales have shown adequate psychometric properties in U.S. women and men (see http:// http://www.healthmeasures.net for details)
Brief Sensation Seeking Scale.
The Brief Sensation Seeking Scale (BSSS)20 consists of 8 items (eg, “I like to do frightening things”), rated from 1 = strongly disagree to 5 = strongly agree. Scores on the BSSS items are averaged (range = 1–5), with higher scores indicating higher levels of sensation-seeking. BSSS scores have shown good internal consistency and construct validity in U.S. female and male undergraduates.
Personal Attributes Questionnaire.
The Personal Attributes Questionnaire (PAQ)38 comprises 24 items that are rated on 5-point bipolar scales. These items are divided into Masculinity (8 items; eg, not at all aggressive to very aggressive), Femininity (8 items; eg, not at all emotional to very emotional), and Androgyny (9 items); eg, very home-oriented to very worldly) subscales.
For each item, the extreme Masculine, Feminine, or Androgynous response is given a score of 4, the next most extreme response is scored 3, etc. Scores on the items of each subscale are summed (range = 0–24), with higher scores indicating a more masculine, feminine, or androgynous gender identity, respectively.
PAQ subscale scores have shown good internal consistency and construct validity. For the purpose of this study, only the items from the Masculine and Feminine subscales were analyzed because they relate to earlier research investigating the importance of gender roles and pain
Body Appreciation Scale-2.
The Body Appreciation Scale-2 (BAS-2)44 comprises 10 items (eg, “I respect my body”), rated from 1 = never to 5 = always. Scores on these items are averaged (range = 1–5), with higher scores reflecting higher levels of body appreciation. BAS-2 scores have shown good 21-day test-retest reliability, construct validity, and internal consistency in U.S. female and male undergraduates.
Balanced Inventory of Desirable Responding-6 Impression Management Subscale.
The Impression Management Subscale40 comprises 20 items (eg, “I sometimes tell lies if I have to”) that are rated from 1 = not at all true to 6 = very true, of which 10 items are reverse scored. Items rated with a 6 or 7 are given 1 extra point before item scores are summed (range = 20–160).
Higher scores indicate a stronger tendency to respond in a socially-desirable manner. Scores on the Impression Management Subscale have shown good internal consistency and construct validity in U.S. female and male undergraduates.
Likelihood of Participating in Pain Research.
Participants were told: “At Maastricht University and the KU Leuven, people have the opportunity to participate in different kinds of studies for financial compensation or participation credits. In the following, please indicate how likely it is that you would participate in the following kinds of studies if given the opportunity.”
Several potential study types were listed in the following order: studies involving brain imaging, computer tasks, painful/unpleasant stimuli, food consumption, physical activity, surveys, and medication.
The study type of interest was “studies involving painful/unpleasant stimuli (eg, electrical stimulation).” Participants responded from 1 = extremely unlikely to 6 = extremely likely for each study type. A higher score on the pain-relevant item indicates a greater perceived likelihood of participating in research involving painful stimuli.
To explore the potential reasons behind participants’ responses, on the following page they were asked, “You indicated that it is [response inserted here] that you would participate in a study involving painful/unpleasant stimuli (eg, electrical stimulation). Please explain why.” To disguise the focus of the study, participants were first asked to explain their response concerning studies involving “consumption of food (eg, chocolate).”
The responses to the question concerning participation in pain research were later independently read by the first and second authors. Both authors identified the most common themes among participants’ responses.
Subsequently, both authors compared their identified themes and agreed on a final set. Last, they categorized participants’ responses according to this final set of themes. Note that this qualitative analysis was post hoc and serves mainly informative purposes.
Study 2 aimed to investigate whether the findings from study 1 translate to behavior: do individuals who are given the chance to sign up for research involving painful stimuli differ from those who do not sign up for research involving painful stimuli?
To this end, we placed advertisements at 2 university campuses for 2 nearly identical studies, one purportedly involving painful stimuli and the other purportedly involving neutral stimuli.
We included the main pain-related outcomes (fear of pain, pain catastrophizing, and illness-injury sensitivity) as well as the additional significant predictors from study 1 (sensation-seeking, age).
On the basis of study 1, we predicted that, compared with participants who signed up for the study involving neutral stimuli, those who signed up for the study involving painful stimuli would be older and demonstrate lower levels of fear of pain, and higher levels of sensation-seeking.
Last, we explored whether a larger number of participants signing up for the study involving painful stimuli had previously participated in pain research.
Group Comparisons on the Main Outcomes
As shown in Table 6, and contrary to our hypothesis, there were no significant differences in pain catastrophizing, fear of pain, illness and injury sensitivity, or age between the pain group and the no-pain group. However, participants in the pain group scored significantly higher on sensation-seeking compared with participants in the no-pain group (t85 = 2.349, P = .021), and the magnitude of this difference was moderate (d = .512 [.350–4.334]).
This study investigated sampling bias in studies involving painful and unpleasant stimuli.
As expected, study 1 showed that fear of pain was associated with reduced perceived likelihood to participate in pain research. However, fear of pain was not associated with actual participation in study 2.
Fear of pain, has been implicated in the development and maintenance of chronic pain and several studies have shown that patients exhibit higher levels of fear of pain than painfree controls.
Specifically, fear of pain has been implicated in the avoidance of pain, so it is unsurprising that individuals with high levels of fear of pain are also likelier to report avoiding pain research. Note that fear of pain scores in our sample were comparable with a similar sample of Dutch healthy undergraduates.
In contrast, pain catastrophizing did not seem to predict the likelihood to participate in pain research. Interestingly, most participants indicated that it is unlikely that they would participate in pain research, demonstrating that it seems to be the rule—rather than the exception—to avoid research involving painful or unpleasant sensations.
Specifically, the 2 most frequent reasons provided to avoid pain research were avoidance of harm and being afraid of pain. Older individuals also indicated being likelier to participate in pain research. This effect was possibly driven by positive previous experiences with pain research, which we did not assess in study 1 but was one of the main reasons given by participants who said they would consider participating in pain research.
Importantly, sensation-seeking was associated with participation in pain research in study 1 as well as in study 2.
Many participants are unfamiliar with painful stimulation in research and might seek out this novel experience because of curiosity. When participants were asked why they would participate in pain research, personal growth and curiosity was indeed one of the most frequent reasons given. Interestingly, this difference in sensation-seeking was not related to sex or gender identity.
Previous research has shown that sensation- and risk-seeking are related to hypermasculinity and that men are more likely to participate in sexuality research. Concerning pain, even though a masculine gender identity is associated with “toughness,” it was not associated with the likelihood of participating in pain research in this study.
There is currently limited research on the relationship between sensation-seeking and pain. However, the extant research suggests that sensation-seeking is associated with reduced pain sensitivity, better coping with pain, and increased placebo responding.
This might have implications for the generalizability of pain research involving volunteers. For instance, sensationseeking could be antithetical to the avoidance of pain, because it is explicitly concerned with seeking out novel and arousing situations or stimuli (including pain).
If this goal is salient in participants in pain research, it would substantially differ from patients with acute and chronic pain who are primarily concerned with the avoidance and minimization of pain.
In fact, this level of sensation-seeking might even differ from the general population and represent an especially resilient subset of individuals who actively seek out novel and sometimes unpleasant stimuli. These notions should be tested in future research.
When examining behavior rather than intention in study 2, differences in fear of pain were not replicated. There are several reasons that this might be. First, the intention assessed in study 1 might not translate into behavior. Research into the “intention-behavior gap” shows that intentions are only translated into actions approximately 50% of the time.
…there was a subtle difference in how painful studies were described in studies 1 and 2. In study 1, the study was explicitly described in terms of involving painful/ unpleasant stimuli, whereas in study 2 “sensory stimuli” that “might” feel unpleasant/painful were mentioned as one of several distractors. Therefore, the pain element might have been less salient in study 2 than in study 1.
Considering that study 2 resembled a frequently-used recruitment strategy, this might be good news: including information about painful sensations in the study description but not making it too salient might help to counteract sampling bias.
Relatedly, results from study 2 suggest that any differences between volunteers and nonvolunteers may be modest.
Nevertheless, the present findings suggest that there are at least some relevant differences between volunteers and nonvolunteers in pain research that are worth considering. If replicated in future investigations, this knowledge will spark important discussions concerning
The risk of making this assumption prematurely is that differences between pain-free individuals and pain patients might be
- misinterpreted (eg, painfree individuals are characterized by increased differential learning abilities rather than a deficit in pain patients),
- exaggerated (eg, the difference between a resilient subset of the population and pain patients is likely larger than between the general population and pain patients), or
- underestimated (ie, correlations on the basis of a restricted range of scores will be smaller than that on the basis of a normative sample).
…the validity of using volunteers in pain research. For example, especially in translational pain research, painfree volunteers are frequently compared with pain patients and differences are interpreted as symptoms or etiologic factors in the development of chronic pain.
Yet, to interpret these differences as symptoms is to assume that the pain-free individuals in these studies indeed represent the general population.
This assumption is implicit in many studies comparing pain patients with pain-free individuals, and in studies assessing “normal” responses to pain in pain-free individuals in areas such as
- learning differences, and
- affective processing.
Sampling bias in pain research might also distort
- estimates of frequency and prevalence,
- the relationship between different variables, or
- the observed natural development of acute and chronic pain conditions.
Our studies might provide initial evidence for the presence of sampling bias in experimental pain research. We hope that this research sparks future empirical investigations of
- the factors that influence individuals’ decision to participate in pain research,
- the possible implications for the validity and generalizability of pain research, and
- what can be done about it.
Selection needs to be more widely addressed, including sample characteristics and motivations for participation in pain research, and the generalizability of studies should be discussed in more detail with this in mind
Supplementary data related to this article can be found at https://doi:10.1016/j.jpain.2018.02.011.