Sampling Bias in Pain Research: Update

Pain, Please: an Investigation of Sampling Bias in Pain Research – Kai Karos, Jessica M. Alleva, Madelon L. Peter – Science Direct – March 2018

I only had access to the abstract of this article in March, but now I have a PDF of the whole study – excerpts follow the abstract below.

The results are quite clear: thrill seekers – the ones who are more susceptible to addiction – are the ones who sign up for pain studies.

Highlights

• Experimental pain research may be especially susceptible for sampling bias.
• Fear of pain was associated with perceived likelihood of participation.
• Sensation seeking was associated with participation in experimental pain research.
• Sampling bias can threaten the external validity and generalizability of pain research.

Experimental pain research frequently relies on the recruitment of volunteers. However, because experimental pain research often involves unpleasant and painful sensations, it may be especially susceptible to sampling bias.

That is, volunteers in experimental pain research might differ from nonvolunteers on several relevant variables that could affect the generalizability and external validity of the research.

This is even more blatant when lab animals are used: Lab animals studied in unrealistic conditions

We conducted two studies to investigate potential sampling bias in experimental pain research.

In Study 1 we assessed participants’ (N = 275; Age: 17 to 30 years) perceived likelihood of participating in pain research. Pain catastrophizing, fear of pain, illness and injury sensitivity, depression, anxiety, sensation seeking, gender identity, body appreciation, and social desirability were also assessed as potential predictors of the likelihood to participate

In Study 2, participants (N = 87; Age: 18 to 31 years) could sign up for two nearly identical studies, with only one involving painful sensations

36 participants signed up for the pain study and

51 participants signed up for the no-pain study.

Study 1 showed that lower levels of fear of pain, higher levels of sensation seeking, and older age predicted the perceived likelihood of participating in pain research

Study 2 demonstrated significantly higher levels of sensation seeking in participants who signed up for the pain study compared to those who signed up for the no-pain study.

The implications of these findings for future research, as well as the clinical conclusions based on experimental pain research, are discussed.

---

Below are excerpts from the full article:

A large body of psychological and medical research relies on the recruitment of volunteers,  often students, making it susceptible to sampling bias. A review of sampling bias (sometimes referred  to as volunteer, recruitment, or self-selection bias), argued that volunteers differ from nonvolunteers and that these  differences can directly affect generalizability and external validity.

There are several areas of research in which the presence of sampling bias is well documented,  such as

• general laboratory research,
• sexuality research, and
• body image research.

Studies concerned with unusual topics have been thought to attract volunteers who may particularly deviate from the norms of the  general population.

Experimental pain research might be especially susceptible to sampling bias, because it commonly involves unpleasant and painful sensations. In fact, volunteers can often choose from a variety of different studies  with comparable compensation, most of which do not involve any unpleasant or painful sensations.

Paradoxically, several traits that are relevant in the development and maintenance of chronic pain, such as pain-related fear, pain catastrophizing, illness and injury sensitivity,  and depression and anxiety might be abnormally low in volunteers participating in pain research.

Further, individuals volunteering in research involving unpleasant, often unknown sensations might do so because  they seek novel sensations more so than the average volunteer or because of social desirability.

Speculatively, gender identity might play a role as well. Individuals  with a more masculine (and less feminine) gender identity might also feel more drawn to pain research, for example in an effort to affirm their “toughness” or “bravery.”

In addition, individuals who avoid participating in research that could cause unnecessary discomfort  to their body might be more protective and appreciative of their body, demonstrated by increased body appreciation

The presence of sampling bias would pose a risk to the  generalizability as well as internal and external validity  of experimental pain research, especially when clinical populations are compared with “healthy controls.”

Differences in such studies are often interpreted as symptoms  of the disorder, even though they could also be caused by abnormalities in the reference group. Despite these serious implications and its susceptibility for sampling bias,  to the best of our knowledge, empirical research investigating sampling bias in pain research has not yet been conducted.

We investigated the presence of sampling bias in experimental pain research in 2 studies.

In study 1, the  relationship between the perceived likelihood of participating in research involving painful sensations and  several pain-specific (eg, fear of pain) and additional (eg, sensation-seeking) outcomes was investigated.

In study  2, using a more ecologically valid paradigm, we compared 2 samples of participants who either signed up for a study involving painful stimuli or a study without such  stimuli that was otherwise identical. We hypothesized that individuals who are likely to participate in experimental pain research have lower levels of fear of pain, pain  catastrophizing, and illness and injury sensitivity. In addition, we also explored possible differences in depression, anxiety, sensation-seeking, gender identity, body appreciation, social desirability, age, and sex.

Study 1

Methods 

Participants were 275 (63 male) undergraduates  between 17 and 30 years old

Measures

Fear of Pain Questionnaire-3.

The Fear of Pain  Questionnaire-3 (FPQ)25 assesses pain-related fear and  consists of 30 items describing painful experiences (eg,  “Being in an automobile accident”).

Participants are asked  to “look at each item and think about how fearful you are of experiencing the pain associated with each item.”  If they had never had a particular painful experience, they are told that they should “answer on the basis of how fearful you expect you would be if you had such  an experience.” Items are rated from 1 = not at all to 5 = extreme. Scores on the 30 items are summed (range = 30–150), with higher scores reflecting greater fear of  pain.

FPQ total and subscale scores have shown good internal consistency and construct validity in healthy U.S. female and male undergraduates

Pain Catastrophizing Scale.

The Pain Catastrophizing  Scale (PCS)41 is a 13-item scale that assesses catastrophic  pain-related cognitions. Participants are asked to reflect on past painful experiences and to indicate the extent  (1 = not at all to 4 = all the time) to which they experienced the thoughts and feelings described (eg, “I feel I can’t go on”). Scores on these 13 items are summed,  with higher scores indicating higher levels of pain catastrophizing (range = 0–52).

PCS scores have shown good internal consistency and construct validity in healthy Canadian female and male undergraduates.

Illness-Injury Sensitivity Inventory-Revised.

The IllnessInjury Sensitivity Inventory-Revised (IISI-R)6 comprises 9  items (eg, “I worry about my physical health”), rated from 0 = agree very little to 4 = agree very much. Item scores are summed (range = 0–36), with higher  scores reflecting greater fear of illness and injury.

IISI-R total and subscale scores have shown good internal consistency and construct validity in U.S. female and male  undergraduates

Patient-Reported Outcomes Measurement Information System Depression and Anxiety Short Forms.

The  Patient-Reported Outcomes Measurement Information System (PROMIS) Depression (eg, “I felt worthless”) and  PROMIS Anxiety (eg, “I felt fearful”) Short Forms1 each contain 4 items rated from 1 = never to 5 = always, and  concern how participants have felt in the past week.

Items were designed to be applicable to clinical as well as general populations. Scores for each short form are averaged (range = 1–5), with higher scores reflecting higher  levels of depression and anxiety, respectively.

Scores on the items of these subscales have shown adequate psychometric properties in U.S. women and men (see http:// http://www.healthmeasures.net for details)

Brief Sensation Seeking Scale.

The Brief Sensation  Seeking Scale (BSSS)20 consists of 8 items (eg, “I like to  do frightening things”), rated from 1 = strongly disagree to 5 = strongly agree. Scores on the BSSS items are  averaged (range = 1–5), with higher scores indicating higher levels of sensation-seeking. BSSS scores have shown  good internal consistency and construct validity in U.S. female and male undergraduates.

Personal Attributes Questionnaire.

The Personal Attributes Questionnaire (PAQ)38 comprises 24 items that  are rated on 5-point bipolar scales. These items are divided into Masculinity (8 items; eg, not at all aggressive to very  aggressive), Femininity (8 items; eg, not at all emotional to very emotional), and Androgyny (9 items); eg, very home-oriented to very worldly) subscales.

For each  item, the extreme Masculine, Feminine, or Androgynous response is given a score of 4, the next most extreme  response is scored 3, etc. Scores on the items of each subscale are summed (range = 0–24), with higher scores  indicating a more masculine, feminine, or androgynous gender identity, respectively.

PAQ subscale scores have shown good internal consistency and construct validity.  For the purpose of this study, only the items from the Masculine and Feminine subscales were analyzed because they relate to earlier research investigating the importance of gender roles and pain

Body Appreciation Scale-2.

The Body Appreciation  Scale-2 (BAS-2)44 comprises 10 items (eg, “I respect my  body”), rated from 1 = never to 5 = always. Scores on these  items are averaged (range = 1–5), with higher scores reflecting higher levels of body appreciation. BAS-2 scores  have shown good 21-day test-retest reliability, construct validity, and internal consistency in U.S. female and male undergraduates.

Balanced Inventory of Desirable Responding-6 Impression Management Subscale.

The Impression  Management Subscale40 comprises 20 items (eg, “I sometimes tell lies if I have to”) that are rated from 1 = not  at all true to 6 = very true, of which 10 items are reverse scored. Items rated with a 6 or 7 are given 1 extra point before item scores are summed (range = 20–160).  

Higher scores indicate a stronger tendency to respond in a socially-desirable manner. Scores on the Impression Management Subscale have shown good internal consistency and construct validity in U.S. female and male  undergraduates.

Likelihood of Participating in Pain Research.

Participants were told: “At Maastricht University and the  KU Leuven, people have the opportunity to participate in different kinds of studies for financial compensation  or participation credits. In the following, please indicate how likely it is that you would participate in the following kinds of studies if given the opportunity.”  

Several potential study types were listed in the following order: studies involving brain imaging, computer tasks, painful/unpleasant stimuli, food consumption, physical  activity, surveys, and medication.

The study type of interest was “studies involving painful/unpleasant stimuli (eg, electrical stimulation).” Participants responded from 1 = extremely unlikely to 6 = extremely likely for each study  type. A higher score on the pain-relevant item indicates a greater perceived likelihood of participating in research involving painful stimuli.

To explore the potential reasons behind participants’ responses, on the following page they were asked,  “You indicated that it is [response inserted here] that you would participate in a study involving painful/unpleasant  stimuli (eg, electrical stimulation). Please explain why.” To disguise the focus of the study, participants were first asked to explain their response concerning studies involving “consumption of food (eg, chocolate).”

The  responses to the question concerning participation in pain research were later independently read by the first and second authors. Both authors identified the most common themes among participants’ responses.

Subsequently, both  authors compared their identified themes and agreed on a final set. Last, they categorized participants’ responses according to this final set of themes. Note that this qualitative analysis was post hoc and serves mainly informative purposes.

Study 2

Study 2 aimed to investigate whether the findings from  study 1 translate to behavior: do individuals who are given  the chance to sign up for research involving painful stimuli differ from those who do not sign up for research involving painful stimuli?

To this end, we placed advertisements at 2 university campuses for 2 nearly identical studies, one purportedly involving painful stimuli and the other purportedly involving neutral stimuli.

We included the main  pain-related outcomes (fear of pain, pain catastrophizing, and illness-injury sensitivity) as well as the additional significant predictors from study 1 (sensation-seeking, age).  

On the basis of study 1, we predicted that, compared with participants who signed up for the study involving neutral stimuli, those who signed up for the study involving painful stimuli would be older and demonstrate lower levels of  fear of pain, and higher levels of sensation-seeking.

Last, we explored whether a larger number of participants signing up for the study involving painful stimuli had previously participated in pain research.

Results

Group Comparisons on the Main Outcomes

As shown in Table 6, and contrary to our hypothesis,  there were no significant differences in pain catastrophizing, fear of pain, illness and injury sensitivity, or age between the pain group and the no-pain  group. However, participants in the pain group scored significantly higher on sensation-seeking compared with participants in the no-pain group (t85 = 2.349, P = .021),  and the magnitude of this difference was moderate (d = .512 [.350–4.334]).

Discussion

This study investigated sampling bias in studies involving painful and unpleasant stimuli.

As expected, study 1 showed that fear of pain was associated with reduced perceived likelihood to participate in pain research. However, fear of pain was not associated with actual participation in study 2.

Fear of pain, has been  implicated in the development and maintenance of chronic pain and several studies have shown that patients exhibit higher levels of fear of pain than painfree controls.

Specifically, fear of pain has been implicated in the avoidance of pain, so it is unsurprising that  individuals with high levels of fear of pain are also likelier to report avoiding pain research. Note that fear of pain scores in our sample were comparable with a similar sample of Dutch healthy undergraduates.

In contrast, pain catastrophizing did not seem to predict the likelihood  to participate in pain research. Interestingly, most participants indicated that it is unlikely that they would participate in pain research, demonstrating that it seems to be the rule—rather than the exception—to avoid research involving painful or unpleasant sensations.

Specifically, the 2 most frequent reasons provided to avoid  pain research were avoidance of harm and being afraid of pain. Older individuals also indicated being likelier to participate in pain research. This effect was possibly driven by positive previous experiences with pain research, which we did not assess in study 1 but was one of the main reasons given by participants who said they  would consider participating in pain research.

Importantly, sensation-seeking was associated with participation in pain research in study 1 as well as in study  2.

Many participants are unfamiliar with painful stimulation in research and might seek out this novel experience because of curiosity. When participants were asked why  they would participate in pain research, personal growth and curiosity was indeed one of the most frequent reasons given. Interestingly, this difference in sensation-seeking  was not related to sex or gender identity.

Previous research has shown that sensation- and risk-seeking are related to hypermasculinity and that men are more likely to participate in sexuality research. Concerning pain,  even though a masculine gender identity is associated with “toughness,” it was not associated with the likelihood of participating in pain research in this study.

There is currently limited research on the relationship between sensation-seeking and pain. However, the  extant research suggests that sensation-seeking is associated with reduced pain sensitivity, better coping  with pain, and increased placebo responding.

This might have implications for the generalizability of pain research involving volunteers. For instance, sensationseeking could be antithetical to the avoidance of pain,  because it is explicitly concerned with seeking out novel and arousing situations or stimuli (including pain).

If this goal is salient in participants in pain research, it would substantially differ from patients with acute and chronic  pain who are primarily concerned with the avoidance and minimization of pain.

In fact, this level of sensation-seeking might even differ from the general population and represent an especially resilient subset of individuals who actively seek out novel and sometimes unpleasant stimuli. These notions should be tested  in future research.

When examining behavior rather than intention in  study 2, differences in fear of pain were not replicated.  There are several reasons that this might be. First, the intention assessed in study 1 might not translate into behavior. Research into the “intention-behavior gap” shows  that intentions are only translated into actions approximately 50% of the time.

…there was a subtle difference in how painful  studies were described in studies 1 and 2. In study 1, the  study was explicitly described in terms of involving painful/  unpleasant stimuli, whereas in study 2 “sensory stimuli” that “might” feel unpleasant/painful were mentioned as  one of several distractors. Therefore, the pain element might have been less salient in study 2 than in study 1.

Considering that study 2 resembled a frequently-used recruitment strategy, this might be good news: including  information about painful sensations in the study description but not making it too salient might help to counteract sampling bias.

Relatedly, results from study 2 suggest that any differences between volunteers and nonvolunteers may be modest.

Nevertheless, the present findings suggest that there  are at least some relevant differences between volunteers and nonvolunteers in pain research that are worth  considering. If replicated in future investigations, this knowledge will spark important discussions concerning

The risk of making this assumption prematurely is that differences between pain-free individuals  and pain patients might be

• misinterpreted (eg, painfree individuals are characterized by increased differential  learning abilities rather than a deficit in pain patients),
• exaggerated (eg, the difference between a resilient subset  of the population and pain patients is likely larger than between the general population and pain patients), or
• underestimated (ie, correlations on the basis of a restricted range of scores will be smaller than that on the  basis of a normative sample).

…the validity of using volunteers in pain research. For  example, especially in translational pain research, painfree volunteers are frequently compared with pain  patients and differences are interpreted as symptoms or etiologic factors in the development of chronic pain.

Yet,  to interpret these differences as symptoms is to assume that the pain-free individuals in these studies indeed represent the general population.

This assumption is implicit  in many studies comparing pain patients with pain-free individuals, and in studies assessing “normal” responses to pain in pain-free individuals in areas such as

• neuroimaging,
• learning differences, and
• affective  processing.

Sampling bias in pain research might also distort

• estimates of frequency and  prevalence,
• the relationship between different variables, or
• the observed natural development of acute and  chronic pain conditions.

Conclusions

Our studies might provide initial evidence for the presence of sampling bias in experimental pain research. We  hope that this research sparks future empirical investigations of

• the factors that influence individuals’ decision  to participate in pain research,
• the possible implications for the validity and generalizability of pain research, and
• what can be done about it.

Selection needs to be  more widely addressed, including sample characteristics and motivations for participation in pain research, and the generalizability of studies should be discussed in more detail with this in mind

Supplementary Data

Supplementary data related to this article can be found  at https://doi:10.1016/j.jpain.2018.02.011.

(July), 2018