Does exposure to political content drive survey satisficing?

Summary and takeaway

In this preregistered survey experiment, I provide evidence that exposure to political content can increase satisficing behavior among survey respondents. Exposure to three questions with affective political content, relative to other topics, increased satisficing behavior in the full sample by roughly 1 percentage point. The minor decrease in data quality suggests that researchers should take care in the ordering of their questionnaires when surveys include content that invokes political actors, political parties and ideologies – especially when these questions deal with charged content rather than purely factual information.

Introduction and background

One perennial consideration in survey science is the phenomenon of satisficing. Satisficing is a form of response that occurs when respondents experience a depletion of cognitive resources.

When respondents’ cognitive resources diminish, they often do not drop out of the survey due to their perceived social obligation to the researcher. Instead, their response quality begins to suffer, as measured by a variety of metrics. Satisficing raises the likelihood that respondents speed through a survey. It raises the failure rate of attention checks, increases the willingness to “straight-line” by selecting the same response option many times in a row, and degrades other related measures of attentiveness. These data quality issues can damage a survey’s ability to accurately measure quantities of interest.

Political content has the potential to rapidly deplete the cognitive resources of U.S.-based survey respondents. Political content can invoke intense affective reactions, especially because politics is often associated with negativity. The combination of cognitive and affect demands inherent in political content may overwhelm respondents’ ability to process further information.

This may be especially true among a subset of politically disengaged respondents. Combined with the complexity of U.S. politics, respondents with the least interest in politics are poised to experience an exceptionally strong demand on their combined cognitive-affective systems when asked to consider political content, thus further driving satisficing behavior in later responses.

Experimental setup

To assess whether political content can increase satisficing, I fielded a survey experiment. Participants were randomly assigned to one of three conditions. In each condition, the respondents were asked to answer a series of three closed-ended survey questions.

The treatments were designed to expose respondents to content that would differentially engage their cognitive-affective systems. In the control condition, respondents saw a series of items that had nothing to do with politics. These questions were still designed to engage respondents’ cognition. As a hard test of the premise, I sought to ensure the control battery contained relatively difficult questions.

In the first treatment (T1), respondents’ cognitive capacity was tested by three basic questions about political institutions. In the second treatment (T2), I also sought to engage respondents’ affective reactions to politics by including references to specific actors, party labels, and polarizing events. ^[1]

After answering these question batteries, respondents were next asked to respond to a series of unrelated questions (hereafter “DV Battery”). This question battery asked respondents about their perceptions of a variety of public figures (without obvious connections to politics).

Research hypotheses

Based on the pre-registration, the research hypotheses were as follows:

• H1. Compared to a control condition with no political content, respondents asked questions about political content will be more likely to exhibit satisficing behavior.
  
• H2. The effects of political content on satisficing behavior will be conditioned by respondents' levels of political interest.
  

Data

The data for the present study were collected from March 29 - April 2, 2024. The study was run on two waves of daily YouGov omnibus surveys, for a combined sample size of N = 3,600. The survey sample conformed to national demographic estimates for age, gender, race/ethnicity, and other basic demography.

Measures of satisficing derived from the DV question battery are described below, and are consistent with the pre-registration. In all cases, larger values on these measures indicates greater satisficing behavior.

1. Attention check failure: Incorrect responses to a special question designed to measure attentiveness. This question included a long preamble, but the final instructions in this text asked respondents to select “Other” from a set of responses, and input the word “attentive” into the associated text box. The question was measured on a binary scale, with 1 indicating failure and 0 indicating passage.
2. Speeding: Speeding was measured by taking the logged response time (in seconds) for respondents completing the full DV Battery. This measure was then normalized using min-max scaling to take values between [0,1], and reverse-coded, so that higher values indicated faster responses (more propensity for speeding).
3. Straightlining: The share of responses to the DV battery that had the identical response selection. This variable was normalized to take values ranging from [0,1] using min-max scaling.
4. An additive index of these measures, rescaled to [0,1] for the purposes of comparison.
   

Results

Hypothesis 1: Average Treatment Effects

The results provide support for H1, primarily for the treatment (T2) that invoked affective political content. When considering the primary outcome variable of interest, the additive index of satisficing behaviors, we see evidence that T1 (Treatment effect = 1 percentage point; p < 0.10) and T2 (Treatment effect = 1.3 percentage points; p < 0.05) showed evidence of small increases in overall satisficing behavior relative to the control condition.

This overall result is mostly attributable to the increase in attention check failure rates associated with the treatments. T2 shows a significant (p < 0.05) increase in attention check failure relative to the control condition of roughly 3.4 percentage points. The remaining estimates for the T2 measure are nonsignificant but in the expected direction. Inattentiveness, it seems, is the most direct and measurable consequence of exposure to political content.

Hypothesis 2: Treatment Effects Among the Politically Disinterested

Does this effect tend to concentrate among individuals with low levels of political interest? The results provide minimal evidence of this interactive phenomenon, and do not provide overall support for H2. The figure below presents the interactive effect estimates of political interest on the overall treatment effect.

Significant negative interactive effects would indicate that increased political interest mitigates the positive treatment effects seen in the figure above. However, the only significant negative estimates come when examining respondent speeding behavior. A unit increase in political interest is associated with a decrease in satisficing behavior of 0.7 percentage points (p < 0.05) among respondents in the T1 condition relative to the control. This relationship shows a smaller interactive effect of 0.5 percentage points for the T2 condition (p < 0.10).

General Discussion

The preceding study has demonstrated that exposure to three political questions in a survey battery, in comparison with relatively more difficult questions about other topics, has the potential to increase satisficing behavior among respondents. An additive estimate of satisficing behavior shows evidence of a roughly 1 percentage point increase in satisficing among respondents exposed to three additional questions that tap affective political considerations. Attentiveness correspondingly declines by roughly 3.4 percentage points. While we see relatively weak evidence that the least politically interested respondents are more susceptible to this phenomenon than others, the presence of an average treatment effect for the general population raises additional concerns for survey scientists.

We should be relieved that the findings presented in this study show small effect sizes. But because the question battery was brief, the results nevertheless caution researchers to consider the possibility that surveys will generate lower response quality on unrelated content when longer political question blocks are introduced early in a survey. Practitioners should take care to consider how to appropriately order question blocks and how best to deploy branches within surveys that differ in terms of the extent of political content.

Bibliography

Alvarez, R. M., Atkeson, L. R., Levin, I., & Li, Y. (2019). Paying attention to inattentive survey respondents. Political Analysis, 27(2), 145-162.

Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3), 213-236.

Krupnikov, Y., & Ryan, J. B. (2022). The other divide. Cambridge University Press.

Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of experimental social psychology, 45(4), 867-872.

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press.

About the author

Dr. Ian Anson is an associate professor of political science at the University of Maryland, Baltimore County. He holds a Ph.D. in political science and an M.S. in applied statistics from Indiana University. When not conducting research on American public opinion, media and politics, and the scholarship of teaching and learning, you can find Ian on a hike, in the pool, or playing with his lab mix, Gus. Originally from Cary, N.C., he received a B.A. at the University of North Carolina.

About Methodology matters

Methodology matters is a series of research experiments focused on survey experience and survey measurement. The series aims to contribute to the academic and professional understanding of the online survey experience and promote best practices among researchers. Academic researchers are invited to submit their own research design and hypotheses.

Footnote

^{[1] _{Readers may also wonder whether the treatments were differentially consequential for the cognitions of Republicans, Democrats, and independents. In a non-preregistered follow-up analysis, I interacted the treatment main effects with partisanship in models analogous to those shown in the figures presented below. The results show that partisanship has little impact on the average treatment effects. Of the tests presented below, only one (the effects of T2 on the attention check measure) yielded a significant positive interaction effect for Democrats relative to independents of around 0.11 points (p < 0.05). This finding shows some evidence that Democrats exhibited greater treatment check failure than other partisans in response to T2, perhaps because the “hot” political objects referenced in T2 contained references to Donald Trump and the January 6, 2021 events–two idea-elements which should cause a strong negative affective reaction among this group. The rest of the tests yielded small and nonsignificant interactive results for both Democrats and Republicans.}}