Information Equivalence in Survey Experiments

Allan Dafoe, Baobao Zhang, Devin Caughey

Research output: Contribution to journalArticlepeer-review

63 Scopus citations

Abstract

Survey experiments often manipulate the description of attributes in a hypothetical scenario, with the goal of learning about those attributes' real-world effects. Such inferences rely on an underappreciated assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE is often violated because subjects, when presented with information about one attribute, update their beliefs about others too. Labeling a country a democracy, for example, affects subjects' beliefs about the country's geographic location. When IE is violated, the effect of the manipulation need not correspond to the quantity of interest (the effect of beliefs about the focal attribute). We formally define the IE assumption, relating it to the exclusion restriction in instrumental-variable analysis. We show how to predict IE violations ex ante and diagnose them ex post with placebo tests. We evaluate three strategies for achieving IE. Abstract encouragement is ineffective. Specifying background details reduces imbalance on the specified details and highly correlated details, but not others. Embedding a natural experiment in the scenario can reduce imbalance on all background beliefs, but raises other issues. We illustrate with four survey experiments, focusing on an extension of a prominent study of the democratic peace.

Original languageEnglish (US)
Pages (from-to)399-416
Number of pages18
JournalPolitical Analysis
Volume26
Issue number4
DOIs
StatePublished - Oct 1 2018
Externally publishedYes

Keywords

  • causal inference
  • natural experiments
  • survey design
  • survey experiments

ASJC Scopus subject areas

  • Sociology and Political Science
  • Political Science and International Relations

Fingerprint

Dive into the research topics of 'Information Equivalence in Survey Experiments'. Together they form a unique fingerprint.

Cite this