Survey experiments are ubiquitous in the behavioral social sciences. A frequent critique of this type of study is that evidence which supports a researcher's expectations occurs due to Experimenter Demand Eects (EDEs)|bias stemming from participants inferring the purpose of an experiment and responding so as to help conrm a researcher's hypothesis. In this paper, we argue that traditional survey experimental designs have several features that make them robust to EDEs. We then explicitly test for the presence of EDEs using a series of experiments which randomly assign participants to receive varying levels of information about each experiment's hypothesis. Replicating three widely used experimental designs, we nd that informing participants of an experiment's purpose has no detectable eect on observed treatment eects, or attenuates them. When informed of a study's purpose, participants do not appear to assist researchers, a nding with important implications for the design and interpretation of survey experiments.
This study develops and tests a theory explaining how government actions can cause citizens to misperceive the social world. Focusing on the case of police militarization, I use survey experiments to show that the appearance of militarized police in a community causes individuals to perceive that violent crime is more severe than in otherwise similar places where traditionally armed police are portrayed. This is problematic, because original data on SWAT team deployments gathered through hundreds of public information requests show that militarized police behavior is an unreliable predictor of violent crime rates. The experiments also show that seeing militarized police weakens support for police funding and lowers confidence in police. Government actions can bias public perceptions of social conditions, and unintentionally undermine the political goals of public agencies, findings with widespread implications for the study of bureaucracy, representation and policy development.
To study how voter identification laws affect participation in elections, Hajnal, Lajevardi and Nielson (2017) examines validated turnout data in five national surveys conducted between 2006 and 2014. The study concludes that strict ID laws cause a large turnout decline among minorities, especially Latinos. Here, we show that the results of this paper are a product of large data inaccuracies, that the evidence does not support the stated conclusion, and that model specifications produce highly variable results. When errors in the analysis are corrected, one can recover positive, negative, or null estimates of the effect of voter ID laws on turnout. Our findings underscore that no definitive relationship between strict voter ID laws and turnout can be established from the validated CCES data. Our analysis highlights more general problems with the way empirical evidence is assembled and reported, but it also offers useful suggestions on the appropriate evidence sources for research on election administration.
When assessing the substantive significance (effect size) of key independent variables, social scientists seek to craft plausible counterfactuals that avoid extrapolating outside the support of the observed data. When causal inference strategies—which confine analyses to comparable sets of treated and untreated units—are employed, the risk of positing such unrealistic scenarios can worsen. The reason is that such strategies discard massive amounts of variation in the data, either before or during the estimation of causal effects. In this paper, we show that failing to account for such reductions in variation after employing one of the most ubiquitous causal inference strategies, fixed effects regression, can and has led researchers to posit unrealistic counterfactuals when discussing substantive significance. We provide a set of best practices for avoiding this interpretive pitfall.
Multiplicative interaction models are widely used in social science to test whether the relationship between an outcome and an independent variable changes with a moderating variable. Current empirical practice overlooks two important problems. First, these models assume a linear interaction effect that changes at a constant rate with the moderator. Second, reliably estimating the conditional effects of the independent variable at all values of the moderator requires sufficient common support. Replicating 46 interaction effects from 22 recent publications in five top political science journals, we find that these core assumptions fail in a majority of cases, suggesting that a large portion of findings across all subfields based on interaction models are modeling artifacts or are at best highly model dependent. We propose simple diagnostics to assess the validity of these assumptions and offer flexible estimation strategies that allow for nonlinear interaction effects and safeguard against excessive extrapolation.
High-profile incidents of police misconduct have led to widespread calls for law enforcement reform. But prior studies cast doubt on whether police commanders can control officers, and offer few policy remedies because of their focus on potentially immutable officer traits like personality. I advance an alternative, institutional perspective, and demonstrate that police officers—sometimes characterized as autonomous—are highly responsive to managerial directives. Using millions of records of police-citizen interactions alongside officer interviews, I evaluate the impact of a change to the protocol for stopping criminal suspects on police performance. An interrupted time series analysis shows the directive produced an immediate increase in the rate of stops producing evidence of the suspected crime. Interviewed officers said the order signaled increased managerial scrutiny, leading them to adopt more conservative tactics. Procedural changes can quickly and dramatically alter officer behavior, suggesting a reform strategy sometimes forestalled by psychological and personality-driven accounts of police reform.
Issue frames are a central concept in studying public opinion, and are thought to operate by foregrounding related considerations in citizens’ minds. But scholarship has yet to consider the breadth of framing effects by testing whether frames influence attitudes beyond the specific issue they highlight. For example, does a discussion of terrorism affect opinions on proximate issues like crime or even more remote issues like poverty? By measuring the breadth of framing effects, we can assess the extent to which citizens’ political considerations are cognitively organized by issues. We undertake a population-based survey experiment with roughly 3,300 respondents which includes frames related to terrorism, crime, health care, and government spending. The results demonstrate that framing effects are narrow, with limited but discernible spillover on proximate, structurally similar issues. Discrete issues not only organize elite politics but also exist in voters’ minds, a finding with implications for studying ideology as well as framing.
Social divisions between American partisans are growing, with Republicans and Democrats exhibiting homophily in a range of seemingly nonpolitical domains. It has been widely claimed that this partisan social divide extends to Americans’ decisions about where to live. In two original survey experiments, we confirm that Democrats are, in fact, more likely than Republicans to prefer living in more Democratic, dense, and racially diverse places. However, improving on previous studies, we test respondents’ stated preferences against their actual moving behavior. While partisans differ in their residential preferences, on average they are not migrating to more politically distinct communities. Using zip-code-level census and partisanship data on the places where respondents live, we provide one explanation for this contradiction: by prioritizing common concerns when deciding where to live, Americans forgo the opportunity to move to more politically compatible communities.
Voters are often uninformed about the political candidates they choose between. Governments, media outlets, and civic organizations devote substantial resources to correcting these knowledge deficits by creating tools to provide candidate information to voters. Despite the widespread production of these aids, it remains unclear who they reach. We collect validated measures of online voter guide use for more than 40,000 newspaper readers during a state primary election. We show this newspaper-produced voter guide was primarily used by individuals with high levels of political interest and knowledge, a finding in contrast to earlier hypotheses that providing guides directly to voters online would reduce disparities in use based on political interest. A field experiment promoting the voter guide failed to diminish these consumption gaps. These results show that the same content preferences that contribute to an unequal distribution of political knowledge also impede the effectiveness of subsequent efforts to close knowledge gaps.
Prior research has demonstrated a preference among partisans for like-minded news outlets, a key mechanism through which the media may be polarizing Americans. But in order for source reputations to cause widespread selective exposure, individuals must prioritize them above other competing attributes of news content. Evaluating the relative influence of various contributors to media choice is therefore critical. This study pits two such factors, source reputation and topic relevance, against one another in conjoint survey experiments offering randomly paired news items to partisans. Making a news source’s reputation politically unfriendly lowers the probability that an individual chooses an item, but this negative effect is often eclipsed by the positive effect of making a news topic relevant to the individual. In many popular modern news consumption environments, where consumers encounter a diverse mixture of sources and topics, the ability of source reputations to contribute to polarization via partisan selective exposure is limited.
Immigrants’ perceptions of discrimination (PD) correlate strongly with various political outcomes, including group consciousness and partisan identity. Here, we examine the hypothesis that immigrants’ PD vary across US localities, as threatened responses by native-born residents may increase perceived discrimination among neighboring immigrants. We also consider the alternative hypothesis that barriers to the expression and detection of discrimination decouple native-born attitudes from immigrants’ perceptions about their treatment. We test these claims by analyzing three national surveys of almost 11,000 first-generation Latino, Asian, and Muslim immigrants. The results indicate that immigrants’ PD hardly vary across localities. While anti-immigrant attitudes are known to be geographically clustered, immigrants’ PD prove not to be. This mismatch helps us narrow the potential causes of perceived discrimination, and it suggests the value of further research into perceived discrimination’s consequences for immigrants’ social and political incorporation.
Understanding how the Tea Party has affected congressional elections and roll call voting helps us understand not only an important political movement, but how movements affect politics more generally. We investigate four channels for the movement to influence political outcomes: activists, constituent opinion, group endorsement activity and elite-level self-identification. We find consistent evidence that activists mattered both electorally and for roll call voting on issues of importance to the movement. Constituent opinion had virtually no impact on either political outcome. Group endorsement activity had possible effects on elections, but mostly no effect on congressional voting. Self-identification among elites did not enhance—or harm—Republican electoral fortunes, but did affect congressional votes important to the movement. These divergent results illustrate how movement politics can influence outcomes through multiple channels and call into question the usefulness of the “Tea Party’’ moniker without important qualifiers.