Survey experiments are ubiquitous in social science. A frequent critique is that positive results in these studies stem from Experimenter Demand Effects (EDEs)---bias that occurs when participants infer the purpose of an experiment and respond so as to help confirm a researcher's hypothesis. We argue that survey experiments have several features that make them robust to EDEs, and test for their presence in studies that involve over 12,000 participants and replicate five experimental designs touching on all empirical political science subfields. We randomly assign participants information about experimenter intent and show that providing this information does not alter the treatment effects in these experiments. Even financial incentives to respond in line with researcher expectations fail to consistently induce demand effects. Research participants exhibit a limited ability to adjust their behavior to align with researcher expectations, a finding with important implications for the design and interpretation of survey experiments.
Multiplicative interaction models are widely used in social science to examine whether the relationship between an outcome and an independent variable changes with a moderating variable. Current empirical practice tends to over- look two important problems. First, these models assume a linear interaction effect that changes at a constant rate with the moderator. Second, estimates of the conditional effects of the independent variable can be misleading if there is a lack of common support of the moderator. Replicating 46 interaction effects from 22 recent publications in five top political science journals, we find that these core assumptions often fail in practice, suggesting that a large portion of findings across all political science subfields based on interaction models are modeling artifacts or are at best highly model dependent. We propose a checklist of simple diagnostics to assess the validity of these assumptions and offer flexible estimation strategies that allow for nonlinear interaction effects and safeguard against excessive extrapolation. These statistical routines are available in both R and STATA.
Fixed effects estimators are frequently used to combat selection bias by eliminating large swaths of variation in observational data. For example, it is well-known that unit fixed effects in panel data discard all between-unit variation, resulting in estimates of an independent variable's effect as it changes within units over time. In this article, we replicate several recent studies which used fixed effects estimators to show how descriptions of the substantive significance of results can be improved by precisely characterizing the variation being studied and presenting plausible counterfactuals---i.e., shifts in the independent variable likely to occur within the variation being used for estimation---when assessing the substantive impact of the treatment. We provide a checklist for the interpretation of fixed effects regression results to help avoid these interpretative pitfalls.
High-profile incidents of police misconduct have led to widespread calls for law enforcement reform. But prior studies cast doubt on whether police commanders can control officers, and offer few policy remedies because of their focus on potentially immutable officer traits like personality. I advance an alternative, institutional perspective, and demonstrate that police officers—sometimes characterized as autonomous—are highly responsive to managerial directives. Using millions of records of police-citizen interactions alongside officer interviews, I evaluate the impact of a change to the protocol for stopping criminal suspects on police performance. An interrupted time series analysis shows the directive produced an immediate increase in the rate of stops producing evidence of the suspected crime. Interviewed officers said the order signaled increased managerial scrutiny, leading them to adopt more conservative tactics. Procedural changes can quickly and dramatically alter officer behavior, suggesting a reform strategy sometimes forestalled by psychological and personality-driven accounts of police reform.
Widespread concern that voter identification laws suppress turnout among racial and ethnic minorities has made empirical evaluations of these laws crucial. But problems with administrative records and survey data impede such evaluations. We replicate and extend Hajnal, Lajevardi and Nielson (2017), which reports that voter ID laws decrease turnout among minorities, using validated turnout data from five national surveys conducted between 2006 and 2014. We show that the results of the paper are a product of data inaccuracies; the presented evidence does not support the stated conclusion; and alternative model specifications produce highly variable results. When errors are corrected, one can recover positive, negative, or null estimates of the effect of voter ID laws on turnout, precluding firm conclusions. We highlight more general problems with available data for research on election administration and we identify more appropriate data sources for research on state voting laws' effects.
Issue frames are a central concept in studying public opinion, and are thought to operate by foregrounding related considerations in citizens’ minds. But scholarship has yet to consider the breadth of framing effects by testing whether frames influence attitudes beyond the specific issue they highlight. For example, does a discussion of terrorism affect opinions on proximate issues like crime or even more remote issues like poverty? By measuring the breadth of framing effects, we can assess the extent to which citizens’ political considerations are cognitively organized by issues. We undertake a population-based survey experiment with roughly 3,300 respondents which includes frames related to terrorism, crime, health care, and government spending. The results demonstrate that framing effects are narrow, with limited but discernible spillover on proximate, structurally similar issues. Discrete issues not only organize elite politics but also exist in voters’ minds, a finding with implications for studying ideology as well as framing.
Voters are often uninformed about the political candidates they choose between. Governments, media outlets, and civic organizations devote substantial resources to correcting these knowledge deficits by creating tools to provide candidate information to voters. Despite the widespread production of these aids, it remains unclear who they reach. We collect validated measures of online voter guide use for more than 40,000 newspaper readers during a state primary election. We show this newspaper-produced voter guide was primarily used by individuals with high levels of political interest and knowledge, a finding in contrast to earlier hypotheses that providing guides directly to voters online would reduce disparities in use based on political interest. A field experiment promoting the voter guide failed to diminish these consumption gaps. These results show that the same content preferences that contribute to an unequal distribution of political knowledge also impede the effectiveness of subsequent efforts to close knowledge gaps.
Prior research has demonstrated a preference among partisans for like-minded news outlets, a key mechanism through which the media may be polarizing Americans. But in order for source reputations to cause widespread selective exposure, individuals must prioritize them above other competing attributes of news content. Evaluating the relative influence of various contributors to media choice is therefore critical. This study pits two such factors, source reputation and topic relevance, against one another in conjoint survey experiments offering randomly paired news items to partisans. Making a news source’s reputation politically unfriendly lowers the probability that an individual chooses an item, but this negative effect is often eclipsed by the positive effect of making a news topic relevant to the individual. In many popular modern news consumption environments, where consumers encounter a diverse mixture of sources and topics, the ability of source reputations to contribute to polarization via partisan selective exposure is limited.
Immigrants’ perceptions of discrimination (PD) correlate strongly with various political outcomes, including group consciousness and partisan identity. Here, we examine the hypothesis that immigrants’ PD vary across US localities, as threatened responses by native-born residents may increase perceived discrimination among neighboring immigrants. We also consider the alternative hypothesis that barriers to the expression and detection of discrimination decouple native-born attitudes from immigrants’ perceptions about their treatment. We test these claims by analyzing three national surveys of almost 11,000 first-generation Latino, Asian, and Muslim immigrants. The results indicate that immigrants’ PD hardly vary across localities. While anti-immigrant attitudes are known to be geographically clustered, immigrants’ PD prove not to be. This mismatch helps us narrow the potential causes of perceived discrimination, and it suggests the value of further research into perceived discrimination’s consequences for immigrants’ social and political incorporation.
Social divisions between American partisans are growing, with Republicans and Democrats exhibiting homophily in a range of seemingly nonpolitical domains. It has been widely claimed that this partisan social divide extends to Americans’ decisions about where to live. In two original survey experiments, we confirm that Democrats are, in fact, more likely than Republicans to prefer living in more Democratic, dense, and racially diverse places. However, improving on previous studies, we test respondents’ stated preferences against their actual moving behavior. While partisans differ in their residential preferences, on average they are not migrating to more politically distinct communities. Using zip-code-level census and partisanship data on the places where respondents live, we provide one explanation for this contradiction: by prioritizing common concerns when deciding where to live, Americans forgo the opportunity to move to more politically compatible communities.
Understanding how the Tea Party has affected congressional elections and roll call voting helps us understand not only an important political movement, but how movements affect politics more generally. We investigate four channels for the movement to influence political outcomes: activists, constituent opinion, group endorsement activity and elite-level self-identification. We find consistent evidence that activists mattered both electorally and for roll call voting on issues of importance to the movement. Constituent opinion had virtually no impact on either political outcome. Group endorsement activity had possible effects on elections, but mostly no effect on congressional voting. Self-identification among elites did not enhance—or harm—Republican electoral fortunes, but did affect congressional votes important to the movement. These divergent results illustrate how movement politics can influence outcomes through multiple channels and call into question the usefulness of the “Tea Party’’ moniker without important qualifiers.