Working Paper
Egami N, Fong CJ, Grimmer J, Roberts ME, Stewart BM. How to Make Causal Inferences Using Texts. Working Paper. ais.pdf
Horowitz M, Stewart B, Tingley D, Bishop M, Resnick L, Roberts M, Chang W, Mellers B, Tetlock P. What Makes Foreign Policy Teams Tick: Explaining Variation in Group Performance At Geopolitical Forecasting. Working Paper.

Copy available here

Lloyd P, Simmons BA, Stewart BM. The Global Diffusion of Law: Transnational Crime and the Case of Human Trafficking. Working Paper.
Stewart BM. Latent Factor Regressions for the Social Sciences. Working Paper.Abstract

In this paper I present a general framework for regression in the presence of complex dependence structures between units such as in time-series cross-sectional data, relational/network data, and spatial data. These types of data are challenging for standard multilevel models because they involve multiple types of structure (e.g. temporal effects and cross-sectional effects) which are interactive. I show that interactive latent factor models provide a powerful modeling alternative that can address a wide range of data types. Although related models have previously been proposed in several different fields, inference is typically cumbersome and slow. I introduce a class of fast variational inference algorithms that allow for models to be fit quickly and accurately.

tensorreg.pdf tensorregappendix.pdf
Roberts ME, Stewart BM, Nielsen R. Matching Methods for High-Dimensional Data with Applications to Text. Working Paper.Abstract

Matching is a popular technique for preprocessing observational data to facilitate causal inference and reduce model dependence by ensuring that treated and control units are balanced along pre-treatment covariates. While most applications of matching balance on a small number of covariates, we identify situations where matching with thousands of covariates may be desirable, such as causal inference where confounders are measured with text. With high-dimensional covariates, traditional matching methods are less effective and may be difficult or impossible to implement. We characterize the problem of matching in a high-dimensional context as a tradeoff between dimension reduction and imbalance bounding. We develop a new method called Topical Inverse Regression Matching (TIRM) that optimizes this tradeoff by including both a low-dimensional projection of covariates and information about the probability of treatment. We illustrate our approach by estimating the effect of censorship on the writing of Chinese bloggers, the effects of gender on citation counts in international relations, and the effects of targeted killings and capture by counterterrorists on the popularity of jihadist writings.

Copy here.

Discourse: MOOC Discussion Forum Analysis at Scale
Kindel A, Yeomans M, Reich J, Stewart B, Tingley D. Discourse: MOOC Discussion Forum Analysis at Scale, in Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale. New York, NY, USA: ACM ; 2017 :141–142. Publisher's Version p141-kindel.pdf
The Civic Mission of MOOCs: Measuring Engagement across Political Differences in Forums
Reich J, Stewart B, Mavon K, Tingley D. The Civic Mission of MOOCs: Measuring Engagement across Political Differences in Forums. Proceedings of the Third (2016) ACM Conference on Learning @ Scale [Internet]. 2016 :1-10. Publisher's VersionAbstract

In this study, we develop methods for computationally measuring the degree to which students engage in MOOC forums with other students holding different political beliefs. We examine a case study of a single MOOC about education policy, Saving Schools, where we obtain measures of student education policy preferences that correlate with political ideology. Contrary to assertions that online spaces often become echo chambers or ideological silos, we find that students in this case hold diverse political beliefs, participate equitably in forum discussions, directly engage (through replies and upvotes) with students holding opposing beliefs, and converge on a shared language rather than talking past one another. Research that focuses on the civic mission of MOOCs helps ensure that open online learning engages the same breadth of purposes that higher education aspires to serve.

A model of text for experimentation in the social sciences
Roberts ME, Stewart BM, Airoldi EM. A model of text for experimentation in the social sciences. Journal of the American Statistical Association [Internet]. 2016;111 (515) :988-1003. Publisher's VersionAbstract

Statistical models of text have become increasingly popular in statistics and computer science as a method of exploring large document collections. Social scientists often want to move beyond exploration, to measurement and experimentation, and make inference about social and political processes that drive discourse and content. In this paper, we develop a model of text data that supports this type of substantive research.
Our approach is to posit a hierarchical mixed membership model for analyzing topical content of documents, in which mixing weights are parameterized by observed covariates. In this model, topical prevalence and topical content are specified as a simple generalized linear model on an arbitrary number of document-level covariates, such as news source and time of release, enabling researchers to introduce elements of the experimental design that informed document collection into the model, within a generally applicable framework. We demonstrate the proposed methodology by analyzing a collection of news reports about China, where we allow the prevalence of topics to evolve over time and vary across newswire services. Our methods quantify the effect of news wire source on both the frequency and nature of topic coverage.


NB: This is a revised version of the working paper previously titled "Structural Topic Models." 

Navigating the Local Modes of Big Data: The Case of Topic Models
Roberts ME, Stewart BM, Tingley D. Navigating the Local Modes of Big Data: The Case of Topic Models. In: Computational Social Science: Discovery and Prediction. New York: Cambridge University Press ; 2016. Publisher's Version

Copy available here

Chuang J, Roberts M, Stewart B, Weiss R, Tingley D, Grimmer J, Heer J. TopicCheck: Interactive Alignment for Assessing Topic Model Stability. North American Chapter of the Association for Computational Linguistics Human Language Technologies (NAACL HLT). 2015.Abstract

Content analysis, a widely-applied social science research method, is increasingly being supplemented by topic modeling. However, while the discourse on content analysis centers heavily on reproducibility, computer scientists often focus more on scalability and less on coding reliability, leading to growing skepticism on the usefulness of topic models for automated content analysis. In response, we introduce TopicCheck, an interactive tool for assessing topic model stability. Our contributions are threefold. First, from established guidelines on reproducible content analysis, we distill a set of design requirements on how to computationally assess the stability of an automated coding process. Second, we devise an interactive alignment algorithm for matching latent topics from multiple models, and enable sensitivity evaluation across a large number of models. Finally, we demonstrate that our tool enables social scientists to gain novel insights into three active research questions.

Computer Assisted Reading and Discovery for Student Generated Text in Massive Open Online Courses
Reich J, Tingley D, Leder-Luis J, Roberts ME, Stewart BM. Computer Assisted Reading and Discovery for Student Generated Text in Massive Open Online Courses. Journal of Learning Analytics. 2015;2 (1) :156-184.Abstract

Dealing with the vast quantities of text that students generate in a Massive Open Online Course (MOOC) is a daunting challenge. Computational tools are needed to help instructional teams uncover themes and patterns as MOOC students write in forums, assignments, and surveys. This paper introduces to the learning analytics community the Structural Topic Model, an approach to language processing that can (1) find syntactic patterns with semantic meaning in unstructured text, (2) identify variation in those patterns across covariates, and (3) uncover archetypal texts that exemplify the documents within a topical pattern. We show examples of computationally- aided discovery and reading in three MOOC settings: mapping students’ self-reported motivations, identifying themes in discussion forums, and uncovering patterns of feedback in course evaluations. 

Computer assisted text analysis for comparative politics.
Lucas C, Nielsen R, Roberts ME, Stewart BM, Storer A, Tingley D. Computer assisted text analysis for comparative politics. Political Analysis. 2015;23 (2) :254-277.Abstract

Recent advances in research tools for the systematic analysis oftextual data are enabling exciting new research throughout the socialsciences. For comparative politics scholars who are often interestedin non-English and possibly multilingual textual datasets, theseadvances may be difficult to access. This paper discusses practicalissues that arise in the the processing, management, translation andanalysis of textual data with a particular focus on how proceduresdiffer across languages. These procedures are combined in two appliedexamples of automated text analysis using the recently introducedStructural Topic Model. We also show how the model can be used toanalyze data that has been translated into a single language viamachine translation tools. All the methods we describe here are implemented in open-source software packages available from the authors.

comparativepoliticstext.pdf compoltextappendix.pdf

Included in Political Analysis virtual issue on Online Research Methods. Software: stm, txtorgtranslateR

Romney D, Stewart BM, Tingley D. Plain Text: Transparency in the Acquisition, Analysis, and Access Stages of the Computer-assisted Analysis of Texts. Qualitative and Multi-Method Research. 2015;13 (1) :32-37. qmmr2015-1.pdf
Chuang J, Wilkerson JD, Weiss R, Tingley D, Stewart BM, Roberts ME, Poursabzi-Sangdeh F, Grimmer J, Findlater L, Boyd-Graber J, et al. Computer-Assisted Content Analysis: Topic Models for Exploring Multiple Subjective Interpretations. Advances in Neural Information Processing Systems Workshop on Human-Propelled Machine Learning. 2014.Abstract

Content analysis, a labor-intensive but widely-applied research method, is increasingly being supplemented by computational techniques such as statistical topic modeling. However, while the discourse on content analysis centers heavily on reproducibility, computer scientists often focus more on increasing the scale of analysis and less on establishing the reliability of analysis results. The gap between user needs and available tools leads to justified skepticism, and limits the adoption and effective use of computational approaches. We argue that enabling human-in-the-loop machine learning requires establishing users’ trust in computer-assisted analysis. To this aim, we introduce our ongoing work on analysis tools for interac- tively exploring the space of available topic models. To aid tool development, we propose two studies to examine how a computer-aided workflow affects the uncovered codes, and how machine-generated codes impact analysis outcome. We present our prototypes and findings currently under submission. 

Coppola A, Stewart BM. lbfgs: Efficient L-BFGS and OWL-QN Optimization in R. 2014.Abstract

This vignette introduces the lbfgs package for R, which consists of a wrapper built around the libLBFGS optimization library written by Naoaki Okazaki. The lbfgs package implements both the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) and the Orthant-Wise Limited-memory Quasi-Newton (OWL-QN) optimization algorithms. The L-BFGS algorithm solves the problem of minimizing an objective, given its gradient, by iteratively computing approximations of the inverse Hessian matrix. The OWL-QN algorithm finds the optimum of an objective plus the L1 norm of the problem’s parameters. The package offers a fast and memory-efficient implementation of these optimization routines, which is particularly suited for high-dimensional problems. The lbfgs package compares favorably with other optimization packages for R in microbenchmark tests.

Structural topic models for open-ended survey responses
Roberts ME, Stewart BM, Tingley D, Lucas C, Leder-Luis J, Gadarian S, Albertson B, Rand D. Structural topic models for open-ended survey responses. American Journal of Political Science. 2014;58 :1064-1082.Abstract

Collection and especially analysis of open-ended survey responses are relatively rare in the discipline and when conducted are almost exclusively done through human coding. We present an alternative, semi-automated approach, the structural topic model (STM) (Roberts, Stewart, and Airoldi 2013; Roberts et al. 2013), that draws on recent developments in machine learning based analysis of textual data. A crucial contribution of the method is that it incorporates information about the document, such as the author’s gender, political affiliation, and treatment assignment (if an experimental study). This article focuses on how the STM is helpful for survey researchers and experimentalists. The STM makes analyzing open-ended responses easier, more revealing, and capable of being used to estimate treatment effects. We illustrate these innovations with analysis of text from surveys and experiments.

topicmodelsopenendedexperiments_0.pdf ajpsappendix.pdf

Awarded the Gosnell Prize for Excellence in Political Methodology for the best work in political methodology presented at any political science conference during the preceding year.  Data at:

Choosing Your Neighbors: Networks of Diffusion in International Relations
Zhukov YM, Stewart BM. Choosing Your Neighbors: Networks of Diffusion in International Relations. International Studies Quarterly. 2013;57 :271-287.Abstract

In examining the diffusion of social and political phenomena like regime transition, conflict, and policy change, scholars routinely make choices about how proximity is defined and which neighbors should be considered more important than others. Since each specification offers an alternative view of the networks through which diffusion can take place, one’s decision can exert a significant influence on the magnitude and scope of estimated diffusion effects. This problem is widely recognized, but is rarely the subject of direct analysis. In international relations research, connectivity choices are usually ad hoc, driven more by data availability than by theoretically informed decision criteria. We take a closer look at the assumptions behind these choices, and propose a more systematic method to asses the structural similarity of two or more alternative networks, and select one that most plausibly relates theory to empirics. We apply this method to the spread of democratic regime change, and offer an illustrative example of how neighbor choices might impact predictions and inferences in the case of the 2011 Arab Spring.


Replication Data: here.  Spatial Weight Data here.

Learning to Extract International Relations from Political Context
O’Connor B, Stewart BM, Smith NA. Learning to Extract International Relations from Political Context. Association of Computational Linguistics. 2013. oconnorstewartsmith.irevents.acl2013.pdf supp.pdf
Psychological and Physiological Responses following Repeated Peer Death
Andersen JP, Silver RC, Stewart BM, Koperwas B, Kirschbaum C. Psychological and Physiological Responses following Repeated Peer Death. PLOS One. 2013;8 :1-9. andersenetal2013.pdf onlineappendix.pdf
The Structural Topic Model and Applied Social Science
Roberts ME, Stewart BM, Tingley D, Airoldi EM. The Structural Topic Model and Applied Social Science. Advances in Neural Information Processing Systems Workshop on Topic Models: Computation, Application, and Evaluation. 2013. stmnips2013.pdf

Peer-Reviewed Conference Workshop. Selected for Oral Presentation.