WIXLIB

Boxes4.15.15.26.16.2The intention-to-treat principleThe Neyman model and the average causal effectEstimating the average causal effectStandard errors under the Neyman modelCode for analysis by cluster meanspage 88111115170181

Preface and acknowledgementsNatural experiments have become ubiquitous in the social sciences. Fromstandard natural experiments to regression-discontinuity and instrumentalvariables designs, our leading research articles and books more and morefrequently reference this label. For professional researchers and students alike,natural experiments are often recommended as a tool for strengthening causalclaims.Surprisingly, we lack a comprehensive guide to this type of research design.Finding a useful and viable natural experiment is as much an art as a science.Thus, an extensive survey of examples—grouped and discussed to highlighthow and why they provide the leverage they do—may help scholars to usenatural experiments effectively in their substantive research. Just as importantly, awareness of the obstacles to successful natural experiments may helpscholars maximize their promise while avoiding their pitfalls. There aresignificant challenges involved in the analysis and interpretation of naturalexperimental data. Moreover, the growing popularity of natural experimentscan lead to conceptual stretching, as the label is applied to studies that do notvery credibly bear the hallmarks of this research design. Discussion of both thestrengths and limitations of natural experiments may help readers to evaluateand bolster the success of specific applications. I therefore hope that this bookwill provide a resource for scholars and students who want to conduct orcritically consume work of this type.While the book is focused on natural experiments, it is also a primer ondesign-based research in the social sciences more generally. Research thatdepends on ex post statistical adjustment (such as cross-country regressions) has recently come under fire; there has been a commensurate shift offocus toward design-based research—in which control over confoundingvariables comes primarily from research design, rather than model-basedstatistical adjustment. The current enthusiasm for natural experimentsreflects this renewed emphasis on design-based research. Yet, how shouldsuch research be conducted and evaluated? What are the key assumptions

xviiiPreface and acknowledgementsbehind design-based inference, and what causal and statistical models areappropriate for this style of research? And can such design-basedapproaches help us make progress on big, important substantive topics,such as the causes and consequences of democracy or socioeconomic development? Answering such questions is critical for sustaining the credibilityand relevance of design-based research.Finally, this book also highlights the potential payoffs from integratingqualitative and quantitative methods. “Bridging the divide” betweenapproaches is a recurring theme in many social sciences. Yet, strategies forcombining multiple methods are not always carefully explicated; and the valueof such combinations is sometimes presumed rather than demonstrated. Thisis unfortunate: at least with natural experiments, different methods do not justsupplement but often require one another. I hope that this book can clarify thepayoffs of mixing methods and especially of the “shoe-leather” research that,together with strong designs, makes compelling causal inference possible.This book grows out of discussions with many colleagues, students, andespecially mentors. I am deeply fortunate to have met David Freedman, towhom the book is dedicated, while finishing my Ph.D. studies at theUniversity of California at Berkeley. His impact on this book will be obviousto readers who know his work; I only wish that he were alive to read it. Whilehe is greatly missed, he left behind an important body of research, with whichevery social scientist who seeks to make causal inferences should grapple.I would also like to thank several other mentors, colleagues, and friends.David Collier’s exemplary commitment to the merger of qualitative andquantitative work has helped me greatly along the way; this book grew outof a chapter I wrote for the second edition of his book, Rethinking SocialInquiry, co-edited with Henry Brady. Jim Robinson, himself a prominentadvocate of natural-experimental research designs, continues to influencemy own substantive and methodological research. I would especially like tothank Don Green and Dan Posner, both great friends and colleagues, whoread and offered detailed and incisive comments on large portions of themanuscript. Colin Elman organized a research workshop at the Institute forQualitative and Multi-Method Research at Syracuse University, where JohnGerring and David Waldner served as very discerning discussants, whileMacartan Humphreys and Alan Jacobs convoked a book event at theUniversity of British Columbia, at which Anjali Bohlken, Chris Kam, andBen Nyblade each perceptively dissected individual chapters. I am grateful toall of the participants in these two events. For helpful conversations andsuggestions, I also thank Jennifer Bussell, Colin Elman, Danny Hidalgo,

xixPreface and acknowledgementsMacartan Humphreys, Jim Mahoney, Ken Scheve, Jay Seawright, Jas Sekhon,Rocío Titiunik, and David Waldner. I have been privileged to teach coursesand workshops on related material to graduate students at the Institute forQualitative and Multi-Method Research and at Yale, where Natalia Bueno,Germán Feierherd, Nikhar Gaikwad, Malte Lierl, Pia Raffler, SteveRosenzweig, Luis Schiumerini, Dawn Teele, and Guadalupe Tuñón, amongothers, offered insightful reactions. I have also enjoyed leading an annual shortcourse on multi-method research at the American Political ScienceAssociation with David Collier and Jay Seawright. These venues have provided a valuable opportunity to improve my thinking on the topics discussedin this book, and I thank all the participants in those workshops and coursesfor their feedback.I would also like to thank my editor on this book, John Haslam ofCambridge University Press, as well as Carrie Parkinson, Ed Robinson, andJim Thomas, for their gracious shepherding of this book to completion. I amparticularly grateful to Colin Elman, John Gerring, and Jim Mahoney, whoapproached me about writing this book for their Strategies of Social Inquiryseries. For their steadfast love and support, my deepest gratitude goes to myfamily.

1Introduction: why natural experiments?If I had any desire to lead a life of indolent ease, I would wish to be an identical twin,separated at birth from my brother and raised in a different social class. We could hireourselves out to a host of social scientists and practically name our fee. For we wouldbe exceedingly rare representatives of the only really adequate natural experiment forseparating genetic from environmental effects in humans—genetically identical individuals raised in disparate environments.—Stephen Jay Gould (1996: 264)Natural experiments are suddenly everywhere. Over the last decade, thenumber of published social-scientific studies that claim to use this methodology has more than tripled (Dunning 2008a). More than 100 articles publishedin major political-science and economics journals from 2000 to 2009 contained the phrase “natural experiment” in the title or abstract—compared toonly 8 in the three decades from 1960 to 1989 and 37 between 1990 and 1999(Figure 1.1).1 Searches for “natural experiment” using Internet search enginesnow routinely turn up several million hits.2 As the examples surveyed in thisbook will suggest, an impressive volume of unpublished, forthcoming, andrecently published studies—many not yet picked up by standard electronicsources—also underscores the growing prevalence of natural experiments.This style of research has also spread across various social science disciplines. Anthropologists, geographers, and historians have used natural experiments to study topics ranging from the effects of the African slave trade to thelong-run consequences of colonialism. Political scientists have explored thecauses and consequences of suffrage expansion, the political effects of militaryconscription, and the returns to campaign donations. Economists, the mostprolific users of natural experiments to date, have scrutinized the workings of121Such searches do not pick up the most recent articles, due to the moving wall used by the online archive,JSTOR.See, for instance, Google Scholar: http://scholar.google.com.

2Introduction: why natural experiments?90Number of published al scienceFigure 1.12000–09EconomicsNatural experiments in political science and economicsArticles published in major political science and economics journals with “natural experiment” inthe title or abstract (as tracked in the online archive JSTOR).labor markets, the consequences of schooling reforms, and the impact ofinstitutions on economic development.3The ubiquity of this method reflects its potential to improve the quality ofcausal inferences in the social sciences. Researchers often ask questions aboutcause and effect. Yet, those questions are challenging to answer in the observational world—the one that scholars find occurring around them.Confounding variables associated both with possible causes and with possibleeffects pose major obstacles. Randomized controlled experiments offer onepossible solution, because randomization limits confounding. However, manycauses of interest to social scientists are difficult to manipulate experimentally.Thus stems the potential importance of natural experiments—in whichsocial and political processes, or clever research-design innovations, create3According to Rozenzweig and Wolpin (2000: 828), “72 studies using the phrase ‘natural experiment’ inthe title or abstract issued or published since 1968 are listed in the Journal of Economic Literaturecumulative index.” A more recent edited volume by Diamond and Robinson (2010) includescontributions from anthropology, economics, geography, history, and political science, though severalof the comparative case studies in the volume do not meet the definition of natural experiments advancedin this book. See also Angrist and Krueger (2001), Dunning (2008a, 2010a), Robinson, McNulty, andKrasno (2009), Sekhon (2009), and Sekhon and Titiunik (2012) for surveys and discussion of recent work.

3Introduction: why natural experiments?situations that approximate true experiments. Here, we find observationalsettings in which causes are randomly, or as good as randomly, assignedamong some set of units, such as individuals, towns, districts, or even countries. Simple comparisons across units exposed to the presence or absence of acause can then provide credible evidence for causal effects, because random oras-if random assignment obviates confounding. Natural experiments can helpovercome the substantial obstacles to drawing causal inferences from observational data, which is one reason why researchers from such varied disciplines increasingly use them to explore causal relationships.Yet, the growth of natural experiments in the social sciences has not beenwithout controversy. Natural experiments can have important limitations,and their use entails specific analytic challenges. Because they are not so muchplanned as discovered, using natural experiments to advance a particularresearch agenda involves an element of luck, as well as an awareness of howthey have been used successfully in disparate settings. For natural experimentsthat lack true randomization, validating the definitional claim of as-if randomassignment is very far from straightforward. Indeed, the status of particularstudies as “natural experiments” is sometimes in doubt: the very popularity ofthis form of research may provoke conceptual stretching, in which an attractive label is applied to research designs that only implausibly meet the definitional features of the method (Dunning 2008a). Social scientists have alsodebated the analytic techniques appropriate to this method: for instance, whatrole should multivariate regression analysis play in analyzing the data fromnatural experiments? Finally, the causes that Nature deigns to assign atrandom may not always be the most important causal variables for socialscientists. For some observers, the proliferation of natural experiments therefore implies the narrowing of research agendas to focus on substantivelyuninteresting or theoretically irrelevant topics (Deaton 2009; Heckman andUrzúa 2010). Despite the enthusiasm evidenced by their increasing use, theability of natural experiments to contribute to the accumulation of substantively important knowledge therefore remains in some doubt.These observations raise a series of questions. How can natural experimentsbest be discovered and leveraged to improve causal inferences in the service ofdiverse substantive research agendas? What are appropriate methods foranalyzing natural experiments, and how can quantitative and qualitativetools be combined to construct such research designs and bolster theirinferential power? How should we evaluate the success of distinct naturalexperiments, and what sorts of criteria should we use to assess their strengthsand limitations? Finally, how can researchers best use natural experiments to

4Introduction: why natural experiments?build strong research designs, while avoiding or mitigating the potentiallimitations of the method? These are the central questions with which thisbook is concerned.In seeking to answer such questions, I place central emphasis on naturalexperiments as a “design-based” method of research—one in which controlover confounding variables comes primarily from research-design choices,rather than ex post adjustment using parametric statistical models. Muchsocial science relies on multivariate regression and its analogues. Yet, thisapproach has well-known drawbacks. For instance, it is not straightforward tocreate an analogy to true experiments through the inclusion of statisticalcontrols in analyses of observational data. Moreover, the validity of multivariate regression models or various kinds of matching techniques depends onthe veracity of causal and statistical assumptions that are often difficult toexplicate and defend—let alone validate.4 By contrast, random or as-if random assignment usually obviates the need to control statistically for potentialconfounders. With natural experiments, it is the research design, rather thanthe statistical modeling, that compels conviction.This implies that the quantitative analysis of natural experiments can besimple and transparent. For instance, a comparison of average outcomes acrossunits exposed to the presence or absence of a cause often suffices to estimate acausal effect. (This is true at least in principle, if not always in practice; onemajor theme of the book is how the simplicity and transparency of statisticalanalyses of natural experiments can be bolstered.) Such comparisons in turnoften rest on credible assumptions: to motivate difference-of-means tests, analysts need only invoke simple causal and statistical models that are oftenpersuasive as descriptions of underlying data-generating processes.Qualitative methods also play a critical role in natural experiments. Forinstance, various qualitative techniques are crucial for discovering opportunities for this kind of research design, for substantiating the claim that assignment to treatment variables is really as good as random, for interpreting,explaining, and contextualizing effects, and for validating the models used inquantitative analysis. Detailed qualitative information on the circumstancesthat created a natural experiment, and especially on the process by which“nature” exposed or failed to expose units to a putative cause, is often essential.Thus, substantive and contextual knowledge plays an important role at every4Matching designs, including exact and propensity-score matching, are discussed below. Like multipleregression, such techniques assume “selection on observables”—in particular, that unobservedconfounders have been measured and controlled.

5Introduction: why natural experiments?stage of natural-experimental research—from discovery to analysis to evaluation. Natural experiments thus typically require a mix of quantitative andqualitative research methods to be fully compelling.In the rest of this introductory chapter, I explore these themes and proposeinitial answers to the questions posed above, which the rest of the bookexplores in greater detail. The first crucial task, however, is to define thismethod and distinguish it from other types of research designs. I do thisbelow, after first discussing the problem of confounding in more detail andintroducing several examples of natural experiments.1.1 The problem of confoundersConsider the obstacles to investigating the following hypothesis, proposed bythe Peruvian development economist Hernando de Soto (2000): granting dejure property titles to poor land squatters augments their access to creditmarkets, by allowing them to use their property to collateralize debt, therebyfostering broad socioeconomic development. To test this hypothesis, researchers might compare poor squatters who possess titles to those who do not.However, differences in access to credit markets across these groups could inpart

1.2 Natural experiments on military conscription and land titles 8 1.3 Varieties of natural experiments 15 1.3.1 Contrast with quasi-experiments and matching 18 1.4 Natural experiments as design-based research 21 1.5 An evaluative framework for natural experiments 27 1.5.1 The plausibil