Transcription
COMPASS COMMISSIONThe Chilean Ministry of Planning asked J-PAL to convene a commission of local andinternational academics. The Compass Commission proposes the implementation of foursocial policies with their respective randomized impact evaluation. The goal is to guidethe design and implementation of social development policy in Chile based on rigorousscientific evidence.
ABOUT J-PALThe Abdul Latif Jameel Poverty Action Lab (J-PAL) is a network of 54 affiliated professors around theworld who are united by their use of Randomized Evaluations (REs) to answer questions critical topoverty alleviation. J-PAL’s mission is to reduce poverty by ensuring that policy is based on scientificevidence. J-PAL works to achieve this by:Conducting Rigorous Impact Evaluations J-PAL researchers conduct randomized evaluations to testand improve the effectiveness of programs and policies aimed at reducing poverty. There are morethan 242 evaluations that have been either completed or are ongoing.Building Capacity J-PAL provides expertise to people interested in rigorous program evaluation, andtraining to others on how to conduct randomized evaluations.Impacting Policy J-PAL’s policy group performs cost-effectiveness analysis to identify the mosteffective ways to achieve policy goals, disseminates this knowledge to policymakers, and workswith governments, NGOs, foundations, and international development organizations to promote thescale-up of highly effective policies and programs around the world.J-PAL was founded in 2003 at the Department of Economics of the Massachusetts Institute ofTechnology (MIT). During 2007, J-PAL opened two regional offices in Europe and South Asia aimingto form connections with local researchers and strengthening the impact of the results obtained fromthe evaluations of local programs. These offices are harbored by the Paris School of Economics (PSE)in France, and the Institute for Financial Management and Research (IFMR), in Chennai, India. Withthe same objective, during 2009 J-PAL inaugurated the Latin America regional office at the CatholicUniversity of Chile (PUC). Finally, during 2011 the African regional office opened at the Universityof Cape Town in South Africa.abdul latif jameel poverty action lab www.povertyactionlab.org
abdul latif jameel poverty action lab www.povertyactionlab.org compass commissionA BOU T T HE C O M P AS S COM M I SSI ONIn 2010 the Ministry of Planning of Chile(MIDEPLAN) asked the Abdul Latif Jameel PovertyAction Lab (J-PAL) to convene a commissionof experts who could identify the main socialpolicy challenges in Chile and propose innovativeprograms that would then be evaluated viarandomized impact evaluations.To this end, J-PAL convened the CompassCommission, a group of international and localacademics with extensive experience in publicpolicy and rigorous impact evaluations. Forsix months, the Commission worked with acoordinating team at J-PAL formed by membersof J-PAL’s Policy Group and J-PAL Latin America,who provided research support and executivecoordination.In addition, representatives of MIDEPLAN informedthe Commission about existing governmentprograms as well as budgetary priorities that couldbe instrumental in implementing the proposedevaluations. MIDEPLAN s representatives facilitateddialogue and coordination with other ministries andpublic institutions that may be involved in the futureimplementation and evaluation of the proposedprograms.During its ten sessions, the Commission’s workprogressed through three stages:identification of social policies:In the first stage, the Commission focusedon identifying key policy questions for socialdevelopment in Chile, which was followed by aseries of proposed interventions and evaluationstrategies that could provide answers to thesequestions. A wide range of topics was covered,including access and quality of education, labormarket intermediation, child development, criminalrehabilitation and prevention of risky behaviorsamong youth.policy proposals:In the second stage, after receiving inputs fromseveral public agencies, the Commission selectedthe four interventions it considered most promisingfor the development of Chile. The selection criteriaincluded the potential impact of the proposedprograms, the administrative and politicalviability of these initiatives, and the feasibility ofimplementing a randomized impact evaluation tomeasure its results.implementation and evaluation design:In the third stage, the Commission worked to designthe implementation and experimental evaluationof the proposed programs in greater detail. To thisend, the Commission was divided into four groups,each in charge of coordinating the progress of theirrespective proposals.
abdul latif jameel poverty action lab www.povertyactionlab.org compass commissionTH E C O M P A S S C O M M I S SI ON P R OP OSA LSthe preferential school subsidy:information and Choiceprograms for rehabilitation and socialintegrationThis proposal aims to understand how greaterchoice in school selection affects the educationaldecisions of low-income families, and ultimately,the academic performance of their children.Households will receive information on theireligibility for the recently created PreferentialSchool Subsidy program, which increased theschool subsidy for poor students by an additional50 percent, and information on schools they cannow access through the benefits of this program.Providing information about a benefit that manyintended beneficiaries are unaware of may improveeducational outcomes.This proposal focuses on public safety via strategiesfor rehabilitation and social reintegration of juvenileand adult offenders. The program would enhanceservices for rehabilitation and social reintegrationat prisons and in juvenile facilities. In the caseof prisons, the prison guards would be trained tosupport the program.support and incentivesfor teachersThis proposal seeks to increase the quality ofeducation by changing teacher incentives to bolsterteaching performance and effort in the classroom.The accompanying study will evaluate the impacton academic outcomes of a program that combinesan increase in the number of hours teachers spendon preparing for classes with individual monetaryincentives tied to student academic performance.opportunities for youthThe fourth proposal describes a multidimensionalprogram implemented in schools that includes jobplacement, business training, mentoring, and accessto health services. The program seeks to facilitateyouth access to the labor market and reduce riskybehaviors that lead to drug use and teen pregnancy.We hope that the work of theCommission serves not only to informthe design and implementation of socialpolicies in Chile, but also as a forum tohighlight the importance of public policydecisions based on rigorous evidence.A joint initiative between a governmentand an organization like J-PAL,committed to fighting poverty throughscientific evidence, could brings positiveresults for Chile and for other countriesthat pursue similar partnerships.
MEMBERS OF THE COMMISSIONHarald BeyerCentro de Estudios Públicos ChileEsther DufloM.I.T. and Director of J-PALFrancisco GallegoUniversidad Católica de Chile andScientific Director, J-PAL LatAmPaul GertlerU.C. Berkeley and J-PAL Global AffiliateCristóbal HuneeusUniversidad de ChileDiana KrugerUniversidad Adolfo Ibáñez ChileOsvaldo LarrañagaPNUD and Universidad de ChileClaudia MartínezUniversidad de Chile andJ-PAL Latam AffiliateAndrea RepettoUniversidad Adolfo Ibáñez ChileAndrés VelascoHarvard Kennedy SchoolJ-PAL MANAGEMENT AND RESEARCHMaría Eugenia BáezResearch Manager, J-PAL LatAmRyan CooperExecutive Director, J-PAL LatAmCristóbal MarshallPolicy Manager, J-PAL GlobalREPRES E N TAT I V E S O F M IN I ST R Y OF P LA N N I N GJeannette FuentesMaría Luisa Vergara
THE PREFERENTIAL SCHOOL SUBSIDY:INFORMATION AND CHOICEPU BLIC P O L I C Y R E L E VA NC E A N D C ON T E X TIn the Chilean educational system, in which familiesreceive a subsidy or voucher and can choose theschool they send their children to, the population issegmented into three socioeconomic groups. Highincome students attend privately funded institutionsthat charge a fee that only high income familiescan afford. Students from middle and upper-middleincome families tend to enroll in private, subsidizedschools that operate mostly in the form of jointfunding with contributions from the public sectorand the families themselves. Finally, lower incomestudents generally attend municipally managedschools free of charge.The Preferential School Voucher (SEP), introduced in2008, substantially modified the structure describedabove. All students of lower socioeconomic statusthat qualify as a priority student are now eligibleto receive a subsidy that is 50 percent higher thanthe existing subsidy. Therefore, SEP significantlyexpands the field of school choice for about 40percent of the poorest students. Many of the privatesubsidized schools that charge a co-payment andthat have joined the new scheme are now open forlow-income families for free.However, it is not clear that families are makingeffective use of this increased range of schoolchoice, and it is highly likely that they are not evenaware of this option.The proposal presented here consists of providinginformation to a group of families eligible forthe SEP as a way to enhance their understandingof the scope of benefits derived from the grantand become aware of the characteristics of theschools available for them to send their childrento. If families have now more choices andmore information on school quality, they couldpotentially choose schools with better academicresults. In this sense, the impact that greatereducational choice exerts in the process of selectionof schools - and eventually in the educationaloutcomes of low-income students - can be assessedthrough an experimental evaluation.E VA LUAT I ON DE SI G NSince the SEP is available to the entire eligiblepopulation but participation rates are still low, theproposed intervention is an induction to treatment,or encouragement strategy. To that effect, theinformation will be provided to a random sample ofthe eligible population.The way in which this information will be compiledand delivered will be determined through focusgroups, in which different alternatives will betested on families with similar socioeconomiccharacteristics to the future beneficiaries of thisintervention.
abdul latif jameel poverty action lab www.povertyactionlab.org compass commissionrandomization strategyThe randomization of the delivery of informationwill take place at two levels: neighborhoods andhousehold. First, around 397 neighborhoods willbe selected. These neighborhoods will have aminimum number of educational establishmentsthat families due to travel distance consider whenchoosing a school for their children. Then, thepercentage of families within the neighborhoodthat receive the information will be randomized.In a first group of neighborhoods, 50 percent ofthe families will receive the information, whilein a second group the information will be sentto only 30 percent of families. A third group ofneighborhoods will serve as a comparison group.These variables, as well as the information relatedto the socioeconomic characteristics of the familiesand students, are available from administrativedata managed by the Ministries of Planningand Education. It is a viable option to considercomplementing the administrative informationavailable with surveys given to both treatment andcomparison groups in each neighborhood. Thesesurveys would inquire about the school electionand application process, details which cannot beidentified using administrative data.Finally, families will be randomly selected to receivethe information so as to meet the percentagesestablished at the neighborhood level. Thisdesign will allow comparison of the families thatreceived information versus those who did not, andcapture potential general equilibrium effects whencomparing results between neighborhoods withdifferent percentages of treated families.To implement the evaluation the cooperation ofthe Ministry of Planning and Ministry of Educationwill be required, particularly to gain access to theadministrative data. Additionally, the evaluationrequires the support of an institution that will serveas the sponsor of the information to be sent tofamilies, which could be an academic entity linkedto some of the country’s universities.The experiment will be carried out among familieswith children entering the first and second yearsof primary education. In the group of first yearstudents, the aim is to influence the choice ofschool that children enter, while the second yearstudent group will measure any possible changes tobetter performing schools.outcomes to measureThe main outcome variables to be measuredare the SIMCE scores. SIMCE is a standardized,countrywide test that measures individual-levelknowledge in the areas of Spanish and mathematics.A second objective of this research is to measure theimpact of the intervention on repetition and dropoutrates; both variables could be relevant in the caseof low-income students, as well as the impact onfamily decisions regarding school choice.potential partnersestimated costsThe total estimated cost of the proposal is 192million Chilean pesos (approximately USD 0.38million). This sum includes: the delivery of theinformation using certified letters, surveys to agroup of families equivalent to 10 percent of theparticipants families, collection of descriptive datausing focus groups, and the expenses related to theadministration of the program evaluation.
SUPPORT AND INCENTIVES FOR TEACHERSPU BLIC P O L I C Y R E L E VA NC E A N D C ON T E X TOne of the most discussed issues surrounding thequality of education in Chile is how best to trainteachers throughout their careers, as well as thebest ways to attract and retain individuals to theprofession. It is perceived that improving boththe equity and quality of education depends on aserious consideration of these topics.Several proposals have been presented to addressthese issues. However, no real evidence of theeffectiveness of these proposals or the amountof time needed to produce observable effectshas been presented. With this aim, this proposalseeks to implement and evaluate an educationalprogram that combines two interventions: First, theintroduction of individual monetary incentives forteachers that improve the academic achievement oftheir students. Second, a reduction in the numberof hours dedicated to teaching in class, and anincrease in the amount of time spent on classpreparation and planning.Given the relevance of the topic and the potentiallylarge amount of resources involved in this program’simplementation, it seems appropriate to evaluate itseffectiveness using a randomized impact evaluation,which could shed light on important policy lessons.E VA LUAT I ON DE SI G NThe proposed program seeks to improve theacademic achievement of students through achange in incentives and the behavior of teachersfor a random sample of subsidized schools. Towardthis end we propose two interventions. The first is anincrease in the time spent in out-of-class activitiesfor fourth grade teachers. Since the teachers areresponsible for imparting all subjects in this grade,the program would increase the time for out-ofclass activities by three hours replacing the teacherin subjects like physical education or informatics.This is equivalent to a 30 percent increase inthe available time for planning and preparationof classes, correction of assignments and tests,and other tasks. This reduction also assumes thepresentation of a plan for the use of these additionalhours, and a monthly supervision of this plan.The second intervention is an individual monetaryincentive linked to students’ academic achievement.This incentive would reward with a bonus varyingbetween 500,000 and 1 million Chilean pesos(approximately USD 1,000 and 2,000), to the upperhalf of treated teachers that add more value to theirstudents. In particular, the first 10 percent wouldbe rewarded with an amount of one million pesos,equivalent to 1.4 times their gross monthly salary.The next 20 percent would be rewarded with abonus of 720,000 pesos and the last 20 percentwould receive 500,000 pesos.
abdul latif jameel poverty action lab www.povertyactionlab.org compass commissionrandomization strategyIn order to evaluate the program’s two components,a sample of schools to guarantee 600 fourth gradeteachers participating will be selected. The schoolswould then be randomly assigned to either thetreatment or comparison group. The treatmentgroup would be divided into three subgroups. Inthe first subgroup, the only intervention would bethe reduction in teaching hours for fourth gradeteachers. In the second subgroup, the incentiveswould be combined to the reduction in teachinghours. In the last subgroup, only the monetaryincentives would be implemented.outcomes to measureThe results would be measured throughstandardized academic tests and a measure ofvalue added. This last measure helps to determinethe specific contribution of the teacher in students’learning. For this purpose, a test will be applied atthe beginning and at the end of the school year. Bymeasuring the value added, the outcome variablecontrols for the effect of other relevant variablesin academic achievement that do not directlydepend on the establishment or the teacher, likesocioeconomic level of students and parents’schooling level. This determines more precisely howmuch the teacher has contributed to the student’slearning.An appropriate instrument to measure theoutcomes of the students who participate in theprogram is the SEPA test which is aligned with thenational curriculum . This test includes a seriesof questionnaires to students, parents and schoolprincipals, providing key additional informationfor the evaluation process related both to studentachievement and variables that illustrate thecharacteristics of teachers, students and schools.Surveying the teachers involved in the program isalso recommended, in order to better interpret theresults. Additionally, it is desirable to complementthe information collected with administrative datafrom the Ministry of Education.potential partnersThe implementation of this program andits evaluation would be benefited with thecollaboration of the Ministry of Education andthe Ministry of Planning, particularly for theidentification of schools and coordination betweenthe evaluation team, participating schools, andprincipals.estimated costsThe total estimated costs for the implementationand evaluation of the proposed program is 425million Chilean pesos (approximately USD 0.85million). This includes the costs associated withteacher replacements, monetary incentives, theapplication of the initial and final tests, datacollection, and a margin equivalent to 32 millionpesos (approximately USD 64,000) to cover theadministrative expenses related to the evaluationprocess.
PROGRAMS FOR REHABILITATIONAND SOCIAL INTEGRATIONPU BLIC P O L I C Y R E L E VA NC E A N D C ON T E X TThe reduction of crime in Chile has long been apolicy priority for both citizens and politicians.Driving this attention is a combination of factorsincluding rising prison populations, a concentrationof crime activities in impoverished areas and highrecidivism rates that have spearheaded the issue tothe top of the political agenda.One of the strategies to control crime and lowerrecidivism is the implementation of rehabilitationand social reinsertion programs for adults andyoung offenders. However, the available programsin this area in Chile are quite limited in scope asthey are only capable of reaching a small subsetof potential beneficiaries. Thus, even when theyare effective, they would likely have no significantimpact on crime reduction. The lack of services foroffenders will only be aggravated by the continualgrowth of prison population, which is additionallycausing problems due to prison overcrowding.In addition, there are a number of issues relatedto the design of these programs that, if fixed andimplemented on a larger scale, might ease theproblem.Many of the existing programs and policies tocontrol crime might conceptually have positiveimpact on reducing recidivism. However, a morepromising strategy is creating a quasi-market forrehabilitation and social reinsertion programs thatmay incentivize the creation of holistic, scalableapproaches to reduce recidivism and increase socialintegration of adult and young offenders.This proposal outlines one such program and anassociated randomized evaluation that can be usedto measure the impact of using holistic, competitiverehabilitation and social reinsertion programscombined, in the case of adult offenders, withtraining and incentives for prison guards.E VA LUAT I ON DE SI G NThe proposal aims to work with two separatedgroups of participants: adults imprisoned in jailsmanaged by Gendarmería, and youths interred indetention centers administrated by SENAME. Thesegroups of beneficiaries are institutionally distinct butcan be served by similar programs centered on theshared goal of lowering recidivism and increasingsocial integration as measured in terms of laborparticipation and educational achievements.For adult participants, the proposed programfeatures the creation of rehabilitation and socialinsertion programs at the prison level. The programsto be offered will be selected via competitivebids. It is expected that current providers of suchservices from the private and public sector will beamong the bidders. In addition to the reinsertionand rehabilitation programs targeted at inmates,participating establishments would also be servedby training and incentive programs for prison guardson their supporting role in this process.In the case of the youth, the proposal consistsof improving and expanding the current supplyof rehabilitation and social reinsertion programsamong juvenile offenders that have either droppedout from school or attend erratically.
abdul latif jameel poverty action lab www.povertyactionlab.org compass commissionrandomization strategyWithin detention centers, prisoners meeting certaincriteria will be invited to apply to participate inthe program. Beneficiaries will then be selectedrandomly from the pool of applicants. Incentive andtraining programs for prison guards will be allocatedrandomly at the prison level to guards working inestablishments with program beneficiaries.This design allows for the creation of threesubgroups within the treated population. The firstsubgroup would consist of individuals who takepart in the rehabilitation programs and whoseprison guards are also participating in the trainingand incentives program. The remaining two groupswould only take part in one of the two programsdescribed above.A similar design is proposed in the case of underageoffenders. They will be invited to participate inthe program and then be randomly assigned to atreatment or control group.outcomes to measureThe primary outcome measure is the recidivismrate of program participants. For this purpose,information about criminal record will be collectedabout both charges and arrests without charges,variables that can be obtained from administrativedata managed by the Ministry of Justice. A secondgroup of variables correspond to social reinsertionindicators such as labor participation, schooldropouts, educational progress, and characteristicsrelated to self-esteem. These variables can beobtained through administrative records and surveysat both the beginning and end of the program, andat a given time after the sentence is served in full.potential partnersThe implementation and evaluation of theseprograms requires the collaboration of the Ministryof Justice and the Ministry of Interior. The Ministryof Planning could play a role in the coordinationof various government agencies, in particular forthe collection of administrative data. The aid andsupport of the Public Security Department of theMinistry of Interior is expected in order to designthe public bid for the selection of rehabilitation andsocial reintegration programs to be implemented.estimated costsThe cost of intervention for adults amounts to 2,817million Chilean pesos a year (approximately USD5.63 million). This amount includes costs associatedwith providing rehabilitation programs for 1,000individuals, and training and incentives for 1,635prison guards. The cost of intervention to treat1,000 underage offenders amounts to 2,132 millionChilean pesos (approximately USD 4.26 million).
OPPORTUNITIES FOR YOUTHA comprehensive approach to address the problems of labor marketparticipation and risky behaviorPU BLIC P O L I C Y R E L E VA NC E A N D C ON T E X TTeens in high school are at a crucial stage of theirlives. On the one hand, they can invest in theirfuture through education or other activities thatenhance their future employability. On the other,they can engage in risky behaviors that generateshort-term benefits, but endanger their ability tosuccessfully enter the labor market, lead to drugaddiction, teen-age pregnancy or dropping out ofschool.Unfortunately, the evidence shows that Chileanyouth face significant problems entering the labormarket and a significant proportion has engaged inactivities associated with risky behaviors. Accordingto figures from the CASEN, a national survey, 17percent of teens between the ages of 15-19 arenot enrolled in an education establishment and donot have a job, a figure that rises to 27 percent forthose aged between 20-29. Further, between 14-17percent of all births in Chile are to teenage mothers(18 years and younger) and the segment of youngpeople between 12- 24 is the age group with thehighest average alcohol consumption.This proposal seeks to increase opportunities foryoung people by improving their employability andreducing the incidence of risky behavior througha multi-system approach. The suggested approachtakes advantage of various initiatives alreadyunderway in Chile, including mentoring programs,job skills training, labor market intermediationservices, and recreational, educational and healthservices. This proposal recommends summoning acommittee of experts from multiple backgrounds in2011, to study the best way to implement this policyand refine the design of implementation describedhere. At this stage, this proposal recognizes theneed for intervention in multiple dimensions andsuggests the expansion of some existing policies inthis regard.Finally, in order to learn lessons about the mosteffective ways to address the problems and makegood use of the resources involved, an evaluationof the effectiveness of the proposed programs usingrandomized controlled trials is proposed.E VA LUAT I ON DE SI G NThis proposal seeks to link programs in variousstages of development. Regarding labor marketintermediation services, the proposition wouldtake advantage of existing job placement programscurrently managed by The Ministry of Labor. Theintervention consisting of training in job skills willbe based on the “Aprendimiento” micro-enterpriseworkshops by Endeavor Chile. The health careproposal will bring the supply of health servicesfor adolescents into spaces designed especially forthem, through“Espacios Amigables,” an initiativeof the Ministry of Health. Finally, the mentoringprograms and information on higher educationcould be implemented through a public agencybased at the Ministry of Interior, which plans tocarry out a specific mentoring program in 2011.
abdul latif jameel poverty action lab www.povertyactionlab.org compass commissionDue to differences in implementation timing of theprograms cited above, the proposal recommends toproceed with a two-part intervention. The first willdeal directly with the job opportunities of teens thathave completed high school no more than a yearago, offering them job placement services.In the second step, and contingent on the successof the first year s intervention, the evaluationwill look to assess the impact of the proposedmultidimensional program. This will combine theprovision of job placement services with the otherproposed programs.randomization strategyDuring the first stage of implementation, therandomization will take place at the individuallevel. The target population that will receive jobplacement services will consist of graduates ofhigher education with less than a year outside theeducation system and who are inactive in the labormarket.In a second phase, the randomization will be doneat the school level. As the rest of the interventionsbegin being implemented, the impact of theprogram as a whole will be evaluated on studentsfrom 7th to 12th grade. Specifically, the “EspaciosAmigables” intervention will treat students in7th and 9th grade students, the entrepreneurshipprogram and mentoring will treat studentsbelonging to 10th and 12th grade, and the labormarket intermediation and job placement serviceswill target recent graduates from high school whoare not pursuing higher education.outcomes to measureSince the proposal includes interventions indifferent dimensions, different outcome variableswill be observed at different points in time. Theoutcome variables of interest include enrollmentrates in higher education, labor force participationand characteristics of the jobs obtained,entrepreneurship, teen pregnancy, and drugand alcohol consumption. The data source forthe evaluation can be obtained through varioussurveys and administrative data from the differentinstitutions involved in implementing the programsdescribed.potential partnersThe programs that would form the proposedmultidimensional intervention already exist. Someexist at the pilot stage, while others are operatingon a small scale. Thus, the support of the respectiveimplementing institutions wil
J-PAL was founded in 2003 at the Department of Economics of the Massachusetts Institute of Technology (MIT). During 2007, J-PAL opened two regional offices in Europe and South Asia aiming . J-PAL convened the Compass Commission, a group of international and local . now access through the benefits of this program.
