WIXLIB

Speedy reflects to herself. "This report is a start, but it does not tell me how poor PROSCOLparticipants are and what impact the program has on schooling. I need hard data." Later Speedyprepares Note 2, summarizing the types of data typically used in impact evaluations.Note 2: Data for impact evaluation**********Know the program well. It is risky to embark on an evaluation without knowing a lot about theadministrative/institutional details of the program; that information typically comes from theprogram administration.It also helps a lot to have a reasonably firm grip on the relevant "stylized facts" about the setting.The relevant facts might include the poverty map, the way the labor market works, the majorethnic divisions, other relevant public programs, etc.Be eclectic about data. Sources can embrace both informal, unstructured, interviews withparticipants in the program as well as quantitative data from representative samples.However, it is extremely difficult to ask counter-factual questions in interviews or focus groups;try asking someone who is currently participating in a public program: "what would you be doingnow if this program did not exist?" Talking to program participants can be valuable, but it isunlikely to provide a credible evaluation on its own.One also needs data on the outcome indicators and relevant explanatory variables. You need thelatter to deal with heterogeneity in outcomes conditional on program participation. Outcomes candiffer depending on whether one is educated, say. It may not be possible to see the impact of theprogram unless one controls for that heterogeneity.Depending on the methods used (Note 1), you might also need data on variables that influenceparticipation but do not influence outcomes given participation. These instrumental variables canbe valuable in sorting out the likely causal effects of non-random programs (Note 1).The data on outcomes and other relevant explanatory variables can be either quantitative orqualitative. But it has to be possible to organize it in some sort of systematic data structure. Asimple and common example is that one has values of various variables including one or moreoutcome indicators for various observation units (individuals, households, firms, communities).The variables one has data on and the observation units one uses are often chosen as part of theevaluation method. These choices should be anchored to the prior knowledge about the program(its objectives of course, but also how it is run) and the setting in which it is introduced.The specific source of the data on outcomes and their determinants, including programparticipation, typically comes from survey data of some sort. The observation unit could be thehousehold, firm, geographic area, depending on the type of program one is studying.Survey data can often be supplemented with useful other data on the program (such as from theproject monitoring data base) or setting (such as from geographic data bases).There is a promising lead in the MSD file. Nine months ago the first national householdsurvey of Labas was done by the Labas Bureau of Statistics (LBS). It is called the Living StandardsSurvey (LSS). The survey was done for a random sample of 10,000 households, and it asked abouthousehold incomes by source, employment, expenditures, health status, education attainments, and5

demographicand other attributes of the family. There is a letter in the file from MSD to LBS, a fewmonths prior to the LSS, asking for a questionto be added on whetler or not the sampled householdhad participatedin PROSCOL.The reply from LBS indicatesthat the listingof income sources in thesurvey schedulewill include a line item for money received from PROSCOL."Wow", says Speedy, and she heads off to LBS.Speedy Analyst already knowsa few things about the LSS, havingused tabulations from itproduced by LBS. But Speedy worriesthat she will not be able to do a good evaluation ofPROSCOLwithout access to the raw household-leveldata. But after a protractedand unsuccessfuldiscussionwith the head of the LBS unit in charge of the survey, and seeminglyendless follow-upphone calls, Speedy starts to worrywhethershe will get the data, and have anything on outcomesworth showingher boss.However,after a formal request from the Minister (which Speedy wrote for him to sign), theSecretaryof Statistics finally agreesto give her the micro data. Then, after a few more phone calls,LBS also gives her a copy of the documentationshe needs to readthe data.Speedy already knows how to use a statistics package called SAPS. After a long and painfulday figuringout how to use the raw LSS data, Speedy starts the real work.She uses SAPS to make across-tabof the average amount receivedfrom PROSCOL by deciles of households, where thedeciles are formed by ranking all householdsin the sample accordingto their income per person. Incalculatingthe latter, Speedy decidesto subtract any monies receivedfrom PROSCOL; this, shereckons, will be a good measure of income in the absence of the program. So she hopes to revealwho gainedaccording to their pre-interventionincome.The cross-tab suggeststhat the cash transfers under the programare quite well targeted to thepoor. By the official LBS povertyline, about30% of NorthwestLabas' population is poor. From hertable, she calculates that the poorest 30% of the LSS sample receive70% of the PROSCOL transfers.This looks like good news for PROSCOL,Speedy reflects.6

What about the impact on schooling? She makes another table, giving average schoolenrollmentrates of various age groupsfor PROSCOL familiesversus non-PROSCOLfamilies. Thissuggestsalmost no difference betweenthe two; the average enrollmentrate for kids aged 6-18 isabout 80% in both cases.Speedy then calculates averageyears of schooling at each age and plots the results separatelyfor PROSCOLfamilies and non-PROSCOLfamilies. The two figures are not identical, but they arevery close."Was there really no impacton schooling,or have I done somethingwrong?" she asksherself. The question is just the beginningof the story of how SpeedyAnalyst solves the mysteryofthe vanishingschooling benefits from PROSCOL.Speedy Analyst visits Mr. Unbiased StatisticaSpeedy decides to show her curiousresults to a couple of trusted colleagues. First she visitsMr. Unbiased Statistica, a senior statisticianat LBS. Speedy likes Statistica,and feels comfortableasking him about statistical problems."Mr Statistica, my calculationsfrom the LSS suggest that PROSCOLkids are no more likelyto be in school than non-PROSCOLkids. Have I done somethingwrong?"Statisticatells her bluntly:"Speedy, I think you may well have a serious bias here. To knowwhat impactPROSCOL has, you needto know what would have happen withoutthe program. Thegain in schoolingattributable to the programis just the difference betweenthe actual schoolattendance rate for participatingkids and the rate for those same kids if the program had not existed."What you are doing Speedyis using non-PROSCOLfamiliesas the comparison group forinferringwhat the schooling wouldbe of the PROSCOLparticipants if the program had not existed.This assumesthat the non-participantscorrectlyreveal, at least on average, schooling without theprogram. Some simple algebra might help make this clear."7

Mr. Statisticastarts writing. "Let Pidenote PROSCOLparticipationof the i'th child. This cantake two possible values, namely Pi I when the child participates in PROSCOLand Pi 0 when shedoes not. When the i'th child does not participate,her level of schooling is S0iwhich stands for childi's schoolingS when P O. When the child does participateher schooling is Sli. The gain inschooling due to PROSCOLfor a child that does in fact participateis:G, SIi-So,JPi l"Why do you need this I "? asks Speedy."That stands for 'given that' or 'conditional on' if you prefer. The''is needed to makeit clear that we are calculating the gain for a child who actually participated. If we want to knowthe average gain we simply take the mean of all the G's. This will give you the sample meangain in schooling amongst all those who participated in PROSCOL. As long as you havecalculated this mean correctly (using the appropriate sample weights from your survey) it willprovide an unbiased estimate of the true mean gain. The latter is what statisticians often call the"expected value" of G, and it can be written as:G E(SI -SoiIPi 1)You can think of this as another way of saying 'mean'. However, it need not be exactly equal to themean you calculatefrom your sample data, giventhat there will be some samplingerror. In theevaluation literature, E(S1-So, IPi l) is sometimes called the 'treatment effect' or the 'averagetreatment effect on the treated'."Speedy thinks to herself that the governmentwould not like to call PROSCOLa "treatment".But she is elated by Statistica's last equation. "Yes Mr. Statistica,that is exactlywhat I want toknow."8

"Ah, but that is not what you have calculatedSpeedy. You have not calculatedG but ratherthe difference in mean schoolingbetween kids in PROSCOLfamilies and those in non-PROSCOLfamilies. This is the sample estimate of:D E(Sli I Pi l) - E(So0 IPi O)There is a simple identitylinking the D and G, namely:D G BThis term 'B' is the bias in your estimate, and it is givenby:B E(SojI Pi 1) - E(So,I P1 0)In other words, the bias is the expecteddifference in schoolingwithout PROSCOLbetween childrenwho did in fact participatein the program and those who did not. You could correct for this bias ifyou knew E(So0 IP, l). But you can't even get a sampleestimate of that. You can't observe what theschooling would have been of kids who actually participatedin PROSCOL had they not participated;that is missing data - it is called a 'counter-factual' mean."Speedy sees that Statisticahas a legitimateconcern.In the absence of the program,PROSCOLparents may well send their kids to school less than do other parents. If so, then there willbe a bias in her calculation.What the Finance Ministerneeds to know is the extra schoolingdue toPROSCOL. Presumablythis only affects those familieswho actually participate.So the Ministerneeds to know how much less schoolingcould be expectedwithout the program.If there is no bias,then the extra schoolingunder the program is the differencein mean schoolingbetweenthose whoparticipatedand those who did not. So the bias arises if there is a difference in mean schoolingbetween PROSCOLparents and non-PROSCOLin the absenceof the program."What can be done to get rid of this bias, Mr. Statistica?""Well, the best way is to assign the programrandomly.Then participantsand nonparticipants will have the same expected schoolingin the absence of the program,i.e., E(So0 I P l)E(So,IPi O). The schoolingof non-participatingfamilieswill then correctlyreveal the counter9

factual, i.e., the schoolingthat we would have observed for participantshad they not had accesstothe program. Indeed, random assignmentwill equate the whole distribution,not just the means.There will still be a bias due to samplingerror, but for large enough samplesyou can safely assumethat any statisticallysignificantdifference in the distributionof schoolingbetween participantsandnon-participants'is due to the program."On recalling what she read in the .PROSCOLfile, Speedyrealizesthat she need look nofurther than the design of the programto see that participationis not random. Indeed, it would be aserious criticism of PROSCOLto find that it was. The very fact of its purposivetargeting to poorfamilies,who are presumably less likely to send their kids to school,would create bias.She tells Mr. Statisticaaboutthe program's purposiveplacement."So, Speedy, if PROSCOLis working well then you shouldexpect participants to have worseschoolingin the absence of the program. Then E(SoiIP, 1) E(SoiIPi 0) and your calculationwillunderestimatethe gain from the program. You may find little or no benefit even though the programis actually working well."Speedy returns to her office, despondent.She sees now that the magnitude of this bias thatMr. Statistica is worried about couldbe huge. Her reasoningis as follows: Supposethat poor familiessend their kids to work rather than school; because they are poor and cannotborrow easily, they needthe extra cash now. Non-poor familiessend their kids to school. The programselects poor families,who then send their kids to school.One observes negligibledifference in mean schoolingbetweenPROSCOLfamilies and non-PROSCOLfamilies; indeed, E(Sj1 I Pi 1) E(Soi I Pi 0) inexpectation. But the impact of the program is positive, and is givenby E(SoiI Pi 0) - E(SoiI Pi 1).The failure to take account of the program's purposive, pro-poor,targeting could well have led to avery substantial under-estimationof PROSCOL's benefits from her comparisonof mean schoolingbetween PROSCOL families and non-PROSCOLfamilies.10

A visit to Ms Tangential EconomisteNext Speedy visits a colleagueat the Ministry of Finance,TangentialEconomiste. Tangentialspecializesin public finance. She has a reputationas a sharp economist in the Ministry, thoughsometimes a little brutal in her commentson her colleague's work.Speedy first shows her the cross-tabof amounts receivedfrom PROSCOLagainst income.Tangential immediatelybrings up a concern,which she chastises Speedy for ignoring."You haveclearly overestimatedthe gains to the poor from PROSCOLbecauseyou have ignored foregoneincome, Speedy. Kids have to go to schoolif the family is to get the PROSCOLtransfer. So theywill not be able to work, either on the familybusiness or in the labor market. Kids aged 15-18 canearn two-thirds or more of the adult wage in agriculture and construction,for example. PROSCOLfamilies will lose this income from their kids' work. You shouldtake account of this foregoneincome when you calculate the net incomegains from the program.And you should subtract this netincome gain, not the gross transfer,to work out pre-interventionincome. Onlythen will you knowhow poor the family would have been is in the absence of the PROSCOLtransfer. I reckon this tablemight greatly overstate the program's gains to the poor.""But why should I factor out the foregone income from child labor? Less child labor issurely a good thing," Speedy says in defense."You should certainly look at the gains from reducing child labor Speedy, of which themain gain is no doubt the extra schooling, and hence higher future incomes of currently poorfamilies. I see your next table is about that. As I see it, you are concerned with the two mainways PROSCOL reduces poverty: one is by increasing the current incomes of the poor, and theother is by increasing their future incomes. The impact on child labor matters to both, but inopposite directions. So PROSCOL faces a trade off."11

Speedy realizes that this is another reason why she needs to get a good estimate of theimpact on schooling; only then will she be able to determine the foregone income that Tangentialis so worried about. Maybe the extra time at school comes out of non-work time.Next, Speedy tells Tangential about Mr. Statistica's concerns about her second table, tosee what she thinks."I think your main problemhere is that you have not allowed for all the other determinantsofschooling,besides participationin PROSCOL. You should run a regressionof years of schoolingona set of control variables as well as whether or not the child's family was covered by PROSCOL.Why not try this regression?" Tangentialwrites. "For the i'th child in your sample let:Si a bPi cX, siHere a, b and c are parameters,X standsfor the control variables,such as age of the child, mother'sand father's education,the size and demographiccompositionof the household and schoolcharacteristics,while E is a residualthat includes other determinantsof schooling,and measurementerrors. You can see Speedy that the estimatedvalue of b gives you the impact of PROSCOLonschooling.""No, I don't see that," Speedyinterjects."Well, if the family of the i'th child participates in PROSCOLthen P 1 and so its schoolingwill be a b cX, si. If it does not participate,then P Oand so its schooling will be a cX1 i.The difference between the two is the gain in schoolingdue to the program,which is just b."This discussion puts Speedy in a more hopeful mood, as she returns to her office to try outTangential's equation. She runs the REGRESScommand in SAPSon the regression with andwithout the control variables Tangentialsuggested. When she runs it without them, she finds that theestimatedvalue of b is not significantlydifferent from zero (usingthe standardt-test given by SAPS).This looks suspiciously like the result she first got, taking the differencein means between12

participants and nonparticipants-suggestingthat PROSCOLis not having any impact onschooling. However, when she puts a bunch of control variables in the regression, she immediatelysees a positive and significantcoefficienton PROSCOLparticipation. She calculates that by 18years of age, the program has addedtwo years to schooling.Speedy thinks that this is startingto look more convincing.But she feels a little unsure aboutwhat she is doing. "Why do these control variables make such a difference? And have I used theright controls? I need more help if I am going to figure out what exactly is going on here, andwhether I should believethis regression."Professor Chisquare helps interpret Speedy's resultsSpeedy decides to visit Professor Chisquare,who was one of her teachers at LNU. Chisquareis a funny little man, who wears old-fashionedsuits and ties that don't match too often. "It isjust notnormal to be so square", she recalls thinking during his classes in econometrics. Speedy also recallsher dread at asking Chisquareanything,because his answerswere sometimes very hard tounderstand."But he knows more about regressions than anyoneelse I know", Speedy reflects.She arranges a meeting. Having heard on the phone what her problem is, Chisquaregreets hisex-studentwith a long list of papers to read, mostly with rather impenetrabletitles, and published inseeminglyobscure places."Thanks very much Professor,but I don't think I will have time to read all this before myreport is due. Can I tell you my problem, and get your reactions now?"Chisquare agrees. Speedyshows him the regressions,thinkingthat he will be pleased that hisex-studenthas been running regressions. He asks her a few questionsabout what she had done, andthen rests back in his chair, ready, it seems, to pronouncejudgement on her efforts so far.13

"One concern I have with your regressionof schoolingon P and X is that it does not allowthe impact of the program to vary with X; the impact is the same for everyone,which does not seemvery likely.""Yes, I wondered about that," chips in Speedy. "Parents with more schoolingwould be morelikely to send their kids to school, so the gains to them from PROSCOLwill be lower"."Quite possibly, Ms Analyst. To allowthe gains to vary with X, let mean schooling of nonparticipantsbe ao coXiwhile that of participantsis a, c1 Xi, so the observedlevel of sc

Sep 10, 1999 · The Mystery of the This entertaining introduction Vanishing Benefits to the concepts and methods of impact evaluation -as seen through the eyes of Ms. Ms. Speedy Analyst's Introduction Speedy Analyst-assumes readers are familiar with basic to Evaluation statistics up to regression analysi