Transcription
Survey Data Collection forImpact EvaluationIt’s Not as Easy as We MightThink!Adam RossSeoul, KoreaDecember 2010
Impact Evaluation ProjectCycleDesign the interventionDesign the impactevaluationCollect Baseline DataRollout the interventionOngoing Monitoringand Process EvaluationScope of thispresentationCollect Follow-up DataAnalysis2
Assumptions of this presentation1.2.3.*We‟re planning a prospective impactevaluationWe need baseline and follow-up surveydata on treatment and control group(s) tomeasure program impact*We are going to collect our own data forthe impact evaluationProcess/implementation data should also begathered throughout the intervention
Before collecting your own data Can we use existing data? Regular surveys (census, DHS survey) Regular monitoring (annual achievement tests) Administrative records (health records, schoolenrollment) In many settings, administrative data isinsufficient, poor quality and low coverage Ex. Turkey vs. Argentina
Before collecting your own dataWho should collect? Bureau of Statistics: has capacity & good place to invest in further capacityUniversity: maybe cheaper, often lessinfrastructure and thus more monitoringExternal firm: depends on capacity andexperienceWhen de we need to start? Procurement, training and data collection alltake time
Objectives of data quality controlTo collect data that:1. accurately reflects the reality of thepopulation2. is representative of the entire targetpopulation3. allows policy makers and analysts tomake real-time, informed decisionsNeed to control sampling and nonsampling error!!!!6
What are Sampling and NonSampling Error? Sampling error: the result of observing asample of n households (the sample size) ratherthan all N households in the target populationNon-sampling error: the result of errors insurvey development and execution. Someexamples are: measurement error - when the answers written onthe questionnaires are different from the actual valuesselection bias - results from imperfections in thesample frame or deficiencies in the sample selectionprocessnon-response - when we don't get an answer tosome or all of our questions from certain households7
The Total Survey Design(TSD) approachTools &MethodsSamplingDataFieldworkmanagement8
Data Collection PlanIncludes:1. Sampling2. Tools and methods 3.4.5.6.questionnaires and instrumentsFieldworkData ManagementBudgetTimeline of activities
1. Sampling Develop sample design - sample size andgeographic distributionGet an updated sample frame andcorresponding cartography Local statistical office, Ministry of HealthDo we have to generate a listing? before the survey or in parallel?Compute sampling weights and samplingerrors as necessary10
2. Tools and Methods:Questionnaires What do you want to know? Tailor your survey to capture outcomes ofinterestUse reliable and valid instruments Be careful: what‟s reliable and valid in onecultural and linguistic context may not be so inanother Adapt it to the country specific reality and locallanguage(s)Save time, money and pain: test yourquestionnaire!
Best Practices for Questionnaires Define your topics and concepts to avoidconfusionQuestion order mattersKeep it short and make it user-friendlyPhrase questions clearlyUse established techniques to minimizerespondent mistakes (e.g. calendars forevent histories)
Best Practices for Questions II A good question is understood consistently byall respondentsA good question is administered consistently toall respondentsA good question elicits the kind of answers theresearcher wants: BadQ: When did you move to Seoul, Korea? BetterQ: In what YEAR did you move to Seoul, Korea? A: In 1964 A: When I was 20 years old A: After I finished college
Best Practices for Questions III A good question is one where therespondents have the necessaryknowledge to answer asking a good question of the wrong person isa source of error in your data A good question is one where therespondent is willing to provide the „true‟answer difficult for sensitive questions
Best Practices for Questions IV Ask about first hand experienceAsk one question at a time BadQ: Are you physically able to do things like walk or carry a full water bucket withoutdifficulty?BetterQ: Are you physically able to carry a fullwater bucket without difficulty?
Who Are Your Respondents? Different respondents require differenttechniques, e.g. youth are a tough crowd: Often mobile Not always well informed Require special consent procedures Cagey about socially undesirable behavior Can have low literacy levels16
Best Practices for SensitiveQuestions Use open questions for frequencies ofundesirable behaviorDesign long questions but short instrumentUse familiar words (know local terms)Ask „have you ever done x‟ before asking „areyou currently doing x‟ for socially undesirablebehaviorEmbed threatening questions in a list of moreor less threatening topics17
How Will You Capture the Data? Self Administered Questionnaire (SAQ)Telephone (CATI,RDD)Paper and pencil (PAPI) Data entry Scannable formsComputer Assisted Personal Interview(CAPI)Audio Computer Assisted Self Interview(ACASI)18
Questionnaire Development The survey researchers‟ axiom: Everyone thinks they can design aquestionnaire Include an experienced survey researchorganization in the development of yourquestionnaireBegin with examples of successfulquestionnaires fielded in similar settings
Questionnaire Testing Evaluate your questions Evaluate your questionnaire andprocedures Conduct focus groups Conduct cognitive testing Conduct full field pretest Revise questionnaire and procedures
3. Fieldwork:Staffing and training Interviewer staffing and training is directlyrelated to data qualityRecruit supervisors, interviewers and dataentry operators very carefullyAll materials must be finalized beforetraining begins: Training materials Field manuals Questionnaires data-entry program
Train interviewers thoroughly Training should have four main elements1. plenary sessions: introduction and lectures2. practical work in small groups: simulated interviews, role playing,interpreting inconsistencies, etc.3. practice fieldwork4. candidate assessmentHigh-quality training takes time Plan to spend roughly 2-4 weeks dependingon questionnaire complexity This is often underestimated
Field Management for QualityData Maximize response rates: Use “Field Team” approach for datacollectionEmploy Computer Assisted Field Entry(CAFÉ ) Schedule visits sensibly Follow up with non-respondents Consider incentives Gather locating data for follow up
Composition of a “Field Team”SupervisorInterviewersAnthropo-metristData entryoperator
Field ManagementMobile teams with integrated data entrySaltaEntre RiosJujuyTeam works withportable computersand printersRegionalOffice
Field ManagementMobile teams with integrated data entrySaltaEntre RiosJujuyOperator travelswith the rest of thefield teamRegionalOffice
Field ManagementMobile teams with integrated data entrySaltaEntre RiosJujuyData entry andvalidation almostimmediateRegionalOffice
Field ManagementMobile teams with integrated data entrySaltaEntre RiosJujuyReduced trips to andfrom Regional Officeto selected PSUsRegionalOffice
Field ManagementMobile teams with integrated data entrySaltaEntre RiosJujuyRegionalOffice
Computer-Assisted Field Entry(CAFÉ ) provides immediate feedback on theperformance of the field staffallows early detection of anyinappropriate behaviorenables inconsistencies to be correctedat their source by re-visiting the HH,rather than ex-post “data cleaning”based on assumptions
4. Data Management Develop a data entry program thatincludes quality control checksInclude data management proceduresthat emphasize confidentiality Remove unique identifiers when transmittingand storing the data Monitor your sample carefully usingfrequent supervision reports
Monitor and assess the quality of 401)HouseholdsNumber ofHouseholdsHouseholdsTotal Nb ofNb of HHreportingcropsreportingreporting fishing durables having creditagriculturalreportedlivestockactivities (Q924) (Section 12) (Section 13)activities (Q901)(Q911)activities (Q918)HH sizemeanNumber oflines with 259,7227057.196,05232
5. Budget Survey data collection can be expensive BUT it isn‟t as expensive as carrying out programsthat are not effective.The long term savings of conducting athorough impact evaluation with quality datacollection can be immeasurable in social termsHigh quality data is vital to conducting aneffective impact evaluation
How much is this going to cost?[hint: quite a bit] 320 primary schools10 studentquestionnairestest of 40 students 1,000 primaryschoolshead teacherquestionnaireteacher questionnaire
How much is this going to cost?[hint: quite a bit] 320 primary schools10 studentquestionnairestest of 40 studentsUSD 35,000USD 200,000
How much is this going to cost?[hint: quite a bit] 1,000 primaryschoolshead teacherquestionnaireteacher questionnaireUSD 50,000 USD 60,000
Large variations in cost320 schools USD 200,0001,000 schools USD 50,000Big differences! Scale of survey Size of country Labor costs Difficulty in getting to schoolsHousehold surveys: More expensive
6. Timeline: Quality datacollection takes timeActivityTiming Needed1. Procurement2-4 months2. Development of SampleDepends on sample design andavailability of sample frame3. Adapt and test questionnaire tothe local conditions4-6 weeks4. Staffing and training2-4 weeks for recruitment andselection, 2-4 weeks for training5. Data management4-6 weeks6. Data Collection2 months38
M1M2M3M4M5M6M7M8M9M10M111 Management, Financing, Logisticsand Institutional Agreements2 Sample3 Questionnaire design4 Data management5 Staffing and training6 Survey in the field. 16 weeks?7 Database preparation8 Analysis. 2 months?39M12M13M14
Sample Project Plan: 30 minutePAPI40
Quality Data Collection:Key Takeaways1.2.3.Pretest data collection plan andquestionnaires, leaving time foradjustmentsSelect interviewers carefully, train themthoroughly and evaluate them dutifullyUse “Field Team” approach and CAFÉwhen possible41
Quality Data Collection:Key Takeaways4.5.Data collection is expensive: chooseyour mode and sample sizes wiselyBuild a realistic timeline with sufficienttime for questionnaire development,testing, training and adjustments
Survey data collection can be expensive BUT it isn‟t as expensive as carrying out programs that are not effective. The long term savings of conducting a thorough impact evaluation with quality data collection can be immeasurable in social terms High quality data is vital to conducting an effective impact evaluation
