WIXLIB

Basic Survey Theory and DesignConstantine BoussalisHarvard Law Schoolcboussalis@law.harvard.edu

Surveys: Not Just Asking People Questions The purpose of a survey is to produce accurate informationthat reflects the views and experiences of a givenpopulation. To achieve valid inferences from a survey, we need toreduce (as best we can) four distinct types of error:1. Coverage Error2. Sampling Error3. Non-response Error4. Measurement Error Theory and empirical evidence can help us tackle thesesources of bias (misrepresentation of the population).2/1/20122

Some Key Terms & ConceptsPopulation: All of the units (e.g. individuals, households,organizations) to which one desires to generalize surveyresults.Sample: Units of the population that are drawn for inclusion inthe survey.Sample frame: The list from which a sample is to be drawn inorder to represent the population.Completed sample: The units that complete thequestionnaire.2/1/20123

Generalizing to the PopulationSamplePopulation2/1/20124

Coverage Error The results of a survey can be biased when:1.2. Not all members of the population have a known, non-zero chanceof being included in the sample.The members of the population who were excluded aresubstantively different than the included members.Coverage error, therefore, stems from the inability of thesurvey to contact segments of the population.Strategies to avoid coverage error:1.2.3.2/1/2012Consider whether the survey mode (i.e. internet, telephone, mail,etc.) is applicable to all units.Be sure that units included in the sample frame are indeedmembers of the population of interest and not from anotherpopulationAvoid unit duplication. This can lead to double-counting which, inturn, is a source of bias.5

Sampling Error The number of properly selectedcompleted responses determinesthe level of precision of ourestimates (margin of error). As the sample size populationsize, the margin of error approacheszero 3 5o This is often too costly .o We do not need to sample theentire population. The size of the sample, not theproportion of the populationsampled, affects precision. Diminishing returns on reducedmargin of error as completedsample size increases.2/1/2012 10Source: Dillman et al. (2009)6

Non-Response Error Systematic error can also arise when not everyone selectedfor the survey completes the questionnaire.Crucially, bias occurs when the units who do not respondare substantively different from those who did completethe survey on the item of interest.o E.g. In a study estimating U.S. political opinion, if nonrespondents are predominantly, say, Republican, theresults of the study will be biased. If the survey was estimating ice cream preferences,this heterogeneity might not be too problematic.Motivating (incentivizing) respondents to complete thesurvey is a good strategy for reducing non-response error.2/1/20127

Measurement Error Bias resulting from inaccurate answers to questions.How can a response be “inaccurate”?oo Poor question wording and designImproper questionnaire constructionSome factors to consider when crafting a question:oo2/1/2012Question format (e.g. open- vs. closed-ended, single vs. multipleanswer, etc.)Wording of the question stem8

Question Formats: Open-ended Type answer in a blank box.Pros:o Allows respondent to freelyanswer questiono Provides rich, detailedinformationo Appropriate for short,numerical responses (e.g. age)Cons:o Respondents more likely toskip open-ended questionso Coding answers into categoriesis resource intensive2/1/2012Open-ended questionHow would you describe the dean’sleadership abilities?How would you describe the dean’sability to innovate?9

Question Formats: Closed-ended Respondents provide an answerafter considering choices provided.Nominal scalar questions provideanswers with no natural order.oo E.g. “check-all-that-apply” andranking questions .Response difficulty increases asnumber of categories increases.Ordinal scalar questions provide anordered set of answer categories.ooResponses fall on some type ofcontinuum.E.g. intensity of satisfaction,frequency of behavior or events2/1/2012Nominal, closed-endedWhich of these five statements best describes the dean?A strong leaderA strong innovatorBoth a strong leader and innovatorNeither a strong leader nor innovatorOrdinal, closed-ended for each conceptTo what extent has the dean demonstrated strongleadership qualities?All of the timeMost of the timeSome of the timeNone of the timeTo what extent has the dean demonstrated an ability toinnovate?All of the timeMost of the timeSome of the timeNone of the time10

Question Formats: Partially closed Hybrid of the open- and closedended formats.Includes set of responses and an“other” category.Pros:o Reduces number of itemsrespondents have to considersimultaneously.o Allows for some freedom.Respondents more likely to answerprovided answers than original answer.o We cannot accurately compareresponses of listed to unlistedoptions2/1/2012Partially closed questionWhich of the following is your favorite college wimmingTennisOther: Please specify11

Make every question require an answerA question that does not require an answer form every respondentA revision that uses a filter questionIf you made dinner at home last night, about how manyminutes did it take to prepare the meal?Did you make dinner at home last night?YesminutesNoIf yes, how many minutes did it take to prepare themeal?minutes2/1/201212

Make sure the question is technically accurateHow many feet tall is your horse?feet2/1/2012How many hands tall is your horse?hands13

Ask one question at a time (double-barreled question)A double-barreled questionA revision to ask each question separately and collect more infoDo you personally subscribe to, or regularly read, anyperiodicals, magazines, newsletters, etc. that arespecifically related to your occupation?Please indicate whether or not you personally subscribeto each of the following sources of informationspecifically related to your occupation.YesNo2/1/2012I do subscribe I do not subscribePeriodicalsMagazinesNewsletters14

Use simple and familiar words1. Exhausted2. Candid3. Top priority4. Leisure5. Employment6. Courageous7. Rectify8. “Occupants of this household”9. “Your responses”10. “Post-school extracurricular activities”11. “Subnational region”2/1/20121. Tired2. Honest3. Most important4. Free time5. Work6. Brave7. Correct8. “People who live here”9. “Your answers”10. “What you do after school”11. “Area of the country”15

Use specific and concrete words to specify concepts clearlyHow many times did you eat together as a family lastweek?# of times2/1/2012How many meals did you sit down to eat at home as afamily last week?# of meals16

Fewer words are better1. Due to the fact that2. At this point in time3. A small number of4. A considerable number of5. Small in size6. Has the ability7. Ascertain the location of8. Concerning the matter of9. If conditions are such that10. In the majority of instances11. Make a decision12. Take into consideration2/1/20121. Because2. Now3. A few4. Many5. Small6. Can7. Locate8. About9. If10. Usually11. Decide12. Consider17

Use complete sentences with simple sentence structuresCommon use of incomplete sentencesA revision using complete sentencesNumber of years lived in MassachusettsHow many years have you lived in Massachusetts?YearsYearsYour city or townCity or TownName of City or TownIn what Massachusetts county do you live?Your countyCounty2/1/2012In what city or town do you live?Name of Massachusetts County18

Make sure “yes” means yes and “no” means noA question containing a double negativeA revision with no double negativeDo you favor or oppose not allowing the state to raisetaxes without approval of 60% of the voters?Do you favor or oppose requiring 60% approval byvoters in order to raise taxes?FavorFavorOpposeOppose2/1/201219

Question must specify the response taskHow many days did you exercise last week?All of the timeMost of the timeSome of the timeNone of the time2/1/2012How many days did you exercise last week?Number of days (0-7)20

State both positive and negative sides in the question stemDo you favor congressional term limits of four years?FavorOpposeTo what extent do you agree with this statement “ ”?Strongly agreeSomewhat agreeSomewhat disagreeStrongly disagreeDo you favor or oppose congressional term limits of fouryears?FavorOpposeTo what extent do you agree or disagree with thisstatement “ ”?Strongly agreeSomewhat agreeSomewhat disagreeStrongly disagree2/1/201221

Answer categories must be mutually exclusiveWhat is your annual income?What is your annual income?Less than 25,000Less than 25,000 25,000 – 50,000 25,000 – 49,999 50,000 -- 75,000 50,000 -- 75,000More than 75,000More than 75,0002/1/201222

Avoid check-all-that-apply questionsWhich of the following varsity sports would youconsider yourself to be a fan of? Please check all thatapplyMen’s baseballWomen’s basketballMen’s basketballWomen’s cross-countryMen’s cross-countryMen’s footballWomen’s golfMen’s golfWomen’s rowingWomen’s soccerWomen’s swimmingWomen’s tennisWomen’s track and fieldMen’s track and fieldWomen’s volleyball2/1/2012Do you consider yourself to be a fan of each of thefollowing varsity sports?yesnoMen’s baseballWomen’s basketballMen’s basketballWomen’s cross-countryMen’s cross-countryMen’s footballWomen’s golfMen’s golfWomen’s rowingWomen’s soccerWomen’s swimmingWomen’s tennisWomen’s track and fieldMen’s track and fieldWomen’s volleyball23

State both positive and negative sides in the question stemDo you favor congressional term limits of four years?FavorOpposeTo what extent do you agree with this statement “ ”?Strongly agreeSomewhat agreeSomewhat disagreeStrongly disagreeDo you favor or oppose congressional term limits of fouryears?FavorOpposeTo what extent do you agree or disagree with thisstatement “ ”?Strongly agreeSomewhat agreeSomewhat disagreeStrongly disagree2/1/201224

Recommended ReadingDillman, D.A., Smyth, J.D., &Christian, L.M. (2009) Internet,Mail, and Mixed-Mode Surveys:The Tailored Design Method (3rdEdition). Hoboken, N.J.: Wiley &Sons.ISBN: 978-0-471-69868-52/1/201225

Qualtrics Demonstrationharvard.qualtrics.com2/1/201226

Harvard Law School cboussalis@law.harvard.edu . Surveys: Not Just Asking People Questions 2/1/2012 2 The purpose of a survey is to produce accurate information that reflects the views and experiences of a given . o E.g. “check