Transcription
Concord Evaluation GroupEvaluation ofCurious GeorgeMAY 2012Contact: Christine Paulsen, Ph.D.cpaulsen@concordevaluation.comPO Box 694 Concord, MA 01742 concordevaluation.com
Table of ContentsBackground. 1Methods and Procedures . 2Study Design . 2Hypotheses and Research Questions . 5Survey Instruments . 7Participants . 9Findings .13Reading Curious George books boosted young children’s science and mathknowledge. .13Curious George books prompted children to use scientific habits of mind whilereading. .15Watching Curious George episodes also boosted young children’s science andmath knowledge. .16Curious George books reinforced the science and math concepts childrenlearned from watching the episodes. .17Curious George books helped parents feel more confident about exploringscience and math with their children. .18Curious George episodes also helped parents feel more confident aboutexploring science and math with their children. .21Curious George resources may inspire young children to read. .23Families enjoyed Curious George, plan to keep using Curious Georgeresources, and would highly recommend Curious George to other families. .26Summary .30AppendicesA: Control Group SurveyB: Book Group SurveyC: Television Survey #1D: Television Survey #2
BackgroundCurious George is a popular children’s character based on theoriginal works of Margret and H. A. Rey, first published byHoughton Mifflin in 1941, about an inquisitive monkey and hisadventures. WGBH Educational Foundation, Universal Studios,and Houghton Mifflin Harcourt have continued the legacy ofCurious George through an animated television series, a bookseries based on the television shows (TV tie-in books), inaddition to other products. According to WGBH, George is thearchetypal preschooler and the perfect vehicle to helppreschoolers learn valuable science, engineering, and mathCurious George skills. WGBH and its partners are interested in studying theeducational impact of the Curious George television series andTV tie-in books.In 2012, Concord Evaluation Group (CEG) was hired to evaluate the impact ofCurious George television episodes and TV tie-in books on their target audience.Specifically, the evaluation was designed to assess the extent to which CuriousGeorge books and television episodes:Page1Helped young children (ages 4-5 years old) to learn science and mathand develop scientific “habits of mind” (e.g., making predictions, askingquestions).Worked together (books and episodes) to reinforce science and mathlearning.Engaged the parents of young children and helped them feel moreconfident about helping their children to learn about science and math.Motivated parents to involve their young children in science and mathactivities at home or in other informal learning environments (likemuseums, parks, etc.).
Methods and ProceduresStudy DesignThe evaluation employed the use of a rigorous research design. In fact, theevaluation included two experimental designs—one to evaluate the impact of theTV tie-in books and one to evaluate the impact of the television episodes. Thestudy design is illustrated in the figures below.CEG recruited study participants using its national panel of families who haveexpressed an interest in previous studies conducted by CEG and by reaching outto preschools and Head Start centers across the country. We provided panelmembers and preschools with information about the study and invited interestedparents to contact us directly to see if their family qualified to participate in thestudy. To qualify for the study, families needed to:Have at least one 4 or 5 year old child living at home,Be able to read and communicate in English, andBe able to play a DVD at home (Television Group only).We did not offer a financial incentive for participation in the study. Instead,families were able to keep the Curious George books and/or episodes as a tokenof gratitude for their participation. Families in the Control Group received apackage of books and a DVD containing Curious George episodes after thestudy ended.Page2From the pool of eligible families, CEG randomly assigned families to one ofthree groups: (1) the Book Group, (2) the Television Group, or (3) the ControlGroup (Figure 1). Random assignment provides rigor to the study designbecause it helps to ensure that the background characteristics of participants aredistributed as fairly and as equally as possible across the groups.1 Equivalencyacross the groups with respect to key variables, like interest in science oreducational level of the parents, enables us to say with confidence that anyobserved differences between two groups with respect to, say, scienceknowledge are likely due to the intervention (reading Curious George books orwatching episodes) and not due to differences in participants’ backgrounds.1Boruch, R.F. (1997). Randomized Experiments for Planning and Evaluation: A PracticalGuide. Thousand Oaks, CA: Sage Publications.
Book GroupNational pool ofeligible families,randomly assigned to1 of 3 groupsControl GroupTelevision GroupFigure 1. The experimental design included random assignment.As illustrated in Figure 2, families that were assigned to the Book Group wereprovided with a set of 4 Curious George books to read over a period of 2-3weeks. We encouraged families to read the books as many times as theywanted. These books (and the science or math concepts they covered) included:Curious George Roller Coaster (measurement)Curious George A Winter’s Nap (hibernation)Curious George Colors Eggs (colors)Curious George Rain or Shine (weather)In fact, 50% of the families in the Book Group reported reading the books severaltimes, 21% reported reading each of the books twice, and the remaining familiesread the books once.After families in the Book Group had an opportunity to read the books, withineach family, the preschool child and one parent completed a survey together toassess their understanding of the science and math constructs that were coveredin the books. Families assigned to the Control Group were not provided withbooks before completing the survey together. The Book Group and ControlGroup surveys contained a set of overlapping questions so that we couldcompare the groups’ understanding of the same science and math constructs aswell as other variables. The surveys, and the constructs they measured, aredescribed in more detail in the next section.Book GroupPage3ControlGroupReadbooksSurveySurveyFigure 2. We compared the survey responses of families that read Curious George booksto survey responses of families that did not read them.
As illustrated in Figure 3, families that were assigned to the Television Groupwere provided with a DVD containing 4 Curious George television episodes.Families were encouraged to watch the episodes as many times as they wantedover a period of 2-3 weeks. These episodes (and the science/math conceptsthey covered) included:Buoy Wonder (buoyancy)Curious George and the Invisible Sound (sound)Curious George, Dog Counter (sorting)Squirrel for a Day (plant life)These episodes included an animated segment (i.e., the Curious Georgecartoon), plus live action “interstitials” that showed “real kids” engaging inscientific explorations that were designed to reinforce the concepts covered inthe animated segments of the episodes. The animated segments averagedabout 11 minutes, while the live action segments ran approximately 2 minutesper episode.58% of the families reported watching the episodes one time, while one-quarterof families (26%) reporting watching the episodes twice. The remaining 16% offamilies watched the episodes several times.After families watched the episodes, the preschool child and his/her parentcompleted a survey together. After completing the survey, families read a set of4 Curious George books together. We encouraged families to read the books asmany times as they wanted over a period of 2-3 weeks. The books were “tied-in”to the television episodes that the families watched at the outset of the study, sothat we could explore the question of whether the books supported or cementedany learning that took place as a result of watching the Curious Georgeepisodes. The 4 books included:Page4Curious George Boat Show (tied in to Buoy Wonder, covered buoyancy)Curious George Finds a Friend (tied in to Curious George and theInvisible Sound, covered sound)Curious George and the Dog Show (tied in to Curious George, DogCounter, covered sorting)Curious George Plants a Seed (tied in to Squirrel for a Day, covered plantlife)Again, families assigned to the Control Group completed a survey that containedquestions which overlapped with the questions in the Television Group survey.But, families in the Control Group did not watch episodes or read books prior tocompleting the survey.
Episode/Book ySurveyFigure 3. We compared the survey responses of families that watched Curious Georgeepisodes to survey responses of families that did not watch them. We also exploredsurvey responses of families that watched the episodes before and after they also read aset of Curious George books.Hypotheses and Research QuestionsBased on the objectives of the evaluation stated earlier, we developed thefollowing hypotheses to be tested in the study:Hypothesis 1: Children who read the Curious George books willdemonstrate significantly better knowledge about the specific science andmath concepts that are covered in the books than children who do notread the Curious George books.Related Research Questions:oCan short-term exposure to Curious George impact children’shabits of mind? Specifically, will parents report that children whoread the Curious George books or watch the Curious Georgeepisodes demonstrated scientific habits of mind (such as askingquestions, making predictions, being curious) while using theresources?Page5Hypothesis 2: Children who watch the Curious George episodes(animated and live action segments) will demonstrate significantly betterknowledge about the specific science and math concepts that arecovered in the episodes than children who do not watch the CuriousGeorge episodes.
Related Research Questions:oooWill learning effects from the Curious George books be additive tothe learning effects from the Curious George episodes? That is,will children’s knowledge scores improve even more after theyread the books?Or, will any learning effects from the episodes remain unchanged?If so, will parents report that the science and math concepts werereinforced by the books?Hypothesis 3: Parents who read the Curious George books with theirchildren will be more likely to report that they know how to engage theiryoung children in STEM activities than parents who did not read theCurious George books with their children. Related to this hypothesis, weexpect that parents who read Curious George books with their childrenwill report greater confidence about engaging in STEM activities afterreading the books.Hypothesis 4: Parents who watch the Curious George episodes withtheir children will be more likely to report that they know how to engagetheir young children in STEM activities than parents who did not watchthe Curious George episodes with their children. Related to thishypothesis, we expect that parents who watch Curious George episodeswith their children will report greater confidence about engaging in STEMactivities after watching the episodes.oRelated Research Question: Will families who use the CuriousGeorge resources report that they are more likely to exploreopportunities for informal science and math exploration thanparents who did not use the Curious George resources with theirchildren?Additional Research Questions:ooPage6oWill children enjoy the Curious George books and episodes?Will parents who read the Curious George books or watched theCurious George episodes be more likely to express an interest indoing more Curious George activities (such as reading books,visiting the website) or purchasing Curious George products in thefuture?Will parents report that they would recommend Curious Georgebooks or episodes to other families?
Note that we did not compare the Book Group to the Television Group on anyvariables. The purpose of the study was to look at the impact of the books andthe impact of the episodes, but not to compare the effects of the two to eachother.Survey InstrumentsWe developed four survey instruments for this evaluation study:1. Control Group Survey (Appendix A)2. Book Group Survey (Appendix B)3. Television Survey #1 (administered after families watched the episodes:Appendix C)4. Television Survey #2 (administered after families read the books:Appendix D)The table below summarizes the questions used to test each of the hypothesesand explore all of the research questions. The table also indicates whichquestions appeared on the different surveys. The surveys are provided inAppendices A-D. Each of the surveys also contained additional items that wereused to provide background or contextual data and were not included in thehypothesis testing.Page7Table 1:Hypotheses, Research Questions, and Corresponding Survey QuestionsHypothesis orResearchQuestionControl GroupSurvey(Appendix A)Hypothesis 1Questions 10-18Hypothesis 2Questions 1-9Hypothesis 3Book GroupSurvey(Appendix B)Questions 1-9,16, 17TelevisionSurvey #1(Appendix C)TelevisionSurvey #2(Appendix D)------Questions 1-9,17Questions 1-9,11, 12Questions 21-23Questions 1214, 16--Question 11Hypothesis 4Questions 21-23--Questions 1214, 16--AdditionalResearchQuestionsQuestions 24-25Questions 18-22Questions 18-20Questions 13-16
Each of the surveys was administered online. Families were provided withcareful instruction about how to access and complete the surveys from home.Due to the young age of the target audience, we instructed parents how to helptheir child click on buttons or type in responses without interfering with thechildren’s answers to the questions. We call this type of survey administration“parent assisted.” Thus, in each family the parent acted merely as a typist for thechild who provided his or her own answers to the questions. The surveys alsocontained questions that were specifically designed for parents themselves. Weprovided written instructions for survey administration to the parents, plus emailfollow-up and, when necessary, telephone support.Page8The survey questions were pilot tested by 10 families prior to conducting theevaluation study to ensure that the survey administration procedures werereliable and that the questions themselves were clear and comprehensible. Wealso used pilot testing data to assess whether the questions were too easy or toodifficult. Pilot testing was iterative—that is, when we made changes to questionsduring pilot testing, we re-tested the new items with other pilot test families toensure that the changes were sufficient.
ParticipantsThe total sample size for the study was 155 families, representing 31 statesacross the country. Approximately half the sample was female (51%). The studyincluded slightly more 4 year olds than 5 year olds (54% versus 46%).The proportion of racial and ethnic groups in our sample mirrors the proportion ofracial and ethnic groups currently living in the U.S. according to the latest U.S.Census, with most reporting they were White or Caucasian (67%), followed byHispanic or Latino (17%), Black or African-American (12%), and Asian (8%).2More than half of the families in our sample (61%) reported that their householdfinancial situation was about the same as the average family. The proportion offamilies that reported their household financial situation was worse than theaverage family was 10% (the proportion of persons below poverty level in theU.S. is currently 14%).The proportion of parents in our sample with a Bachelor’s Degree (32%) wasonly slightly higher than the national average of 28%. The distribution ofdemographic and background characteristics across the three groups issummarized below. There were no statistically significant differences betweenthe Control Group and the Book Group or the Television Group with respect tothese characteristics, thus, it appears that randomization was successful atdistributing key variables across the groups.3Table 2:Participant Demographic and Background CharacteristicsTotal sample sizeGenderGirlBoyAge45Race/ethnicityWhite or CaucasianHispanic or roup5032 (60.4%)21 (39.6%)24 (46.2%)28 (53.8%)23 (46.0%)27 (54.0%79 (51.0%)76 (49.0%)29 (54.7%)24 (45.3%)24 (46.2%)28 (53.8%)31 (62.0%)19 (38.0%)84 (54.2%)71 (45.8%)36 (67.9%)11 (20.8%)43 (82.7%)6 (11.5%)25 (50.0%)10 (20.0%)104 (67.1%)27 (17.4%)TOTALS155U.S. Census Bureau (Jan 17, 2012) State and County QuickFacts. Available tml3We did not compare the differences between the Book Group and the Television Groupbecause that was not the purpose of this study.
ControlGroup5 (9.4%)6 (15.1%)1 (1.9%)Black or African-AmericanAsian or Pacific IslanderNative AmericanHousehold financial situationWorse than average6 (11.3%)Average25 (47.2%)Better than average21 (39.6%)Unknown1 (19.9%)Highest household education levelHigh school graduate or1 (1.9%)lessAssociate degree or6 (11.3%)certificate programSome college credit, but no3 (5.7%)degreeBachelor’s degree12 (22.6%)Master’s degree12 (22.6%)Doctorate degree or13 (24.5%)Professional degreeUnknown6 (11.3%)BookGroup6 (11.5%)3 (5.8%)1 (1.9%)TelevisionGroup7 (14.0%)3 (6.0%)0 (0.0%)18 (11.6%)12 (7.7%)2 (1.3%)3 (5.8%)36 (69.2%)13 (25.0%)0 (0.0%)7 (14.0%)33 (66.0%)10 (20.0%)0 (0.0%)16 (10.3%)94 (60.6%)44 (28.4%)1 ( 1%)1 (1.9%)1 (2.0%)3 (1.9%)10 (19.2%)7 (14.0%)23 (14.8%)4 (7.7%)5 (10.0%)12 (7.7%)24 (46.2%)8 (15.4%)14 (28.0%)17 (34.0%)50 (32.3%)37 (23.9%)4 (7.7%)2 (4.0%)19 (12.3%)1 (1.9%)3 (6.0%)10 (6.6%)TOTALSOur original intent during study recruitment was to only include families with noprior television exposure to Curious George. However, we discovered thatalmost no families reported that they had never seen or heard of CuriousGeorge. So, we amended our recruitment plan to include families with differentlevels of exposure to Curious George on TV. We asked families to report, on ascale from 0 (never) to 2 (a lot), the extent to which their child watched CuriousGeorge. We found that there were no differences in reported exposure toCurious George viewing prior to the study. The average levels of CuriousGeorge exposure are summarized below.Page10Table 3:Exposure to Curious George Prior to the StudyControl Group AverageComparison GroupAverage1.08(standard deviation .70)Book Group 0.94(standard deviation .42)Television Group 1.10(standard deviation .62)StatisticallySignificantDifference?NoNo
We also wanted to ensure that families had an equal predisposition to engagingin STEM (science, technology, engineering, and math) activities across the threegroups. So, we asked parents to indicate, from among the following activities,which were the Top 3 that they would most like to do with their children. TheSTEM-related activities are marked as such:Listen to or play music with your childDraw picturesLook at the stars at night (STEM)Observe insects and animals outdoors (STEM)Play sportsCollect acorns or rocks or other things from nature (STEM)Grow a garden or indoor plants (STEM)Read storiesBuild with blocks (STEM)We gave each parent one point for each STEM-related activity they chose andtotaled the number of STEM activities for each family. Since families could onlychoose up to 3 activities, scores ranged from 0 to 3 points. As summarized inthe table below, there were no significant differences between the groups. Thus,the families in the groups had an equal predisposition to engage in STEMactivities.Table 4:Family Predisposition to Engage in STEM ActivitiesControl Group AverageComparison GroupAverage1.85(standard deviation .89)Book Group 1.65(standard deviation .81)Television Group 1.70(standard deviation inally, we asked parents to indicate the extent to which their children werenaturally curious or enjoyed science or math-related activities such as:CountingAsking questionsArranging toys in different ways, sorting things by color or shape or inother waysSolving problems or doing puzzlesComing up with his or her own ideas about why things work the way theydo
Learning how things workMaking guesses and predictions about thingsBeing curious about thingsParents indicated on a scale of 1 (Strongly Disagree) to 4 (Strongly Agree) theextent to which each of the above characteristics described their children. Wecompared the Control Group to the Book Group and the Television Group toexplore whether there were differences with respect to any of the items listedabove. We found no significant differences between the groups. Since none ofthe individual items was significantly different, the table below reports theaggregate averages for each group rather than each individual item.Page12Table 5:Children’s Predisposition to STEM-Related ThinkingControl Group AverageComparison GroupAverage3.56(standard deviation .36)Book Group 3.59(standard deviation .33)Television Group 3.53(standard deviation .37)StatisticallySignificantDifference?NoNo
FindingsReading Curious George books boosted young children’sscience and math knowledge.The first hypothesis we tested was:Hypothesis 1: Children who read the Curious George books willdemonstrate significantly better knowledge about the specific science andmath concepts that are covered in the books than children who do notread the Curious George books.To test this hypothesis, we compared Book Group to Control Group responseson a set of 9 questions designed to assess children’s knowledge of the 4 specificscience and math concepts covered in the Curious George books:MeasurementHibernationColorsWeatherChildren had the possibility of earning 1 point for each correct response – for apossible total score of 9 points. We compared Book Group Scores to ControlGroup scores to find that the Book Group children demonstrated statisticallygreater knowledge of the specific science and math concepts measuredthan the kids in the Control Group. Children in the Book Group scored anaverage of 6.29 points (standard deviation 1.18) while children in the ControlGroup scored an average of 5.21 points (standard deviation 1.80).4 In additionto finding a statistically significant difference between the groups, the effect sizewas also moderately large (d .71).5,6Page134t (89.780) 3.647, p .000. The “t-test” assesses whether the averages of the two groupsare statistically different from one another. The “p value” associated with the t statistictells us the probability of rejecting the null hypothesis (that the two groups are equal)when the null is actually true (a false positive). In this case, the probability of a falsepositive is very slim: A probability lower than p .05 is considered “statistically significant”in education research.5Effect size is a measure of the strength of the relationship between two variables and isan alternate measure (versus the t-statistic) of the magnitude of that relationship. Someresearchers consider effect size to be more relevant than reporting inferential statisticsand their associated p values because the effect size cannot be influenced by samplesizes (whereas inferential statistics can be).6When comparing the averages of two independent groups, .20 is generally considereda small effect size, .50 is a medium effect size, and .80 is a large effect size [Cohen, J.(1992). A power primer. Psychological Bulletin, 112(1), 155-159.]
Given that the Book Group and the Control Group were equally matched on allkey background and demographic variables, we are confident that thedifferences observed between the groups is attributable to the intervention—thatis, reading Curious George books resulted in a knowledge boost for thechildren in the Book Group.Parents reported:We did a weather journal and my daughter really enjoyed this activity. Itwas great getting her to pay attention to the clouds. We both learned.It was awesome to see her know answers to things and figure things out!“My son learnedwhich animalshibernate (henow sleeps in hissleeping bag inbed like it’s acave.”“I wish I had thiswhen I was alittle girl. Wewere notencouraged toexcel in scienceand math.”My son was excited to measure himself after we read about how Georgeused the candy to measure his height.The books made learning easy and they also encouraged thinking.My daughter talks a lot about the weather now, and if and when it willrain.We watch every day and try to figure out if it's going to be sunny all day orif it might rain. We also try to figure out if it's going to be cold or warm.My daughter learned how to use various household items to measuredifferent things. She also learned about clouds and that there are peoplewho are weather scientists. We never discussed these things before thebooks.My son learned which animals hibernate (he now sleeps in his sleepingbag in bed like it's a cave).I wish I had this when I was a girl. We were not encouraged to excel inscience and math.Page14I hadn't yet exposed my 4-year old much to mixing colors andmeasurement, these books were a great introduction.The weather book was something new for him. We are a VERY curiousfamily and love talking about and learning new things. even thoughthese were things we had talked about at some point, Curious Georgepresented them in a new way, so they were a fun addition and a new wayto look at the topics. we are already wanting more of the books!
Curious George books prompted children to use scientifichabits of mind while reading.Related to the first hypothesis we tested, we explored the following researchquestions:Can short-term exposure to Curious George impact children’s habits ofmind?Specifically, will parents report that children who read the Curious Georgebooks or watch the Curious George episodes demonstrate scientifichabits of mind (such as asking questions, making predictions, beingcurious) while using the resources?Parents in the Book Group and the Television Group reported on the extent towhich their children demonstrated scientific habits of mind while they werereading their respective sets of books. Parents indicated their agreement on ascale of 1 (Strongly disagree) to 4 (Strongly agree) with statements about theirchildren’s behaviors.As indicated in the table below, for each of the behaviors listed, parents were inagreement (most scores were around “3”) that their children demonstrated thesebehaviors while reading the Curious George books. The behaviors observedmost frequently were that children talked about the things they were seeing whilereading the books and tried to make predictions about what would happen next inthe stories. Most children also asked a lot of questions while reading and cameup with theories about why certain things were happening in the stories as theyread them.Table 6:Scientific Habits of Mind Observed During Reading Curious George BooksNAs we read together, my child talked about the thingsshe/he was seeing.As we read together, my child tried to guess what wemight discover/what might happen next.As we read together, my child asked a lot of questions.Page15As we read together, my child came up with ideas aboutwhy certain things were happening.MeanStd Dev1003.50.5951003.24.656993.12.7651013.05.711
Watching Curious George episodes also boosted youngchildren’s science and math knowledge.Our second hypothesis was:Hypothesis 2: Children who watch the Curious George episodes(animated and live action segments) will demonstrate significantly betterknowledge about the specific science and math concepts that arecovered in the episodes than children who do not watch the CuriousGeorge episodes.To test this hypothesis, we compared Television Group to Control Groupresponses on a set of 9 questions designed to assess children’s knowledge ofthe 4 specific science and math concepts covered in the Curious Georgeepisodes:BuoyancySoundSortingPlant lifeChildren had the possibility of earning 1 point for each correct response – for apossible total score of 9 points. We compared Television Group Scores toControl Group scores to find that the Television Group children demonstratedstatistically greater knowledge of the specific science and math conceptsmeasured than the kids in the Control Group. Children in the TelevisionGroup scored an average of 8.04 points (standard deviation 1.09) whilechildren in the Control Group scored an average of 7.43 points (standarddeviation 1.61).7 The ef
read the Curious George books. Related Research Questions: o Can short-term exposure to Curious George impact children's habits of mind? Specifically, will parents report that children who read the Curious George books or watch the Curious George episodes demonstrated scientific habits of mind (such as asking
