Transcription
Reading Street - Final ReportTHE E FF E C T S O FR E AD I N G S T RE E TON READIN G AC H I E V E M E N T:A F OCUS ON S E C O N D Y E A RCURRICU LU M U SE R SFinalRepSubmitted by:CLAREMONT GRADUATE UNIVERSITYInstitute of Organizational and Program Evaluation ResearchSeptember 4, 2009Tiffany Berry, PhDKatherine K ByrdKrista CollinsWith Special Acknowledgement to:ortAdiam condimentumPurus, in consectetuer Proinin sapien. Fusce urnamagna,neque eget lacus.Mae cenas felis nunc,aliquam ac, consequatvitae, feugiat at, blanditvitae, euismod vel, nunc.Maecenas felis nunc,aliquam ac vitae, feugiat at.Aenean ut erat ut nibhcommodo suscipit.Maecenas non quam.Xiaoxia Newton, PhD, University of California, BerkeleyClaremont Graduate University123 East Eighth Street Claremont CA 91711 9096071540
Reading Street - Final ReportCO N TE N T SExecutive Summary .1Acknowledgements .7Introduction .8Methodology .10Power Calculations .10Site Recruitment .10Implementation Measures .11Student Level Measures .13General Study Procedures and Timeline .14Sample Description .16Schools .16Teachers .17Students .17Summary of Study Sample and Comparability between Groups .21Curriculum Implementation . 23Log Completion .23Description of Reading Street Implementation .23Description of Comparison Implementation .32Implementation Comparisons .37Results . 41Question 1: Do Students Exposed to Reading Street Improve Their Reading Scores over the School Year?.41Question 2: Do Reading Street Students’ Reading Scores Vary by Their Background Characteristics? .43Question 3: Do Students Exposed to Reading Street Improve Their Reading Scores More than StudentsExposed to a Different Elementary Reading Curriculum? .45Question 4: How Does Curriculum Implementation Relate to Gains in Students’ Reading Scores? .51Overall Summary for Results .54Conclusions . 55Key Evaluation Findings .55Study Limitations .58References . 59Appendices . 60Appendix A: Study Overview .60Appendix B: Provided Reading Street Materials .62Appendix C: School Demographics .63Appendix D: Treatment Implementation Log .64Appendix E: Implementation Guidelines .68Claremont Graduate University123 East Eighth Street Claremont CA 91711 9096071540
Reading Street - Final ReportCO N TE N T SAppendix F: Curriculum Materials for Treatment.70Appendix G: Content Coverage for Treatment.72Appendix H: Materials and Content for Comparison .73Appendix I: Growth in Reading Achievement by Condition, Grade Level, and Demographic Group .75Appendix J: Variables Explored during HLM Data Analysis .77Appendix K: HLM Final Models (Second Grade) .79Appendix L: HLM Final Models (Fourth Grade).83List of TablesTable 1: Sample Demographic Information .2Table 2: Average Days per Week Met with Small Groups: 2nd Grade.3Table 3: Average Days per Week Covering Content: 2nd Grade.3Table 4: Description of GRADE Assessment .14Table 5: Timeline for Data Collection across Implementation and Outcome Measures .15Table 6: Teacher Demographic Information .17Table 7: Sample Size by Grade Level and Condition .18Table 8: Student Demographic Characteristics for All Students .19Table 9: Student Demographic Characteristics for All Students with a Posttest.20Table 10: Reading Street Implementation Recommendations .24Table 11: Average Days per Week Used Reading Street Materials .26Table 12: Treatment Teachers Who Met or Exceeded Recommended Coverage: 2nd Grade .27Table 13: Average Days per Week Treatment Teachers Covered Recommended Content.28Table 14: Treatment Teachers Who Met or Exceeded Recommended Coverage: 4th Grade .28Table 15: Average Days per Week Treatment Teachers Met in Small Groups .28Table 16: How Often and How Long Treatment Teachers Met in Small Groups .29Table 17: Average Days per Week Comparison Teachers Covered Recommended Content .34Table 18: Average Time per Week Comparison Teachers Met in Small Groups .35Table 19: Average Days per Week Comparison Teachers Met in Small Groups .35Table 20: Average Days per Week Implementing the Literacy Block: Treatment vs Comparison .37Table 21: Average Days per Week Teachers Covered Recommended Content: Treatment vs Comparison .38Table 22: Most and Least Used Materials: Treatment vs Comparison .38Table 23: Average Time per Week Teachers Met in Small Groups: Treatment vs Comparison .39Table 24: Average Days per Week Teachers Met in Small Groups: Treatment vs Comparison.39Table 25: Second Grade Treatment Students’ Reading Gains .42Table 26: Fourth Grade Treatment Students’ Reading Gains .43Table 27: Second and Fourth Grade Comparison Students’ Reading Gains .46Table 28: Description of HLM Variables .48Claremont Graduate University123 East Eighth Street Claremont CA 91711 9096071540
Reading Street - Final ReportCO N TE N T STable 29: Second Grade HLM Summary .49Table 30: Fourth Grade HLM Summary .50Table 31: Correlation Matrix between Achievement and Implementation: Overall Sample .52Table 32: Correlations across Conditions and Grade Levels .53List of FiguresFigure 1: Total Reading Achievement Scores: 2nd Grade .4Figure 2: Total Reading Achievement Scores: 4th Grade .4Figure 3: Treatment Teachers’ Perceptions of Reading Street.25Figure 4: Average Time Treatment Teachers Spent Planning Each Week’s Lesson .25Figure 5: Average Days per Week Treatment Teachers Implemented the Literacy Block.26Figure 6: Treatment Teachers’ Adherence to Differentiated Instruction Guidelines .30Figure 7: Treatment Teachers’ Use of Reading Assessments .30Figure 8: Treatment Teachers’ Use of Progress Monitoring Strategies .31Figure 9: Comparison Teachers’ Curricula .32Figure 10: Comparison Teachers’ Perceptions of Their Curriculum .33Figure 11: Average Time Comparison Teachers Spent Planning Each Week’s Lesson .33Figure 12: Average Days per Week Comparison Teachers Implemented the Literacy Block.34Figure 13: Comparison Teachers’ Use of Reading Assessments .36Figure 14: Comparison Teachers’ Use of Progress Monitoring Strategies .36Figure 15: Days per Week Teachers Implemented the Literacy Block.37Figure 16: Second Grade Treatment Students’ Reading Gains .42Figure 17: Fourth Grade Treatment Students’ Reading Gains .43Figure 18: Treatment Students’ Reading Gains by Gender and Grade Level .44Figure 19: Treatment Students’ Reading Gains by Ethnicity and Grade Level .44Figure 20: Treatment Students’ Reading Gains by Lunch Status and Grade Level .45Figure 21: Differential Reading Gains for Caucasian vs Non-Caucasian Students by Condition .50Claremont Graduate University123 East Eighth Street Claremont CA 91711 9096071540
Reading Street - Final ReportEX E C U T I V E SU MMARYIntroductionReading Street (2008) is an empirically-based, basal reading program for pre-Kindergarten to sixth grade classrooms.Reading Street is thematically organized to emphasize development of priority skills in five core reading areas: phonemicawareness, phonics, fluency, vocabulary, and comprehension. Reading Street is based on Three Tier Instruction which isdesigned to meet the needs of students using primary, secondary, and tertiary intervention (Linan-Thompson & Elbaum,2005).Two previous randomized control trials (RCTs) of Reading Street were conducted by Magnolia Consulting in 2005-2006and 2006-2007 (Wilkerson, Shannon, & Herman, 2006; Wilkerson, Shannon, & Herman, 2007). Results from their studiesindicated that students who used the Reading Street program displayed overall literacy improvements across the school year;nevertheless, treatment students did not academically outperform students who were using a different basal readingproduct. Wilkerson et al. (2007) postulated that one school year was insufficient for Reading Street teachers to feelcomfortable implementing a brand new curriculum with high fidelity.Building upon these previous studies, Claremont Graduate University (CGU) was contracted to conduct a quasiexperimental matched pairs study to test the effectiveness of the Reading Street curriculum during the 2008-2009 academicyear. Schools using Reading Street for at least one year prior to September 2008 were matched to comparison schools onschool demographic variables. This report contains the findings from this year long investigation of the Reading Streetcurriculum.Evaluation Design and MethodsResearch QuestionsThis study was designed to answer four main research questions, each of which was developed collaboratively betweenCGU and Pearson:1. Do students exposed to Reading Street significantly improve their reading achievement over the school year?2. Do Reading Street students’ reading gains vary by their background characteristics (e.g., ethnicity, socioeconomicstatus, grade levels, etc.)?3. Do students exposed to Reading Street improve their reading scores more than students exposed to a differentelementary reading curriculum?4. How does curriculum implementation and teacher variables relate to gains in students’ reading scores?MeasuresThe primary dependent variable in this study, reading achievement, was measured by the Group Reading andDiagnostic Evaluation (GRADE) in both grades 2 and 4. This assessment included composite Normal Curve Equivalent(NCE) scores in vocabulary, reading comprehension, and total reading achievement across both grade levels. Thisassessment is nationally recognized and has strong psychometric properties.Claremont Graduate University1
Reading Street - Final ReportEX E C U T I V E SU MMARYFour implementation measures were used to describe curriculum implementation as well as facilitate interpretation ofthe achievement findings revealed in this study: Monthly implementation logs were developed to track treatment and comparison teachers’ implementation,specifically focusing on content coverage, small group instruction, and use of curricula materials. A teacher survey was designed to capture teachers’ demographic information, professional background, literacyblock structure, and attitudinal indicators of teacher’s classroom environment (school environment, instructionalstrategies employed, parent support, etc.). Structured interviews of treatment and comparison teachers (and classroom observations of all treatment teachers)were conducted to obtain in-depth information about typical teacher practices and environmental factors that may haveinfluenced curriculum implementation.Sample DescriptionThere were a total of 26 participating elementary schools (15 treatment, 11 comparison), 81 teachers (45 treatment,36 comparison), and 1,594 students (901 treatment, 693 comparison). Study schools represented three geographic regions,six states, and 16 school districts in the U.S.TA B L E 1: SAMPLE DEMOGRAPHICINFORMATIONThe majority of teachers were female(93%) and identified as Caucasian (96%).Approximately 63% of teachers obtainedGrade Level2nd and 4thStates6Geographic RegionsNortheast, Midwest, SoutheastSchool Districts16years within their current school district,Schools26 (15 treatment, 11 comparison)These teacher characteristics did not varyTeachers81 (45 treatment, 36 comparison)across condition.Students1,594 (901 treatment, 693 comparison)a Master’s degree, 27% earned at least aBachelor’s Degree, and the remaining 8%had teaching credentials. On average,teachers taught for 16 years overall, 13and 8 years at their current grade level.The majority of students were Caucasian(73.6%), roughly half were male, and60% did not receive free/reduced lunchat school. At fourth grade, the treatment group had a higher proportion of non-Caucasian students (33.6%) than thecomparison group (24.2%). At second grade, a significantly higher proportion of students in the comparison groupreceived free/reduced lunch (42.0%) than the treatment group (29.7%). These estimates remained relatively constant afteraccounting for sample attrition, which was less than 8% of the total sample.Although sample attrition was low, there was differential attrition across groups. In the treatment group, studentswithout a posttest were more likely to be low achieving, male, non-Caucasian, and more economically disadvantaged. InClaremont Graduate University2
Reading Street - Final ReportEX E C U T I V E SU MMARYthe comparison group, there were few significant differences between those with a posttest compared to those missing aposttest.Curriculum ImplementationTreatment teachers implemented Reading Street as intended: they implemented the literacy block daily, met or exceededweekly coverage expectations for each content area (except spelling at the second grade level), met with below-level smallgroups twice as long as on- andTA B L E 2 : AVERAGE DAYS PER WEEK MET WITHS MALL GROUPS - 2N D 74.315 daysOn-Level**2.783.882 daysAbove-Level*2.533.392 daysabove-level groups, and used theTeacher Edition as a daily resource.In general, second grade treatmentteachers were extremely satisfiedwith the curriculum, while fourthgrade teachers felt that there was toomuch material to cover, requiring the*Comparison is significantly higher than treatment, p .05 (2-tailed)pace of instruction to move too**Comparison is significantly higher than treatment, p .001 (2-tailed)quickly.Comparison teachers used a range ofdifferent reading curricula, including research-based basal and non-basal programs. On average, second grade comparisonteachers covered vocabulary, spelling, and phonics an average of 1 day more per week than second grade treatmentteachers. Additionally, second gradecomparison teachers met with small groupsat each ability level 1 day more per weekTABLE 3: AVERAGE DAYS PER WE E KCOVERING CONTENT - 2ND GRA D Ethan second grade treatment teachers.However, both treatment and comparisonteachers at each grade level met or exceededReading Street implementationrecommendations for the same content areasand for the on- and above-level small groups.Overall, treatment and comparisonteachers implemented their curriculasimilarly, as most comparison teachers usedresearch-based curricula that emphasizedcontent coverage similar to Reading Street.However, second grade comparison ency*3.394.192Listening and2.473.492Phonemic *covered content and met with small groups*Comparison is significantly higher than treatment, p .05 (2-tailed)more days per week than second grade**Comparison is significantly higher than treatment, p .01 (2-tailed)treatment teachers.Claremont Graduate University3
Reading Street - Final ReportEX E C U T I V E SU MMARYResultsYes, Reading Street students significantly increased their reading achievementover one academic year. Second grade students improved, on average, 20NCE points from pretest to posttest across all three subtests (vocabulary d .28, comprehension d 1.0, total test d .91). Fourth grade treatmentDo students exposed to ReadingStreet significantly improvetheir reading achievement overthe school year?students improved, on average, 22 NCE points from pretest to posttestacross all three subtests (vocabulary d .52, comprehension d .62, totaltest d .67). By the end of the school year, second grade students wereperforming at close to a fourth grade reading level (Grade Equivalent 3.81) and fourth grade students were performing at nearly a seventh gradelevel (Grade Equivalent 7.04).Yes, in general, students across all demographic subgroups (ethnicity, gradelevel, lunch status, and gender) significantly improved their readingachievement from pretest to posttest on the GRADE by one grade level inDo Reading Street students’reading gains vary by theirthe second grade and by two grade levels in the fourth grade. Descriptively,background characteristics?showed large gains in reading over the school year. These findings suggeststudents who were male, non-Caucasian, or received free or reduced lunchthat the Reading Street curriculum may be particularly advantageous forstudents who are male, from minority backgrounds, and who areeconomically underprivileged.No, as indicated by the HLM analyses across grade level and subtest, thereDo students exposed to ReadingStreet improve their readingwas no overall difference between treatment and comparison students’reading achievement. Reading Street students at second and fourth gradescores more than studentsexposed to a differentelementary reading curriculum?performed comparably to grade level students in the comparison groupsacross vocabulary, comprehension, and total reading achievement subtests(see Figures 1 and tmentPosttestComparisonClaremont Graduate UniversityFigure 2: Total Reading Achievement Scores: Fourth GradeNormal Curve Equivalent (NCE) ScoreNormal Curve Equivalent (NCE) ScoreFigure 1: Total Reading Achievement Scores: Second eatmentPosttestComparison4
Reading Street - Final ReportEX E C U T I V E SU MMARYHowever, there was a significant cross-level interaction between condition and ethnicity at second grade. Specifically,the non-Caucasian to Caucasian gap in the comparison group was much bigger at the end of the school year than thenon-Caucasian to Caucasian gap in the treatment group, everything else being equal. This suggests that for second grade,Reading Street may be particularly beneficial for non-Caucasian students and may help close the achievement gap for thesestudents.Curriculum implementation was systematically related to changes in students’How does curriculumimplementation andteacher variables relate togains in students’ readingscores?reading scores. For content coverage, phonics was significantly related to growth inreading comprehension and total reading achievement (particularly in thetreatment group). The number of days teachers covered phonemic awareness wasnegatively related to students’ reading achievement in the comparison group. Thisnegative correlation was likely the result of teachers spending extensive timecovering phonemic awareness with low achieving students.For small groups, the number of days spent working in small groups was significantly correlated with students’ growthin reading achievement, however, this correlation was only significant in the comparison group and in fourth gradeclassrooms.For use of materials, using leveled readers was significantly correlated to students’ class growth in readingachievement, however, these correlations only remained significant in comparison classes and second grade classes.However, in treatment classrooms, the more days teachers reported using the Teacher’s Edition, the greater the growth instudents’ total reading achievement (although this result was marginally significant).ConclusionsBased on the evaluation findings, six notable themes emerged from the data:Sample Comparable Across Schools, Not StudentsOverall, treatment and comparison groups appeared comparable at the school and teacher level; however, there weresome demographic differences at the student level across grade level and condition. Also, more low performing studentsleft the treatment than the comparison group. Thus, while schools were well matched at the onset of the study, variation instudent characteristics later emerged as a distinguishing factor between treatment and comparison groups.Moderate to Strong Curriculum Implementation Fidelity by Treatment TeachersIn general, treatment teachers implemented the Reading Street curriculum as intended. As recommended by ReadingStreet, teachers met with below-level groups twice as long as on- and above-level groups each week, although they met withbelow-level groups less than five days per week. Although fidelity to content coverage was high, teachers struggled to meetdaily expectations with Reading Street recommended content, likely because of the sheer amount of material teachersneeded to cover with students.Teachers were highly satisfied with the organization of the curriculum, especially how important reading skills wereinterwoven throughout all the materials at appropriate levels for each reader. Second grade teachers were more satisfiedClaremont Graduate University5
Reading Street - Final ReportEX E C U T I V E SU MMARYwith Reading Street than fourth grade teachers, but across grade levels most teachers felt they were able to meet theirstudents’ needs most of the time.Similar Use of Materials, Content Coverage, and Small Group Instruction in Treatment andComparison ClassroomsAlthough treatment and comparison teachers had access to different curricular resources, treatment and comparisonteachers used their Teacher Edition, Student Edition, and leveled readers most often, and both used CDs the least often.Also, although both treatment and comparison teachers used Student Editions at similar frequencies, treatment teachersused Teacher Editions (or similar teacher’s manual) more often than comparison teachers.Content coverage was comparable for treatment and comparison teachers at both the second and fourth grades,except in a few instances. While there was no difference in content coverage at fourth grade between treatment andcomparison teachers, comparison teachers at the second grade level covered vocabulary, spelling, and phonics 1 day moreper week than treatment teachers.Treatment teachers met with below-level students more frequently than on-level or above-level students, althoughcomparison teachers met more frequently with below-level students overall. Based on our classroom observati
Reading Street (2008) is an empirically-based, basal reading program for pre-Kindergarten to sixth grade classrooms. Reading Street is thematically organized to emphasize development of priority skills in five core reading areas: phonemic awareness, phonics, fluency, vocabulary, and comprehension.
