WIXLIB

NCEE 2012-4009U . S . D E PA R T M E N T O F E D U C AT I O NEffectiveness of a Programto Accelerate VocabularyDevelopment in Kindergarten(VOCAB): First GradeFollow-up Impact Reportand Exploratory Analyses ofKindergarten Impacts

Effectiveness of a Program to Accelerate VocabularyDevelopment in Kindergarten (VOCAB): First GradeFollow-up Impact Report and Exploratory Analyses ofKindergarten ImpactsFinal ReportDecember 2011Authors:Barbara Goodson, Principal InvestigatorAbt Associates Inc.Anne Wolf, Director of EvaluationAbt Associates Inc.Steve Bell, Task 2 Methodological LeaderAbt Associates Inc.Herb Turner, Technical ConsultantANALYTICAPamela B. Finney, Research Management LeaderRegional Education Laboratory SoutheastProject Officer:Sandra GarciaInstitute of Education SciencesNCEE 2012–4009U.S. Department of Education

U.S. Department of EducationArne DuncanSecretaryInstitute of Education SciencesJohn Q. EastonDirectorNational Center for Education Evaluation and Regional AssistanceRebecca A. MaynardCommissionerDecember 2011This report was prepared for the National Center for Education Evaluation and RegionalAssistance, Institute of Education Sciences, under contract ED–06-CO–0028.IES evaluation reports present objective information on the conditions of implementation andimpacts of the programs being evaluated. IES evaluation reports do not include conclusions orrecommendations or views with regard to actions policymakers or practitioners should take inlight of the findings in the report.This report is in the public domain. Authorization to reproduce it in whole or in part is granted.While permission to reprint this publication is not necessary, the citation should read: Goodson,B., Wolf, A., Bell, S., Turner, H., & Finney, P. B. (2011). Effectiveness of a Program toAccelerate Vocabulary Development in Kindergarten (VOCAB): First Grade Follow-up ImpactReport and Exploratory Analyses of Kindergarten Impacts (NCEE 2012–4009). Washington,DC: National Center for Education Evaluation and Regional Assistance, Institute of EducationSciences, U.S. Department of Education.This report is available on the Institute of Education Sciences website at http://ncee.ed.gov andthe Regional Educational Laboratory Program website at http://edlabs.ed.gov.Alternate Formats. Upon request, this report is available in alternate formats, such as Braille,large print, audiotape, or computer diskette. For more information, please contact theDepartment’s Alternate Format Center at 202-260-9895 or 202-205-8113.

DISCLOSURE OF POTENTIAL CONFLICT OF INTEREST1None of the authors or other staff involved in the study from the Regional EducationalLaboratory Southeast at the SERVE Center at the University of North Carolina atGreensboro, Abt Associates Inc., ANALYTICA, or Empirical Education Inc. hasfinancial interests that could be affected by the content of this report.1Contractors carrying out research and evaluation projects for IES frequently need to obtain expert adviceand technical assistance from individuals and entities whose other professional work may not be entirelyindependent of or separable from the tasks they are carrying out for the IES contractor. Contractorsendeavor not to put such individuals or entities in positions in which they could bias the analysis andreporting of results, and their potential conflicts of interest are disclosed.iii

ACKNOWLEDGEMENTSMany individuals and organizations contributed to the evaluation of the K-PAVE vocabularyintervention. We want to thank Dr. Hank Bounds, former State Superintendent of Education forMississippi, and the other staff in the Mississippi Department of Education for their support inallowing us to conduct a random assignment evaluation and for communicating with districtsuperintendents and school principals to facilitate recruitment of districts and schools for thestudy. In addition, we greatly appreciate the many school district superintendents, schoolprincipals, and teachers who were willing to open their classrooms and share their time with us.Many people from the SERVE Center at the University of North Carolina at Greensboro, theprime contractor for the Regional Educational Laboratory-Southeast that funded the study, werecritical to the success of the project. Frankie Walton White, Alexa Edwards, Melissa Williams,Fain Barker, Lynn Amwake, Lucy Wynn, De Fuller, Glyn Brown, Teri White, Beth Thrift, andTeresa Lewis recruited participants, coordinated the intervention training and materials, traineddata collectors, collected data, and identified first grade students for the follow up study. DavidSchaefer was critical in the document review and editing process.In addition, we want to thank the staff from Empirical Education Inc. for their accurate databaseentry of student assessments and classroom observations.Staff from Abt Associates was invaluable to many aspects of the project, including training datacollectors and assisting with impact analyses. Carter Epstein developed rigorous trainingsessions for child assessors and classroom observers, and she lead a team of excellent trainers,including Kenyon Maree, Rachel McCormick, Cay Bradley, Alyssa Rulf Fountain, MeganTiano, Laurie Bozzi, and Rachel Luck. Carolyn Layzer assisted with pilot testing of datacollection instruments. Amanda Parsad was a crucial member of the impact analysis team.Thanks also go to Amanda Parsad, David Cook, and Tom McCall for their assistance inpreparing the restricted-use data release.iv

CONTENTSDISCLOSURE OF POTENTIAL CONFLICT OF INTEREST . IIIACKNOWLEDGEMENTS . IVSUMMARY . 11. INTRODUCTION AND STUDY OVERVIEW . 4ROLE OF VOCABULARY KNOWLEDGE IN READING COMPREHENSION .4THE K-PAVE VOCABULARY INSTRUCTION PROGRAM .5WHY K-PAVE? .6THEORY OF CHANGE FOR K-PAVE .7EXPERIMENTAL AND QUASI-EXPERIMENTAL EVIDENCE OF IMPACTS OF VOCABULARY INSTRUCTION .8THE FOCUS ON THE MISSISSIPPI DELTA REGION .12PRIOR FINDINGS AND RESEARCH QUESTIONS .13Results of the kindergarten study .14Analysis of sustained impacts on students in grade 1 .15Exploratory analysis of different impacts on subgroups of students and schools .16Exploratory analysis of kindergarten impacts on other outcomes .17ORGANIZATION OF THE REPORT .182. STUDY DESIGN AND METHODOLOGY .19SAMPLE RECRUITMENT AND RANDOM ASSIGNMENT .20Recruitment of eligible districts, schools, and teachers .20Random assignment within blocks .23Random selection of classrooms .24Random selection of students .26Characteristics of schools in final study sample.27ATTRITION AND ANALYTIC SAMPLE IN THE KINDERGARTEN STUDY .30Attrition of schools and classrooms .30Attrition of students.31Nonparticipants and crossovers during the kindergarten intervention year .33ATTRITION AND ANALYTIC SAMPLE IN THE GRADE 1 FOLLOW-UP STUDY .33MEASURES AND DATA COLLECTION .36Student assessments .36Student characteristics and school characteristics (as covariates) .41DESCRIPTION OF ANALYTIC SAMPLE .43Characteristics of teachers and students in intervention and control schools .44Student outcomes at baseline for intervention and control schools.45DATA ANALYSIS METHODS .47Confirmatory analysis .47Exploratory analysis of subgroup differences in impacts of K-PAVE on students.523. CONFIRMATORY ANALYSIS OF IMPACTS ON STUDENTS ONE YEAR AFTER K-PAVE INTERVENTION.57IMPACTS ON STUDENTS’ EXPRESSIVE VOCABULARY ONE YEAR AFTER INTERVENTION .57IMPACTS ON STUDENTS’ ACADEMIC KNOWLEDGE AND PASSAGE COMPREHENSION ONE YEAR AFTER INTERVENTION .584. EXPLORATORY ANALYSES OF DIFFERENCES IN IMPACTS OF K-PAVE FOR SUBGROUPS OF STUDENTS ANDSCHOOLS .61v

ANALYSES OF IMPACT DIFFERENCES FOR SUBGROUPS OF STUDENTS .62Student gender .62Kindergarten pretest score .64ANALYSES OF IMPACT DIFFERENCES FOR SUBGROUPS OF SCHOOLS .66Expressive vocabulary.68Academic knowledge .68Listening comprehension.68Passage comprehension .695. SUMMARY OF FINDINGS AND STUDY LIMITATIONS .70EFFECT OF K-PAVE ON READING-RELATED OUTCOMES IN GRADE 1 .70DIFFERENCES IN EFFECTS OF K-PAVE FOR SUBGROUPS OF STUDENTS AND SCHOOLS .70STUDY PARAMETERS .70STUDY LIMITATIONS .71APPENDIX A. K-PAVE MATERIALS PROVIDED TO TEACHERS .72APPENDIX B. SAMPLE WEEKLY UNIT FROM K-PAVE PROGRAM .73TRANSPORTATION .73Books.73Quick Definitions .73CAR TALK.74On the Go .74From Kalamazoo to Timbuktu.74PAIRED WORDS FOR TRANSPORTATION .74EXTENSION ACTIVITIES .75Activity 1: Sorting Transportation by Characteristics.75Activity 2: How would you get there from here?.76APPENDIX C. LIST OF K-PAVE TARGET WORDS .77APPENDIX D. MISSISSIPPI COUNTIES WITH STUDY SCHOOLS .79APPENDIX E. EXPLORATORY ANALYSES OF KINDERGARTEN IMPACTS ON COMPONENTS OF CLASSROOMVOCABULARY AND COMPREHENSION SUPPORT, STUDENT LEXICAL DIVERSITY, AND TEACHER LEXICAL DIVERSITY80COMPONENTS OF CLASSROOM VOCABULARY AND COMPREHENSION SUPPORT .80MEASURES AND DATA COLLECTION .81Training of data collectors .83Data collection response rates for classroom vocabulary and comprehension support measures .84Vocabulary and comprehension support at baseline for intervention and control schools.84DATA ANALYSIS METHODS .85Analytic model .85Statistical power .88Handling missing data .88SENSITIVITY ANALYSES .88IMPACTS OF K-PAVE ON COMPONENTS OF VOCABULARY AND COMPREHENSION SUPPORT AT END OF INTERVENTION YEAR(KINDERGARTEN).88ALTERNATIVE MEASURES OF STUDENT VOCABULARY DEVELOPMENT AND CLASSROOM VOCABULARY SUPPORT .90Measure of student lexical diversity .90Measure of teacher lexical diversity .91vi

Data collection response rates for student and teacher lexical diversity .91Student and teacher lexical diversity at baseline for intervention and control schools .92DATA ANALYSIS METHODS .93Student lexical diversity .93Teacher lexical diversity .93Statistical power .93Handling missing data .93Sensitivity analyses .94APPENDIX F. STATISTICAL POWER ANALYSIS .97STATISTICAL POWER FOR DETECTING IMPACTS ON STUDENTS .97ACTUAL MINIMUM DETECTABLE EFFECT SIZES FOR STUDENT OUTCOMES AT END OF KINDERGARTEN .99ESTIMATED MINIMUM DETECTABLE EFFECT SIZES FOR STUDENT .101OUTCOMES AT END OF GRADE 1 .101ACTUAL MINIMUM DETECTABLE EFFECT SIZES FOR STUDENT OUTCOMES AT END OF GRADE 1 .102ESTIMATED MINIMUM DETECTABLE EFFECT SIZES FOR EXPLORATORY ANALYSES .103Variation in impacts for subgroups of students and schools .103Impacts on students’ lexical diversity .107Impacts on teachers’ lexical diversity .107Impacts on components of vocabulary and comprehension support in the classroom .108ACTUAL MINIMUM DETECTABLE EFFECT SIZES FOR EXPLORATORY ANALYSES .109APPENDIX G. RANDOM ASSIGNMENT . 111MATCHING OF SCHOOLS WITHIN BLOCKS FOR RANDOM ASSIGNMENT.111PROCESS OF RANDOM ASSIGNMENT: SEQUENCE GENERATION .112CONCEALMENT OF ALLOCATION .114APPENDIX H. RECRUITMENT AND RANDOM SELECTION OF THE STUDENT SAMPLE . 116APPENDIX I. COMPARISON OF STUDENTS MISSING AND NOT MISSING BASELINE ASSESSMENT. 117APPENDIX J. MODEL SPECIFICATIONS . 118THREE-LEVEL MODEL USED TO ESTIMATE OVERALL IMPACTS ON STUDENTS .118THREE-LEVEL MODEL USED TO ESTIMATE DIFFERENCES IN IMPACTS FOR SUBGROUPS OF STUDENTS.122THREE-LEVEL MODEL USED TO ESTIMATE DIFFERENCES IN IMPACTS IN STUDENT OUTCOMES FOR SUBGROUPS OF SCHOOLS .125TWO-LEVEL MODEL USED TO ESTIMATE IMPACTS ON CLASSROOM INSTRUCTION AT THE END OF INTERVENTION YEAR(KINDERGARTEN) .126APPENDIX K. SENSITIVITY ANALYSES . 129CONFIRMATORY ANALYSIS OF IMPACTS ON STUDENTS ONE YEAR AFTER INTERVENTION .129Students’ expressive vocabulary at follow-up .129Students’ academic knowledge and passage comprehension at follow-up .130EXPLORATORY ANALYSES OF IMPACTS AT THE END OF THE INTERVENTION YEAR .135Students’ lexical diversity in kindergarten .135Teachers’ lexical diversity at end of intervention year .136Components of classroom vocabulary and comprehension support at end of intervention .138APPENDIX L. IMPUTATION OF MISSING DATA . 141DUMMY VARIABLE ADJUSTMENT FOR MISSING COVARIATES .141vii