WIXLIB

DIFFERENT SKILLSvein did not identify any significant effects of ELL-specific professional development, perhaps dueto imperfect teacher implementation (Short et al., 2011). Other work relies on school-levelaggregates of teacher characteristics, rather than teacher-level data, compounding the likelihood ofbias due to differential sorting of students, by ability, into schools (Williams et al., 2005). In additionto methodological limitations, empirical research in this area has tended to rely on tests of languageproficiency as outcomes, rather than students’ academic achievement gains. Math outcomes inparticular have received scant attention (Genesee et al., 2006). Finally, the extant empirical researchdoes not typically examine whether ELL-specific skills yield differential benefits for ELLs, or insteadare universally effective instructional practices that would benefit all students equally.Other research suggests the possibility that more foundational teacher characteristics, such assecond-language fluency, motivation, or cultural affiliation, might be differentially relevant in ELLinstruction. The available research regarding these characteristics primarily consists of case studies ofeffective second-language teachers, or of survey data on teachers’ perspectives about teaching thatare linked to student outcomes (Dixon et al., 2012). These studies suggest that some knowledge of asecond language may aid in instruction, and that motivation to teach second-language learners mayalso be beneficial, though the evidence in support of either contention is limited. Another line ofresearch has examined how innate teacher characteristics, such as race or gender, influenceeffectiveness differentially for certain students, for example through positive “same-race” effects(Dee, 2005; Wayne & Youngs, 2003). However, there are no comparable studies explicitly linkingteacher characteristics such as cultural or race affiliation to ELL achievement gains.Investigating differential teacher effectiveness with ELL students. The results of the priorliterature on teacher effectiveness generate three suppositions concerning the effectiveness ofteachers of English language learners. First, we might expect that characteristics associated with6

DIFFERENT SKILLSmore effective teachers for students in general - such as test performance, content knowledge andteaching experience - would also be associated with more effective teachers for ELL students.Second, specific instructional skills and strategies that teachers can learn from training or practicemay support differential effectiveness with ELL students, relative to non-ELL students, in theirmath instruction. Third and finally, in addition to general aspects of teacher quality, somefoundational teacher attributes - such as second language proficiency, motivation to teach ELLs, andrelevant cultural affinity - might differentially benefit math instruction for English learners.In this study we use an unusually rich data set to examine these suppositions. We improve uponexisting studies by including a greater range of teacher characteristics, such as rich survey items thatidentify the quantity and quality of the ELL-related training that individual teachers received; byusing student and teacher-level longitudinal data rather than school-level aggregates and crosssectional data; and by employing analytical methods that, while imperfect, more directly addressconcerns of omitted variables bias than do prior studies.Data and MethodsThis study explores several research questions that bear on the teaching of English languagelearners. In keeping with the discussion above, we assesses the extent to which teachercharacteristics that predict math achievement growth for non-ELL students, such as years ofexperience and test performance, also predict math achievement growth for ELL students. We alsoassess whether there are other characteristics of teachers, including second language fluency andlearning experiences focused on ELL students, that differentially predict math achievement growthfor ELL students more than for non-ELL students. We specifically consider the following questionsof interest:7

DIFFERENT SKILLS1. Do teacher characteristics that predict math achievement growth for non-ELL students alsopredict math achievement growth for ELL students?a. Does teachers' own test performance predict ELL and non-ELL math achievement gains?b. Does teachers' teaching experience predict ELL and non-ELL math achievement gains?2. Do specific teacher experiences that support learning to teach ELL students differentially predicteffectiveness with ELL students in math instruction?a. Does past teaching experience with ELL students predict differential efficacy teaching ELLs?b. Does pre-service teacher preparation that addresses specific instructional skills for teachingELLs predict differential efficacy in teaching ELLs?c. Does in-service teacher professional development that addresses specific instructional skillsfor teaching ELLs predict differential efficacy in teaching ELLs?d. Does certification to teach English as a Second Language, certification as a bilingualEducation teacher, or teacher preparation via an alternative pathway predict differentialefficacy in teaching ELLs?3. Do teacher background characteristics predict differential effectiveness with ELL students in mathinstruction?a. Does a teacher’s ability to speak Spanish fluently, either native or learned, predict differentialefficacy in teaching Hispanic English language learner students?b. Does a teacher’s preference for teaching at sites with more ELLs predict differential efficacyin teaching ELLs?In the following sections, we detail the data that we use to answer these questions, and providedescriptive information about the population of study. We then describe our methodology in depth.8

DIFFERENT SKILLSDataContext and overview of the study sample. The data for this study come from the New YorkCity (NYC) public school system, which includes a large and diverse population of ELLs. We utilizerich administrative information about teachers and students in NYC, as well as unique survey dataabout teachers’ preparation experiences, to predict differences in ELL students’ math achievementgains in grades four through eight. Specifically, the New York City Department of Education(NYCDOE) provided student and teacher demographic data files and a student exam data file foreach school year from 2001-2002 through 2007-2008 for the study. To further enrich our data aboutteachers, we also match NYC teachers to employee data from New York State EducationDepartment (NYSED) databases. Finally, we leverage a survey of all first year NYC teachers,conducted in the spring of 2005, which asked teachers detailed questions about their pre-servicepreparation experiences, in-service training in their first year of teaching, teaching practices, andpreferences.Limiting our analysis to math teachers and outcomes. Although we have access to someEnglish language arts (ELA) achievement data, we ultimately chose to examine and report only onmath outcomes in this study, due to limitations in the available ELA data. First, ELL-designatedstudents were not consistently tested in ELA in NYC during much of this period (2001 to 2008).Towards the end of the period, testing requirements were reformed with more inclusive mandatesfor ELL student testing. For example, starting in 2007, the ELA exam was required after one year inthe district for ELL students, rather than after three years as had been the practice before. Thus, formuch of the study period, ELLs who were tested in ELA were likely quite dissimilar from those whowere not. In addition, during this time period, New York administered ELA exams, unlike mathexams, at mid-year, rather than near the end of the school year. Even with these limitations we dorun similar analyses using ELA scores to those presented in this study about math performance. In9

DIFFERENT SKILLSgeneral, the results are directionally similar, but markedly attenuated. We would be happy to sharethese results or a summary of findings upon request.Administrative data on students, teachers, and schools. Our administrative data include arange of demographic characteristics about students, teachers, and students’ classroom peers. Inaddition, the combination of NYC and NYCDOE databases yields data on teachers’ certificationstatus, their performance on licensing exams, and their experience level within the NYC schoolsystem.Our primary student outcome measures consist of annual student achievement scores in math,based on exams given in third through eighth grades to most NYC students. For each year, the datainclude scores for approximately 65,000 to 80,000 NYC students in each grade. Using these data, weconstruct a set of records with a student’s current math exam score standardized within each gradelevel and year, as well as his or her lagged exam score. We then match individual students andteachers across the district using a process recommended by the NYCDOE.Identifying ELL students. In NYC, the vast majority (98 percent) of students are initiallydesignated as ELLs or non-ELLs based on a home survey that determines whether English is theprimary language spoken at home, followed by a Language Assessment Battery (LAB) examadministered to students whose home language was not English, in order to determine their level ofEnglish proficiency. Our data include an indicator for ELL status in each year in which a studentwas present in the data set. The district was unable to provide the language proficiency scores thatstudents received on their initial LAB exam, or on subsequent exams that re-assess their status. Ourdata also do not specify which ELL-specific programs students may have participated in.Survey of first-year teachers. In addition to utilizing administrative data from 2001-2002 to2007-2008, we also conducted a survey of all first year NYC teachers in the spring of 2005 whichasked detailed questions about teacher preparation experiences, in-service training in their first year10

DIFFERENT SKILLSof teaching, teaching practices, and preferences. Of particular interest for this analysis, the surveyincluded questions related to the quantity and efficacy of teachers’ ELL-specific pre-service and inservice training, as well as their reported preferences to teach at school sites with more ELLstudents. It also asked teachers about their fluency in languages other than English, includingSpanish. The overall response rate for this survey was 71.5 percent, representing 4303 teachersacross all grades and subjects. Of these, 1221 were math teachers present in our administrative data.Descriptive information about students and teachers. We describe the characteristics anddistribution of ELL students across the district in Tables 1 and 2. In line with national trends in ELLperformance, district-wide ELL math achievement is substantially lower than that of non-ELLs inNYC. The bulk of ELL students (70 percent) are classified as Hispanic, while the second largestsubgroup is those of Asian descent (18 percent). In comparison, 35 percent of non-ELLs areHispanic and 13 percent are Asian. ELLs are more likely to receive free or reduced price lunch (76percent) than non-ELLs (68 percent).The majority of ELLs in NYC (80 percent) attend math classrooms composed of both ELL andnon-ELL peer students. Table 3 details the proportion of classrooms district-wide that contain none,minority, and majority populations of ELL students, and the demographic composition of thoseclassrooms. Forty-four percent of all ELLs attend classrooms that are predominantly (94 percent ormore) ELL students. In these classrooms, students are on average 78 percent Hispanic and 73percent eligible for subsidized lunch.While we do not find substantial differences in teachers of ELLs and non-ELLs in terms ofcharacteristics associated with generic teacher quality, we do see key differences in the assignment ofteachers with ELL-specific training or language skills. Table 4 shows that, on average, 25.7 percentof non-ELL students in grades 4-8 were taught by a first or second year teacher, compared to 23.7percent for ELL students. Liberal Arts and Sciences Test (LAST) scores, New York's general11

DIFFERENT SKILLSknowledge certification exam, were lower for the teachers of ELL students, 237 on average,compared to 246 for non-ELL students. However, ELL students are more likely to be taught by amath teacher who is certified as an ESL or bilingual education instructor, with 6.6 percent of ELLsreceiving math instruction from an ESL-certified teacher, and 21.7 percent from a bilingualeducation certified teacher, in comparison to 1.6 and 4 percent, respectively, for non-ELLs. Finally,within our sample of surveyed first year teachers, there is a clear tendency for Hispanic ELLs to betaught by teachers who are fluent in Spanish compared to other students (38.8 percent versus 11.7percent).Table 5 describes the questions of interest from our survey of new teachers that addressed ELLsand provides the distribution of teacher responses for each question. In line with previous researchon teacher preparation to teach ELLs (Boyd et. al., 2009), the proportion of teachers reportingtraining to teach ELLs is low relative to the proportion likely to be supporting ELLs in theirclassrooms. Only 14.1 percent reported a meaningful pre-service opportunity to learn instructionalstrategies for teaching ELLs, and only 13.9 percent reported having more than nine hours of inservice professional development focused on ELL instruction by the middle of their first year. Notethat this survey occurred during a period of substantial ELL-related professional development andcoaching across NYC, particularly in elementary and middle school grades (Horwitz et al., 2009).MethodsThis section details the empirical approach we use to answer each of our research questions. Inparticular, we address two primary challenges to generating accurate results. First, teachers with acharacteristic that affects student learning may also have another set of characteristics thatindependently affect student learning. For example, teachers who speak Spanish may also havegreater overall academic ability. If we do not adjust for this difference, we might attribute to Spanish12

DIFFERENT SKILLSfluency what is really the effect of academic ability. Second, teachers with given characteristics mayteach students with different propensities to learn. For example, if teachers who speak Spanish aredifferentially assigned to students with more learning difficulties, we might see lower gains in thoseclassrooms even if teachers who speak Spanish are more effective. Under an ideal scenario, wewould test effects of teacher characteristics using a randomized experiment. Specifically, we wouldrandomly assign relevant skills (or training) to teachers to ensure comparable underlying teacherability across treatment and control teachers, and we would randomly assign students to teachers toensure comparable student ability across teacher groups.We are unable to conduct an ideal experiment, but use both rich data and two empiricalapproaches to reduce the likelihood of bias. First, we reduce the potential that we are attributingeffectiveness to one teacher characteristic when it is really a correlated teacher characteristic drivingthe association by including theoretically appropriate controls in the models. Second, we reduce thebias associated with the sorting of teachers to schools by comparing the achievement gains ofstudents within the same school but in classrooms with teachers who have different characteristics.Third, we reduce the bias associated with students (both ELL and non-ELL) being assigneddifferentially to teachers within a school by comparing the gains of ELL and non-ELL studentstaught by the same teacher. We describe these methods in more detail below.Model 1: A within-school achievement growth model. As a baseline, we consider howstudent achievement outcomes in math vary across teachers with different characteristics bycomparing the math achievement gains of students within the same school. Equation 1 describesthis specification.Here, the standardized achievement (A) of student i in year t with teacher j in school s is afunction of his or her prior achievement (A at t-1), time varying and fixed student characteristics (X),13

DIFFERENT SKILLScharacteristics of the classroom (C), characteristics of the teacher (T), a fixed-effect for the school( ), a fixed-effect for the grade level of the student ( ), a fixed effect for the year ( ), and a randomerror term ( ). When controlling for prior achievement, we include both a linear and quadratic termto represent the student’s standardized prior achievement result. Also at the student level, we includeobservable characteristics that tend to predict differential achievement, including race and ethnicity,gender, eligibility for free or reduced-price lunch, the number of school absences in the previousyear, and the number of suspensions in the previous year. At the classroom level, we control forpotentially relevant peer effects by including the average of all the student characteristics alreadymentioned, as well as the percent of students in the classroom that are designated ELLs, and themean and standard deviation of student test scores in the prior year. At the teacher level, we includeobservable teacher characteristics that tend to be associated with instructional efficacy, includingyears of teaching experience in NYC (as a proxy for total years of experience), teacherrace/ethnicity, and teacher test scores on the Liberal Arts and Science Test (LAST) generalknowledge exam that teachers must pass to earn certification. In addition to these generic teacherlevel controls, we examine, in separate models detailed below, various teacher characteristics relevantto our research questions. Including fixed effects at the school level reduces the potential biasassociated with teacher assignment to schools, and our controls for student, classroom, and teachercharacteristics reduce potential bias associated with student assignment to teachers within schools.Differential effectiveness with ELLs. In order to assess whether the teacher characteristics inquestion predict differential effectiveness for ELL students, we model, as specified in Equation 1,the effects of ELL and non-ELL characteristics separately, across a sample of students of only thoseteachers who teach mixed classes of ELLs and non-ELLs in a given year, grade, and school. Wethen conduct an F-test on the coefficients of interest to see whether the effect size associated withELL students is significantly different from the effect associated with non-ELL students.14

DIFFERENT SKILLSModel 2: Within-teacher differential effectiveness with ELLs. While

specific set of skills - who are relatively more effective at teaching English learners. If a good teacher is a good teacher for all then the most promising policies for closing ELL achievement gaps will aim to place effective teachers with English learners. If specific skills are more important, then efforts to