WIXLIB

RESEARCH REPORT 2016-1The Redesigned SAT PilotPredictive Validity Study:A First Look By Emily J. Shaw, Jessica P. Marini, Jonathan Beard, Doron Shmueli,Linda Young, and Helen Ng

Emily J. Shaw is a senior director in the College Board’s researchdepartment.Jessica P. Marini is an assistant research scientist in the CollegeBoard’s research department.Jonathan Beard is an associate research scientist in the CollegeBoard’s research department.Doron Shmueli is a research statistician II in the College Board’sresearch department.Linda Young is a senior database statistician in the College Board’sresearch department.Helen Ng is a senior research analyst in the College Board’s researchdepartment.About the College BoardThe College Board is a mission-driven not-for-profit organization thatconnects students to college success and opportunity. Founded in1900, the College Board was created to expand access to highereducation. Today, the membership association is made up of over6,000 of the world’s leading educational institutions and is dedicatedto promoting excellence and equity in education. Each year, theCollege Board helps more than seven million students prepare fora successful transition to college through programs and services incollege readiness and college success — including the SAT and theAdvanced Placement Program . The organization also serves theeducation community through research and advocacy on behalf ofstudents, educators, and schools. For further information,visit www.collegeboard.org. 2016 The College Board. College Board, Advanced Placement Program, AP,SAT, and the acorn logo are registered trademarks of the College Board. ACESand Admitted Class Evaluation Service are trademarks owned by the CollegeBoard. PSAT/NMSQT is a registered trademark of the College Board andNational Merit Scholarship Corporation. Visit the College Board on the Web:www.collegeboard.org.VALIDITYFor more information onCollege Board research and data,visit research.collegeboard.org.

ContentsExecutive Summary . 4Introduction . 5Methodology . 6Study Design. 6Participants. 7Measures. 10Analysis. 11Results . 12Discussion. 19References. 21

TablesTable 1. Comparison of Institutional Study Sample to Population of Institutionsfor Recruitment and Previous SAT Validity Research Sample. 8Table 2. Comparison of Student Sample to Population of Students andPrevious SAT Validity Research Student Sample . 10Table 3. Descriptive Statistics for Study Variables . 13Table 4. Corrected (Raw) Correlation Matrix of Redesigned SAT Sections and HSGPA. 13Table 5. Correlations of Predictors with FYGPA. 14Table 6. Comparison of Correlations of Predictors with FYGPA from RedesignedSAT Pilot Validity Study Sample and 2012 National SAT Validity Study Sample . 16

FiguresFigure 1. Mean FYGPA by SAT total score band. . 14Figure 2. Mean FYGPA by SAT total score band, controlling for HSGPA. 15Figure 3. Relationship between SAT Evidence-Based Reading and Writing scoresand course grades in the same domain. . 17Figure 4. Relationship between SAT Math Section scores and course grades inthe same domain. 17Figure 5. Relationship between SAT Analysis in Science cross-test scores andcourse grades in the same domain. 18Figure 6. Relationship between SAT Analysis in History/Social Studiescross-test scores and course grades in the same domain. 19

The Redesigned SAT Pilot Predictive Validity StudyExecutive SummaryThe College Board conducted a pilot predictive validity study to provide colleges anduniversities with early information about the relationship between the redesigned SAT andcollege grades. Fifteen four-year institutions were recruited to administer a pilot form of theredesigned SAT to between 75 and 250 first-year, first-time students very early in the fallsemester of 2014. Measures were taken to ensure that the redesigned SAT was administeredto students under standardized conditions and that students were motivated to perform wellon the test. In June 2015, participating institutions provided the College Board with first-yearperformance data for those students participating in the fall 2014 administration of theredesigned SAT so that relationships between SAT scores and college performance could beanalyzed. Results of study analyses show that the redesigned SAT is as predictive of collegesuccess as the current SAT, that redesigned SAT scores improve the ability to predict collegeperformance above high school GPA alone, and that there is a strong, positive relationshipbetween redesigned SAT scores and grades in matching college course domains, suggestingthat the redesigned SAT is sensitive to instruction in English language arts, math, science, andhistory/social studies.4College Board Research Reports

The Redesigned SAT Pilot Predictive Validity StudyIntroductionIn February of 2013, the College Board announced it would undertake a redesign of theSAT in order to develop an assessment that better reflects the work that students will doin college, focusing on the core knowledge and skills that evidence has shown to be criticalin preparation for college and career. The redesigned test will be introduced in March 2016and will include a number of important changes (for a full description of these changes, /test-design).The redesigned SAT Evidence-Based Reading and Writing section and (optional) Essay portionswill incorporate key design elements supported by evidence, including: The use of a range of text complexity aligned to college- and career-ready reading levels; An emphasis on the use of evidence and source analysis; The incorporation of data and informational graphics that students will analyze alongwith text; A focus on relevant words in context and on word choice for rhetorical effect; Attention to a core set of important English language conventions and to effective writtenexpression; and The requirement that students interact with texts across a broad range of disciplines.The key evidence-based design elements that will be incorporated into the redesigned SATMath section include: A focus on the content that matters most for college and career readiness (rather than avast array of concepts); An emphasis on problem solving and data analysis; and The inclusion of “Calculator: Permitted” questions as well as “Calculator: Not Permitted”questions and attention to the use of the calculator as a tool.Instead of the SAT having three sections, each on a 200–800 scale, the redesigned SAT willnow have two broad sections: Evidence-Based Reading and Writing, and Math, each on a200–800 scale. Within the Evidence-Based Reading and Writing section, there will be two testscores: a Reading Test score and a Writing and Language Test score, each on a 10–40 scale.The Math section will also produce a Math Test score on a 10–40 scale. The Essay will nowbe optional, and students will have 50 minutes instead of 25 minutes to write. There will alsobe a number of subscores and cross-test scores produced to provide richer information tostudents, schools, and institutions on student performance. Another notable change is thatstudents will earn points for the questions they answer correctly and will not lose points forincorrect answers as they had on the previous SAT.As with the redesign of all assessments, it is important to examine and understand how thechanges to the content and format of the test impact the inferences made from the test’sscores for their intended uses. One primary use of the SAT is for admission and placementCollege Board Research Reports5

The Redesigned SAT Pilot Predictive Validity Studydecisions and, therefore, it was important to examine the relationship between the scoresfrom the redesigned test with college outcomes such as first-year grade point average(FYGPA) and college course grades. In order to conduct such an analysis a pilot study wasinitiated because the test is not yet operational.This paper describes the research efforts and the results of the first predictive validity studyon a pilot form of the redesigned SAT. The findings should inform the higher educationcommunity with regard to any expected changes in the predictive validity of the redesignedSAT in college admission.MethodologyStudy DesignA typical operational admission validity study would use students’ recorded SAT scores andtheir reported FYGPA to examine the statistical association between the two. Because thiswas a pilot study and not an operational validity study, it was necessary to first administer apilot form of the redesigned SAT to students who had just begun their first year of college.We would then follow those students through their first year of college and collect theirgrades and FYGPA as the outcome for analyses. In order to do this, the College Boardpartnered with four-year institutions in the U.S. to administer the test and then collect studentgrades. The general process for institutional participation was to: Determine a preferred date and time early in the first semester to hold the testadministration. Recruit between 75 and 250 students to participate in the study. For students to beeligible to sit for the test/participate in the study, they had to be first-time, first-yearstudents who had previously taken the SAT.o Students received a 100 gift card for participating in the study immediately followingtheir test participation. To increase test-taking motivation, students were also madeaware that they would receive a 50 gift card, mailed at a later date, if their scores onthe redesigned SAT met or exceeded their most recent SAT scores on record at theCollege Board. Reserve a testing room(s) based on planned recruitment numbers and SAT roomrequirements/sample seating plans. Assist with the handling of test materials and test day administration (along with CollegeBoard and ETS staff). Deliver the student data file to the College Board at the end of the 2014–15 school yearwith student participants’ first-year course work and grades.Institutions were asked to assign a Study Coordinator as the point person for the study. TheStudy Coordinator or the institution was eligible to receive a fee for coordinating the study. Inaddition, each participating institution received an institution-specific validity report based onits data.6College Board Research Reports