Transcription
Yale CollegeMetropolitan Business Academy: A Case Study in Performance-based AssessmentEthan YoungBerkeley CollegeAdviser Professor Richard HershYale University Education Studies Program, Senior Capstone Project
Metropolitan Business Academy: A Case Study in Performance-based Assessment2Table of ContentsAbstract3Introduction4Scope and Research Questions9Literature Review11Methodology20Findings and Analysis24Ensuring Rigor: The Moderation Study24Alignment with Curriculum and Instruction: The Performance-Based Assessment28Recognizing Cognitive Processes: The SAT and PBA31Limitations34Discussion and Conclusion35Appendix A: Teacher Interview Questions38Appendix B: Student Survey39Appendix C: Moderation Study Rubric40Appendix D: Moderation Study Planning 2017-2018 (abridged)41Acknowledgements42Works Consulted43
Metropolitan Business Academy: A Case Study in Performance-based AssessmentAbstractThis paper examines a New Haven, CT high school where all students take performance-basedassessments (PBA) in addition to standardized tests, including the Scholastic Aptitude Test(SAT). Widely cited by scholars as pedagogically sound and logistically unworkable, PBApresents an alternative to standardized tests for evaluating student learning. Beyond potentialpedagogical benefits, Bloom’s Taxonomy suggests the promise of PBA to capture complexthought. Using student survey data and teacher interviews, this analysis considers rigor,alignment, and students’ metacognition to explain higher student success rates on performancebased tasks than SAT indicators at Metropolitan Business Academy (MBA). MBA studentsrecognized differences between these two assessments, and strongly preferred PBA for its pace,fairness, and opportunities for feedback. Although students found the SAT more difficult toprepare for and take, they felt that PBA more closely correlated with instruction and moreaccurately reflected their knowledge and skills. These findings suggest the value of authenticityin performance-based tasks to drive student motivation. MBA students viewed performancebased assessment as a meaningful exhibition of their learning and this belief motivated higherperformance rates than those exhibited during comparable SAT tasks. Further research shouldinvestigate these trends in different subject areas (namely mathematics) and among differentstudent populations.3
Metropolitan Business Academy: A Case Study in Performance-based Assessment4IntroductionIn seventh grade, I first became involved in education policy by writing the TennesseeDepartment of Education regarding its mandatory, state-wide assessments. Each year, I sat forthree full schooldays bubbling in answers and chewing peppermints meant to stimulate myhippocampus. The Tennessee Comprehensive Assessment Program (TCAP) was a dramaticevent: local news media interviewed students, teachers waited anxiously as their score-alignedsalaries were determined, and politicians anticipated a new year of results that could improveTennessee’s national reputation. Suffice it to say that I grew up regularly taking “high-stakes”tests. Despite the fanfare surrounding TCAP, its test questions were often ambiguous, poorlydesigned, and reductionist. Moreover, the TCAP “experience” was an intrusion into my learning,not an enrichment of it. I noted these complaints in a four-page letter to Nashville, but neverreceived a reply. My frustration persisted: how could this low-quality exam convey myintellectual capacity or growth?Since middle school, I discovered educational research that articulated my frustrationwith standardized tests. In 1956, Benjamin Bloom published his Taxonomy of EducationalObjectives, which classified cognitive processes and forms of knowledge into hierarchies ofcomplexity (see Figure 1). Since then, Bloom’s Taxonomy has been revised and critiqued, but itscentral contribution endures: educational tasks require different degrees of cognitive processing,the most complex of which is creation—a new idea, investigation, or product (Anderson et al.,2001). In other words, completing a multiple-choice exam demands different levels of cognitionthan writing an essay or articulating an original thought. At best, standardized exams askstudents to remember, understand, apply, analyze, and occasionally even evaluate; but, theyrarely gauge creation, the highest form of cognition according to Bloom’s Taxonomy (Anderson
Metropolitan Business Academy: A Case Study in Performance-based Assessment5et al., 2001). So, perhaps my seventh-grade self reacted to the agony of enduring the TCAP forthree days, the pressure of the examination environment, or its poorly-crafted materials. Or,perhaps I reacted to a deeper flaw: standardized tests were assessing lower-level cognitiveprocesses, tasks I found dull and uninteresting. My experience in Tennessee’s public schools,however, was hardly unique.Figure 1. Bloom’s Taxonomy, Used by Permission from the Vanderbilt University Center for Teaching.Throughout the past two decades, standardized tests have come to dominate Americaneducation. In the pursuit of accountability, legislators have increasingly sought discrete,quantifiable means by which to evaluate schools. To generate data about something as complexand mysterious as learning, assessments necessarily rely on reductionism—most commonlyexemplified by multiple choice questions. Meanwhile, educators ask their students to do just theopposite: to broaden one’s mind and formulate complex ideas. Often, both parties end upfrustrated. Policymakers call for better data or dramatic reforms based primarily upon test scores.Teachers bemoan the intrusion of testing into their instructional time. Buried beneath this debatesit students, expected to learn and perform on a range of assessments. Most young Americans
Metropolitan Business Academy: A Case Study in Performance-based Assessment6take tests just like the TCAP—high-stakes, comprehensive, and authoritative in the minds ofmany policymakers and educational leaders.Standardized assessments prevail due to the acceptance of a pedagogical narrative inAmerican culture and politics: these tests enable a social meritocracy by consistently and fairlymeasuring academic ability. Correlatively, students’ performance on prevalent standardized testsindicates, and often serves as the only legitimate indication of, certain content and skill mastery.This narrative not only permeates education scholarship but also fundamentally shapes ourunderstanding of human worth and social possibility. Although literature supports the validityand reliability of standardized testing in what they purport to measure, dissident voices haveproliferated in recent decades—educators, researchers, and policymakers (Darling-Hammond,1994; Gordon & Rajagopalan, 2016; Hursh, 2005; Koretz, 2017; Ravitch, 2013; Rothstein, 2004;Schneider, 2017) questioning the validity of such testing.Their critiques vary. Some lament a conflict of interest linked to for-profit testingbehemoths driving much of today’s standardized test production and evaluation (Ravitch, 2013).Others point to the significant loss of instructional hours due to frequent in-class testingrequirements (Wilczak et al., 2014). Still more critique the way in which test data has become anend of teaching in its own right, surpassing more essential matters like learning and growth(Koretz, 2017). But some have leveled more fundamental critiques: standardized tests areintrinsically unfair and inconsistent; their very design preordains that certain students performwell, and others do not (Darling-Hammond, 2007; Lee, 1998; Rothstein, 2004). According tothese critics, performance divisions fall along familiar categories of race, class, and gender.Many also lend credence to my seventh-grade observation. Referencing Bloom’s Taxonomy,they question the efficacy of standardized exams to assess the most complex learning outcomes,
Metropolitan Business Academy: A Case Study in Performance-based Assessment7such as creativity and problem solving (Gordon & Rajagopalan, 2016; Koretz, 2017). If true,these criticisms reanimate a time-honored question: how should we go about judging humanachievement and potential? Can assessment provide robust accountability while maintainingfidelity to our educational aspirations, such as equal opportunity and transformative learning forall students?While educators, researchers, and policymakers debate these questions, a movementbegan in New York City twenty years ago. Thirty-two public schools formed a coalition—theNew York Performance Standards Consortium (NYPSC)—to support a common conviction:performance-based assessments (PBA) offer superior insight and learning opportunities tostudents than those of standardized tests. These performance-based assessments ask students tocreate products—essays, oral presentations, portfolios—that demonstrate their mastery ofstandards like argumentation, source evaluation, and contextualization. After developingrigorous testing materials and lobbying the New York Department of Education, NYPSC schoolswere exempted from New York State Board of Regents exams—New York’s TCAPequivalent—allowing performance-based tests to serve as indicators of student growth andachievement. Moreover, NYPSC schools averaged a ten percent gain in 4- and 6-year graduationrates, as well as a twenty-percent gain in college enrollment 18 months post-graduation(Redefining Assessment, 2017). Their success inspired other educators to develop alternatives tohigh-stakes testing programs.In 2013, educators from a Connecticut high school visited a NYPSC school and adaptedits methods. Metropolitan Business Academy (MBA), an interdistrict magnet high school locatedin New Haven’s Wooster Square community, features performance-based assessment as arequired component of every student’s education. I first learned about MBA’s unique assessment
Metropolitan Business Academy: A Case Study in Performance-based Assessment8program in conversation with Principal Judith Puglisi, who pioneered the initiative. She showedme videos of high school seniors delivering independently-researched presentations tocommunity members, teachers, and peers. These oral presentations, along with extended researchpapers and round-table discussions, culminate four years of work at MBA. Students choose theirsubject and presentation format, but everyone must produce a substantial artifact thatdemonstrates mastery of specified competencies. Although students continue to take some statemandated standardized tests, MBA assigns significant value to their in-house PBA program forits ability to improve instruction and empower students by equalizing the playing field. LikeNYPSC schools, MBA saw broader student improvements after its assessment shift.Four years into the program, an extraordinary trend emerged: students performed betteron PBA than the College Board’s Scholastic Aptitude Test (SAT). All eleventh graders atMetropolitan complete a performance-based Moderation Study and the Evidence-Based Readingand Writing portion of the SAT. These two exams purport to test nearly identical outcomes:reading comprehension skills, evidence-based arguments, and evaluation of sources. In 2016,37.1 percent of eleventh graders met a College Ready standard as assessed by SAT’s Englishsection.1 Meanwhile, 60.0 percent of those same students were identified as “Competent” orabove on Metropolitan’s performance-based Moderation Study.2 Moreover, the ModerationStudy is a diagnostic test administered to gauge students’ abilities before they enter aninstructional year. Thus, nearly a third of Metropolitan’s eleventh graders were given twoverdicts about their preparedness for the future.1Data from “New Haven Public Schools, Metropolitan, High School Balanced Progress Report.”From the outset, it is important to clarify the impossibility of direct comparisons between PBAand SAT scores given the two assessments’ distinct methods. I treat this subject in greater detailin the Methodology section.2
Metropolitan Business Academy: A Case Study in Performance-based Assessment9Why are students succeeding at higher rates on an assessment that is, in theory, morechallenging? Three explanations come to mind. Maybe the performance-based test is easier ormore lenient, so more students earn higher scores. The performance-based test might better alignwith students’ experiences in the classroom, giving them a conceptual advantage going into thetask. Or, perhaps students view these two tasks very differently, and apply themselves more fullytowards the performance-based task than the SAT. These explanations yield research questionsabout rigor, alignment, and student metacognition.3 Are Metropolitan Business Academy’sperformance-based assessments rigorous, or can these data be ascribed to lower standards? Arestudents benefiting from clearer correspondence among curriculum, instruction, and assessment,or do performance-based assessments and the SAT align similarly? Are students aware of thecognitive processes being probed in performance-based assessments as compared to SAT, or dothey understand both assessments to be similar? By addressing these three questions, I willconstruct an explanation for these unexpected data.Scope and Research QuestionsAssessment is among the most widely debated, heavily researched, and politicallycharged topics in education. Consequently, it is important to address this study’s narrow scopefrom the outset. My project focuses on one performance-based assessment administered to allMBA students. Regardless of Academic Pathway, everyone completes a writing-based PBAduring their junior year.4 My research evaluates the rigor of Metropolitan’s performance-based3These terms are further defined in the Literature Review section, but to offer preliminarydefinitions: rigor refers to the fidelity of an exam’s results to the purported objectives beingtested; alignment refers to the correspondence among curriculum, teaching, and assessment;student metacognition refers to the ability of students to recognize their own cognitive processes.4MBA students choose an Academic Pathway, options of which include Law and PoliticalScience, Allied Health and Science, Digital Arts and Technology, and Finance.
Metropolitan Business Academy: A Case Study in Performance-based Assessment10assessment, contextualize program alignment in existing scholarship, and evaluate students’metacognitive distinctions between performance-based assessments and standardized tests,namely the SAT. With this information, I attempt to make sense of the finding that more studentsperform well on performance-based assessments. In doing so, I contextualize Metropolitan’sprogram design in broader research, and offer a case study of performance-based assessment tothe larger field of Education Studies.
Metropolitan Business Academy: A Case Study in Performance-based Assessment11Literature ReviewThis analysis intersects scholarship concerning assessment theory, educationalpsychology, and the history of education policy. Before describing specific assessments, oneshould define the term. An assessment, or test, is a formal attempt to determine a student’srelationship to specified variables (i.e., learning outcomes, attitudes). As Popham (2003)outlines, educators rely on assessment to enable decisions about “the nature and purpose ofcurriculum,” “students’ prior knowledge,” “how long to teach something,” and the “effectivenessof instruction” (p. 9). Thus, assessment lies at the core of educational practice, facilitatingexchange between curriculum and instruction (Anderson et al., 2001; Popham, 2003). Alignmentdescribes the extent to which these three elements correspond and interact (Anderson et al.,2001). As the cohesion between instruction and evaluation, curriculum and instruction, orevaluation and curriculum improve, so too does alignment.Assessments range considerably, but they can be classified by their role in aneducational system and the kind of learning they evaluate. Most assessments are diagnostic,formative, or summative (Luongo-Orlando, 2003). Diagnostic assessments, such as MBA’sModeration Study, provide educators with baseline information about students’ knowledge andskills to create effective teaching plans. Formative assessments, such as MBA’s performancebased assessment, function within the learning process throughout a school year, offeringiterative feedback that allows students and teachers to adjust their methods to meet performancegoals. Summative assessments, such as the SAT, occur after a period of teaching and askstudents to demonstrate their knowledge and skills on-demand. Diagnostic, formative, andsummative assessments achieve different goals, and it is important to distinguish performancebased assessments and the SAT since, “formative assessment is used primarily to improve
Metropolitan Business Academy: A Case Study in Performance-based Assessment12student learning; summative assessment is used primarily to assign grades” (Anderson, et al.,2001, p. 102). So, how do these different assessments evaluate student learning?Educators designassessments to evaluatecertain cognitive processes.Anderson, et al. (2001)revised Bloom’s Taxonomy(Bloom, 1956) and presenta Taxonomy Table forclassifying assessments interms of cognitive processesand types of knowledge.5The Cognitive ProcessDimension describes theform of cognitive workFigure 2. The Taxonomy Table. Reprinted from Anderson et al., 2001, p. 28.required by a student in agiven task, ranging from memorization on the least difficult end to creation on the most difficult.The Knowledge Dimension describes the type of knowledge required by a task, ranging fromfactual knowledge on the least difficult end to metacognitive on the most difficult. As visualizedin Figure 2, each tier in in the taxonomy is thought to be more difficult and, by extension, less5There are two essential merits to describing assessments with the Taxonomy Table. First, theTable enables detailed analysis of alignment by displaying the correspondence amongcurriculum objectives, how they were taught, and assessment of student work. In other words, itreveals the distance between theory and practice in education. Second, the Table assistseducators in eliciting high-level cognitive processes among students.
Metropolitan Business Academy: A Case Study in Performance-based Assessment13commonly achieved by students. Kreitzer and Madaus (1994) support the hierarchical theory ofBloom’s Taxonomy, which suggests that students progress through each step of cognitiveprocessing sequentially. This hierarchy further suggests that only by considering the long-viewof a given educational task is it possible to guide students to creation and evaluation. Using theTaxonomy Table, comparisons of performance-based assessments and the SAT revealunderlying differences in their educational philosophies and abilities to test cognitive processes.6Performance-based assessment (sometimes called ‘performance assessment’) has alongstanding and robust presence in educational research and pedagogical literature (Rudner &Boston, 1994; Pecheone et al., 2010; Darling-Hammond, 1994; Baker et al., 1993). PBA’sunderlying philosophy appeared as early as John Dewey, who wrote: “The child’s own instinctsand powers furnish the material and give the starting-point” (Dewey, 1964, p. 428). Althoughperformance-based assessments range considerably in format and scope, they consistently entailstudent-produced artifacts (i.e., portfolio, oral presentation, research paper) and timely feedbackfrom educators (Rudner & Boston, 1994). In the United States, performance-based assessmentsrose to national awareness in the 1980s as indicated by the U.S. Department of Education’sreview of performance-based assessment in its Educational Resource Information Center (ERIC)publication (Rudner & Boston, 1994). Research literature offers numerous d
New York Performance Standards Consortium (NYPSC)—to support a common conviction: . These performance-based assessments ask students to create products—essays, oral presentations, portfolios—that demonstrate their mastery of standards like argumentation, source evaluation, and contextualization. . (MBA), an interdistrict magnet high .
