Transcription
THE DEEPEST LEARNERS:WHAT PISA CAN REVEAL ABOUT THE LEARNING THAT MATTERSDECEMBER 2013
At a time when the United States hascommitted itself to ensuring that all studentsgraduate from high school ready for collegeand careers, there is concern that far toofew high school students currently have theknowledge and skills they need to succeedin postsecondary education and training.However, there is little evidence about theextent to which U.S. students possess thecompetencies needed for success, andwhether other education systems are betterable to equip their youth with those abilities.One place to look for such evidence is the Programmefor International Student Assessment (PISA), a testadministered every three years by the Organisation forEconomic Co-operation and Development (OECD) tofifteen-year-olds in more than sixty countries. In contrastto other international assessments, PISA is designedto measure whether students can use their knowledgeto think critically, solve problems, and communicatewhat they understand. These abilities, known as deeperlearning, are precisely the kinds of abilities neededfor college and the workplace. PISA thus provides animportant window into deeper learning and nations’success in preparing students for life after high school.This report will examine what PISA reveals about deeperlearning. It will describe the competencies associated withdeeper learning and the evidence of their importance instudents’ lives after high school. It will show the ways thatPISA measures these competencies. And it will show the2012 results: which countries appear to be more successfulin getting more of their students to develop deeper learningcompetencies, and how these results compare with those inthe United States. It will also show which countries appearto produce greater equity in their educational outcomes—those countries with the narrowest gaps betweenhigh-performing and low-performing students.The report will look at evidence about the countries withlarge numbers of top performers to discover policies andpractices that seem to be associated with their abilityto enable more students to develop deeper learningcompetencies. It will also show new data from the 2012mathematics assessment that indicates the extent to whichstudents have had the opportunity to learn the kinds ofskills measured by PISA. It will conclude with implicationsfor the United States and recommendations for policy.A note of caution is in order. PISA is a test administeredat a single time. It does not measure all of the abilitiesstudents should demonstrate. And it provides a snapshotof a nation’s performance; it cannot definitively linkIn 2012, the OECD administered PISA to students in sixtyfive countries and education systems, as well as three U.S.states.1 The test measured students’ abilities in readingliteracy, mathematical problem solving, and science.Much of the attention to the results will likely focus on theinternational rankings, which show which countries, onaverage, performed best, and which countries improvedtheir relative performance (or declined) since the lastadministration of the test, in 2009. In addition, the OECD isproducing a report that will examine the results in light of theU.S. Common Core State Standards in mathematics.THE DEEPEST LEARNERS: WHAT PISA CAN REVEAL ABOUT THE LEARNING THAT MATTERSALL4ED.ORG1
policies and practices to results. Yet through the resultsWhy? In large part because technology has changed theand the extensive data gathered from student and schoolworkplace. The ubiquity of computers means that routinequestionnaires, it provides important insights into policiestasks—the kind that can be performed by people with littleand practices that can inform policymakers as theyeducation—have declined in importance; computers cancontinue to refine their education systems. The findings onperform those tasks or they can be outsourced to countriesdeeper learning can provide some guidance to the Unitedthat pay workers far less than the United States. But tasksStates as states move forward to implement standards forcollege and career readiness.that require expert thinking and complex communicationare increasingly important. That means that workers withhigher levels of educational attainment and skills are inTHE DEEPER LEARNINGCOMPETENCIES AND WHYTHEY MATTERDeeper learning competencies are not new. For centuries,schools have taught students to develop deep contentknowledge, to use that knowledge to think critically andsolve problems, to be able to communicate in a variety ofmedia, to collaborate with peers, to reflect on their ownlearning, and to develop appropriate academic mind-sets.An extensive study by the National Research Councilfound that these competencies are associated withimproved educational, career, and health outcomes foradults.2 The study found that deeper learning, whichthe report defined as the ability to transfer knowledge tonew settings, a critical goal for education, can developdemand, while those who lack such skills are not.4Virtually all states have embraced college and careerreadiness—and, by implication, deeper learning—as agoal for the education system. Forty-six states and theDistrict of Columbia have adopted the Common CoreState Standards, which spell out the knowledge andskills all students should demonstrate each year. Thestandards include numerous expectations for deeperlearning; for example, in English language arts/literacy,they require students to be able to construct and evaluateevidence-based arguments and to demonstrate complexcommunications. In mathematics, the standards requirestudents to solve non-routine problems and to reasonfrom evidence.The states that have adopted the standards are currentlythese competencies. The report also found that cognitiveimplementing them in classrooms by developing andcompetencies can be taught in ways that promote transfer.acquiring curriculum materials, preparing teachers for theThere also is increasing evidence that such abilities arenecessary for all students, not just for a select few. A studyby Anthony Carnevale and his colleagues at GeorgetownUniversity’s Center for Education and the Workforce foundthat in 2018 two-thirds of U.S. jobs will require somepostsecondary education, including four-year colleges,two-year colleges, and workplace training programs,compared with just over half in 1992 and one third in 1973.Only 10 percent of jobs will be able to be done by highschool dropouts, the study found.3new expectations, and developing new assessments tomeasure students’ abilities against the standards. Twostate consortia, the Partnership for the Assessment ofReadiness for College and Careers (PARCC) and theSmarter Balanced Assessment Consortium, are developingassessments intended to measure student learning againstthe standards. These assessments, which are expectedto be administered for the first time in the 2014–15 schoolyear, are likely to assess deeper learning competenciesmore extensively than do current state tests.5THE DEEPEST LEARNERS: WHAT PISA CAN REVEAL ABOUT THE LEARNING THAT MATTERSALL4ED.ORG2
HOW PISA MEASURESDEEPER LEARNINGUnlike many tests, including other internationalassessments, PISA was designed from the outset explicitlyto measure many deeper learning competencies. PISAquestions test whether students can apply their knowledgeto real-world problems “and to analyse, reason andcommunicate effectively as they pose, interpret and solveproblems in a variety of situations.”6In addition, PISA also includes an extensive studentquestionnaire that asks students about their motivationsto learn and their attitudes about learning, which are alsokey deeper learning competencies. In 2012, for the firsttime, the questionnaire included items that asked studentswhether they had encountered problems that asked themto apply mathematics in real-world settings, the kind ofproblems PISA emphasizes.PISA QUESTIONS TESTWHETHER STUDENTS CANAPPLY THEIR KNOWLEDGETO REAL-WORLD PROBLEMS“AND TO ANALYSE, REASONAND COMMUNICATEEFFECTIVELY AS THEYPOSE, INTERPRET ANDSOLVE PROBLEMS IN AVARIETY OF SITUATIONS.”students who had reached Level 5 or above in readingStudents who score at the highest levels on PISAon PISA were much more likely than those who did notdemonstrate deeper learning competencies. PISA scoresreach that level to be in college at age twenty-one. Highare reported in two ways: scale scores and performanceperformance in reading was also associated with higherlevels. The scale scores are similar to those used on theearnings in the workplace.7National Assessment of Educational Progress (NAEP).They measure student performance on a scale of 0 to1,000, with a mean score of 500.Deeper learning in mathematicsThe descriptions of student knowledge and skills atIn addition, PISA also reports the proportion of students whothe top performance levels show the deeper learningperform at six performance levels, which are based on thecompetencies that they can demonstrate. For example,difficulty of the tasks students are asked to perform. Thosein mathematics, a student at Level 5 canat Levels 5 and 6, who can complete the most cognitivelycomplex tasks, are considered “top performers.”develop and work with models for complex situations,identifying constraints and specifying assumptions.A study conducted by the OECD found that top performanceThey can select, compare and evaluate appropriateon PISA, particularly in reading, leads to positive outcomesproblem-solving strategies for dealing with complexfor young people. For six years the study followed Canadianproblems related to these models. Students at this levelstudents who had taken PISA in 2000 at age fifteen. It foundcan work strategically using broad, well-developedthat, even after controlling for background characteristics,thinking and reasoning skills, appropriate linkedsuch as parental education and other demographic factors,representations, symbolic and formal characterisations,THE DEEPEST LEARNERS: WHAT PISA CAN REVEAL ABOUT THE LEARNING THAT MATTERSALL4ED.ORG3
and insights pertaining to these situations. TheyThe following is an example of a problem typifyingbegin to reflect on their work and can formulate andperformance at Level 6:communicate their interpretations and reasoning.Helen rode her bike from her home to the river, which isAs an example of a task that exemplifies performance at4 km away. It took her 9 minutes. She rode home usingLevel 5, consider the following, from a set of tasks calleda shorter route of 3 km. This took her only 6 minutes.“Climbing Mount Fuji”:What was Helen’s average speed, in km/h, for her tripToshi wore a pedometer to count his steps on his walkto the river and back?along the Gotemba trail. His pedometer showed thatHere, students must know the mathematical definition ofhe walked 22,500 steps on the way up.“average speed,” or apply proportional reasoning, andEstimate Toshi’s average step length for his walkconvert units from minutes to hours.up the 9 km Gotemba trail. Give your answer incentimeters (cm).Although the calculation for this problem is relativelystraightforward, students must be able to rearrangethe conventional formula (distance average steplength x number of steps) and convert units fromkilometers to centimeters.Level 6 is even more cognitively challenging. Studentswho perform at that level in mathematics canDeeper learning in readingIn reading, tasks at top-performing levels require studentsto demonstrate deeper learning as well. At Level 5,[t]asks . . . that involve retrieving information requirethe reader to locate and organise several piecesof deeply embedded information, inferring whichinformation in the text is relevant. Reflective tasksrequire critical evaluation or hypothesis, drawingon specialised knowledge. Both interpretativeconceptualise, generalise and use information basedand reflective tasks require a full and detailedon their investigations and modelling of complexunderstanding of a text whose content or form isproblem situations, and can use their knowledgeunfamiliar. For all aspects of reading, tasks at thisin relatively non-standard contexts. They can linklevel typically involve dealing with concepts that aredifferent information sources and representationscontrary to expectations.and move flexibly among them. Students at this levelare capable of advanced mathematical thinking andAt Level 6,reasoning. These students can apply this insight and[t]asks . . . typically require the reader to makeunderstanding, along with a mastery of symbolic andmultiple inferences, comparisons and contrastsformal mathematical operations and relationships, tothat are both detailed and precise. They requiredevelop new approaches and strategies for addressingdemonstration of a full and detailed understandingnovel situations. Students at this level can reflectof one or more texts and may involve integratingon their actions, and can formulate and preciselyinformation from more than one text. Tasks maycommunicate their actions and reflections regardingrequire the reader to deal with unfamiliar ideas, in thetheir findings, interpretations and arguments, and canpresence of prominent competing information, andexplain why they were applied to the original situation.to generate abstract categories for interpretations.THE DEEPEST LEARNERS: WHAT PISA CAN REVEAL ABOUT THE LEARNING THAT MATTERSALL4ED.ORG4
Reflect and evaluate tasks may require the reader tobe found in the text itself, it is located in the middle ofhypothesise about or critically evaluate a complex textthe passage, rather than at the beginning, where moston an unfamiliar topic, taking into account multiplestudents making a literal interpretation would look for it.criteria or perspectives, and applying sophisticatedunderstandings from beyond the text. A salientcondition for access and retrieve tasks at this level isprecision of analysis and fine attention to detail that isinconspicuous in the texts.An example of a task exemplifying this level ofperformance asks students to read the first few pages of aplay by the Hungarian dramatist Ferenc Molnár and asks,“What were the characters in the play doing before thecurtain went up?”This task is particularly challenging because it requiresstudents to make an interpretation based on theirunderstanding of what the play states. While a clue canDeeper learning in scienceTop performers in science, likewise, can demonstratedeeper learning. At Level 5,students can identify the scientific components ofmany complex life situations, apply both scientificconcepts and knowledge about science to thesesituations, and can compare, select and evaluateappropriate scientific evidence for respondingto life situations. Students at this level can usewell-developed inquiry abilities, link knowledgeappropriately, and bring critical insights to situations.They can construct explanations based on evidenceand arguments based on their critical analysis.A problem that exemplifies performance at that levelshows two graphs, one presenting trends in carbon dioxideemissions over time and the other presenting trends inaverage earth temperatures over time. The problem states,“André concludes from these two graphs that it is certainthat the increase in the average temperature of the Earth’satmosphere is due to the increase in the carbon dioxideemission.” It then asks, “Another student, Jeanne, disagreeswith André’s conclusion. She compares the two graphsand says that some parts of the graphs do not support hisconclusion. Give an example of a part of the graphs thatdoes not support André’s conclusion. Explain your answer.”This question involves critical thinking, because it asksstudents to use evidence to support a conclusion; in thiscase, students must use evidence to refute a conclusion.Students must also communicate effectively in order toexplain their reasoning. Students who do not explain theiranswer sufficiently receive partial credit.THE DEEPEST LEARNERS: WHAT PISA CAN REVEAL ABOUT THE LEARNING THAT MATTERSALL4ED.ORG5
Level 6 indicates a higher level of deeper learning.This problem requires students to identify scientific issuesAt Level 6,and to explain phenomena scientifically. It thus requires astudents can consistently identify, explain and applyscientific knowledge and knowledge about sciencein a variety of complex life situations. They canlink different information sources and explanationsand use evidence from those sources to justifydecisions. They clearly and consistently demonstratedeep understanding of core content, in addition to criticalthinking and communication.THE DEEPEST LEARNERSON THE 2012 PISAadvanced scientific thinking and reasoning, andIn almost all countries, only a small percentage of studentsthey use their scientific understanding in support ofin 2012 reached the top levels and demonstrated deepersolutions to unfamiliar scientific and technologicallearning competencies. However, there were widesituations. Students at this level can use scientificvariations among nations. In Shanghai-China, 56 percentknowledge and develop arguments in support ofof students were top performers in at least one subjectrecommendations and decisions that centre onarea, and 19.6 percent were top performers in all three.personal, social or global situations.Among the industrialized nations in the OECD, on averageA problem that exemplifies this level is also related to theclimate change graphs. In this case, the problem states thatAndré persists in his conclusion that the averagetemperature rise of the Earth’s atmosphere is causedby the increase in the carbon dioxide emission. ButJeanne thinks that his conclusion is premature. Shesays: “Before accepting this conclusion you mustbe sure that other factors that could influence thegreenhouse effect are constant”. Name one of thefactors that Jeanne means.IN SHANGHAI-CHINA,56 PERCENT OF STUDENTSWERE TOP PERFORMERS INAT LEAST ONE SUBJECT AREA,AND 19.6 PERCENT WERE TOPPERFORMERS IN ALL THREE.16.2 percent were top performers in at least one subjectand 4.4 percent were top performers in all three. However,there were some variations among the countries. InFinland, for example, 24 percent were top performers inat least one subject and 7.4 percent were top performersin all three; in Japan, 30 percent were top performers inat least one subject and 11.3 percent in all three; and inCanada, 21.9 percent were top performers in at least onesubject and 6.5 percent in all three. In the U.S., 12 percentof students were top performers in at least one subject and4.7 percent were top performers in all three.Mathematics had the highest proportion of studentsperforming at top levels. On average among OECDnations, 9.3 percent of students performed at Level 5and 3.3 percent performed at Level 6. Many countrieshad much higher proportions of students at top levels.In Shanghai-China, 24.6 percent of students performedat Level 5 and 30.8 percent performed at Level 6. InSingapore, 21 percent performed at Level 5 and 19percent performed at Level 6.THE DEEPEST LEARNERS: WHAT PISA CAN REVEAL ABOUT THE LEARNING THAT MATTERSALL4ED.ORG6
Among European nations, the Netherlands stood out,In the U.S., 6.9 percent reached Level 5 and 1 percentwith 14.9 percent at Level 5 and 4.4 percent at Level 6.attained Level 6. Those proportions were similar to thoseIn Switzerland, 14.6 percent performed at Level 5 and 6.8reached in Sweden and slightly lower than the proportionpercent at Level 6. Traditionally high-performing Finlandof top performers in the United Kingdom.came in somewhat lower, with 11.7 percent at Level 5and 3.5 percent at Level 6. In the Western Hemisphere,Canada led the pack with 12.1 percent at Level 5 and 4.3percent at Level 6.In the U.S., 6.6 percent of students attained Level 5, aboutthe same proportion as Latvia, Lithuania, and Sweden,while 2.2 percent performed at Level 6, comparable toHungary, Israel, and Italy.In several countries, the number
key deeper learning competencies. In 2012, for the first time, the questionnaire included items that asked students whether they had encountered problems that asked them to apply mathematics in real-world settings, the kind of problems PISA emphasizes. Students who score at the highest levels on PISA d
