Transcription
Programme for International Student AssessmentPISA 2003Data Analysis ManualSPSS UsersOECDORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENTPISA2003manual-fauxtitre-SPSS.indd 15/06/05 20:39:53
ForewordForewordThe OECD’s Programme for International Student Assessment (PISA) surveys, which take place everythree years, have been designed to collect information about 15-year-old students in participatingcountries. PISA examines how well students are prepared to meet the challenges of the future, rather thanhow well they master particular curricula. The data collected during each PISA cycle are an extremelyvaluable source of information for researchers, policy makers, educators, parents and students. It is nowrecognised that the future economic and social well-being of countries is closely linked to the knowledgeand skills of their populations. The internationally comparable information provided by PISA allowscountries to assess how well their 15-year-old students are prepared for life in a larger context and tocompare their relative strengths and weaknesses.The PISA 2003 database, on which this manual is focused, contains information on over a quarter ofa million students from 41 countries. It includes not only information on their performance in thefour main areas of assessment – reading, mathematics, science and problem solving – but also theirresponses to the Student Questionnaire that they complete as part of the assessment. Data from theschool principals are also included.The PISA 2003 Data Analysis Manual has evolved from the analytical workshops held in Sydney,Vienna,Paris and Bratislava, which exposed participants to the various techniques needed to correctly analysethe complex databases. It allows analysts to confidently replicate procedures used for the production ofthe PISA 2003 initial reports, Learning for Tomorrow’s World – First Results from PISA 2003 (OECD, 2004a)and Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003(OECD, 2004b), and to accurately undertake new analyses in areas of special interest. In addition tothe inclusion of the necessary techniques, the manual also includes a detailed account of the variablesconstructed from the student and school questionnaires. This information was previously published inthe Manual for the PISA 2000 Database (OECD, 2002a).The PISA 2003 Data Analysis Manual is in four parts – the first two sections give a detailed theoreticalbackground and instructions for analysing the data; the third section lists the program codes (syntaxesand the macros), which are needed to carry out the analyses; and the fourth section contains a detaileddescription of the database.PISA is a collaborative effort by the participating countries, and guided by their governments on thebasis of shared policy-driven interests. Representatives of each country form the PISA Governing Boardwhich decides on the assessment and reporting of results in PISA.There are two versions of this manual – one for SPSS users and one for SAS users. The OECDrecognises the creative work of Christian Monseur in preparing the text for both versions of the manualin collaboration with Sheila Krawchuk and Keith Rust, as well as his preparation of the program codingfor the SAS users’ manual.The coding for the SPSS users’ manual was prepared by Wolfram Schulz andEveline Gebhardt. The main editorial work was completed at the OECD Secretariat by Miyako Ikeda,Sophie Vayssettes, John Cresswell, Claire Shewbridge and Kate Lancaster. The PISA assessmentsand the data underlying the manuals were prepared by the PISA Consortium under the direction ofRaymond Adams. OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-foreword-SPSS.indd 335/06/05 20:47:12
Table of ContentsTable of ContentsUsers’ Guide . 9CHAPTER 1The OECD’s Programme for International Student Assessment . 11An overview of PISA . 12What makes PISA unique? . 13How the assessment takes place . 15About this manual. 16CHAPTER 2Sample Weights . 19Introduction . 20Weights for simple random samples . 21Sampling designs for education surveys . 22Why do the PISA weights vary? . 27Conclusions . 30CHAPTER 3Replicate Weights . 31Introduction . 32Sampling variance for simple random sampling . 32Sampling variance for two-stage sampling . 38Replication methods for simple random samples . 44Resampling methods for two-stage samples. 46The Jackknife for unstratified two-stage sample designs . 47The Jackknife for stratified two-stage sample designs. 48The Balanced Repeated Replication method . 49Other procedures for accounting for clustered samples . 51Conclusions . 51CHAPTER 4The Rasch Model . 53Introduction . 54How can the information be summarised? . 54The Rasch model for dichotomous items . 56Other Item Response Theory models . 69Conclusions . 69 OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-ToC-SPSS.indd 555/06/05 20:47:58
Table of ContentsCHAPTER 5Plausible Values. 71Individual estimates versus population estimates . 72The meaning of plausible values . 72Comparison of the efficiency of Warm Likelihood Estimates, Expected A Posterioriestimates and plausible values for the estimation of some population statistics . 76How to perform analyses with plausible values. 78Conclusions . 80CHAPTER 6Computation of Standard Errors . 81Introduction . 82The standard error on univariate statistics for numerical variables . 82The SPSS macro for computing the standard error on a mean . 85The standard error on percentages. 87The standard error on regression coefficients . 90The standard error on correlation coefficients . 92Conclusions . 93CHAPTER 7Analyses with Plausible Values . 95Introduction . 96Univariate statistics on plausible values . 96The standard error on percentages with plausible values . 101The standard error on regression coefficients with plausible values . 101The standard error on correlation coefficients with plausible values . 104Correlation between two sets of plausible values . 104A fatal error shortcut . 108An unbiased shortcut . 109Conclusions . 110CHAPTER 8Use of Proficiency Levels . 111Introduction . 112Generation of the proficiency levels. 112Other analyses with proficiency levels. 118Conclusions . 121CHAPTER 9Analyses with School Level Variables . 123Introduction . 124Limits of the PISA school samples . 125Merging the school and student data files . 126Analyses of the school variables . 126Conclusions . 1286 OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-ToC-SPSS.indd 65/06/05 20:47:59
Standard Error on a Difference . 129Introduction . 130The standard error of a difference without plausible values . 132The standard error of a difference with plausible values. 137Multiple comparisons . 140Conclusions . 141Table of ContentsCHAPTER 10CHAPTER 11OECD Average and OECD Total. 143Introduction . 144Recoding of the database for the estimation of the OECD total and OECD average . 144Duplication of the data for avoiding three runs of the procedure . 147Comparisons between OECD average or OECD total estimates and a country estimate . 147Conclusions . 150CHAPTER 12Trends . 153Introduction . 154The computation of the standard error for trend indicators on variables otherthan performance . 155The computation of the standard error for trend indicators on performance variables . 158Conclusions . 164CHAPTER 13Multilevel Analyses . 165Introduction . 166Simple linear regression . 166Simple linear versus multilevel regression analyses . 170Fixed effect versus random effect . 173Some examples with SPSS . 174Limitations of the multilevel model in the PISA context . 190Conclusions . 192CHAPTER 14Other Statistical Issues . 193Introduction . 194Analyses by quarters . 194The concepts of relative risk and attributable risk . 198Instability of the relative and attributable risks. 200Computation of the relative risk and attributable risk . 201Conclusions . 201CHAPTER 15 SPSS Macros . 203Introduction . 204Structure of the SPSS macros . 204 OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-ToC-SPSS.indd 775/06/05 20:48:00
Table of ContentsAppendix 1: PISA 2003 International Database . 237Appendix 2: Student Questionnaire . 251Appendix 3: Educational Career Questionnaire . 265Appendix 4: Information Communication Technology (ICT) Questionnaire . 267Appendix 5: School Questionnaire . 271Appendix 6: Student Questionnaire Data File Codebook . 285Appendix 7: School Questionnaire Data File Codebook . 331Appendix 8: Student Cognitive Test Data File Codebook. 345Appendix 9: Student and School Questionnaire Indices . 375Appendix 10: Scores Allocated to the Items . 401References . 4078 OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-ToC-SPSS.indd 85/06/05 20:48:01
Users’ GuideUSERS’ GUIDEPreparation of data filesAll data files (in text format) and the SPSS control files are available on the PISA Web site(www.pisa.oecd.org).SPSS usersThe PISA 2003 student data file and the PISA 2003 school data file need to be saved under“c:\PISA\Data2003” before running the syntax in the following chapters.Student data file: c:\PISA\Data2003\INT stui 2003.savSchool data file: c:\PISA\Data2003\INT schi 2003.savThe SPSS macros presented in Chapter 15 need to be saved under “c:\PISA\macros”.SPSS syntax and macrosAll syntaxes and macros used in this manual can be copied from the PISA Web site (www.pisa.oecd.org). Each chapter of the manual contains a complete set of syntaxes, which must be donesequentially, for all of them to run correctly, within the chapter.Rounding of figuresIn the tables and formulas, figures were rounded to a convenient number of decimal places,although calculations were always made with the full number of decimal places.Country abbreviations used in this striaBelgiumBrazilCanadaSwitzerlandCzech DNIRLISLITAJPNFranceUnited KingdomGreeceHong w lRussian ayUnited StatesSerbiaSocio-economic statusThe highest occupational status of parents (HISEI) is referred to as the socio-economic statusof the students throughout this manual. It should be noted that occupational status is only oneaspect of socio-economic status, which can also include education and wealth. The PISA 2003database also includes a broader socio-economic measure called the index of Economic, Socialand Cultural Status (ESCS), which is derived from the highest occupational status of parents,the highest educational level and an estimate related to household possessions. OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-guide-SPSS.indd 995/06/05 20:48:42
Users’ Guide10Further documentationFor further information on the PISA 2003 results, see the PISA 2003 initial reports: Learning forTomorrow’s World – First Results from PISA 2003 (OECD, 2004a) and Problem Solving for Tomorrow’sWorld – First Measures of Cross-Curricular Competencies from PISA 2003 (OECD, 2004b). Forfurther information on the PISA assessment instruments and the method used in PISA, see thePISA 2003 Technical Report (OECD, forthcoming) and the PISA Web site (www.pisa.oecd.org). OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-guide-SPSS.indd 105/06/05 20:48:45
1The OECD’s Programmefor InternationalStudent AssessmentAn overview of PISA . 12What makes PISA unique?. 13How the assessment takes place . 15About this manual . 16 OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-ch1-SPSS.indd 11115/06/05 20:49:33
The OECD’s Programme for International Student Assessment1AN OVERVIEW OF PISAThe OECD’s Programme for International Student Assessment (PISA) is a collaborative effort, involvingall OECD countries and a significant number of partner countries, to measure how well 15-year-oldstudents are prepared to meet the challenges of today’s knowledge societies.The assessment looks to thefuture, focusing on young people’s ability to use their knowledge and skills to meet real-life challenges,rather than on the mastery of specific school curricula. This orientation reflects a change in the goalsand objectives of curricula themselves, which are increasingly concerned with knowledge applicationrather than merely knowledge acquisition. The age of 15 is used because in most OECD countries it isthe age at which students are approaching the end of compulsory schooling.PISA is the most comprehensive and rigorous international effort to date to assess student performanceand to collect data about the student, as well as about the family and institutional factors potentiallyaffecting performance. Decisions about the scope and nature of the assessment and the backgroundinformation to be collected were made by leading experts in participating countries and steeredjointly by their governments on the basis of shared, policy-driven interests. Substantial efforts andresources were devoted to achieving wide cultural and linguistic coverage in the assessment materials.Stringent quality assurance mechanisms were applied in translation, sampling and data collection. As aconsequence, the results of PISA have a high degree of validity and reliability, and they can significantlyimprove understanding of the outcomes of education in a large number of the world’s countries.PISA is based on a dynamic model of lifelong learning in which new knowledge and skills necessary forsuccessful adaptation to a changing world are continuously acquired throughout life. PISA focuses onskills that 15-year-olds will need in the future and seeks to assess their ability to perform them. PISAdoes assess students’ knowledge, but it also examines their potential to reflect on their knowledge andexperiences, and to apply that knowledge and those experiences to real-world issues. For example,Table 1.1 Participating countries in PISA 2000 and in PISA 2003PISA 2000PISA 2003OECDcountriesAustralia, Austria, Belgium, Canada,Czech Republic, Denmark, Finland, France,Germany, Greece, Hungary, Iceland,Ireland, Italy, Japan, Korea, Luxembourg,Mexico, Netherlands,a New Zealand,Norway, Poland, Portugal, Spain, Sweden,Switzerland, United Kingdom,United States.Australia, Austria, Belgium, Canada,Czech Republic, Denmark, Finland, France,Germany, Greece, Hungary, Iceland,Ireland, Italy, Japan, Korea, Luxembourg,Mexico, Netherlands, New Zealand,Norway, Poland, Portugal, Slovak Republic,Spain, Sweden, Switzerland, Turkey,United Kingdom,b United States.PartnercountriesAlbania, Argentina, Brazil, Bulgaria, Chile,Hong Kong-China, Indonesia, Israel, Latvia,Liechtenstein, Macedonia, Peru, Romania,Russian Federation, ThailandBrazil, Hong Kong-China, Indonesia,Liechtenstein, Latvia, Macao-China,Russian Federation, Thailand, Tunisia,Uruguay, Serbia.ca.b.c.12Response rate is too low to ensure comparability. See Annex 3 in Literacy Skills for the World of Tomorrow – FurtherResults From PISA 2000 (OECD, 2003a).Response rate is too low to ensure comparability. See Annex 3 in Learning for Tomorrow’s World – First Results from PISA2003 (OECD, 2004a).For the country Serbia and Montenegro, data for Montenegro are not available in PISA 2003. The latter accountsfor 7.9 per cent of the national population. The name “Serbia” is used as a shorthand for the Serbian part of Serbiaand Montenegro. OECD 2005 PISA 2003 Data Analysis Manual: SPSS UsersPISA2003manual-ch1-SPSS.indd 125/06/05 20:49:36
in order to understand and evaluate scientific advice on food safety, an adult would not only needto know some basic facts about the composition of nutrients, but should also be able to apply thatinformation. The term “literacy” is used to encapsulate this broader concept of knowledge and skills.PISA is an ongoing survey with a data collection every three years.The first PISA survey was conductedin 2000 in 32 countries, using written tasks answered in schools under independently supervisedtest conditions following consistently applied standards. Another 11 countries participated in thesame survey in late 2001 or early 2002. The second survey was conducted in 2003 in 41 countries.Table 1.1 gives the list of participating countries for PISA 2000 and PISA 2003.PISA mainly assesses reading, mathematical and scientific literacy. For each data collection, oneof these three domains is chosen as the major domain, while the others are considered as minordomains. PISA 2000 focused on reading, while the major domain for PISA 2003 was mathematicalliteracy. About 70 per cent of the testing time is devoted to the major domain and the remainder isshared by the minor domains.Table 1.2 Assessment domains covered per data collectionPISA 2000Major domainReading literacyPISA 2003Mathematical literacyPISA 2006Scientific literacyMinor domainsMathematical literacy, scientific literacyReading literacyScientific literacyProblem solvingMathematical literacyReading literacyThe OECD’s Programme for International Student Assessment1In 2009, the major domain will again be reading literacy.WHAT MAKES PISA UNIQUE?PISA is not the first international comparative survey of student achievement. Others have beenconducted over the past 40 years, primarily developed by the International Association for theEvaluation of Educational Achievement (IEA) and by the Education Testing Service’s InternationalAssessment of Educational Progress (IAEP).These surveys have concentrated on outcomes linked directly to those parts of the curriculum thatare essentially common across the participating countries. Aspects of the curriculum unique to onecountry or a smaller number of countries have usually not been taken into account in the assessments,regardless of how significant those parts of the curriculum are for the countries involved.Key features associated with PISA include: Its policy orientation, with design and reporting methods determined by the need of governmentsto draw policy lessons; Its innovative “literacy” concept, which is conc
PISA 2003 Data Analysis Manual SPSS Users Programme for International Student Assessment OECD ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT PPISA2003manual-fauxtitre-SPSS.indd 1ISA2003manual-fauxtitre-
