Transcription
SEA-PLM 2019 Data User ManualThis report has been prepared by experts from the Australian Council for EducationalResearch: Louise Ockwell and Martin Murphy.1
Contents1Introduction . 32Exploratory analysis procedures . 43Replicate weights . 54Secondary analyses . 64.14.25Secondary analyses with PVs . 135.15.26Generating proficiency levels .25Analyses with proficiency levels .27School-level analyses . 348.18.28.39Computing group differences (and SEs) without PVs .19Computing group differences (and SEs) with PVs .21Proficiency levels . 257.17.28Computing a mean (and SE) with PVs .13Computing percentiles (and SEs) with PVs .15Computing group differences . 196.16.27Computing a mean and its standard error . 6Computing group percentages (and SEs).10Combining school- and student-level data .34Analyses with school-level data .35Teacher-level data .35Summary. 36References . 37Appendix A: Installation instructions of predefined macros for SPSS. 38Appendix C SPSS macro for computing standard error on a mean . 392
1 IntroductionThe South East Asian Primary Learning Metric (SEA-PLM) uses complexmethodologies that affect the way data should be analysed. These methodologies arenot included in the standard procedures within analytical software packages such asSAS or SPSS. The procedures described in this manual assume that data has beencollected using a complex sampling design requiring analysis with replicate weights andthat student performance has been estimated using plausible values (PVs).Standard data analysis software procedures assume a simple random sample ofstudents drawn from an exhaustive list. When the sampling design is more complex,such as with SEA-PLM, standard errors (SEs) are underestimated by these procedures,leading to non-significant results being reported as significant. The analysis proceduresdescribed in this manual aim to correct this.In addition, SEA-PLM uses plausible values (PVs) to estimate of student performancein reading, writing and maths, in order to gain unbiased population estimates of theseabilities. However, using PVs also requires more complicated analytical proceduresthan those provided by standard data analysis software.This manual describes the methodologies that should be employed to correctly reportresults from SEA-PLM, and how to use them in SPSS. The main focus of this manual ishow to correctly estimate a selection of commonly required population statistics (e.g.group means, group percentages, correlations and regression coefficients) and theirassociated SEs, so that they can be used for reporting purposes.Data used in the examples in this manual were taken from the SEA-PLM 2019database, which had not been finalised at the time of writing. It is possible that somevariable names used in the provided examples will differ slightly to those in the finaldatabase. Additionally, so that no single country’s data was able to be identified, arandom selection of cases was selected from each country’s data that was available atthe time of writing this manual. These were combined into a single fictitious entity,Country X, and used in the example analyses provided.3
2 Exploratory analysis proceduresA data analysis plan is of critical importance when deciding the first steps to take inany analysis. The data analysis plan outlines the proposed analyses to be undertakenand how these inform the research questions posed at the beginning of the assessmentdesign stage. Unfortunately, research questions may not always be directly addressedby the planned analyses, and it may be necessary to modify these analyses followingpreliminary investigations. Unexpected results may also direct research questions ina new direction that was not anticipated at the beginning of an assessment program.Data should always be weighted when running analyses, as unweighted data will returnbiased estimates. Datasets from complex surveys contain a final student weight, andusually a number of replicate weights (the number depends on the replication methodused when calculating the weights). The final student weight is used when estimatingpopulation outcomes. The replicate weights are used (with the final student weight)to estimate the standard error (SE) to reflect the uncertainty associated with theestimated outcomes. In addition to student weights, SEA-PLM datasets also include afinal teacher weight with associated replicate weights.Additionally, when analysing data with plausible values (see Section 5), a single analysisneeds to be repeated for each of the PVs provided, which is usually a set of five.1 Thesteps describing how secondary analyses using the complete set of PVs can be doneare described in Section 5.1 of this manual. For preliminary analyses, researchers canuse one PV instead of five to obtain quick unbiased estimates of population statistics,as well as their unbiased sampling variances. This is only sufficient to identifypotentially significant results that may warrant further investigation. However, thismethod does not estimate the measurement variance (see Section 5.1), and istherefore not sufficient for reporting purposes.While exploratory analysis procedures can be used to identify preliminary results,they should certainly not be used to produce any results that will be reported.Exploratory procedures are intended to direct further analyses, even if they identifykey findings that will almost certainly be reported, those findings must be confirmedusing the proper analytical procedures. For more detailed information aboutexploratory analysis procedures with large scale assessment data, refer to the PISAData Analysis Manaual (2nd Edition) [2]. The remaining sections in this manual willdescribe these procedures.1The choice of five PVs is based on the recommendations by Little, R. J. A., & Rubin, D. B. (1987).Statistical analysis with missing data. New York: J. Wiley & Sons [1].4
3 Replicate weightsSEA-PLM employed a two-stage stratified sample design. At the first stage, schoolswere organised (sorted) by stratification variables, such as socio-economicbackground and geographic location that were expected to be correlated with surveyoutcomes, and then systematically sampled with probability proportional to the grade5 enrolment at the school. At the second stage, an intact class (or classes) of studentswas selected. Because of the complexities of this two-stage sample design, simpleformulae for estimating the sampling variance do not exist. Instead, other methodsare used to estimate the sampling variance. SEA-PLM employed a replication methodknows as ‘Jackknife 2’ (JK2), which involves the calculation of the variance between anoutcome estimated from the full sample and that same outcome estimated from aseries of subsamples. Replicate weights have been added to the SEA-PLM databasewhich are used to define the subsamples used in the replication approach.Further information about replicate weights can be found in the PISA Data AnalysisManual (2nd edition) [2].5
4 Secondary analysesStratified two-stage sample designs such as SEA-PLM generally use a replicationmethod for the computation of sampling variance.The following sections describe the steps involved in computing various statistics ofinterest during the secondary data analysis stage, and their respective SEs, usingpredefined macros in SPSS.4.1 Computing a mean and its standard errorThe general procedure for computing any statistic of interest and its standard erroris essentially the same regardless of the type of statistic. Predefined macros have beendeveloped by the Australian Council for Educational Research (ACER) for the SPSSstatistical analysis software package. A set of installation instructions for thepredefined macros for SPSS is provided in Appendix A.The following example demonstrates how to estimate the average Asian identity(ASIDENT) scale score for students in Country X. The dataset used in the analysiscontains only students from Country X.Figure 1 SE on a mean1. Click to open dialog box2. Click to select data file3. Click to confirm file selection6
4. After filling in the details as shown, the“OK” button will be activated. Click to run.5. Or, click to paste syntax into SPSSsyntax file (.sps).6. To run the syntax in thedesignated SPSS syntax window, clickon the syntax window, highlight theset of pasted syntax and clickrun.7. If running the replicate add-in using the dialogbox, this may open when the run is complete.Click “Yes” to clear all logs from output. Or click“No” to check the logs for possible errors in theoutput window.7to
4.1.1 Interpreting the resultsTable 1 is an example of the structure of the SPSS output data computed by the SPSSmacro. As a reference, a hardcopy of the macro for computing the SE on a mean hasbeen provided in Appendix B.Table 1 Output data for SE on a meanstatistic1MEANASIDENT46.969se ASIDENTN cases.207405369966.26NU casesNU psu4743337A confidence interval (CI) can then be built around this mean. With a risk of type Ierror equal to 0.05, usually denoted α, a two-sided 95% CI will be equal to:[46.969 (1.96 0.207); 46.969 (1.96 0.207)][46.563; 47.375]A 95% CI has a 0.95 probability of containing the true value of the population mean.In other words, there are 5 chances out of 100 that the calculated CI will fail tocapture the population mean. It also indicates that the population mean for ASIDENTfor students in Country X is significantly lower than, for example, 50.00 (theinternational mean), as this number is not included in the CI. Therefore, it can beconcluded that students in Country X report significantly lower levels of Asian identitythan students internationally.The following example shows how to add a grouping variable (gender) to the previousanalysis (between steps 3 and 4) to estimate the average Asian identity (ASIDENT)scale score for subgroups of students (males and females) in Country X.8
Figure 2 SE on a mean by subgroupTable 2 Output data for SE on a mean by groupGenderstatisticASIDENTse ASIDENTN casesNU casesNU psu11 FemaleMEAN46.319.269094193890.22250433322 MaleMEAN47.685.277295176076.042239329The 95% CI for female students from Country X will be equal to:[46.319 (1.96 0.269); 46.319 (1.96 0.269)][45.792; 46.846]The 95% CI for male students from Country X will be equal to:[47.685 (1.96 0.277); 47.685 (1.96 0.277)][47.142; 48.228]9
!! CAUTION !!Researchers may be tempted to use this methodology (comparing CIs) toascertain whether subgroups have significantly different mean scores on aparticular measure (e.g. whether males or females have higher ASIDENTlevels). However, this method can fail to capture certain sources ofvariance. Section Error! Reference source not found. provides thecorrect way to compute the SE for comparisons between two or moresubpopulation groups4.2 Computing group percentages (and SEs)For reporting categorical variables such as gender, the statistic of interest is usuallythe percentage of students per category. Predefined SPSS macros will compute thepercentages and their respective SEs for each category of the gender variable. Thesteps for using the SPSS predefined macro for SE on group percentages aredemonstrated below.Figure 3 SE on group percentages1. Click to open dialog box10
2. Click to select data file3. Click to confirm file selection4. Select REMOVE missing if there are missing values inthe data that do not need to be included in the analysis.5. After filling in the details as shown, the“OK” button will be activated. Click to run.6. Or, click to paste syntax intoSPSS syntax file (.sps).7. To run the syntax in thedesignated SPSS syntaxwindow, click on the syntaxwindow, highlight the set ofpasted syntax and clickto run.11
8. If running the replicate add-in using the dialogbox, this may open when the run is complete.Click “Yes” to clear all logs from output. Or click“No” to check the logs for possible errors in theoutput window.Table 3 is an example of the structure of the SPSS output data computed by the SPSSmacro. This example shows the percentages (STAT) and the associated SEs of maleand female students from Country X in the SEA-PLM 2019 dataset.Note that when estimating group percentages, missing values for the variable ofinterest (in this case gender) should be removed using the checkbox in the dialog boxabove (step 3). This only works if the missing values are declared on the variable viewscreen in SPSS. The proportion of missing values would not normally be explicitlyreported for an analysis such as this. Removing them from the analysis ensures thatthe reported percentages estimated by the macro will always add to 100.Table 3 Output data for SE on group percentages macroGenderSTATSEN casesNU casesNU psu11 Female52.079.938002198409257333322 Male47.921.9380021825672337330The last three columns of output in Table 3, N cases, NU cases and NU psu list theweighted number of cases, the unweighted number of cases, and the unweightednumber of primary sampling units (i.e. schools), respectively, in each of the gendergroups.As with estimating the SE of a mean described above, 95% CIs can be constructedaround the percentages estimated by the macro. However, this procedure can stillonly be used to estimate and report the percentages of students in each group, notto evaluate whether the percentages of students in each group are significantlydifferent from each other. The procedure to correctly make these comparisons willbe described in Section Error! Reference source not found.12
5 Secondary analyses with PVsPlausible values (PVs) allow unbiased estimation of student performance for a givenpopulation. PVs should always be used when the focus of an assessment is on providingresults that describe the performance at the population level (i.e. how students inCountry X performed in mathematics in 2019). For a detailed description of why PVsshould be used over individual estimates (such as weighed maximum likelihoodestimates and expected a-posteriori estimates) refer to the PISA Data Analysis Manual(2nd Edition) [2].Using PVs to estimate the SE associated with the statistic of interest allowscomputation of an additional source of variability in student scores: the uncertaintydue to the lack of precision in the measurement test, known as the measurementvariance. SEs estimated with PVs are computed using both the sampling variance andthe measurement variance.5.1 Computing a mean (and SE) with PVsThere are SPSS macros specially developed for computing estimates and theirassociated SEs using PVs. The steps for using the SPSS predefined macro for SE on amean (PVs) are demonstrated below.Figure 4 SE on a mean (PVs)1. Click to open dialog box2. Click to select data file3. Click to confirm file selection13
4. After filling in the details as shown, the“OK” button will be activated. Click to run.5. Or, click to paste syntaxinto SPSS syntax file (.sps).6. To run the syntax in thedesignated SPSS syntaxwindow, click on the syntaxwindow, highlight the set ofpasted syntax and clickto run.7. After clicking “OK” or “Paste” this dialog boxwill appear confirming the set of PVs to beanalysed. If correct, click “Yes” to continue. Orclick “No” to make changes.14
8. If running the replicate add-in using the dialogbox, this may open when the run is complete.Click “Yes” to clear all logs from output. Or click“No” to check the logs for possible errors in theoutput window.Table 4 Output data from SE on a mean (PVs)statistic1MEANAs can be seen inSTAT304.209SEN cases.684346NU cases3809764910NU psu337Table 4 Output data from SE on a mean (PVs), the average performance of studentsin Country X on the Mathematics scale can be reported as 304 points, with a SE of0.68. This same procedure can be used to estimate and report the averageMathematics achievement for subgroups of students (e.g. by state/jurisdiction, gender)by specifying a grouping variable in the variable selection window after step 3 (see!! CAUTION !!When analysing data with PVs, researchers may be tempted to computethe mean of the five PVs at the student level, before further analysis.However, computing the mean of the five PVs at the student level ismore or less equal to the expected a-posteriori (EAP) estimate, which isbiased [2].Figure 4).As described in Section 2 of this manual, analysing one PV instead of five providesunbiased population estimates as well as the unbiased sampling variance but does notestimate the measurement variance. This quicker analysis can be used to quicklyidentify results that may warrant further investigation, but the full set of PVs shouldalways be used for reporting purposes.5.2 Computing percentiles (and SEs) with PVsAn alternative way to report results for continuous variables (such as Mathematicsachievement), is to calculate percentiles (i.e. the value(s) below which a givenpercentage(s) of students scored). This can provide a representation of thedistribution of student scores on the given measure.Predefined SPSS macros calculate user-defined percentiles and associated SEs. Thesteps for using the SPSS predefined macro for SE on percentiles are demonstratedbelow.15
Figure 5 SE on percentiles1. Click to open dialog box2. Click to select data file3. Click to confirm file selection16
4. After filling in the details as shown, the“OK” button will be activated. Click to run.5. Or, click to paste syntax into SPSSsyntax file (.sps).6. To run the syntax in thedesignated SPSS syntax window, clickon the syntax window, highlight theset of pasted syntax and clickrun.to7. After clicking “OK” or “Paste” this dialogbox will appear confirming that the set ofPVs to be analysed. If correct, click “Yes” tocontinue. Or click “No” to make changes.8. If running the replicate add-in using the dialog box,this may open when the run is complete. Click “Yes”to clear all logs from output. Or click “No” to checkthe logs for possible errors in the output window.The structure of the output data file is given in Table 5. This example shows thecalculated scale scores that correspond to the 5th, 25th, 50th, 75th and 95thpercentiles for Mathematics for students from Country X in the 2019 SEA-PLMdatabase.17
Table 5 Output data for SE on percentiles macro1STAT.5SE.5264.5401.14450STAT.25SE.25287.485 .91293STAT.50SE.50304.507 .92705STAT.75SE.75321.284 .98078STAT.95342.965SE.951.68114As can be seen in Table 5, the SE is larger for the 95th percentile (1.68) than the lowerones (between 0.91 and 1.14). This is due to smaller percentages of students achievingscores at the upper level, and can be interpreted as there being less certainty orprecision in the estimates for this level.18
6 Computing group differencesDue to the design of SEA-PLM 2019 sample, the SPSS Compare Means procedure forcomputing a difference between two means underestimates the associated SE. Incertain circumstances (such as when the groups being compared are independent),the SE can be calculated manually without needing to run the predefined SPSS macro.Refer to the PISA Data Analysis Manual (2nd edition) [2] for these formulas and thecircumstances in which they can be applied. The predefined SPSS macros developedby ACER can easily estimate the correct SE when comparing the difference betweentwo means, both with and without PVs.6.1 Computing group differences (and SEs) without PVsTo compute the SE on a difference without PVs, consider the earlier example ofASIDENT scores for male and female students in Country X (Figure 2). The macrofor computing SE on a mean was used to estimate the mean scores for both groupsof students (the results are repeated in Table 6). The steps for using the SPSSpredefined macro for SE on a mean by group are demonstrated in Section 4.1.Table 6 Output data for SE on mean macro for ASIDENTGenderstatisticASIDENTse ASIDENTN casesNU casesNU psu11 FemaleMEAN46.319.269094193890.22250433322 MaleMEAN47.685.277295176076.042239329On average, male students in Country X appear to score higher on ASIDENT thanfemale students. However, the difference cannot be fully interpreted until the SE ofthe difference is calculated.As mentioned above, the SPSS macros for computing SE on a difference easily calculatethe unbiased estimate – which is the recommended estimate for comparing resultsbetween groups. The steps for using the SPSS predefined macro for SE on a differencewithout PVs are demonstrated below.Figure 6 SE on a difference without PVs1. Click to open dialog box19
2. Click to select data file3. Click to confirm file selection4. After filling in the details as shown, the“OK” button will be activated. Click to run.5. Or, click to paste syntax into SPSS syntax file (.sps).6. To run the syntax in the designated SPSS syntax window, click on the syntaxwindow, highlight the set of pasted syntax and click20to run.
8. If running the replicateadd-in using the dialogbox, this may open whenthe run is complete. Click“Yes” to clear all logs fromoutput. Or click “No” tocheck the logs for possibleerrors in the outputwindow.7. After clicking “OK”or “Paste” this dialogbox will appearconfirming whetherthe dependentvariable is a singlevariable or a set ofPVs. If correct, click“Yes” to continue. Orclick “No” to makechanges.The output data for this macro is presented in Table 7.Table 7 Output data for SE on a difference without PVs (ASIDENT)DIFFSTATse1GENDER.1 2-1.366.3531192GENDER.1 2-1.366.353119As can be read from the above table, the average difference in Asian identity scalescores between male and female students in Country X is 1.366, with a SE of 0.353.As the 95% CI around this estimate [-2.058, -0.674] does not include 0, it can beconcluded that the two groups are significantly different. Additionally, since the macrocomputed this difference by subtracting the male (gender 2) mean score from thefemale (gender 1) mean score, it can be concluded that, on average, male students inCountry X score just over one point higher than female students on the Asian identityscale. This should be reported in the context of the scale, which has an internationalmean of 50 and standard deviation of 10, so a one point difference is very small.6.2 Computing group differences (and SEs) with PVsThe steps for using the SPSS predefined macro for SE on a difference with PVs aredemonstrated below.21
Figure 7 SE on a difference with PVs1. Click to open dialog box2. Click to select data file3. Click to confirm file selection22
4. After filling in the details as shown, the“OK” button will be activated. Click to run.5. Or, click to paste syntax into SPSSsyntax file (.sps).6. To run the syntax in the designatedSPSS syntax window, click on thesyntax window, highlight the set ofpasted syntax and clickto run.7. After clicking “OK” or “Paste” thisdialog box will appear confirmingwhether the dependent variable is asingle variable or a set of PVs. Ifcorrect, click “Yes” to continue. Orclick “No” to make changes.8. If running the replicate add-in using the dialog box,this may open when the run is complete. Click “Yes”to clear all logs from output. Or click “No” to checkthe logs for possible errors in the output window.23
Table 8 shows the structure of the results of these macros when estimating the SE onthe difference in performance in Mathematics between male and female students inCountry X.Table 8 Output data for SE on a difference (PVs) on Mathematics scaleGENDERSTATSE11 22.1001.02608921 22.1001.026089As can be read from Table 8, the difference in Mathematics mean scores betweenmale and female students in Country X is 2.1, with a SE of 1.02. The 95% CI [0.09,4.11] does not include zero, therefore it can be concluded that the two groups aresignificantly different. The macro computed this difference by subtracting the malemean score from the female mean score, it can be concluded that, on average, femalestudents are performing about 2 points higher than male students on the Mathematicsscale. This analysis can be repeated for other subgroups (e.g. state/jurisdiction) in theSEA-PLM 2019 database2.2If family-wide or simultaneous multiple comparisons are being made, a Bonferroni correction on theα level should be made. Refer to the PISA Data Analysis Manual (2nd edition) [2] for more information.24
7 Proficiency levelsProficiency scales allow the reporting of assessment results to be more accessible topolicymakers and educators. They should be developed for all assessment domains.The scales should be divided according to levels of difficulty and performance, so thata ranking of student performance can be obtained, as well as a description of the skills(as measured by the assessment test items) associated with each proficiency level.Higher proficiency levels should be associated with tasks of increasing difficulty.Further information about proficiency levels can be found in the PISA Data AnalysisManual (2nd edition) [2]. This section shows how to derive proficiency levels from adatabase and how to use them.7.1 Generating proficiency levelsProficiency levels can be derived from PVs provided in a database. Predefined cutpoints that frame the proficiency levels in each domain need to have been establishedprior to analysis. These cu tpoints are then applied to each PV given to each studentin a database, so that five plausible proficiency levels are assigned to each studentrespectively according to their five PVs. The SPSS macros developed to generateproficiency levels make this process simple and easy to perform. The steps for usingthe SPSS predefined macro for generating proficiency levels are demonstrated below.Figure 8 Generation of proficiency levels1. Click to open dialog box2. Click to select data file3. Click to confirm file selection25
4. After filling in the details as shown,the “OK” button will be activated. Clickto run.5. Or, click to paste syntax into SPSSsyntax file (.sps).6. To run the syntax in the designatedSPSS syntax window, click on the syntaxwindow, highlight the set of pasted syntaxand clickto run.7. After clicking “OK” or “Paste”this dialog box will appearconfirming a location to save anew data file that includes thegenerated proficiency levelvariables. Click “Browse” tospecify a new location. Click“OK” to continue.Example SPSS syntax for the generation of plausible proficiency levels in Mathematicsis provided in Figure 9.26
Figure 9 SPSS syntax for the generation of proficiency levels in MathematicsGET FILE "C:/TEMP/ANALYSIS.sav".DO REPEATSTUDEST MATHPV1 MATHPV2 MATHPV3 MATHPV4 MATHPV5/PROFLEV PL MATHPV1 PL MATHPV2 PL MATHPV3 PL MATHPV4PL MATHPV5.If (StudEstIf (StudEstIf (StudEstIf (StudEstIf (StudEstIf (StudEstIf (StudEstIf (StudEstEND REPEAT. 269) ProfLev 2. 269 & StudEst 283 & StudEst 297 & StudEst 311 & StudEst 325 & StudEst 339 & StudEst 353) ProfLev 9.283)297)311)325)339)353)ProfLev 3.ProfLev 4.ProfLev 5.ProfLev 6.ProfLev 7.ProfLev 8.FORMATS PL MATHPV1 PL MATHPV2 PL MATHPV3 PL MATHPV4PL MATHPV5 (F1.0).SAVE OUTFILE "C:/TEMP/ANALYSIS PL.sav".In this syntax, two vectors are defined. The first, labelled STUDEST, includes the fivePVs for the Mathematics scale. The second, labelled PROFLEV, will create five newvariables, labelled PL PV1MATH to PL PV5MATH. The iterative process will recodeeach PV variable into a new variable with eight categories labelled 2 to 9 formathematics (no data existed in 2019 for proficiency levels below 2).7.2 Analyses with proficiency levelsOnce generated in the dataset, proficiency levels can be used to compute thepercentage of students at each proficiency level.7.2.1 Computing the percentage of students in each proficiency level (andSEs)The computation of the percentage of students at each proficiency level and thecorresponding SEs is similar to the computation of a mean estimate with PVs and itscorresponding SE.SPSS macros have been developed for computing the percentage of students at eachproficiency level as well as its respective SE in one run. The steps for using the SPSSpredefined macro for SE on percentages by proficiency level are demonstrated below.27
Figure 10 SE on perce
box, this may open when the run is complete. Click "Yes" to clear all logs from output. Or click "No" to check the logs for possible errors in the output window. 5. Or, click to paste syntax into SPSS syntax file (.sps). 6. To run the syntax in the designated SPSS syntax window, click on the syntax window, highlight the
