WIXLIB

Growth NormsNorms and Instructional ContentThe real power of MAP and MPG assessments is inmeasuring growth over time and comparing eachstudent’s measured growth to growth norms. NWEAmakes this easy for educators with the ConditionalGrowth Index (CGI), which allows for comparisonsbased on subject, grade level, instructional weeks,and students’ starting RIT score (the score given bythe MAP test). Using growth norms, teachers canunderstand not only that a student grew, but howmuch that student grew in comparison to otherstudents who had similar starting RIT scores andweeks of instruction. Accounting for instructionalprogress in growth norms is something only NWEAoffers. Growth norms provide teachers the necessarycontext for setting individual student growth targets,a powerful way to help students take ownershipof their own learning. At the school level, growthnorms express the progress of groups of studentsand can be useful for making inferences aboutprograms and instructional approaches.Over the last couple of years, NWEA has leveragedits norms data to help teachers differentiateinstruction by linking our data to content in both openeducational resources like Khan Academy and otherinstructional content that schools have purchased.These links can create individualized learning pathsfor each student by suggesting content that is in thestudent’s zone of proximal development. Equallyimportant, these individualized learning paths canhelp parents and guardians support students at home.6 John Wood is Senior Analystfor Assessment and Educationon the Academic Servicesteam at NWEA. He has workedextensively with the CommonCore State Standards (CCSS) sincethey were issued in draft form,and frequently gives presentations to NWEA partnerson the relationship of MAP to the CCSS. He providessubject matter expertise to many teams at NWEA, aswell as to partners. He is constantly working on newways to connect MAP data to instructional content inorder to help educators create assessment-informedlearning paths for each student.Partnering to Help All Kids Learn NWEA.org 503.624.1951 121 NW Everett St., Portland, OR 97209

Making Sense of StandardError of MeasurementNate Jensen, Ph.D., Research Scientist, NWEAImagine for a moment that you’re starting a newexercise and diet program to lose ten pounds in thenext year. In order to track your weight loss progress,you decide to purchase a new scale. The store,however, only carries two different styles of scales—thefirst is much cheaper, but only guarantees its estimatesof weight within 5 pounds (a range of 10 pounds),whereas the second one, while more expensive,guarantees its estimates to within 0.1 pounds (arange of 0.2 pounds). Which scale would you buy?The answer seems obvious. It wouldn’t be veryhelpful if every time you stepped on the scale youcouldn’t be certain if your weight was actually fivepounds heavier (bummer), five pounds lighter(victory!), or somewhere in between. In order totrack your progress over the weeks and months,it would be important to have a scale that wouldreturn an accurate estimate of what you actuallyweigh every time you step on the scale.This same principle holds with student test scores.If you want to track student progress over time,it’s critical to use an assessment that provides youwith accurate estimates of student achievement—assessments with a high level of precision. When werefer to measures of precision, we are referencingsomething known as the Standard Error ofMeasurement (SEM).Before we define SEM, it’s important to rememberthat all test scores are estimates of a student’s truescore. That is, irrespective of the test being used,all observed scores include some measurementerror, so we can never really know a student’s actualachievement level (his or her true score). But wecan estimate the range in which we think a student’strue score likely falls. And, in general, the smaller therange, the greater the precision of the assessment.7 SEM, put in simple terms, is a measure of precisionof the assessment—the smaller the SEM, the moreprecise the measurement capacity of the instrument.Consequently, smaller standard errors translate tomore sensitive measurements of student progress.On MAP assessments, student RIT scores are alwaysreported with an associated SEM, with the SEM oftenpresented as a range of scores around a student’sobserved RIT score. On some reports, it lookssomething like this:Student Score Range: 185-188-191So what information does this range of scoresprovide? First, the middle number tells us thata RIT score of 188 is the best estimate of thisstudent’s current achievement level. It also tells usthat the SEM associated with this student’s score isapproximately 3 RIT—this is why the range aroundthe student’s RIT score extends from 185 (188 - 3)to 191 (188 3). A SEM of 3 RIT points is consistentwith typical SEMs on the MAP tests (which tend to beapproximately 3 RIT for all students).The observed score and its associated SEM can beused to construct a “confidence interval” to anydesired degree of certainty. For example, a rangeof 1 SEM around the observed score (which, inthe case above, was a range from 185 to 191) isthe range within which there is a 68% chance thata student’s true score lies, with 188 representingthe most likely estimate of this student’s score.Intuitively, if we specified a larger range aroundthe observed score—for example, 2 SEM, orapproximately 6 RIT—we would be much moreconfident that the range encompassed the student’strue score, as this range corresponds to a 95%confidence interval.Partnering to Help All Kids Learn NWEA.org 503.624.1951 121 NW Everett St., Portland, OR 97209

So, to this point we’ve learned that smaller SEMsare related to greater precision in the estimationof student achievement, and, conversely, that thelarger the SEM, the less sensitive is our ability todetect changes in student achievement. But why isthis fact important to educators?An example of how SEMs increase in magnitude forstudents above or below grade level is shown in thefollowing figure, with the size of the SEMs on anolder version of the Florida 5th grade reading testplotted on the vertical axis relative to student scalescores on the horizontal axis.5 What is apparent fromthis figure is that there is a tremendous amountof imprecision associated with the test scores forlow- and high-achieving students. In this example,the SEMs for students on or near grade level (scalescores of approximately 300) are between 10 to15 points, but increase significantly for studentsthe further away they get from grade level. Thispattern is fairly common on fixed-form assessments,with the end result being that it is very difficultto measure changes in performance for thosestudents at the low and high end of the achievementdistribution. Put simply, this high amount ofimprecision will limit the ability of educatorsto say with any certainty what the achievementlevel for these students actually is and how theirperformance has changed over time.If we want to measure the improvement of studentsover time, it’s important that the assessment usedbe designed with this intent in mind. And in orderto do this, the assessment must measure all kidswith similar precision, whether they are on, above,or below grade level. Recall, a larger SEM means lessprecision and less capacity to accurately measurechange over time, so if SEMs are larger for high- andlow-performing students, this means those scoresare going to be far less informative, especially whencompared to those students who are on grade level.Educators should consider the magnitude of SEMsfor students across the achievement distributionto ensure that the information they are using tomake educational decisions is highly accurate for allstudents, regardless of their achievement level.Grade 5 Reading SEMStandard Error of Measurement100Figure 1: Standard Errors ofMeasurement by Student AchievementLevel, 5th Grade Reading, FloridaComprehensive Assessment Test 500Scale Score8 Partnering to Help All Kids Learn NWEA.org 503.624.1951 121 NW Everett St., Portland, OR 97209

Of course, the standard error of measurement isn’tthe only factor that impacts the accuracy of the test.Accuracy is also impacted by the quality of testingconditions and the energy and motivation thatstudents bring to a test. In fact, an unexpectedlylow test score is more likely to be caused bypoor conditions or low student motivation thanto be explained by a problem with the testinginstrument. To ensure an accurate estimate ofstudent achievement, it’s important to use asound assessment, administer assessments underconditions conducive to high test performance, andhave students ready and motivated to perform.In short, just as you would want an accurate scale ifyou were tracking your progress in an exercise anddiet program, it is critical that the assessment usedin your school to track student progress providesachievement estimates for all students that areas precise as possible. And, as we have discussedin this paper, the key metric on which to focus to9 determine if that is the case is the standard error ofmeasurement. The SEM of a test score isn’t alwaysthe easiest statistical term to understand. But, if youremember that smaller SEMs lead to more accurateestimates of student achievement and growth, youwill be better able to interpret and evaluate theperformance of your students.Prior to joining NWEA threeyears ago, Nate Jensen put hisskills to use as a teacher and asa senior research associate atthe office for education policyat the University of Arkansas.Over a decade of experiencein the education field has given him a profoundappreciation of the importance of data in theclassroom. His passion lies in helping teachers andschool leaders understand how to use metrics todeeply understand what each student needs in orderto grow.Partnering to Help All Kids Learn NWEA.org 503.624.1951 121 NW Everett St., Portland, OR 97209

Measuring Growth with a Stable ScaleA strong, stable scale is key to accurately measuring student growthNicole A. Zdeb, Senior Manager, Academic Services, NWEAAt NWEA, we are passionate about growth and whatit means in the educational journeys of the studentswe serve. Growth means that learning is happening.Students are acquiring the concepts and skills theyneed to flourish in the world and be the authors oftheir own stories. Measuring that growth is crucial,and the stability of a measurement scale over time isnecessary to measure growth accurately. But what isa scale and what does stability mean?Measuring Latent TraitsScale is one of those words that befuddles learnersof English because there are various and unrelatedmeanings, from the covering on certain animals, to acause of blindness, to an instrument with graduateddegrees. When we talk about scales in educationalmeasurement, we’re talking about something akinto the latter definition: a construct that indicates thedegree of student ability in a certain area, such asmathematical reasoning. We call this ability a latenttrait because we can’t see the amount of a student’smathematical reasoning ability directly, like wecan see a student’s hair color or the shoes she iswearing. Even though we can’t physically interactwith it, we know mathematical reasoning is a realthing. It is similar to a psychological state such ashappiness—it exists in the interiority of a person.Quantifying, or measuring, latent traits is a subtleendeavor to which we bring to bear the power ofstatistical modeling.To measure a latent trait, we must first elicit it. Weelicit evidence of a latent trait through the use ofinstruments such as test items, performance tasks,10 and writing samples. The evidence is a proxy, orrepresentative, of the trait itself. We infer that a personis happy by certain indicators such as body language,laughter, or smiling. Likewise, we infer the degree ofa latent trait such as mathematical reasoning by theevidence we get from eliciting that trait.Stable, Equal Interval ScaleOnce we elicit evidence of a latent trait, we measurethat evidence by using a scale. The scale used byMAP is an equal interval measurement called theRIT scale. An equal interval scale provides a specifickind of information about order, namely that thereis the same distance between points on the scale,or the same amount of underlying quality. Anotherexample of an equal interval scale is a thermometer.The value of an equal interval scale is that it isconsistent and objective; such qualities help make ascale stable over time.Scale stability means that scales maintain theirmeasurement characteristics, allowing forcomparisons of assessment scores among groups ofstudents, growth estimates, and longitudinal studies.For example, a RIT score of 215 in 1975 would beequivalent to a RIT of 215 in 1995. Maintainingthis stability of scale—so that our scale retains itsmeasurement properties in exactly the same way yearafter year—is a crucial part of what we do at NWEA.Guarding Against Scale DriftNot all educational measurement scales areinherently stable. Many scales can and do drift overtime, meaning that the underlying quantity of what isPartnering to Help All Kids Learn NWEA.org 503.624.1951 121 NW Everett St., Portland, OR 97209

being measured shifts. Item calibrations can becomemore difficult or less difficult as time goes on. This isdisastrous for measuring growth or making any sortof longitudinal comparisons. There are a number ofreasons a scale might drift over time: Curricular changes (including pedagogicalapproaches) Changes in standards or what is valued in acontent area Changes in testing populations over time Changes in the stakes of a test or assessment,its social context, or the meanings andconsequences attached to it Changes in the intended purposes ofthe assessment11 Guarding our scale’s integrity in the face of suchchanges allows NWEA to maintain a scale withremarkable stability over time. This stability givesus confidence in the educational decisions that aremade as a result of the data gleaned from our MAPassessments. We are aware of the importance ofmaintaining scale stability, so we make it a priorityto monitor our scales for drift. The scale connects itall—items to scores to students. We ensure our dataare the gold standard in measuring growth becausewe know that every student can grow. Reporting thatgrowth accurately makes a difference in the lives ofstudents across the country and world.Making a difference in the lives ofchildren, teachers, and familiesis what has driven Nicole Zdeb’swork at NWEA over the pastdecade. As a specialist in contentand curriculum, she’s broughther prodigious experience as aneducator to bear on the Academic Services departmentof NWEA. Outside the office, Nicole devotes her timeto supporting student literacy, and has had poemspublished in numerous literary journals.Partnering to Help All Kids Learn NWEA.org 503.624.1951 121 NW Everett St., Portland, OR 97209

How Deep Is the Pool—andWhy Does it Matter?How item pool depth affects your assessment’s accuracyApril Roethel, Senior Analyst, Assessment and Education, NWEATo dive safely into a swimming pool, the watermust be deep enough to extend past your headand your feet. Similarly, to ensure the validity andeffectiveness of computer adaptive tests (CATs), theitem pool must be deep enough to stretch above andbelow a student’s entry point.An essential part of a CAT is a well-constructed itempool. One crucial requirement of a well-designeditem pool is that it includes enough items to enablethe building of numerous individualized tests thatalign to students’ varying ability levels—this is whatwe mean by the depth of the pool. The ideal itempool should also include enough breadth to coverthe scope of the content domain.Why is item pool depth so important incomputer adaptive assessments?Unlike paper-and-pencil tests that are fixedin content, CATs adapt to individual studentperformance. They get harder or easier dependingon how a student is performing on the test, whichrequires a deep item pool from which many differenttests can be drawn.A student’s grade level is not necessarily his or herinstructional readiness point; this is why a CAT mustadapt to measure on-, above-, and below-gradeabilities. An assessment that informs educatorsabout each student’s instructional readiness drawson content that spans across grades. A deep itempool can provide this because it will be stocked withitems that correspond to many different grade levels.12 How many items are enough toassess a skill?A common question that comes up whenevaluating an assessment tool is, “How manyitems should it include?”The appropriate size of the item pool depends onfour main factors.Precision is the first factor to consider, as it relates tothe “estimate of student achievement that is desired.”6The more precision you desire, the larger your itempool needs to be. If you are aiming to get just a roughestimate, you can use a smaller item pool. Range is another significant factor. How broador narrow is the range of achievement to bemeasured? A larger item pool will be requiredfor assessment that is very broad, since it willinclude items with a large range of difficulty.For example, if an assessment is being usedto measure students’ performance at multipledepth of knowledge (DOK) levels, it will requirea greater range of items than an assessmentconcerned with only one DOK level. Stakes are a third factor that will determine theitem pool size requirement. If a CAT is very highstakes, students might be more likely to gamethe test. Large item pools improve the chancethat examinees receive a different set of testitems for every test administration, making itimpossible to cheat the system. Number of times a CAT is administered is afourth factor of importance. If an assessmentis administered to the same students multiplePartnering to Help All Kids Learn NWEA.org 503.624.1951 121 NW Everett St., Portland, OR 97209

times a year, for instance, the item pool must belarge enough to ensure that a student doesn’tsee any item more than once.The goal is to have a sufficient number of items ineach desired content area to assemble an individualtest with the balanced content coverage required bythe test.7The importance of field testingand calibrationA deep pool of items isn’t very valuable if the i

the MAP test). Using growth norms, teachers can understand not only that a student grew, but how much that student grew in comparison to other students who had similar starting RIT scores and weeks of instruction. Accounting for instructional progress in growth norms is something only NWEA offers. Growth norms provide teachers the necessary