WIXLIB

NBME CUSTOMIZEDASSESSMENT SERVICES PROGRAM GUIDECreate ExaminationsTailored to Your Curriculum

CONTENTS03Overview05Policies for Use of Customized Assessment Services05Access06Key Medical Staff Roles06Faculty Participants06User Account Administrator09Test Construction Activities09Additional Features12Score Reporting12Scaling Group Application12Sample Score Report ComponentsCopyright 2007, 2011-2016, 2019-2021 NBME. All rights reserved.2

OVERVIEWNBME offers the Customized Assessment Services (CAS) program to enablemedical school faculty to build high-quality examinations tailored to local curriculausing NBME test questions. We offer pre-clerkship and clerkship content in CAS tosupport integrated clinical curricula at medical schools.be interpreted in light of other available information.Likewise, curriculum evaluation cannot be based onassessment results alone. The quality of teachingcan and should be evaluated by frequent peerobservation and student feedback, not inferred solelyfrom the level of assessment results.CAS helps support local organ-system, disciplinebased, problem-based or other curricularapproaches. To do so, NBME has assembled a largepool of test items covering the topics commonlytaught and tested in coursework. These test itemshave been permanently transferred from the UnitedStates Medical Licensing Examination (USMLE )or NBME Subject Examination pool, and thereforehave undergone the same rigorous quality assuranceprotocols as all USMLE test items.The success of this program depends on vigorousadherence to NBME policies and protocols, includingthe implementation of the strictest securitymeasures.Of equal importance, NBME is committed toprotecting the high-quality reliability, validity, andconfidentiality of the examinations it creates andmanages. We work with our partners, customers, andexaminees to continuously identify the right solutionsto improve exam security. We have a customizedsecurity approach for each of our assessments.NBME collaborates with partners, clients, andtest administrators to meet agreed upon securitystandards.Policies for Use of CustomizedAssessment ServicesParticipating schools must agree to use CustomizedAssessment Services in compliance with NBMEpolicies and procedures. Moreover, participatinginstitutions must understand that NBME customizedassessments are intended to complement othersources of information about the educationalprogress of students. The assessment results should3

AccessGo online to the Customized Assessment ServicesFees page to view current fees for the service.CAS is available through the MyNBME ServicesPortal, the primary source for authorized medicalschool representatives to obtain information abouttheir students. Access enables a variety of activitiesrelated to USMLE, NBME Subject Examinations, andother services.Logging into CASTo access the CAS system, faculty will follow NBME’ssecurity process. This validation of identity currentlyrequires an initial configuration of a cell phone,landline, or tablet. Two-factor authentication isalso used to access CAS’ online item analysis, whichis available for review when the score reports areposted to the MyNBME Services Portal.An annual subscription fee for CAS includes:u program setupu support for test construction and testadministration activitiesTest AdministrationCAS examinations must be proctored and areadministered either locally at your institution orremotely. CAS tests are not available for delivery atPrometric test centers. For more details about TestAdministration, please go to the NBME ExaminationsRules and Conduct page on the NBME website.u construction of all CAS examinations for studentsu score reports, individual student performanceprofiles, andu item analyses for each CAS examinationFaculty ParticipantsKey Medical School Staff RolesListed below are the key players that supportthe construction and administration of theCustomized Assessment Services program.Any faculty member may participate in the testconstruction process; however, those who willreview NBME test items must agree to the termsand conditions when first accessing the CustomizedAssessment Services system. Faculty membersshould review the terms and conditions carefullyprior to deciding whether to agree and comply ornot. Individuals who do not agree to the terms andconditions will not be able to access the CustomizedAssessment Services system.Executive Chief ProctorThe Executive Chief Proctor (ECP), nominated bythe dean, has access to the MyNBME Services Portaland has overall responsibility for ordering servicesonline, coordinating test administration activities, andassuring that the security of the test items reviewedby faculty and seen by examinees is maintained.Specific ECP responsibilities are outlined below.User Account Administrator (UAA)The ECP plays a key role in all local aspects of theCAS program.The UAA creates and manages accounts in theMyNBME Services Portal for eligible and qualifyingstaff who need to access the test constructionsystem, score reports, or item analysis data.Order a SubscriptionThe first step is for the ECP to order a subscriptionto the service and identify the estimated month andyear that test construction activities will begin. NBMEmust receive payment for the subscription fee beforethe subscription period starts unless it is paid bycheck. The subscription becomes active on the firstday of the month entered as the starting point fortest construction activities.OTHER RESOURCES: For moreinformation about all NBME examinations,please visit the Assess & Learn pageon the NBME website.4

TEST CONSTRUCTION ACTIVITIESThe main menu of the customized assessmenttest construction system is displayed below.u Item Preview and SelectionThe components of the test construction processinclude:Continue to the Item Preview and Selection stageto review and select your test itemsw Review your item content.u Build Your Blueprintw Create a selection of test items using a varietyof filters (such as organ system, discipline,vignette type, and medication)uand/orA variety of attributes (such as itemdifficulty and associated source exam,default score categories, and contentoutline categorization) are provided.u You may also filter and sort test items invarious ways.w Search by keyword for specific concepts andadd additional filters to refine your searchw Drop and replace unwanted test items usingthe Refresh button, as necessaryw Assign a target number of test items for eachselectionw Create and assign any custom score categories,if desiredw Finalize your test5

u Review the Testw The test should be reviewed in advance of the testadministration as a final content check.u Order the TestwAfter the test has been finalized, it can be ordered foradministration by the ECP or other medical school facultywith the Ordering Services permission. Your exam will belisted under the ‘Custom’ header in your list of exams inthe ordering system, which is accessed via the Orderingtab in the MyNBME Services Portal.Once an examination has been constructed and finalizedby faculty, the faculty member who finalized the exam willreceive an e-mail notification that it can be requested for testadministration through the online ordering system.Additional FeaturesCombine Your Blueprints/Tests This feature includes the capability to combine blueprints or testsinto one exam form. For example, if you are building a test that covers content across differentcourses, separate exams can be generated from different blueprints and then merged into one tocreate a new test.Custom Score Categories Create custom score categories by individually tagging test itemsor merging default score categories to receive extra feedback on your score reports. Scorecategories must contain a minimum of 20 test items to generate a numeric subscore toappear on the Percent Correct and Scaled Score Reports, and a minimum of 10 test itemsto generate a profile band to appear on the Individual Examinee Performance Profiles.Item Usage Easily see when a test item is on other blueprints and tests at your school.Favorites Add items to your Favorites as you view them to create a personal repositoryof your Favorite Items. Use the tagging feature to Tag your Favorite items to a course,curriculum, or learning objective. Tags can also be used to blueprint and createcustom score categories.CAS Community The CAS Community allows you to share your blueprints andtests with all medical school faculty that use CAS. You can also view blueprintsand tests posted by faculty at other schools. Blueprints and Tests postedto the CAS Community can be downloaded to use within your school,and can be edited to better fit your course or curriculum.6

SCORE REPORTINGNBME customized assessments provide institutions with a tool for measuringexaminees’ understanding of a series of content areas as defined by the institution.Because course objectives vary, customized assessments allow faculty to buildassessments that target the specifics of a given curriculum.NBME neither sets nor recommends a “passing”score. We encourage customized assessments beused in conjunction with other indicators of studentperformance to determine grades.There are two stages of score reporting:1. Immediately After the Test Session Closes –Total Test Percent Correct ScoresAfter the test session is ended by the proctorthrough the proctoring system, a roster confirmationwill appear listing the examinees along with theirtotal test percent correct scores. The percent correctmean, standard deviation, and minimum/maximumscores are also shown. The proctor may excludeany records that are invalid (e.g., if an examinee leftdue to illness) and then confirm the group to besubmitted for final scoring.Percent correct scores are computed and providedfor the total test and the content areas definedduring the test construction process. Scaled scoresare also reported for the total test and content areaswhen a scaling group is defined*. Scaled scores arecomputed to have a mean of 70 and a standarddeviation of 8 for the your defined scaling group. Thescale provides a useful tool for comparing the scoresof students to one another as well as identifying therelative strengths and weaknesses within the contentareas defined for the exam.The individual total test percent correct scores andsummary statistics may be printed, downloaded toExcel, or saved as an Adobe Reader (PDF) file.Feedback is based only on the performance ofstudents from your school who took the customizedassessment. (Norm data based on a large,representative group of examinees is not availablefor customized assessments, since the exams aretargeted specifically to an individual curriculum.)Results should not be generalized to other cohorts,either within the institution or across otherinstitutions.*2. Within 2-3 Business Days of the TestAdministration Date – Full Score ReportsNBME will send an e-mail notification to the ECPwhen the score report is posted to the MyNBMEServices Portal.The various score reports included with CAS aredescribed on the following page.A scaling group is automatically created if there areat least 25 examinees. Scaling groups can also bedefined through the Scaling Group Application.A scaling group excludes examinees whodid not take the exam under standard pacingconditions, scored greater than 3 standarddeviations below the mean, or omittedmore than 10% of the items. Additionally,any examinee that is removed fromthe examinee group by the ChiefProctor will be removed fromthe scaling group.7