Transcription
NBMESELF-ASSESSMENTSERVICES VOUCHER PROGRAM GUIDEHigh quality self-assessment toolsfor medical schools, residency programs,and medical education organizations
CONTENTS03Overview03Background03Self-Assessment Services03Ownership and Copyright04Terms, Conditions, and Disclaimers04Performance Feedback Reports06Voucher Ordering06NBME Store Account Setup08Payment Methods09Order Cancellations09NBME Store Account Credit10Voucher Information Program (VIP)11Form Assignment11Download and Distribution of Voucher IDs12Voucher Redemption and Administration12Voucher Redemption13Administration and Support13Standard Setting Administration14Examinee Administration14Contact UsCopyright 2006, 2009-2019. 2020-2022 NBME. All rights reserved.2
OVERVIEWBackgroundThe NBME Self-Assessment Services Voucher Program is offered to medical schools,residency programs, and medical education organizations interested in purchasingself-assessments for their students.Institutions are able to: Self-assessments for the International Foundationsof Medicine (IFOM ) Clinical Science Examinationare for examinees preparing to take that exam orfor anyone wishing to evaluate medical knowledgeand understanding of clinical science consideredessential internationally for the provision of safeand effective patient care. Order vouchers for self-assessments throughan online NBME Store Assign a specific form number to voucher IDs Download prepaid voucher IDs and distribute themto students so they can request a self-assessment Track the redemption of vouchers (redeemedversus unredeemed)Ownership and Copyright Track voucher expirationThe self-assessments are owned and copyrighted byNBME. Participants may not transfer or reproduceself-assessment materials in any way withoutpermission from NBME. Any unauthorized transferor reproduction of these materials, by any means,including but not limited to, storage in a retrievalsystem, transmission, printing, memorization, ordistribution is strictly prohibited. If NBME believesthat any participant has engaged in or is engaging inthe foregoing behavior, NBME reserves the right totake any and all further action necessary to protectthe integrity of the examination, including but notlimited to notifying the participant’s medical schoolor residency program of the suspected unauthorizeddisclosure of copyrighted materials and taking legalaction. Any actions taken by a medical school orresidency program on the basis of such informationare completely the responsibility of the medicalschool or residency program. See full disclaimer. Monitor the completion of self-assessments(launched or completed) View performance feedback for completedself-assessmentsSelf-Assessment ServicesSelf-Assessment Services are available to bothdomestic and international medical students andgraduates.Web-based self-assessments include the following: Comprehensive Basic Science, Clinical Science,and Clinical Medicine Self-Assessments allowexaminees to evaluate their readiness to take theUnited States Medical Licensing Examination (USMLE ) Step 1, Step 2 Clinical Knowledge (CK),and Step 3 examinations. The Clinical Science Mastery Series helpsexaminees to gauge their comprehension ofthe clinical sciences covered during a clerkshipor medical education course.3
Terms, Conditions and DisclaimersPerformance feedback for the ComprehensiveClinical Science Self-Assessments includes:The material presented in the self-assessmentsis provided by NBME for educational purposesonly. The self-assessments are not intended topredict performance on the USMLE or the NBME Subject Examinations. Rather, the self-assessmentsare intended to be used as a tool to determine aparticipant’s strengths and weaknesses in generaltopic areas. An assessment score on the same score scaleas Step 2 CK Performance explanations and illustrations thatalign with the USMLE Step 2 CK Score Report Diagnostic feedback that highlights areas ofstrengths and weaknesses relative to overallperformance Diagnostic feedback that highlights areas ofstrengths and weaknesses relative to a USMLEcomparison groupPerformance Feedback ReportsStudents Accessing Performance Feedback Reports The ability for you to review answer explanationsfor all itemsPerformance feedback for the ComprehensiveClinical Medicine Self-Assessments includes:Participants receive immediate feedback in theform of performance profiles that indicate relativestrengths and weaknesses in general topic areas.These reports can be accessed and downloaded bythe participant on their MyNBME account. The ability for you to review answer explanationsfor all items Diagnostic feedback that highlights areas ofstrengths and weaknesses relative to borderlineperformancePerformance feedback for the Comprehensive BasicScience Self-Assessments includes: A total equated percent correct score (EPC) andcontent area EPCs that represent the percentageof content mastered on the overall exam and onthe content areasPerformance feedback for the Clinical ScienceMastery Series includes: A score interpretation guide that will help youapproximate a Subject Examination score The estimated probability of passing Step 1 ifyou test within a week (i.e., with a similar levelof knowledge) The ability to review answer explanations forall items Longitudinal feedback is provided when multipleassessments are completed Information about where your CBSSA score islocated relative to the low-pass range on Step 1Performance feedback for the IFOM Clinical ScienceSelf-Assessment includes: Information about your likely score range if youtest again without learning or forgetting anything A performance summary containing a total scoreand a list of all questions on the self-assessmentwith the correct answer displayed, by contentcategory The ability for you to review answer explanationsfor all items Diagnostic feedback that highlights areas ofstrengths and weaknesses relative to a USMLEcomparison group A score interpretation guide that includesa conversion table that a participant can useto “translate” the self-assessment score to anapproximate score on the InternationalFoundations of Medicine Clinical ScienceExamination score scale Longitudinal feedback provided when multipleassessments are completed** This will be available sometime in April 2022.4
Institutions Accessing Performance FeedbackReportsMyNBME Services Portal users will be able to viewthe examinee name, voucher ID used, form selected,timing mode, and scaled score. When users clickon the View link, they can view and downloadperformance feedback reports in a PDF format.Users can also download the information they see onscreen (excluding the PDF report) by clicking on theExcel icon.Institutions will be able to view performancefeedback for completed self-assessments byaccessing the Redeemed link in the VIP system.However, it will take up to 24 hours for the feedbackresults to be posted to the system.NOTE: An aggregate reportis not provided for the NBMEself-assessments since thisservice is based on individualperformance.5
VOUCHER ORDERINGThe MyNBME Services Portal is a secure website and a primary source for authorizedusers to obtain information about their students for a variety of activities related tothe USMLE, Subject Examinations, IFOM, Self-Assessment Voucher Program, andother services.NBME Store Account SetupEach medical school has at least one User AccountAdministrator who is responsible for assigning usersthe privileges they need to access areas of theportal appropriate for their responsibilities. The UserAccount Administrator must assign users the SelfAssessment Voucher Service permission to accessthe NBME Self-Assessment Services portlet, whichhouses the NBME Store and Voucher InformationProgram system.Self-assessment vouchers are ordered through anonline NBME Store. The store will allow you to shopfor a variety of NBME services. Users will not beable to place an order if they do not have access toa store billing account. User Account Administratorscan create institution store billing accounts andassign local users access to them. NBME AccountAdministrators can do the same for residencyprograms and medical organizations.NBME requires that an institution store billingaccount be set up for the purposes of placing anorder and applying credit.NOTE:NBME Account Administratorsare responsible for creatingaccounts for residencyprograms and medicalorganizations as well asassigning user privilegesto the Self-AssessmentVoucher Service.6
The NBME Store can be accessed on the Purchase Exams page of the portal, click on the Purchase buttonwithin the Self-Assessment Vouchers section.After entering the online store, users will be able to browse by category and view a description of eachproduct, add a product to their shopping cart, select a method of payment, track the status of orders, as wellas view and download invoices. Click here for self-assessment voucher fees.7
Payment Methods15 minutes of order placement. When using accountcredit, the credit must be equal to or less than thebalance due. Payments cannot be split into twodifferent payment methods. Payment methodscannot be changed once the order is submitted.If payment is remitted with a check or wire, thevouchers will not be made available until the order ispaid in full.Payment for services may be made by: Credit card (Visa, Mastercard and AmericanExpress) Check Wire transferA confirmation e-mail with an invoice is providedimmediately after order submission. Users can viewall current and past order history from the orderhistory tab. Available credit (Institution account with a creditbalance)Self-assessment vouchers ordered with a creditcard or account credit are usually available withinNOTE: Since self-assessment vouchers are ordered through a separate platform than theweb-based NBME Subject Examinations, funds cannot be transferred from one platformto another.User Account Administrators and staff with permissions to order vouchers can reviewaccount credit by accessing the Purchase Exams page and clicking on View Accountsfrom the NBME Store Accounts tile.8
Order CancellationsNBME Store Account CreditOrder cancellations may be made through the NBMEStore. After users access the Order History page, theywill be able to click on the order number and selectthe Cancel button. However, if the order has beenfulfilled and the vouchers have been generated, userswill not be able to select this option. They will need tosubmit a cancellation request to support@nbme.org.Vouchers are good for one year from the NBMEdate of issue. Institutions will receive credit forunused vouchers associated with orders submittedthrough the NBME store. This credit is applied tothe institution store account once the vouchers haveexpired. The credit may then be used toward futureorders.9
VOUCHER INFORMATION PROGRAM (VIP)The VIP system allows institutions to assign a form to vouchers, download anddistribute voucher IDs to students, track voucher activity, monitor completion ofassessments, and view performance reports.When users access VIP, they will see a Voucher Order History page and by default be able to access voucherIDs for orders they have placed. Users have the option to select All Vouchers from the Select View drop-downmenu located in the left-hand corner above the voucher history table to view the entire voucher order historyof their institution.10
Download and Distributionof Voucher IDsThe Not Redeemed link associated with an orderallows a user to view unused voucher IDs and theirend or expiration date. The Redeemed link will allowa user to track voucher activity – examinee name,voucher ID used, self-assessment status (launchedversus completed), form selected, timing mode,scaled score, and a performance feedback report.After users click on the Not Redeemed link to viewtheir voucher IDs, they can select the PDF iconin the right-hand corner. This feature will allowusers to download and merge voucher IDs withredemption instructions into a PDF. Institutions maythen distribute a voucher ID along with instructionsto each examinee. Users also have the optionto download voucher IDs into a spreadsheet byselecting the Excel icon.Form AssignmentAll vouchers are processed without a formdesignation. However, the institution has thecapability to assign a specific form to voucher IDsthrough the VIP system. A drop-down menu willprovide the number of the forms available; the higherthe form numbers, the more current. Located withinVIP, under the General Instructions drop-down menu,are instructions for form assignment.Vouchers are good for one year from the NBME dateof issue. Institutions will receive credit for unused,expired vouchers associated with orders submittedthrough the NBME store. This credit is applied tothe institution store account once the vouchers haveexpired and may then be used toward future orders.NOTE: If a form is not assigned, then the voucher IDs may be redeemed for “any form” ofthe self-assessment requested.11
VOUCHER REDEMPTIONAND ADMINISTRATIONVoucher RedemptionNBME self-assessments are purchased and launched through the MyNBME website.Resources and materials available on the website include a sample assessment,system requirements, and sample performance reports. Since the self-assessmentsare a non-secure service, a proctoring system is not required for administration.To start their self-assessment, examinees will be required to: Create and activate their account; Select the type of assessment and form foradministration; Choose the timing mode (standard-pacedversus self-paced); and Select Voucher as the methodof payment and enter thevoucher ID.12
Administrationand Support1.Examinees should beprovided a voucher IDand voucher redemptioninstructions in advance.There are two differentapproaches toadministration.Standard Setting AdministrationIf you are going to have examinees take the self-assessment in a classroomsetting, you will need to allow enough time for account setup (examineeswill need to activate their account via e-mail), purchasing, the tutorial, andadministration of the assessment. Below is a table that displays the timingmodes (excluding the 15-minute tutorial).Standard-paced is selected as the default as the timing mode when aself-assessment is chosen. The timing mode cannot be changed after anassessment is started.Examinees will have to review an online tutorial before they begin theirself-assessment. The tutorial will describe how to answer items, skip items,review responses, and change answers.Timing infoSession TimeStandard-PacedTotal Session TimeSelf-PacedAssessment(does not includeoptional, tutorial)(does not includeoptional, tutorial)Tutorial (optional)15 minutes1 hourComprehensive BasicScience Self-Assessment5 hours20 hours200Comprehensive ClinicalScience Self-Assessment5 hours20 hours200Comprehensive ClinicalMedicine Self-Assessment4 hours 26 minutes17 hours 44 minutes176Clinical Science Mastery Series1 hour 15 minutes5 hours50IFOM Clinical Science Self-Assessment2 hours8 hours80Number of Items--Here are a few helpful tips for institutions:During AdministrationAfter students calibrate their monitor, they will then be presented withthe tutorial. The “SkipTutorial” option is located at the bottom of thescreen, on the left-hand side. Students should only skip the tutorial ifthey are familiar with taking a self-assessment.If examinees “Pause” a section and resume it later, they will return tothe same section and item in which they were working, provided thetotal administration time allotted has not run out. If participants select“Pause” while working in a section, the time clock stops for that sectiononly and the browser will close.Before examinees complete a section, they must select “End” in order toproceed to the next section. If they end a section, or if time expires for asection, they will not be able to return to that section to review items orchange answers.13
Administrationand SupportImmediately Following Administration(continued)After examinees complete their assessment, they will be able to viewtheir performance immediately in the online application.Once the students advance through the online application to reviewtheir feedback, they will have the option to complete a survey or select“End.” If they select “End,” this will prompt the system to end the sessionand close the window. Examinees will have 2 years to access and viewtheir performance report from their MyNBME account.2.Examinee AdministrationThe same rules apply to both standard setting and examinee administration.Examinees will have to set up an account as well as select an assessment,form, and timing mode. Standard-paced is selected as the default as thetiming mode when a self-assessment is chosen. However, examinees maychoose self-paced.Once the examinees redeem the voucher, they will have 90 days to launchand complete the self-assessment (an expiration date is provided within theapplication). Examinees will have to review an online tutorial before theybegin their self-assessment.CONTACT USPlease feel free to contact us if you have any questions. Various NBME Staff canprovide the information you need or help solve a problem.NBME Self-Assessment Voucher Program Questions OnlyAvailable Monday-Friday between 9:00am-5:00pm,except for holidays.Support@nbme.orgOperational Support and All Other InquiriesAvailable 24 hours a day, 7 days a week, except for , option 1
The NBME Self-Assessment Services Voucher Program is offered to medical schools, residency programs, and medical education organizations interested in purchasing self-assessments for their students. OVERVIEW Institutions are able to: Order vouchers for self-assessments through
