WIXLIB

Webinars ExpandAudience ReachIn 2019, NBME hosted several webinars to foster deeperinteractions with medical school faculty and students.These webinars provided opportunities for faculty andstudents to learn about NBME products and services.They also provided a forum for our staff to engage inmeaningful dialogue about our assessment servicessuch as Customized Assessments, the AmbulatoryCare Subject Examination, and comprehensive andsubject examination score reports. NBME also hosted alive webinar on USMLE score reporting to foster a richconversation about this important topic.New Health SystemsScience Exam Bridgesthe GapIn collaboration with the American Medical Association,the Health Systems Science Examination was madeavailable to all medical schools. This high-quality,standardized assessment supports the implementationof health system science curricula and educationalinitiatives. Together with clinical and basic scienceinstruction, this new assessment prepares medicalstudents for effective clinical practice during residencyand throughout their careers.82019 NBME Annual ReportHealth and WellnessCoaching Field SeesGrowthNBME has been collaborating with the National Boardfor Health & Wellness Coaching (NBHWC) since 2016to create and administer a national board certificationassessment for health and wellness coaching. In 2019: NBHWCreceived a grant from the Centers for DiseaseControl and Prevention for a coaching programdesigned to help prevent or delay type 2 diabetesmellitus. The organization will partner with theAmerican College of Preventative Medicine to developgroup coaching competencies, a training program, anda certificate examination to advance this work.The American Medical Association approved the use ofnew Category III Current Procedural Terminology (CPT )Codes for health and wellness coaching beginningin January 2020. NBHWC and the US Department ofVeterans Affairs (VA) applied for these new trackingcodes to be based off NBHWC standards, and the VAplans to utilize them to evaluate the effectiveness ofcoaching within its health care system. As the VA usesthe new codes to track services provided by coacheswho are trained by the VA and certified by the NBHWC,both organizations hope to increase recognition of thevalue of health and wellness coaching in improvingpublic health.

Invitational Conference on USMLE Scoring (InCUS)In early 2019, NBME co-sponsored the InvitationalConference on USMLE Scoring (InCUS) to collaborativelyreview the USMLE program’s practice of numeric scorereporting within the context of its primary use (initialmedical licensure), and to discuss secondary uses ofscores, such as residency selection.The meeting was also co-sponsored by the AmericanMedical Association, Association of American MedicalColleges, Educational Commission for Foreign MedicalGraduates, and Federation of State Medical Boards.In all, 65 attendees participated in a professionallyfacilitated two-day meeting in order to connect, advise,and inform each other. Prior to the meeting, additionalinput was gathered from more than 200 organizationsto ensure that a complete range of perspectives wasconsidered. Areas of consensus include: Thecurrent medical school-to-residency transition is notmeeting stakeholders’ needs because each stakeholdergroup has optimized its part of the system over time. Unilateralchanges to USMLE will not “fix” the entiresystem, absent changes in other parts of the system. Changes—bothsystemic and specific to USMLE—must be explored, identified, and implemented on areasonable timeline.Four preliminary recommendations emerged from InCUSthat begin to address the complex challenges of a flawedsystem of residency selection.Stakeholders were invited to submit feedback aboutthe preliminary recommendations through an opencomment period. More than 20,000 responded to asurvey that accompanied the four recommendations.A summary of themes that emerged from InCUS andpublic commentary was published on USMLE.org in fall2019. These comments provided valuable input thathas helped to inform our efforts as we continue thisimportant conversation.In 2020, the USMLE program announced three futurepolicy changes: ChangingStep 1 score reporting from a three-digitnumeric score to reporting only pass/fail; Reducingthe allowable number of exam attempts oneach Step or Step Component from six to four; and Requiringall examinees to successfully pass Step 1 as aprerequisite for taking Step 2 Clinical Skills.Learn more about the policy changes on USMLE.org.2019 NBME Annual Report9

Reimagining Assessment PracticesWhen Amy Morales, Director of Test MaterialsDevelopment at NBME, attended a meetingof the Committee on Clerkship Directors ofthe Association for Surgical Education (ASE)in April 2017, she learned a hard truth: thecommittee was unhappy with the Surgerysubject examination. Its content did not alignwith the core curriculum for surgery.This could have hurt the relationship betweenNBME and the clerkship directors, but instead,it strengthened it. Listening to the clerkshipdirectors’ concerns sparked the reimagining ofhow to better align the Surgery subject examwith the curriculum. NBME realized that theexam questions needed to be curated moreclosely by the surgeons who best understoodthe subject matter, and who could ensurethat the exam content aligned with the corecurriculum.From that original conference, Marc de Moya,MD, Chief of Trauma/Acute Care Surgery atthe Medical College of Wisconsin/FroedtertHospital, invited Amy to attend the American102019 NBME Annual ReportCollege of Surgeons (ACS)/ASE MedicalStudent Core Curriculum Steering Committee.In that meeting, Amy was introduced to theindividuals responsible for developing thenational surgery curriculum and learned moreabout the core topics.The next improvement in the collaborationwas for NBME’s Test Development staff tohost an item-writing workshop and itemreview session in Philadelphia. There, CarlyDaniels and Miranda Gipe sat down with 13surgery clerkship directors to teach thembest practices for writing multiple-choicequestions. Afterward, the surgeons wrotenew Surgery subject exam content and vettedexisting content. Typically, new subject examcontent is reviewed remotely by clerkshipdirectors, so this was a different approach.Many of the writers noted that it was helpfulto be in the room with one another becausethe conversations that took place providednew perspectives, and at times, resulted inmaking decisions they would not have madehad they reviewed the content alone.

“It has been a fantastic collaboration. NBME’s opennessto improving the Surgery subject exam has beenimpressive. Working with NBME has been a personalpleasure. I’ve learned a great deal.Jesse Moore, MD“When you have a group of people in a room,that collective voice is helpful,” Amy said.“Additionally, it creates an environment thatenables editorial staff to interact with subjectmatter experts and ask questions. There islearning on both sides.”This collaboration has now evolved into an NBMEtask force that writes and reviews new examcontent. Jesse Moore, MD, an Associate Professorof Surgery at the Larner College of Medicine atthe University of Vermont, chairs the group. Heunderscored the richness of this collaboration.“It has been a fantastic collaboration. NBME’sopenness to improving the Surgery subjectexam has been impressive,” he said. “Workingwith NBME has been a personal pleasure. I’velearned a great deal.”And for NBME, the feeling is mutual.“There’s no way that we could create a qualityproduct if we didn’t have the subject matterexperts work together with us,” Amy said.“Collaboration is not just an enhancement ofwhat we do, it’s essential to what we do.”We take pride in collaborating with medical education professionals whosevalues echo our own. Their experience and expert viewpoints respectfullychallenge, and ultimately deepen, our commitment to innovating andimproving our offerings.2019 NBME Annual Report11

improvingassessmentIMPROVING ASSESSMENT THROUGHRESEARCH AND DEVELOPMENTTo keep pace with rapid changes both in medical educationand in the delivery of patient care, NBME innovates tocreate new products and enhance existing ones. In doingso, we better meet the needs of our customers.In 2019, collaborations with University of Wolverhamptonand the University of Pennsylvania enabled NBME’sCenter for Advanced Assessment to develop capabilitiesbased on Natural Language Processing (NLP). Thesecapabilities have led to improved assessment practices:scoring of the patient note forUSMLE Step 2 Clinical Skills Examination combinesjudgments from licensed, board-certified physicianraters with NLP-based computer-generated scores.Use of NLP enables NBME to apply the best available Computer-assisted122019 NBME Annual Reporttechnology to ensure accuracy and validity for all ourexamination programs. Automaticgeneration of multiple-choice questiondistractors—incorrect yet plausible alternatives to thecorrect answer—can facilitate the test-item writingprocess. NLP enables review of existing test contentand the generating of a list of distractors that can beused by item writers. This technology will increase theefficiency and quality of the item writing process. Identifyingitems that should not be placed on thesame test form because of an overlap in content can beachieved using NLP-based procedures. This will allowfor savings in test development time and will result inimproving the quality and fairness of our assessments.

In 2019, Psychometrics and Data Analysis (PADA) workedto enhance assessment-related products and services,inform best practices, and promote evidence-baseddecisions about students and health care professionals: Anew method of computing and reporting subscoresresulted in the design of new score reports forUSMLE, subject examinations, and self-assessments.The new reports were extensively evaluated priorto their adoption; evaluation included input frommultiple stakeholders across medical education. Thesereports enable examinees to appropriately interprettheir strengths, address more challenging areas, andfunction as overall guides for remedial study. NBMEresearchers have led, independently or togetherwith collaborating organizations, a number of studiesaround fairness and equity in assessment: Use of expert review to identify potential bias in testquestions; A study in collaboration with a number of medicalschools to understand the potential effects ofdelaying the taking of Step 1 to after clinical clerkship. NBMEhas continued to support medical specialtyboards by assessing the degree that performance onUSMLE and in-training examinations predicts successon respective board certification examinations. In collaboration with the American Board of InternalMedicine, NBME has evaluated these relationships.Findings have overwhelmingly shown that in-trainingexamination scores serve as a useful indicator offuture board certification success and can helpprogram directors identify at-risk trainees.2019 NBME Annual Report13

SUPPORTINGTHE COMMUNITYHealth care educators receive support from NBME through several avenues. First, theStrategic Educator Enhancement Fund serves the next generation of medical educatorsthrough professional development and a newly introduced fellowship program. Grantsbestowed through the Edward J. Stemmler Medical Education Research Fund and theLatin America Grants Program contribute to excellence in medical education assessmentresearch and faculty development. Item writing workshops facilitate knowledge sharing tohelp educators enhance the quality of their own assessments.Strategic Educator Enhancement Fund (SEEF)Two opportunities for medical educators availablethrough this fund are the NBME Invitational Conferencefor Educators (NICE) and the SEEF Medical EducationResearch Fellowship. TheNBME Invitational Conference for Educators fostersskill development in assessment and provides a venuefor networking for medical school faculty. The secondof these conferences was held in Indianapolis, Indiana,on May 15-16, 2019, with 240 faculty participating from135 US allopathic and osteopathic medical schools.With help from current and past NBME test committeemembers, NBME staff presented interactive workshopson written and performance assessments.142019 NBME Annual Report TheMedical Education Research Fellowshipwas introduced for the first time in 2019 and is aproject-based faculty development program. It wasestablished to support a cohort of individuals who willwork with a senior mentor and NBME staff to identifyand develop questions, and a research project. Thefellowship provides an opportunity for medical schoolfaculty interested in developing skills in medicaleducation and assessment research and who havea commitment to working with a team of interestedcolleagues. Eight individuals have been selected toform the inaugural cohort.

Edward J. Stemmler Medical EducationResearch Fund2020 marks the 25th year of the Stemmler Fund, andplans are in place to acknowledge and celebrate thecontributions by grant recipients since its inception in1995. The fund promotes advancements in the theory,knowledge, or practice of assessment at any point alongthe continuum of medical education.In 2019, NBME awarded a total of 449,000 to theseresearch projects: AssessingResidents’ Clinical Performance UsingResident-Sensitive Quality Measures – Daniel Schumacher,MD; University of Cincinnati College of Medicine UsingNLP to Unleash the Potential of Narrative Datato Assess a Student’s Readiness to Progress in aCompetency-Based Curriculum – Celia Laird O’Brien,PhD; Northwestern University Feinberg School ofMedicine StandardizedPatient Assessment of CommunicationSkills: A Standardized Examinee Audit Study ofGendered and Racial Differences – Laura Hirshfield,PhD; University of Illinois at Chicago College ofMedicineLatin AmericaGrantsItem WritingWorkshopsThe Latin America Grants program continues to supportthree consortia that were selected as part of the 2018-2020grant cycle and are making great progress in these projects:To enhance the quality of assessments, NBME facilitatedapproximately 30 in-person and virtual workshops in2019 for medical school faculty and others, includingMeharry Medical College, Carle Illinois College ofMedicine, Marshall University, and University of Texasat Tyler. The workshops are designed for those whowould like to increase their knowledge and skills, and tolearn to utilize tools to improve their own assessments.Professional development benefits include: TeacherDevelopment for Programmatic Evaluation ofStudents in Health Degree Courses in Brazil submittedby a consortium of six Brazilian medical schools MOOCDesign for Training Teachers in EducationalAssessment in Health Sciences submitted by aconsortium of three medical schools in Mexico Assessmentof Abilities and Skills by DirectObservation in Medical Internship submitted by twomedical schools in BrazilThe recipients of the 2020-2022 grant cycle will beannounced on April 1, 2020. All grants serve to improveassessment tools through funding and counsel from NBME. Techniquesto improve the quality of test items inassessments across health professions education Checklists,templates, and a copy of NBME’sConstructing Written Questions for the Basic and ClinicalSciences, 4th edition, to support writing test items. Networkingwith colleagues who are also learningabout and developing assessments2019 NBME Annual Report15

TALENT ANDRECOGNITIONNBME WelcomesKen Ridgley as VicePresident of TestDevelopment servicesIn February 2019, NBMEwelcomed Ken Ridgleyas its new Vice Presidentof Test Development. Achampion of innovativeassessment practices andorganizational development,Ken cultivated his expertisein assessment at Prometricwithin a decade-long tenureas the Vice President of TestDevelopment, and as the leader of its Global Test CenterSupport organization. Ken’s commitment to NBME’smission and his knowledge of progressive contentmanagement technologies enrich NBME’s values andvision for excellence in assessment.Ken earned a BA in psychology from the University ofMaryland, Baltimore County, and an MS in industrial/organizational psychology from the University of Baltimore.162019 NBME Annual ReportMedical Advisorsin the CommunityNBME medical advisors are immersed in thesame settings as our customers and students—they practice and teach clinical medicine andcontribute scholarship to respected, peerreviewed journals and educational venues.Through their multifaceted relationships,our medical advisors connect NBME withthe community. They remain engaged inmedical education’s ever-changing, complexconversations. In doing so, NBME cancontinuously align assessments with therealities that customers encounter.In 2019, Miguel Paniagua, MD, and ThomasRebbecchi, MD, presented at several venues toconnect with the communities we educate andlearn from: “Evidence-BasedAssessment: Envisioningthe Future” - University of South CarolinaSchool of Medicine, Greenville Departmentof Emergency Medicine Grand Rounds

“Re-envisioningExams: NBME’sEffort on Wellness (RENEW)Project,” Short Communications- Student Wellbeing, Associationfor Medical Education in Europe(AMEE) Conference, Vienna, Austria “Analytical Approaches for ImprovingMiguel Paniagua, MDthe Quality of ObservationalWorkplace-based Assessments,”International Association ofMedical Regulatory Authorities(IAMRA) Symposium: ContinuedCompetency: Balancing Assuranceand Improvement, Chicago, IL “HowDo they Know What IKnow? ‘Steps’ to becoming aCompetent Physician” - Universityof South Carolina USMLE/NBMEUpdate - TempleUniversity School of MedicineFaculty Retreat, Philadelphia, PA OverviewThomas Rebbecchi, MDof NBME Offerings atthe Council on Medical StudentEducation in Pediatrics (COMSEP)Annual Meeting, St. Petersburg, FL USMLE/NBMEUpdate at theAssociation of Program Directorsin Internal Medicine (APDIM) fallmeeting, Denver, CO Historyof Assessment at theNBME - American Board ofMedical Specialties (ABMS)Visiting Scholars Program,Webinar, November 20192019 JohnP. HubbardAwardThe 2019 John P.Hubbard Award forexcellence in thefield of evaluationin medicine wasawarded to WilliamC. McGaghie, PhD,Professor of MedicalEducation and Preventative Medicineat Northwestern University FeinbergSchool of Medicine in Chicago, Illinois.“Dr. McGaghie has made significantand sustained contributions tothe assessment of professionalcompetence in medicine and thedevelopment of medical educationprograms for more

They also provided a forum for our staff to engage in meaningful dialogue about our assessment services such as Customized Assessments, the Ambulatory Care Subject Examination, and comprehensive and subject examination score reports. NBME also hosted a live webinar on USMLE score reporting to foster a rich conversation about this important topic.