Transcription
2014 Continuing Compliance Master SeriesBest Practices in Alternative Assessment of PerformanceBrad S. Karon, MD, PhD, FCAPNovember 19, 2014www.cap.org
Today’sTodays PresenterBrad S. Karon,, MD,, PhD,, FCAPDr. Karon is currently chair of the College of AmericanPathologists (CAP) Continuous Compliance Committeeand a member of the Commission on LaboratoryAccreditation. He is an Associate Professor of LaboratoryMedicine and Pathology at Mayo ClinicCRochester. 2014 College of American Pathologists. All rights reserved.2
Disclosure No disclosures Special thanks to Dr. Christina Wojewoda for somecontent 2014 College of American Pathologists. All rights reserved.3
Objectives Define alternative performance assessment and the elements of analternative performance assessment system.system Identify tests or analytes performed in your laboratory that may beassessed using an alternative performance assessment system. Describe Laboratory Accreditation Program (LAP) requirements foralternative performance assessment systems. List best practices for alternative performance assessment that helplaboratories improve the quality of testing while optimizing laboratoryresources. 2014 College of American Pathologists. All rights reserved.4
What is Proficiency Testing (PT)? The CAP checklists define PT as “the determination oflaboratory testing performance by means ofinterlaboratory comparisons.”g of unknown samplesp CMS defines PT as the “testingsent to a laboratory by a CMS approved PT program.”The PT program grades the results using CLIAgrading criteria and sends the laboratory scoresreflectingfl ti howhaccuratelyt l it performedfd ttesting.ti 2014 College of American Pathologists. All rights reserved.5
What is Alternative Performance Assessment (APA)? Determination of laboratoryy testinggpperformance byymeans other than PT:o Split-sample testing with reference or other laboratoryor byb a diffdifferentt method.th do Split samples with established in-house method,assayed samples or materials, etc.o Clinical validation by chart review.o Participation in graded/ungraded/educational PTchallenges.o Also (previously) known as Alternative Assessment ofPerformance 2014 College of American Pathologists. All rights reserved.6
CMS PT Requirements Clinical Laboratory Improvement Amendments (CLIA)o http://www.cms.gov/CLIA/03 Interpretive Guidelineshtt ///CLIA/03 I ttiG id lifor Laboratories.asp#TopOfPageo Appendix C, Interpretive Guidelines Subpart H, PT requirements for non-waived testingo Regulated analytes defined by discipline (specialty)o Enrollment by analyte (chemistry, immunology,hematology, immunohematology)o Enrollment by subspecialty (microbiology) 2014 College of American Pathologists. All rights reserved.7
CMS PT Requirements - Regulated Analytes Eighty-threeg yanalytesylisted in the CLIA Regulationsg Criteria for evaluation described in CLIA regulations(% or SD from target or peer mean) Five Microbiologygy Subspecialtiespo Bacteriologyo Mycologyo MycobacteriologyMb t i lo Parasitologyo Virology 2014 College of American Pathologists. All rights reserved.8
CMS PT Requirements CLIA PT rules, continuedo Enrollment required for each “lab”lab (CLIA number)o Enrollment required for primary instrument/methodeach analytey Secondary instrument reporting eliminated 2014o Regulated analyte PT 5 samples X 3/yearo Must be CMS-approved PT providero Passing score 80% (100% immunohematology) 2014 College of American Pathologists. All rights reserved.9
CMS PT Requirements CLIA PT rules, continuedo OneO failedf il d PT ttestt (( 80%) unsatisfactoryti f tperformancef Must investigate cause CMS can direct cease testing if patient dangero 2 consecutive or 2/3 failed unsuccessful performance CMS mayyppermit technical assistance or retraining,g, orcease testingo 3 consecutive or 3/4 failed (repeat unsuccessful) cease testingt ti (regulated(l t d analytes)l t ) 2014 College of American Pathologists. All rights reserved.10
CMS PT RequirementsCLIA PT rules, continued What about all other non-waived testing? Non-regulated, non-waived testingo Twice annually verify accuracy of results What about waived testing?o NoN CLIA/CMS requirementsit forf PT or accuracyverification PT handlingo Handle as patient specimeno Incorporate PT into routine patient workload 2014 College of American Pathologists. All rights reserved.11
Accrediting Agency PT Requirements Accrediting agencies must at minimum meet CLIAo Frequency, scoring of regulated analyte PTo Each agency puts its own “spin” on PT BestB t practiceti Focus of accrediting agency Scientific/medical input on significance of testing 2014 College of American Pathologists. All rights reserved.12
Accrediting Agency PT Requirements Laboratory Accreditation Program (LAP)o College of American Pathologists (CAP) Key differences between LAP and CLIA PTo LAP does not distinguish waived vs. non-waived For enrollment, not necessarily scoringo PT required for predictive markers (ER/PR, HER2)o Emphasis on evaluating ungraded or no consensusPTo PT Programs must be CAP-accepted Currently 9 CAPCAP-acceptedaccepted PT programs 2014 College of American Pathologists. All rights reserved.13
Accrediting Agency PT Requirements LAP vs. CLIA PTo Analytes with required PT enrollment ( 300) Waived and non-waived Regulated and nonnon-regulatedregulatedo Similar to CLIA by analytevs. urine Some analytes defined by matrix (serum vshCG)o Frequency/number PT samples 5 X 3 for regulated analytes Varies for all other analytes (5X3, 3X2, 2X3) 2014 College of American Pathologists. All rights reserved.14
CAP (LAP) Oversight of PT Performance Continuous Compliance Committee (CCC)o Defines which nonregulated and waived analytesrequire PT Clinical relevance/significancegof analytesy Availability of graded PT Number labs performing testo AuditsA dit andd acceptst PT providersido Monitors enrollment, participation, and performance inrequired PT by LAP laboratorieso Sends out notification when a laboratory must CEASETESTING due to non-enrollment, non-participation, orcritical performance failure 2014 College of American Pathologists. All rights reserved.15
PT Terminology Regulated analytes Non-regulated analytes GradedG d d Non-graded Graded, Not-required PT,, Not available 2014 College of American Pathologists. All rights reserved.16
PT Terminology Non-gradedgPTo Intended as educational challengeo Lack of participant or referee consensuso Small number enrolled ( 10 quantitative or 5qualitative)o For both regulated and non-regulated analytes,laboratory must have policy/procedure for gradingitself 2014 College of American Pathologists. All rights reserved.17
Example of PT Evaluation-Ungraded PT Challenges 2014 College of American Pathologists. All rights reserved.18
PT Terminology Graded,, not requiredqo Laboratory enrolls in available PT product as analternative performance assessment 2014 College of American Pathologists. All rights reserved.19
How to Find Out What or if PT is Required? Master Activityy Menu with PT Optionsp((in e-Lab Solutions)) Laboratory’s Activity Menu (in e-Lab Solutions)[[COM.01200 The laboratory’sy current CAP activityy menushould accurately reflect the testing performed] Analyte/ProcedureyIndex of the Surveysy or EXCELCatalog (on www.cap.org) 2014 College of American Pathologists. All rights reserved.20
Master Activity Menu with PT Options 2014 College of American Pathologists. All rights reserved.21
Master Activity Menu with PT Options22 2014 College of American Pathologists. All rights reserved.
Laboratory’sLaboratorys Activity Menu 2014 College of American Pathologists. All rights reserved.23
Analyte/Procedure Index of Surveys and EXCEL Catalogs 2014 College of American Pathologists. All rights reserved.24
What is Alternative Performance Assessment (APA)? Determination of laboratory testing performance bymeans other than PT:– Split-sample testing with reference or other laboratoryor byy a different method.– Split samples with established in-house method,assayed materials, etc.– ClinicalCvalidation by chart review.– Participation in graded/ungraded/educational PTchallenges.challenges Alternative assessment must be performed semiannually/evaluation criteria established. 2014 College of American Pathologists. All rights reserved.25
When Would You Need To Do APA? For non-regulated, not required analyteso Porphobilinogen, antigliadin Ab (quant)o May use PT to satisfy APA requirement For analytes in which PT is not availableo By definition not required analyteso Stone analysis For in vivo testing:go Bleeding time 2014 College of American Pathologists. All rights reserved.26
LAP AAP Requirements Laboratory Accreditation ManualIf enrollment in a CAPCAP-acceptedaccepted PT program is not required for aparticular test, the laboratory must perform and document analternative assessment method semiannually to assess its analyticperformance for that test. Alternative assessment may include:o Participation in a PT program (graded or educational)o Split sample analysis with reference or other laboratorieso Split samples with an established in-house method, assayedmaterials or regional poolso Clinical validation by chart review, or other suitable anddocumented meansAlternative assessment that allows for comparison of results withexternal laboratories may provide more information than split sampleanalysis using internal methods. The laboratory must defineacceptable criteria for alternative assessment (eg(eg, results within10% of a reference method) 2014 College of American Pathologists. All rights reserved.27
LAP APA Requirements 2014 CAP Checklists COM.01500 Alternative Performance Assessmento For tests for which CAP does not require PT, the laboratory atleast semi-annually exercises an alternative performanceassessment system for determining the reliability of analytictesting.o NOTE 1: Appropriate alternative performance assessmentprocedures include participation in an external PT program notrequired by CAP; participation in an ungraded/educational PTprogram; split sample analysis with reference or otherlaboratories,, splitp samplespwith an established in-house method,,clinical validation by chart review, or other suitable anddocumented means. It is the responsibility of the laboratorydirector to define such alternative assessment procedures andthe criteria for successful performance in accordance with goodclinical and scientific laboratory practice. 2014 College of American Pathologists. All rights reserved.28
What You Need To Do Choose process for APA ProcedurePdtto documentdt planl andd acceptabilityt bilit lilimitsitbefore testing is performedo (( /- 2 or 3 SD or based on variation in the literature)) Document performance Evaluate trends Corrective action if needed Performance and evaluation automatic if using graded PT Evaluation more robust if using ungraded PT, still mustdefine performance required (SDI, % from target or mean) 2014 College of American Pathologists. All rights reserved.29
Example: PT Evaluation - Alternative AssessmentALTERNATIVE PERFORMANCE ASSESSMENTUrine MyoglobinSAMPLEXYZ LAB REF LAB12345POSAcceptableY/NPOSCRITERIA / or -/-Reviewed by: Date: 2014 College of American Pathologists. All rights reserved.30
FAQ sFAQ’s If a laboratory is enrolled in a PT Program but thel blaboratorytdoesdnott testt t theth specimensiforf ashipment (testing event), is the laboratory requiredto perform an alternative assessment? Yes. The laboratory should document why the PTspecimens were not analyzed (ie, instrument notfunctioning or reagents not available,available etcetc.).) Thelaboratory should contact the PT program to determine ifreplacement specimens are available. If not, thelaboratory should perform/document alternativeassessment at the same number of challenges for theperiod(s) that the PT specimens were not tested. 2014 College of American Pathologists. All rights reserved.31
Split-SampleSplitSample Testing External – reference laboratory Internal – with a different established method orpreviously tested samples 2014 College of American Pathologists. All rights reserved.32
Split-SampleSplitSample Testing: Quantitative Determine type of statistical analysis to use Determine criteria for acceptability Test samples representing AMRAMR, each side of cut-offcut off Test enough samples Use patient sampleso Avoids matrix effects, probes pre-analytic erroro StStorage, processingi may stilltill diffdiffer ffrom routinetipatient testing 2014 College of American Pathologists. All rights reserved.33
Split-SampleSplitSample testing: Quantitative If onlyy 2 of 3 match,, the samplep size must be increasedto 6 5 of the 6 must match to achieve the same level ofconfidence 95% confidence 2014 College of American Pathologists. All rights reserved.34
Audit-Sample Procedure (Previously Tested Samples) Can be used for stable analytesy Aliquots of patient sample stored AnalyzedA ld periodicallyi di ll over titime Assesses reproducibility and stability of calibration Does not assess trueness 2014 College of American Pathologists. All rights reserved.35
Direct Observation of Technique-DependentTechnique Dependent Tests Sweat test Bleeding time ObservedObdbby experiencedid seniori analystl t or supervisori Checklist with factors to be observed 2014 College of American Pathologists. All rights reserved.36
Clinical Correlation Studies Limited applicationppto routine test assessmento Imperfect correlation of clinical events to laboratoryresultso Test referral biaso Disease classification bias Can be used in certain circumstanceso If the presence of the disorder can be independentlyddeterminedi d at a reasonablebl pointi iin timeiafterf testingi 2014 College of American Pathologists. All rights reserved.37
Scenario 1 You are inspectingpg an anatomic ppathologygy laboratoryy andyou are told that they are performing HER2, brightfield insitu hybridization (CISH). You ask the staff what are theydoing for AAP and they show you a binder of results fromsamples that were split with a local laboratory. You noticesome discrepancies in the results between the laboratoryyou are in and the referenceflaboratory. When you ask howthese results were evaluated, you are told “We think wewere rightg and theyy were wrong”g althoughg no correctiveaction or evaluation was documented. 2014 College of American Pathologists. All rights reserved.38
Scenario 1This laboratoryy is: Meeting the requirement NotN t meetingti ththe requirementit Performing Best Practice AAP 2014 College of American Pathologists. All rights reserved.39
Scenario 1 2014 College of American Pathologists. All rights reserved.40
Scenario 1 2014 College of American Pathologists. All rights reserved.41
Scenario 2 You are inspectingpg a limited services laboratoryy and seeon their Activity Menu that they are performing nonwaived fecal occult blood and that it requires APA. Whenasked the technician shows you the past 2 years worthasked,of results (from 3 challenges twice/year) with signaturesof the laboratory director. APA was done by split sampleanalysis with a neighbor laboratory and accuracy criteria( / or -/-) are defined. Both positive and negativesamplespincluded each time. 2014 College of American Pathologists. All rights reserved.42
Scenario 2This laboratoryy is: Meeting the requirement NotN t meetingti ththe requirementit Performing Best Practice AAP 2014 College of American Pathologists. All rights reserved.43
Scenario 2 2014 College of American Pathologists. All rights reserved.44
CLSI GP29GP29-A2A2 CLSI GP29-A2 ((2008).) Assessment of Laboratoryy TestsWhen Proficiency Testing Is Not Availableo Situations in which PT may not be available: Unstable material/analyte (RBC osmotic frag,some micro)fluid, Analytes in unusual matrices (interstitial fluidstool) In vivo testing (bleeding time, Schilling test)o APA using patient samples avoids matrix effects andassesses pre-analytic factors, best practice 2014 College of American Pathologists. All rights reserved.45
Other Methods From CLSI Analysis of manufacturer’s product calibrator or controlmaterialo Use different lot calibrator or QC, limitations AnalysisA l i off iinterlaboratoryt l b tQC ddatat Analysis of patient averages/medianso Both forms of continuous QC, could be used for AAPif nothing else exists Re-evaluation of reference interval or interpreted resulto Periodic re-evaluation RI can detect changesoRRe-interpretationi tt ti relevantlt tto iinterpretivetti testingt ti(morphology) 2014 College of American Pathologists. All rights reserved.46
Sample Exchange Registry Internet-based service Connects laboratories performing testing where noformal pproficiencyy testingg ((PT)) is available Participate in the registry service at any time WhWhen 3 llaboratoriesb t i are ididentifiedtifi d as ttestingti ffor ththesame analyte, the CAP will facilitate the sampleexchange Primarily used for rare molecular tests 2014 College of American Pathologists. All rights reserved.47
Sample Exchange Registry CAP will distribute the samplespto pparticipatingpglaboratories Laboratories will test the samplespand send their resultsto CAP Data will be anonymizedyand each individual laboratoryywill receive its own results along with an anonymoussummary report of all the participants 2014 College of American Pathologists. All rights reserved.48
CAP Resources Contact the Compliance Groupo LAP PT Compliance Groupo Phone: 800-323-4040, ext. 6052o Fax: 847-832-8174o Email: ptcn@cap.org OOr you may use theth followingf ll i address:ddo LAP PT Compliance GroupCollege of American Pathologists325 Waukegan RoadNorthfield, IL 60093-2750 2014 College of American Pathologists. All rights reserved.49
Thank you!Questions? 2014 College of American Pathologists. All rights reserved.50
o One failed PT test ( 80%) unsatisfactory performance Must investigate cause CMS can direct cease testing if patient dangerCMS can direct cease testing if patient danger o 2 consecutive or 2/3 failed unsuccessful performance CMS mayyp g, permit technical as
