Transcription
UCLA CTSI-EvaluationPerformance Measurement: Accelerating Discovery,Improving Systems and OrganizationsPamela Davidson, PhDLeader, CTSI-Evaluation1
Evaluation FacultyDr. Pamela DavidsonDr. Mohsen BazarganDr. Lourdes GuerreroDr. Moira InkelasDr. Gerald KominskiDr. Deborah Koniak-GriffinDr. Loren MillerDr. Jack NeedlemanDr. Michael Weisman2
Program Area Presentation Outline forExternal Advisory Board MeetingI. Review of Program Area GoalsII. Response to EAB RecommendationsIII. ProgressIV. Future plansAreas most important to our renewalInformation not covered in presentation submitted assupplemental material3
I. Program Area Goals1. Longitudinally track and evaluate initiative andprogram outcomes2. Implement a performance measurement andimprovement initiative3. Create a transdisciplinary center for evaluationresearch4. Collaborate with local, regional and national CTSAinstitutions and support NCATS initiatives4
II. Summary of Major EAB Recommendations Digital Dashboarding Tool.– Plans to select software and co-design V2.0 dashboardwith Bioinformatics– Capture high priority/novel metrics in emerging strategicplan Ratio of Outputs to Outcomes.– Balance of dashboard metrics– Output 10/ Outcome 7 Cost-Outcome Evaluation.- 2014 Focus on Return on Investment (ROI)5
II. Summary of Major EAB Recommendations (continued) Tracking Career Outcomes.– Stay with RU Graduate Tracking System Survey(GTSS) in CTSAcycle– Migrate to ORCHID post-2015/16 KL2 Education and Training Dashboard.- At end of year 4, conduct additional analyses to design metricsbased strategic plan for renewal application- Measure career performance and pathway by type of trainingand track metrics associated with success IRB Process Mapping and Outputs.- Stratified IRB metrics by partner site- UC BRAID IRB workgroup- Interdependent IRB metrics and IRB Directors are critical6
II. Summary of Major EAB Recommendations (continued)EAB Strengths/Assets Executive Dashboard Initiative– to monitor and improve organizational outputsand outcomes– guide timely decision-making is another strength Cross-component performance metrics linked tostrategies in the strategic framework– including Return on Investment (ROI) is a great asset7
NCATS/CTSA Context for Evaluation IOM Report1. Formalize and standardize evaluation processes2. Lead in developing evaluation methodologies and metrics3. Critical for accountability, transparency, informeddecision-making, and communication regardingoutcomes and value of investment8
III. ProgressGoal 1Longitudinally track andevaluate initiative andprogram outcomes UCLA CTSI Executive Dashboard Continue to collect data and update 17metrics Metrics definition and interpretation9
CTSI Executive DashboardCTSI Performance Measurement StrategyGoals:Conduct performance measurement to Accelerate discovery Enhance performance Systems/Organizations/Programs Metrics based strategic plan Interfaces and interdependenciesAims:Facilitate a collaborative process to:1. Align metrics with strategic plan2. Set performance targets3. Plan high impact interventions to optimize performance10
UCLA CTSI Executive DashboardEAB Supplement I provides metric definitions, interpretations, and format for setting performance targets.11
ROI/Return Dashboard MetricsCTSI Goal 2: Team ScienceReturn on Investment/ReturnGoal 2MetricReturn on Investment (ROI):Vouchers, pilot grantsReturn /CTSI GrantFacilitation (includes RRT)*Data Source(s)Scientific Achievement surveyPeriodicityAnnualOffice of the Institute AdminrecordsAnnualROI metrics reflect cumulative yield from 2011 voucher and pilot grant awardees only12
III. Progress (continued)Goal 2Implement aPerformanceMeasurement andImprovement Initiative Template for setting metric targets Continue collaborations, e.g.:- CTSI-Regulatory Program to track IRB- CTRC needs assessment to identifyinvestigator needs/interests- UC BRAID to measure and improvecontracting- UC BRAID needs assessment on virtualbiobank- UC CAI performance metrics andinnovator knowledge and skillsdevelopment13
III. Progress (continued)Goal 3Create atransdisciplinary centerfor evaluation research U54 Pending (Norris and Davidson):NIH BUILD/NRMN Coordination andEvaluation Center- BUILD: Building InfrastructureLeading to Diversity- NRMN: National Research MentoringNetwork F5LA Dental Home Project/ Evaluateaccess to care in 22 safety net clinics J. Macy: Innovative tools for EvaluatingInterprofessional Competencies14
III. Progress (continued)Goal 4Collaborate with local,regional and nationalCTSA institutions andsupport NCATSinitiativesUC BRAID NHLBI Center for AcceleratingInnovation (CAI) Evaluation Network Contracting Workgroup Biobanking Workgroup IRB Workgroup Standardize Research Education Metrics Bibliometrics15
IV. Future PlansI.Continue developing performance measurementsystem in collaboration with CTSI leaders and BIP1.Continue reporting and refining existing metrics2.Set performance targets3.Develop high impact interventions to acceleratediscovery and enhance org performance4.Expand dashboard to reflect metrics-based strategic plan16
IV. Future Plans (continued)II. Collaboratively design metrics based strategic planIII. Develop capabilities and competencies inperformance enhancement in the researchenterprise.IV. Continue expansion in UC BRAID, UC CAI, andNCATS/CTSA evaluation activities.V. Support and sustain evaluation program throughextramural funding.UC BRAID: Biomedical Research Acceleration, Integration, and DevelopmentUC CAI: Center for Accelerating Innovation funded by National Heart Lung Blood Institute17
Supplementary MaterialsSupplement I. UCLA CTSI Executive Dashboard MetricsDefinitions, Interpretation, and Format for SettingPerformance TargetsSupplement II. Detailed Responses to 2013 EABRecommendations18
based strategic plan for renewal application - Measure career performance and pathway by type of training and track metrics associated with success IRB Process Mapping and Outputs. - Stratified IRB metrics by partner site - UC BRAID IRB workgroup - Interd
