Transcription
PBC Contract GroupAnnual UpdateJuvenile CourtsProvider Services Quality & Availability Accountability Group & PBC Support TeamApril 20, 2022www.dcyf.wa.gov1
Juvenile CourtsBackground InformationContract GroupMission: Juvenile Courts serves the highest-risk youth with theintent to prevent further offending; by connecting youth to servicesusing research-based treatment approaches that span a continuumof careLead: Cory Redman, Juvenile Courts Program AdministratorClients Served (FY21): 1,021Members: Bob Leonard, Program Administrator; ChadConnors, Juvenile Court Administrator; David Sattler,CMAP CoordinatorAnnual Contract Spending (FY21): 3,745,632PBC Staff: Stacey Gillette, PI AdministratorCurrent Contract Cycle (FY22): July 2021 – June 2023Data Partner(s): Arina Gertseva, Senior Research Associate,AOC – Data Consultation ONLY# of Contracts (client services) (FY22): 56Implementation Status: Phase 2 PBC Service Standard: Currently tracking services delivered in FY22-23 contracts PBC Quality Standard: Will incorporate new quality standards (2) with PMT in FY24-25 contracts PBC Outcome Standard: Will incorporate at least one (1) outcome metric, target, with PMT in FY24-25 contracts2
Juvenile Courts: FY21-22 PBC Metrics, Targets & PMTsQUALITY MEASURESProgram StartMetricNumber of youth who startan EBPTarget100% of youth identified incontractor’s application tostart an EBPPerformanceManagement Tool (PMT)N/A; DCYF will collect baseline dataduring the FY22-23 contract period tovalidate the target and identify a PMTfor the FY24-25 contract period.3
Juvenile Courts: FY21 PBC ResultsQUALITY MEASURESMETRICProgram StartersTARGETNumber of youth who start an EBP100% of youth identified to start an EBPPBC RESULTS BY PROGRAMTARGETNumber of StartersACTUALNumber of Starters%Actual / ROGRAMTOTAL4
Juvenile Courts: FY21 PBC Results ContinuedPBC RESULTS BY JUVENILE COURTTarget RangeNumber of Courts withinRangeTARGETNumber of StartersACTUALNumber of Starters100% 4253175.0 – 99.0%313912850.0 – 74.9%997159710.0 – 49.9%12807265 9.9%5540TOTAL331,9961,021*Performance Management Tools were not required. This is baseline data.5
Juvenile Courts: Performance Feedback LoopEach Court reports client demographic data,risk level, and program starts and completionsthrough PACT (juvenile court risk assessment.AOC houses and processes the data.The inclusion of starter targets in thecontract has enabled the courts and JR tolook at service gaps and where best to directresources.CollectDataAnalyzeDataAOC reports performance data to each Courtannually and recently started sharing thedata with DCYF to support PBC data analysis.ImprovePerformanceShareResultsDCYF currently produces an annual report, including program starts and completions by EBPwith results disaggregated by race & ethnicity.Future Plans Design and develop a quarterly PBC data dashboard to include program starts and program completes, disaggregated byrace/ethnicity and risk level to examine disparities.Ensure Data Share agreement provides the data necessary to achieve continuous improvement in PBC metricsCreate a data sharing and review process for continuous communication and improvement with the Courts.6
Juvenile Courts: Updated PBC Logic ModelClient Population: Justiceinvolved youth served by localcounty juvenile courts andtheir families.Logic Model Statement:Juvenile Courts help justiceinvolved youth and theirfamilies by delivering EBPswith timely and quality riskassessments and modelfidelity in order to reduce riskfactors / increase protectivefactors and reduce recidivismin support of youthmental/behavioral health.7
Juvenile Courts: FY22-23 PBC Metrics, Targets & PMTsSERVICE MEASURESPerformanceManagement Tool (PMT)MetricTargetProgram StartNumber of youth who start an EBP100% of youth identified incontractor’s application to startan EBPN/A; not required for PBCservice metricsProgram Completion% of youth who start an EBPcomplete the programTBDN/A; not required for PBCservice metrics.QUALITY MEASURESMetricTargetPerformanceManagement Tool (PMT)Timely RiskAssessmentsAssessments completed withinpolicy and standardsTBDN/A; DCYF will collectbaseline data during theFY22-23 contract period.Quality RiskAssessmentsJuvenile Court staff must becertified in the assessmentTBD“ “KEYBlue text included in FY22-23 contractsGreen text measures of interest FY24-25 contracts8
Juvenile Courts: FY24-25 PBC Metrics, Targets & PMTsPerformance ManagementTool (PMT)OUTCOME MEASURES (Proximal)MetricTargetReduction is risk factors /increase in protective factorsChange in target domain scoresfrom initial assessment to finalassessmentN/A; DCYF will collect baseline data during theFY22-23 contract period.OUTCOME MEASURES (Distal)MetricTargetReduced Recidivism*No new convictions oradjudications for 18 monthsfrom the start of communitysupervisionN/A; not required for PBC distal outcomes*Pending Accountability Group approvalKEYPerformance ManagementTool (PMT)Green text measures of interest FY24-25 contracts9
Juvenile Courts: FY22 Stakeholder Engagement ResultsSummary of Stakeholder FeedbackEngagement HistoryStakeholdersMethodsPresentations:Semi annually (Sept – Feb) WAJCA Block Grant OversightCommitteeAnnually (Dec) CJAA AdvisoryCommitteeQuarterly Stakeholders agree with the new direction of PBC – focusingon timely and quality risk assessments. They feel this work isalready underway and is a direct connection to programoutcomes Stakeholders are excited about the plan for data dashboardsas a tool to track progress on quality metrics and eventuallyoutcomes. AOC is currently experimenting with technology todo this. Courts have given feedback to the Performance ImprovementPlan (PMT) template and understand the importance ofaccountability for services.Next Steps Work with AOC and WAJCA on the initial iteration of PBC data dashboard and collect feedbackEnsure data exchange with AOC is current and on going in alignment with the DSAObtain data from AOC to establish targets for Quality and Outcome Metrics10
Juvenile Courts: FY22 ReflectionsCelebrating Success Navigating ChallengesStakeholder Engagement: On going support fromJuvenile Courts stakeholders regarding PBC.Focus on Data and Learning: Courts are excited to usedata to support learning and improve performance.AOC taking the lead on data dashboard: This work is ongoing but AOC is willing and able to create thesedashboards. The hope is to have for use in the nextfiscal year.Updated Data Sharing Agreement Data Access Support: AOC provides data to DCYF butthe original data exchange did not include all necessarydata points for determining appropriate risk-adjustedperformance targets on current and future metrics. Thisresulted in an incomplete retrospective analysis. Lack of Research and Data Analytic Support: AOC canonly provide a minimum level of support, and DCYF haslimited resources to provide PBC support. COVID-19 has impacted the juvenile courts’ ability todeliver programs as they once have.Support Needs Expanded Research and Data Analytic Support – either through AOC (additional funding) or DCYFBegin PBC process for Disposition Alternatives program as a continuation of the Juvenile Courts Workgroup.11
PBC Contract GroupAnnual UpdateOffice of Juvenile Justice (OJJ)Provider Services Quality & Availability Accountability Group & PBC Support TeamApril 20, 2022www.dcyf.wa.gov1
Office of Juvenile JusticeBackground InformationMission: OJJ provides leadership, education and coordination of juvenile justicesystem improvement and reform efforts. OJJ leads innovation and collaborationto reduce the impact of racial and ethnic disparities throughout the juvenilejustice system. By providing compliance monitoring, funding and training, OJJacts as a resource to juvenile justice system partners.Annual Contract Spending (FY21): 1.3 millionContract GroupLead: Jenny Young, Juvenile Justice ManagerMembers: Alice Coil, Deputy DirectorPBC Staff: Stacey Gillette, PI AdministratorCurrent Contract Cycle (FY22): July 2021-June 2022# of Contracts (client services) (FY22): 13Implementation Status: Phase 3 PBC Service Standard: Currently tracking services to implement in FY21-22 contracts PBC Quality Standard: Currently tracking at least one (1) quality metric and target in FY21-22 contractsX PBC Outcome Standard: Will incorporate at least one (1) outcome metric and target in FY22-23 contracts2
Invest in systemImprovementInitiativesCurrently e-improvement/ojj3
The PartnershipCouncil of JuvenileJusticeOrganizationalStructureState Advisory GroupWA Partnership Council on Juvenile JusticeJuvenile Justice andDelinquency Prevention Act(Federal)DesignatedStateAgency:Executive Order20-02(State)DCYFOffice ofJuvenileJusticeState Advisory Group:Partnership Council onJuvenile Justice26 members appointedby the Governor, including fiveyouth ovement/ojjJuvenile JusticeSystemImprovementInitiatives4
Priority CommitmentFFY 2020 Disparity Chart at Five Contact PointsEliminate the Impactof Racial & EthnicDisparitiesAge 10-17Source: Washington State Center for Court Research
2020 Governor’s provement/ojj6
JDAI Data Report provement/ojj/jdai/jdai-outcomesAdmissions to Detention by Calendar ,6044,1554,0003,0002,1272,0001,0000JDAI SitesNon JDAI Sites20162017201820192020Note: 2020 CY detention admissions have not yet beenvalidated, this is a close estimate, and only for JDAI sites.7
OJJ: Updated PBCLogic ModelLogic Model Statement:The OJJ Contract Groupprovides funding andtechnical assistance tosystem and communitypartners in order to achievethe outcomes of reducing theimpact of racial and ethnicdisparities in the juvenilejustice system and alsoprovides capacity building forbest practices and/orpromising programs insupport of improved youthdevelopment and youthmental/behavioral health.SERVICESQUALITYOUTCOMES(Activities and Outputs)(Process Measures)(Results)PERFORMANCEMANAGEMENT TOOL(PMT)Contractors are expected to track client-level service data (including eligible, enrolled, and completed) forinternal program management purposes, but may not be required to report client-level service data to DCYF.Description of Services- Contractors will work withOJJ to create and adoptpolicies outlining bestpractice requirements forstaff in the collection ofrace and ethnicity datafrom clients.- Submission of requiredquarterly progress anddata reports(disaggregated by gender,race and ethnicity)- Attendance at requiredsteering committeeand/or contractormeetings- Contractors willcollaborate andcoordinate with CBOsand/or system partners toachieve implementationplan goalsProximal Outcome(s):- 50% of contracts will havea completed and approveddata collection policy forrace and ethnicity beforethe first quarterly datareport is due.- 80% of contractors willsubmit quarterly progress& data reports on time &accurately- 90% of contractors willattend and activelyparticipate in requiredmeetings- 85% of contractors willprovide evidence inquarterly reports ofcollaboration with CBOsand/or system partnersDescription of ServicesFinancial reports (invoices)are held until the quarterlyprogress reports arereceived.Final invoices are held untilthe annual evaluation issubmitted.supported bycontracted servicesResilience Children and youth aresupported by healthyrelationships with adults Parents and caregiversare supported to meet theneeds of children andyouth Family economicsecurityEducation Kindergarten readiness Youth schoolengagement High school graduationJDAI Contracts- Increase in referrals ofyouth to CBO’s forcommunity basedservices/interventions- Reduction in racial andethnic disparities- Increased use ofalternatives to detentionHealth Healthy birthweight Child/youthdevelopment Youth mental/behavioral healthDistal Outcome(s):- Technical Assistance to beprovided to contractorsupon request- 90% of contractors willconfirm the TA receivedmet the identified need intheir request- Ensure Youth and Familyvoices are valued andhonored- 80% of Youth and Familyparticipants surveyedreport positiveinteractions with OJJ andall partners- Ensure continuous andstrong partnershipAll Contracts- Improved accuracy in raceand ethnicity datareporting- Increase in shared learningacross systems and/orCBO partnersDCYF GOAL(s)- Contractor capacityexpands and success ofprograms grow- Model approach forengaging with youth andfamilies is fully integrated- CBOs are valued as vitalpartners to improvedsystem outcomes.N/A; does not applyto system-level8
OJJ: PBC Quality Metrics, Targets & PMT (All Contracts)QUALITY MEASURESMetricData collection policy for raceand ethnicity data.Creation and submission of the policy to OJJData ReportingSubmit quarterly reports on time &accuratelyKEYTargetPMT50%If not met: Technicalassistance provided80%If not met: Invoices heldfor payment until reportsreceivedGreen text measures of interest FY22-23 contractsBlue text already included in FY21-22 Contracts9
OJJ: PBC Quality Metrics, Targets & PMT (System Level)QUALITY MEASURESMetricTargetTechnical Assistance to be providedto contractors upon requestContractors will confirm the TA receivedmet the identified need in their request90%Ensure Youth and Family voices arevalued and honoredYouth and Family participants surveyedreport positive interactions with OJJ andall partners80%Ensure continuous and strongpartnership between community andsystem partnersPublic meetings will have CBOsrepresented90%Third Party Evaluations submittedwith a consistent outlineAll requested sections included in theevaluations80%KEYPMTIf not met: improverecruitment, meaningfulengagementWhen met: Evaluatorsplaced on a preferred list.Green text measures of interest FY22-23 contracts10
OJJ: PBC Outcome Metrics, Targets & PMT (Contract Level)OUTCOME MEASURESMetricAll ContractsImproved accuracy in race and ethnicity datareporting.TBDIncrease in shared learning and capacity building.TBDJDAI ContractsReduction in racial and ethnic disparities.Detention AdmissionsTargetPMTIncrease in referrals of youth to CBOs for communityDiversion Countsbased services/interventions.Increased use of alternatives to detention.KEYDetention Admissions 5% annualreductionIf not met: RevisedImplementation PlanGreen text measures of interest FY22-23 contracts11
OJJ: PBC Outcome Metrics, Targets & PMT (System Level)OUTCOME MEASURESMetricTargetPMTContractor capacity expandsand success of programsService capacity increases.grow.Model approach forengaging with youth andfamilies is integrated.Policies and processes havebeen agreed upon andoperationalized.Youth and familiesconsistently serve onover 50% of OJJ/WAPCJJCommitteesIf not met: improverecruitment, meaningfulengagement and capacitybuilding activities.1 program every 3 yearsIf not met: identifyadditional fundingCBOs are a valued partner in Increase in # of DCYF contractsachieving outcomes.with CBOsBest practices andpromising programs areidentified.KEYFederal & State fundedprograms added to the WISPPEBP ListGreen text measures of interest FY22-23 contracts12
OJJ: Performance Feedback LoopData is submitted on a quarterly basis from allclient service contractors to the contractmanager in a provided/approved format. Title IIContractor data is submitted to OJJDP quarterly.CollectDataContinued investment in collaboration andImproverelationships with system partners, programs, andCBOs. Working with contractors to submit reports Performanceon time.AnalyzeDataShareResultsTitle II Contracts provide a third party programevaluation report to OJJ. System level data isanalyzed and published every other year throughthe Governor’s Report with the assistance ofWSCCR. JDAI data is analyzed every other yearthrough a contract with WSCCR.All reports are posted on the OJJ website and anannouncement email is sent to WA-PCJJ and OJJpartners when a new data report is published.Future Plans Development of PBC Data dashboard Policy and practice improvement of client self-identification of race and ethnicity13
OJJ: FY22 Stakeholder EngagementStakeholdersContractors - CBOsContractors – CourtsMethodsQuarterly meetingsOne-on-one meetingsContractor orientationAnnual Contractor Site VisitsReview of Contractor Annual EvaluationsJDAI Site Implementation Plan Reviews14
OJJ: FY22 ReflectionsCelebrating Success Robust Contractor OrientationQuarterly Report TemplateReports include Race/Ethnicity DataCompleted Tracking Process for ReportSubmissionNavigating Challenges Accurate representation of OJJ contract fundingimpact on total clients served (some contractorshave multiple funders)Support Needs Presentation from DCYF: OIAA on race and ethnicity data identification15
Appendix16
System-Level PBC ContractsFocus onCapacity BuildingSome DCYF contracts focus on building thecapacity of contractors and/or entire systems ofcare (e.g. juvenile justice system) rather thanonly funding direct service provision.Variation in Servicesand/or ContractorsThese contract groups operate more like grantprograms, funding different services acrosscontractors and/or significantly changingcontractors each cycle, making it difficult tomeasure standardized PBC metrics over time.17
System-Level PBC RequirementsContract groups in this category must adopt the following PBC components:PBCLOGIC MODELClient-Level(Service, Quality, & Outcomes ORMANCEMANAGEMENT---Contractor-Level(Quality & Outcomes Standards)System-Level-(Quality & Outcomes Standards)Legend:PBC RequirementApproved by the PBC SteeringCommittee, 03.11.20Developed with support fromThird Sector Capital Partners, Inc.18
PBC Contract GroupAnnual UpdateReentry (Employment Services)Provider Services Quality & Availability Accountability Group & PBC Support TeamApril 20, 2022www.dcyf.wa.gov1
Reentry (Employment Services)Background InformationContract GroupMission: Support the successful transition of reentry youth throughworkforce skills and job training opportunitiesClients Served (FY21):165 (PacMtn); 17 (ANEW); 24 (AJAC); 24(HERO); 22 (BFA)Annual Contract Spending 733,000 (FY21): 526,000 (FY20PacMtn); 73,000 (FY21 ANEW); 23,000 (FY21 AJAC), 56,000(FY21 HERO); 55,000 (FY21 BFA)Current Contract Cycle (FY22): July 2021-June 2022Lead: Lisa McAllister, Office Chief: Reentry and TransitionMembers: Karena McGovern, Contracts Manager; DesireeCheung; Business Relations Administrator; Gerdon Jones,Post-Secondary AdministratorPBC Staff: Stacey GilletteData Partner(s): Andrew Fox - consultation only# of Contracts (client services) (FY22): 5Implementation Status: 2 PBC Service Standard: Collects client level service data from billing and ACT PBC Quality Standard: Quality metrics currently included in contractsX PBC Outcome Standard: Possible outcome metric, target and PMTs identified for FY23 contracts2
Reentry: FY21 PBC Metrics, Targets & PMTsQUALITY MEASURESReentry EmploymentReadiness ServicesCompletionMetric% of youth who completethe programTarget85%Performance ManagementTool (PMT)N/A: DCYF will collectbaseline data andwork with thecontractor to identifyand implementneeded performanceimprovementstrategies.3
Reentry: FY21 PBC ResultsQUALITY MEASUREProgram CompletionMETRICTARGET% of youth who complete theprogram85%RESULTS BY CONTRACTORNumber of Completers /Starters% of youth who completeMet / Did Not MeetMyJOB56 / 16533%Did not meet targetANEW13 / 1775%Did not meet target*AJAC20 / 2483%Did not meet target*HERO20 / 2483%Did not meet target*BFA16 / 2273%Did not meet target*PROGRAM*Less than 25 data points4
Reentry: Performance Feedback LoopCompletion data (one time requests) collectedthrough ACT by researcher. Need to determineon-going data access from Digital Innovation.Developing a data dashboard has brought tolight a need for data analytics for all JRservices in DCYF. See Support ResultsInitial pull of ACT data was analyzed for quality.On-going data collection continues –reporting has been paused.Results will be shared with contractors andDCYF facility staff during existing quarterlymeetings to drive reporting improvements anddeveloping equity driven opportunities.Future Plans Identify process for obtaining JR data in ACT on a reoccurring basis.PBC data dashboard will be developed and shared with contractors.5
Reentry: Updated PBC Logic ModelClient Population: Young adultsreleasing from incarcerationinto the community.Logic Model Statement:Reentry helps young adultsreleasing from incarcerationinto the community bydelivering job readiness servicesand pre-apprenticeship trainingwith dosage and certificationsand/or college credit in order toachieve reduction in need,reduced recidivism, andemployment and/or enrollmentin educational programs insupport of DCYF’s resilience,education, and health goals.6
Reentry: FY23 PBC Metrics, Targets & PMTsSERVICE MEASURESProgram CompletionMetric% of youth who complete theprogramQUALITY MEASURESPerformanceManagement Tool (PMT)Target85%MetricTBDPerformanceManagement Tool (PMT)TargetDosageTBDTBDN/A; DCYF will collectbaseline data during theFY23 contract period.Certifications and/orcollege creditobtained (whenapplicable)TBDTBD“ “KEYBlue text included in FY22 contractsGreen text measures of interest FY23 contracts7
Reentry: FY23 PBC Metrics, Targets & PMTsPerformance ManagementTool (PMT)OUTCOME MEASURES (Proximal)MetricTargetReduction in needTBD; as measured by change inscore on the IntegratedTreatment Assessment (ITA)N/A; DCYF will collect baseline data during theFY23 contract period.OUTCOME MEASURES (Distal)MetricTargetReduced Recidivism*No new convictions oradjudications for 18 monthsfrom the start of communitysupervisionN/A; not required for PBC distal outcomesPerformance ManagementTool (PMT)*Pending Accountability Group approvalKEYGreen text measures of interest FY23 contracts8
Reentry: FY22 Stakeholder Engagement ResultsEngagement History reement regardingreporting dosage anddefinition of programcompletion Summary of Stakeholder FeedbackMethodsFirst annual all-contractormeeting together April 2022Quarterly meetings set forall contractorsMonthly meetings forindividual contractors Contractors are most excited about understandinglonger-term employment after a youth exits the JRsystem. Contractors value program completion, but definitionsvary for each contracted service. Use of ACT as a centralized data source varies acrossstakeholders, which requires additional support fromDigital Innovation to collect data for reporting purposes.Next Steps Continue to work to access ACT data effectively within OIAA-Digital Innovation.Engage with contractors and JR facility staff to refine reporting processes for all contracted services.Engage with contractors and JR facility staff to develop a strong performance feedback loop.9
Reentry: FY22 ReflectionsCelebrating Success Navigating ChallengesAdditional Contracts: Reentry is aligning expectationsacross employment services contracts to focus on keyperformance metric(s) which allows for easyadaptation into additional contracts that are beginningin FY23.Stakeholder Engagement: Reentry staff have activelyengaged contractors, DVR partners and DCYF facilitystaff throughout the process, strengtheningrelationships and improving general buy-in. Data Quality: Improving data quality has been a focus in FY22given the limitations of ACT and will continue to be a need inFY23 – see Support Needs.Staff Turnover: Continuous staff turnover with key positionswith providers has created delays in the PBC implementationprocessData Reporting: As data quality has improved, getting accessto the data has been difficult due to the shift of responsibilityto access going from JR IT to OIAA-Digital Innovation wherethere is limited knowledge in the way ACT functions.Support Needs Finalize FY23 contract metrics, targets & PMTsData: Reporting ACT CAST data is needed. OIAA has full data access but no resources to assist or develop reporting features.Continuous Improvement: Support for the development of a PBC dashboard and ongoing performance management.10
Juvenile Courts: FY22-23 PBC Metrics, Targets & PMTs SERVICE MEASURES Metric Target Performance Management Tool (PMT) Program Start Number of youth who start an EBP 100% of youth identified in contractor's application to start an EBP N/A; not required for PBC service metrics Program Completion % of youth who start an EBP complete the program TBD
