WIXLIB

subset of Medicaid recipients who qualify under the MAGI standard. 5 OHA spent approximately 57 million on design, development, and implementation for the existing ONE system. As ofJanuary 2019, an additional 73 million has been spent on maintenance and operations.While the Integrated Eligibility Project team performs enhancements to the existing ONE system,it is known as integrated ONE; however, after implementation, it will again be known simply asthe ONE system. The integrated ONE system is scheduled to be fully implemented by November2020 and the Integrated Eligibility Project is scheduled to wrap up in January 2021.Another motivating factor to develop and implement the integrated ONE system was anopportunity from the federal government where states could obtain a funding match rate of upto 90% for developing integrated eligibility systems. The Integrated Eligibility Project initiationwas accelerated in order to maximize enhanced federal funding that was to expire December 31,2018. DHS and OHA made every effort to take advantage of this funding opportunity. Overall, theproject was able to achieve a federal funds matching rate of approximately 77%, with theremaining monies coming from the state General Fund and bond sales.Significant resources have been invested into the new eligibility systemThe integrated ONE system is a major investment of state resources, with a design, development,and implementation budget of over 380 million. Approximately one hundred fifty stateemployees and three hundred contractors are working to pilot the integrated ONE system inspring of 2020. 6 Figure 2 shows how the integrated ONE system’s budget has increased overtime. Additional discussion of factors that have impacted the budget is included in our March2019 risk assessment letter in Appendix A. The state will spend a total of about 510 million toimplement an integrated eligibility system, including past spending on the existing ONE system.The integrated ONE system will also incur additional ongoing maintenance and operations costsper year. 7Figure 2: The integrated ONE system’s budget for implementation has steadily increasedIn Millions 400 300 200 100 0July 2015December 2015 September 2016July 2017December 2017January 2019Source: Auditor prepared from Integrated Eligibility Project documentation. The July 2015 estimate was solely for a Non-MAGI Medicaideligibility system. Approximately 77% of system funding is from the federal government.Computer code for both the existing ONE and integrated ONE systems was acquired at no costfrom the state of Kentucky. Although the base code was free, there were costs associated withupdating the systems to meet Oregon’s needs and to develop various interfaces with existingcomputer systems. This work was handled in part by the state and in part by a contractedAs noted in report no. 2017-09 that system is largely functioning as intended. For reporting purposes, this system will be known asthe “existing ONE system” to help distinguish it from the Integrated Eligibility Project and the “integrated ONE system.”6 Staffing levels are as of June 2019. Staffing levels change over time in relation to the project’s workload need.7 The final cost for continued maintenance and operation of integrated ONE is subject to negotiation, but is likely to be similar to theexisting ONE system, which incurred about 73 million in expenses over approximately 4 years.5Oregon Secretary of State Report 2019-37 October 2019 Page 3

System Integrator. The System Integrator for both the existing ONE and integrated ONE systemsis Deloitte Consulting LLP (Deloitte).Many external stakeholders are involved or impacted by the Integrated Eligibility Project. Thisincludes citizens, local governments, the federal government, nonprofit organizations, advocacygroups, the Legislature, and other entities. Approximately 1 million recipients will be impactedby the implementation of the integrated ONE system.The implementation of the integrated ONE system will fundamentally change day-to-dayoperations for approximately 4,600 workers in over 100 DHS field offices across the state. 8 Theproject also involves integrating data from multiple existing legacy computer systems. Large,complex projects, such as the Integrated Eligibility Project, always have inherent risks given thesize and scope of the work involved. As the Legislative Fiscal Office has noted, agencymanagement has limited control over several of these risks, including the number of staffneeded to complete the project, the project’s complexity, changing scopes, and the requirementthat the project serve the needs of two major state agencies, DHS and OHA, through a sharedcomputer system.The Integrated Eligibility approach will provide a better experience for clients andopportunities for improved efficiencyIf successfully implemented, the integrated ONE system will allow most applicants to fill out asingle application for multiple public assistance programs instead of requiring separateapplications and visits to multiple offices. 9 Successful implementation of the integrated ONEsystem should help reduce: The amount of time Oregonians spend applying for assistance;The wait time between completing an application and receiving benefits; andThe number of manual processes state staff perform.DHS and OHA have also adopted a “no wrong door” approach where applicants can apply inperson, over the phone, by fax, and through a web portal.Figure 3: The no wrong door approach of the integrated ONE systemPhoneField OfficePaperCommunity PartnersOnline/Webintegrated ONESNAPTANFERDCMedicaidA 2017 agreement delegated authority to DHS to perform eligibility determinations and other frontline work on behalf of OHA.A number of specialized DHS and OHA programs may still require separate applications or multiple contacts with different staff.Examples include: Pre-TANF, Post-TANF, Breast and Cervical Cancer, Older American Act, Oregon Project Independence, and 24-HRMental Health Residential. Programs offered by other agencies, such as housing or energy assistance, will still require separateapplications.89Oregon Secretary of State Report 2019-37 October 2019 Page 4

In addition to a potentially better client and caseworker experience, the integrated ONE systemshould help reduce administrative errors, fraud, waste, and abuse in assistance programs byautomating eligibility determinations and creating a more accurate master client index. If thesystem can reduce errors, fraud, waste, and abuse, it could potentially yield millions of dollars ormore in annual savings.A master client index is a centralized databaseof all client records. Rather than housing clientdatabases in multiple computer systems, amaster client index consolidates this informationand reduces the risk of duplication and errors inclient information.An eligibility determination is the formalapproval or denial of benefits that occursafter the application materials are reviewed.It is a crucial milestone in the process ofapplying for public assistance.Data conversion is a crucial effort to implementing the integrated ONE systemData from multiple legacy systems and the existing ONE system will be used to populate datainto the new integrated ONE system. By using existing data, the project team can reduce theneed to perform extensive and time-consuming manual data entry into the new system.The Integrated Eligibility Project team developed a process to convert the data from existinglegacy systems into the new integrated ONE system, which included developing plans to specifyhow the data is being converted. The planning documents identify the data that is most accurateand valid for conversion. The planning documents also describe how data from legacy systemsmap to data in the integrated ONE system. See Figure 4 for a simplified example of the programdata being converted and its conversion process.Figure 4: Simplified model of data conversion process for the integrated ONE systemProgram DataERDCSNAPTANFPrecedence LogicA C, B A, C B, integratedONEMedicaidData maps show how data inone system relates to data inanother system. They are a keydocument used in dataconversions.Precedence logic is a definedranking of data elementsbuilding off data maps.Rankings are based on thequality of the information. Theprecedence logic is used duringconversion to select which dataelements should be convertedfrom each system.Defining the precedence logic and which data elements are to be converted was importantbecause DHS and OHA may have multiple datasets containing different information. Forexample, the agencies’ have more than one mailing address for many clients and will need todetermine which system housed the most accurate and up-to-date information. The project teamalso needed to develop mapping documents to show how data elements in the legacy systemswould relate to data elements in the new integrated ONE system. For example, in the legacysystem, something might be formatted numerically, as 1, 2, or 3, but in the integrated ONEsystem that same data might be formatted alphabetically, as a yes or a no. Lastly, a testing planwas developed to help ensure that the conversion followed the specifications provided by DHSand OHA.Oregon Secretary of State Report 2019-37 October 2019 Page 5

To a large extent, the plan is to rely upon information within the existing ONE system, as thatdata had a robust collection process in recent years. DHS and OHA program managers and theIntegrated Eligibility Project team believe the information within the existing ONE system issufficiently complete and accurate. When client information does not exist within the existingONE system, data will be pulled from older legacy systems. The precedence logic contains aranking for each data element showing which legacy system contains the best information to beconverted.Oregon Secretary of State Report 2019-37 October 2019 Page 6

Audit ResultsThis real-time audit was conducted in alignment with the Oregon Audits Division’s strategicfocus of being timely and responsive. Real-time auditing focuses on evaluating front-endstrategic planning, service delivery processes, controls, and performance measurementframeworks before or at the onset of significant projects or public policy implementations bystate agencies.We found the Integrated Eligibility Project team generally followed industry standards to helpensure that data is converted completely and accurately. For example, the project teamdeveloped plans to map out how information was to flow from existing systems into theintegrated ONE solution and has developed testing plans to identify errors that could result fromthe data conversion process. Although not all of the planned activities are complete at the time ofthis report, these efforts should help minimize the risk of significant issues arising during systemimplementation. 10The Integrated Eligibility Project team is generally adhering to industry practices for dataconversion planning. However, at the time of the audit, we found that staff resource planningwas insufficient, especially relating to work requirements that will likely need to be completedafter the conversion process. A lack of adequate staffing could negatively impact clientexperience through long waits. Furthermore, though it did so for other elements of the project,the Integrated Eligibility Project team did not consistently ensure that access to sensitive datagoing through the conversion process was appropriately restricted and monitored in accordancewith industry security standards.The project team developed appropriate implementation plansIndustry standards call for detailed plans when new information technology systems are beingdeveloped. Plans should include considerations for system and data conversion, acceptancetesting criteria, communication, promotion from development to production, early productionsupport, a fallback plan, training, and a post-implementation review. In addition, organizationsshould involve key technical and business stakeholders and clearly define roles andresponsibilities.The Integrated Eligibility Project team and the contracted System Integrator worked together todevelop various plans for the integrated ONE system project. One of the key phases of theIntegrated Eligibility Project is data conversion, a process where data from an older legacysystem is extracted, transformed, and loaded into the new system. This complex processrequires adequate planning to help reduce data errors and implementation problems.Multiple planning documents were created that cover various elements of the data conversionprocess. These plans were reviewed by an independent, third-party quality assurance contractorand approved by the project team and the Office of the State Chief Information Officer.We reviewed over 30 documents related to system implementation, focusing on elementsrelated to data conversion. A selection of reviewed documents is highlighted in Figure 5. Wefound the Integrated Eligibility team generally followed the broad principles outlined byindustry standards. By performing sufficient due diligence in their planning efforts, theIntegrated Eligibility Project team has helped to reduce the risk of issues with data conversion. Asuccessful conversion will help minimize staffing needed to perform manual data entry or cleanup activities, freeing up resources to focus on improving the client experience.10It is normal for work to be in progress during a real-time audit.Oregon Secretary of State Report 2019-37 October 2019 Page 7

The Integrated Eligibility Project team developed testing plans, which contained neededprocedures and guidelines for defect management and data validation. These plans areimportant because defect management includes identifying, logging, and performing triage tomanage and close defects that have been discovered during testing. Performing data validationprovides metrics for success rates, but also seeks to validate that the data is as expected —converted whole with no data loss. The Integrated Eligibility Project team also developed officesimulations and automated testing that help to simulate real-life scenarios and efficiently ensurethat data has been appropriately converted from existing legacy systems.Figure 5: Reviewed plans included design, development, planning, and testingDesignDevelopmentFunctional DetailConversion SpecificationsConversion PlanApplication SecurityApplication DevelopmentApplication ConfigurationSystem ImplementationInformation Resource Mgmt.Configuration ManagementPlanningProject CommunicationsTechnology Mgmt.TestingProject Mgmt. FrameworkBusiness OperationsTrainingSystem SecurityPeople Readiness CommunicationOverall TestingFunctional SecurityTest ExecutionConversionPerformance & StressAccessibilityNote: The above list does not contain all documents that were reviewed.Although the plans generally met industry standards, we did note some elements could beimproved. A few testing plans, including the Functional Security test plan dated October 16,2018, did not always establish clear criteria for measuring the successful completion of each testplan. The Functional Security test plan stated “The entrance and exit criteria only applies to FinalSIT [System Integration Testing] as mutually agreed upon by State and Deloitte. There is no exitcriteria to functional security test activities defined in this plan.” In addition, plans generally didnot establish remediation procedures when success criteria is not met. The Integrated EligibilityProject team is continuing to finalize various plans and documents such as the Cutover Plan,Accessibility Test Plan, System Security Plan, Environment De-Provisioning Plan, and DisasterRecovery Plan. These are important planning documents that the Integrated Eligibility Projectteam has scheduled for completion later in the project, so they were not available for reviewduring the audit.Throughout the project, the Integrated Eligibility team also generally followed appropriatestandards around change management. For example, the team has processes in place todocument changes to policy, procedures, or the computer systems when needed. These changesmay be needed due to updates in laws or to remediate errors found during testing or toaccommodate design changes.We noted during our review that the Integrated Eligibility Project team did not always updateproject documentation and plans or ensure key stakeholders had the most up-to-dateinformation. Although substantial documentation issues were not identified during our review,outdated documentation increases the risk of missing critical steps during the finalimplementation of the project. The Integrated Eligibility team has scheduled a review of criticaldocuments for October 2019.Oregon Secretary of State Report 2019-37 October 2019 Page 8

Processes are in place to help ensure appropriate data conversionIn order to help identify errors and reduce production issues, industry standards call forperforming data conversion practice runs, also known as mock runs. The Integrated EligibilityProject team’s plans include seven mock runs of the data conversion process. Mock runssimulate converting the data from legacy systems being replaced by the integrated ONE system.After completion of each mock run conversion, the Integrated Eligibility Project team validatesand verifies the accuracy of the data through multiple reports. The reports include two keyperformance indicators: conversion success rate and benefit match success rate.A conversion success occurs when the integrated ONE system receives data through the dataconversion process. For example, if the existing computer system lists a client’s name as JohnSmith and any alphanumeric data is converted into the name field in the integrated ONE system,a target conversion success has occurred. This metric only measures if data passes through theconversion process, not the quality of the converted data. Therefore, even if inaccurate data isconverted into integrated ONE, it can still achieve a successful metric for target conversion. 11A benefit match occurs when the integrated ONE system calculates the same level of benefits asidentified in existing computer systems. For example, if a family of four is currently receiving 250 per month in food assistance and the integrated ONE system calculates the benefit rate tobe 250 per month, a successful benefit match has occurred. If the value is higher or lower thanthe existing benefits, a benefit mismatch occurs.The benefit match success rate is important because it helps measure the accuracy of the databeing converted. However, there are valid reasons for why a benefit mismatch occurs. In fact,due to the nature of data conversion, benefit mismatch is expected because income data wascollected at different times in different systems. A benefit mismatch is not always a negativeoutcome. DHS and OHA have made policy decisions to delay requesting additional informationfrom existing clients until the integrated ONE system has been fully implemented. Rather thanupdating information in legacy systems to go through the conversion process, the agencydecided it was best to wait until the integrated ONE solution is working to perform the dataentry. In these situations, there are known clients in a mismatch state, but as long as thoseexceptions are tracked and remediated later there is minimal risk associated with the mismatch.The Integrated Eligibility Project director is currently tracking policy decisions, such as thosedescribed above, and reported they will be monitored during system implementation.Individuals with benefit mismatch will continue to receive the benefits they currently receivefrom legacy systems until agency staff can review their case in the integrated ONE system.The Integrated Eligibility Project team intends to achieve acceptable data quality by performingseven mock conversion runs. After each run, performance metrics are compiled and data issuesare identified, triaged, and resolved. For example, during one mock run, employees reported tous that an issue with the precedence logic was identified and corrected for future runs. Somedata issues are flagged for immediate remediation by a team of four business analysts, whileother items need additional research, and various data issues are being held off until theintegrated ONE system is implemented.Following testing during the mock runs, the project team plans to hold a phased rollout thatincludes piloting the new integrated ONE system in two Oregon counties: Jackson and Josephine.The pilot is scheduled to run from April 6, 2020, to July 31, 2020. After the pilot there will be twoadditional waves, as shown in Figure 6. Wave 1 will encompass the Willamette Valley, Southern,and Eastern Oregon as well as most of the coast and is scheduled for August 3, 2020, to11 Although the conversion success metric does not review the data content specifically, the IE Project Team has other mitigatingcontrols to validate the quality of the converted data, such as examining benefit match rates and data validation reports. The dataconversion test plan also includes the use of computer scripts to validate data mapping and identify data quality issues.Oregon Secretary of State Report 2019-37 October 201

The integrated ONE system will also incur additional ongoing maintenance and operations costs per year. 7. Figure 2: The integrated ONE system's budget for implementation has steadily increased . Source: Auditor prepared from Integrated Eligibility Project documentation. The July 2015 estimate was solely for a Non-MAGI Medicaid eligibility .