Transcription
April 7, 2017AUDIT OF THE CLOUD MIGRATION & UPGRADE TOPEOPLESOFT SYSTEMST. Bert Fletcher, CPA, CGMACity AuditorHIGHLIGHTSHighlights of City Auditor Report #1706, a report to the CityCommission and City managementWHY THIS AUDIT WAS CONDUCTEDThis audit was conducted to evaluate and report onmanagement’s efforts to transition the City’s PeopleSoftFinancials and Human Resources (HR) systems to a cloudenvironment and to subsequently upgrade those two majorsystems. Because this major information technology project(Project) is anticipated to take more than two years tocomplete, this audit is being conducted in two phases. Thisreport covers the first phase, which is represented by theperiod from the start of the Project through completion of thetransition, or migration, of the two systems from aninternally managed environment to a cloud environment. Thesecond phase will address the City’s remaining efforts toupgrade the two systems subsequent to the cloud migration.WHAT WE FOUND1. The City’s PeopleSoft Financials and HR systems weresuccessfully migrated to and are currently operating in acloud environment.2. The City followed and met most of the best practicesidentified for successful migrations to a cloudenvironment, and did not suffer any identifiable adverseimpacts for the practices not initially followed or met.3. Efforts to subsequently upgrade the two City systems tocurrent versions were suspended due to challengesresulting from a lack of clarity and specificity in certaincontractual terms and conditions regarding roles andresponsibilities of City and contractor staff.4. Payments totaling 1.2 million made by the City tocontracted Project vendors were generally correct;however, enhanced Project planning and scheduling, aswell as stronger negotiation and contractual restrictions,would likely have reduced those costs. Also, aninadequate understanding of certain contractualprovisions resulted in an undetected overbilling andoverpayment for hosting services.5. It likely would have been more beneficial to the City ifcompetitive proposals had been solicited and evaluatedin connection with the selection of Project contractors.6. The contract for upgrade services should have beenstructured differently to reduce the City’s exposure tocertain financial risks.7. Cloud hosting costs associated with the transitionedsystems are expected to exceed initial Project estimates.The full report may be obtained from the City Auditor’s g-auditreports.aspx. Formore information, contact us by e-mail at auditors@talgov.com orby telephone at 850/891-8397.The City successfully migrated (transitioned) thePeopleSoft Financials and Human Resources systems to acloud environment. However, efforts to subsequentlyupgrade the two City systems to current versions weretemporarily suspended.WHAT WE RECOMMENDED1. The City should establish a formal policy or procedure togovern City information technology systems operated in a cloudenvironment. That policy or procedure should address industrybest practices.2. The City should continue efforts to execute contractamendments that are in the best interest of the City such that theupgrade services can be resumed. Among other things, thoseamendments should establish a maximum price (fee) that will bepaid for the remaining upgrade services. In the event thoseefforts are not successful, the City should develop alternativeplans to timely upgrade the PeopleSoft Financials and HRsystems.3. Planning and scheduling of contractor site work on City projectsshould be enhanced to reduce associated vendor travel costs thatare reimbursed by the City. Future contractual provisionsregarding reimbursable vendor travel costs should establishmaximum amounts that will be reimbursed.4. Vendor invoices should be timely reviewed and paid, withsupport for amounts paid properly retained in City records.Supervisory review and approvals of those invoices should bedocumented.5. City staff responsible for reviewing and approving contractorinvoices should obtain complete and proper understandings ofapplicable contractual terms to ensure future overbillings do notoccur.6. For future projects of the nature addressed by this audit,competitive proposals from multiple vendors should be solicitedand evaluated in connection with the acquisition of neededservices.7. Enhancements should be made to the City’s procurement policyto help avoid perceived (or actual) conflicts of interest in thevendor selection process.8. For future projects of the nature addressed by this audit,contracts for needed services should be structured to providemaximum amounts the City will pay for those services. Suchcontracts should also establish milestones, and provide forpenalties when those milestones are not met.9. For future projects of the nature addressed by this audit,contracts for needed services should require performance bondsinsuring the City for the value of the contracted services.Insurance provisions should require adequate levels of propercoverage and provide for adequate protections of the City. TheCity’s Risk Management Section should be consulted in makingthose determinations.10. Management should continue efforts to reduce hosting costs.We would like to thank staff in the City’s Technology andInnovations Department and in Procurement Services for theircooperation and assistance during this audit.Office of the City Auditor
Audit of the Cloud Migration &Upgrade to PeopleSoft SystemsReport #1706April 7, 2017
This page intentionally left blank.Copies of this audit report #1706 may be obtained from the City Auditor’s website eports.aspx), by telephone (850 / 891-8397), by FAX (850 / 891-0912), by mail, or in person (City Auditor, 300 S. AdamsStreet, Mail Box A-22, Tallahassee, FL 32301-1731), or by e-mail (auditors@talgov.com).Audit conducted by:Patrick A. Cowen, CPA, CISA, CIA, Senior IT AuditorT. Bert Fletcher, CPA, CGMA, City Auditor
Cloud Migration & Upgrade to PeopleSoftReport #1706Table of ContentsExecutive Summary . 1Objectives, Scope, and Methodology . 9Background. 11Audit Objective #1: Vendor Selection & Contract Execution 18Audit Objective #2: Project Expenditures . 33Audit Objective #3: Best Practices . 40Audit Objective #4: Project Status &Successes & Challenges 47Overall Conclusion . 55Appointed Official’s Response . 55Appendix A: Additional Best Practices . 59Appendix B: Management Action Plan . 61i
Report #1706Cloud Migration & Upgrade to PeopleSoftThis page intentionally left blank.ii
Audit of the CloudMigration & Upgrade toPeopleSoft SystemsReport #1706ExecutiveSummaryThis audit was conductedto evaluate and report onmanagement’s efforts totransition the City’sPeopleSoft Financialsand Human Resourcessystems to a “cloud”environment and tosubsequently upgradethose two major systems.T. Bert Fletcher, CPA, CGMACity AuditorApril 7, 2017With the assistance of a contracted vendor, the City successfullymigrated (transitioned) the PeopleSoft Financials and HumanResources systems to a cloud environment. For the most part,City staff followed industry best practices during the migrationprocess. Efforts to subsequently upgrade the two City systemsto current versions after the transition to a cloud environmentwere, however, suspended due to challenges resulting from alack of clarity and specificity in certain contractual terms andconditions regarding roles and responsibilities of City andcontractor staff. Payments made to the contracted vendors inconnection with the transition to the cloud environment andsubsequent upgrade efforts were generally appropriate andcorrect. Enhanced project planning and scheduling as well asstronger negotiation and contractual restrictions would likelyhave reduced those costs. Furthermore, while applicablecontracts were executed with legitimate vendors for the neededservices through authorized processes, it likely would have beenmore beneficial to the City if vendors had been selected using adirect solicitation of proposals through a competitive process.Lastly, hosting costs associated with the transition and upgradeefforts are expected to exceed initial City projections.This audit was conducted to evaluate and report on management’sefforts to transition the City’s PeopleSoft Financials and HumanResources systems to a “cloud” environment and to subsequentlyupgrade those two major systems. Because this major informationtechnology project (Project) is anticipated to take more than twoyears to complete, this audit is being conducted in two phases. Thisreport covers the first phase, which is represented by the period1
Report #1706Cloud Migration & Upgrade to PeopleSoftfrom the start of the Project through completion of the transition, ormigration, of the two systems from an internally managedenvironment to a cloud environment. The second phase will addressthe City’s remaining efforts to upgrade the two systems subsequentto the cloud migration.Four objectives wereestablished for this audit.The specific audit objectives included the following: Determine if the vendors associated with the Project wereselected in accordance with best practices and if contractsexecuted with those vendors were appropriate, adequate, and inthe best interests of the City.Determine if payments to Project vendors were reasonable,appropriate, supported, properly approved, and in accordancewith controlling contractual provisions.Identify best practices relating to cloud computing anddetermine if the migration of the two PeopleSoft systems to acloud environment was conducted in accordance with thosepractices.Determine and report on the overall status of the Project, toinclude successes and challenges.Vendor Selection and Related ContractsAn evaluation andanalysis by the Cityshowed retaining andupgrading the twoPeopleSoft systems wasthe City’s best option.After determining Oracle Corporation (Oracle) would no longersupport the PeopleSoft Financials and Human Resources (HR)versions used by the City, management researched industry trendsand identified options. One option considered by the City was thereplacement of the two PeopleSoft systems with non-PeopleSoftsystems. However, evaluation by City staff showed available nonPeopleSoft systems did not have all desired functions and theanticipated implementation costs were higher than the anticipatedcosts of upgrading the two PeopleSoft ERP systems. Additionally,City management determined the need to train City employees inthe use of new systems would further increase the costs ofimplementing non-PeopleSoft systems. Accordingly, managementdecided to retain and upgrade the two PeopleSoft systems to currentversions that are supported by Oracle, and to also transition thosesystems to a cloud environment.2
Cloud Migration & Upgrade to PeopleSoftThe City executedcontracts with twovendors related to thisProject, one with Ciberand one withCenturyLink.Issues were identified inregard to the vendorselection process andadequacy of contractualprovisions.Report #1706The City executed contracts with Ciber, Inc. (Ciber) to assist theCity in transitioning to a cloud environment and to subsequentlyupgrade those two systems to current versions. The City executed acontract with CenturyLink, LLC (CenturyLink) for the cloudhosting services. Those vendors were selected and applicablecontracts were executed through authorized processes. However, itlikely would have been more beneficial to the City if the vendorshad been selected using a direct solicitation of proposals through acompetitive process. Because the City did not solicit competitiveproposals, the City cannot demonstrate the services were acquiredunder the most favorable terms and prices. Additionally, the mannerin which the contract for upgrade services was structured increasedthe risk the City will pay more for those services. Lastly, for each ofthe three contracts for Project services (transition, upgrade, andcloud hosting), enhanced terms and provisions requiring insuranceand liability protection would have better safeguarded the City fromcertain risks. Recommendations were made to help managementensure future services are acquired using competitive procurementmethods, and to ensure future contracts contain appropriate termsand conditions to better protect the City’s interests.Payments to Project VendorsPayments made by theCity related to thisProject were generallycorrect. However,enhanced Projectplanning and schedulinglikely would have reducedsome of the City’s costs.As of the date of our audit tests, the City had paid Project vendorsapproximately 1.2 million for their services in connection withtransitioning to the cloud environment and ongoing management ofthat environment, upgrade of the two PeopleSoft systems, and cloudhosting services. Payments by the City for those services weregenerally correct. However, enhanced Project planning andscheduling likely would have reduced some costs incurred by theCity. Additionally, stronger negotiation and enhanced contractualrestrictions regarding vendor travel costs would likely have furtherreduced Project costs. Also, enhanced understandings by Projectstaff of billing provisions within the respective contracts and theinvoices submitted by the two contracted vendors would have betterensured the payments to contractors were proper and correct.Lastly, we noted better efforts are needed to ensure contractors are3
Report #1706Cloud Migration & Upgrade to PeopleSoftpaid timely, and evidence is prepared to demonstrate Projectmanagement is reviewing and authorizing invoices prior to Citypayment. We made recommendations to address each of theseareas.Best Practices for Transitioning to a Cloud EnvironmentThe City generallyfollowed industry bestpractices during themigration process.Through research we identified 35 best practices consideredapplicable to the migration of the City’s PeopleSoft Financials andHuman Resources systems to a cloud environment. Of the 35 bestpractices, we considered 14 as the most critical to the successfulmigration of the two City ERP systems. Of the 14 more criticalpractices, we determined the City successfully followed and meteight practices; partially followed and met two practices; and didnot follow the four remaining practices. Of the six practices notfollowed or only partially followed, subsequent actions and effortswere made, after the dates those practices should have beenimplemented, that showed the City did not suffer any adverseeffects as a result of not initially following those practices.In addition to the 14 critical best practices addressed above, weidentified 21 other less critical practices applicable to the City’smigration to and use of a cloud environment. We determined all butone of those 21 practices were followed and/or met. While the Cityhas enacted other measures to protect and secure City datamaintained in the CenturyLink-provided cloud environment,subsequent implementation of the one practice not followed/metwill provide an additional measure to secure City data.Recommendations were made as appropriate.Project Accomplishments, Challenges, and Current StatusThe City working throughthe contracted vendors,Ciber and CenturyLink,created a reasonablysecure cloud environmentand successfully migratedthe two PeopleSoft ERPsystems to thatenvironment.Accomplishments. The City working through the contractedvendors, Ciber and CenturyLink, created a reasonably secure cloudenvironment and successfully migrated the two PeopleSoft ERPsystems to that environment. City staff are now operating in thatenvironment. Specific activities performed to achieve that successincluded:4
Cloud Migration & Upgrade to PeopleSoft Report #1706Development of primary and disaster recovery cloudenvironments in two separate data centers.Development and availability of adequate computing capacityfor City operations.Migrating City systems and data to the cloud environment.Performance of appropriate testing of the two City systemswithin the cloud environment to ensure the systems functionedadequately.Several activities relative to the subsequent upgrade of the twoPeopleSoft systems have also been successfully completed prior tothe date those efforts were suspended (as explained below under“Challenges”). Specifically, as of the date the systems weresuccessfully migrated to the cloud environment and operational (inuse by City staff), there were 21 Project tasks/deliverables that wereto be completed by Ciber and/or City Project staff in regard toupgrading the two ERP systems and implementing additionalsystem modules. We determined each of those 21 tasks/deliverableshas been completed. (Note: The City’s upgrade contract with Ciberestablished 61 Project tasks/deliverables; the remaining 40tasks/deliverables were due for completion subject to the successfulmigration. The completion of those remaining 40 tasks/deliverableswill be addressed in a subsequent progress audit of the Project,conducted by our office, in the event the upgrade activities areresumed.)A lack of clarity andspecificity in certaincontractual terms andconditions, includingtasks and expected rolesand responsibilities ofCiber and City staff, aswell as differences ininterpretations of certaincontractual terms andconditions, have causedconfusion,communication issues,and delays in completionof subsequent Projectphases.Challenges. A lack of clarity and specificity in certain contractualterms and conditions, including tasks and expected roles andresponsibilities of Ciber and City staff, as well as differences ininterpretations of certain contractual terms and conditions, havecaused confusion, communication issues, and delays in completionof subsequent Project phases. Based on discussions with Projectmanagement and staff and review of related records andcorrespondence, there have been several areas where the City andCiber differed as to expectations regarding roles and responsibilitiesof both parties. For example, City management asserted that to keepthe Project on schedule, City Project staff completed certain tasksthat it interpreted to be the responsibility of Ciber, including5
Report #1706Cloud Migration & Upgrade to PeopleSoftestablishing the Virtual Private Network (VPN) that allows securetransmission of data between the City and the cloud host’s datacenter. Also, Ciber initially asserted to the City that the fees for thecloud host vendor (CenturyLink) would be 18,500 monthly.However, the City subsequently determined that fee wassignificantly higher. Notwithstanding the City should have obtaineda proper understanding of those fees before it executed the contractsand that Ciber has agreed to provide the City a credit due to themisunderstanding, this occurrence furthered management’sconcerns as to the adequacy of Ciber’s communications with theCity. Other contract interpretation differences pertain to disasterrecovery terms and conditions.Our audit shows that based on activity as of December 31, 2016, thehosting costs over the initial three-year period will likely exceedanticipated costs by approximately 327,000. Of that amount, Ciberhas agreed to provide services at the end of the upgrade of the twoPeopleSoft systems, valued at 276,000, at no charge to the City.Project management is currently exploring options to reduce futurehosting costs. We recommend those efforts be continued.Because of ongoingconcerns regardingCiber’s provision ofmanaged and upgradeservices, the City directedCiber on January 10,2017, to suspend furtherProject activities inregard to the upgrade ofthe two PeopleSoftsystems.Project Status. The City’s PeopleSoft Financials and HumanResources systems were migrated to and are currently operating in acloud environment. Accordingly, the first phase of the Project hasbeen successfully completed. However, because of ongoingconcerns regarding Ciber’s provision of managed and upgradeservices (see “Challenges” above), the City directed Ciber onJanuary 10, 2017, to suspend further Project activities in regard tothe upgrade of the two PeopleSoft systems. In that correspondencethe City informed Ciber the upgrade services were being suspendedto allow the City to develop and execute amendments to thecontract that will clarify the roles and responsibilities of each party,address Project milestones, establish more clearly defineddeliverables, and provide penalties in the event Ciber does not meetthe established milestones or provide the required deliverables.City management indicated that it is currently working on thosecontract amendments, which have been proposed to Ciber forexecution. Management indicated that if favorable amendments6
Cloud Migration & Upgrade to PeopleSoftReport #1706cannot be executed, it will terminate the upgrade contract inaccordance with existing contractual provisions.We recommend the City continue efforts to develop and executecontract amendments that are in the best interests of the City. Aspart of those efforts, we also recommend City management considerestablishing a maximum price (fee) that will be paid for theremaining services. In the event the City is not successful innegotiating appropriate contract amendments and the upgradecontract is terminated, the City should develop alternative plans totimely upgrade the two PeopleSoft ERP systems to current versions.AcknowledgmentsWe would like to thank staff in the City’s Technology andInnovations Department and in Procurement Services for theircooperation and assistance during this audit.7
Report #1706Cloud Migration & Upgrade to PeopleSoftThis page intentionally left blank8
Audit of the CloudMigration & Upgrade toPeopleSoft SystemsT. Bert Fletcher, CPA, CGMACity AuditorReport #1706Objectives,Scope, andMethodologyBecause this majorinformation technologyproject (Project) isanticipated to take morethan two years tocomplete, this audit isbeing conducted in twophases. This reportcovers the first phase,which is represented bythe period from the startof the Project throughcompletion of thetransition, or migration,of the two systems froman internally managedenvironment to a cloudenvironment.April 7, 2017This audit was conducted to evaluate and report on management’sefforts to transition the City’s PeopleSoft Financials and HumanResources systems to a “cloud” environment and to subsequentlyupgrade those two major systems. Because this major informationtechnology project (Project) is anticipated to take more than twoyears to complete, this audit is being conducted in two phases. Thisreport covers the first phase, which is represented by the periodfrom the start of the Project through completion of the transition, ormigration, of the two systems from an internally managedenvironment to a cloud environment. The second phase will addressthe City’s remaining efforts to upgrade the two systems subsequentto the cloud migration.The specific audit objectives included the following: Four specific objectiveswere established for thisaudit. Determine if the vendors associated with the Project wereselected in accordance with best practices and if contractsexecuted with those vendors were appropriate, adequate, and inthe best interests of the City.Determine if payments to Project vendors were reasonable,appropriate, supported, properly approved, and in accordancewith controlling contractual provisions.Identify best practices relating to cloud computing and todetermine if the migration of the two PeopleSoft systems to acloud environment was conducted in accordance with thosepractices.Determine and report on the overall status of the Project, toinclude successes and challenges.As previously noted, the scope of this first phase covered the startof the Project through migration of the two PeopleSoft systems to9
Report #1706Cloud Migration & Upgrade to PeopleSoftthe cloud. During this audit we reviewed selected Projectmanagement activities with an emphasis on vendor selection andcontracting, contract payments and other financial activities, bestpractices, and contract compliance.Audit procedures performed to meet our stated objectives included: We conductedappropriate auditprocedures to meet ourobjectives. Interviewing City and contractor management and staff toobtain a detailed understanding of the Project’s purpose, goals,plans, and activities.Reviewing Project-related contracts and records to ascertainhow vendors were selected and whether best practices werefollowed in regard to vendor selection and contracting activities.Selecting and testing a sample of Project-related expenditures.Researching and identifying best practices for migration to acloud computing environment and evaluating Project activitiesto determine compliance with those best practices.Identifying key Project deliverables and reviewing relatedrecords to ascertain if they were provided by applicablecontractors.Attending periodic Project meetings held by management andstaff to help ascertain the Project’s status, accomplishments andchallenges.We conducted this audit in accordance with the InternationalStandards for the Professional Practice of Internal Auditing andGenerally Accepted Government Auditing Standards. Thosestandards require we plan and perform the audit to obtain sufficient,appropriate evidence to provide a reasonable basis for our findingsand conclusions based on our audit objectives. We believe theevidence obtained provides a reasonable basis for our findings andconclusions based on our audit objectives.10
Cloud Migration & Upgrade to PeopleSoftBackgroundReport #1706Project Definition and PurposeAn Enterprise Resource Planning (ERP) system is defined as a suiteof computerized applications (modules) that are integrated tocollect, store, manage, and interpret data from business activities.Companies that create, own, and/or sell ERP systems often improveor enhance (upgrade) those systems periodically; and, offer thoseupgraded system versions to customers (entities) using thosesystems. Customers generally must expend resources to implementthe upgraded versions. Furthermore, once a newer version has beenavailable for an extended period of time, the owner companies oftenstop providing ongoing customer support for the older versions. Asa result, customers that do not obtain the newer versions within adefined period will no longer be able to obtain ongoing support oftheir older versions for a reasonable fee.The City currently utilizesthree separate PeopleSoftERP systems. Of thosethree systems, one is usedfor financial managementand accounting(PeopleSoft Financials),a second one formanaging humanresources activities(PeopleSoft HR), and athird one for managingcustomer utility activities(PeopleSoft CustomerInformation System, orCIS).The PeopleSoftFinancials and HRsystems were initiallyacquired andimplemented by the Cityin 2001 and 1999,respectively.The City currently utilizes three separate PeopleSoft ERP systems.Of those three systems, one is used for financial management andaccounting (PeopleSoft Financials), a second one for managinghuman resources activities (PeopleSoft Human Resources, or HR),and a third one for managing customer utility activities (PeopleSoftCustomer Information System, or CIS). The Project addressed inthis audit was established by management to upgrade thePeopleSoft Financials and HR systems to more current versions ascreated and made available by the company (Oracle Corporation)that owns those two ERP systems. Management may consider asimilar upgrade of the PeopleSoft CIS system at a future time aspart of a separate project.The PeopleSoft Financials and HR systems were initially acquiredand implemented by the City in 2001 and 1999, respectively. Bothsystems have been upgraded by the City twice since their initialimplementations. The Financials system was last upgraded to theversion currently being used (version 9.0) in 2009. The HR systemwas last upgraded by the City to the version currently in use(version 8.9) in 2006. Since those last City upgrades, OracleCorporation (Oracle) has created and released additional upgradedversions of the two systems. The City intentionally did not11
Report #1706The PeopleSoftFinancials and HRsystems have not beenupgraded in recent years.Management determinedthe costs to replace thetwo PeopleSoft systemsranged from 200,000 to 2,200,000 more than thecosts to retain andupgrade those twosystems.Cloud Migration & Upgrade to PeopleSoftimplement those subsequent upgrades for the purpose of savingCity resources during the economic downturn resulting from theGreat Recession. Notwithstanding that reason, because of thosesubsequent versions, Oracle announced it would no longer supportthe versions currently used by the City.After determining Oracle would no longer support the PeopleSoftFinancials and HR versions used by the City, managementresearched industry trends and identified options. One optionconsidered by the City was the replacement of the two PeopleSoftsystems with non-PeopleSoft systems. The two non-PeopleSoftsystems given significant consideration were “Workday” and“Fusion.” However, evaluation by City staff showed those nonPeopleSoft systems did not have all desired functions and theanticipated implementation costs were higher than the anticipatedcosts of upgrading the two PeopleSoft ERP systems. Specifically,management’s analysis showed the cost differential ranged from 200,000 (Fusion) to 2,200,000 (WorkDay). Additionally, Citymanagement determined the need to train City employees in
hosting services. Those vendors were selected and applicable contracts were executed through authorized processes. However, it likely would have been more beneficial to the City if the vendors had been selected using a direct solicitation of proposals through a competitive process. Because the City did not solicit competitive
