WIXLIB

IntroductionWe conducted this performance audit of the Department ofInformation Technology’s general controls pursuant to Chapter 6 ofthe Atlanta City Charter, which establishes the City of Atlanta AuditCommittee and the City Auditor’s Office and outlines their primaryduties. The Audit Committee reviewed our audit scope in March2010.A performance audit is an objective analysis of sufficient,appropriate evidence to assess the performance of an organization,program, activity, or function. Performance audits provideassurance or conclusions to help management and those chargedwith governance improve program performance and operations,reduce costs, facilitate decision-making and contribute to publicaccountability. Performance audits encompass a wide variety ofobjectives, including those related to assessing programeffectiveness and results; economy and efficiency; internal controls;compliance with legal or other requirements; and objectives relatedto providing prospective analyses, guidance, or summaryinformation.1We undertook this audit because prior audits related to specificinformation system applications identified problems. The city’schief information officer also expressed concerns about inadequatestaffing, risks to network security, and lack of disaster recovery andbusiness continuity plans.Our audit focuses on the Department of Information Technology’sgeneral controls using the COBIT (Control Objectives for Informationand related Technology) framework. General controls relate toaccess, security, disaster recovery, change management, anddocumentation requirements that cut across information technologyapplications and systems. COBIT is a set of generally accepted bestpractices related to information technology that covers 34 businessprocesses grouped into four broad areas:21Comptroller General of the United States, Government Auditing Standards, Washington, DC: U.S.Government Accountability Office, 2007, p. 17-18.2IT Governance Institute, IT Assurance Guide Using COBIT, 2007, p. 25.Information Technology General Controls1

Plan and Organize – development of an organization’soverall technology strategy, management and investment tomeet strategic goals Acquire and Implement – identification of requirements,acquiring and implementing the technology, and developing amaintenance plan Deliver and Support – operation of applications and supportincluding security and training Monitor and Evaluate – assessment of whether the currentsystem meets the designed objectives and if controls aresufficient to comply with regulatory requirementsWe reviewed selected policies and controls for 20 of the 34 businessprocesses covered in the COBIT framework. While we reviewedaspects of all four broad organizational areas covered in theframework, we focused most attention on service delivery andsupport.We also followed up on the department’s progress in implementing23 audit recommendations open as of July 2009. The city auditor’soffice is responsible for assessing the implementation of prior auditrecommendations and reporting on management’s corrective actionsand significant findings that management has not fully addressed.We made these recommendations in three reports issued in 2007 and2008: Police Computer Aided Dispatch Data Reliability, April 2008 Review of the Oracle ERP First Payroll Run, April 2008 ERP Implementation Assessment for the City of Atlanta,June 2007 (conducted by KPMG LLP)Based on our assessment, seven of the recommendations — all madein the 2007 ERP Implementation Assessment — are no longerrelevant because they dealt with risks the city faced whiletransitioning to the new Oracle system. Most of the 16recommendations we followed up on dealt with processimprovements or strengthening controls to safeguard informationassets. Appendix B shows detail of each recommendation and statusby risk category. Management agreed with all of therecommendations.2Information Technology General Controls

BackgroundThe Department of Information Technology’s mission is tocollaborate with other departments to facilitate cost-effective useof technology. The department supports the city’s informationtechnology infrastructure and an estimated 175 applications thatother city departments use to provide services to the public ormanage administrative functions. As of October 2009, thedepartment had 111 positions (see Exhibit 1), including 30contractors. The department is organized into nine units thatprovide the following services: Business Strategic Services focuses on program and projectmanagement best practices, strategy development,governance, and using existing technology. Telecommunications is responsible for the hardware,telephone infrastructure, and vendor management forcitywide telecommunications. Network and Server Operations maintains the general fundservers, network equipment and inventory, data storage, andbackup equipment. Data Center Operations provides continuous mainframesupport for departments, prints large scale forms, handlesfile transfers, and processes checks. The Data Center acts asthe help desk after normal business hours End User Support and Help Desk work to resolve technologyissues for city users. Systems and Programming supports the city’s applicationsand databases. Enterprise Resource Planning manages the city’s Oraclesystem, which integrates procurement, finance, humanresources, and payroll transactions. Information Security protects the city’s computer data andinformation assets from security threats. Business Office provides departmental administrativesupport.Information Technology General Controls3

Exhibit 1 Department of Information Technology Organizational ChartSource: Department of Information Technology, as of October 27, 2009Note: Green text represents vacant positionsThe city budgeted almost 55 million for information technology infiscal year 2010, including the Department of InformationTechnology and information technology groups within thedepartments of Aviation and Watershed Management. About 19million of the budget was for personnel and almost 16 million wasfor consultants (see Exhibit 2).Exhibit 2 Information Technology Budgets Fiscal Year ntTOTALTotal IT Budget 29,686,392 10,997,367 14,234,823 54,918,582Personnel 6,422,553 4,409,947 8,124,638 18,957,138Consulting 9,872,800 2,491,713 3,536,495 15,901,008Source: City of Atlanta’s Oracle Financials application and FY 2010 IT budget detail as of April 20104Information Technology General Controls

Previous Audits Identified Information Technology ControlWeaknessesWeak information technology controls increase the risk of inaccurateor lost data. Previous audits identified weaknesses in changemanagement, lack of validation controls between the city’stimekeeping and payroll systems, and inadequate procedures toremove system access from former employees. The city’s financialauditor also noted control deficiencies in management lettersaccompanying the city’s fiscal year 2008 and 2009 audited financialstatements and recommended strengthening general informationtechnology and application-specific controls.Change management policies not followed. The Department ofInformation Technology did not follow its change management policywhen attempting to fix a faulty interface between Oracle — thecity’s financial management system — and the Department ofAviation’s invoicing system. We reported in our August 2009performance audit, Airport Terminal Leases, that the Department ofInformation Technology agreed to reconfigure coding in the Oracleaccounts receivable module to prevent problems affecting theposting of about one percent of aviation invoicing transactions.However, the department developed the proposed solution withoutinvolving Department of Finance functional staff or the ERP(Enterprise Resource Planning) steering committee, which isresponsible for ensuring that the system meets the city’s goals andobjectives.The change management policy calls for “owner departments,”those directly responsible for data, to approve all changes beforethe Department of Information Technology proceeds with thechange. When multiple departments own the data, as is the case inOracle, all owner departments must approve the change. Failure toinclude key stakeholders in change decisions increases the risk thatchanges will introduce new problems that could destabilize a criticalsystem. We recommended the Department of InformationTechnology involve key stakeholders and application owners early inthe change management process in order to provide time formeaningful analysis of options and identify risk to the system toaddress future problems with the system.We reported in our December 2009 performance audit, Departmentof Watershed Management Customer Information System, thatwatershed management stated that change management was thesole responsibility of the contractor that maintains its billing system.However, the city’s maintenance agreement with the contractorInformation Technology General Controls5

identified portions of change management, including testing andapproving changes before they are put into production, as the city’sresponsibility. Because watershed management misunderstood itschange management responsibilities, neither city staff nor thecontractor adequately tested a new program to calculate and applyback-billed water and sewer charges. Consequently, theDepartment of Watershed Management incorrectly applied latepenalties to 40,000 accounts before catching and correcting theerror.Overtime errors not detected. The city miscalculated overtime fornearly 1,700 employees in its first payroll processed in Oracle. Wereported in our April 2008 performance audit, Review of the OracleERP First Payroll Run, that the city overpaid about 243,000 inovertime due to improper data entries in the Kronos timekeepingsystem. The interface between Kronos and Oracle requireddepartment timekeepers to account for time more precisely thanthey had when Kronos was interfaced with PeopleSoft. Sometimekeepers had developed a practice of entering all of anemployee’s overtime for a pay period into one entry rather thanrecording the actual hours worked per day, which resulted inovertime calculation errors in Oracle. Kronos technicians wereunaware of the informal timekeeping practice and therefore didn’taddress it when testing the interface with Oracle or in training. Werecommended the chief information officer develop automated orsemi-automated controls to detect errors and validate timekeepingtotals originating in Kronos.Employees retained system access after leaving city employment.We identified accounts of former employees that were still active,which increases risk of unauthorized access or changes to sensitivedata. We reported in the Airport Terminal Leases audit that 14 of73 user accounts in the department’s billing system belonged toformer employees. Four of these accounts provided access tocreate, update, and delete both lease agreements and invoices. Inthe Department of Watershed Management Customer InformationSystem audit, we identified three of a random sample of 25 useraccounts as belonging to former employees. Failure to inactivateaccounts provides opportunities for misuse of data or fraud.Watershed management removed access for accounts that weidentified. We recommended the departments periodically reviewand recertify application users’ level of access and removeterminated users.External audits recommended additional controls. The city’sfinancial auditor recommended strengthening information6Information Technology General Controls

technology controls in management letters accompanying the city’s2008 and 2009 audited financial statements. In 2009, the financialauditor recommended that the Department of InformationTechnology regularly rotate personnel, classify confidential data,and secure administrator manuals. In 2008, the financial auditorrecommended that the department formalize its security testingprocess and save test results for at least 90 days. The financialauditor also made several recommendations to the watershedmanagement technology group, including establishing a changecontrol process, tracking employee transactions, using software tomanage updates, and regular security monitoring. We followed upon the status of the external audit recommendations and will reportlater in November.Audit ObjectivesThis report addresses the following objectives: Are controls in place to maintain data integrity and datasecurity for critical city systems? Are the staffing needs identified in the Department ofInformation Technology’s November 2009 transition planreasonable? To what extent has the department implemented previousaudit recommendations?Scope and MethodologyWe conducted this audit in accordance with generally acceptedgovernment auditing standards. We focused our tests of controls onthe network, the AIX operating system housing the Oracleapplication and the city’s two enterprise-wide systems: Oracle andKronos. Failure in any of these systems could affect all cityoperations. Oracle and Kronos are the city’s only enterprise widefinancial applications and the city’s network provides access toother citywide applications. We conducted our analysis of thenetwork, Kronos, and Oracle from January through May 2010. Ouranalysis of former employees with user access covered all employeeswho left city employment beginning December 2008 throughDecember 2009. We did not evaluate whether enterpriseInformation Technology General Controls7

information technology functions should be consolidated under theDepartment of Information Technology.Our audit methods included: reviewing Department of Information Technology policies tounderstand intended control procedures interviewing department personnel to understand howprocedures are followed in practice walking through key controls to identify areas of high risk comparing department policies and practices to the COBITframework analyzing access to Oracle, Kronos, the network, and restrictedareas of city hall to determine whether any active IDsbelonged to former city employees reviewing security settings for the Oracle operating platform reviewing the methods, calculations, and supporting data forthe department’s November 2009 staffing plan following up on the status of previous recommendations,including:ointerviewing department management and staff tounderstand the status of the recommendationsoreviewing change documentationoreviewing reports on contractor workochecking the validation process for 911 reportingointerviewing payroll to discuss the validation processoanalyzing time entries in Oracle and Kronos for thepay period ending April 14, 2010Generally accepted government auditing standards require that weplan and perform the audit to obtain sufficient, appropriateevidence to provide a reasonable basis for our findings andconclusions based on our audit objectives. We believe that theevidence obtained provides a reasonable basis for our findings andconclusions based on our audit objectives.8Information Technology General Controls

We provided the chief information officer a detailed description ofthe results of our AIX security assessment April 2010. This reportsummarizes the findings and recommendations related to security,but excludes details about specific vulnerabilities. Vulnerabilityassessments for technology infrastructure are not subject todisclosure under the Georgia Open Records Act.33O.C.G.A. § 50-18-72(15)(A)(i).Information Technology General Controls9

10Information Technology General Controls

Findings and AnalysisStronger Controls Needed to Protect Critical SystemsThe Department of Information Technology has implementedsufficient controls in 59% of the areas we evaluated, but significantrisks remain. Effective entity-wide controls help ensure thedepartment is able to support the city’s critical informationtechnology infrastructure and applications such as Oracle andKronos. Control weaknesses present risks to the city’s ability tocomply with data regulations, maintain data integrity, continueoperations or recover from a disaster, provide needed services tothe departments, and restore lost email.Department policies and procedures protect against external attacksfrom viruses, worms, spyware, adware and Trojan horses. Thedepartment has also implemented controls that protect the physicalenvironment where critical systems are housed. Its organizationalstructure aligns with key business areas and duties are segregated toreduce the possibility for a single individual to compromise a criticalprocess. The department’s use of state negotiated contractsprovides opportunities for the city to save money on hardware andsoftware purchases.We also identified areas where policies were insufficient or did notmatch practices. The department lacks disaster recovery andbusiness continuity plans, procedures to monitor security logs,assessment of legal and regulatory requirements and serviceagreements with other departments. The city has a sound changemanagement policy but technical documents were incomplete for arandomly selected change to the Oracle system. While someprocesses to manage user accounts are strong, the department doesnot enforce the city’s guidelines for strong passwords in Oracle andmore than 200 employees who no longer work for the city retainedaccess to Oracle and the network.Implemented Controls Reduce Several RisksThe Department of Information Technology has implementedpolicies and practices to reduce the city’s risk. The departmentestablished policies that govern the city’s use of firewall andantivirus programs, implemented measures to protect technologyInformation Technology General Controls11

assets, structured organizational units along business functions,developed procedures for granting and removing user access fromsystems, and has implemented tools and methods to managevendors. The department also protects Oracle through limitingchange authorization, segregating change duties, and testingchanges with the system users.Exhibit 3 summarizes the results of our assessment of departmentcontrol policies and practices compared to the COBIT framework. Inhalf of the control objectives we evaluated, the department’spolicies were consistent with COBIT and practices were consistentwith the department’s policies. In another 9% of the controlobjectives we evaluated, the department’s practices wereconsistent with COBIT, but differed from policy or a policy is not y

Our audit focuses on the Department of Information Technology's general controls using the COBIT (Control Objectives for Information and related Technology) framework. General controls relate to access, security, disaster recovery, change management, and documentation requirements that cut across information technology applications and systems.