Transcription
United States Government Accountability OfficeGAOReport to Congressional RequestersAugust 2004INFORMATIONTECHNOLOGYFAA Has ManyInvestmentManagementCapabilities in Place,but More Oversight ofOperational SystemsIs NeededGAO-04-822a
August 2004INFORMATION TECHNOLOGYHighlights of GAO-04-822, a report tocongressional requestersThe Federal AviationAdministration’s (FAA) mission isto promote the safe, orderly, andexpeditious flow of air traffic in theUnited States airspace system,commonly referred to as theNational Airspace System (NAS).To maintain its ability to effectivelycarry out this mission FAAembarked, in 1981,on a multibillion dollar effort to modernize itsaging air traffic control (ATC)system, the principle technologycomponent of the NAS. Yet theNAS modernization has continuedto be plagued by cost increases,schedule delays, and performanceshortfalls. To gain insight into howFAA is meeting its managementchallenges, congressionalrequesters asked GAO to evaluateFAA’s processes for making ITinvestment management decisions.The objectives of this reviewincluded (1) evaluating FAA’scapabilities for managing its ITinvestments and (2) determiningwhat plans, if any, the agency mighthave for improving thesecapabilities.FAA Has Many Investment ManagementCapabilities in Place, but More Oversightof Operational Systems Is NeededJudged against the criteria of GAO’s framework for information technologyinvestment management (ITIM), which measures the maturity of anorganization’s investment management processes, FAA has establishedabout 80 percent of the basic selection and control practices that it needs tomanage its mission-critical investments (see table below). For example,business lines actively monitor projects throughout their life cycles.However, the agency’s senior IT investment board does not regularly reviewinvestments that are in the “in-service management,” or operational, phase oftheir life cycles, and this creates a weakness in FAA’s ability to oversee morethan 1 billion of its IT investments. In addition, the agency has not yetestablished the key practices that would allow it to manage all of itsinvestments as one portfolio—an integrated set of competing options. UntilFAA has established the practices that would enable it to effectively manageits annual IT budget of about 2.5 billion, agency executives lack assurancethat they are selecting and managing the mix of investments that best meetsthe agency’s needs and priorities.The agency has initiated efforts to improve its investment managementprocesses, but it has not yet developed and implemented a comprehensiveplan—supported by management—to guide all of its improvement efforts.Such a plan is crucial in helping FAA to coordinate and prioritize itsimprovement efforts and sustain its commitment to the efforts it already hasunder way. Without such a plan—and controls for implementing it—FAAwill be unlikely to develop a mature investment management capability.Summary of Results for Foundational Critical Processes and Key PracticesCritical processTo strengthen FAA’s investmentmanagement capability, GAOrecommends that FAA develop andimplement a plan to address theweaknesses identified in thisreport. In commenting on a draft ofthis report, the Department ofTransportation commented that thereport was balanced and fair,showing where FAA has manycapabilities in place and identifyingareas that need Instituting aninvestment boardMeeting businessneedsSelecting nvestmentinformationOverallSource: GAO.To view the full product, including the scopeand methodology, click on the link above.For more information, contact David Powner,202-512-9286, pownerd@gao.gov or LesterDiamond, 202-512-7957, diamondl@gao.gov.PurposeTo define and establish an appropriate IT investmentmanagement structure and the processes for selecting,controlling, and evaluating IT investments.To ensure that IT projects and systems support theorganization’s business needs and meets users’ needs.To ensure that a well-defined and disciplined process isused to select new IT proposals and reselect ongoinginvestments.To review the progress of IT projects and systems, usingpre-defined criteria and checkpoints, in meeting cost,schedule, risk, and benefit expectations and to takecorrective action when these expectations are not beingmet.To make available to decision makers information toevaluate the impacts and opportunities created byproposed (or continuing) IT investments.Key practicesexecuted forNAS systems88%86%70%57%100%79%
ContentsLetter123Results in BriefBackgroundFAA Has Established an IT Management Structure to Manage ItsNAS InvestmentsFAA Has Initiated Efforts to Improve Its Investment ManagementProcessDOT is Taking Steps to Integrate Oversight of FAA’s ITInvestmentsConclusionsRecommendations for Executive ActionAgency Comments41424344Objectives, Scope, and Methodology45Investment Management Process Usedby Some Organizational Units to ManageNon-NAS Investments481940AppendixesAppendix I:Appendix II:TablesFiguresTable 1: NAS and Non-NAS IT InvestmentsTable 2: Stage 2 Critical Processes—Building the InvestmentFoundationTable 3: Summary of Results for Stage 2 Critical Processes and KeyPractices for NAS InvestmentsTable 4: Instituting the Investment BoardTable 5: Meeting Business NeedsTable 6: Selecting an InvestmentTable 7: Providing Investment OversightTable 8: Capturing Investment InformationTable 9: Stage 3 Critical Processes—Developing a CompleteInvestment PortfolioTable 10: Status of Stage 3 Critical ProcessesFigure 1: FAA’s Life Cycle Management ProcessFigure 2: Detailed Breakdown of FAA’s Life Cycle ManagementProcessFigure 3: The Five ITIM Stages of Maturity with CriticalProcessesPage i518192125283134373881216GAO-04-822 FAA’s IT Investment Management Process
OMBOPSPIRVCSUVSCSFinancial Services unitHuman Resource Management unitInformation Services unitAcquisition Management Systemacquisition program baselineResearch and Acquisition unitRegion and Center Operations unitair traffic controlAir Traffic OrganizationAir Traffic Services unitRegulation and Certification unitChief Information OfficerDepartment of TransportationEn Route Communications Gatewayfacilities and equipment (predeployment stage of system lifecycle)Federal Aviation AdministrationFAA Telecommunications InfrastructureInvestment Analysis Teaminformation technologyInformation Technology Executive BoardInformation Technology Investment Management frameworkInformation Technology Investment Portfolio SystemJoint Resources CouncilNational Airspace SystemNAS Support Integration ProcessOffice of Management and Budgetoperations (postdeployment stage of system life cycle)postimplementation reviewVSCS Control Subsystem UpgradeVoice Switching and Control SystemThis is a work of the U.S. government and is not subject to copyright protection in theUnited States. It may be reproduced and distributed in its entirety without furtherpermission from GAO. However, because this work may contain copyrighted images orother material, permission from the copyright holder may be necessary if you wish toreproduce this material separately.Page iiGAO-04-822 FAA’s IT Investment Management Process
AUnited States Government Accountability OfficeWashington, D.C. 20548August 20, 2004LeterThe Honorable Tom DavisChairmanCommittee on Government ReformU.S. House of RepresentativesThe Honorable Adam H. PutnamChairmanSubcommittee on Technology, Information Policy,Intergovernmental Relations and the CensusCommittee on Government ReformU.S. House of RepresentativesThe Federal Aviation Administration’s (FAA) mission is to promote thesafe, orderly, and expeditious flow of air traffic in the United Statesairspace system, commonly referred to as the National Airspace System(NAS). To maintain its ability to effectively carry out this mission FAAembarked, in 1981, on a multibillion dollar effort to modernize its aging airtraffic control (ATC) system, the principle technology component of theNAS. Over the past 2 decades, individual FAA modernization projects haveexperienced cost overruns, schedule delays, and performance shortfalls oflarge proportions. Because of the size, complexity, cost, and problemplagued past of FAA’s modernization program, we have designated it a highrisk information technology investment since 1995.1This report is one in a series of reports responding to your request toevaluate FAA’s efforts to address the information technology (IT)management challenges it faces as it continues to modernize the ATCsystem. It focuses on FAA’s processes for making IT investmentmanagement decisions and uses our Information Technology InvestmentManagement (ITIM) framework,2 which was released at a hearing of thesubcommittee on March 3, 2004. The framework provides a method for1See GAO, High-Risk Series: An Overview, GAO/HR-95-1 (Washington, D.C.: February1995); GAO, High-Risk Series: Information Management and Technology, GAO/HR-97-9(Washington, D.C.: February 1997); GAO, High-Risk Series: An Update, GAO/HR-99-1(Washington, D.C.: January 1999); GAO, High-Risk Series: An Update, GAO-01-263(Washington, D.C.: January 2001); and GAO, High-Risk Series: An Update, GAO-03-119(Washington, D.C.: January 2003).2GAO, Information Technology Investment Management: A Framework for Assessing andImproving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004).Page 1GAO-04-822 FAA’s IT Investment Management Process
evaluating and assessing how well an agency is selecting and managing itsIT resources. As agreed, our objectives were to (1) evaluate FAA’scapabilities for managing its IT investments, (2) determine what plans theagency might have for improving these capabilities, and (3) describe theDepartment of Transportation’s (DOT) oversight of FAA’s investments andinvestment management process. To address these objectives we analyzeddocuments and interviewed agency officials to (1) validate and updateFAA’s self-assessments of the key practices in the framework, (2) evaluateFAA’s plans for improving its capabilities, and (3) describe the department’soversight role. We performed our work from October 2003, through July2004, in accordance with generally accepted government auditingstandards. Appendix I contains further details on our objectives, scope, andmethodology.Results in BriefFAA has established most—about 80 percent—of the basic practicesneeded to manage its mission critical investments, including many of thefoundational practices for selecting and controlling IT investments. Thesekey practices provide additional assurance that the investments selectedwill meet organizational needs and will be completed on time and withinbudget. The practices also will enable the agency to manage its ITinvestments as a portfolio, or integrated set of competing options.Even with these many capabilities in place, weaknesses remain in severalareas. Specifically, FAA does not involve its senior IT investment board in regular reviews ofinvestments that have entered the in-service management phase, that is,those systems that have completed development and becomeoperational; does not have standard practices for managing its mission-support andadministrative investments; has not developed a process where the senior IT investment boardregularly reviews the full portfolio of investments; and has not implemented postimplementation reviews of its majorinvestments to validate that they are providing the expected benefitsafter they become operational.Page 2GAO-04-822 FAA’s IT Investment Management Process
FAA has begun to act to resolve the weaknesses described above, but untilFAA establishes the practices it needs to effectively manage its ITinvestments, executives cannot be assured that they are selecting andmanaging the mix of investments that best meets the agency’s needs andpriorities. Establishing the capabilities needed to effectively manageinvestments requires the development and implementation of a plan,supported by management, that defines and prioritizes improvements tothe investment process. While FAA has initiated a series of efforts toimprove its investment management processes, it does not have such aplan. Without this plan—and controls for implementing it—it is unlikelythat the agency will effectively establish mature investment managementcapabilities.The Department of Transportation has recently initiated several efforts thatcan serve to provide better departmental oversight of FAA investments. Forexample, DOT is finalizing capital investment guidance for all of itsoperating administrations to follow in implementing their investmentmanagement processes, and it has initiated a process for reviewing thefiscal year 2006 budget justifications for major programs, including those ofFAA. The department has also identified about a dozen programs it plans tomonitor on a regular basis and has asked FAA to report cost, schedule, andperformance data on some of its programs quarterly.To further strengthen FAA’s investment management capability, we arerecommending that the agency develop and implement a plan aimed ataddressing the weaknesses identified in this report.In commenting on a draft of this report, the Department of Transportation’sDirector of Audit Relations stated that GAO did a good job of keeping thereport balanced and fair, showing where FAA has many capabilities inplace and identifying areas that need improvement. The agency alsoprovided a technical comment, which we have incorporated into the report.BackgroundFAA’s Mission andOrganizational StructureAs an agency of the Department of Transportation, FAA’s mission is topromote the safe, orderly, and expeditious flow of air traffic in the nationalairspace. To fulfill its mission requires the extensive use of technology. Theachievement of the agency’s mission is also dependent in large part on thePage 3GAO-04-822 FAA’s IT Investment Management Process
skills and expertise of its workforce. Its workforce of nearly 50,000 peopleprovides aviation services that include air traffic control; maintenance ofair traffic control equipment; and certification of aircraft, airlineoperations, and pilots.FAA is organized into several staff support offices (examples include theOffice of Information Services and the Office of Human ResourceManagement) and five lines of business, which include Airports, Regulationand Certification, Commercial Space Transportation, the Office of Securityand Hazardous Materials, and the newly formed Air Traffic Organization(ATO). The ATO was formed on February 8, 2004, to better provide safe,secure, and cost-effective air traffic services now and into the future. TheAir Traffic Services and the Research and Acquisitions units, which hadbeen primarily responsible for managing air traffic services within FAA,were combined into one performance-based organization to create ATO.ATO is led by FAA’s Chief Operating Officer and consists of 10 serviceunits.3FAA’s Use of ITFAA relies extensively on information technology to carry out its NASoperations. It constantly depends on the adequacy and reliability of thenation’s ATC system, which comprises a vast network of radars; automateddata processing, navigation, and communications equipment; and ATCfacilities.4 Through this system, FAA provides services such as controllingtakeoffs and landings and managing the flow of traffic between airports.For example, the Integrated Terminal Weather System is employed to allowmaximum use of airport runways in all kinds of weather through a varietyof weather sensors. The Wide Area Augmentation System is used to provide3The 10 service units that make up the ATO include Safety, Communications, OperationsPlanning, Finance, Acquisition & Business Services, En Route and Oceanic Services,Terminal Services, Flight Services, System Operations Services, and Technical OperationsServices.4FAA uses three types of facilities to control traffic: airport towers, terminal radar approachcontrol facilities, and en route centers. Airport towers direct traffic on the ground, beforelanding, and after takeoff within 5 nautical miles from the airport and about 3,000 feet abovethe airport. Terminal radar approach control facilities sequence and separate aircraft as theyapproach and leave airports, beginning about 5 nautical miles and ending about 50 nauticalmiles from the airport and generally up to 10,000 feet above the ground. Air route trafficcontrol centers, called en route centers, control planes in transit and during approaches tosome airports, generally controlling air space that extends above 18,000 feet for commercialaircraft.Page 4GAO-04-822 FAA’s IT Investment Management Process
vertically guided landing to aircraft at thousands of airports and airstripswhere there is currently no vertically guided landing capability.FAA also relies on IT to carry out its mission-support and administrativeoperations (non-NAS operations). For example, FAA uses IT to supportaccident and incident investigations, security inspections, and personneland payroll functions.With an IT budget of about 2.5 billion for fiscal year 2004, FAA accountsfor over 90 percent of the Department of Transportation’s IT budget. Theamount of investments in both NAS and non-NAS IT is shown in the table 1below.Table 1: NAS and Non-NAS IT InvestmentsType ofinvestmentNASFunding of IT investmentsFacilities and Equipment (F&E)(development through 2 years of operations)Operations (OPS) (through the rest of the lifecycle)Non-NASOperationsTotal IT investment infiscal year 2004 inbillions of dollars1.4640.8340.350-0.500aSource: FAA.aAccording to FAA, these numbers will be verified via baselining by the end of Fiscal Year 2004 toreflect non-NAS IT assets and their costs.Prior Reviews IdentifiedWeaknesses in the Agency’sIT Investment ManagementProcessIn 1995, we designated FAA’s modernization of its air traffic control system,the principle technology component of the NAS, as a high-risk area becauseof the size and complexity of the program and FAA’s many failures inmeeting projects’ cost, schedule, and performance goals. In our latest HighRisk Series, issued in January 2003,5 we addressed the critical need for FAAto continue to improve its investment management practices—themanagement processes the agency uses to select, control and evaluate thebenefits realized from its IT spending—because the agency would bespending nearly 16 billion more through FY 2007, after having alreadyspent 35 billion since 1981. Other reports have also noted weaknesses in5GAO-03-119.Page 5GAO-04-822 FAA’s IT Investment Management Process
FAA’s IT investment management processes and have made a number ofrecommendations to address this area.6 For instance, last year we reportedthat while FAA had improved its processes, several issues remainedunresolved. We noted, for example, that the agency had not yetimplemented processes for evaluating projects after implementing them, inorder to identify lessons learned and improve the investment managementprocess.FAA’s Current Approach toInvestment ManagementFAA’s process for managing an IT investment varies depending on the typeof investment—NAS systems in development through the second year ofoperation (F&E), NAS systems in operation after the second year (OPS),and non-NAS systems each follow different processes. NAS investmentsare managed through a standardized process, the FAA AcquisitionManagement System (AMS), and non-NAS investments are managedthrough a number of different processes.Process for Managing NASInvestmentsIn April 1996, FAA implemented its AMS in response to legislation thatdirected the agency to develop a new acquisition management system.7Because of FAA’s contention that some of its modernization problems werecaused by federal acquisition regulations, the Congress enacted legislationin November 1995 that exempted the agency from most federalprocurement laws and regulations and directed FAA to develop andimplement a new acquisition management system that would address theunique needs of the agency. AMS was intended to reduce the time and costfor fielding new products and services by introducing (1) a new investmentmanagement system that spans the entire life cycle of an acquisition, (2) anew procurement system that provides flexibility in selecting andmanaging contractors, and (3) organizational and human capital reformsthat support the new investment and procurement systems.AMS provides high-level acquisition policy and guidance for selecting andcontrolling FAA’s NAS investments through all phases of the acquisition lifecycle, which is organized into a series of phases and decision points that6GAO, Air Traffic Control: FAA’s Modernization Investment Management Approach CouldBe Strengthened, GAO/RCED/AIMD-99-88 (Washington, D.C.: Apr. 30, 1999); GAO, AirTraffic Control: FAA’s Modernization Efforts—Past, Present, and Future, GAO-04-227T(Washington, D.C.: Oct. 30, 2003).749 U.S.C. 40110(d).Page 6GAO-04-822 FAA’s IT Investment Management Process
include (1) mission analysis, (2) investment analysis, (3) solutionimplementation, and (4) in-service management. To select investments,FAA has established two processes—mission analysis and investmentanalysis—which together constitute a set of policies, procedures, andguidance that enhance the agency’s ability to screen projects that aresubmitted for funding. Also, through these two processes FAA is to assessand rank each project based on its relative costs, benefits, risks, andcontribution to FAA’s mission, and a senior, corporate-level decisionmaking group selects projects for funding. After a project has beenselected, FAA officials are required to formally establish the life cycle cost,schedule, benefits, and performance baselines that are used to monitor theproject’s status throughout the remaining phases of the acquisitionmanagement life cycle. See figure 1 for a graphic depiction of FAA’s lifecycle management process.Page 7GAO-04-822 FAA’s IT Investment Management Process
InvestmentdecisionsioSolMisReMission needdecisionisent AnalysisIn v e st earcFigure 1: FAA’s Life Cycle Management ProcessFAALife CycleManagement ProcessNewserviceneedsntioIn-serv ice Manage m e tnIn-servicedecisionSelect phases - to choose and confirm investment choicesControl phases - to oversee ongoing investmentsSource: GAO based on FAA documents.Note: During the front end of the life cycle, research and system analysis activities are undertaken todiscover applications of new technology for FAA’s present services, explore new opportunities forservice delivery, solve problems within current operations, and define requirements.Several groups are involved in managing FAA’s NAS investments; theyperform functions from analysis of mission needs and alternativeinvestments through system development, implementation, operation, and,ultimately, disposal. The roles and responsibilities of each group aredescribed below:Joint Resources Council (JRC)—This board makes corporate-levelresource and investment decisions and establishes investment programs.Members include Associate Administrators representing FAA’s lines ofPage 8GAO-04-822 FAA’s IT Investment Management Process
business, the FAA Acquisition Executive,8 the Chief Financial Officer, theChief Information Officer (CIO), and the Assistant Administrators forSystem Safety, for Policy, Planning and International Aviation, and forRegion and Center Operations. The board is supported by the JRCSecretariat Team, a group that facilitates the board’s processes bymaintaining the meeting calendar and guidance documents, developingrecords of decisions, and providing advisory and liaison support toprograms.Systems Engineering/Operational Analysis Team—This team performsaffordability assessments for newly proposed investments and preparesrecommendations for the reprogramming of funds from lower priorityprograms. It also prepares annual budget submissions for approval by theJRC. This team is composed of representatives from each line of businessand from other functional disciplines and is chaired by the Director, SystemArchitecture and Investment Analysis.Investment Analysis Team (IAT)—This team is assembled for a relativelyshort period for each specific investment being considered, to conduct thedetailed analysis of alternatives that will lead to selecting andrecommending a preferred acquisition solution. The team draws expertsfrom the integrated product teams,9 the organizational unit with the need,the investment analysis staff,10 and other organizations.Corporate Mission Analysis Organization—Performs agency-level missionanalysis and coordinates service area analysis, an activity that is conductedduring mission analysis to (1) identify capability shortfalls for or in8May be delegated to an associate administrator.9These teams may operate as entities or be organized into subintegrated product teams orproduct teams to develop, procure, and deliver products and services for users orcustomers. They are responsible for the acquisition of new or improved capability forservices and products throughout their life cycles and for developing cost and schedulebaselines for candidate solutions during investment analysis.10The investment analysis staff assists and oversees the work of all the investment analysisteams, is responsible for all investment analyses, and is responsible for developing the tools,techniques, and databases to ensure quality performance of investment analysis on behalf ofthe JRC.Page 9GAO-04-822 FAA’s IT Investment Management Process
conjunction with service organizations,11 (2) ensure alignment with agencystrategic goals, and (3) eliminate redundant activity, duplicate benefits,service gaps, and service overlaps. It also develops and maintainsstandards and tools for conducting service area analysis, and it assistsservice organizations in establishing a service area analysis capability.In addition to identifying the roles and responsibilities of the groupsinvolved in the management process, AMS provides guidance on thedocuments and decisions that result from each of the life cycle phases. Forexample, through the mission analysis phase, FAA identifies critical needsthat the agency must meet for improving the safety, capacity, efficiency,and effectiveness of the NAS. Approval of a mission need statement by theJRC signifies that the agency agrees that the need is critical enough toproceed to the next phase—investment analysis. During the investmentanalysis phase, the IAT is to analyze and recommend a solution that bestsatisfies FAA’s performance goals and customer service needs. This team isthen to rank each proposed project based on a number of factors, includinghow well it meets mission needs compared to other projects and whether ithas a favorable cost-benefit ratio. As part of the JRC selection process, thelife cycle cost, schedule, benefits, and performance baselines areestablished in a formal document called the acquisition program baseline(APB), which is designed to be used by program offices to monitor aproject’s status in achieving those baselines throughout the remainingphases of the acquisition management life cycle.The solution implementation phase begins when the JRC approves andfunds a project, establishes its acquisition program baseline, and authorizesthe service organizations to implement and manage the project over its lifecycle. After the project has been implemented and is in operation (FAA’s inservice management phase), the service organizations monitor and assessoperational performance. Also during this phase, the project is monitoredto determine whether the current capability satisfies the demand forservices or whether another solution offers the potential for improving11A service organization is any organization within FAA that delivers a service, whether it isa business unit, project office, program directorate, or integrated product team, or whetherit is engaged in air traffic services, security, regulation, certification, operations, commercialspace transportation, or airport development. These organizations are responsible andaccountable for managing service delivery throughout the life cycle. Investment decisionsare made to support service delivery. Specifically, after the investment decision has beenmade, the service organization assumes responsibility for the investment program,implements the selected solution, and manages the product throughout the in-servicemanagement phase of its life cycle.Page 10GAO-04-822 FAA’s IT Investment Management Process
safety or effectiveness or for significantly lowering costs. If the currentcapability is lacking, FAA initiates a process whereby the mission needwould be revalidated and the investment analysis process begun again,possibly leading to a new investment decision. Figure 2 provides detail onthe phases of FAA’s IT investment management process and decisionpoints. The highlighted decision points represent those for which the JRCmust make an approval decision before a project can move forward.Senior executives have stated that with the reorganization of the ATO inFebruary 2004, discussions have been held about realigning the investmentmanagement process to make the heads of the service units responsibleand accountable for managing programs’ capital investments and operatingcosts from inception to retirement. In the past, the business units havebeen organized to manage either capital investments or operating costs,but not both. These discussions have not yet led to specific changes inFAA’s investment management processes and responsibilities.Page 11GAO-04-822 FAA’s IT Investment Management Process
Figure 2: Detailed Breakdown of FAA’s Life Cycle Management Proces
Diamond, 202-512-7957, diamondl@gao.gov. Highlights of GAO-04-822, a report to congressional requesters August 2004 INFORMATION TECHNOLOGY FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed The Federal Aviation Administration's (FAA) mission is to promote the safe, orderly, and
