WIXLIB

DFAS End-to-End Testing. DFAS identified its critical business processes,and developed plans to test those business processes. Critical processes aredefined as those that, if not performed, would preclude or immediately impairthe disbursal, pay, and accounting functions. Specifically, DFAS identified thefollowing seven critical business processes: civilian pay, military pay, contractand vendor pay, transportation pay, travel pay, accounting, and disbursing.DFAS has organized its end-to-end testing into seven testing events, one foreach critical mission or business process. DFAS has further divided each eventor business process into "threads." A thread is a system or set of systems thatperforms the functions within the business process. Threads may span a seriesof systems owned by DFAS or other DoD organizations, as well as externalinterfaces, such as the Federal Reserve Bank, the U.S. Postal Service, and otherorganizations. Each event can include one or more threads that track to acritical business process. DFAS organized its events to include all 45 1 of itsY2K mission-critical systems, as well as certain non mission-critical systemsthat were essential to the critical business processes.DFAS Y2K End-to-End Master Plan. DFAS issued the "DFAS Y2KEnd-to-End Master Plan," revision 2.3, on June 21, 1999, specifically foraccomplishing Y2K-related end-to-end testing of its critical business processes.The Master Plan identifies roles and responsibilities; assumptions andconstraints related to testing; requirements for interfaces with non-DFASorganizations; and requirements for planning, testing, and reporting on testresults.Roles and Responsibilities. DFAS designated a Y2K project managerand functional proponent at DFAS Headquarters with overall Y2K testingresponsibility. DFAS delegated assignment of event leaders and thread leadersto its functional proponents to execute the end-to-end testing. Also, DFASassigned roles and responsibilities to system managers for controlling theirsegment of the end-to-end testing process.Assumptions and Constraints. Because of limited time and resources,the DFAS Master Plan acknowledges constraints and identifies assumptionsrelated to Y2K end-to-end testing. These include the assumptions thatthird-party software and computing platforms are Y2K compliant, thatoperations and compliance testing takes precedence over end-to-end testing, thatpartner organizations will conduct their own internal end-to-end tests andprovide input for DFAS, and that all mission critical systems will havecontingency plans in place.1We identified 42 DFAS mission critical systems in previous audit reports. DFAS recentlyadded three systems to its mission critical list: the Standard Accounting and Reporting System One Bill Pay (STARS-OP), the Standard Army Financial Inventory Accounting and ReportingSystem (STARFIARS), and the Standard Base Supply System (SBSS).2

Interface Requirements. The DFAS Master Plan states that each testevent will include critical automated interfaces with other Departments andagencies. However, because of sizing limitations within the DoD Megacentersthat support testing, DFAS and the other DoD organizations may not be able torun true end-to-end tests simultaneously. Rather, each organization willmaintain sufficient control of their segment of the end-to-end testing process toensure the integrity of the data flow from one system to the other.Planning, Testing, and Reporting. The DFAS Master Plan specifiesrequirements for the following: Live Versus Simulation. DFAS plans to test its businessprocesses under normal operating conditions when possible.Otherwise, DFAS will use a "time machine" or simulatedoperating environment and document the reasons and theassociated risks. 2 Critical dates. Although the Master Plan does not designatespecific dates for testing, it recommends that testing cover thefollowing five time periods: the fiscal year 2000 crossover,calendar year 2000 crossover, fiscal year 2001 crossover,calendar year 2001 crossover, and leap year(February 29, 2000). DFAS recommends that whatever dateseach event tests should be consistent with dates being testedby interfacing systems. Baselines. The Master Plan states that after testing the dates,DFAS organizations should compare their test results tooutcomes previously determined as the baseline. (Thebaseline is the set of known end-to-end test inputs and outputsextracted from systems that have been certified as Y2Kcompliant). Each DFAS organization will document thediscrepancies between each of the tests and the baseline. Data Analysis and Documentation. The Master Planrequires that each DFAS organization develop, and documentin their test plan, a data collection and analysis strategy thatprovides sufficient information to support end-to-end testdesign, results and analysis. The Master Plan leaves thedetails for data analysis or documentation to the organizationsresponsible for testing.2A time machine test strategy involves setting system clocks to the year 2000 and operating under testingconditions. Simulation is a program that allows testers to simulate changing dates rather than actuallychanging the dates during normal operations.3

Preliminary Assessment. On May 28, 1999, the Office of the InspectorGeneral, DoD, provided DFAS with preliminary audit results in a memorandumregarding DFAS planning for end-to-end testing in general. On June 8, 1999,the Director, Information and Technology, DFAS, responded to the issues ofroles and responsibilities, Master Plan checklists, interfacing systems, criticalcrossover dates, data collection and analysis, and the transportation pay event.Our initial assessment and DFAS comments, as appropriate to the civilian paybusiness area, are discussed in the finding section of this report. Appendix Bcontains a copy of our memorandum and Appendix C is the DFAS response.ObjectiveThe overall audit objective was to evaluate the effectiveness of the plannedend-to-end testing in the DoD civilian pay area. This report addresses plans forconducting Y2K end-to-end testing for DoD civilian pay. Other reports willaddress other DFAS critical business processes. See Appendix A for adiscussion of the audit scope and methodology, the management controlprogram, and prior audit coverage related to the audit objective.4

Civilian Pay Year 2000 End-to-EndTesting Event PlansThe civilian pay Y2K end-to-end event planners took the initiative todevelop an end-to-end event plan before requirements were defined forDFAS as a whole, and developed a sound testing methodology.However, DFAS Headquarters did not require revisions to the civilianpay event plan as overarching guidance was issued. The event planlacked requirements for data collection and data analysis. Despite theactions taken by DFAS during the audit, some concerns remain about thelevel of assurance that can be derived from the test results.Civilian Pay Functional AreaThe mission of the DFAS civilian pay functional area is to process payroll forDoD civilian employees, culminating in a civilian payday. The DefenseCivilian Pay System (DCPS) is the payroll system for civilian employees inDoD. DCPS maintains pay and leave entitlement records, deductions andwithholdings, time and attendance data, and other personnel data, for use inprocessing bi-weekly payroll. DCPS provides standard interface support tovarious DoD accounting, financial, disbursing, and personnel systems.DCPS makes payments via checks and electronic funds transfers, and providesentitlement information to the disbursing systems, which then make thepayments via checks and electronic funds transfers. DCPS processes theentitlements for 708,000 DoD civilian employees totaling approximately 37 .6 billion annually.During normal processing, DCPS receives inputs from 46 systems and sendsoutputs to 67 systems belonging to the military Services, Defense agencies, theOffice of Personnel and Management, the Social Security Administration, stateand local taxing authorities, and other agencies. Civilian pay event plannersdetermined that there were seven critical business processes, or threads,involved in civilian pay. Critical processes are defined as those that, if notperformed, would preclude or immediately impair the civilian payments andentitlements. The seven threads involved in the civilian pay event are: payment processing; garnishment; taxes and Social Security withholdings; bonds and sending electronic funds transfers to the Federal ReserveBank;5

accounting; processing with the Automated Time, Attendance and ProductionSystem (ATAAPS); and processing with the Defense Civilian Personnel Data System Modification (DCPDS-MOD).During end-to-end testing, DCPS planned to receive inputs from 9 systems andsend outputs to 21 systems. DCPS also planned to conduct additional interfacetests with 19 systems, receiving inputs from 14 systems and sending outputs to16 systems (some systems send and receive information to/from DCPS). Thecivilian pay event planners selected the systems they considered most critical totimely and accurate payments for input/output testing. For those systems notincluded in the end-to-end or interface tests, the civilian pay event plannersrelied on the results of interface testing already performed by DFAS during theDCPS Y2K certification process.Civilian Pay End-To-End Event PlanThe DCPS System Manager and personnel at the Systems EngineeringOrganization, Pensacola, FL, began developing the "Civilian Payroll Y2K EndTo End Test Plan" in August 1998, prior to issuance of the DFAS Y2KEnd-to-End Master Plan. According to event planners, the plan was based ontesting experience and common sense. The event planners developed the testplan based on the June 1998 draft of the General Accounting Office's"Year 2000 Computing Crisis: A Testing Guide," which was finalized inNovember 1998.As part of the planning, civilian pay event planners determined that there wereseven critical business processes (threads) involved in civilian pay. The civilianpay event planners identified the major trading partners that are critical to timelyand accurate payments, and planned to conduct end-to-end testing with thosepartners. The event planners also planned interface testing with other majorinterface partners, and accepted the risk of relying on validation testing resultsfor the remaining partners. The event planners developed a sound testingmethodology for the end-to-end testing of the civilian pay event. As part of thetesting methodology, the civilian pay event planners included re-testing ofsystems and threads as necessary - for example, if there was a failure during thetest, or if there were configuration management changes to systems involved inthe event.6

DFAS End-to-End Project Office ActionsAs previously stated, event planners began developing the civilian pay eventplan in August 1998, prior to the DFAS end-to-end project office issuing thefirst version of the Master Plan in May 1999. The civilian pay event leader metwith the DFAS end-to-end project office concerning the need to modify theplans based upon the guidance issued subsequent to completion of the civilianpay event plan. DFAS project office officials stated that the civilian pay eventplan did not need to be revised, but that the plan should adhere to the newrequirements.Effectiveness of Civilian Pay Event PlanWhile the civilian pay event plan was comprehensive and employed anappropriate testing methodology, the plan had shortfalls regarding datacollection and data analysis.Data Collection and Data Analysis Plans. The DoD Y2K Management Plan,Appendix I, requires that Y2K event plans and procedures specify in detail whattype of data must be collected, who will analyze it, how it will be analyzed, andthe mechanism for sending the Y2K data to the analysis site. In addition, theDFAS Master Plan exit criteria for the test planning phase requires that eachevent identify pass/fail criteria for all of its tests and that data collectionprocedures be in place and mechanisms needed to capture data be installed priorto test execution. The Master Plan, however, does not provide guidance as towhat types of data should be collected to ensure consistency in reporting testresults. The Master Plan states that each DFAS organization will document thedata collection process in the appropriate event plan.Civilian Pay Data Collection Plan. Although the DFAS Master Planstates that plans for data collection should be written at the event level, thecivilian pay event plan did not contain a data collection plan. During theplanning phase of end-to-end testing, the civilian pay event planners and theirinterface partners defined the Y2K dates that would be tested, and ensured thattime and attendance systems were matched with accounting systems; however,we could not document any other data collection planning. According tocivilian pay event planners, they were not familiar with the requirements for a"data collection plan" and they felt that it was not within their authority torequire the managers of systems participating in the event to follow theirinstructions. Instead, the event planners relied on the participating systems toprovide the appropriate information. Consequently, results obtained from theparticipating systems during testing may not be consistent or appropriate fordata analysis purposes. Without this organized approach, the civilian pay eventplanners will not have a standard group of test data from which to conduct theiranalysis to determine if civilian pay is able to function in a Y2K environment.7

A plan for data collection is critical to define how the system providesinput/output, interactions, and data to be recorded at all principal phases of theassessment. As such, the civilian pay event plan should be amended to definespecific data collection procedures, including defining: necessary tools and instrumentation; required data elements for when, where, how, by whom, and inwhat format data should be collected; and how the data will be transferred, archived and controlled forentry into the centralized operational evaluation database.Subject matter experts should review the requirements for each dataelement desired, and then determine how the data can be captured or recorded ateach site. When multiple sites are used, requirements must be consistent for allof the sites to assure quality data collection and analysis.Testing for the civilian pay event began March 1, 1999. As ofJuly 20, 1999, event planners had tested the DCPS segment for six of the seventhreads. However, event planners had received confirmations from all partnersin the thread for only one of the seven threads; they had received partialconfirmation from partners for three of the remaining six threads. Eventplanners stated that they do not plan to provide the final test reports to theDFAS project office until October 1999.Civilian Pay Data Analysis Plan. The civilian pay event plan also didnot include a data analysis plan as required by the DoD Management Plan. Thedata analysis plan should have been written during the planning process toidentify an organized approach to understand data requirements; identifypotential issues, conflicts, or difficulties; and to evaluate test data to determinewhether the test was successful. The evaluation section should have indicated"who, what, where, when, how, and how much" to analyze the data collectedthrough the data collection plan.Civilian Pay event planners stated they planned to determine the successof each test by the correct passing of the data from the inputting system toDCPS, or from DCPS to the receiving systems. Event planners also stated thatthey received information from inputting partners (prior to the test) thatindicated what data DCPS should receive, and would use that information todetermine whether the data passed correctly. However, even with thisinformation to use as a starting point, there was no way to uniformly andobjectively analyze the test results or ensure that the data processed accurately.Given the nature of end-to-end testing, with its large numbers ofparticipating systems, it is prudent to ensure that the data collection is asconsistent as possible for each test, and that the analysis of the test data is8

objective. Without the definition of data collection and data analysis plans, thisis impossible. The civilian pay event planners can strengthen the end-to-endtesting by documenting uniform, detailed data collection and data analysisprocedures.Recent Management Actions TakenMemorandum to DFAS. We sent a memorandum on May 28, 1999, to theDFAS Director, Information and Technology, on issues concerning DFAS Y2Kend-to-end event planning (see Appendix B). The DFAS Director, Informationand Technology, responded in a memorandum dated June 8, 1999 (seeAppendix C). The areas of concern included issues identified in this report,including data collection and analysis. The DFAS Director, Information andTechnology, generally concurred with the issues identified, and discussedactions underway and planned to correct deficiencies in the planning process forDFAS end-to-end testing.Our memorandum suggested that DFAS event leaders ensure that detailed datacollection, analysis, and reporting requirements are clearly defined prior totesting. The DFAS project office concurred with the suggestion, and stated thatDFAS agrees that current plans lack specific exit criteria and that it is takingaction to strengthen this area of the plans. The project office further stated thatDFAS Regulation 8000.1-R, "Information Management Policy andInstructional Guidance," October 15, 1998, provides guidance concerning datacollection and analysis. In addition, the project office stated that DFAS hadcontracted with the Joint Interoperability Test Command for an independentverification and validation analysis of DFAS end-to-end test planning andtesting. DFAS believes that the Joint Interoperability Test Commandanalysis/evaluation will document specific risks associated with data collectionand analysis procedures in sufficient time for the project office to takecorrective action.In order to mitigate the risks related to data collection and analysis, it isessential to the testing process to determine the data collection and analysisprocedures before testing or re-testing begins, so that the collection andanalysis may be conducted in a consistent, uniform manner, and ensureaccurate results.Meeting with Civilian Pay

The Defense Civilian Pay System processes the entitlements for 708,000 DoD civilian employees totaling approximately 37 .6 billion annually. The civilian pay functional area Y2K end-to-end event.