Information Technology Disaster Recovery Plan - Guam

Transcription

INFORMATION TECHNOLOGY DISASTER RECOVERY PLANPOLICY# OTECH-POL2017-005GOVERNMENT OF GUAM, OFFICE OF TECHNOLOGY211 ASPINAL AVENUE HAGATNA, GUAM 96910Otech.guam.govAUGUST 4, 2017

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-005ContentsOverview . 2Revision History . 2Introduction . 3Definition of a Disaster . 3Purpose . 3Scope . 41.0 Disaster Recovery Teams & Responsibilities. 41.1Disaster Recovery Lead . 41.2Disaster Management Team . 51.3Network Team. 51.4Server Team . 61.5Applications Team . 71.6Operations Team. 82.0 Disaster Recovery Call Tree . 83.0 Recovery Facilities . 103.1 Description of Recovery Facilities . 103.2 Data and Backups . 114.0 Communicating During a Disaster . 114.1 Communicating with the Authorities . 114.2 Communicating with Employees . 124.3 Communicating with Clients . 124.4 Communicated with Vendors . 125.0 Dealing with a Disaster . 135.1 Disaster identification and Declaration . 135.2 DRP Activation. 145.3 Communicating the Disaster . 145.4 Assessment of Current and Prevention of Further Damage . 145.5 Standby Facility Activation . 145.6 Restoring IT Functionality . 155.7 Repair & Rebuilding of Primary Facility . 151 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-005OverviewPolicy Number:OTECH-POL2017-005Title:Information Technology Disaster Recovery PlanPurpose:To define protocols and procedures in the event OTECH experiences a disaster.Publication Date:August 4, 2017Policy Approval:Frank L.G. Lujan, Jr.OTECH Chief Technology Officer (CTO)Target Audience:Office of Technology employeesContact Details:Office of Technology211 Aspinall AvenueHagatna, Guam 96910O: 671.475.1113F: 671.472.9508otech.guam.govRevision HistoryDate of ChangeResponsibleSummary of ChangeJuly 2017OTECH Systems SupportDraft policyJuly 2017OTECH Data Processing Manager and CTOReview policyAugust 2017CTOApprove policy2 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-005IntroductionThis Disaster Recovery Plan (DRP) captures, in a single repository, all of the information that describes the Office ofTechnology’s (OTECH) ability to withstand a disaster as well as the processes that must be followed to achieve disasterrecovery within the Government of Guam’s (GovGuam) information technology infrastructure and systems.OTECH is committed to providing a highly reliable, secure and cost effective oversight, leadership and direction foractivities relating to information technology to all agencies across GovGuam.Definition of a DisasterA disaster can be caused by man or nature and results in any of the GovGuam line agencies not being able to perform allor some of their regular roles and responsibilities for a period of time. OTECH defines disasters as the following: One or more vital systems are non-functionalThe building is not available for an extended period of time but all systems are functional within itThe building is available but all systems are non-functionalThe building and all systems are non-functionalThe following events can result in a disaster, requiring this Disaster Recovery document to be activated: FireTsunamiPandemicPower OutageTheftTerrorist AttackTyphoonEarthquakeCyber AttackPurposeThe purpose of this DRP document is twofold: first to capture all of the information relevant to OTECH’s ability towithstand a disaster, and second to document the steps that OTECH will follow if a disaster occurs.Note that in the event of a disaster the first priority of OTECH is to prevent the loss of life. Before any secondarymeasures are undertaken, OTECH will ensure that all employees, and any other individuals on the organization’spremises, are safe and secure.After all individuals have been brought to safety, the next goal of OTECH will be to enact the steps outlined in this DRP tobring all of the organization’s groups and departments back to business-as-usual as quickly as possible.This includes: Preventing the loss of the organization’s resources such as hardware, data and physical IT assetsMinimizing downtime related to ITKeeping the business running in the event of a disaster3 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-005ScopeThe OTECH DRP takes all of the following areas into consideration: Network InfrastructureServers InfrastructureTelephony SystemData Storage and Backup SystemsData Output DevicesEnd-user ComputersOrganizational Software SystemsDatabase SystemsIT DocumentationThis DRP does not take into consideration any non-IT, personnel, Human Resources and real estate related disasters. Forany disasters that are not addressed in this document, please refer to the respective GovGuam agency businesscontinuity plan.1.0 Disaster Recovery Teams & ResponsibilitiesIn the event of a disaster, different groups will be required to assist the IT department in their effort to restore normalfunctionality to the employees of OTECH. The different groups and their responsibilities are as follows: Disaster Recovery Lead(s)Disaster Management TeamNetwork TeamServer TeamApplications TeamOperations TeamThe lists of roles and responsibilities in this section have been created by OTECH and reflect the likely tasks that teammembers will have to perform. Disaster Recovery Team members will be responsible for performing all of the tasksbelow. In some disaster situations, Disaster Recovery Team members will be called upon to perform tasks not describedin this section.1.1 Disaster Recovery LeadThe Disaster Recovery Lead is responsible for making all decisions related to the Disaster Recovery efforts. This person’sprimary role will be to guide the disaster recovery process and all other individuals involved in the disaster recoveryprocess will report to this person in the event that a disaster occurs at OTECH regardless of their department andexisting managers.1.1.1 Role and Responsibilities Make the determination that a disaster has occurred and trigger the DRP and related processes. Initiate the DR Call Tree. Be the single point of contact for and oversee all of the DR Teams. Organize and chair regular meetings of the DR Team leads throughout the disaster. Present to the Management Team on the state of the disaster and the decisions that need to be made. Organize, supervise and manage all DRP test and author all DRP updates.4 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-0051.1.2 Contact InformationNameFrank L.G. Lujan, Jr.Joseph C. ManibusanRole/TitlePrimary Disaster LeadChief Technology OfficerSecondary Disaster LeadData Processing -475-1113671-482-0036Email otech.guam.gov1.2 Disaster Management TeamThe Disaster Management Team that will oversee the entire disaster recovery process. They will be the first team thatwill need to take action in the event of a disaster. This team will evaluate the disaster and will determine what stepsneed to be taken to get the organization back to business as usual.1.2.1 Role & ResponsibilitiesSet the DRP into motion after the Disaster Recovery Lead has declared a disasterDetermine the magnitude and class of the disasterDetermine what systems and processes have been affected by the disasterCommunicate the disaster to the other disaster recovery teamsDetermine what first steps need to be taken by the disaster recovery teamsKeep the disaster recovery teams on track with pre-determined expectations and goalsKeep a record of money spent during the disaster recovery processEnsure that all decisions made abide by the DRP and policies set by OTECHGet the secondary site ready to restore business operationsEnsure that the secondary site is fully functional and secureNotify the relevant parties once the disaster is over and normal business functionality has been restored1.2.2Contact InformationNameRole/TitleFrank L.G. Lujan, Jr.Chief Technology OfficerJoseph C. ManibusanData Processing ManagerBeatrice A. SantosSystems & Email otech.guam.govbea.santos@otech.guam.gov1.3 Network TeamThe Network Team will be responsible for assessing damage specific to any network infrastructure and for provisioningdata and voice network connectivity including WAN, LAN, and any telephony connections internally within theenterprise as well as telephony and data connections with the outside world. They will be primarily responsible forproviding baseline network functionality and may assist other IT DR Teams as required.1.3.1 Role & ResponsibilitiesIn the event of a disaster that does not require migration to standby facilities, the team will determine whichnetwork services are not functioning at the primary facility5 of 15 P a g e

IT Disaster Recovery Plan 1.3.2POLICY#: OTECH-POL2017-005If multiple network services are impacted, the team will prioritize the recovery of services in the manner andorder that has the least business impact.If network services are provided by third parties, the team will communicate and co-ordinate with these thirdparties to ensure recovery of connectivity.In the event of a disaster that does require migration to standby facilities the team will ensure that all networkservices are brought online at the secondary facilityOnce critical systems have been provided with connectivity, employees will be provided with connectivity in thefollowing order:o All members of the DR Teamso All Executive Staffo All IT employeeso All remaining employeesInstall and implement any tools, hardware, software and systems required in the standby facilityInstall and implement any tools, hardware, software and systems required in the primary facilityContact InformationNameRole/TitleRoman M. PalomoSystems Analyst IIGerard B. CalvoSystems ProgrammerJennie T.S. QuintansSystems 671-635-4502671-482-5219671-638-3806Email ch.guam.govjennie.quintans@otech.guam.gov1.4 Server TeamThe Server Team will be responsible for providing the physical server infrastructure required for the enterprise to run itsIT operations and applications in the event of and during a disaster. They will be primarily responsible for providingbaseline server functionality and may assist other IT DR Teams as required.1.4.1 Role & ResponsibilitiesIn the event of a disaster that does not require migration to standby facilities, the team will determine whichservers are not functioning at the primary facilityIf multiple servers are impacted, the team will prioritize the recovery of servers in the manner and order that hasthe least business impact. Recovery will include the following tasks:o Assess the damage to any serverso Restart and refresh servers if necessaryEnsure that secondary servers located in standby facilities are kept up-to-date with system patchesEnsure that secondary servers located in standby facilities are kept up-to-date with application patchesEnsure that secondary servers located in standby facilities are kept up-to-date with data copiesEnsure that the secondary servers located in the standby facility are backed up appropriatelyInstall and implement any tools, hardware, and systems required in the standby facilityInstall and implement any tools, hardware, and systems required in the primary facility6 of 15 P a g e

IT Disaster Recovery Plan1.4.2POLICY#: OTECH-POL2017-005Contact InformationNameBeatrice A. SantosRole/TitleChristine A. BazaRoman M. PalomoSystems & ProgrammingAdministratorSystems ProgrammerSystems Analyst IIGerard B. CalvoSystems ProgrammerJennie T.S. QuintansSystems 1-638-3806Email @otech.guam.govjennie.quintans@otech.guam.gov1.5 Applications TeamThe Applications Team will be responsible for ensuring that all enterprise applications operates as required to meetbusiness objectives in the event of and during a disaster. They will be primarily responsible for ensuring and validatingappropriate application performance and may assist other IT DR Teams as required.1.5.1 1.5.2Role & ResponsibilitiesIn the event of a disaster that does not require migration to standby facilities, the team will determine whichapplications are not functioning at the primary facilityIf multiple applications are impacted, the team will prioritize the recovery of applications in the manner andorder that has the least business impact. Recovery will include the following tasks:o Assess the impact to application processeso Restart applications as requiredo Patch, recode or rewrite applications as requiredEnsure that secondary servers located in standby facilities are kept up-to-date with application patchesEnsure that secondary servers located in standby facilities are kept up-to-date with data copiesInstall and implement any tools, software and patches required in the standby facilityInstall and implement any tools, software and patches required in the primary facilityContact InformationNameBeatrice A. SantosRole/TitleJoseph J. NededogSystems & ProgrammingAdministratorSystems ProgrammerProgrammer AnalystSupervisorSystems Analyst IShirley L.G. MunozSystems Analyst IIMaryann D. MendiolaSystems Analyst IIChristine A. BazaChristine M. San mail a@otech.guam.gov7 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-0051.6 Operations TeamThis team’s primary goal will be to provide employees with the tools they need to perform their roles as quickly andefficiently as possible. They will need to provision all OTECH employees in the standby facility and those working fromhome with the tools that their specific role requires.1.6.1 1.6.2Role & ResponsibilitiesMaintain lists of all essential supplies that will be required in the event of a disasterEnsure that these supplies are provisioned appropriately in the event of a disasterEnsure sufficient spare computers and laptops are on hand so that work is not significantly disrupted in a disasterEnsure that spare computers and laptops have the required software and patchesEnsure sufficient computer and laptop related supplies such as cables, wireless cards, laptop locks, mice, printersand docking stations are on hand so that work is not significantly disrupted in a disasterEnsure that all employees that require access to a computer/laptop and other related supplies are provisioned inan appropriate timeframeIf insufficient computers/laptops or related supplies are not available the team will prioritize distribution in themanner and order that has the least business impactThis team will be required to maintain a log of where all of the supplies and equipment were usedContact InformationNameRole/TitleElaine J. CruzComputer Operations Supervisor(Lead)Systems Analyst ITeleprocessing Network CoordinatorTeleprocessing Network CoordinatorAdministrative AideComputer Operator IIComputer Image RecorderKenneth J. CruzEric H. RosellVictor S.N. SottoPhillip WintterleJonathan PerezJessica 5-7649671-635-1780Email 2.0 Disaster Recovery Call TreeIn a disaster recovery or business continuity emergency, time is of the essence so OTECH will make use of a Call Tree toensure that appropriate individuals are contacted in a timely manner.In the event a team member is unavailable, the initial caller assumes responsibility for subsequent 71-635-4500671-482-5183HomeDR LeadFrank L.G. Lujan, Jr.DR Management Team LeadFrank L.G. Lujan, Jr.8 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-005DR Management TeamJoseph C. Manibusan671-475-1113671-482-0036DR Management TeamBeatrice A. Santos671-635-4501Network Team LeadRoman M. 5219Network TeamGerard B. CalvoNetwork TeamJennie T.S. Quintans671-638-3806Server Team Lead 1Beatrice A. Santos671-635-4501Server Team Lead 2Gerard B. Calvo671-635-4502671-482-5219Server TeamChristine A. Baza671-638-3803Server TeamRoman M. Palomo671-635-4503671-483-1496Server TeamJennie T.S. Quintans671-638-3806Applications Team LeadChristine A Baza671-638-3803App TeamBeatrice A. Santos671-635-4501App TeamChristine M. San Agustin671-635-1810App TeamJoseph J. Nededog671-475-1253671-488-56159 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-005App TeamShirley L.G. 524671-638-3801671-480-3418App TeamMaryann D. MendiolaOperations Team LeadElaine J. CruzOperations TeamKenneth J. Cruz671-475-1203Operations TeamEric H. Rosell671-475-1203Operations TeamVictor S.N. Sotto671-638-3802Operations TeamPhillip Wintterle671-475-1203Operations TeamJonathan Perez671-635-7649Operations TeamJessica Naputi671-635-17803.0 Recovery FacilitiesIn order to ensure that OTECH is able to withstand a significant outage caused by a disaster, it has provisioned separatededicated standby facilities. This section of this document describes those facilities and includes operational informationshould those facilities have to be used.3.1 Description of Recovery FacilitiesThe DR Standby facility will be used after the Disaster Recovery Lead has declared that a disaster has occurred. Theaffected system will determine which Standby system is activated and promoted to production. OTECH operates three(3) production datacenters to support GovGuam IT operations. Should the Disaster Recovery Lead declare a disaster atone of the production datacenters, the identified standby datacenter facility will be restored, rebuilt or promoted toreinstate end-user IT operations and support. The availability of resources will determine how the standby facility will berestored. Refer to diagram below for standby facilities.System: Power 7 (Partial automatic failover in place)10 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-005DOA Power 7RevTax Power 7211 Aspinal Avenue Hagatna, Guam 969101240 Army Drive Barridaga, Guam 96913DOA LPAR - ProductionDOA LPAR - StandbyDPHSS LPAR - ProductionDPHSS LPAR - StandbyREVTAX LPAR - StandbyREVTAX LPAR - ProductionSystem: Virtual Environments (Manual process to rebuild and restore servers based on availability of resources)DOA vCenter211 Aspinal Avenue Hagatna, Guam 96910DPHSS vCenterREVTAX vCenter123 Chalan Kareta, Mangilao, Guam 969131240 Army Drive Barrigada, Guam 96913The standby facility will be used by the IT department and the Disaster Recovery teams; it will function as a centrallocation where all decisions during the disaster will be made. It will also function as a communications hub for OTECH.3.2 Data and BackupsRefer to the OTECH Backup Policy and Procedures for information on the backups. Servers will be restored based on theorder of criticality.4.0 Communicating During a DisasterIn the event of a disaster, the DR Lead or DR Management Lead will need to communicate with various parties to informthem of the effects on the business, surrounding areas and timelines. They will be responsible for contacting allstakholders.4.1 Communicating with the AuthoritiesThe DR Lead or DR Management Lead’s first priority will be to ensure that the appropriate authorities have been notifiedof the disaster, providing the following information: The location of the disasterThe nature of the disaster11 of 15 P a g e

IT Disaster Recovery Plan POLICY#: OTECH-POL2017-005The magnitude of the disasterThe impact of the disasterAssistance required in overcoming the disasterAnticipated timelines4.2 Communicating with EmployeesThe DR Lead or DR Management Lead’s second priority will be to ensure that the entire company has been notified ofthe disaster. The best and/or most practical means of contacting all of the employees will be used with preference onthe following methods (in order): Text Message (via Whatsapp group chat)E-mail (via corporate e-mail where that system still functions)Telephone to employee work numberTelephone to employee mobile phone numberThe employees will need to be informed of the following: Whether it is safe for them to come into the officeWhere they should go if they cannot come into the officeWhich services are still available to themWork expectations of them during the disaster4.3 Communicating with ClientsAfter all of OTECH’s employees have been informed of the disaster, the DR Lead or DR Management Lead will beresponsible for informing clients of the disaster and the impact that it will have on the following: Anticipated impact on service offeringsAnticipated impact on delivery schedulesAnticipated impact on security of client informationAnticipated timelinesCrucial clients will be made aware of the disaster situation first. All other clients will be contacted only after all crucialclients have been contacted.4.4 Communicated with VendorsAfter all of the organization’s employees have been informed of the disaster, the DR Lead or DR Management Lead willbe responsible for informing vendors of the disaster and the impact that it will have on the following: Adjustments to service requirementsAdjustments to delivery locationsAdjustments to contact informationAnticipated timelines12 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-005Crucial vendors will be made aware of the disaster situation first. All other vendors will be contacted only after all crucialvendors have been contacted.Vendors encompass those organizations that provide everyday services to the enterprise, but also the hardware andsoftware companies that supply the IT department.5.0 Dealing with a DisasterIf a disaster occurs in OTECH, the first priority is to ensure that all employees are safe and accounted for. After this, stepsmust be taken to mitigate any further damage to the facility and to reduce the impact of the disaster to theorganization.Regardless of the category that the disaster falls into, dealing with a disaster can be broken down into the followingsteps:1)2)3)4)5)6)7)Disaster identification and declarationDRP activationCommunicating the disasterAssessment of current and prevention of further damageStandby facility activationEstablish IT operationsRepair and rebuilding of primary facility5.1 Disaster identification and DeclarationSince it is almost impossible to predict when and how a disaster might occur, OTECH must be prepared to find out aboutdisasters from a variety of possible avenues. These can include: First hand observationSystem Alarms and Network MonitorsEnvironmental and Security Alarms in the Primary FacilitySecurity staffFacilities staffEnd users3rd Party VendorsMedia reportsOnce the Disaster Recovery Lead has determined that a disaster had occurred, s/he must officially declare that thecompany is in an official state of disaster. It is during this phase that the Disaster Recovery Lead must ensure thatanyone that was in the primary facility at the time of the disaster has been accounted for and evacuated to safetyaccording to the company’s Evacuation Policy.While employees are being brought to safety, the Disaster Recovery Lead or the Disaster Recovery Management Leadwill begin contacting the Authorities and all employees not at the impacted facility that a disaster has occurred.13 of 15 P a g e

IT Disaster Recovery PlanPOLICY#: OTECH-POL2017-0055.2 DRP ActivationOnce the Disaster Recovery Lead has formally declared that a disaster has occurred s/he will initiate the activation of theDRP by triggering the Disaster Recovery Call Tree. The following information will be provided in the calls that theDisaster Recovery Lead makes and should be passed during subsequent calls: That a disaster has occurredThe nature of the disaster (if known)The initial estimation of the magnitude of the disaster (if known)The initial estimation of the impact of the disaster (if known)The initial estimation of the expected duration of the disaster (if known)Actions that have been taken to this pointActions that are to be taken prior to the meeting of Disaster Recovery Team LeadsScheduled meeting place for the meeting of Disaster Recovery Team LeadsScheduled meeting time for the meeting of Disaster Recovery Team LeadsAny other pertinent informationIf the Disaster Recovery Lead is unavailable to trigger the Disaster Recovery Call Tree, that responsibility shall fall to theDisaster Management Team Lead.5.3 Communicating the DisasterRefer to the “Communicating During a Disaster” section 4.0 of this document.5.4 Assessment of Current and Prevention of Further DamageBefore any employees from OTECH can enter the primary facility after a disaster, appropriate authorities must firstensure that the premises are safe to enter.During each team’s review of their relevant areas, they must assess any areas where further damage can be preventedand take the necessary means to protect OTECH’s assets. Any necessary repairs or preventative measures must be takento protect

This Disaster Recovery Plan (DRP) captures, in a single repository, all of the information that describes the Office of Technology's (OTECH) ability to withstand a disaster as well as the processes that must be followed to achieve disaster recovery within the Government of Guam's (GovGuam) information technology infrastructure and systems.