WIXLIB

The Definitive Disaster RecoveryPlan ChecklistEvery day there are new horror stories of tech outages,downtime, and data loss — even at the best of companies. Whendisaster strikes, engineering teams are dispatched to repair thedamage, while PR teams work overtime to restore customerconfidence. It’s a time-consuming and often expensive effort.No matter what the cause of the disaster, the organizationsthat manage them most effectively, and with the least amountof collateral damage, are those with a comprehensive, easyto-follow, and regularly tested disaster recovery (DR) plan.Whether you already have a DR plan or you are just beginningthe process of creating one for your organization, this DefinitiveDisaster Recovery Plan Checklist will help you ensure you’veincluded all the crucial components in your plan.#1: Determine Recovery Objectives(RTO and RPO)The main goal of DR is to keep your business operating as usual,all the time. This means you need to determine which workloadsare the most mission-critical to your organization, and whatRecovery Time Objective (RTO) and Recovery Point Objective(RPO) is required for these workloads. RTO is the amount oftime required to recover from a disaster after notification ofbusiness disruption. A reliable DR plan must contain a clearlystated allowable RTO. If your business cannot withstand anhour of downtime without losing customers to competitorsor paying penalty fees due to service-level agreements (SLAs),it’s mission-critical to your business to be operational beforean hour has expired. In this case, your RTO would be onehour. RPO is the window of time that data loss is tolerable.If your business can only survive four hours of data loss andyou perform nightly backups, you would have a catastrophicloss of important data if disaster strikes the afternoon aftera full backup. In this case, your RPO would be four hours. Acompany’s RTO and RPO will affect its DR strategy as well as1associated expenses. While a simple file-level backup systemmight be sufficient for some applications, your mission-criticalapplications will likely need a DR technology based on realtime continuous data replication, to enable you to achievenear-zero RPO and RTO.#2: Identify StakeholdersThe next step is to identify all those who need to be updatedonce disaster strikes. In addition to those involved withperforming the actual recovery from a disaster (e.g., engineers,support, executives), you should also pinpoint members ofyour PR and marketing team, vendors, third-party suppliers,and even key customers. Many companies keep a register ofstakeholders, a good starting point for identifying all of thestakeholders you’ll want to notify if there is a disaster.#3: Establish Communication ChannelsOrganizations should keep a list of all teams responsible forDR, along with their roles and contact information. Establisha complete chain of command, including relevant executiveleadership and accountable individuals from each of theengineering teams (e.g., network, systems, database, andstorage). Assign a designated contact person from the supportteam as well. You should also set up dedicated communicationchannels and hubs, such as an on-site room where everyonewill gather or an online information-sharing tool to use forinstant messaging.#4: Collect All Infrastructure DocumentationAlthough your engineering teams that are dispatched to activateDR procedures possess the required skills and knowledgefor shifting operations to your target DR site, infrastructure 2019 CloudEndure Ltd. All Rights Reservedwww.cloudendure.com

documentation is still a must, especially with of the pressure thatcomes with a disaster. Even the most highly trained engineersprefer to follow infrastructure documentation line by line andcommand by command during a disaster. The documentationshould list all of your mapped network connections (withfunctioning devices and their configurations), the entire setupof systems and their usage (OS and configuration, applicationsrunning, installation and recovery procedures), storage anddatabases (how and where the data is saved, how backupsare restored, how the data is verified for accuracy), and cloudtemplates. It should contain everything IT-related that yourbusiness relies upon. Of course, always keep hard copies of thedocumentation, as outages may knock your internal systemsoffline.#5: Choose the Right TechnologyThere are many effective solutions for business continuitybeyond traditional in-house, on-premises or outsourcedDisaster Recovery as a Service (DRaaS) solutions. Anotheroption is to utilize cloud-based DR, where you can spin upyour DR site on a public cloud such as AWS in minutes usingan automated DR solution. Before selecting a DR solution, youshould consider total cost of ownership (which is much higherwith an on-premises DR strategy because of the duplicatehardware and software licensing costs), scalability, ability torecover to previous points in time, maintenance requirements,recovery objectives, and ease of testing. You should also takeyour current production setup into consideration (the hardwareand software that you run in a production environmentevery day).#6: Define Incident Response ProcedureAn incident response procedure is a must in every DR plan.This is where companies define in detail what is considered adisaster. For example, if your system is down for five minutes,should you declare a disaster? Does it matter what the cause is?In addition to listing the events that will be declared a disaster,the plan should indicate how you will verify the disaster isreally happening and how the disaster will be reported —by an automatic monitoring system, raised by calls from site2reliability engineering (SRE) teams, or reported by customers?To verify that a disaster is taking place, check the status ofcritical network devices, application logs, server hardware, orany other critical components in your production system thatyou monitor proactively. If something is odd or not working,such as customers being unable to reach your online shop oraccess their data, then you definitely have a disaster on yourhands. Being able to quickly detect the failure and verify thatit’s not a false alarm will impact your ability to meet your RTO.#7: Define Action Response ProcedureAfter declaring a disaster, the recovery environment should beactivated as soon as possible. An action response procedureoutlines how to perform failover to the DR target site, with allnecessary steps. Even if your recovery process uses a DR toolor DRaaS provider that launches your DR site automatically, youshould still prepare the action response procedure in writingto be completely certain how the necessary services will bestarted, verified, and controlled. In addition, it is not enoughto simply spin up production services in another location. Averification process in which you make sure that all the requireddata is in place, and all the required business applications arefunctioning properly, is critical.#8: Prepare for Failback to PrimaryInfrastructureFor most companies, the DR site is not designed to run dailyoperations, and a lot of effort may be required to implementthe moving of data and business services back to the primaryenvironment once the disaster is over. You may need to planfor downtime or a partial disruption of your business duringthe failback process. Fortunately, there are DR solutions thatprovide seamless failback to your primary location, triggeredeither automatically or manually after you have completedverification of your primary environment.#9: Perform Extensive TestsTesting your DR plan in action is essential, but is often neglected.Many organizations don’t test on a regular basis because their 2019 CloudEndure Ltd. All Rights Reservedwww.cloudendure.com

failover procedures are too complex and there are legitimateconcerns that failover tests will lead to a disruption of theirproduction environment or even data loss. Despite theseconcerns, it is important to schedule regular (minimally, oncea quarter) failover tests to your DR site. If you never test yourDR plan, you are putting your entire business at risk, sinceyou might not be able to recover in time (or at all) if disasterstrikes and your recovery plan doesn’t work. Not only will DRdrills demonstrate whether your DR solution is adequate, butit will also prepare your engineers and supporting teams torespond quickly and accurately to a disaster. Performance testsare also important to assess whether or not your secondarylocation is sufficient to withstand the business load.#10: Stay Up-to-DateKeeping all of your DR documentation updated is as importantas regularly scheduled testing of your target infrastructure. Afterevery test (or worse, every incident), review what happened,how your teams handled the test or event, and document yourfindings. Many companies keep a risk register that, in additionto listing potential risks to business continuity, include analysesof previous disasters and lessons learned.Disaster Recovery Plan ExampleHere’s an example summary of a DR plan for a modern company, running 200 physical and virtual servers in an on-premisesdata center. (Note: The plan below is an overview. An organization’s full DR plan would run anywhere from 10 to more than 100pages.) The company relies on its production environment being available 24/7 to customers, which is why their DR strategyneeds to function perfectly with minimal downtime. This company uses AWS as their target DR infrastructure in order to cutcosts and improve their RTO and RPO.RTO: 5 minutesAccording to their RTO, the production environment must be shifted from the on-premisesdata center to AWS within 5 minutes.RecoveryObjectivesRPO: 0 minutesRPO is near-zero because the business cannot tolerate any data loss. This is why data iscontinuously replicated from the on-premises environment to the cloud.Required Documents:- Stakeholder Register- Risk Register- Communication Plan3 2019 CloudEndure Ltd. All Rights Reservedwww.cloudendure.com

Sources of Incident Reporting:- Automatic monitoring service- External (customers) or internal incident reporting (support, engineering)When Incident Is Reported:Incident Reporting- Gather responsible teams and implement chain of command- Perform required production checks to establish whether it is a real threat- Determine if the production environment can be repaired within the defined RTO, or if theDR plan should be triggeredRequired Documents:- Incident handling documentationOps/Sys Admin Teams Should:1. Verify data replication and diagnose potential loss of data2. Check network connectivity3. Route traffic to disaster recovery siteAction Response4. Verify secondary production before startingRequired Documents:- Infrastructure documentation of physical environment- Failover procedures- AWS infrastructure documentation and logging procedureFallback Procedure:1. Perform verification of primary site when disaster has finished2. Perform verification of other components, such application/web servers, load balancers,network connectionsOperation Restore3. Prepare for failback by reversing data replication from the target to the source environment4. Perform final run tests before going live on the primary siteRequired Documents- Failback procedures- Findings and lessons learned documentation4 2019 CloudEndure Ltd. All Rights Reservedwww.cloudendure.com