Transcription
Sei ceaicpa.org/soc4soOrganizations SeSecure Cloud ServicesManaged & Compliant Infrastructurer vitionizafo ceO888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide \\ Table of ContentsTable of ContentsPurposeBusiness Associate AgreementRecovery Time Objective (RTO)Recovery Point Objective (RPO)Hosted Infrastructure Redundancy ProtectionThe Disaster Recovery PlanHow to Activate the Disaster Recovery PlanThe Disaster Recovery ProcessEstablish a Command CenterEstablish a Communication PlanRecovering Servers and ServicesTechnical Teams Recover ServicesMonitoring the Recovery ProgressTestingRoot Cause Analysis (RCA)Plan Service Relocation and Return to BAUSecure Cloud ServicesManaged & Compliant Infrastructure44556667788910111112888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide \\ ForewordThis whitepaper details the key personnel,processes, and procedures required for thepreparation and planning of a disasterrecovery (DR) scenario. DR is achieved byproviding healthcare clients with a resilientand HIPAA-compliant cloud infrastructureservice.This service utilizes a leading disasterrecovery technology platform, providingthe healthcare client with compute, network, storage, and an application stackwhich is seamlessly replicated to a targetdata center location using encrypted transfer techniques.This technology allows for some of thebest-in-class Recovery Point Objectives(RPO) and Recovery Time Objectives (RTO).Data is replicated in near real-time, and thecloud service enables us to utilize an on-demand compute infrastructure to rebuildproduction environments within a disparate cloud data center location.Secure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide4Purpose This document aims to serve as aclient reference point; it is an overview of the disaster recovery plan(DRP) and the processes establishedto ensure a smooth transition intoDR operations. We will also discussthe planning of failing back of infrastructure services to source production location.The aim of the Disaster RecoveryPlan (DRP) is to explain: What is a disaster recovery scenario? When to declare a disaster? How does an organizationinvoke disaster recovery? How does communication flowduring a disaster scenario? What are the key roles andresponsibilitiesofpersonnelassigned to the recovery team? How does the business keeprunning in the event of a disaster? Recovery time objectives – howlong can the business operate without critical IT systems?Recovery point objectives – towhat point in the processing cyclecan the provider recover? What are the post DR activities?(root cause analysis) How can the organization continuously test and evolve the DRP?The Business AssociateAgreementHIPAA compliance demands that thecloud provider have a full understanding of the client’s cloud hostingrequirements and offer specificguarantees of compliance to thephysical, technical, and administrative safeguards required for HIPAAcertification.Before any disaster recovery planning, work is completed to understand our clients’ hosted infrastructure and cloud infrastructurerequirements for disaster recovery,as well as the what important ePHIprotected data is in scope and mustbe protected.It is important to work with health-Secure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide5care clients to understand: What electronic patient healthinformation (ePHI) data is in scope.and the servers, databases, andapplications associated with thatinformation What is the source computinginfrastructure which requires restoration to the recovery locationduring a DR scenario (this is typicallyreferred to as in scope productionservers)The BAA outlines the sharedresponsibilities for data protectionunder HIPAA between the cloudservice provider and the client,within which the disaster recoveryservice can be designed. It enablesthe creation of service processes,proceduresandautomationblueprints for recovering services. Itenables the provider to design thefailover and failback process, andalso helps with the agreement ofservice levels.Disaster recovery (DR) infrastructureas a service (IaaS) is usually onlyapplicable to client production servers. Test and development infrastructure is typically out of the scopeof most DR offerings.Recovery Time Objective (RTO)RTO is the maximum tolerable timetaken to recover customer systemsafter a disaster scenario is declared.RTO is measured from the time thehosting services are declared to havebecome unavailable until the timesuch services become available andoperational per applicable servicelevel agreements.Recovery Point Objective(RPO)RPO is the measure of the maximumacceptable data loss recorded bytime. It is the maximum allowed ageof data files when recovering theservice. RPO is measured from thetime the first transaction is lost, orfrom the time the hosting servicesbecame unavailable to the timeservices returned to operational.Secure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide6Hosted InfrastructureRedundancy ProtectionThe cloud provider builds and maintains a minimum of N 1 redundancyin their data center technology. Thisensures resiliency in system availability in the event of a componentfailure. N 1 means components (N)have twice the amount required forcontinued operation. This helpsensure the highest levels of reliability and availability in primary andrecovery data centers if any components were to fail.Regular data center checks are completed on top of the automatedmonitoring which is in place toensure the maximum redundancy ofthe hardware.Key areas where N 1 is employedinclude: Networks – dual feed powersupply (PSU) and failover paired networking stack Storage – varying RAID levels toensure disk redundancy, dual feedPSU and multisite replication tech-nology Storage which is highly redundant across multiple regions Servers – dual feed PSU, clustering, high availability for criticalapplications (where applicable) Real-time monitoring Network load balancing on multiple layers for separation of webfrontends, applications, and databasesEven despite all these redundancyprotection systems, an outage maystill happen. Consideration must begiven to a total loss of the primarysite, such as a lightning strike, tornado, or flood.The Disaster Recovery PlanHow to Activate the DisasterRecovery PlanCloud providers often request thatthe client appoints a DR coordinatorfor all communications relating tothe disaster recovery process. It isalso advisable to have backup con-Secure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide7tact in case the usual person is onvacation or unwell.Invocation of the disaster recoveryplan must be agreed upon by one ofthe pre-approved authorized personnel who must contact the cloud provider using the emergency contactdetails provided in the DRP.The Disaster Recovery ProcessOnce the disaster recovery plan(DRP) has been declared, the casewill be escalated at the highest priority to the disaster recovery lead(DRL). The DRL will start the processof activating DR and declaring anofficial state of disaster. It is duringthis phase that the DRL must ensure,if applicable, that anyone that is inthe primary location at the time ofthe disaster has been accounted forand evacuated to safety.An all-hands call will be establishedby the DR Lead with the aim of: Confirming to all that a disasterhas occurred Explain what has happened and(if known) the nature of the disaster Understanding the extent of thedisaster Understanding the impact theDR scenario has had on productionservices If possible, estimate the expected time it will take to recover fromthe disaster Document the steps taken priortoo, and since declaring a disasterEstablish a Command CenterIf applicable, the provider mustestablish a business continuity officeseparate from the primary datacenter location. Typically, remoteworkers will be contacted and maybe asked to attend the commandcenter. If the primary productionlocation is not physically affected,then operations can continue fromthe usual headquarters.The command center is the focalpoint for all recovery actions andoperations. The command center willalso be where all communications toSecure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide8key stakeholders will be provided.The cloud provider is responsible forkeeping the client’s senior management team and all command centerpersonnel informed on all availablecommunications channels during theDR.Establish a CommunicationPlanAs part of the HIPAA complianceprocedures, up-to-date call trees willbe used to contact all relevant support teams, client contacts, andimportant personnel.The DR lead will need to communicate with various parties to informthem of the disaster and the impacton the business. This may includecontacting employees, customers,third-party vendors, and suppliers.Depending on the DR scenario, itwill usually be the technical teamswho will receive the first alerts orcommunications about a disaster.These teams will then contact therequired escalation persons using adetailed escalation procedure.Modern communication methodscan be used, including automatedtext alert systems for critical systemoutages; employees may also have acompany mobile with access to employee contact details, emailaddresses, and emergency procedures.To cover all scenarios, the DRPemergency procedures, the DRPescalation path, and the emergencycontact details for employees, customers, and third party providersshould be stored on physical mediain two separate physical locationswithin fire safes. All DRP proceduresmust be regularly reviewed andupdated accordingly in all locations.Recovering Servers and ServicesAfter the disaster has been declaredand approval has been granted toinvoke the DRP, the technical teamswill then begin to recover clientservers and services. The replicationSecure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide9technology used by the cloud provider leverages vast reserves of computation within the recovery location.This process might be system-automated, triggering when certainpre-defined disaster conditions aremet. The disaster recovery solutionutilized by the cloud provider variesas per provider but is typically asource-to-target replication solution.Technical Teams Recover ServicesTechnical operational playbooks anddisaster recovery procedures are avital part of HIPAA compliance.When in a DR scenario, these documents contain the building blocks torecover critical business services.Despite recovery often being anautomated process, it is still vital tohave a step-by-step recovery guideavailable if issues arise.The technical playbooks give adetailed step-by-step process for thetechnical teams to follow. This is acomprehensivetechnicalguideexplaining how to restore serviceinto the DR location.To summarize, this process may contain the following steps: Wait for approval from the DRteam lead to activate the disasterrecovery plan All teams will report into thecommand center via the assignedbridge number and receive recoveryactivities pertinent to their teams The technical infrastructureteam will be informed which clientservices are affected by the outageand are being moved to their alternate DR site The operations team will identify the appropriate client credentialsand validate that infrastructure systems are being failed over too They must log in to the designated DR infrastructure and manually restore the required infrastructureservers They must monitor the designated client services and manuallyrestore the required client servicesSecure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide 10The operations teams will bringup all shared systems and validatethat all services are running andoperational The operations teams will confirm monitoring tools are set up forall of the shared servers/applicationsand networks The operations team will confirm backup scripts are in place andbackups are scheduled The operations network teamwill cut over the client DR networktunnels to the alternate DR site to“switch” client secured VPNs to therecovery (DR) location The operations network teamwill modify any DNS entries andchange internet routing to send allinfrastructure related traffic over tothe alternate DR failover site Designated system administrators will work with the operationsteams to validate all connectivity toall application services and that theyare operating as expected Validate that all ePHI data is nowavailable in the DR location andavailable within RTO and RPO service levelsThe disaster recovery proceduresshould contain information about: Details of each client serverinstance types, operating systems,and applications List of critical servers with ePHI The selected regional failoversite for the client’s primary site Documented VPC subnets andreplication network information The startup order of the infrastructure services if manual restoration is neededMonitoring the Recovery ProgressThe disaster recovery lead will trackthe progress, report into the command center and make the necessarynotifications to the appropriateteams while in a disaster recoverysituation. Updates will be providedusing the communication plandetailed above.The technical teams will track theprogress of the recovery using anymonitoring tools available. These willmonitor the health of the applica-Secure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide11tions, network, and infrastructure.SNMP monitoring agents are typically already deployed to each systemso base level monitoring of theserver uptime, operating systeminformation, database informationcan be checked and cross-referenced with playbooks and procedures.available and protected and that thehealthcare organization can accessthe data and associated applicationsfrom the recovery location.Monitoring solutions vary from eachcloud provider. Monitoring dashboards can provide a visual representation of IT systems that aredown, pending, or in a failover state.Root Cause Analysis (RCA)TestingAfter the service has been recoveredinto the DR site using the cloud provider’s replication technology, theprovider must work with its clients totest essential infrastructure and ITservices to ensure data consistencyand that production services are inline with RTO and RPO objectives.Arguably the most important stephere is the data integrity checks –these ensure that all electronicpatient health information (ePHI) isTesting is a two-way process wherethe client and the provider will needto work together to ensure the service is fully restored.During a disaster scenario, therecovery team’s objectives willalways be to implement the disasterrecovery plan and recover production services as quickly as possible.Post-scenario, the questions of whathappened and why can be asked.Within seven days of the DR scenario, all of the participants involved,including management, DR leads,technical teams, etc. must meet todiscuss what happened. These meetings aim to create a root cause analysis (RCA) report and get a full understanding of the disaster incidentfrom start to finish.It is crucial this meeting takes placeSecure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide12as soon as possible to ensure thefacts are fresh in everyone’s minds,but this also serves as a critical factorof HIPAA compliance, as the lessonslearned during the DR incident needto be translated into updating theprocess and procedures followed.The goals of the meetings are to discover and document: Start date & time of the incidentEnd date & time of the incidentDetails of the support teamsinvolved Description of initial detailswhich caused the decision to invokeDR Symptoms of incident Impact of incident Full analysis of the DR scenario Discuss the causes of the incident Discuss the resolution details What worked well? What could be done better? What lessons have beenlearned? Can any preventative measuresbe introduced? Discuss the resolution details ofhow the incident was resolved List timings of milestones duringincident Create the root cause analysisreport Update and make any changesthat may be required to the disasterrecovery plan.Plan Service Relocation and/orReturn to BAUDependent on the disaster scenario,there may be a requirement to keepservices running in the DR site for alengthy period. During this time, it isessential to carefully plan a strategyfor returning service to the primaryproduction site/region. Data in theprimary site will most likely be outof sync, so consideration must bemade to ensure a complete resyncof data so a smooth failback ofrecovery can take place.Once the primary site has beenbrought back online, the technicalteams will identify the primary siteand start to replicate data back fromthe recovery site to the primary site.Once data integrity is available, theSecure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
HIPAA Disaster Recovery Guide13replication technology will allow youto failback services from the recovery (DR) site back to the primarysite. The failback process is usuallyan automatic job that runs onceinvoked by the technical teams.Again, the failback will be monitoredand thoroughly tested upon successful failback. The final step is toensure that the replication fromprimary to failover site is reactivatedand in sync.Get Help with HIPAA ComplianceHIPAA Compliant Hosting by Atlantic.Net is SOC 2 & SOC 3 certified and HIPAA &HITECH audited, designed to secure and protect critical healthcare data andrecords. Get a free consultation today! Call 888-618-3282 or review our solutionsat Secure Cloud ServicesManaged & Compliant Infrastructure888-618-DATA (3282)sales@atlantic.netwww.atlantic.net
The Disaster Recovery Plan 6 How to Activate the Disaster Recovery Plan 6 The Disaster Recovery Process 7 Establish a Command Center 7 Establish a Communication Plan 8 . HIPAA Disaster Recovery Guide \\ Table of Contents Purpose This document aims to serve as a client reference point; it is an over-
