Transcription
Disclaimer: This document is for informational purposes. It is not a commitment to deliver any material, code, orfunctionality, and should not be relied upon in making purchasing decisions. The development, release, and timing ofany features or functionality described in this document remains at the sole discretion of Oracle.ORACLE DAT A SHEETOracle Fusion Cloud ServiceBusiness Continuity and Disaster Recovery Status Report 2019CONTENT Test PreparednessTest ProcessTest CasesKey FindingsGlossaryThis document summarizes the aggregated test results of various DisasterRecovery (DR) switchover and failover scenarios for the Oracle Fusion CloudService and Applications: Oracle Customer Relationship Management (CRM),Oracle Fusion Enterprise Resource Planning (ERP), Oracle Fusion HumanCapital Management (HCM) and Oracle Fusion Supply Chain Management(SCM). These tests were conducted to verify the DR capabilities of the OracleFusion Cloud Service. Each test case was completed one or more times. Testresults and timings may vary based on real-time scenarios. The results belowdocument the two Oracle Fusion Cloud Service DR exercises which werecompleted on May 25, 2019Disaster Recovery Test PreparednessThe disaster recovery test preparedness process has been developed followingindustry standards and best practices. This ensures that Oracle Fusion CloudService follows a repeatable defined process to ensure proper adherence toOracle standards for Disaster Recovery. During Disaster Recovery testing thefollowing components are validated in addition to application recovery.**All DR Plan documentation is for Oracle internal use onlyCONFIDENTIAL – ORACLE RESTRICTED1
Oracle Fusion Cloud Service Disaster Recovery PlanDocument ComponentDocument Verified as Accurate andMost Recent Review DateCompletePrerequisites and DependenciesDocumentation reviewed quarterly,May 25, 2019verified as complete.Resource RequirementsDocumentation reviewed quarterly,May 25, 2019verified as complete.Technical Staff Roles and ResponsibilitiesDocumentation reviewed quarterly,May 25, 2019verified as complete.Components and ArchitectureDocumentation reviewed quarterly,May 25, 2019verified as complete.Disaster Recovery WorkflowDocumentation reviewed quarterly,May 25, 2019verified as complete.Recovery ProceduresDocumentation reviewed quarterly,May 25, 2019verified as complete.Application Verification ProceduresDocumentation reviewed quarterly,May 25, 2019verified as complete.Reconstitution procedureDocumentation reviewed quarterly,May 25, 2019verified as complete.Oracle Fusion Cloud Service Communication PlanDocument ComponentDocument Verified as Accurate andMost Recent Review DateCompleteDisaster Declaration FlowDocumentation reviewed quarterly,May 25, 2019verified as complete.Declaration Roles and ResponsibilitiesDocumentation reviewed quarterly,May 25, 2019verified as complete.Conference Bridges and Distribution ListsDocumentation reviewed quarterly,May 25, 2019verified as complete.Escalation PathsDocumentation reviewed quarterly,May 25, 2019verified as complete.Internal Communication FlowDocumentation reviewed quarterly,May 25, 2019verified as complete.External Communication FlowDocumentation reviewed quarterly,verified as complete.CONFIDENTIAL – ORACLE RESTRICTED2May 25, 2019
All documentation is in a standard format for all Oracle Cloud Services to ensurecontinuity. Documentation is updated in conjunction with production changes as partof the Oracle Fusion Cloud Service and Applications (CRM – ERP – HCM and SCM)standard change management processes. Disaster Recovery governanceprocesses are in place to ensure that the Oracle Fusion Cloud Service is continuallyrecoverable in the event of disaster declaration.Disaster Recovery Test ProcessThe disaster recovery testing includes all components of the cloud service. Thisensures that full end to end testing is vetted. The switchover tests include bothpositive and negative test cases. Switchover/Failover tests are exercised annually,and tabletop tests are exercised quarterly.DR support occurs between a production and an alternate facility that aregeographically separated. Oracle does not disclose the physical details of its datacenters or POD level details about any of its customer deployments. DR occurswithin a particular geographic region and Oracle may make changes to productionand alternate sites as needed. Oracle cannot disclose any details beyond this.In the event of an Oracle declared disaster, Oracle will resume production servicesat an alternate facility.To ensure thoroughness, a full failover of Fusion’s was conducted to thecorresponding recovery facility. The Oracle Fusion Cloud Service disaster recoverytest consists of the following activities: Disaster Simulation: Simulation of a disaster, such as Production Datacenterfailure. Database Failover: Recovery and associated activities necessary to bringdatabases online. Application Failover: Application recovery and reconfiguration to ensure fullfunctional service resumption including DNS, configuration and content backups. Service Restoration: A series of configuration management tools maintainedby replicating configuration changes and preventing configuration drift, theapplication consistency between the production site and alternate sites. Validation Testing: Post restoration functional testing to ensure applicationfunctionality. Service Reconstitution: Fail back to original siteCONFIDENTIAL – ORACLE RESTRICTED3
Failover / Switchover Test CasesThe below highlight the outcome of various component failovers pertaining to therecovery of the service. Successful completion ensures the ability to successfullyrecover all aspects of the service. Also note that several of the tests are executed inparallel to ensure optimal RTO.Test lannedswitchoverusing OracleSite GuardGUIPre-checks andswitchover shouldsucceedPassed 120minutes2Plannedswitchoverusing theEnterpriseManagerCommandLineInterface(EMCLI) toolPre-checks andswitchover shouldsucceedPassed 62minutes3Test DNSConnectionsTraffic redirects toAlternate siteshould succeedPassed 15minutes4RTO/RPOConstraintsRPO of 1 hourand RTO of 12hours shouldsucceedPassed 1 hrCommentsRPO & 12 hrRTO5Functionaltesting – Appstack isperformingas designedFunctional testingshould succeedPassed 15minutesDisruptive Test CasesThese test cases are designed to ensure that Oracle Site Guard can successfullyhandle invalid input, unexpected user behavior, and unexpected systemconfiguration.CONFIDENTIAL – ORACLE RESTRICTED4
EMCCThe EMCC-related test cases represent scenarios around abnormal consolestatuses or Enterprise Manger agent activity.EMCC RELATED DISRUPTIVE TEST CASESCase NoTest ScenarioExpected BehaviorTestResultsTest Duration1Switchover when one ormore agents on theproduction or alternatehosts are downPre-checks should failPassed3 minutes, 41 seconds2Switchover or failover isattempted when theunderlying database isalready role reversedPre-checks shoulddetect this state andfail with appropriateerror messagePassed1 minute3Switchover is attemptedwith errors in DBTransport service or DBApply Log servicePre-checks should failPassed4 minutes, 23 seconds4Switchover or failover isperformed when onlyone instance of thestandby RAC databaseis upSwitchover or failovershouldsucceedPassed1 hour 20 minutes5Switchover or failover isattempted when allinstances of the standbyRAC database are downPre-checks should failPassed3 minutes 55 seconds6Switchover is attemptedwhen the production oralternate database isnot reachablePre-checks should failPassed3 minutes 55 seconds7Switchover or failover isattempted with invalidDB sysdba credentialsfor Alternate database inEM credential storePre-checks should failPassed4 minutes 9 seconds8Switchover operation isattempted when OracleData Guard Brokerreports any error orwarning in brokerconfigurationPre-checks should failPassed3 minutes 20 seconds9Switchover operation isattempted when DataGuard Monitor (DMON)process is down onproduction or alternatedatabasePre-checks should failPassed3 minutes 41 secondsCONFIDENTIAL – ORACLE RESTRICTED5
Key FindingsThese test cases are designed to ensure that Oracle can successfully recover theFusion environment and Applications (CRM – ERP – HCM and SCM) to an alternatefacility and restore business operations. Testing resulted in both a SuccessfulFailover exercise and a Successful Switchover exercise on May 25th, 2019.Glossary of TermsAsynchronous (replication) - Process by which data is written to the productionstorage first and then data is copied to the replica.Dataguard – Provides a set of services that create, maintain, manage, and monitorone or more standby databases to enable production Oracle databases to survivedisasters and data corruptions.RPO – Recovery Point Objective is defined as the maximum targeted period inwhich data might be lost from an IT service due to a disaster. For instance, if theRPO is set to one-hour, off-site replication must be continuously maintained toensure that data loss will be less than one hour.RTO – Recovery Time Objective, is the targeted duration of time within which abusiness process must be restored after a disaster. It includes the time for therecovery, testing, and communication to the service subscribers.ZFS – File system and logical volume manager used for highly scalable storagesystems.CONTACT USFor more information about Fusion, visit oracle.com or call 1.800.ORACLE1 to speak to an Oraclerepresentative.CONNECT W ITH r.com/oracleoracle.comCopyright 2019, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and thecontents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warrantiesor conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for aparticular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed eitherdirectly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic ormechanical, for any purpose, without our prior written permission.Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and aretrademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo aretrademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0619CONFIDENTIAL – ORACLE RESTRICTED6
In the event of an Oracle declared disaster, Oracle will resume production services at an alternate facility. To ensure thoroughness, a full failover of Fusion's was conducted to the corresponding recovery facility. The Oracle Fusion Cloud Service disaster recovery test consists of the following activities:
