WIXLIB

Oracle RAC with VMware virtualized environmentThis reference architecture uses VMware virtual machines to create two Oracle RAC clusters with four nodeseach. Although this solution guide describes a two 4-node configuration, this architecture can support scalableall-NVMe Cisco HyperFlex configurations, as well as scalable RAC nodes and scalable virtual machine countsand sizes as needed to meet your deployment requirements.Note: For best availability, Oracle RAC virtual machines should be hosted on different VMware ESXservers. With this setup, the failure of any single ESX server will not take down more than a single RACvirtual machine and node with it.Figure 2 shows the Oracle RAC configuration used in the solution described in this document.Figure 2.Oracle Real Application Cluster configurationOracle RAC allows multiple virtual machines to access a single database to provide database redundancy whileproviding more processing resources for application access. The distributed architecture of the Cisco HyperFlexsystem allows a single RAC node to consume and properly use resources across the Cisco HyperFlex cluster.The Cisco HyperFlex shared infrastructure enables the Oracle RAC environment to evenly distribute theworkload among all RAC nodes running concurrently. These characteristics are critical for any multitenantdatabase environment in which resource allocation may fluctuate.The Cisco HyperFlex all-NVMe cluster supports large cluster sizes, with the capability to add compute-onlynodes to independently scale the computing capacity of the cluster. This approach allows any deployment tostart with a small environment and grow as needed, using a pay-as-you-grow model.This reference architecture document is written for the following audience: Database administrators Storage administrators IT professionals responsible for planning and deploying an Oracle Database solutionTo benefit from this reference architecture guide, familiarity with the following is required: Hyperconvergence technology Virtualized environments SSD and flash storage Oracle Database 19c Oracle Automatic Storage Management (ASM) Oracle Enterprise Linux 2020 Cisco and/or its affiliates. All rights reserved.Page 6 of 49

Oracle Database scalable architecture overviewThis section describes how to implement Oracle RAC database on a Cisco HyperFlex system using two 4-nodeclusters. This reference configuration helps ensure proper sizing and configuration when you deploy a RACdatabase on a Cisco HyperFlex system. This solution enables customers to rapidly deploy Oracle databases byeliminating engineering and validation processes that are usually associated with deployment of enterprisesolutions.This solution uses virtual machines for Oracle RAC nodes. Table 1 summarizes the configuration of the virtualmachines with VMware.Table 1.Oracle virtual machine configurationResourceDetails for Oracle virtual machineVirtual machine specifications24 virtual CPUs (vCPUs)150 GB of vRAMVirtual machine controllers4 Paravirtual SCSI (PVSCSI) controllerVirtual machine disks1 500-GB VMDK for virtual machine OS4 500-GB VMDK for Oracle data3 70-GB VMDK for Oracle redo log2 80-GB VMDK for Oracle Fast Recovery Area3 40-GB VMDK for Oracle Cluster-Ready Services and voting diskFigure 3 provides a high-level view of the environment.Figure 3.High-level solution design 2020 Cisco and/or its affiliates. All rights reserved.Page 7 of 49

Solution componentsThis section describes the components of this solution. Table 2 summarizes the main components of thesolution. Table 3 summarizes the HX220c M5 Node configuration for the cluster.Hardware componentsThis section describes the hardware components used for this solution.Cisco HyperFlex systemThe Cisco HyperFlex system provides next-generation hyperconvergence with intelligent end-to-endautomation and network integration by unifying computing, storage, and networking resources. The CiscoHyperFlex HX Data Platform is a high performance, flash-optimized distributed file system that delivers a widerange of enterprise-class data management and optimization services. HX Data Platform is optimized for flashmemory, reducing SSD wear while delivering high performance and low latency without compromising datamanagement or storage efficiency.The main features of the Cisco HyperFlex system include: Simplified data management Continuous data optimization Optimization for flash memory Independent scaling Dynamic data distributionVisit Cisco's website for more details about the Cisco HyperFlex HX-Series.Cisco HyperFlex HX220c M5 All NVMe NodesNodes with all-NVMe storage are integrated into a single system by a pair of Cisco UCS 6400 or 6300 SeriesFabric Interconnects. Each node includes an M.2 boot drive (240G), an NVMe drive (500G) for data-loggingdrive, a single Optane NVMe SSD serving as write-log drive, and up to eight 1-TB NVMe SSD drives, for acontribution of up to 8 TB of raw storage capacity. The nodes use the Intel Xeon Gold 6248 processor familywith Cascade Lake-based CPUs and next-generation DDR4 memory and offer 12-Gbps SAS throughput. Theydeliver significant performance and efficiency gains as well as outstanding levels of adaptability in a 1-RackUnit (1RU) form factor.This solution uses eight Cisco HyperFlex HX220c M5 All NVMe Nodes for an eight-node server cluster toprovide two-node failure reliability as the Replication Factor (RF) is set to 3.See the Cisco HyperFlex HX220c M5 All NVMe Node data sheet for more information. 2020 Cisco and/or its affiliates. All rights reserved.Page 8 of 49

Cisco UCS 6400 Series Fabric InterconnectsThe Cisco UCS 6400 Series Fabric Interconnects are a core part of the Cisco Unified Computing System,providing both network connectivity and management capabilities for the system. The Cisco UCS 6400 Seriesoffer line-rate, low-latency, lossless 10/25/40/100 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE), andFibre Channel functions. The Cisco UCS 6400 Series Fabric Interconnects provide the management andcommunication backbone for the Cisco UCS B-Series Blade Servers, UCS 5108 B-Series Server Chassis, UCSManaged C-Series Rack Servers, and UCS S-Series Storage Servers. All servers attached to a Cisco UCS 6400Series Fabric Interconnect become part of a single, highly available management domain. In addition, bysupporting a unified fabric, Cisco UCS 6400 Series Fabric Interconnects provide both LAN and SANconnectivity for all servers within their domain. From a networking perspective, the Cisco UCS 6400 SeriesFabric Interconnects use a cut-through architecture, supporting deterministic, low-latency, line-rate10/25/40/100 Gigabit Ethernet ports, switching capacity of 3.82 Tbps for the 6454, 7.42 Tbps for the 64108,and 200 Gbps bandwidth between the 6400 Series Fabric Interconnect and the Cisco UCS I/O Model 2408(IOM 2408) per UCS 5108 blade chassis, independent of packet size and enabled services. The product familysupports Cisco low-latency, lossless 10/25/40/100 Gigabit Ethernet unified network fabric capabilities, whichincrease the reliability, efficiency, and scalability of Ethernet networks. The fabric interconnect supports multipletraffic classes over a lossless Ethernet fabric from the server through the fabric interconnect. Significant TCOsavings come from an FCoE-optimized server design in which Network Interface Cards (NICs), Host BusAdapters (HBAs), cables, and switches can be consolidated.Table 2.Reference architecture componentsHardwareDescriptionQuantityCisco HyperFlex HX220c M5 AllNVMe NodesCisco 1-Rack-Unit (1RU) hyperconverged node thatallows for cluster scaling with minimal footprintrequirements8Cisco UCS 6400 Series FabricInterconnectsFabric interconnects2Table 3.Cisco HyperFlex HX220c M5 Node configurationDescriptionSpecificationCPU2 Intel Xeon Gold 6248 CPU @ 2.50GHzMemory24 32-GB DIMMsCisco Flexible Flash (FlexFlash)Secure Digital (SD) card240-GB M.2Boot drivesSSD500-GB NVMeConfigured for housekeepingtasks375-GB Optane SSDConfigured as cache8 x 1-TB NVMe SSDCapacity disks for each nodeVMware vSphere, 6.5.0Virtual platform for CiscoHyperFlex HX Data PlatformsoftwareHypervisor 2020 Cisco and/or its affiliates. All rights reserved.NotesPage 9 of 49

DescriptionSpecificationCisco HyperFlex HX DataPlatform softwareCisco HyperFlex HX Data Platform Release 4.0(2a)Replication factor3NotesFailure redundancy from twosimultaneous, uncorrelatedfailuresSoftware componentsThis section describes the software components used for this solution.VMware vSphereVMware vSphere helps you get performance, availability, and efficiency from your infrastructure while reducingthe hardware footprint and your capital expenditures (CapEx) through server consolidation. Using VMwareproducts and features such as VMware ESX, vCenter Server, High Availability (HA), Distributed ResourceScheduler (DRS), and Fault Tolerance (FT), vSphere provides a robust environment with centralizedmanagement and gives administrators control over critical capabilities.VMware provides the following product features that can help manage the entire infrastructure: vMotion – vMotion allows nondisruptive migration of both virtual machines and storage. Its performancegraphs allow you to monitor resources, virtual machines, resource pools, and server utilization. Distributed Resource Scheduler – DRS monitors resource utilization and intelligently allocates systemresources as needed. High availability – HA monitors hardware and OS failures and automatically restarts the virtual machine,providing cost-effective failover. Fault Tolerance – FT provides continuous availability for applications by creating a live shadow instanceof the virtual machine that stays synchronized with the primary instance. If a hardware failure occurs, theshadow instance instantly takes over and eliminates even the smallest data loss.For more information, visit the VMware website.Oracle Database 19cOracle Database 19c now provides customers with a high-performance, reliable, and secure platform tomodernize their transactional and analytical workloads on-premises easily and cost-effectively. It offers thesame familiar database software running on-premises that enables customers to use the Oracle applicationsthey have developed in-house. Customers can therefore continue to use all their existing IT skills and resourcesand get the same support for their Oracle databases on their premises.For more information, visit the Oracle website.Note: The validated solution discussed here uses Oracle Database 19c Release 3. Limited testing showsno issues with Oracle Database 19c Release 3 or 12c Release 2 for this solution. 2020 Cisco and/or its affiliates. All rights reserved.Page 10 of 49

Table 4.Reference architecture software componentsSoftwareVersionFunctionCisco HyperFlex HX Data PlatformRelease 4.0(2a)Data platformOracle Enterprise LinuxVersion 7.6OS for Oracle RACOracle UEK Kernel4.14.35-1902.3.1.el7uek.x86 64x86 64Kernel version in Oracle LinuxOracle Grid and (ASM)Version 19c Release 3Automatic storage managementOracle DatabaseVersion 19c Release 3Oracle Database systemOracle Silly Little Oracle Benchmark(SLOB)Version 2.4Workload suiteSwingbench, Order Entry workloadVersion 2.5Workload suiteRecovery Manager (RMAN)Version 19c Release 3Backup and recovery manager forOracle DatabaseOracle Data GuardVersion 19c Release 3High availability, data protection, anddisaster recovery for Oracle DatabaseStorage architectureThis reference architecture uses an all-NVMe configuration. The HX220c M5 All NVMe Nodes allow eight NVMeSSDs. However, two per node are reserved for cluster use. NVMe SSDs from all eight nodes in the cluster arestriped to form a single physical disk pool. (For an in-depth look at the Cisco HyperFlex architecture, see theCisco white paper, Deliver Hyperconvergence with a Next-Generation Platform. A logical datastore is thencreated for placement of Virtual Machine Disk (VMDK) disks. The storage architecture for this environment isshown in Figure 4. This reference architecture uses 1-TB NVMe SSDs.Figure 4.Storage architecture 2020 Cisco and/or its affiliates. All rights reserved.Page 11 of 49

Storage configurationThis solution uses VMDK disks to create shared storage that is configured as an Oracle Automatic StorageManagement, or ASM, disk group. Because all Oracle RAC nodes must be able to access the VMDK disksconcurrently, you should configure the Multi-writer option for sharing in the virtual machine disk configuration.For optimal performance, distribute the VMDK disks to the virtual controller using Table 5 for guidance.Note: In general, both HyperFlex (HX) and Oracle ASM provide an RF factor. In our test environment, theresiliency provided by HX is applicable to all disk groups, and in addition, the DATA disk group isconfigured with normal redundancy provided by Oracle ASM. Table 6 shows the data disk group capacityafter applying normal redundancy. The capacities vary depending on the RF factor being set. (For instance,if RF is set to 2, actual capacities are one-half of raw capacity. If RF is set to 3, actual capacities are onethird of raw capacity.)Table 5.Assignment of VMDK disks to SCSI controllers; storage layout for each virtual machine (all disks are shared withall four Oracle RAC nodes)SCSI 0 (Paravirtual)SCSI 1 (Paravirtual)SCSI 2 (Paravirtual)SCSI 3 (Paravirtual)500 GB, OS disk500 GB, Data1500 GB, Data270 GB, Log1500 GB, Data3500 GB, Data470 GB, Log280 GB, FRA180 GB, FRA270 GB, Log340 GB, CRS140 GB, CRS240 GB, CRS32000 GB, RMANConfigure the settings on all VMDK disks shared by Oracle RAC nodes as outlined in Figure 5.Figure 5.Settings for VMDK disks shared by Oracle RAC nodes 2020 Cisco and/or its affiliates. All rights reserved.Page 12 of 49

For additional information about the Multi-writer option and the configuration of shared storage, read thisVMWare knowledgebase article.Table 6 summarizes the Oracle Any-Source Multicast (ASM) disk groups for this solution that are shared by allOracle RAC nodes.Table 6.Oracle ASM disk groupsOracle ASM disk groupPurposeStripe sizeCapacityDATA-DGOracle database disk group4 MB1000 GBREDO-DGOracle database redo group4 MB210 GBCRS-DGOracle RAC Cluster-Ready Servicedisk group4 MB120 GBFRA-DGOracle Fast Recovery Area diskgroup4 MB160 GBOracle Database configurationThis section describes the Oracle Database configuration for this solution. Table 7 summarizes the configurationdetails.Table 7.Oracle Database configurationSettingsConfigurationSGA TARGET50 GBPGA AGGREGATE TARGET30 GBData files placementASM and DATA DGLog files placementASM and REDO DGRedo log size30 GBRedo log block size4 KBDatabase block8 KB 2020 Cisco and/or its affiliates. All rights reserved.Page 13 of 49

Network configurationThe Cisco HyperFlex network topology consists of redundant Ethernet links for all components to provide thehighly available network infrastructure that is required for an Oracle Database environment. No single point offailure exists at the network layer. The converged network interfaces provide high data throughput whilereducing the number of network switch ports. Figure 6 shows the network topology for this environment.Figure 6.Network topologyStorage configurationFor most deployments, a single datastore for the cluster is sufficient, resulting in fewer objects that need to bemanaged. The Cisco HyperFlex HX Data Platform is a distributed file system that is not vulnerable to many ofthe problems that face traditional systems that require data locality. A VMDK disk does not have to fit within theavailable storage of the physical node that hosts it. If the cluster has enough space to hold the configurednumber of copies of the data, the VMDK disk will fit because the HX Data Platform presents a single pool ofcapacity that spans all the hyperconverged nodes in the cluster. Similarly, moving a virtual machine to adifferent node in the cluster is a host migration; the data itself is not moved. 2020 Cisco and/or its affiliates. All rights reserved.Page 14 of 49

In some cases, however, additional datastores may be beneficial. For example, an administrator may want tocreate an additional HX Data Platform datastore for logical separation. Because performance metrics can befiltered to the data-store level, isolation of workloads or virtual machines may be desired. The datastore is thinlyprovisioned on the cluster. However, the maximum datastore size is set during datastore creation and can beused to keep a workload, a set of virtual machines, or end users from running out of disk space on the entirecluster and thus affecting other virtual machines. In such scenarios, the recommended approach is to provisionthe entire virtual machine, including all its virtual disks, in the same datastore and to use multiple datastores toseparate virtual machines instead of provisioning virtual machines with virtual disks spanning multipledatastores.Another good use for additional datastores is to assist in throughput and latency in high-performance Oracledeployments. If the cumulative IOPS of all the virtual machines on a VMware ESX host surpasses 10,000 IOPS,the system may begin to reach that queue depth. In ESXTOP, you should monitor the Active Commands andCommands counters, under Physical Disk NFS Volume. Dividing the virtual machines into multiple datastorescan relieve the bottleneck. The default value for ESX queue depth per datastore on a Cisco HyperFlex system is1024.Another place at which insufficient queue depth may result in higher latency is the Small Computer SystemInterface (SCSI) controller. Often, the queue depth settings of virtual disks are overlooked, resulting inperformance degradation, particularly in high-I/O workloads. Applications such as Oracle Database tend toperform a lot of simultaneous I/O operations, resulting in virtual machine driver queue depth settings insufficientto sustain the heavy I/O processing (the default setting is 64 for PVSCSI). Hence, the recommended approachis to change the default queue depth setting to a higher value (up to 254), as suggested in thisVMware knowledgebase article.For large-scale and high-I/O databases, you always should use multiple virtual disks and distribute those virtualdisks across multiple SCSI controller adapters rather than assigning all of them to a single SCSI controller. Thisapproach helps ensure that the guest virtual machine accesses multiple virtual SCSI controllers (four SCSIcontrollers maximum per guest virtual machine), thus enabling greater concurrency using the multiple queuesavailable for the SCSI controllers.Paravirtual SCSI (PVSCSI) queue depths settingsLarge-scale workloads with intensive I/O patterns require adapter queue depths greater than the PVSCSIdefault values. Current PVSCSI queue depth default values are 64 (for devices) and 254 (for adapters). You canincrease the PVSCSI queue depths to 254 (for devices) and 1024 (for adapters) in a Microsoft Windows orLinux virtual machine.The following parameters were configured in the design discussed in this document: vmw pvscsi.cmd per lun 254 vmw pvscsi.ring pages 32For additional information about PVSCSI queue depth settings, read this VMware knowledgebase article. 2020 Cisco and/or its affiliates. All rights reserved.Page 15 of 49

Engineering validationThe performance, functions, and reliability of this solution were validated while running Oracle Database in aCisco HyperFlex environment. The Oracle SLOB and Swingbench test kit was used to create and test an onlinetransaction processing (OLTP)-equivalent database workload.Performance testingThis section describes the results that were observed during the testing of this solution.The test includes:

Oracle RAC 19c Database on Cisco HyperFlex systems 5 . (PCIe) bus, they eliminate the latency of disk controllers and the CPU cycles needed to process SAS and SATA protocols. Without a disk controller to insulate the CPU . Software-as-a-Service (SaaS) management platform that can support all your clusters—from the cloud .