WIXLIB

Experiences with implementing OracleDatabase with Oracle Extended RAC 11gRelease 2 (11.2.0.2) on IBM AIX 6.1.Francisco RiccioIBM PeruMay 31, 2012In collaboration with the IBM Oracle International Competency Center Copyright IBM Corporation, 2012. All Rights Reserved.All trademarks or registered trademarks mentioned herein are the property of their respective holders.

Table of contentsAbstract. 1Introduction . 11. Architecture . 2NFS Server . 6NTP Server . 72. Configuration & Installing Oracle Grid Infrastructure 11.2.0.2 . 93. Configurating & Installing Oracle Database . 28Summary. 33Resources. 34About the author. 34Trademarks and special notices . 35Experiences with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation

AbstractThis paper is based on our experiences implementing an Oracle Extended Real Application Clusterssolution with Oracle Database 11.2.0.2 on an IBM AIX 6.1 platform. It is intended for IBM fieldsupport, IBM Business Partners and IBM customers planning to implement this offering from Oracle.IntroductionOracle Extended RAC is an architecture where the nodes in the cluster are separated by different DataCenter. An Oracle Extended RAC is an architecture that provides extremely fast recovery from a sitefailure and allows for all nodes, at all sites, to actively process transactions as part of single databasecluster.For example, for a business that has a corporate campus, the Oracle Extended RAC configuration couldconsist of individual Oracle Real Application Clusters nodes located in separate buildings. Oracle RealApplication Clusters on an extended cluster provides greater availability than a local Oracle RealApplication Cluster due to the fact that if we have a disaster in a Data Center our solution can continueoperating.We recommend reading the following document to understand all the concepts and architectures that areinvolved in an implementation of Oracle Extended periences with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation1

1. ArchitectureOur implementation had the following architecture:Figure 1: General architectureEach server must have two network cards, one for the public IP and the other for the private IP (known asinterconnect). Each IP must work in a separate VLAN or a separate device network. The private IP mustensure a communication speed of at least 1 Gbit. Each node pings at an interval of between three msecto three sec to each node of the cluster through the private network. If a node pinge is not acknowledgedwithin 30 seconds, the unresponsive node is forcibly removed from the cluster (evicted).Also we recommend setting up in each LPAR at least two core of POWER7 (in my case an IBM Power750 server). If the server does not have enough CPU power then an eviction action may be initiated.Oracle does not recommend having more than one LPAR on the same physical environment as aproduction node. If you need the processing power try to use a bigger LPAR per physical system.We recommended review the following es with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation2

Figure 2 shows the architecture more detail.Figure 2: Communication between switchesThere should always be redundancy of communications between each device. If this is not possible, itpresents a risk. In my implementation I have used Port Channel Cisco’s technology for providingredundancy.Experiences with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation3

Mirrroring Storage – Host Based Mirroring (Active/Active Storage)VIOSIBM Datacenter–La MolinaVIOSIBM Datacenter–San IsidroDS 5300Array - RAID 5DS 5300Array – RAID 5Replication by Oracle ASM – Normal RedundancyFigure 3: Configuration of storage between sitesNote: The VIOS shown in Figure 3 above means that the SAN storage is attached to the Host BaseAdapter, which is owned by VIOS and the SAN LUNs are presented to the AIX LPARs through VirtualSCSI adapters.We can use two kinds of mirroring of storage, Host Based Mirroring (Active/Active Storage) and ArrayBased Mirroring (Active/Failover Storage). In this test case I implemented with Host Based Mirroring foreach pair of disks (one at each data center). Normal redundancy is provided through Oracle AutomaticStorage Management (Oracle ASM)When using Host Based Mirroring we should not have both storage separated by more than 100 Km.Also, if the distance is greater than ten kilometers, we should use dark fiber.For each two Km of separation we recommend creating one buffer credit. The buffer credit allows datacommunication in a Fibre Channel storage area network (SAN) where there are long spans of fiber opticcable.Experiences with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation4

Figure 4: Storage configuration at each siteIt is very important that we can establish redundancy between the SAN switches, HBA’s and the path’s tothe storage as shown in Figure 4.Also, it is important to determine the transfer of data (throughput) sent to each device to identify potentialbottlenecks in the general architecture.DNS ServerIn Oracle Real Application Clusters (RAC) 11g Release 2, we need a DNS server that can resolve theSingle Client Access Name (SCAN) of the cluster. The SCAN provides a single name to the clientsconnecting to Oracle Real Application Clusters. It does not change throughout the life of the cluster, evenif you add or remove nodes from the cluster. Clients connecting with SCAN can use a simple connectionstring, such as a thin JDBC URL or EZConnect, and still achieve the load balancing and client connectionfailover. If we want to use all features of SCAN, we must use Oracle client 11g Release 2.It is important to highlight that the network administrator must create a single name that resolves to threeseparate IP addresses using round-robin algorithms.In the test implementation we used a DNS server running on Microsoft Windows 2008 R2 .Experiences with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation5

Figure 5Note: In our test case, the DNS server is a Microsoft DNS and the clients are on AIX. With thisconfiguration it is necessary in the DNS to register in the Forward Lookup Zones section and in theReverse Lookup Zones section.Also the server DNS must have some mechanisms to ensure high availability. In Microsoft DNS, we canhave high availability through replication DNS (Active/Active) or zone transfers (Active/Passive - thereplication is one way). For our testing we used the zone transfer method.Also our DNS clients (Oracle Database Server – AIX) must register both DNS servers as primary andsecondary, respectively.NFS ServerIn an extended cluster we need a NFS server where each server on every site can mount a folder. In thisfolder a voting file will exist to host the quorum disk. This voting file will be a witness between both sites.The quorum voting file allows greater latency.We must remember that if a node can’t read its local voting file (it makes pings to the voting file everyinterval of three ms to three sec), it then takes a timeout of 200 seconds by default when I use OracleClusterware and 600 seconds may at the most be the CSS timeout when third party cluster software isExperiences with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation6

used; if this time is completed and the server can’t read the voting file so the servers get into an evictedstate and then the clusterware reboots the server.The recommendation is to have the NFS server in a third data center or else it is risk due to the fact aproblem such as unavailable data center, where the NFS Server reside; all our solution of our cluster willnot be working. Two of three voting files always must be available to be read for each node in the cluster.Always each node in the cluster must able to read 50 percent of our total of voting files created.In our test implementation, each node has a folder mounted to the NFS Server.Example:In the file /etc/filesystems of each node:The following options for the mount command should be used for IBM AIX:cio,rw,bg,hard,intr,rsize 32768,wsize 32768,timeo 600,vers 3,proto tcp,noac,sec sysIn the following document, we can see the correct mount parameters that the clusterware needs for fs-131158.pdfNote 1: The document linked to above does not show use of the “cio” parameter for an AIX environment,but if it is omitted it will give an error condition.Note 2: The directory in the AIX NFS client must have the user “oracle” as owner before and after it ismounted with the published NFS server.NTP ServerIn previous releases of Oracle Real Application Clusters an NTP server was required, but with OracleGrid Infrastructure 11g Release 2 an NTP server is no longer required. Oracle Grid Infrastructure 11gRelease 2 automatically provides this service for the nodes that belong to the cluster solution. Thisservice is now named “Oracle Cluster Time Synchronization Service” and is provided through the “ctssd”daemon.If we want to use the NTP of our company, we must execute the following steps:a) Open the /etc/rc.tcpip file, and locate the following line: start /usr/sbin/xntpd " src running"b) Change the line to the following: start /usr/sbin/xntpd " src running" "-xc) Save the fileThe option –x insures that the NTP clients never back off time. In previous releases of Oracle RealApplication Clusters this event caused a reboot of the node.Experiences with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation7

In the test implementation, we used the Oracle Cluster Time Synchronization Service because the set upis automatic.Experiences with implementing Oracle Database Extended RAC 11.2.0.2 on IBM AIXhttp://www.ibm.com/support/techdocs Copyright 2012, IBM Corporation8