Transcription
VSPEX ISV Solution Deployment GuideTHE LENOVO THINKSERVER – EMC VSPEXPRIVATE CLOUD VALIDATED REFERENCEARCHITECTURE DEPLOYMENT GUIDEScalable VMware vSphereTM 5.1 for Up To 100 VirtualMachinesEMC VSPEX powered by Lenovo ThinkServerAbstractThe Lenovo ThinkServer - EMC VSPEX Private Cloud solution for VMwarevSphere 5.1 supports up to 100 virtual machines with redundantserver/network topology and highly available storage. This validated andmodular reference architecture built with proven best-of-breedtechnologies is a complete virtualization solution for Private Clouddeployments.July 2013
Copyright in formation here Doc Title Doc Type Part Number Part # 2 Document Title Document Type
ContentsContentsChapter 1 Introduction7Purpose of this guide . 8Business value . 8Scope . 9Audience . 9Terminology . 9Solution Tested . 10Chapter 2 Solution Architecture14Architecture Overview . 15Logical Design View . 17Chapter 3 Solution Deployment20Deployment Overview. 21Deployment and Interoperability Prerequisites . 21Pre-Deployment Tasks . 22Solution Deployment . 22Network Layer Installation and Configuration . 22Server Installation and Configuration . 27Virtual Layer Installation and Configuration . 28Installation and Configuration . 28Storage Layer Installation and Configuration. 39Chapter 4 Acknowledgements53Acknowledgements . 54Chapter 5 References55References . 56Appendix AVLAN and IP Address Schema58VLAN and IP Address Schema . 59Appendix B Compute Card Schema60CNA Network Assignments . 61Appendix CSample Network Configuration62Sample Network Configuration . 63 Document Title Document Type 3
ContentsFabric A . 63Fabric B . 63Appendix DSample Cabling Guide65Sample Cabling Guide . 66Appendix E Sample Switch Port Labeling Guide67Sample Switch Port Labeling Guide . 68Appendix F Compute Firmware and Hardware Installation69PCIe Slot Assignment. 70Recommended Firmware Release (TBC) . 70Appendix GPower and Rack Position Planning71Power and Rack Position Planning. 724 Document Title Document Type
ContentsFiguresFigure 1.Solution Architecture Logical View . 15Figure 2.Solution Architecture Physical View . 15Figure 3.Customer Logical View . 18Figure 4.Storage Logical View . 18Figure 5.Management Logical View . 19Figure 6.Cluster/HA Logical View . 19Figure 7.Attaching Extreme Summit 670 Rack Mount Bracket . 23Figure 8.Installing the Extreme Summit x670 Series Switch . 23Figure 9.Example CLI for Port Display String Validation . 27Figure 10.Configure SP Interface Properties . 41Figure 11.Configure iSCSI Virtual Port Properties . 42Figure 12.Check for VNXe MAC Addresses . 43Figure 13.Check for Host Initiator MAC Addresses . 43Figure 14.Confirm Host and VNXe MACs . 44Figure 15.Create Storage Pool . 44Figure 16.Create Storage Pool . 46Figure 17.Create LUNs . 47Figure 18.Configure VMware iSCSI Software Adapter . 48Figure 19.iSCSI SW Initiator Dynamic Discovery . 48Figure 20.Configure iSCSI SW Initiator Target IP . 48Figure 21.iSCSI SW Initiator Static Discovery . 49Figure 22.iSCSI SW Initiator Static Discovery Populated . 49Figure 23.VNXe Discovered. 50Figure 24.Paths to VNXe . 50Figure 25.VNXe Discovered Initiators . 50Figure 26.VNXe Host Initiator Registration . 51Figure 27.Confirm Initiator Registration . 51Figure 28.VNXe Storage Group Configuration . 52Figure 29.Confirm VNXe Storage Group Hosts and LUNs. 52Figure 30.Power and Rack Position Planning. 72TablesTable 1.Abbreviations . 9Table 2.Hardware Summary . 11Table 3.VNXe Raid Group Layout . 46Table 4.Sample IP Address Schema . 59 Document Title Document Type 5
Contents6 Document Title Document Type Table 5.CNA Network Assignments. 61Table 6.Cabling Guide . 66Table 7.Sample Switch Port Labeling Guide . 68Table 8.PCIe Slot Assignment. 70Table 9.Recommended Firmware Release . 70
Chapter 1: IntroductionChapter 1 IntroductionThis chapter presents the following topics:Purpose of this guide . 8Business value . 8Scope . 9Audience . 9Terminology . 9Solution Tested . 10 Document Title Document Type 7
Chapter 1: IntroductionPurpose of this guideThe Lenovo ThinkServer - EMC VSPEX Private Cloud solution for VMwarevSphere 5.1 provides a system architecture capable of supporting up to100 virtual machines with a redundant server/network topology and highlyavailable storage. This validated and modular reference architecture builtwith proven best-of-breed technologies is a complete virtualization solutionenabling informed hypervisor, compute, and networking layer decisions forPrivate Cloud deployments. When embarking on server virtualization, theLenovo ThinkServer - EMC VSPEX Private Cloud Solution accelerates your ITtransformation by enabling faster deployments, greater efficiency, andlower risk.This document describes the key components, logical design anddeployment instructions for this highly available virtualized convergedsolution for business applications.Business valueBusiness applications are moving into virtualized and consolidatedcompute, network, and storage environments. The choices available tobuild such an environment are vast and often lacking proven guidelines atevery stage of the design: Vendor and component selection Component interoperability Solution Scalability Solution Performance Solution Robustness and Availability Physical and Logical System Design and ConfigurationThe Lenovo ThinkServer - EMC VSPEX Private Cloud solution using VMwareeliminates the complexity of designing and building virtualized convergedinfrastructure solutions by providing a proven and documented referencearchitecture:8 Document Title Document Type Trusted component vendors Interoperability testing already done Assured performance and high availability Reliable, flexible, and scalable reference design Virtualization planning and configuration burdens reduced Physical cable connectivity guidelines
Chapter 1: IntroductionScopeThis document contains the key design principles and deploymentinstructions required to build a highly available virtualized convergedLenovo ThinkServer - EMC VSPEX solution for business applications. Thedocument provides: a base understanding of the design principles applied to the LenovoThinkServer - EMC VSPEX solution deployment instructions to build the Lenovo ThinkServer – EMC VSPEXsolutionThis document does not provide the guiding principles to size and matchan internal or external customer’s business need to a Lenovo ThinkServer EMC VSPEX platform. It assumes that this has been completed prior todeployment as per the Lenovo ThinkServer – EMC VSPEX SolutionArchitecture Guide.AudienceThis document is intended for channel partner professional services teamswho will be deploying the Lenovo ThinkServer – EMC VSPEX platform.Expected Level of Technical Knowledge: High for devices andtechnologies described in this document.TerminologyThe abbreviations used in this document are summarized in the tablebelow.Table 1.AbbreviationsAbbreviationDescriptionACLAccess Control ListCNAConverged Network AdapterDRSDistributed Resource Scheduling within VMwareHAHigh AvailabilityiBMCIntegrated Baseboard Management ControlleriKVMIntegrated Keyboard, Video and Mouse Remote ControlIOPsPerformance metric for number of inputs or outputs persecondiSCSIInternet Small Computer Systems InterfaceLACPLink Aggregation Control Protocol Document Title Document Type 9
Chapter 1: IntroductionAbbreviationDescriptionLUNLogical Unit NumberPCIePeripheral Component Interconnect ExpressPDUPower Distribution UnitQoSQuality of ServiceSANStorage Area NetworkSASSerial Attached SCSISPService ProcessorvCPUVirtual CPUVLANVirtual LANVMVirtual MachineSolution TestedThe Lenovo ThinkServer - EMC VSPEX Private Cloud solution core elementsorganized by virtualization, compute, networking and storage layers are:Virtualization VMware vSphere 5.1 ESXi Hypervisor and VMware vCenter Server VMware vSphere High AvailabilityCompute Lenovo ThinkServer RD630 Lenovo ThinkServer RD330 (Management) QLogic 8300 Series Converged Network AdaptersNetworking Extreme Networks Summit Series x670 with ExtremeXOS TMStorage EMC VNXe Series StorageThe table below provides a summary of the hardware and itsconfiguration.10 Document Title Document Type
Chapter 1: IntroductionTable 2.Hardware SummaryHardwareConfigurationNotesCompute:3 ThinkServer RD630LenovoThinkServerCPU: 2 x socket 8 cores eachConfigured asa singlevSphere clusterOne vCPU per virtual machineFour vCPUs per physical coreMemory:2 GB RAM per virtual machine256 GB RAM across all servers2GB Ram reservation per vSphere hostNetwork:Two QLogic 8362 CNANetworkinfrastructure:ExtremeSummit Series2 Extreme Summit x670 48-portFour 10GbE ports used per vSphereServerRedundantLANconfigurationFour 10Gbe ports used for VNXeStorage:VNXe 3300 Chassis 3UVNXe SeriesTwo storage processors (active/active)iSCSI StorageFive 12 Slot 3.5” 2U Disk-ArrayEnclosuresEighty-Five (total) 300GB 15KRPM 3.5”SAS ver RD330CPU: 2 x socket 4 cores eachTwo vCPU per virtual machineMemory:64 GB RAM8GB Ram reservation per vSphere hostNetwork:Motherboard GbE interfaceVirtualization VMware vSphere 5.1 ESXiHypervisor and VMware vCenter Server VMware vSphere HighAvailabilityVmwareConfigured asa single ESXhost to run thefollowingInfrastructureVM’s Vcenter,SQL arevSphere 5Enterpriserecommended Document Title Document Type 11
Chapter 1: IntroductionThe guiding principles for the solution design are: FunctionGeneral purpose business applications that run in a virtualenvironment. UsabilitySimplified debug, maintenance, upgrade and monitoring with predetermined network, storage and compute configurations, allvalidated and documented with procedures. PerformanceTargets a general-purpose workload of up to 100 virtual machineswith a typical workload. Example workloads would be medium-tolight database, web application or mail server processing. Resilience / ReliabilityThis design is intended to provide N 1 redundancy throughout thesystem to recover from typical single points of failure at each layer ofthe system. Compute, network and storage all provide fail over andfail back of services without impacting business applications. Ease of DeploymentChannel partners trained on the platform can deploy a system readyfor business application service enablement in days versus weeks ormonths for a similar solutions that are not pre-engineered ScalabilityDesigned from the ground up to enable pay as you grow scaling.Compute and storage layers have assigned ratios that allowcustomers to incrementally grow each tier to match new businessdemands. For example, a pre-engineered server to disk ratio isdesigned into the architecture. Manageability / MaintainabilityThe best practices to configure each component in the system aredocumented to maximize the value of the tools used to manage aconverged Infrastructure platform. Some examples are: BIOS settings,MTU size, onboard IPMI capabilities, power consumption and rackdesign. Economic and technical constraints, and tradeoffsN 1 design requires powered and connected compute, storage andnetwork infrastructure to be lightly loaded during normal businessproduction hours. This architecture includes an optional server toenable high availability at the compute layer, so if a server fails, thesystem has enough resources to maintain business operations. The12 Document Title Document Type
Chapter 1: Introductiondesign is an active/passive design aimed at availability and reducedcomplexity. Document Title Document Type 13
Chapter 2: Solution ArchitectureChapter 2 Solution ArchitectureThis chapter presents the following topics:Architecture Overview . 15Logical Design View . 1714 Document Title Document Type
Chapter 2: Solution ArchitectureArchitecture OverviewThe Lenovo ThinkServer – EMC VSPEX Validated Private Cloud Solutionoffers a scalable solution for up to 100 VMware based virtual machineswith full hardware and software redundancy.Figure 1. Solution Architecture Logical ViewFigure 2. Solution Architecture Physical View Document Title Document Type 15
Chapter 2: Solution ArchitectureThe solution architecture was designed and tested based on the goalsbelow to enable a robust compute platform for typical businessapplications without sacrificing performance or availability: Matched Compute Cores and Memory to disk spindles for typicalbusiness work loads Converged network adaptors equipped with hardware offload forstorage processing, allowing more CPU cycles for businessapplications Simple HA design (Active/Passive) N 1 Simple growth rules to increase capacity In band VMware ESX management path over redundant interfaces;other components are managed out of band (network, storage,compute hardware) Reduced cabling requirements Converged layer 2 Ethernet fabric all storage, network and highavailability traffic.Both of the storage and network communications fabrics have at least twophysical ten-gigabit Ethernet paths with one active and one passive path.Network traffic is segregated on these physical links into at least five virtualLANs (VLANs): ComputeiSCSI and TCP processing off loaded to QLogic 8300 ConvergedNetwork Adapters with hardware ASIC’s to reduce CPU utilizationand maximize compute layer CPU for business applications ManagementManagement path QOS to ensure reachability of all devices underall network conditions. StorageDedicated high bandwidth/low latency physical links exclusively forEthernet based iSCSI storage traffic. Storage network traffic isdesigned to provide access to storage at less than 3ms latency. vMotionHigh-bandwidth links for migrating and live migrating virtual machinesfrom one physical server to another User AccessBandwidth dedicated to application communication networks foruser to machine and machine to machine designated traffics. ManyUser Access VLANs are typically provisioned.16 Document Title Document Type
Chapter 2: Solution Architecture Fault ToleranceLow latency/low bandwidth/high quality network utilized forhypervisor to hypervisor health checks and resource scheduling. Thisis used with the optional VMware Fault Tolerance host to hostreplication of storage and state to a 2nd physical host.Quality of Service (QoS) is applied to each shared VLAN to predictablymanage the behavior of these networks and to ensure shared linksprioritize application and storage traffic as needed.With this VMware based solution, common infrastructure components arevirtualized and hosted on a separate dedicated management server:VMware vSphere management server, Microsoft Windows 2012 ActiveDirectory, DHCP server, DNS server and Microsoft Windows SQL Server2008R2 Standard.Lenovo RD630 enterprise class servers are deployed in an N 1configuration to provide full failover capabilities for all hosted virtualapplication servers. A minimum of one physical core is allocated for everyfour virtual CPUs (vCPU), with one vCPU to each virtualized applicationserver. This gives a final ratio of no more than four virtualized applicationservers for each physical CPU core. At least two gigabytes of PhysicalRAM is reserved for each virtual application.Overall the solution is designed to be resilient to multiple simultaneousfailures at the network, computing, storage and power layer.Best practices are documented throughout this document. However,advanced design elements are highlighted but not described in detail, asthe goal for the solution is reduced complexity for install, support andmanagement of the system.Logical Design ViewOnce a system is physically connected, it is often difficult to predict how orwhere data will go without a logical component layer view to highlight therelevant design elements. The Lenovo ThinkServer – EMC VSPEX ValidatedPrivate Cloud Solution is designed to use an active/passive failure plane tosimplify the configuration, debugging, deployment and dependencies.This logical view of the design is an important tool to simplify solutiondebugging and for reliably predicting impacts of typical maintenanceactivities. For example, the effect of rebooting an Ethernet switch must beunderstood in advance to avoid any disruption to business applicationsthat are running in a live environment. The logical view for Customeraccess, Management access, Storage access and Cluster/HA design areprovided below for reference. Document Title Document Type 17
Chapter 2: Solution ArchitectureFigure 3. Customer Logical ViewCustomer Basic Design: Active Path and Passive PathFigure 4. Storage Logical ViewStorage Basic Design: 18 Document Title Document Type Active Path and Passive Path based on Hypervisor Multi-Pathing
Chapter 2: Solution ArchitectureFigure 5. Management Logical ViewManagement Basic Design: All devices have connection to management network withdedicated management port out of bandFigure 6. Cluster/HA Logical ViewCluster/HA Basic Design: Active trunk LACP L2 or L2 Load Share Document Title Document Type 19
Chapter 3: Solution DeploymentChapter 3 Solution DeploymentThis chapter presents the following topics:Deployment Overview. 21Deployment and Interoperability Prerequisites . 21Pre-Deployment Tasks . 22Solution Deployment . 2220 Document Title Document Type
Chapter 3: Solution DeploymentDeployment OverviewThe deployment process has been designed to simplify system turn up at acustomer site. The high level steps are based on how the system was builtand tested and provide checkpoints throughout the process to ensurethat the system is deployed and ready for business applications oncecomplete. This document also serves as documentation repository for keysystem variables that are required to operate the system and to provide tocustomers.Steps to Deployment:1.Review all planning checklists in the Appendices2.Complete Physical Rack installation based on Appendix G3.Complete Network Layer installation and configuration4.Complete Server installation and configuration5.Complete Virtualization Layer installation and configuration6.Complete Storage Layer installation and configuration7.Add Storage to Virtualization Layer8.Validate Solution Deploymenta.Guest VM to disk traffic testb.Guest VM to network traffic testc.Guest VM to Guest VM traffic testd.Compute to Compute traffic testFollowing these steps will reduce the time to install and debug all pathsbetween components and will ensure that the building blocks are built inthe correct order such that traffic tests can be successfully completed andand the solution can be handed off to the customer.Deployment and Interoperability PrerequisitesAll components were tested with a basic configuration that ensures thehighest level of interoperability. For example, VLAN tagging was used toseparate traffic and vswitches. It is not expected that software specificdependencies exist in the design and it is therefore recommended toupdate all components to the latest versions available before going intoproduction. Each section outlines when software or firmware should beupdated before moving to the next building block. Document Title Document Type 21
Chapter 3: Solution DeploymentPre-Deployment TasksIt is important to ensure all sections for planning are complete beforedeployment begins. Key checklists that should be reviewed before startinga deployment are listed below. All checklists are found in the Appendices.1.Power and Rack position planning2.IP Address and VLAN planning3.Server CNA and slot planning4.Software version check on all components with required softwareversions downloaded and availableIt is highly recommended to keep a record of actual deployment datathat can be provided to the customer as part of the deployment handoffprocess. The planning tables should be used as a reference for the datathat needs to be documented.Solution DeploymentThis section describes the deployment steps for the Lenovo ThinkServer –EMC VSPEX Solution. The deployment must follow the order as described inthis section.Network LayerInstallation andConfigurationThis section describes the installation and configuration steps for theNetwork Layer of the Lenovo ThinkServer – EMC VSPEX Solution.Pre-Installation RequirementsThe following additional tools and equipment should be on-hand prior tostarting the Network Layer Installation and Configuration:22 Document Title Document Type #1 Phillips screwdriver Rack mounting screws 4x AC Power Cables Extreme management console cable Twin-Axial or SFP /Fiber Cables for system connectivity Complete and review actual IP Address and VLAN table fordeployment
Chapter 3: Solution DeploymentNetwork Layer Installation and Configuration StepsPhysical Switch Installation1.Identify the switch space designation within the target rack (refer toAppendix G for rack layout plan).2.Attach each mounting bracket to the switch housing with thescrews.Figure 7.Attaching Extreme Summit 670 Rack Mount Bracket3.Slide the switch into the rack and secure the mounting brackets intothe rack.4.Repeat steps 1 - 3 for the second switch.Figure 8.Installing the Extreme Summit x670 Series Switch Document Title Document Type 23
Chapter 3: Solution DeploymentPower Cables1.Power is supplied to all equipment in this design by redundant PDUswith each PDU supplied from a discrete and independent source.2.Attach power cords from the separate PDUs to both powersupplies located at the back of the switch.3.Make sure the cables are neatly stowed to the side away from fanintakes and exhaust.4.Repeat steps 1 - 3 for the second switch.SFPs and CablingPassive electrical twin-axial cables at 1M and 3M are recommended for allinterconnects although a mix of Extreme SFP and fibre optic cable can beused as an alternative.1.Review suggested cabling configuration included in Appendix Dand proceed with connecting equipment2.Installing passive electrical twin-axial cable:3.a.Holding the connector by its sides, insert the connector into theport on the switch.b.Push the connector into the port until you hear it click intoplace.Installing SFP and fibre optic cable combination:4.a.Seat SFP modules into designated ports on the switch.b.Connect SFPs via a fibre optic cable of appropriate length toensure tidy deployment.Complete cabling for both switches.Switch Initial configuration24 Document Title Document Type 1.Use the Summit Family Switches Hardware Installation Guide tocomplete the initial management access section for the Summitx670.2.Use the ExtremeXOS Concepts Guide Software Version 15.1 toconfigure: Switch Name SNMP Information NTP3.Save the configuration.4.Repeat steps 1 - 3 for the second switch.
Chapter 3: Solution DeploymentUpgrade Firmware1.Download image to secondary partition.Example:# download image 172.16.41.1 summitX-12.6.1.3.xos vr "VRDefault" secondary2.Set secondary partition as the boot image.# use image secondary3.Reboot extreme switch to restart with new installed firmware.# reboot4.Verify the firmware version.# show version5.Repeat steps 1 - 4 for the second switch.Configure Jumbo FramesJumbo frames are used between endstations that support larger framesizes for more efficient transfers of bulk data. Both endstations involved inthe transfer must be capable of supporting jumbo frames. Jumbo framesare disabled by default. The default size setting is 9216 which is themaximum size that can be set and is the required setting for this solution.Jumbo frames must be enabled on ports that are used in this deployment.1.Login into the switch CLI interface.2.Configure preferred jumbo frame size of 9216.configure jumbo-frame-size 92163.Enable jumbo frames on ports that are used in this solution:Example: Enable jumbo frames on a particular portenable jumbo-frame ports 5Example: Enable jumbo frames on multiple ports:enable jumbo-frame ports 5,6,104.Repeat steps 1 - 3 for the second switch.Configure Sharing1.Login into switch ClI.2.To configure sharing add ports using the following syntax:configure sharing master port add ports port list Example:configure sharing 3:9 add ports 3:133.Repeat steps 1 - 2 for the second switch. Document Title Document Type 25
Chapter 3: Solution DeploymentConfigure QoS QueuesQuality of Service (QoS) allows traffic groups higher priority access tonetwork resources in the event of network contention. Bandwidth can bereserved for special groups
The Lenovo ThinkServer - EMC VSPEX Private Cloud solution for VMware vSphere 5.1 supports up to 100 virtual machines with redundant server/network topology and highly available storage. This validated and modular reference architecture built with proven best-of-breed
