Transcription
SUN SPARC ENTERPRISE M-SERIESSERVERSCONFIGURATION CONCEPTSJames Hsieh, Sun Systems GroupSun BluePrints OnlinePart No 820-7132-10Revision 1.0, 12/17/08
Sun Microsystems, Inc.Table of ContentsIntroduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1Basic Configuration Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2Types of Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Overview of Configurable Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4CPUs and System Boards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Memory and Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7I/O Units. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Important Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Physical System Boards (PSBs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10The eXtended System Board (XSB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Understanding Connectivity Between Components . . . . . . . . . . . . . . . . . . . . . . . 13Basic Configuration Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Adding Components to the Sun SPARC Enterprise M4000 and M5000 Servers . . . 15Adding Components to the Sun SPARC Enterprise M8000 and M9000 Servers . . . 22Advanced Configuration Concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Quad-XSB Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Quad-XSB Mode and IOUs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Dynamic System Domain Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 27Domain Configurations and RAS Concerns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Advanced Memory Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Memory Configuration Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Adding DIMMs with Different Capacities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Memory Mirroring Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35I/O Performance Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36PCIe Slot Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36PCIe Relaxed Ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Software Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38XSCF Units. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38XSCFU Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Email Alerts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39XCP Software Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Solaris OS Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Required Solaris Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Related Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
1IntroductionSun Microsystems, Inc.Chapter 1IntroductionThe Sun SPARC Enterprise M-series server line is the newest addition to SunMicrosystem’s mid-range and high-end server families. Designed for the enterpriseclass datacenter, all members of the server line offer a great deal of configurationflexibility in addition to high performance and reliability. The configuration flexibilityoffered by these platforms provides multiple deployment options that require carefulconsideration. This Sun Blueprints article covers basic and advanced configurationconcepts and goes through several sample configurations for the Sun SPARC Enterprise M4000, M5000, M8000, and M9000 servers. This article should help readersavoid common configuration mistakes. Please note, the entry-level Sun SPARCEnterprise M3000 server is not covered in this article.This document is not intended to serve as a complete reference for configuring andoperating the Sun SPARC Enterprise M-series servers and should not be considered as areplacement for the official product documentation located at docs.sun.com. ThisSun Blueprints article should be used as a supplement to explain the key concepts andhelp navigate through the official product documentation.The intended audiences for this document are end-users and system resellers that needto understand how to best configure the highly flexible Sun SPARC Enterprise serverline. The reader should be familiar with basic computing principles and computerhardware at the component level such as CPUs, memory, and I/O busses.
2Basic Configuration ConceptsSun Microsystems, Inc.Chapter 2Basic Configuration ConceptsThis chapter presents the fundamental concepts for properly configuring the Sun SPARCEnterprise M-series servers. The base configurations of the systems are defined alongwith several examples describing the installation of additional components such asCPU and memory. An overview of the major configurable components is presented aswell.Types of SystemsWhile all of the systems in the Sun SPARC Enterprise M-series server line share acommon architecture, they can be grouped into two sets, the rackmount systems andthe systems that occupy one or more full-sized cabinets. The Sun SPARC EnterpriseM4000 server is 6 rack units (RU) high. It can have a maximum of four CPUs. The SunSPARC Enterprise M5000 server is 10 RU high, and can have a maximum of eight CPUs.These servers are very similar and share many common components. Table 1 provides asummary of the rackmount systems.Table 1. Rackmount systems summary.DescriptionSizeSun SPARCSun SPARCEnterprise M4000 Enterprise M5000ServerServer6 RU10 RU488/1616/32Maximum CPUM(CPU modules, 2 CPUs each)24Total Dual In-line Memory Modules (DIMMs)sockets326432/64/128 GB64/128/256 GBMaximum MEMBs(memory boards, 8 DIMM sockets each)48Total DIMM sockets3264Maximum I/O Units (IOUs)12Maximum PCI Express (PCIe) slots4 per IOU (without external expansion)48Maximum PCI Extended (PCI-X) slots1 per IOU (without external expansion)12Maximum internal 2.5-inch serial attachedSCSI (SAS) disks(2 per IOU)24Physical System Boards (PSBs) equivalents12Total CPU socketsMaximum CPU cores (dual-core/quad-core)Maximum memory (1/2/4 GB DIMMs)
3Basic Configuration ConceptsSun Microsystems, Inc.Sun SPARCSun SPARCEnterprise M4000 Enterprise M5000ServerServerDescriptionMaximum Dynamic System Domains24eXtended Service Control Facility Units(XSCFU)11Power Supply Units (PSUs)24The Sun SPARC Enterprise M8000 and M9000 servers are housed in one or more fullrack-sized cabinets. These systems share many common components with each other,but not with the rackmount systems. Table 2 lists the details for the full cabinet-sizedsystems.Table 2. Full cabinet-sized systems.Sun SPARCEnterpriseM8000ServerSun SPARCEnterpriseM9000-32ServerSun SPARCEnterpriseM9000-64ServerNumber of cabinets112Maximum CPU sockets16326432/6464/128128/256128256512512 GB1 TB2 TBMaximum CMUs4816Maximum IOUs4816Maximum PCIe slots3264128Maximum 2.5-inch SAS internaldisks(2 x 2 per IOU)163264Maximum physical system boards4816Maximum Dynamic SystemDomains162424eXtended Service Control Facility Uunits222DescriptionMaximum CPU cores (dual/quad)Maximum DIMMsMaximum memory (4 GB DIMMs)Given the differences between the system types, the discussions in this article aregrouped into separate sections for the rackmount or cabinet-sized systems. Moredetails on the Sun SPARC Enterprise servers can be found on the Sun Web site at:sun.com/sparcenterprise.
4Basic Configuration ConceptsSun Microsystems, Inc.Overview of Configurable ComponentsThe primary components that are typically considered when configuring a system areCPUs, memory, and I/O. While the underlying technology and many of theconfiguration rules are the same across all of the servers in this line, there aredifferences in the physical packaging of the components between the rackmount SunSPARC Enterprise M4000 and M5000 servers and the larger full cabinet-sized Sun SPARCEnterprise M8000 and M9000 servers.CPUs and System BoardsThe Sun SPARC Enterprise M-series servers utilize SPARC64 processors. Two processorsare available, the SPARC64 VI dual-core CPU, which provides two processing cores in asingle physical CPU chip, and the SPARC64 VII, which provides four cores. Throughoutthis blueprint article the term CPU refers to the physical chip, regardless of how manyprocessing cores it contains.CPUs for the Sun SPARC Enterprise M4000 and M5000 servers are built into a modulecalled a CPUM, that contains two physical CPUs as illustrated in Figure 1. The CPUMsplug into slots on the motherboard.Figure 1. CPUM for the Sun SPARC Enterprise M4000 or M5000 server.The Sun SPARC Enterprise M4000 server’s motherboard accepts up to two CPUMs,which allows either two or four CPUs. See Figure 2 for a picture of the Sun SPARCEnterprise M4000 server’s motherboard. Figure 3 shows a block diagram of a fullypopulated system.
5Basic Configuration ConceptsSun Microsystems, Inc.-%-"#/.%#4/23-%-" #05- #05- #05#/.%#4/23Figure 2. Sun SPARC Enterprise M4000 server motherboard.MEMB 0(8 DIMMS)CPU 0DVDSAS Disk 0Opt. DATSAS Disk 1PCI-X Slot 0PCle Slot 1PCle Slot 2CPUM 0MEMB 1(8 DIMMS)CPU 1PCle Slot 3PCle Slot 4Uni-XSB 00-0(PSB 0 and IOU 0)MEMB 2(8 DIMMS)CPU 2CPUM 1CPU 3MEMB 3(8 DIMMS)M5000 MBUM5000 IOU 0Figure 3. Block diagram of a fully populated Sun SPARC Enterprise M4000 server.The Sun SPARC Enterprise M5000 server includes sockets for up to four CPUMs, for amaximum configuration of eight CPUs, providing up to 16 processor cores whenutilizing dual-core SPARC64 VI CPUs or 32 cores when using SPARC64 VII CPUs. While thesystem has a single motherboard, it is the logical equivalent of two Sun SPARC
6Basic Configuration ConceptsSun Microsystems, Inc.Enterprise M4000 system boards with up to two CPUMs each. The distinction isimportant for understanding configuration rules that are covered in the next section.See Figure 4 for a block diagram of a fully populated Sun SPARC Enterprise M5000server.MEMB 0(8 DIMMS)CPU 0DVDSAS Disk 0Opt. DATSAS Disk 1PCI-X Slot 0PCle Slot 1PCle Slot 2CPUM 0MEMB 1(8 DIMMS)CPU 1PCle Slot 3PCle Slot 4Uni-XSB 00-0(PSB 0 and IOU 0)MEMB 2(8 DIMMS)CPU 2CPUM 1CPU 3MEMB 3(8 DIMMS)CPU 4MEMB 4(8 DIMMS)M5000 IOU 0SAS Disk 2PCI-X Slot 0PCle Slot 1SAS Disk 3PCle Slot 2CPUM 3CPU 5MEMB 5(8 DIMMS)CPU 6MEMB 6(8 DIMMS)PCle Slot 3Uni-XSB 01-0(PSB 1 and IOU 1)PCle Slot 4CPUM 4MEMB 7(8 DIMMS)CPU 7M5000 MBUM5000 IOU 1Figure 4. Block diagram of a fully populated Sun SPARC Enterprise M5000 server.CPUs for the full cabinet-sized systems are installed on a board that contains both CPUsand memory, which is referred to as a CPU and Memory Unit (CMU). Either two or fourCPUs can be installed on a CMU board. The Sun SPARC Enterprise M8000 server allowsup to 4 CMU boards to be installed for a maximum configuration of 16 physical CPUs.The single-cabinet Sun SPARC Enterprise M9000-32 server can hold up to 8 CMU boards,while the dual-cabinet Sun SPARC Enterprise M9000-64 server doubles that number to16 CMUs for a maximum of 64 CPUs. The CMU boards are hot-swappable, which isimportant for reliability, availability, and serviceability (RAS). Figure 5 shows a pictureof a CMU board.
7Basic Configuration Concepts 8 )--3Sun Microsystems, Inc. #05 3OCKETSFigure 5. CMU for Sun SPARC Enterprise M8000 or M9000 servers.Memory and Memory ModulesSimilar to the way the Sun SPARC Enterprise M4000 and M5000 servers use easilyinstalled modules for CPUs, an assembly referred to as an MEMB (memory board),shown below in Figure 6, is used to house the memory. The Sun SPARC EnterpriseM4000 system board, shown above in Figure 2, has slots for up to four MEMBs. The SunSPARC Enterprise M5000 server has slots for up to eight MEMBs. Each MEMB slot has arelationship to a particular CPU. There are a number of rules for MEMB placement thatare explained later in this article.Figure 6. Picture of a Sun SPARC Enterprise M4000 or M5000 server MEMB.
8Basic Configuration ConceptsSun Microsystems, Inc.Each MEMB has sockets for up to eight Dual In-line Memory Modules (DIMMs) that arearranged in two groups of four. All four sockets in a group must be populated usingidentical DIMMs. Therefore, there are only two valid DIMM configurations, halfpopulated with four DIMMs in the first group (Group A) and fully populated with alleight DIMMs completing both groups (Groups A and B).The DIMMs for the Sun SPARC Enterprise M8000 and M9000 servers are installeddirectly on the CPU and memory boards (CMUs). Each CMU has 32 DIMM socketsarranged in two groups of 16 DIMMs each. Each group must be completely populatedwith identical DIMMs. Therefore, there are only two valid configurations, 16 DIMMs inthe first group (Group A) or 32 DIMMs completing both groups (Group A and B). One GB,2 GB, or 4 GB DIMMs can be utilized. Generally, the same capacity DIMMs need to beused. There are a few specific cases where different sized DIMMs can be installed.Additional information on memory configuration is covered in “Advanced MemoryConfiguration” on page 32.I/O UnitsI/O is added to the SPARC Enterprise M-series servers through assemblies called I/OUnits (IOUs) that contain a number of slots for PCI Express (PCIe) cards. The Sun SPARCEnterprise M4000 server has one IOU that provides four PCIe slots and one PCI Extended(PCI-X) slot. The IOU also has several on-board devices that don’t take up any slots: two1 gigabit Ethernet ports, a controller for two Serial Attached SCSI (SAS) 2.5-inch diskdrives, and a serial ATA (SATA) controller for the DVD and optional DAT drive. The drivesare installed in the system chassis, but the IOU is needed to provide connectivity to therest of the system. The Sun SPARC Enterprise M5000 server can house up to two IOUs.The second IOU is required for connectivity to the third and fourth built-in disk drives.There is a fixed relationship between the IOU and specific CPU/memory modules. Thefirst IOU is associated with CPUM 0 and 1, and MEMB 0 through MEMB 3. The secondIOU is associated with CPUM 2 and 3, and MEMB 4 through MEMB 7. The configurationrules are covered in detail later in this article.The Sun SPARC Enterprise M8000 and M9000 server IOUs (see Figure 7) have eight PCIeslots and can hold up to four 2.5-inch SAS disks in a 2 x 2 configuration. Unlike the IOUsin the Sun SPARC Enterprise M4000 and M5000 servers, there are no on-boardcontrollers. The IOU Adapter card (IOUA) provides two 1 gigabit Ethernet ports and aSAS controller for up to two drives. IOUA card can be placed in any of the evennumbered PCIe slots. However, an IOUA is required in slot 0 in order to utilize disks 0and 1, and slot 4 for disks 2 and 3. The system chassis contains a DVD drive and optionalDAT drive. An IOU with an IOUA is required to access these peripherals.
9Basic Configuration ConceptsSun Microsystems, Inc. 8 ISK "AYS 8 0#)E 3LOTSFigure 7. IOU for a Sun SPARC Enterprise M8000 or M9000 server.The number of slots available for IOUs in the Sun SPARC Enterprise M8000, M9000-32,and M9000-64 servers are the same as the number of slots for CMUs: 4, 8, or 16. Thereis a fixed relationship between the CMU slot and the IOU slot. An IOU cannot beutilized by the system unless there is CPU and memory installed on the CMU in thecorresponding slot. However, the IOU is not required for CPU and memory to operate.An external I/O expansion box is available for use with all of the Sun SPARC EnterpriseM-series servers that can provide additional PCIe or PCI-X slots. The external I/Oexpansion box requires that a PCIe link card be installed in the appropriate IOU forconnectivity. Details on the external I/O expansion box are beyond the scope of thisSun BluePrint. More information can be found at:sun.com/servers/midrange/external io.Important ConceptsConfiguring servers is most often done by thinking in terms of individual numbers ofCPUs, amounts of memory, and I/O slots. The flexibility, performance, and reliabilityaspects of the Sun SPARC Enterprise M-series servers require a more holistic view inthinking about CPU, memory, and I/O together as a system. In these servers, resourcesare grouped into physical system boards. There are fixed relationships between CPUsand DIMMs on the physical system board. Additionally, there are fixed relationships
10Basic Configuration ConceptsSun Microsystems, Inc.between IOUs and system boards. The eXtended System Board (XSB) comprised of CPU,memory, and optionally I/O, is the fundamental configuration unit for the Sun SPARCEnterprise M-series systems. Many configuration mistakes can be avoided by thinkingin terms of XSBs.Physical System Boards (PSBs)The Sun SPARC Enterprise M-series servers use a concept of a Physical System Board(PSB) that consists of up to four CPUs and 32 DIMMs. The Sun SPARC Enterprise M8000and M9000 server CMU board is perhaps the clearest example of a PSB. The Sun SPARCEnterprise M4000 server’s motherboard represents one physical system board. The SunSPARC Enterprise M5000 server has the equivalent of two PSBs even though it only hasone motherboard. While the packaging of the PSB is different across these systems, thecapacity for components, and therefore, the configuration rules, are very similar.The PSB also contains two or four System Controller (SC) Application Specific IntegratedCircuits (ASICs) that are the communication hubs of the system. The SCs connect CPUs,memory controllers, and I/O systems. On the Sun SPARC Enterprise M8000 and M9000servers, the four System Controllers also provide connectivity to other system boards byconnecting to the cross-bar switch that provides a high bandwidth and low latency databackbone. Figure 8 shows a block diagram of a Sun SPARC Enterprise M4000 server IMMsMemoryBoardMAC38DIMMsSC1Figure 8. Block diagram of a Sun SPARC Enterprise M4000 system PSB.The eXtended System Board (XSB)The eXtended System Board (XSB) is the combination of a CPU, memory, and I/O whenavailable. For example, on a Sun SPARC Enterprise M8000 server, the combination ofCMU 0 and IOU 0 make up an XSB. The XSB is the primary unit used in configurationand other system operations. Multiple XSBs can be combined to build domains thathave more than four CPUs. The XSB is a very important concept for understanding howto configure the Sun SPARC Enterprise M-series servers. It is best to think of XSBs andnot individual components. A fundamental rule is that every XSB must have both CPUand memory. An XSB that only has CPU or only has memory cannot be used by thesystem. An IOU cannot be used unless both its associated CPU and memory are
11Basic Configuration ConceptsSun Microsystems, Inc.present, thus forming a complete XSB. See Figure 9 for an example of a complete XSBfor a Sun SPARC Enterprise M8000 or M9000 server. An example showing a Sun SPARCEnterprise M5000 server with two complete XSBs can be found in Figure 4 on page 6.The XSB can be set to one of two logical configurations, Uni-XSB or Quad-XSB mode.Uni-XSB mode uses all of the resources of the entire PSB and IOU as a single unit.Uni-XSB mode is illustrated in Figure 9.SAS Disk 0CPU 0SAS Disk 18 DIMMSPCIe Slot 0IOUA CardPCIe Slot 1PCIe Slot 28 DIMMSCPU 1PCIe Slot 3Uni-XSB 00SAS Disk 28 DIMMSCPU 2SAS Disk 3PCIe Slot 4IOUA CardPCIe Slot 5PCIe Slot 68 DIMMSCPU 3PCIe Slot 7CMUIOUFigure 9. A Sun SPARC Enterprise M8000 or M9000 server CMU and IOU in Uni-XSBmode.Quad-XSB mode divides the available resources into quarters to make four XSBs withone CPU and four or eight DIMMs each. This allows finer-grained configuration choices.If an IOU is present, it is part of the same XSB and operates in the same mode, Uni- orQuad-XSB. For example, a Sun SPARC Enterprise M8000 or M9000 server IOU provideseach quad in Quad-XSB mode with two PCIe slots, as shown in Figure 10.
12Basic Configuration ConceptsSun Microsystems, Inc.SAS Disk 0CPU 08 DIMMSQuad-XSB 00-0SAS Disk 1PCIe Slot 0IOUA CardPCIe Slot 1PCIe Slot 28 DIMMSCPU 1Quad-XSB 00-1PCIe Slot 3SAS Disk 28 DIMMSCPU 2Quad-XSB 00-2SAS Disk 3PCIe Slot 4IOUA CardPCIe Slot 58 DIMMSCPU 3PCIe Slot 6Quad-XSB 00-3PCIe Slot 7IOUCMUFigure 10. A Sun SPARC Enterprise M8000 or M9000 server CMU and IOU inQuad-XSB mode.The IOUs used in the Sun SPARC Enterprise M4000 and M5000 servers do not haveenough resources to distribute among all four parts of the Quad-XSB. Instead the I/Oresources are allocated to the first two quads as shown in Figure 11. The third and forthQuad-XSBs do not have any I/O. Note that I/O is not a requirement for an XSB, onlyCPU and memory are required. However, in order to boot and be functional, those XSBswould have to be combined with at least one other XSB that does have I/O. The sectionon “Dynamic System Domain Configuration Examples” on page 27 provides additionaldetails on utilizing XSBs in domains.MEMB 0(8 DIMMS)CPU 0Quad-XSB 00-0DVDSAS Disk 0Opt. DATSAS Disk 1PCle Slot 1CPUM 0MEMB 1(8 DIMMS)Quad-XSB 00-1CPU 2MEMB 2(8 DIMMS)Quad-XSB 00-2MEMB 3(8 DIMMS)Quad-XSB 00-3PCle Slot 4CPUM 1M4000 MBUPCle Slot 2PCle Slot 3CPU 1CPU 3PCI-X Slot 0M4000 IOUFigure 11. Sun SPARC Enterprise M4000 server in Quad-XSB mode.
13Basic Configuration ConceptsSun Microsystems, Inc.The flexibility of using Quad-XSBs for fine-grained allocation of resources comes withthe price of additional configuration rules that increase installation complexity.For example, there are configurations that are valid in Uni-XSB mode, but are not validfor Quad-XSB mode because they lack CPU or memory. This chapter only providesguidance on Uni-XSB configurations. Quad-XSBs are discussed in more detail in thesection “Quad-XSB Configurations” on page 24.Understanding Connectivity Between ComponentsMany of the configuration rules for the Sun SPARC Enterprise M-series systems aredetermined by the way components are dependent on each other. Invalidconfigurations occur when components are isolated with no directly usable path.Therefore, it is particularly important to understand the connections within an XSB.CPU and MemoryCPUs and memory are connected by the SC on their PSBs as illustrated in Figure 8 onpage 10. A CPU can only operate with memory that is on the same XSB. A CPU on anXSB with no memory cannot be used. Similarly, memory on an XSB without a CPUcannot be used. In Uni-XSB mode, a CPU can utilize all of the memory on the sameboard.In Quad-XSB mode the resources are partitioned into quarters. The SC is responsible forthe partitioning. This mode creates an even tighter relationship between CPUs andmemory. For example, on a Sun SPARC Enterprise M4000 server, the first CPU onCPUM 0 only has access to the memory on MEMB 0. Similarly, the second CPU onCPUM 0 only has access to MEMB 1. If MEMB 1 isn’t present the second CPU doesn’thave memory, and is therefore not a valid configuration for Quad-XSB mode (Figure 12).This topic is discussed in further detail in “Adding Components to the Sun SPARCEnterprise M4000 and M5000 Servers” on page 15 and “Quad-XSB Configurations” onpage 24.
14Basic Configuration ConceptsMEMB 0(8 DIMMS)CPU 0Sun Microsystems, Inc.Quad-XSB 00-0DVDSAS DISK 0Opt. DATSAS DISK 1PCI-X Slot 0PCle Slot 1PCle Slot 2CPUM 0MEMB Slot 1EmptyCPU 1MEMB Slot 2EmptyPCle Slot 3Quad-XSB 00-1PCle Slot 4Quad-XSB 00-2CPUMSlot 1EmptyMEMB Slot 3EmptyQuad-XSB 00-3M4000 MBUM4000 IOUFigure 12. Invalid Quad-XSB configuration for a Sun SPARC Enterprise M4000server: CPU 1 cannot be used.I/O Units and Physical System BoardsEach IOU is tied to a specific PSB by its position in the system. The System Controllersprovide the connections between the CPU, memory, and the IOU. An IOU can only beutilized by the system if there is CPU and memory in the associated slots. On the SunSPARC Enterprise M8000 and M9000 servers, there is a direct relationship betweenCMU slot numbers and IOU slot numbers. For example, IOU 3 is connected to CMU 3.The Sun SPARC Enterprise M5000 server can have two IOUs. This system has theequivalent of two PSBs, even though they are packaged into a single motherboard.PSB 0 is connected to the first IOU. The second IOU is connected to PSB 1. As a result,there must be CPUs and memory on PSB 1 in order to utilize the I/O provided by thesecond IOU. At a minimum, CPUM slot 2 and MEMB slot 4 must be populated in orderto utilize IOU1. See Figure 4 on page 6 for a block diagram.Internal 2.5-inch SAS Disk DrivesThe Sun SPARC Enterprise M-series servers have bays for 2.5-inch SAS disk drives thatare typically utilized as boot disks. The drives are used as mirrored pairs for higheravailability. The IOUs on the Sun SPARC Enterprise M4000 and M5000 servers providethe SAS controller for these drives. The Sun SPARC Enterprise M4000 server has roomfor two drives, while the Sun SPARC Enterprise M5000 server has room for an additionalpair of drives. The first two drives are connected to the first IOU. In order to use thesecond pair of drives on the Sun SPARC Enterprise M5000 server, the second IOU mustbe present to provide the controller. This also means that CPU and memory must be
15Basic Configuration ConceptsSun Microsystems, Inc.present on the second PSB as detailed in the section above. If the system board is usedin Quad-XSB mode, the drives are attached to the first XSB (Figure 11). Additionaldetails can be found in “Quad-XSB Configurations” on page 24.The IOUs for the Sun SPARC Enterprise M8000 and M9000 servers have four drive baysthat provide for up to two pairs of boot disks. The IOUs for these systems do not haveon-board controllers. An IOUA PCIe card needs to be installed for each pair of drives thatare used, in PCIe slot 0 for disk 0 and 1, and PCIe slot 4 for disk 2 and 3. In Quad-XSBmode, the first pair of disks are part of the first XSB (XSB 0) and the second pair are partof the third (XSB 2), as illustrated in Figure 10 on page 12.On-Board DVD and Optional DAT DrivesThe on-board DVD and optional DAT drives are connected in a similar manner to the SASdisk drives described in the previous section. On the Sun SPARC Enterprise M4000 andM5000 servers, the DVD and DAT drives have a fixed association with the first XSB onthe first PSB. The second IOU on the Sun SPARC Enterprise M5000 server does not havethe options for DVD or DAT drives. If the drives are needed by another domain, XSB 00-0must be temporarily assigned to that domain. More information is available in“Dynamic System Domain Configuration Examples” on page 27.The DVD and DAT drives on the Sun SPARC Enterprise M8000 and M9000 servers can bedynamically assigned to the XSB that requires it by using the XSCF as long as the XSBhas an IOUA card available in an even numbered PCIe slot. See Table 4 on page 26Basic Configuration ExamplesThis section covers some basic configuration examples for each of the systems. Thebase configurations as they appear on the current Sun price list are used as a startingpoint. CPU, memory, and I/O are then added to demonstrate some of the keyconfiguration rules and highlight some common configuration mistakes.In the basic examples, only Uni-XSB configurations are used to keep thingsstraightforward. In Uni-XSB mode, many of the configuration rules are somewhatrelaxed. However, there are valid Uni-XSB configurations that are not valid for QuadXSB mode. If the requirements call for XSBs that can operate in either Uni- or Quad-XSBmode, then all of the Quad-XSB configuration rules need to be adhered to as well. See“Advanced Configuration Concepts” on page 24 for more information.Adding Components to the Sun SPARC Enterprise M4000 and M5000ServersThis section provides guidelines on adding and configuring components to therackmount systems.
16Basic Configuration ConceptsSun Microsystems, Inc.Sun SPARC Enterprise M4000 and M5000 Server Base ConfigurationsThe Sun SPARC Enterprise M4000 server base configuration includes one CPUM (whichincludes two dual-core CPUs) and 8 GB of memory in two MEMBs. Each MEMB has four1 GB DIMMs installed in the first group, Group A. The modules are all installed in thelowest numbered slots, the CPUM in slot 0, the MEMBs in slots 0 and 1 (Figure 13).The base configuration for the Sun SPARC Enterprise M5000 server is the same, butthere is a second PSB that has no CPU or memory installed.MEMB 0(4 x 1 GB)CPU 0DVDSAS DISK 0Opt. DATSAS DISK 1PCI-X Slot 0PCle Slot 1PCle Slot 2CPUM 0CPU 1PCle Slot 3MEMB 1(4 x 1 GB)PCle Slot 4Uni-XSB 00MEMB Slot 2EmptyCPUMSlot 1EmptyMEMB Slot 3EmptyM4
the systems that occupy one or more full-sized cabinets. The Sun SPARC Enterprise M4000 server is 6 rack units (RU) high. It can have a maximum of four CPUs. The Sun SPARC Enterprise M5000 server is 10 RU hi gh, and can have a maximum of eight CPUs. These servers are very similar an d share many common components. Table 1 provides a
