Transcription
Third party information broughtto you courtesy of Dell.ConnectX Dual-Port InfiniBand Mezzanine I/Ocards for Dell PowerEdge M1000e-series BladeServers User ManualP/N: ConnectX MDI, MCQH29-XCCRev 1.3Mellanox Technologies
2 Copyright 2009. Mellanox Technologies, Inc. All Rights Reserved.Mellanox Technologies, InfiniHost, and ConnectX are registered trademarks for Mellanox Technologies, Inc.Dell and the Dell logo are trademarks of Dell Inc. All other marks and names mentioned herein may be trademarks oftheir respective companies.ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade ServersDocument Number:2850Mellanox Technologies, Inc.350 Oakmead Parkway Suite 100Sunnyvale, CA 94085U.S.A.www.mellanox.comTel: (408) 970-3400Fax: (408) 970-3403Mellanox Technologies LtdPO Box 586 Hermon BuildingYokneam 20692IsraelTel: 972-4-909-7200Fax: 972-4-959-3245Rev 1.3Mellanox Technologies
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual3Table of ContentsRevision History5About this Manual6Chapter 1 Overview71.1Chapter 2Mezzanine I/O card7I/O Card Installation82.12.2Hardware and Software RequirementsInstallation Instructions882.2.1 Safety Warnings8Chapter 3 Driver Software and Firmware3.13.2Drivers and Clustering SoftwareUpdating the Mezzanine I/O card FirmwareChapter 4 Mezzanine I/O card Interfaces4.14.24.34.410101011InfiniBand InterfacePCI Express InterfaceFlash Jumper .51111121314System MemoryFlashConnectX MDI EEPROM VPDMCQH29-XCC EEPROM VPDFRU (Field Replacement Unit) ConnectX MDI and MCQH29-XCC EEPROMAppendix A Specifications15A.1 Board Mechanical Drawing and DimensionsA.2 EMC Certification StatementsA.3 Specifications151618Appendix B Avertissements de sécurité d’installation (French)19Appendix C Installation - Sicherheitshinweise (German)21Mellanox TechnologiesRev 1.3
4List of TablesTable 1:Revision History Table5Table 2:Documents List6Table 3:ConnectX InfiniBand Mezzanine I/O card details7Table 4:Hardware and Software Requirements8Table 5:MDI VPD12Table 6:MCQH29-XCC VPD13Table 7:ConnectX MDI Specifications18Table 8:ConnectX MCQH29-XCC Specifications18Rev 1.3Mellanox Technologies
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual5Revision HistoryThis document was first printed on 12/21/07.Table 1 - Revision History TableDateRevComments/ChangesMar. 20091.3Added Dell LogoRemoved WatermarkFeb. 20091.2Changed IO Card to mezzanine I/O cardReplaced pic in Table 3Dec. 20081.1Added MCQH29-XXC QDR InfiniBand mezzanine I/O cardFixed linksDec. 20071.0Initial ReleaseMellanox TechnologiesRev 1.3
6About this ManualThis User’s Manual describes ConnectX Dual Port InfiniBand mezzanine I/O cards for Dell PowerEdge M1000eseries Blade Servers.It provides details as to the interfaces of the board, specifications, required software and firmware for operating thecards, and relevant documentation.Intended AudienceThis manual is intended for the installer and user of the mezzanine I/O cards listed in “Overview” on page 7.The manual assumes basic familiarity with the Infiniband architecture specifications.Related DocumentationTable 2 - Documents ListInfiniBand Architecture Specification Volume 1 Release 1.2 and Volume 2 release 1.2.1– Infiniband Architecture Specifications DescriptionsPCI Express Base 2.0 Specification (1.1 compatible)PCI Local Bus Specification Rev 2.3Online Resources Mellanox Technologies Web pages: http://www.mellanox.comDell Support Web pages: http://support.dell.comDocument ConventionsorThese symbols indicate a situation, status, or condition that may cause harm to peopleor damage to the equipment.When discussing memory sizes, MB and MBytes are used in this document to mean size in mega bytes. The use ofMb or Mbits (small b) indicates size in mega bits.Rev 1.3Mellanox Technologies
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual71OverviewThis document is a User’s Manual for the Mellanox ConnectX 20 and 40Gb/s InfiniBand dual port Mezzanine I/Ocards for Dell PowerEdge M1000e-series Blade Servers. The cards described in this manual (see Table 3, on page 7)have the following main features: 1μs MPI ping latency20 or 40Gb/s InfiniBand portsCPU offload of transport operationsEnd-to-end QoS and congestion controlHardware-based I/O virtualization1.1 Mezzanine I/O cardTable 3- ConnectX InfiniBand Mezzanine I/O card detailsOrdering PartNumber (OPN)InfinibandLink SpeedRoHSComplianceConnectXMCQH29-XCC40 Gb/sQDRRoHS-R5(with exemption)ConnectX MDI20 Gb/sDDRRoHS-R5(with exemption)Mezzanine I/O card PhotoMellanox TechnologiesRev 1.3
82 I/O Card Installation2.1 Hardware and Software RequirementsBefore installing the InfiniBand Mezzanine I/O card, please make sure that the system meets the hardware and software requirements listed in Table 4.Table 4 - Hardware and Software RequirementsRequirementHardwareDescriptionUsed with Dell PowerEdge M1000e-seriesSoftware OperatingSystems/DistributionsSoftware StacksRefer to the PowerEdge M1000e-series ManualsMellanox OpenFabric software package (either MLNX OFED for Linux orMLNX WinOF for Microsoft Windows)2.2 Installation InstructionsFollow the instructions in the Installation Guide for ConnectX InfiniBand Mezzanine I/O Adapter Card for DellPowerEdge M1000e-series Blade Servers.2.2.1 Safety Warnings1.Installation InstructionsRead all installation instructions before connecting the equipment to the power source.2.Over TemperatureThis equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: 40 C (104 F).To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation openings.3.Stacking the ChassisThe chassis should not be stacked on any other equipment. If the chassis falls, it cancause bodily injury and equipment damage.Rev 1.3Mellanox Technologies
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual94.Lightening – Electrical HazardDuring periods of lightning activity, do not work on the equipment.5.Rack Mounting and ServicingWhen this product is mounted or serviced in a rack, special precautions must be takento ensure that the system remains stable. In general you should fill the rack with equipment starting from the bottom to the top.6.Installation of EquipmentThis equipment should be installed, replaced, or serviced only by trained and qualifiedpersonnel.7.Disposal of EquipmentDisposal of this equipment should be in accordance to all national laws and regulations.8.Compliance with Local and National CodesThis equipment should be installed in compliance with local and national electricalcodes.Mellanox TechnologiesRev 1.3
103Driver Software and Firmware3.1 Drivers and Clustering SoftwareDownload and install on all nodes of the Mellanox OpenFabric software package for Linux, Windows, or other operating systems from the Mellanox software website. Go to www.mellanox.com Products InfiniBand SW/drivers.This software package provides server drivers enabling connectivity for server and storage systems utilizing HighPerformance Computing (HPC) or enterprise data center (EDC) applications across an InfiniBand fabric. It also provides a Subnet Manager for simple network configuration and network administration and diagnostic tools for network management.3.2 Updating the Mezzanine I/O card FirmwareThe Mezzanine I/O card is shipped with the latest version of qualified firmware at the time of manufacturing. Newfirmware versions will be posted on the Mellanox web site. Go to http://www.mellanox.com Support Dell.You will need the Mellanox Firmware Tools package available in the Mellanox OpenFabrics software package, toupdate firmware for this switch. It can also be downloaded from the Management Tools section from the MellanoxDownload web site. In the directory that holds the latest firmware, run the following commands:lspci grep Mellanox ;identifies PCI ID to be used in next commandmstflint -d PCI ID, for example 05:00.0 -i .bin file bRev 1.3Mellanox Technologies
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual114Mezzanine I/O card InterfacesThe Mezzanine I/O card attaches to the blade using a press feed connector which connects both the InfiniBand andPCI Express interfaces.4.1 InfiniBand InterfaceThe ConnectX (MT25408) device is compliant with the InfiniBand Architecture Specification, Release 1.2. It hastwo compliant 4X InfiniBand ports, ports 1 and 2, each having four Tx/Rx pairs of SerDes. The Mezzanine I/O cardbased on this device provides access to these ports through a board to board press feed connector.4.2 PCI Express InterfaceThe Mezzanine I/O card has eight Tx/Rx pairs of SerDes providing for a PCI Express x8 interface, version 2.0 compliant and compatible with base 1.1 chipsets. The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations.4.3 Flash Jumper ConfigurationThere is a jumper on each Mezzanine I/O card that indicates to the device whether an on-board Flash device exists (oris to be used). See Figure 1 on page 15 for the jumper location.4.4 MemoryThe Mezzanine I/O card supports multiple memory devices through the PCI Express, Flash, and I2C-compatibleinterfaces.4.4.1 System MemoryThe Mezzanine I/O card utilizes the PCI Express interface to store and access InfiniBand fabric connection information on the system memory.4.4.2 FlashThe Mezzanine I/O card includes two 2MB SPI Flash devices (P/N M25P16-VME6G by ST Microelectronics) accessible via the Flash interface of the MT25408 ConnectX device.Mellanox TechnologiesRev 1.3
124.4.3 ConnectX MDI EEPROM VPDEach board incorporates an EEPROM that is accessible through the I2C-compatible interface. The EEPROM is usedfor storing the Vital Product Data (VPD) and FRU. The VPD format adheres to the PCI Local Bus specification rev2.3 VPD definition. The EEPROM capacity is 512 bytes.Table 5 - MDI VPDOffset(Decimal)ItemValue0Large Resource Type ID String Tag (0x02)0x821Length0x73Data“IB Mezz”10Large Resource Type VPD-R Tag (0x10)0x9011Length0x3F13VPD Keyword“PN”15Length0x616Data“0H288M”22VPD Keyword“EC”24Length0x325Data"X01"28VPD MDSSSS”according to the board label51VPD Keyword“V0”Misc Information53Length0x1254Data"20G I/O card"72VPD Keyword“RV”74Length0x175DataChecksum76Large Resource Type VPD-W Tag (0x11)0x9177Length0xB079VPD Keyword“V1”81Length0x682Data“N/A”88VPD umericShort description / IDRead Only AreaNumbersAdd in Card Part NumberAlphanumericEngineering Change Level of thecard (rev)AlphanumericSerial NumberPCB revisionRead / Write AreaDriver versionNumberAsset Tag91Data“N/A”123VPD Keyword“RW”125Length0x81126DataReserved (0x00)255Small Resource Type END Tag (0x11)0x78256Mellanox Read Only Mask0x0.0Numbers335Mellanox Read/Write Mask0x1.1Numbers511Mellanox Read Only Mask0x0NumbersRev 1.3Mellanox TechnologiesAlphanumericRemaining read/write area
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual134.4.4 MCQH29-XCC EEPROM VPDTable 6- MCQH29-XCC VPDOffset(Decimal)ItemValue0Large Resource Type ID String Tag (0x02)0x821Length0xB3Data“IB QDR Mezz”14Large Resource Type VPD-R Tag (0x10)0x9015Length0x4317VPD Keyword“PN”19Length0x620Data“0Y773M”26VPD Keyword“EC”28Length0x329Data”A00”32VPD MDSSSS”according to the board label55VPD Keyword“V0”Misc Information57Length0x1658Data"40G IO card"80VPD Keyword“RV”82Length0x183DataChecksum84Large Resource Type VPD-W Tag (0x11)0x9185Length0xA887VPD Keyword“V1”89Length0x690Data“N/A”96VPD Keyword“YA”98Length0x2099Data“N/A”131VPD Keyword“RW”133Length0x79134DataReserved (0x00)255Small Resource Type END Tag (0x11)0x78256Mellanox Read Only Mask0x0.0Numbers343Mellanox Read/Write Mask0x1.1Numbers511Mellanox Read Only Mask0x0NumbersMellanox TechnologiesFormatDescriptionAlphanumericShort description / IDRead Only AreaNumbersAdd in Card Part NumberAlphanumericEngineering Change Level of thecard (rev)AlphanumericSerial NumberPCB revisionRead / Write AreaDriver versionNumberAsset TagAlphanumericRemaining read/write areaRev 1.3
144.4.5 FRU (Field Replacement Unit) ConnectX MDI and MCQH29-XCC EEPROMThe FRU is used by chassis management for identifying the Mezzanine I/O card. Each board incorporates anEEPROM that is accessible through the I2C-compatible interface. The EEPROM is used for storing the Vital ProductData (VPD) and FRU.Rev 1.3Mellanox Technologies
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual15Appendix A: SpecificationsA.1 Board Mechanical Drawing and DimensionsThe ConnectX InfiniBand Mezzanine I/O card mechanical drawing is depicted in Figure 1.Note: All dimensions are in millimeters.Figure 1: ConnectX Mezzanine I/O card6.5517.5D1D2D3D435.8983.99 15.4017.56.5588.00J1 is the I2C Connector.Mellanox TechnologiesRev 1.3
16A.2 EMC Certification StatementsA.2.1 FCC Statements (USA)Class A Statements:§ 15.21StatementWarning! Changes or modifications to this equipment not expressly approved by the party responsible for compliance (Mellanox Technologies) could void the user's authority to operate the equipment.§15.105(a)StatementNOTE: This equipment has been tested and found to comply with the limits for a Class A digital device,pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protectionagainst harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operationof this equipment in a residential area is likely to cause harmful interference in which case the user willbe required to correct the interference at his own expense.A.2.2 EN Statements (Europe)EN55022 Class A Statement:WarningThis is a class A product. In a domestic environment this product may cause radiointerference in which case the user may be required to take adequate measures.A.2.3 ICES Statements (Canada)Class A Statement:“This Class A digital apparatus complies with Canadian ICES-003.Cet appareil numérique de la classe A est conforme à la norme NMB-003 du Canada.”A.2.4 VCCI Statements (Japan)Class A Statement:Rev 1.3Mellanox Technologies
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual17(Translation - "This is a Class A product based on the standard of the Voluntary Control Council forInterference by Information Technology Equipment (VCCI). If this equipment is used in a domesticenvironment, radio interference may occur, in which case the user may be required to take correctiveactions.")A.2.5 MIC Notice (Republic of Korea Only)The MIC label may be located separately from the other regulatory markings applied to your product.Class A devices are for business purposes.Class A DeviceMIC Class A Regulatory LabelIf the regulatory label includes the following marking, your device is a Class A product:Mellanox TechnologiesRev 1.3
18A.3 Specifications.Table 7 - ConnectX MDI SpecificationsPhysicalPower and EnvironmentalSize:88.00mm X 83.99 mmVoltage:Typ Power:Maximum Power:Temperature:Protocol SupportInfiniBand:QoS:RDMA Support:Data Rate:PCI Express12V, 3.3V11.05W13.09W0 C to 40 CRegulatoryIBTA v1.2, Auto-Negotiation 10 or20Gb/s per port8 InfiniBand Virtual Lanes for eachportYes, All PortsDDRBase 2.0 compliant, backwards compatible with 1.1Safety:EMC (Emissions):US/Canada: cTUVusEU: IEC60950International: CBUSA: FCC, Class ACanada: ICES, Class AEU: CE Mark (EN55022 Class A,EN50024, EN61000-3-2, EN61000-3-3)Japan: VCCI, Class AKorea: MIC Class AAustralia/ New Zealand: C-Tick Class ATable 8 - ConnectX MCQH29-XCC SpecificationsPhysicalPower and EnvironmentalSize:88.00mm X 83.99 mmVoltage:Typ Power:Maximum Power:Temperature:Protocol SupportInfiniBand:QoS:RDMA Support:Data Rate:PCI ExpressRev 1.312V, 3.3V12.23W14.32W0 C to 40 CRegulatoryIBTA v1.2, Auto-Negotiation 10, 20, or40Gb/s per port8 InfiniBand Virtual Lanes for eachportYes, All PortsQDRBase 2.0 compliant, backwards compatible with 1.1Safety:EMC (Emissions):Mellanox TechnologiesUS/Canada: cTUVusEU: IEC60950International: CBUSA: FCC, Class ACanada: ICES, Class AEU: CE Mark (EN55022 Class A,EN50024, EN61000-3-2, EN61000-3-3)Japan: VCCI, Class AKorea: MIC Class AAustralia/ New Zealand: C-Tick Class A
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual19Appendix A: Avertissements de sécuritéd’installation (French)1.Instructions d’installationLisez toutes les instructions d’installation avant de brancher le matériel à la sourced’alimentation électrique.2.Température excessiveCe matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 40 C (104 F). Un flux d’air de 200LFM à cettetempérature ambiante maximale est nécessaire. En outre, pour garantir un bonécoulement de l’air, laissez au moins 8 cm (3 pouces) d’espace libre autour des ouvertures de ventilation.3.Empilage du châssisLe châssis ne doit pas être empilé sur un autre matériel. Si le châssis tombe, il peut provoquer des blessures corporelles et des dégradations de biens.4.Orages – dangers électriquesPendant un orage, il ne faut pas utiliser le matériel.5.Montage et entretien sur baieLorsque ce produit est monté ou entretenu sur baie, il faut prendre des précautionsspéciales pour s’assurer que le système reste stable. En général, il faut remplir la baieavec du matériel de bas en haut.6.Installation du matérielCe matériel ne doit être installé, remplacé ou entretenu que par du personnel formé etqualifié.7.Elimination du matérielL’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations etréglementations nationales en vigueur.Mellanox TechnologiesRev 1.3
208.Codes électriques locaux et nationauxCe matériel doit être installé dans le respect des codes électriques locaux et nationaux.Rev 1.3Mellanox Technologies
ConnectX Dual-Port InfiniBand Mezzanine I/O cards for Dell PowerEdge M1000e-series Blade Servers User Manual21Appendix A: Installation - genLesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die s Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über dermaximal empfohlenen Temperatur von 40 C (104 F) betrieben werden. Außerdemsollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um eineneinwandfreien Luftstrom zu gewährleisten.3.Stapeln des ChassisDas Chassis sollte nicht auf andere Geräte gestapelt werden. Wenn das Chassis herunterfällt, kann es zu Verletzungen und Beschädigungen an Geräten führen.4.Bei Gewitter - Elektrische GefahrArbeiten Sie während eines Gewitters und Blitzschlag nicht am Gerät.5.Rack-Montage und WartungWenn dieses Produkt in einem Rack montiert oder gewartet wird, sind besondere Vorsichtsmaßnahmen zu ergreifen, um die Stabilität des Systems zu gewährleisten. ImAllgemeinen sollten Sie das Gestell von unten nach oben mit Geräten füllen.6.GeräteinstallationDiese Gerät sollte nur von geschultem und qualifiziertem Personal installiert, ausgetauscht oder gewartet werden.7.GeräteentsorgungDie Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen.Mellanox TechnologiesRev 1.3
228.Regionale und nationale elektrische BestimmungenDieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischenBestimmungen installiert werden.Rev 1.3Mellanox Technologies
Rev 1.3 Mellanox Technologies 6 About this Manual This User's Manual describes ConnectX Dual Port InfiniBand mezzanine I/O cards for Dell PowerEdge M1000e- series Blade Servers. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the
