Transcription
UCS Networking Deep DiveBRKCOM-2003Michael Ciesla, Customer Support Engineer#clmel
Agenda UCS Overview Chassis Connectivity Server Connectivity Fabric Forwarding TopologiesBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
Cisco Unified Computing System (UCS)Single Point of ManagementLogical Building BlocksBRKCOM-2003Hardware/Software(Service Profiles)4Cisco Public Abstraction 2015 Cisco and/or its affiliates. All rights reserved.
UCS ComponentsLANMGMTSANFabric InterconnectUCS ChassisHeartbeat link (No Data)BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public5
UCS ComponentsLANMGMTSANFabric InterconnectUCS ChassisHeartbeat link (No Data)IO ModuleBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public6
UCS ComponentsLANMGMTSANFabric Interconnect4 KR lanes to each halfwidth blade slotUCS ChassisHeartbeat link (No Data)IO ModuleBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public7
UCS ComponentsLANFabric InterconnectMGMTSAN4 KR lanes to each halfwidth blade slotUCS ChassisHeartbeat link (No Data)IO ModuleBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public8
UCS ComponentsFabric InterconnectCiscoVICIO ModuleHeartbeat link (No Data)UCS BladeBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public9
UCS 6248 Hardware DiagramCarmel 1Carmel 2Carmel 3SouthBridgePCIe x8FlashSerial12 GigIntelJasper ForestMemoryNVRAMPEX 85254 port PCIESwitch12 GigCarmelCPUUnified Crossbar FabricSunnyvale12 Gig12 GigCarmel 4Carmel 5Carmel 610 GigBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public10PCIe x401PCIEDual GigDDR3 x2Carmel 1Carmel 2Carmel 6Carmel cpuSunnyvale10 GigPCIe x4PCIe x4PCIEDual Gig01Xcon1MgmtXcon2ConsolePCIEDual Gig01N/C
Cisco UCS 6200 Series Fabric InterconnectsFlexibilityProduct Features and SpecsUCS 6248UPUCS 6296UP960 Gbps1.92 Tbps1RU2RU1 Gigabit Ethernet Port Density489610 Gigabit Ethernet Port Density48968G Native FC Port Density4896Port-to-Port Latency2.0us2.0usActive # of VLANs20002000Switch Fabric ThroughputSwitch FootprintScalabilityMultipurposeBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS Mini: 6324 Fabric Interconnect UCS BIO Modules 6248 or 6296 FabricFabric InterconnectsUCS 5108 ChassisSupports existing and future blades UCS Mini6324 Fabric InterconnectUCS 5108 ChassisSupports existing and future bladesBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
TAC Tip: Carmel ASIC Port MappingBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
Chassis Connectivity Options
“Californian Octopus”BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS Fabric TopologiesChassis Bandwidth Options2208XPonly2x 1 Link2x 2 Link2x 4 Link2x 8 Links20 Gbps per Chassis40 Gbps per Chassis80 Gbps per Chassis160 Gbps per ChassisBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public16
UCS 2200 IO Module (FEX) UCS-IOM-2204XP UCS-IOM-2208XP 40G to the Network 80G to the Network 160G to the Hosts 320G to the Hosts– 2x10G Half width slot– 4x10G Full width slotBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.– 4x10G Half width slot– 8x10G Full width slotCisco Public
220x-XP ArchitectureFabric Ports to deWoodsideFabric Ports(NIF)48Host Ports(HIF)1632CoS88Latency 500ns ide ASICIOSwitch2204ChassisSignals2208Internal backplane ports to bladesNo Local Switching – ever!Traffic goes up to FIBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public18
NIF/HIF Interfaces Output from “show fex detail” of the NXOS shellFI ports connectingto the FEXBackplane portsconnecting toserver 1/3Link between the FIand IOM/FEXwhich the serverwill be usingAdditional interface?BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS Internal Block DiagramUCS 6248UCS 6248FabricInterconnects16x SFP 16x SFP Expansion Module16x SFP 16x SFP Expansion ModuleFabric Uplinks (NIFs)IO Modules2208XP2208XPBackplane Ports (HIFs)MidplaneAdaptermLOMMezzx8 Gen 3CPU 0Server Bladex8 Gen 3CPU 1QPI LinkUCS Blade ChassisBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
CLI Block Diagram Output from “show platform software woodside sts” of the IOM(FINAL POSITION TBD)1 2 3 4 5 6 7 8 - -- -- -- -- -- -- -- - SFP:[ ][ ][ ][ ][ ][ ][ ][ ] - -- -- -- -- -- -- -- - N N N N N N N N I I I I I I I I 0 1 2 3 4 5 6 7 NI (0-7) ------------ ----------- ------------------------- ------------- ------------- --------------------------- ------------ ----------- ----------- ------------ ------------ ----------- ------------- ---------- HI (0-7) HI (8-15) HI (16-23) HI (24-31) H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I 0 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - -- -- -- -- -- -- -- - - -- -- -- -- -- -- -- - - -- -- -- -- -- -- -- - - -- -- -- -- -- -- -- - [ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ] - -- -- -- -- -- -- -- - - -- -- -- -- -- -- -- - - -- -- -- -- -- -- -- - - -- -- -- -- -- -- -- - - - - - - - - 1 11 11 11 98 76 54 32 16 54 32 10\ \ / / \ \ / /\ \ / / \ \ / /\ \ / / \ \ / /\ \ / / \ \ / RKCOM-2003Uplink #:Link status: 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public21Eth1/1/1Port # from theFEX Port
CLI Command Output from “show platform software woodside rate” of the IOMBlade 1fex-1# sh platform software woodside rate -------- ------------ ----------- ------------ ------------ ----------- ------------ ------- ------- --- Port Tx Packets Tx Rate Tx Bit Rx Packets Rx Rate Rx Bit Avg Pkt Avg Pkt (pkts/s) Rate (pkts/s) Rate (Tx) (Rx) Err -------- ------------ ----------- ------------ ------------ ----------- ------------ ------- ------- --- 0-BI 25 5 5.04Kbps 14 2 3.09Kbps 106 118 0-CI 65 13 20.20Kbps 49 9 25.44Kbps 174 304 0-NI3 2 0 3.48Kbps 63 12 12.98Kbps 1072 108 0-NI2 35 7 12.05Kbps 6 1 2.75Kbps 195 267 0-NI1 3 0 3.69Kbps 14 2 4.22Kbps 750 168 0-NI0 17 3 6.86Kbps 20 4 12.38Kbps 232 367 0-HI31 7 1 8.00Kbps 1 0 768.00 bps 695 464 0-HI30 5 1 1.28Kbps 1 0 200.00 bps 141 106 0-HI23 5 1 1.22Kbps 1 0 352.00 bps 133 200 0-HI19 13 2 2.48Kbps 1 0 1.19Kbps 99 728 0-HI11 3 0 560.00 bps 1 0 128.00 bps 98 64 0-HI7 4 0 616.00 bps 1 0 560.00 bps 77 332 -------- ------------ ----------- ------------ ------------ ----------- ------------ ------- ------- --- BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public22
Fabric Link Connectivity23
Chassis Connectivity PolicyBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public24
IO Module HIF to NIF Pinning2208XP – 1 LinkSlot 1HIF1-4HIF1-4Slot 2NIF1HIF5-8HIF5-8Slot 3HIF9-12HIF9-12Slot 4HIF13-16HIF13-16Slot 5HIF17-20HIF17-20Slot 6HIF21-24HIF21-24Slot 7HIF25-28HIF25-28Slot 8HIF29-32BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.HIF29-32Cisco Public25NIF1
IO Module HIF to NIF Pinning2208XP – 2 LinksSlot 1HIF1-4HIF1-4Slot 2NIF1HIF5-8HIF5-8Slot 3NIF2HIF9-12NIF2HIF9-12Slot 4HIF13-16HIF13-16Slot 5HIF17-20HIF17-20Slot 6HIF21-24HIF21-24Slot 7HIF25-28HIF25-28Slot 8HIF29-32BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.HIF29-32Cisco Public26NIF1
IO Module HIF to NIF Pinning2208XP – 4 LinksSlot 1HIF1-4HIF1-4Slot 2NIF1HIF5-8Slot 3NIF2NIF3HIF5-8HIF9-12NIF2HIF9-12Slot 4NIF4HIF13-16Slot 5HIF17-20Slot 6HIF21-24HIF21-24Slot 7HIF25-28HIF25-28Slot 8HIF29-32BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.HIF29-32Cisco Public27NIF3NIF4HIF13-16HIF17-20NIF1
IO Module HIF to NIF Pinning2208XP – 8 LinksSlot 1HIF1-4HIF1-4Slot 2NIF1HIF5-8Slot 3NIF2NIF3NIF4HIF9-12HIF13-16Slot 5HIF17-20HIF21-24NIF6HIF21-24NIF7NIF8HIF25-28Slot 8HIF29-32BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.HIF29-32Cisco Public28NIF4NIF5Slot 7HIF25-28NIF3HIF17-20Slot 6NIF7NIF8HIF9-12HIF13-16NIF1NIF2Slot 4NIF5NIF6HIF5-8
CLI MappingFEX Level commandInterface Level command Output from “show fex detail” of theNXOS shell Output from “show interfaceethernet mod/port fex-intf output truncated Fabric interface state:Eth1/9 - Interface Up. State: ActiveEth1/10 - Interface Up. State: ActiveEth1/11 - Interface Up. State: ActiveEth1/12 - Interface Up. State: ActiveFex PortState Fabric PortEth3/1/1UpEth1/9Eth3/1/2UpEth1/10Eth3/1/3 /6 DownNoneEth3/1/7 DownEth1/11Eth3/1/8 --Eth1/9Eth3/1/1Eth3/1/5 output truncated BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public29
IOM Link Failure ScenarioSlot 1HIF1-4HIF1-4Slot 2NIF1HIF5-8Slot 3NIF2NIF3Link FailureHIF5-8HIF9-12NIF2HIF9-12Slot 4NIF4HIF13-16Slot 5HIF17-20Slot 6HIF21-24HIF21-24Slot 7HIF25-28HIF25-28Slot 8HIF29-32BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco PublicHIF29-3230NIF3NIF4HIF13-16HIF17-20NIF1
IOM Link Failure ScenarioSlot 1HIF1-4HIF1-4Slot 2NIF1HIF5-8Slot 3NIF2NIF3HIF5-8HIF9-12NIF2HIF9-12Slot 4NIF4HIF13-16Slot 5HIF17-20Slot 6HIF21-24HIF21-24Slot 7HIF25-28HIF25-28Slot 8HIF29-32BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco PublicNIF3NIF4HIF13-16HIF17-20NIF1HIF29-32
IOM Link Failure ScenarioSlot 1HIF1-4HIF1-4Slot 2NIF1HIF5-8HIF5-8Slot 3NIF2NIF3HIF9-12NIF2HIF9-12Slot 4HIF13-16Slot 5HIF17-20Slot 6HIF21-24HIF21-24Slot 7HIF25-28HIF25-28Slot 8HIF29-32BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.HIF29-32Cisco PublicNIF3NIF4HIF13-16HIF17-20NIF1
Increased Bandwidth Access to Blades4 links, Discrete - Todayslot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8FEXFabricInterconnect8 links, Discreteslot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8FEXFabricInterconnectUp to 8 links, Port-channelFEXFabricInterconnect Available bandwidth perblade – 10Gb Available bandwidth perblade – 20Gb Available bandwidth perblade – up to 160Gb Statically pinned toindividual fabric links Statically pinned toindividual fabric links Statically pinned to Portchannel Deterministic Path Deterministic Path Increased and sharedbandwidth Guaranteed 10Gb toeach bladeBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public33 Higher Availability
Port-channel Pinning No slot based pinning No invalid link count for NIF ports (no “power of 2” rule)HIFsVIC 1200/1300 adaptor withDCE links in Port-Channel2200-IOMPinnedto PoGen-1 adaptor withsingle 10G linkBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.NIFHIFCisco Public34
Server Connectivity
Cisco Virtual Interface Cards (VIC)1st Gen2nd Gen3rd Gen M81KR, P81E 1240, 1280, 12xx 1340, 1380 128 PCIe devices 256 PCIe Device Dual 8x PCIe Gen 3 Dual 10Gb Dual 40Gb (4 x 10Gb) Native 40Gb Support 16x PCIe Gen1 16x PCIe Gen 2 VXLAN & NVGREBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public RoCE
UCS Cisco 1200/1300 VIC AdapterUCS 2208 IOMUCS 2208 IOMSide BSide AUCS 1200/1300 VIC256 PCIe devices1240SerenoBRKCOM-20031280Sereno 2015 Cisco and/or its affiliates. All rights reserved.1340CruzCisco Public371380Cruz
VIC 1240/1340 Plus Port Expander Card Base option supports dual 2x10Gb Option to enable all port of ASIC(Sereno) Fits in the Mezzanine slot of B200M3/M4 Port Expander has no PCIe presence It is a “passive connector” deviceBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public38
Connectivity IOM to Adapter2208 IOM2208 IOM Implicit Port-channel between VIC 1200/1300and UCS 2200 IOM 7-Tuple Flow based hash, 10 Gbps per flow A vNIC is active on side A or BSide BSide AUCS 1200/1300 VICvNIC1VMBRKCOM-2003VM Flows1.10 Gb FTP traffic2.10 Gb UDP traffic 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public39
VIC 1240/1340 to IOM ConnectivityMLOM onlyUCS 6248UCS 6248FabricInterconnects16x SFP 16x SFP Expansion Module16x SFP 16x SFP Expansion ModuleIO Modules2208XP2208XPMidplane Dual 2x10 Gb port-channel from VIC 1240/1340 to2208 IO ModulesAdapter1340 VICx8 Gen 3Server BladeCPU 0Emptyx8 Gen 3CPU 1QPI LinkB200 M3/M4BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.UCS Blade ChassisCisco Public40
VIC 1240/1340 to IOM ConnectivityMLOM plus Port ExpanderUCS 6248UCS 6248FabricInterconnects16x SFP 16x SFP Expansion Module16x SFP 16x SFP Expansion ModuleIO Modules2208XP2208XPPort Channel 1MidplaneAdapterPort Channel 21340 VICx8 Gen 3Server BladeCPU 0 Port Expander Passive Increase BW to 80Gbps Dual 4x10Gbps Port-channelPort Expx8 Gen 3CPU 1QPI LinkB200 M3/M4BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.UCS Blade ChassisCisco Public41
What Does The OS See?BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public42
VIC 1x40 & 1x80 to IOM ConnectivityUCS 6248UCS 6248FabricInterconnects16x SFP 16x SFP IO ModulesExpansion Module16x SFP 2208XP16x SFP Expansion Module2208XPMidplaneAdapter1340 VICx8 Gen 3CPU 0Server BladeVIC1380 Adapter Redundancy Split vNIC across adapters 4 2x10 Gb Port-channelsx8 Gen 3CPU 1QPI LinkB200 M3/M4BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.UCS Blade ChassisCisco Public43
Full Width Blade to IOM ConnectivityMLOM, Port Expander, VIC1x80UCS 6248UCS 6248FabricInterconnects16x SFP 16x SFP 16x SFP Expansion Module16x SFP Expansion ModuleIO Modules2208XP2208XPPort Channel 1Port Channel 2Midplane4x10Adapter1340 VICx8 Gen 3Server BladeCPUx8 Gen 3CPUCPUQPI Link Total BW is 160G Four 40G port-channelsVIC1380Port Expx8 Gen 34x10QPI LinkB420 M3 / B260 M4BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.UCS Blade ChassisCisco Public44
UCS Mini: Fabric to Server Connectivity Same server-side connectivityas the 2204XP IOM 20G per half width bladeBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
Topology Designs For Maximum BandwidthUCS 6248UPUCS 6248UPSide BSide A1240 or M81KR Shared IOM uplinkbandwidth of 10Gbps vNIC Burst up to 10Gbps Shared IOM uplinkbandwidth of 80Gbps vNIC Burst up to 10GbBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Side ASide BM81KRSide \B12xx / 13xx Dedicated IOM uplinkbandwidth of 10Gbps vNIC Burst up to 10GbpsCisco Public46UCS 6248UPUCS 2208 IOMUCS 2208 IOMUCS 2208 IOMUCS 2104 IOMSide AUCS 6248UPSide ASide B12xx / 13xx Shared IOM uplinkbandwidth of 80Gbps
Fabric Interconnect VIF CalculationUPCFI-A1UPC2UPC3UPC6UPC1UPC2UPC3UPC6UCS 2208XPUCS 2208XPUCS 2208XPUCS 2208XP11122334455667788122334455667788IOM-B2208 XPIOM-A2208 XPIOM-A2208 XP Recommended Not recommended Maximise number of available VIFs to thehost Minimal number of VIFs to the hostBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco PublicIOM-B2208 XP
Virtual Interfaces
Cisco UCS: Infrastructure VirtualisationSwitchport Virtualisation (vEth, vFC)Fabric InterconnectvFC1vEth1vFC2Eth 1/1Cable Virtualisation (VNTag)vEth2DCB EthernetEth 1/2IndividualEthernetsIndividualStorage(iSCSI, NFS, FC)Blade or RackService Profile # AdaptersIdentity (MAC / WWN)FirmwareSettingsCPUMEMI/OAdapterPCIePCIeServer AbstractionBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Adapter Virtualisation (NIV)Cisco Public49
Abstracting the Logical AvEth1vFC1vFC1vEth1 Dynamic, RapidProvisioning State abstraction LocationIndependence Blade or RackEth 1/1IOM AIOM ACable10GEA10GEAAdapterPhysical CablevHBA1BladeBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.vNIC1Service Profile(Server)Cisco Public50vNIC1vHBA1(Server)Virtual Cable(VN-Tag)
Fabric Extension (FEX) ConceptVirtualising the Network PortLegacy multi-tier architectureFEX architectureLANSwitchSwitch 2015 Cisco and/or its affiliates. All rights reserved.Logical SwitchSwitchBRKCOM-2003Switch port extended overFabric ExtenderLANCisco PublicFEX
Fabric Extender Evolution Cisco VN-TAG is the prestandard to IEEE 802.1BRPort ExtensionNetw orkAdministratorVN-TAG/IEEE 802.1BRFEXMany applicationsrequiremultiple interfacesLegacyBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public The 802.1BR Architectureprovides the ability to extendthe bridge (switch) interface todownstream devices
VN-TAGFEX EthernetFrameVNTAGFrameFEXVN-TAG Ethertyped plBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Publicrdestination virtual interfaceversource virtual interface
Fabric Extender Evolution – Virtual Interfaces 802.1BR associates the LogicalInterface (LIF) to a VirtualInterface (VIF)LIFNetw orkAdministratorFEXVIFAdapter FEXBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
VN-Tag at the Adapter (Mezz Card) LevelBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public55
BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public56
Fabric Extender Evolution (VM-FEX)Netw orkAdministratorVN-TAG/IEEE 802.1BR*FEX Each VM assigned dedicatedNIC on ESXi hosts PCIE bus Hypervisor arbitrates mappingbetween VM and PCI vNIC Each VM gets a dedicatedswitch port on the FabricInterconnectIEEE 802.1BR* VN-TAG/IEEE 802.1BR*HypervisorLegacyBRKCOM-2003Adapter FEX 2015 Cisco and/or its affiliates. All rights reserved.VM-FEXCisco PublicVM networkmanaged byServeradministrator
Fabric FailoverEnd Host Mode (only)SAN ALAN Fabric provides NIC failover capabilitieschosen when defining a service profileSAN B Traditionally done using NIC bondingdriver in the OSUCS FabricInterconnects Provides failover for both unicast andmulticast trafficChassisFabric Extender Works for any OS onbare metal and hypervisorsFabric ExtenderUCS-6200-A /chassis/server/adapter/host-eth-if # show vifAdapterCiMCHalf Width ----12011202CiMCHalf Width Blade 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public58Fabric ID--------ABTransport TagStatus--------- ----- ----------Ether0 AllocatedEther0 AllocatedOper State---------ActivePassive
Fabric Forwarding - Ethernet
Ethernet Fabric Forwarding Mode of OperationsLAN End-host mode (EHM): Default mode Switch mode: User configurable– No spanning-tree protocol (STP)– Fabric Interconnects behave likeregular ethernet switches– STP parameters are lock– VLAN/Mac based forwardingBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved. Active/Active for all links and VLANs– Port definitions– Policy based forwarding No unknown unicast forwardingCisco Public60
End Host ModeLANSpanningTree– Presents itself as a bunch of hosts tothe networkFI AvEth 3Fabric A Completely transparent to thenetworkMACLearning No STP – simplifies upstreamconnectivityMACLearning All uplinks ports are forwarding –never blockedvEth 1VLAN 10L2SwitchingVNIC 0VNIC 0Server 2Server 1BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public61
End Host ModeUnicast Forwarding MAC/VLAN plus policy basedforwardingLANServer 2– Server pinned to uplink portsUplink PortsDeja-Vu Policies to prevent packet loopingRPF– déjà vu check– RPF– No uplink to uplink forwardingFIvEth 1VLAN 10vEth 3VNIC 0VNIC 0Server 2Server 1BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved. No unknown unicastCisco Public62
End Host ModeMulticast Forwarding Broadcast traffic for a VLAN ispinned on exactly one uplink port (orport-channel) i.e., it is dropped whenreceived on other uplinksLANBBBroadcastListenerper VLANUplinkPorts Server to server multicast traffic islocally switchedFIvEth 1 RPF and déjà vu check also appliesfor multicast trafficvEth 3BVNIC 0VNIC 0Server 2Server 1BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public63
Switch ModeRootLAN Fabric Interconnect behaves like anormal L2 switch Rapid-STP to prevent loops– STP parameters are not configurableMACLearningvEth 3 Server vNIC traffic follows STPforwarding statesvEth 1VLAN 10– Use VPC to get around blocked ports VTP is not supportedL2SwitchingVNIC 0VNIC 0Server 2Server 1BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved. MAC address learning on bothuplinks and server linksCisco Public64
Uplink Pinning
End Host Mode - Dynamic Pinning UCSM manages the vEth pinning tothe uplinkLAN UCSM will periodically vEthdistribution and redistribute thevEths across the uplinksFI AvEth 2PinningvEth 3vEth 1VLAN 10SwitchingVNIC 0VNIC 0VNIC 0Server 2Server 3Server 1BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public66
End Host Mode – Individual UplinksDynamic Re-pinning of failed uplinksFI-ASub-second re-pinningvEth 3Fabric AvEth 1 2015 Cisco and/or its affiliates. All rights reserved.VNIC stays upVNIC 0MAC AVNIC 0BRKCOM-2003SwitchingVLAN 10L2Switching All uplinks forwarding for all VLANs GARP aided upstream convergence No STP Sub-second re-pinning No server NIC disruptionPinningCisco Public67Server 2vSwitch / N1KESX HOST 1VM 1VM 2MAC BMAC C
End Host Mode – Port Channel UplinksNo disruptionRecommended: Port Channel UplinksNo GARPsneededFI-ASub-second convergencevEth 3Fabric A More Bandwidth per Uplink Per flow uplink diversity No Server NIC disruption Fewer GARPs needed Faster bi-directional convergence Fewer moving partsRECOMMENDEDBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.PinningvEth 1SwitchingVLAN 10L2SwitchingNIC stays upVNIC 0MAC AVNIC 0Cisco Public68Server 2vSwitch / N1KESX HOST 1VM 1VM 2MAC BMAC C
End Host Mode – Static PinningLANAdministrator Pinning DefinitionFI AvEth 2vEth 3UplinkvEth 1BluevEth 2BluevEth 3PurplePinningvEth 1VLAN 10vEth Interfaces Administer controls the vEth pinningSwitching Deterministic traffic flow Pinning configuration is done under theLAN tab - LAN Pin groups andassigned under the vNICVNIC 0Server 2BRKCOM-2003VNIC 0Server 3 No re-pinning with in the same FIVNIC 0 Static and dynamic pinning can coexistServer 1 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public69
Which uplink is the servers’ vEth pinned to?TME-UCS6100-A(nxos)# sh pinning border-interfaces-------------------- --------- ---------------------------------------Border InterfaceStatusSIFs-------------------- --------- th1093 Veth1094 Veth1099Po2Activesup-eth2 Veth1103Eth1/6DownEth1/7DownEth1/8DownTotal Interfaces : 5BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public70
Fabric Forwarding - Multicast
IGMP Querier? Three Options:1. Upstream IGMP Querier / PIM Router2. Fabric Interconnect IGMP Querier3. IGMP Snooping disabledBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS MulticastIGMP Querier UpstreamIGMP Querier / PIM Router1. IGMP QuerierLANUplinkPorts3. IGMP ReportBroadcastListenerper VLANFIvEth 1vEth 32. IGMP ReportBRKCOM-2003VNIC 0VNIC 0Server 2Server 1 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
G-pinned?BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS MulticastInternal QuerierLANUplinkPortsBroadcastListenerper VLANFIIGMP QueriervEth 1vEth 31. IGMP Querier2. IGMP ReportBRKCOM-2003VNIC 0VNIC 0Server 2Server 1 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS Multicast Configuration (2.1 )BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS & Microsoft Network Load Balancing (NLB) Unicast Mode– Ethernet Switching mode only– Nexus 1000v (no mac auto-static-learn) Multicast Mode IGMP Multicast– Requires igmp querierBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
Fabric Forwarding - QoS
UCS Congestion Management8 Classes: 1 FCoE, 1 best effort, 4 user-definable, 8 Classes: 2 reserved for controlBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS QoS – Marking / ClassificationBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
Pause!LANPFC or 802.3x PauseUCS 6200PFC PauseUCS IOMPFC or802.3xPauseSide AAdapterBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco PublicSide B
ossless rubric will continue leveraging their QoS/CoS semantics to ensure reliability.below displays the differences in the format of the legacy PAUSE frame with that defined inQbb. Note how the PFC frame now has fields targeting different traffic classes.Priority Flow ControlClassical Ethernet Pause vs. Data Centre Ethernet PFC PausePriority Flow ControlTransmit QueuesEthernet LinkReceive OPPAUSESixSevenSevenEightEight Enables lossless Fabrics for each class of service PAUSE sent per virtual lane when buffers limit exceededBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public82EightVirtualLanes
PFC Pause: What does it look like?BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
UCS QoS Identifying CongestionUCSB-2-B(nxos)# show interface priority-flow-control PortMode Oper(VL bmap) RxPPPTxPPP Ethernet1/1Auto Off00Ethernet1/2Auto Off00Ethernet1/3Auto Off396483597850Ethernet1/4Auto Off467844069738Ethernet1/5Auto Off00fex-1# show platform software woodside loss ------- ------------------------------------- ------------ - ----------------------------------- --------------------------------------- frm to Port Extra --------------------------------------- RMON Drop S SS Loss Counters COS XOFF ------------ ----------- ------------ ------------ S ----------- ----------- ----------- --------------------------------------- Port Tx Pause Rx Pause Errors Counters x RX SS Tx SS SS Total 0 1 2 3 4 5 6 7 0 1 ------- ------------ ----------- ------------ ------------ - ----------- ----------- ----------- --- --- --- --- --- --- --- --- --- --- 0- NI1 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ------- ------------------------------------- ------------ - ----------- ----------- ----------- --- --- --- --- --- --- --- --- --- --- 0-HI19 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ------- ------------------------------------- ------------ - ----------- ----------- ----------- --- --- --- --- --- --- --- --- --- --- fex-1#BRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
Fabric Forwarding - Storage
SAN “End Host” NPV ModeN-Port Virtualisation ForwardingFLOGIFDISCSAN ASAN BNPIVNPIVF PortF PortVSAN 1VSAN 1N Proxy6100-A6100-BvFC 1vFC 2F ProxyBRKCOM-2003N ProxyvFC 1vFC 2F ProxyN PortvHBAvHBA01N PortvHBAvHBA01Server 1Server 2VSAN 1VSAN 1 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
SAN “End Host” NPV ModeN-Port Virtualisation Forwarding with MDS, Nexus 5000F PortChannel &TrunkSAN ASAN BNPIVNPIVF PortVSAN1,2VSAN1,2N Proxy6100-A6100-BvFC 1vFC 2vFC 1vFC 2F ProxyN PortvHBAvHBA01BRKCOM-2003vHBA0vHBA1Server 1Server 2VSAN 1VSAN 2 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
SAN FC Switch ModeDirect Attach FC & FCoE Storage to UCSFCSANFCoEOptionalMDSN PortMDSVSAN 2VSAN 1TE PortF Port6100-A FC SwitchvFC 1vFC 26100-B FC SwitchvFC 1vFC 2F PortN PortvHBAvHBA01BRKCOM-2003vHBA0vHBA1Server 1Server 2VSAN 1VSAN 2 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
Multi-Hop FCoEFCoE STORAGE Supports MDS, N5K & N7K “Unified Uplink” port typeFCoE FI in ENM Switching ModeFCoEMDS/N5K/N7KMDS/N5K/N7K– VNP port typeFCoE/EthernetUnified Uplinks FI in FC Switching ModeNPV/EHMUCS FI– VE port typeUCS FIUCS B-SeriesBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco PublicFCoE/Ethernet
Unified Appliance PortsDirect attach FCoE, iSCSI, NFS & CIFS storageStorageFCoEiSCSINFSCIFSUnified AppliancePortUCS FIUCS FIUCS B-SeriesBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco Public
FCoE with Adapter FEXVFC interface bound to 802.1BR / VN-Tag virtual ethernet interfacevfcXvfcXFabric InterconnectsSAN BSAN AvethXvethXCan be FC or FCoECan be FC or FCoEIOM802.1BR / VN-TagIOM802.1BR / VN-TagvHBA AAdapterFC1FC0OSBRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Cisco PublicvHBABBinding
Topologies92
Recommended Topology for Upstream ConnectivityAccess/Aggregation LayervPC/VSSFabric Interconnect ABRKCOM-2003 2015 Cisco and/or its affiliates. All rights reserved.Forwarding Layer 2 linksFabric Interconnect BCisco Public93
Layer 2 Disjoint TopologyProductionVLAN 10-20 A vNIC can only participate in oneL2 network upstreamBackupVLAN 30-40 Both dynamic and static pinningmethods are supportedFabric Interconnect - AEnd Host ModeFabric Interconnect - BEnd Host ModeIOM-BIOM
Cisco Public UCS Internal Block Diagram mLOM CPU 0 CPU 1 Gen 3x8 Gen 3 16x SFP 16x SFP Expansion Module 16x SFP 16x SFP UCS Blade Chassis UCS 6248 Expansion Module UCS 6248 2208XP 2208XP Fabric Uplinks (NIFs) Adapter Server Blade Midplane IO Modules Fabric Interconnects Backplane Ports (HIFs)
