Dell EMC ECS: Networking And Best Practices

Transcription

Best PracticesDell EMC ECS: Networking and Best PracticesAbstractThis white paper describes networking and related best practices for ECS, theDell EMC software-defined cloud-scale object storage platform.April 2021H15718.7

RevisionsRevisionsDateDecember 2016August 2017February 2019March 2020April 2021DescriptionInitial releaseUpdated based on ECS 3.1Add Gen3 detailsUpdated based on ECS Special Feature Configuration SupportUpdated based on ECS 3.6.1AcknowledgmentsThis paper was produced by the following member of the Technical Marketing Engineering and SolutionArchitects team in the Unstructured Data Storage division of Dell Technologies:Author: Jarvis ZhuSupport: Chris KraftThis document may contain certain words that are not consistent with Dell's current language guidelines. Dell plans to update the document oversubsequent future releases to revise these words accordingly.This document may contain language from third party content that is not under Dell's control and is not consistent with Dell's current guidelines for Dell'sown content. When such third-party content is updated by the relevant third parties, this document will be revised accordingly.The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in thispublication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.Use, copying, and distribution of any software described in this publication requires an applicable software license.Copyright 2016–2021 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC and other trademarks are trademarksof Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. [4/27/2021] [Best Practices] [H15718.7]2Dell EMC ECS: Networking and Best Practices H15718.7

Table of contentsTable of contentsRevisions.2Acknowledgments .2Executive summary .51Introduction .61.1Audience .61.2Scope .62ECS overview .73ECS network overview .83.14Traffic types .8ECS network hardware .94.1.1 Public switches .94.1.2 Private switches .104.1.3 Aggregation switches for EXF900 .124.1.4 Customer-provided switches .135ECS network configurations .145.1Production network .145.1.1 ToR public switch configuration and node connectivity .155.1.2 Customer uplink configuration .175.1.3 Network configuration custom requests .215.2Internal private network .225.2.1 NAN topologies .225.2.2 Segment LAN .245.2.3 Private.4 LAN.245.2.4 RMM/iDRAC access from customer network (optional) .256ECS network separation .266.1.1 Network separation configurations .276.1.2 ECS switch configuration for network separation .307ECS network performance .358Tools .3698.1ECS portal .368.2ECS designer and planning guide .368.3Secure remote services .368.4Linux or HAL tools .36Network services .3910 Conclusion .403Dell EMC ECS: Networking and Best Practices H15718.7

Table of contentsA4Technical support and resources .41Dell EMC ECS: Networking and Best Practices H15718.7

Executive summaryExecutive summaryECS is the Dell EMC cloud-scale, object storage platform for traditional, archival, and next-generationworkloads. It provides geo-distributed and multi-protocol (Object, HDFS, and NFS) access to data. In anECS deployment, a turn-key appliance or industry standard hardware can be utilized to form the hardwareinfrastructure. In either type of deployment, a network infrastructure is required for the interconnectionbetween the nodes and customer environments for object storage access. This paper describes ECSnetworking and some best practices to consider.5Dell EMC ECS: Networking and Best Practices H15718.7

IntroductionIntroductionThis white paper provides details on ECS networking. ECS network hardware, network configurations, andnetwork separation are discussed. It will also describe some ECS networking best practices. This papershould be used as an adjunct to the following Dell EMC ECS documentation: 1.1ECS Hardware Guide - for Gen1 and Gen2 hardwareECS EX-Series Hardware GuideNetworks Guide for D- and U- Series (Gen 1 and Gen 2) HardwareNetworks Guide for EX300 and EX3000 (EX-Series) HardwareAudienceThis document is targeted for Dell EMC field personnel and customers interested in understanding ECSnetworking infrastructure and the role networking plays within ECS as well as how ECS connects to customerenvironments.1.2ScopeThis white paper explains ECS network configurations and topologies and provides some best practices. Itdoes not cover ECS network installation and administration. Refer to official Dell EMC ECS productdocumentation for information on ECS installation and administration.Updates to this document are done periodically and often coincides with new features and functionalitychanges.6Dell EMC ECS: Networking and Best Practices H15718.7

ECS overviewECS overviewECS features a software-defined architecture that promotes scalability, reliability and availability. ECS wasbuilt as a completely distributed storage system to provide data access, protection and geo-replication. Themain use cases for ECS include storage for modern applications and as secondary storage to free up primarystorage of infrequently used data while also keeping it reasonably accessible.ECS software and hardware components work in concert for un-paralleled object and file access. Thesoftware layers are presented in Figure 1 along with the underlying infrastructure and hardware layers. It canbe viewed as a set of layered components consisting of the following: ECS Portal and Provisioning Services - provides an API, CLI and Web-based portal that allowsself-service, automation, reporting and management of ECS nodes. It also handles licensing,authentication, multi-tenancy, and provisioning services.Data Services - provides services, tools and APIs to support Object, and HDFS and NFSv3.Storage Engine - responsible for storing and retrieving data, managing transactions and protectingand replicating data.Fabric - provides clustering, health, software and configuration management as well as upgradecapabilities and alerting.Infrastructure - uses SUSE Linux Enterprise Server 12 as the base operating system for the turnkeyappliance or qualified Linux operating systems for industry standard hardware configuration.Hardware - industry standard hardware composed of x86 nodes with internal disks or attached todisk array enclosures with disks, and top-of-rack (ToR) switches.For more in-depth architecture of ECS, refer to the ECS Architecture and Overview white paper.ECS Software Layers7Dell EMC ECS: Networking and Best Practices H15718.7

ECS network overviewECS network overviewECS network infrastructure consists of a set of ToR switches which allow for the following two types ofnetwork connections: Public Network - connection between the customer production network and ECS.Private Network - for management of nodes and switches within and across racks.The ToR switches are dedicated to either the public (production) network, or to the private, internal-to-ECSonly network. For the public network a pair of 10/25 GbE network switches are used which service data andinternal communication between the nodes. For the private network, depending on the hardware generation,either a single 1 GbE switch for Gen1 or Gen2 (Gen1/2) is used, or a pair of 25 GbE switches (Gen3) areused. The private network is used for remote management, console access and PXE booting which enablesrack management and cluster-side management and provisioning. From these set of switches, uplinkconnections are presented to the customer production environment for storage access and management ofECS. The networking configurations for ECS are recommended to be redundant and highly available.Note: Gen1/2 and Gen3 EX300, 500, 3000 series use public network for internal communication betweennodes. Gen3 EXF900 series use private network for internal communication between nodes.3.1Traffic typesUnderstanding the traffic types within ECS and the customer environment is useful for architecting thenetwork physical and logical layout and configuration for ECS.The public network carries the following types of traffic: Data - customer data and I/O requests.Management - provisioning and/or querying ECS via the portal and/or ECS Rest Management APIsas well as network services traffic such DNS, AD and NTP.Inter-node - messages are sent between nodes to process I/O requests depending on owner of dataand inter-node checks.Replication - data replicated to other nodes within a replication group.In a single site single-rack deployment, inter-node traffic stays within the ECS rack switches; whereas insingle site multi-rack deployment, inter-node traffic traverses from one rack set of switches up to the customerswitch and to the other rack switches to process requests. In a multi-site or geo-replicated deployment, all theabove traffic will also go across the WAN.Note: The inter-node traffic of EXF900 running in a private network due to the NVMe-oF architecture ofEXF900.The private network, which is under Dell EMC control, is entirely for node and switch maintenance and thustraffic types include: 8Segment Maintenance Management - traffic associated with administration, installation or setup ofnodes and switches within rack.Private.4 network - interconnects multiple, co-located ECS intra-rack networks into a single interrack network through VLAN 4. The private.4 network is also referred to as the Nile Area Network(NAN).Dell EMC ECS: Networking and Best Practices H15718.7

ECS network hardware4ECS network hardwareEach ECS appliance rack contains three, four or six switches. Gen1/2 appliances have three switches, twofor the public network and one for the private network. Gen3 EX300, EX500 and EX3000 has two publicswitches and two private switches. Gen3 EXF900 have another two dedicated aggregation switches forprivate switches ensure all the EXF900 nodes have line rate performance to any node in any rack.Switch details, including model numbers, along with designated switch port usage and network cablinginformation, can be found in the ECS Hardware Guide for Gen1/2 appliances and the ECS EX-SeriesHardware Guide for Gen3 appliances.4.1.1Public switchesPublic (a.k.a. production or front-end) switches are used for data transfer to and from customer applicationsas well as internal node-to-node communication for Gen1/2 and Gen3 EX300, 500, 3000 series. The internode traffic of Gen3 EXF900 will go through inside the private switch. These switches connect to the ECSnodes in the same rack. For Gen1/2 appliances, two 10 GbE, 24-port or 52-port Arista switches are used.For Gen3 appliances, two 10/25 GbE (EX300) or two 25 GbE (EX500, EX3000 and EXF900) 48-port Dellswitches are used. To create a High Availability (HA) network for the nodes in the rack, the public switcheswork in tandem using LACP/MLAG, with the Arista switches in Gen1/2 appliances, and Virtual Link Trunking(VLT), with the Dell EMC switches in Gen3 appliances. This pairing is for redundancy and resiliency in caseof switch failure.Across all generations of hardware, Gen1-3, each ECS node has two Ethernet ports that directly connect toone of the ToR public switches. Due to NIC bonding, the individual connections of a node appear to theoutside world as one. The nodes are assigned IP addresses from the customer’s network either statically orvia a DHCP server. At a minimum one uplink between each ToR public switch in the ECS appliance to thecustomer network is required. The public switch management ports connect to the ToR private switch(es).Public network for ECS Gen2 nodes9Dell EMC ECS: Networking and Best Practices H15718.7

ECS network hardwarePublic network for ECS Gen3 nodesBest Practices For redundancy and to maintain a certain level of performance, have two uplinks per switch to customerswitch, or, four uplinks per rack minimum. Use 25 GbE switches for optimal performance when using customer-provided public switches. Have dedicated switches for ECS and do not use “shared ports” on customer core network.4.1.2Private switchesPrivate switches are used by ECS for node management. For Gen1/2 appliances, the private switches alsoallow for out-of-band (OOB) management communication between customer networks and RemoteManagement Module (RMM) ports in individual ECS nodes. Gen1/2 appliances have a 52-port 1 GbE Aristaswitch, or a Cisco switch for organizations with strict Cisco only requirements. Gen3 appliances contain two25 GbE 48-port Dell private switches identical in model to the public switches.Note: Gen3 does not allow for OOB management communication from customer networks.The management ports in each node connect to the private switch(es). They use private addresses such as192.168.219.x. Each Gen1/2 node also has a connection between its RMM port and the private switch, whichin turn can have access to the customer network to provide OOB management of the nodes. Gen3 nodesalso have a connection between their iDRAC (Integrated Dell Remote Controller) and one of the privateswitches, however, there is no customer-facing OOB management for Gen3 ECS nodes.Note: Dell EMC switches are required for the private network. Private switches cannot be customer-provided.10Dell EMC ECS: Networking and Best Practices H15718.7

ECS network hardwarePrivate network for Gen2 ECS NodesPrivate network for Gen3 ECS NodesBest Practices When physically connecting nodes to the management switch, do so in an ordered and sequentialfashion. For instance, node 1 should connect to port 1, node 2 to port 2 and so on. Connecting nodes toan arbitrary port between 1 through 24 can cause installation issues. RMM/iDRAC Connections are optional and best practice is to ask customer requirements for theseconnections.11Dell EMC ECS: Networking and Best Practices H15718.7

ECS network hardware4.1.3Aggregation switches for EXF900The aggregation switches can be installed in the Dell EMC rack or a customer provided rack. The aggregationswitch allows to connect up to seven racks of EXF900 nodes in the same cluster.Aggregation network topologyNote: For more information about the network cabling, please refer to ECS EXSerires Hardware guide andECS network guide.12Dell EMC ECS: Networking and Best Practices H15718.7

ECS network hardware4.1.4Customer-provided switchesThe flexibility of ECS allows for variations of network hardware and configurations which should meet the DellEMC standards like ECS Appliance- Special Feature Configuration Support. However, An RPQ is a requestfor approval or review of non-standard configuration.ECS Appliance- Special Feature Configuration Support is an internal-only authenticated reference, customersneed ask Presales/Sales help if they want to read this reference.Regarding customer provided switches, configuration and support are the responsibility of the customer.These switches should be dedicated to

This white paper provides details on ECS networking. ECS network hardware, network configurations, and network separation are discussed. It will also describe some ECS networking best practices. This paper should be used as an adjunct to the following Dell EMC ECS documentation