Transcription
Technical White PaperECS Overview and ArchitectureAbstractThis document provides a technical overview and design of the Dell EMC ECS software-defined cloud-scale object storage platform.February 2021h14071.18
RevisionsRevisionsDateDescriptionDecember 2015Initial releaseMay 2016Updated for 2.2.1September 2016Updated for 3.0August 2017Updated for 3.1March 2018Updated for 3.2September 2018Updated for Gen3 HardwareFebruary 2019Updated for 3.3September 2019Updated for 3.4February 2020Updated ECSDOC-628 changesMay 2020Updated for 3.5November 2020Updated for 3.6February 2021Updated for 3.6.1AcknowledgementsThis paper was produced by the following:Author: Zhu, JarvisThe information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in thispublication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of anysoftware described in this publication requires an applicable software license.This document may contain certain words that are not consistent with Dell's current language guidelines. Dell plans to update the document oversubsequent future releases to revise these words accordingly.This document may contain language from third party content that is not under Dell's control and is not consistent with Dell's current guidelines for Dell'sown content. When such third-party content is updated by the relevant third parties, this document will be revised accordingly.Copyright 2015–2021 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or itssubsidiaries. Other trademarks may be trademarks of their respective owners. [2/24/2021] [Technical White Paper] [h14071.18]2ECS Overview and Architecture h14071.18
Table of contentsTable of contentsRevisions.2Acknowledgements .2Table of contents .3Executive summary .51Introduction .61.1Audience .61.2Scope .62Value of ECS .73Architecture .93.1Overview .93.2ECS portal and provisioning services .103.3Data services .123.3.1 Object .123.3.2 HDFS .133.3.3 NFS .163.3.4 Connectors and gateways .163.4Storage engine .173.4.1 Storage services .173.4.2 Data .173.4.3 Data management .193.4.4 Data flow .213.4.5 Write optimizations for file size .223.4.6 Space reclamation .223.4.7 SSD metadata caching .233.4.8 Cloud DVR .243.5Fabric .243.5.1 Node agent .243.5.2 Lifecycle manager.253.5.3 Registry .253.5.4 Event library .253.5.5 Hardware manager .253.6Infrastructure.253.6.1 Docker .2543Appliance hardware models .27ECS Overview and Architecture h14071.18
Table of contents4.1EX-series .274.2Appliance networking .294.2.1 S5148F - front-end public switches .294.2.2 S5148F - back-end private switches .304.2.3 S5248F - front-end public switches .314.2.4 S5248F - back-end private switches .314.2.5 S5232 – aggregation switch .325Network separation .336Security .346.1Authentication .346.2Data services authentication .356.3Data-at-rest encryption (D@RE) .356.3.1 Key rotation.366.4ECS IAM .366.5Object tagging.376.5.1 Additional information about object tagging .387Data integrity and protection .397.18Compliance .40Deployment .418.1Single site deployment .428.2Multisite deployment .438.2.1 Data consistency .448.2.2 Active replication group .448.2.3 Passive replication group .458.2.4 Geo-caching remote data .478.2.5 Behavior during site outage .4798.3Failure tolerance .498.4Disk replacement automation .518.5Tech refresh.51Storage protection overhead .5210 Conclusion .54A4Technical support and resources .55ECS Overview and Architecture h14071.18
Executive summaryExecutive summaryOrganizations require options for consuming public cloud services with the reliability and control of a privatecloud infrastructure. Dell EMC ECS is a software-defined, IPv6-supported, cloud-scale, object storageplatform that delivers S3, Atmos, CAS, Swift, NFSv3, and HDFS storage services on a single, modernplatform.With ECS, administrators can easily manage globally distributed storage infrastructure under a single globalnamespace that provides anywhere access to content. ECS core components are layered for flexibility andresiliency. Each layer is abstracted and independently scalable with high availability.Simple RESTful API access for storage services are being embraced by developers. Use of HTTP semanticslike GET and PUT simplifies the application logic required when compared with traditional, but familiar, pathbased file operations. In addition, ECS’s underlying storage system is strongly consistent, which means it canguarantee an authoritative response. Applications that are required to guarantee authoritative delivery of dataare able to do so without complex code logic by using ECS.5ECS Overview and Architecture h14071.18
Introduction1IntroductionThis document provides an overview of Dell EMC ECS object storage platform. It details the ECS designarchitecture and core components such as the storage services and data protection mechanisms.1.1AudienceThis paper is intended for anyone interested in understanding the value and architecture of ECS. It aims toprovide context with links to additional information.1.2ScopeThis document focuses primarily on ECS architecture. It does not cover installation, administration, andupgrade procedures for ECS software or hardware. It also does not cover specifics on using and creatingapplications with the ECS APIs.Updates to this document are done periodically and generally coincide with major releases or new features.6ECS Overview and Architecture h14071.18
Value of ECS2Value of ECSECS provides significant value for enterprises and service providers seeking a platform architected to supportrapid data growth. The main advantages and features of ECS that enable enterprises to globally manage andstore distributed content at scale include: Cloud Scale - ECS is an object storage platform for both traditional and next-gen workloads. ECS’ssoftware-defined layered architecture promotes limitless scalability. Feature highlights are:- Flexible Deployment - ECS has unmatched flexibility with features such as:- -Data-at-rest (D@RE) with key rotation and external key management.Encrypted inter-site communicationDisables ports 9101/9206 by default to empowers organizations to meet compliance policiesReporting, policy- and event-based record retention and platform hardening for SEC Rule 17a4(f) compliance including advanced retention management such as litigation hold and min-maxgovernanceCompliance with Defense Information Systems Agency (DISA) Security Technical ImplementationGuide (STIG) hardening guidelinesAuthentication, authorization and access controls with Active directory and LDAPIntegration with monitoring and alerting infrastructure (SNMP traps and SYSLOG)Enhanced enterprise capabilities (multi-tenancy, capacity monitoring and alerting)TCO Reduction - ECS can dramatically reduce Total Cost of Ownership (TCO) relative to bothtraditional storage and public cloud storage. It even offers a lower TCO than tape for long-termretention. Features include:-7Appliance deploymentSoftware-only deployment with support for certified or custom industry standard hardwareMultiprotocol support: Object (S3, Swift, Atmos, CAS) and File (HDFS, NFSv3)Multiple workloads: Modern apps and long-term archiveSecondary storage for Data Domain Cloud Tier and Isilon using CloudPoolsNon-disruptive upgrade paths to current generation ECS modelsEnterprise Grade - ECS provides customers more control of their data assets with enterprise classstorage in a secure and compliant system with features such as:- Globally distributed object infrastructureExabyte scale without limits on storage pool, cluster or federated environment capacityNo limits exist on the number of objects in a system, namespace or bucketEfficient at both small and large file workloads with no limits to object sizeGlobal namespaceSmall and large file performanceSeamless Centera migrationFully compliant with Atmos RESTLow management overheadSmall data center footprintHigh storage utilizationECS Overview and Architecture h14071.18
Value of ECSThe design of ECS is optimized for the following primary use cases: 8Modern Applications - ECS designed for modern development such as for next-gen web, mobileand cloud applications. Application development is simplified with strongly-consistent storage. Alongwith multi-site, simultaneous multi-user read/write access, as the ECS capacity changes and grows,developers never need to recode their apps.Secondary Storage - ECS is used as secondary storage to free up primary storage of infrequentlyaccessed data, while also keeping it reasonably accessible. Examples are policy-based tieringproducts such as Data Domain Cloud Tier and Isilon CloudPools. GeoDrive, a Windows-basedapplication, gives Windows systems direct access to ECS to store data.Geo-Protected Archive - ECS serves as a secure and affordable on-premise cloud for archival andlong-term retention purposes. Using ECS as an archive tier can significantly reduce primary storagecapacities. To allow for better storage efficiencies for cold archive use cases a 10 2 erasure coding(EC) scheme is available in addition to the default of 12 4.Global Content Repository - Unstructured content repositories containing data such as images andvideos are often stored in high cost storage systems making it impossible for businesses to costeffectively manage massive data growth. ECS enables consolidation of multiple storage systems intoa single, globally accessible and efficient content repository.Storage for Internet of Things - The Internet of Things (IoT) offers a new revenue opportunity forbusinesses who can extract value from customer data. ECS offers an efficient IoT architecture forunstructured data collection at massive scale. With no limits on the number of objects, the size ofobjects or custom metadata, ECS is the ideal platform to store IoT data. ECS can also streamlinesome analytic workflows by allowing data to be analyzed directly on the ECS platform withoutrequiring time consuming extract, transform and load (ETL) processes. Hadoop clusters can runqueries using data stored on ECS by another protocol API such as S3 or NFS.Video Surveillance Evidence Repository - In contrast to IoT data, video surveillance data has amuch smaller object storage count, but a much higher capacity footprint per file. While dataauthenticity is important, data retention is not as critical. ECS can be a low-cost landing area orsecondary s
This document provides an overview of Dell EMC ECS object storage platform. It details the ECS design architecture and core components such as the storage services and data protection mechanisms. 1.1 Audience This paper
