WIXLIB

DriveScale-ClouderaReference Architecture

Drivescale-ClouderaREFERENCE ARCHITECTURETable of Contents1.Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.Audience and Scope. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.DriveScale Cloudera Enterprise Solution Overview. . . . . . . . . . . . . . . . . . . . . . . . 44.DriveScale Components Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64.1 Hardware: DriveScale SAS Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65.Reference Architecture Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75.1 Physical Cluster Component List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75.2 Logical Cluster Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85.3 Physical Cluster Topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105.4 Cluster Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.5 Enabling Hadoop Virtualization Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.6 Disk and Filesystem Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.7 OS Supportability/Compatibility Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.8 JBOD Supportability/Compatibility Matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.Rack Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2019 DriveScale Inc. All Right Reserved.2 of 54

Drivescale-ClouderaREFERENCE ARCHITECTURE7.References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208.Bill of Materials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2110. Appendix A: Glossary of Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2211. Appendix B: DriveScale Cluster Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2411.1 Configure Your Domains with DriveScale Central (DSC). . . . . . . . . . . . . . . . . . 2411.2 Set up DriveScale Composer nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2611.3 Set up DriveScale Adapter (DSA). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2611.4 Start the Composer login to Composer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2711.5 Set up Servers/DataNodes/MasterNodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2811.6 Tagging JBOD and drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2811.7 Creating Server Nodes and Clusters from templates . . . . . . . . . . . . . . . . . . . . 2911.7.1 Creating Node and Cluster Template. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2911.7.2 Creating Cluster Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3411.7.2 DriveScale Cluster Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3412. Appendix B: Cloudera Manager Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3512.1 Cloudera Manager Installation Procedure for Reference Architecture . . . . . . 36 2019 DriveScale Inc. All Right Reserved.3 of 54

REFERENCE ARCHITECTUREDRIVESCALE-CLOUDERA1. Executive SummaryThis document is a high-level design reference architecture guide for implementing ClouderaEnterprise on a DriveScale solution with industry standard servers and JBOD.The reference architecture introduces all the high-level hardware, and software that are included inthe stack. Each high-level component is then described individually. This reference architecture doesnot describe the Cloudera data components or their applications.DriveScale Technology OverviewDriveScale is leading the charge in bringing hyperscale computing capabilities to mainstreamenterprises. It’s composable data center architecture transforms rigid data centers into flexibleand responsive scale-out deployments. Using DriveScale, data center administrators can deployindependent pools of commodity compute and storage resources, automatically discover availableassets, and combine and recombine these resources as needed. The solution is provided througha set of on-premises and SaaS tools that coordinate between multiple levels of infrastructure. WithDriveScale, Hadoop architects can more easily support Hadoop deployments of any size as well asother modern application workloads.DriveScale provides hardware and software technology that allows separate deployment of computeand storage using commodity servers with minimal drives for Operating System and JBODs (Justa Bunch of Disks), with flexible binding of storage-to-compute resources in any ratio required by anapplication. As needs change, these bindings can be dissolved and reconfigured on demand, allunder software control.DriveScale makes it possible for data-driven companies to deploy high scale, high performanceserver infrastructure through composability for data-intensive computing. The DriveScale ComposablePlatform enables agility at cloud scale by creating flexible pools of disaggregated heterogeneouscompute nodes and storage systems and composing them into secure, highly available serversoptimized to each workload. DriveScale empowers IT to quickly and cost-effectively expand their dataand analytics deployments and drive revenue.2. Audience and ScopeThis reference architecture guide is for Hadoop and IT architects who are responsible for the designand deployment of Cloudera Enterprise solutions on premises, as well as for Apache Hadoopadministrators and architects and data center architects/engineers who collaborate with specialists inthat space.3. DriveScale Cloudera Enterprise Solution OverviewApache Hadoop is designed to address the ever so changing hardware requirements from customersfor a more flexible and dynamic hardware infrastructure that provides significant cost and operationalbenefits. It is designed with composability as the primary goal, saving money, improving utilization andgreatly simplifying the deployment of Hadoop clusters. 2017 DriveScale Inc. All Right Reserved.4 of 54

Drivescale-ClouderaREFERENCE ARCHITECTUREHadoop is an Apache project being developed in the Java programming language by a globalcommunity of contributors. Yahoo!, has been the largest contributor to this project, and uses ApacheHadoop extensively across its businesses. Core committers on the Hadoop project include employeesfrom Cloudera, eBay, Facebook, Getopt, Hortonworks, Huawei, IBM, InMobi, INRIA, LinkedIn, MapR,Microsoft, Pivotal, Twitter, UC Berkeley, VMware, WANdisco, and Yahoo!, with contributions from manymore individuals and organizations.Although Hadoop is popular and widely used, installing, configuring, and running a productionHadoop cluster involves many concerns, including: Choosing the appropriate Hadoop software distribution and extensions Installing monitoring and management software Allocation of Hadoop services to physical nodes Selection of appropriate server hardware Rightsizing the storage configuration Implementing data locality Design of the network fabric Sizing and system scalability Overall performanceThese concerns are complicated by the need to understand the workloads that will be running on thecluster, the fast-moving pace of the core Hadoop project, and the challenges to managing a systemdesigned to scale to thousands of nodes in a single cluster.The DriveScale Cloudera Solution was designed by DriveScale in collaboration with Cloudera, andembodies all the hardware, software, resources and services needed to run Hadoop in a productionenvironment. This end-to-end solution approach means that you can be in production with Hadoop ina shorter time than is typically possible with homegrown solutions. The solution is based on ClouderaEnterprise Data Hub 5.x (including Cloudera Distributed Hadoop), DriveScale hardware and software,industry standard servers, network switches and JBODs.This solution includes components that span the entire solution stack: Reference architecture and best practices Optimized storage configurations Optimized network infrastructure Cloudera Enterprise Data Hub including Cloudera Distributed HadoopThis solution is designed to address the clear majority of Apache Hadoop use cases including, but notlimited to: 2019 DriveScale Inc. All Right Reserved.Big data analytics5 of 54

Drivescale-ClouderaREFERENCE ARCHITECTURE ETL Offload Data Warehouse Optimization Batch processing of unstructured data Big data visualization Search and predictive analysis4. DriveScale Components OverviewDriveScale system is composed of one hardware component and four software components whichare described below:4.1 Hardware: DriveScale SAS AdapterThis is a 1U appliance with adapters that connect to servers via 10Gb Ethernet interfaces and toJBOD’s via SAS interfaces.4.2 SoftwareThere are four principal components of the DriveScale software:a)b) 2019 DriveScale Inc. All Right Reserved.DriveScale Composer The server running the DriveScale Composer (Composer) bundle is called the Composer node. A typical deployment consists of three Composer's in a clustered configuration for high availability(HA). The software manages and configure resources and contains the inventory/configurationinformation repository and database:3Inventory: Composer's, DS Adapters, switches, JBOD chassis, disks, server nodes3Configuration: node templates, cluster templates, configured clusters3Composer's Database: used as a message bus to communicate with the end points.DriveScale Server Agent6 of 54

Drivescale-ClouderaREFERENCE ARCHITECTURE c)DriveScale Server Agent discovery action provides inventory for hardware and servers, and createsmappings between server nodes and the disks they consume.DriveScale Server AgentCloud-based software management portal that acts as the:d)osoftware distribution repositories for subscribersoDriveScale keys repositoryocentralized log file repositoryouser documentation repositoryolicense managerDriveScale SAS Adapter Software DriveScale SAS Adapter software enables the JBODs to be mapped to the servers and over thenetwork to be used as local drives.5. Reference Architecture Details5.1 Physical Cluster Component ListThe following table lists the physical components for the riveScale SASAdapterDHCP, Jumbo frameenabled1U appliance withadapters that connectto servers via Ethernet,and to JBOD’s via SAS.2DriveScale AdapterDHCP, Jumbo frameenabledProvides the datanetwork.4 for each chassisDriveScale ComposerDriveScale Composerrunning as a VMManages andconfigures the nodesand cluster and alsostores the inventory/configurationrepository of everyhardware in thecluster.Min 1, for HA3 DriveScaleComposer's should beconfigured as masterand slaveServers2 socket CPU andmemory according tothe individual Hadoopcluster requirementsCommodity x86servers that house allthe Node Manager,compute instancesand DriveScaleagents.Min 3 Master nodes 5 Data nodesHDD for Servers2 drives configured inRAID 1The internal drives areused for OS install.2 for each server 2019 DriveScale Inc. All Right Reserved.7 of 54