WIXLIB

Introduction1.4The advantages of Spark and Hadoop on PowerStoreBoth Spark and HDFS share a similar distributed architecture that requires a powerful, highly scalable, andflexible infrastructure. PowerStore is performance-optimized for any workload, and its adaptable scale-up andscale-out architecture complements the distributed model of these applications. This section highlights thePowerStore features that benefit and extend the application environment.1.4.1AppsON brings applications closer to the infrastructure and storageBringing applications closer to data increases density and simplifies infrastructure operations. ThePowerStore AppsON capability integrates with VMware vSphere , resulting in streamlined management inwhich storage resources plug directly into the virtualization layer. Using VMware as the onboard applicationenvironment results in unmatched simplicity, since support is inherently available for any standard VM-basedapplications. When a new PowerStore X model is deployed, the VASA provider is automatically registered,and the datastore is created, eliminating manual steps and saving time. PowerStore seamlessly integrates theVMware ESXi software into the same hardware. Two ESXi nodes are embedded inside the appliance whichhas direct access to the same storage resources. This close integration allows applications such as Sparkand Hadoop to take full advantage of server and storage virtualization with simplified deployment andmanagement. AppsON is available on the PowerStore X model exclusively.1.4.2Agile infrastructure, flexible scaling on a high-performing storage and computeplatformPowerStore provides flexible scaling with ease of management that compliments the Spark and Hadoopscale-up and scale-out distribution model. The integrated hypervisor dynamically scales up the cluster nodeswhen the workload requires it, while you can rapidly provision new nodes on the same or on other appliancesin a different location.Big-data applications require large amount of data and computational power for analytics, machining learning,model training, and other workloads. With a PowerStore appliance, administrators can scale up the storagecapacity by adding disks and disk expansion enclosures without service interruption at any time. You can alsoconfigure multiple PowerStore appliances into a cluster to increase CPUs, memory, storage capacity, andfront-end connectivity. Clustering simplifies and centralizes the management of multiple appliances fromPowerStore Manager, a single HTML5-based management interface. A cluster can consist of up to fourPowerStore T appliances or four PowerStore X appliances. Each appliance within the cluster can havedifferent configurations of CPUs, memory, NVMe drives, and expansion enclosures.The NVMe architecture is designed for the next-generation NVMe-based storage and takes advantage of lowoverhead NVRAM cache. PowerStore is engineered to handle the most demanding workloads.1.4.3Mission-critical high availability and fault-tolerant platformPowerStore provides a high level of stability and reliability for Spark and HDFS. At the hardware level,PowerStore is highly available and fault tolerant. It monitors the storage devices continuously, and itautomatically relocates data from failing devices to avoid data loss. The PowerStore X model applianceincludes two ESXi nodes and redundant hardware components. The nondisruptive upgrade (NDU) featurefurther increases overall PowerStore availability. The updates take place on the nodes in a rolling fashion.NDU supports PowerStore software releases, hotfixes, and hardware and disk firmware.The dynamic resiliency engine (DRE) feature automatically protects and repairs the underlying storage fromdrive failures. Administrators are not required to manually configure or manage the protection settings for thedrives.9Dell EMC PowerStore: Apache Spark Solution Guide H18663

IntroductionTo support high-value business workloads and service requirements on the application level, it is essential toprotect and ensure the availability of the Spark and HDFS nodes. The Spark master node and the HDFSNameNode are central to all operations of the application in the cluster. When they become inaccessible, allapplications and storage operations are affected. Also, if a DataNode is not reachable for an extended period,the Namenode determines the blocks on the failed node and starts making copies of the blocks from otherreplicas until the replication factor is met.With standard VMware vSphere High Availability (HA) integrated into PowerStore, the embedded VMwareESXi hypervisor automatically restarts or migrates failed Spark and HDFS servers to a different ESXi node.This process resumes operations by helping restore Spark and HDFS to its full operation capacity, and itminimizes the chance of the DataNodes being marked dead.To achieve an even higher level of redundancy and application availability, you can deploy the Spark clusterand HDFS cluster across multiple PowerStore appliances in different racks, floors, or locations. PowerStoreimproves application availability and provides unparalleled flexibility and mobility to relocate and move acrossdata centers and appliances.1.4.4PowerStore inline data reduction reduces storage consumption and costData science and big-data applications continuously pull in a tremendous amount of data from varioussources. To help reduce storage consumption and cost, the PowerStore inline data-reduction featuremaximizes space savings by combining both software data deduplication and hardware compression. Datareduction works seamlessly in the background, is always enabled, and cannot be disabled. Since datareduction is always active in PowerStore, enabling application or operating system compression may notprovide additional savings.1.4.5Efficient and convenient snapshot data backupPowerStore provides Spark and HDFS with extra data protection through array-based snapshots. APowerStore snapshot is a point-in-time copy of the data. The snapshots are space efficient and requireseconds to create. Snapshot data are exact copies of the source data and can be used for application testing,backup, or DevOps. Because of the tight integration with VMware vSphere, PowerStore can take vVol VMsnapshots directly from PowerStore Manager using a protection policy schedule or on demand. You can viewthe VM snapshot information in PowerStore and vCenter.1.4.6Secure data protection with ease of mindWith high-value data driving business applications, data security is a top concern for all organizations. Lost orstolen data can seriously damage the reputation of an organization and result in huge financial costs and lossof customer trust. Dell Technologies engineered PowerStore with Data at Rest Encryption (D@RE) whichuses self-encrypting drives and supports array-based, self-managed keys. When D@RE is activated, data isencrypted as it is written to disk using the 256-bit Advanced Encryption Standard (AES). PowerStore D@REprovides this data security benefit to Spark applications while eliminating application overhead, performancepenalties, and administrative overhead that is typically associated with software-based solutions.1.4.7Unified infrastructure and services managementPowerStore provides deep integration with VMware management tools and services with Dell EMC VirtualStorage Integrator (VSI), VMware vRealize Operations Manager (vROps), VMware vRealize Orchestrator(vRO), and VMware Storage Replication Adapter (SRA). You can easily incorporate ESXi on PowerStore Xmodels into your existing vCenter and manage all VMware infrastructure and services from a unifiedmanagement platform.10Dell EMC PowerStore: Apache Spark Solution Guide H18663

Introduction1.4.8Spark value and future expansionBig-data platforms such as Spark and Hadoop create enormous value for organizations. As the value andscale of this data grows, it is critical to have a future-proof platform that is easy to manage. The platform mustalso provide technical innovation for future growth, and support the application architecture. Spark andHadoop on PowerStore bring IT organizations the ability to be agile, efficient, and responsive to businessdemands.1.5TerminologyThe following terms are used with PowerStore.Appliance: Solution containing a base enclosure and attached expansion enclosures. The size of anappliance could be only the base enclosure or the base enclosure plus expansion enclosures.PowerStore node: Storage controller that provides the processing resources for performing storageoperations and servicing I/O between storage and hosts. Each PowerStore appliance contains two nodes.Base enclosure: Enclosure containing both nodes (node A and node B) and 25 NVMe drive slotsExpansion enclosure: Enclosures that can be attached to a base enclosure to provide additional storage.Fibre Channel (FC) protocol: Protocol used to perform SCSI commands over a Fibre Channel network.iSCSI: Provides a mechanism for accessing block-level data storage over network connections.NDU: A nondisruptive upgrade (NDU) updates PowerStore and maximizes its availability by performing rollingupdates. This includes updates for PowerStore software releases, hotfixes, and hardware and disk firmware.NVMe: Non-Volatile Memory Express is a communication interface and driver for accessing nonvolatilestorage media such as solid-state drives (SSD) and SCM drives through the PCIe bus.NVMe over Fibre Channel (NVMe-FC): Allows hosts to access storage systems across a networkfabric with the NVMe protocol using Fibre Channel as the underlying transport.NVRAM: Nonvolatile random-access memory is persistent random-access memory that retains data withoutan electrical charge. NVRAM drives are used in PowerStore appliance as additional system write caching.Volume: A block-level storage device that can be shared out using a protocol such as iSCSI or FibreChannel.Snapshot: A point-in-time view of data that is stored on a storage resource. You can recover files from asnapshot, restore a storage resource from a snapshot, or provide access to a host.Storage container: A VMware term for a logical entity that consists of one or more capability profiles andtheir storage limits. This entity is known as a vVol datastore when it is mounted in vSphere.PCIe: Peripheral Component Interconnect Express is a high-speed serial computer expansion bus standard.PowerStore Manager: An HTML5 management interface for creating storage resources and configuring andscheduling protection of stored data on PowerStore. PowerStore Manager can be used for all management ofPowerStore native replication.11Dell EMC PowerStore: Apache Spark Solution Guide H18663

IntroductionPowerStore T model: Container-based storage system that is running on purpose-built hardware. Thisstorage system supports unified (block and file) workloads, or block-optimized workloads.PowerStore X model: Container-based storage system that runs inside a virtual machine that is deployed ona VMware hypervisor. Besides offering block-optimized workloads, PowerStore also allows you to deployapplications directly on the array.RecoverPoint for Virtual Machines: Protects VMs in a VMware environment with VM-level granularity

1.3 Apache Hadoop Distributed File System overview Apache Hadoop is an open-source software suite and framework for big-data processing. Hadoop Distributed File System (HDFS) is one of the core components of Hadoop. It is a distributed file system designed to be massively scalable, fault tolerant, and have high throughput.