Transcription
Cloudera-Intel-CiscoHadoop BenchmarkTOI (External)What matters in aHadoop Cluster?Floris Grandvarlet (Cisco) floris.grandvarlet@cisco.comPatrick Schotts (Intel) patrick.schots@intel.comWoody Christy (Cloudera) wchristy@cloudera.com
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?AcknowledgmentsThe authors acknowledge the contributions of:Intel:Stephen G. Anderson, stephen.g.anderson@intel.comRob Kypriotakis, rob.kypriotakis@intel.comJacob A. Ohara, jacob.a.ohara@intel.comGert Pauwels, Gert.Pauwels@intel.comRichard B. Pilling, richard.b.pilling@intel.comCisco:Arnaud Bassaler, abassale@cisco.comPeter Ruttens, pruttens@cisco.comMichel Sumbul, msumbul@cisco.comKarthik Kulkarni, kkulkar@cisco.comCloudera:Sandeep Brahmarouthu, sandeep@cloudera.comJonathan Cooper, jcooper@cloudera.comRob Johnson, rj@cloudera.comKunal Kusoorkar, kkusoorkar@cloudera.comDwai Lahiri, dlahiri@cloudera.comJonathan Seidman, jseidman@cloudera.comALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLEC TIVELY, “DESIGNS”)IN THIS PAPER ARE PRESENTED “AS IS,” WITH ALL FAULTS. CISCO AND ITS SUP PLIERS DISCLAIM ALL WARRANTIES,INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE ANDNONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCOOR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THEDESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATIONOF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITSSUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THEDESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo,DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Storeare service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP,CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, CiscoSystems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center,Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, theIronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, NetworkingAcademy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert,StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registeredtrademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.All other trademarks mentioned in this document or website are the property of their respective owners. The use of the wordpartner does not imply a partnership relationship between Cisco and any other com pany. (0809R) 2014 Cisco Systems, Inc. All rights reserved.Cloudera-Intel-Cisco v2.0 Public Page 2
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?Contents1. Introduction.4Executive Summary.52. Benchmark Test oftware post-installation configuration.8Architecture.9Server Configuration and Cabling. 10Rack. 113. CPU Benchmark.123.1. Overview. 123.2. CPU Test Architecture. 123.3. CPU Benchmarks Caveats. 133.3.1. Cloudera Manager Architecture. 153.3.2. Power measurements. 163.4. Results. 183.4.1. Tera Results for CPU. 183.4.2. Word Count for CPU. 193.4.3. Power Results for CPU. 203.4.4. Consolidated Results with Pricing. 203.5. CPU Benchmark Results Conclusion. 214. Cluster Benchmark.234.1. Overview.4.2. Benchmark Caveat.4.2.1. Benchmark Caveat : Raid Configuration.4.2.2. Benchmark Caveat : Network Bandwidth.4.3. Benchmark Hyper-Threading.4.3.1. Hyper-Threading details.4.4. Benchmark Network Bandwidth.4.4.1. TeraGen and TeraSort details.4.5. Benchmark Hyper-Threading/Networking results conclusion.4.6. Benchmark Data Nodes Scale-out.4.7. Benchmark HDD scaling.4.8. Benchmark HDD/Scaling results conclusion.2323232426272829313233355. Security assessment.365.1. Overview.5.2. Servers.5.3. Hadoop.5.3.1. Environment.5.3.2. Attack Surface.5.3.3. Additional Notes.3636373738406. Appendix.417. References.41Cloudera-Intel-Cisco v2.0Public Page 3
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?Cloudera-Intel-CiscoHadoop Benchmark TOI (External)What matters in a Hadoop Cluster?1. IntroductionThis benchmark intends to look at the physics of Hadoop. Today, it is pretty easy to set up a working Hadoop environment wherethe base default configuration works and the jobs are completed.But, are you really happy if the job completes in three hours at 10% utilisation?Also, success can easily be a curse, with management realizing the benefits and looking for richer insight, meaning larger morecomplex and more numerous jobs, but Hadoop does not self-optimize.You now have 3 options :1. Buy more hardware2. Buy expertise3. Optimize the platform as-is to get the best performanceBecause of this, an idea came about in April 2014 as we (Intel/Cisco) discussed the setup of a demonstration Hadoop cluster.We decided to extend the initial setup work to include a benchmark of the physics of Hadoop before launching the demoplatform. When we talk about the physics of Hadoop, we are looking at various aspects of the platform: Platform processor choice Base Hardware platform Network stack Disk partitioning/filesystem Base Operating System Cloudera software Data workloadThis paper provides all the necessary information to reproduce the same test-bed, as well as a summary of the results and thelesson learned. This information is valuable in designing and building Hadoop clusters.The intent of this benchmarking effort is not compete with other published record benchmarks; rather it done mainly todemonstrate the relative impacts of the choices of CPU, Network BW, HDD, etc. for your Hadoop platform. The final results areexpressed as a relative percentage of the baseline. (Raw data available in the Appendix section).Cloudera-Intel-Cisco v2.0Public Page 4
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?Executive SummaryThis benchmark intends to look at the physics of Hadoop and how to optimize your Hadoop platform.When we talk about the physics of Hadoop, we are looking at various aspects of the platform, including: Platform processor choice Base Hardware platform Network stack Disk partitioning/filesystem Base Operating System Cloudera software Data workloadThe intent of this benchmarking effort is not to compete with other published record benchmarks; rather it was done mainly todemonstrate the relative impacts of the choices of CPU, Network BW, HDD, etc., for your Hadoop platform. The final results areexpressed as a relative percentage of the baseline. (Raw data available in the Appendix section).As such, we benchmarked CPU, HyperThreading, Networking BW (10/5/1 Gb), HDD (12/24) as well as scalability (4/5/6 datanodes), using the usual suspects—TeraGen, TeraSort, TeraValidate (1TB and 2TB), as well as wordcount ( 12GB). We adopted amulti-run (x3, average value) benchmarking strategy with a max deviation of 20% between runs, which if it occurred triggered afull re-test.The test-bed :Figure 1Test-BedCloudera-Intel-Cisco v2.0Public Page 5
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?All details have been provided to enable you to reproduce/modify these tests for your own purposes, with the results expressedas a relative percentage of the baseline. (Raw data available in the Appendix section).In conclusion, it appears that:The best CPU is the E5-2690 v2, which presents the best balance of cores and frequency.Network BW is definitively on the 10 GE side.Hyper-Threading needs to be set to “ON”.HDD of 24 provides a benefit, as expected.Scalability is proven, also as expected with Hadoop.Security is transversal and provided as a gentle reminder, as we did not consider security when setting up the benchmark, but itis mandatory for the operational environment.Overall, the main lesson learned is that knowledge and review of the entire Hadoop platform is a must for decent performance(feel free to read the epilogue to learn about our initial results).Cloudera-Intel-Cisco v2.0Public Page 6
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?2. Benchmark Test bed2.1. HardwareThe Cloudera-Intel-Cisco cluster for this benchmark is composed of 6 Data Nodes and 2 Name Nodes (active/standby). 8 x Cisco C240 M3 rack server. 256GB total memory for each node (16GB DDR3-1600-MHz RDIMM/PC3-12800/dual rank/1.35v) Data Nodes: 24 individual RAID0 volume with 1 HDD per volume. Name Nodes: 8 HDDs configured as RAID6 with 4 hot-spare HDDs (12 HDDs free) Each HDD is a 600GB 6 GbpsSAS 10K RPM SFF HDDWe also used a couple of UCS rack servers to host Monitoring/Management platform in the backend (Cacti, Ganglia, Intel DCM,UCS Manager, Cloudera Manager).The network is architectured around: Cisco UCS 6200 Series Fabric Interconnects provide high-bandwidth, low-latency connectivity for servers, with integrated,unified management provided for all connected devices by Cisco UCS Manager. Cisco Nexus 2200 Series Fabric Extenders (FEX) provide an optional extended single point of management (extending thenetwork into each rack, acting as remote line cards for fabric interconnects and providing highly scalable and extremely costeffective connectivity for a large number of nodes). For the high-performance BW test, we did not use the FEX. Cisco UCS Virtual Interface Cards (VICs) are unique to Cisco. Cisco UCS Virtual Interface Cards incorporate next-generationconverged network adapter (CNA) technology from Cisco, and offer dual 10-Gbps ports designed for use with Cisco UCSC-Series Rack-Mount Servers.Using a Nexus 5548 switch to connect all management workstations (Cloudera Manager, UCS Manager, monitoring tools/suites )2.2. SoftwareCloudera Distribution of Hadoop (known as CDH) is a popular enterprise-grade, hardened distribution of Apache Hadoop andrelated projects. CDH is 100 percent Apache-licensed open source and offers unified batch processing, interactive SQL, andinteractive search, and role-based access controls.Similar to Linux distribution, which gives you more than Linux, CDH delivers the core elements of Hadoop; scalable storage anddistributed computing, along with additional components such as a user interface, plus necessary enterprise capabilities, such assecurity and integration with a broad range of hardware and software solutions.The integration and the entire solution is thoroughly tested and fully documented. By taking the guesswork out of building aHadoop deployment, CDH provides a streamlined path to success in solving real business problems.For more information about what projects are included in CDH, see CDH Version and Packaging Information.html CentOS: 6.5 (Final) UCSM version: 2.2(1d) BIOS C240 M3: 3.1.5.4f.0.1113201 Cloudera Hadoop 5.0.1Cloudera-Intel-Cisco v2.0Public Page 7
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?2.3. Software post-installation configurationAll other values are at their specified defaults except for the followingConfiguration itemFinal urce.memory-mb (all resource C during some testsyarn scheduler minimum allocation mapreduce.map.java.optsCD manager Defaultsmapreduce.reduce.java.optsCD manager Defaultsmapred compress map outputTRUEmapred map output compression ed reduce parallel 64CDH version5.0.1Additional Linux and cluster changes: CPU scaling set to performance to force the CPU to always run at the highest frequency Please note that installing unnecessary services will cause resources to be assigned to these services rather than be freefor use by services needed to process the workload. We chose to disable these unnecessary services for all phases oftesting;unused services such as impala and Hbase were disabled;Impala/Hbase systems need to be set to performance vm.swappiness set to vm.swapiness 0 Transparent Huge Pages Defrag set to OFF. Network tuning Set ring buffer to 4096 Tuned memory settings in sysctl.conf Enabled jumbo frames on the LAN and the servers lowered IRQ rate and increased throughputCloudera-Intel-Cisco v2.0Public Page 8
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?Changes in sysctl.conf:#10 Gbps settingsTCP/IP memory tuningnet.ipv4.tcp timestamps 0net.ipv4.tcp sack 0net.ipv4.tcp rmem 4096 134217728 134217728net.ipv4.tcp wmem 4096 134217728 134217728Core memory tuningnet.core.rmem max 134217728net.core.wmem max 134217728net.core.rmem default 134217728net.core.rmem default 134217728net.core.optme max 134217728net.core.netdev max backlog 250000# recommended default congestion control is htcpnet.ipv4.tcp congestion control htcp# recommended for hosts with jumbo frames enablednet.ipv4.tcp mtu probing 1vm.swappiness 0 onBoot.sh script added to /etc/rc.local to apply on every boot.onBoot.sh script:#Set all CPU to max freqfor i in {0.47};do echo performance /sys/devices/system/cpu/cpu i/cpufreq/scalinggovernor;done#Disable Transparent Huge Page defragecho never /sys/kernel/mm/transparent hugepage/defrag#Transmit to NIC card 4096 Bytes at a time#setpci -v -d 1137:0043 e6.b 2e#Jumbo framesifconfig eth0 mtu 9000ifconfig eth0 txqueuelen 10002.4. ArchitectureThis benchmark architecture is based on CPAv2 for Big Data with Cloudera-based architecture, scaled down to a total of 8nodes.The Cisco UCS solution for Cloudera is based on Cisco UCS Common Platform Architecture Version 2 (CPAv2) for Big Data,a highly scalable architecture designed to meet a variety of scale-out application demands with seamless data integration andmanagement integration capabilities built using the following components: Cisco UCS 6200 Series Fabric Interconnects provide high-bandwidth, low-latency connectivity for servers, with integrated,unified management provided for all connected devices by Cisco UCS Manager. Deployed in redundant pairs, Cisco fabricinterconnects offer the full active-active redundancy, performance, and exceptional scalability needed to support the largenumber of nodes that are typical in clusters serving big data applications. Cisco UCS Manager enables rapid and consistentserver configuration using service profiles, automating ongoing system maintenance activities such as firmware updatesacross the entire cluster as a single operation. Cisco UCS Manager also offers advanced monitoring with options to raisealarms and send notifications about the health of the entire cluster.Cloudera-Intel-Cisco v2.0Public Page 9
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster? Cisco UCS 2200 Series Fabric Extenders extend the network into each rack, acting as remote line cards for fabricinterconnects and providing highly scalable and extremely cost-effective connectivity for a large number of nodes. For thehigh-performance BW test, we did not us the FEX(see 3.5). Cisco UCS C240 M3 Rack-Mount Servers are 2-socket servers based on Intel Xeon E5-2600 v2 series processors andsupporting up to 768 GB of main memory. The 24 Small Form Factor (SFF) disk drives are supported in a performanceoptimized option and 12 Large Form Factor (LFF) disk drives are supported in the capacity option, along with 4 GigabitEthernet LAN-on-motherboard (LOM) ports. While the UCS Manager provides global manageability of both Rack and Bladeservers (same profiles, templating, etc.), we use a rack-mounted solutionaligned to the essence of Hadoop, clustering tightlycoupled CPU-Storage capacity for data locality and processing efficiency. Cisco UCS Virtual Interface Cards (VICs) are unique to Cisco. Cisco UCS Virtual Interface Cards incorporate next-generationconverged network adapter (CNA) technology from Cisco, and offer dual 10-Gbps ports designed for use with Cisco UCSC-Series Rack-Mount Servers. Optimized for virtualized networking, these cards deliver high performance and bandwidthutilization and support up to 256 virtual devices. Cisco UCS Manager resides within the Cisco UCS 6200 Series Fabric Interconnects. It makes the system self-aware andself-integrating, managing all of the system components as a single logical entity. Cisco UCS Manager can be accessedthrough an intuitive graphical user interface (GUI), a command-line interface (CLI), or an XML application-programminginterface (API). Cisco UCS Manager uses service profiles to define the personality, configuration and connectivity of all of theresources within Cisco UCS, radically simplifying provisioning of resources so that the process takes minutes instead of days.This simplification allows IT departments to shift their focus from constant maintenance to strategic business initiatives.2.5. Server Configuration and CablingThe Cisco UCS C240 M3 Rack Server is by default equipped with Intel Xeon E5-2660 v2 processors, 256 GB of memory, CiscoUCS Virtual Interface Card 1225 Cisco, Cisco LSI MegaRAID SAS 9271 CV-8i storage controller and 24 x 1TB 7.2K SAS diskdrives.Figure 2 Network Cabling both 10GE and 5/1GECloudera-Intel-Cisco v2.0Public Page 10
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?For more information on physical connectivity and single-wire management see:http://www.cisco.com/en/US/docs/unified computing/ucs/c-series integration/ucsm2.1/b UCSM2-1 C-Integrationchapter 010.htmlFor more information on physical connectivity illustrations and cluster setup, see:http://www.cisco.com/en/US/docs/unified computing/ucs/c-series integration/ucsm2.1/b UCSM2-1 C-Integrationchapter 010.html#reference FE5B914256CB4C47B30287D2F9CE3597Figure 3 depicts a 64-node cluster, and each link represents 8 x 10 Gigabit links.2.6. RackCloudera-Intel-Cisco v2.0Public Page 11
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?3. CPU Benchmark3.1. OverviewThe CPU selection process considered the attribute of: core count, base and effective turbo frequency as well as powerconsumption in a limited set of processors to be tested, and GPL pricing1 added on the side for information. You can see belowthe 5 CPU selected, trying to be representative of Highest-frequency, Highest number of core and mean pricing. 2336 2057 2057 1389 1552See below some information about the TurboBoost technology and effect between # of core active and frequency.Processor SKU(TDP (W))E5-2690BaseFrequency(GHz)Cores1 CoreActive2 CoresActive3 CoresActive4 CoresActive5 CoresActive6 CoresActive7 CoresActive8 CoresActive2.983.8 ( 9)3.6 ( 7)3.6 ( 7)3.4 ( 5)3.4 ( 5)3.3 ( 4)3.3 ( 4)3.3 ( 4)(135W)The E5-2660 v2 was chosen as the baseline, all other results have been scored relative to the baseline.For example, ifthe time to completion for the E5-2660 v2 is 00:40:00 (so 40 minutes),then the following results:E5-2695 v2 00:44:00E5-2690 v2 00:20:00The comparative results are given as follows:E5-2695 v2 : 10%E5-2690 v2 : - 50%3.2. CPU Test ArchitectureAll CPUs used in testing have the same architecture; only yarn.nodemanager.resource.cpu-vcores Cloudera configuration settingwas changed to match the Logical cores available in the processor installed for that specific test sequenceThis is pricing we found at the time of the benchmark, we can reasonably assume that pricing will go down in time or can bedifferent per region/theatre; however, we can also assume that differences/evolutions would be proportional for each CPU/solution.1Cloudera-Intel-Cisco v2.0Public Page 12
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?The entire platform Config was locked down, with the only changes being those being measured – tests 1-5 on cpu (with onlyVcores changes), test 6 platform attributes (BW, # of disks, HyperThreading, #of NameNode)TestnameCPUDataNode/node manager1Gb/10GbHT ON / OFFCluster Tuning Vcores(default group)Nbr. of HDDLine rateDCM power loggingTest 1E5-2695v26/510GbON38244 links - 5Gb/sGroupBDtest2E5-2660v26/510GbON32244 links - 5Gb/sGroupBDtest3E5-2643v26/510GbON19244 links - 5Gb/sGroupBDtest4E5-2667v26/510GbON26244 links - 5Gb/sGroupBDtest5E5-2690v26/510GbON32244 links - 5Gb/sGroupBD3.3. CPU Benchmarks CaveatsThe open-public benchmark was selected so that results could be reproduced by anyone. In addition, one could tune or modifythem to better reflect their specific needs and be able to make the best choice of Hadoop cluster architecture.The following well-known and popular benchmark processes/protocols were used: TeraGen, TeraSort, TeraValidate2, and PigWordcount3.Note: The Transaction Processing Performance Council recently published TPCx-HS for standard Hadoop benchmarking. TPCExpress Benchmark(tm)HS (TPCx-HS) was developed to provide an objective measure of the hardware, operating systemand commercial Apache Hadoop File System API-compatible software distributions, and to provide the industry with verifiableperformance, price-performance and availability metrics. The benchmark models continuous system availability 24 hours a day, 7days a week. More information on TPCx-HS can be found here: http://www.tpc.org/tpcx-hs/default.aspThere are two required Jar files4 to execute for this benchmark Jar 1 - hadoop-mapreduce-client-jobclient.jar Jar 2 - hadoop-mapreduce-examples.jarCopy these files from /opt/cloudera/parcels/ distribution /lib/hadoop-mapreduce to the working directory where the tests willbe run (where benchmark-test.sh is located)Unpack and copy the inputtext.zip file to hdfs as follows: Unzip inputtext.xzipNote the location where the files are extracted.Edit the benchmark-v2.sh script line as follows: For hadoop fs put pattern /user/root/wordcount/in/ Once script has run once, put a # in front of the command to avoid the copy action happening for every script tutorial.html4All Jar scripts can be found in the Appendix section.Cloudera-Intel-Cisco v2.0Public Page 13
Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster?The benchmark script started 3 Hadoop reference benchmarks: TeraGen, TeraSort, TeraValidate (10 billion rows, 1TB) TeraGen, TeraSort, TeraValidate
Cloudera-Intel-Cisco v2.0 Public Page 7 Cloudera-Intel-Cisco Hadoop Benchmark TOI (External) What matters in a Hadoop Cluster? 2. Benchmark Test bed 2.1. Hardware The Cloudera-Intel-Cisco cluster for this benchmark is composed of 6 Data Nodes and 2 Name Nodes (active/standby). 8 x Cisco C240 M3 rack server.
