Transcription
Mastering HadoopGo beyond the basics and master the next generationof Hadoop data processing platformsSandeep KaranthBIRMINGHAM - MUMBAI
Mastering HadoopCopyright 2014 Packt PublishingAll rights reserved. No part of this book may be reproduced, stored in a retrievalsystem, or transmitted in any form or by any means, without the prior writtenpermission of the publisher, except in the case of brief quotations embedded incritical articles or reviews.Every effort has been made in the preparation of this book to ensure the accuracyof the information presented. However, the information contained in this book issold without warranty, either express or implied. Neither the author, nor PacktPublishing, and its dealers and distributors will be held liable for any damagescaused or alleged to be caused directly or indirectly by this book.Packt Publishing has endeavored to provide trademark information about all of thecompanies and products mentioned in this book by the appropriate use of capitals.However, Packt Publishing cannot guarantee the accuracy of this information.First published: December 2014Production reference: 1221214Published by Packt Publishing Ltd.Livery Place35 Livery StreetBirmingham B3 2PB, UK.ISBN 978-1-78398-364-3www.packtpub.comCover image by Poonam Nayak (pooh.graphics@gmail.com)
CreditsAuthorSandeep KaranthReviewersProject CoordinatorKranti BerdeProofreadersShiva AchariSimran BhogalPavan Kumar PolineniMaria GouldUchit VyasAmeesha GreenYohan WadiaPaul HindleCommissioning EditorEdward GordonAcquisition EditorRebecca YouéIndexerMariammal ChettiyarGraphicsAbhinash SahuValentina DsilvaContent Development EditorRuchita BhansaliProduction CoordinatorArvindkumar GuptaTechnical EditorsBharat PatilRohit Kumar SinghParag TopreCopy EditorsSayanee MukherjeeVikrant PhadkayCover WorkArvindkumar Gupta
About the AuthorSandeep Karanth is a technical architect who specializes in building andoperationalizing software systems. He has more than 14 years of experience in thesoftware industry, working on a gamut of products ranging from enterprise dataapplications to newer-generation mobile applications. He has primarily worked atMicrosoft Corporation in Redmond, Microsoft Research in India, and is currently acofounder at Scibler, architecting data intelligence products.Sandeep has special interest in data modeling and architecting data applications.In his area of interest, he has successfully built and deployed applications, cateringto a variety of business use cases such as vulnerability detection from machine logs,churn analysis from subscription data, and sentiment analyses from chat logs. Theseapplications were built using next generation big data technologies such as Hadoop,Spark, and Microsoft StreamInsight and deployed on cloud platforms such asAmazon AWS and Microsoft Azure.Sandeep is also experienced and interested in areas such as green computingand the emerging Internet of Things. He frequently trains professionals andgives talks on topics such as big data and cloud computing. Sandeep believesin inculcating skill-oriented and industry-related topics in the undergraduateengineering curriculum, and his talks are geared with this in mind. Sandeep hasa Master's degree in Computer and Information Sciences from the University ofMinnesota, Twin Cities.Sandeep's twitter handle is @karanths. His GitHub profile is https://github.com/Karanth, and he writes technical snippets at https://gist.github.com/Karanth.
AcknowledgmentsI would like to dedicate this book to my loving daughter, Avani, who has taughtme many a lesson in effective time management. I would like to thank my wife andparents for their constant support that has helped me complete this book on time.Packt Publishing have been gracious enough to give me this opportunity, and I wouldlike to thank all individuals who were involved in editing, reviewing, and publishingthis book. Questions and feedback from curious audiences at my lectures have drivenmuch of the content of this book. Some of the subtopics are from experiences I gainedworking on a wide variety of projects throughout my career. I would like to thank myaudience and also my employers for indirectly helping me write this book.
About the ReviewersShiva Achari has over 8 years of extensive industry experience and is currentlyworking as a Big Data architect in Teradata. Over the years, he has architected,designed, and developed multiple innovative and high-performing large-scalesolutions such as distributed systems, data center, Big Data management,SaaS cloud applications, Internet applications, and data analytics solutions.He is currently writing a book on Hadoop essentials, which is based on Hadoop,its ecosystem components, and how we can leverage the components in differentphases of the Hadoop project life cycle.Achari has experience in designing Big Data and analytics applications, ingestion,cleansing, transformation, correlating different sources, data mining, and userexperience using Hadoop, Cassandra, Solr, Storm, R, and Tableau.He specializes in developing solutions for the Big Data domain and possessesa sound hands-on experience on projects migrating to the Hadoop world, newdevelopment, product consulting, and POC. He also has hands-on expertiseon technologies such as Hadoop, Yarn, Sqoop, Hive, Pig, Flume, Solr, Lucene,Elasticsearch, Zookeeper, Storm, Redis, Cassandra, HBase, MongoDB, Talend, R,Mahout, Tableau, Java, and J2EE.Shiva has expertise in requirement analysis, estimations, technology evaluation,and system architecture, with domain experience of telecom, Internet applications,document management, healthcare, and media.
Currently, he supports presales activities such as writing technical proposals (RFP),providing technical consultation to customers, and managing deliveries of Big Datapractice group in Teradata.He is active on LinkedIn at http://in.linkedin.com/in/shivaachari/.I would like to thank Packt Publishing for helping me out with thereviewing process and the opportunity to review this book, whichwas a great opportunity and experience. I will wish the publicationand author best of luck for the success of the book.Pavan Kumar Polineni is working as Analytics Manager at Fantain Sports.He has experience in the fields of information retrieval and recommendationengines. He is a Cloudera certified Hadoop administrator. His is interested inmachine learning, data mining, and visualization.He has a Bachelor's degree in Computer Science from Koneru Lakshmaiah College ofEngineering and is about to complete his Master's degree in Software Systems fromBITS, Pilani. He has worked at organizations such as IBM and Ctrls Datacenter. Hecan be found on Twitter as @polinenipavan.
Uchit Vyas is an open source specialist and a hands-on lead DevOps of ClogenyTechnologies. He is responsible for the delivery of solutions, services, and productdevelopment. He explores new enterprise open source and defining architecture,roadmaps, and best practices. He has consulted and provided training on variousopen source technologies, including cloud computing (AWS Cloud, Rackspace,Azure, CloudStack, Openstack, and Eucalyptus), Mule ESB, Chef, Puppet andLiferay Portal, Alfresco ECM, and JBoss, to corporations around the world.He has a degree in Engineering in Computer Science from the Gujarat University.He worked in the education and research team of Infosys Limited as senior associate,during which time he worked on SaaS, private clouds, virtualization, and now,cloud system automation.He has also published book on Mule ESB, and is writing various books on opensource technologies and AWS.He hosts a blog named Cloud Magic World, cloudbyuchit.blogspot.com,where he posts tips and phenomena about open source technologies, mostlycloud technologies. He can also be found on Twitter as @uchit vyas.I am thankful to Riddhi Thaker (my colleague) for helping me a lotin reviewing this book.Yohan Wadia is a client-focused virtualization and cloud expert with 5 years ofexperience in the IT industry.He has been involved in conceptualizing, designing, and implementing large-scalesolutions for a variety of enterprise customers based on VMware vCloud, AmazonWeb Services, and Eucalyptus Private Cloud.His community-focused involvement enables him to share his passion forvirtualization and cloud technologies with peers through social media engagements,public speaking at industry events, and through his personal blog at yoyoclouds.com.He is currently working with Virtela Technology Services, an NTT communicationscompany, as a cloud solutions engineer, and is involved in managing the company'sin-house cloud platform. He works on various open source and enterprise-levelcloud solutions for internal as well as external customers. He is also a VMwareCertified Professional and vExpert (2012, 2013).
www.PacktPub.comSupport files, eBooks, discount offers, and moreFor support files and downloads related to your book, please visit www.PacktPub.com.Did you know that Packt offers eBook versions of every book published, with PDFand ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get intouch with us at service@packtpub.com for more details.At www.PacktPub.com, you can also read a collection of free technical articles, signup for a range of free newsletters and receive exclusive discounts and offers on Packtbooks and ion/packtlibDo you need instant solutions to your IT questions? PacktLib is Packt's online digitalbook library. Here, you can search, access, and read Packt's entire library of books.Why subscribe? Fully searchable across every book published by Packt Copy and paste, print, and bookmark content On demand and accessible via a web browserFree access for Packt account holdersIf you have an account with Packt at www.PacktPub.com, you can use this to accessPacktLib today and view 9 entirely free books. Simply use your login credentials forimmediate access.
Table of ContentsPrefaceChapter 1: Hadoop 2.XThe inception of HadoopThe evolution of HadoopHadoop's oop's timelineHadoop 2.XYet Another Resource Negotiator (YARN)Architecture overviewStorage layer enhancementsHigh availabilityHDFS FederationHDFS snapshotsOther enhancementsSupport enhancementsHadoop distributionsWhich Hadoop geabilityAvailable distributionsCloudera Distribution of Hadoop (CDH)Hortonworks Data Platform (HDP)MapRPivotal 2627282929293030
Table of ContentsChapter 2: Advanced MapReduce31Chapter 3: Advanced Pig69MapReduce inputThe InputFormat classThe InputSplit classThe RecordReader classHadoop's "small files" problemFiltering inputsThe Map taskThe dfs.blocksize attributeSort and spill of intermediate outputsNode-local Reducers or CombinersFetching intermediate outputs – Map-sideThe Reduce taskFetching intermediate outputs – Reduce-sideMerge and spill of intermediate outputsMapReduce outputSpeculative execution of tasksMapReduce job countersHandling data joinsReduce-side joinsMap-side joinsSummaryPig versus SQLDifferent modes of executionComplex data types in PigCompiling Pig scriptsThe logical planThe physical planThe MapReduce planDevelopment and debugging aidsThe DESCRIBE commandThe EXPLAIN commandThe ILLUSTRATE commandThe advanced Pig operatorsThe advanced FOREACH 70717273737475767676777777The FLATTEN operatorThe nested FOREACH operatorThe COGROUP operatorThe UNION operatorThe CROSS operator7879808182[ ii ]
Table of ContentsSpecialized joins in Pig82User-defined functionsThe evaluation functions8586The Replicated joinSkewed joinsThe Merge join838485The aggregate functionsThe filter functionsThe load functionsThe store functionsPig performance optimizationsThe optimization rulesMeasurement of Pig script performanceCombiners in PigMemory for the Bag data typeNumber of reducers in PigThe multiquery mode in PigBest practicesThe explicit usage of typesEarly and frequent projectionEarly and frequent filteringThe usage of the LIMIT operatorThe usage of the DISTINCT operatorThe reduction of operationsThe usage of Algebraic UDFsThe usage of Accumulator UDFsEliminating nulls in the dataThe usage of specialized joinsCompressing intermediate resultsCombining smaller filesSummaryChapter 4: Advanced HiveThe Hive architectureThe Hive metastoreThe Hive compilerThe Hive execution engineThe supporting components of HiveData typesFile formatsCompressed filesORC files[ iii 103103103103104104105106106107107107108109109109
Table of ContentsThe Parquet filesThe data modelDynamic partitions110111114Indexes on Hive tablesHive query optimizersAdvanced DMLThe GROUP BY operationORDER BY versus SORT BY clausesThe JOIN operator and its types116118119120120120Advanced aggregation supportOther advanced clausesUDF, UDAF, and UDTFSummary122123123127Semantics for dynamic partitioningMap-side joinsChapter 5: Serialization and Hadoop I/OData serialization in HadoopWritable and WritableComparableHadoop versus Java serializationAvro serializationAvro and MapReduceAvro and PigAvro and HiveComparison – Avro versus Protocol Buffers / ThriftFile formatsThe Sequence file formatReading and writing Sequence filesThe MapFile formatOther data structuresCompressionSplits and compressionsScope for compressionSummaryChapter 6: YARN – Bringing Other Paradigms to HadoopThe YARN architectureResource Manager (RM)Application Master (AM)Node Manager (NM)YARN clients[ iv 52153154155157158158160161162
Table of ContentsDeveloping YARN applicationsWriting YARN clientsWriting the Application Master entityMonitoring YARNJob scheduling in YARNCapacitySchedulerFairSchedulerYARN commandsUser commandsAdministration commandsSummaryChapter 7: Storm on YARN – Low LatencyProcessing in HadoopBatch processing versus streamingApache StormArchitecture of an Apache Storm clusterComputation and data modeling in Apache StormUse cases for Apache StormDeveloping with Apache StormApache Storm 0.9.1Storm on YARNInstalling Apache Storm-on-YARNPrerequisitesInstallation 91192193194196197198206207207207208218Chapter 8: Hadoop on the Cloud219Chapter 9: HDFS Replacements239Cloud computing characteristicsHadoop on the cloudAmazon Elastic MapReduce (EMR)Provisioning a Hadoop cluster on EMRSummaryHDFS – advantages and drawbacksAmazon AWS S3Hadoop support for S3Implementing a filesystem in HadoopImplementing an S3 native filesystem in HadoopSummary[v]219221222223237240240241243244256
Table of ContentsChapter 10: HDFS Federation257Chapter 11: Hadoop Security273Limitations of the older HDFS architectureArchitecture of HDFS FederationBenefits of HDFS FederationDeploying federated NameNodesHDFS high availabilitySecondary NameNode, Checkpoint Node, and Backup NodeHigh availability – edits sharingUseful HDFS toolsThree-layer versus four-layer network topologyHDFS block placementPluggable block placement policySummaryThe security pillarsAuthentication in HadoopKerberos authenticationThe Kerberos architecture and workflowKerberos authentication and HadoopAuthentication via HTTP interfacesAuthorization in HadoopAuthorization in HDFSIdentity of an HDFS userGroup listings for an HDFS userHDFS APIs and shell commandsSpecifying the HDFS superuserTurning off HDFS 74275275276277278279279280281281282283Limiting HDFS usageService-level authorization in HadoopData confidentiality in HadoopHTTPS and encrypted shuffle283285286287Audit logging in HadoopSummary293295SSL configuration changesConfiguring the keystore and truststoreChapter 12: Analytics Using HadoopData analytics workflowMachine learningApache MahoutDocument analysis using Hadoop and MahoutTerm frequency[ vi ]287289297298299302304304
Table of ContentsDocument frequencyTerm frequency – inverse document frequencyTf-Idf in PigCosine similarity distance measuresClustering using k-meansK-means clustering using Apache ix: Hadoop for Microsoft Windows319Index339Deploying Hadoop on Microsoft WindowsPrerequisitesBuilding HadoopConfiguring HadoopDeploying HadoopSummary[ vii ]320321326328333338
PrefaceWe are in an age where data is the primary driver in decision-making. With storagecosts declining, network speeds increasing, and everything around us becomingdigital, we do not hesitate a bit to download, store, or share data with others aroundus. About 20 years back, a camera was a device used to capture pictures on film.Every photograph had to be captured almost perfectly. The storage of film negativeswas done carefully lest they get damaged. There was a higher cost associated withtaking prints of these photographs. The time taken between a picture click and toview it was almost a day. This meant that less data was being captured as thesefactors presented a cliff for people from recording each and every moment of theirlife, unless it was very significant.However, with cameras becoming digital, this has changed. We do not hesitate toclick a photograph of almost anything anytime. We do not worry about storageas our externals disks of a terabyte capacity always provide a reliable backup. Weseldom take our cameras anywhere as we have mobile devices that we can use totake photographs. We have applications such as Instagram that can be used to addeffects to our pictures and share them. We gather opinions and information about thepictures, and we click and base some of our decisions on them. We capture almostevery moment, of great significance or not, and push it into our memory books. Theera of big data has arrived!This era of Big Data has similar changes in businesses as well. Almost everything in abusiness is logged. Every action taken by a user on the page of an e-commerce page isrecorded to improve quality of service and every item bought by the user are recordedto cross-sell or up-sell other items. Businesses want to understand the DNA of theircustomers and try to infer it by pinching out every possible data they can get aboutthese customers. Businesses are not worried about the format of the data. They areready to accept speech, images, natural language text, or structured data. These datapoints are used to drive business decisions and personalize experiences for the user.The more data, the higher the degree of personalization and better the experience forthe user.
PrefaceWe saw that we are ready, in some aspects, to take on this Big Data challenge.However, what about the tools used to analyze this data? Can they handle thevolume, velocity, and variety of the incoming data? Theoretically, all this data canreside on a single machine, but what is the cost of such a machine? Will it be ableto cater to the variations in loads? We know that supercomputers are available,but there are only a handful of them in the world. Supercomputers don't scale. Thealternative is to build a team of machines, a cluster, or individual computing unitsthat work in tandem to achieve a task. A team of machines are interconnected via avery fast network and provide better scaling and elasticity, but that is not enough.These clusters have to be programmed. A greater number of machines, just like ateam of human beings, require more coordination and synchronization. The higherthe number of machines, the greater the possibility of failures in the cluster. How dowe handle synchronization and fault tolerance in a simple way easing the burden onthe programmer? The answer is systems such as Hadoop.Hadoop is synonymous with Big Data processing. Its simple programming model,"code once and deploy at any scale" paradigm, and an ever-growing ecosystem makeHadoop an inclusive platform for programmers with different levels of expertise andbreadth of knowledge. Today, it is the number-one sought after job skill in the datasciences space. To handle and analyze Big Data, Hadoop has become the go-to tool.Hadoop 2.0 is spreading its wings to cover a variety of application paradigms andsolve a wider range of data problems. It is rapidly becoming a general-purpose clusterplatform for all data processing needs, and will soon become a mandatory skill forevery engineer across verticals.This book covers optimizations and advanced features of MapReduce, Pig, and Hive.It also covers Hadoop 2.0 and illustrates how it can be used to extend the capabilitiesof Hadoop.Hadoop, in its 2.0 release, has evolved to become a general-purpose cluster-computingplatform. The book will explain the platform-level changes that enable this. Industryguidelines to optimize MapReduce jobs and higher-level abstractions such as Pig andHive in Hadoop 2.0 are covered. Some advanced job patterns and their applicationsare also discussed. These topics will empower the Hadoop user to optimize existingjobs and migrate them to Hadoop 2.0. Subsequently, it will dive deeper into Hadoop2.0-specific features such as YARN (Yet Another Resource Negotiator) and HDFSFederation, along with examples. Replacing HDFS with other filesystems is anothertopic that will be covered in the latter half of the book. Understanding these topicswill enable Hadoop users to extend Hadoop to other application paradigms and datastores, making efficient use of the available cluster resources.[2]
PrefaceThis book is a guide focusing on advanced concepts and features in Hadoop.Foundations of every concept are explained with code fragments or schematicillustrations. The data processing flow dictates the order of the concepts in each chapter.What this book coversChapter 1, Hadoop 2.X, discusses the improvements in Hadoop 2.X in comparison toits predecessor generation.Chapter 2, Advanced MapReduce, helps you understand the best practices and patternsfor Hadoop MapReduce, with examples.Chapter 3, Advanced Pig, discusses the advanced features of Pig, a framework to scriptMapReduce jobs on Hadoop.Chapter 4, Advanced Hive, discusses the advanced features of a higher-level SQLabstraction on Hadoop MapReduce called Hive.Chapter 5, Serialization and Hadoop I/O, discusses the IO capabilities in Hadoop.Specifically, this chapter covers the concepts of serialization and deserializationsupport and their necessity within Hadoop; Avro, an external serializationframework; data compression codecs available within Hadoop; their tradeoffs;and finally, the special file formats in Hadoop.Chapter 6, YARN – Bringing Other Paradigms to Hadoop, discusses YARN(Yet Another Resource Negotiator), a new resource manager that has beenincluded in Hadoop 2.X, and how it is generalizing the Hadoop platform toinclude other computing paradigms.Chapter 7, Storm on YARN – Low Latency Processing in Hadoop, discusses the oppositeparadigm, that is, moving data to the compute, and compares and contrasts it withbatch processing systems such as MapReduce. It also discusses the Apache Stormframework and how to develop applications in Storm. Finally, you will learn howto install Storm on Hadoop 2.X with YARN.Chapter 8, Hadoop on the Cloud, discusses the characteristics of cloud computing andHadoop's Platform as a Service offering across cloud computing service providers.Further, it delves into Amazon's managed Hadoop services, also known as ElasticMapReduce (EMR) and looks into how to provision and run jobs on a HadoopEMR cluster.[3]
PrefaceChapter 9, HDFS Replacements, discusses the strengths and drawbacks of HDFSwhen compared to other file systems. The chapter also draws attention to Hadoop'ssupport for Amazon's S3 cloud storage service. At the end, the chapter illustratesHadoop HDFS extensibility features by implementing Hadoop's support for S3'snative file system to extend Hadoop.Chapter 10, HDFS Federation, discusses the advantages of HDFS Federation and itsarchitecture. Block placement strategies, which are central to the success of HDFSin the MapReduce environment, are also discussed in the chapter.Chapter 11, Hadoop Security, focuses on the security aspects of a Hadoop cluster.The main pillars of security are authentication, authorization, auditing, and dataprotection. We will look at Hadoop's features in each of these pillars.Chapter 12, Analytics Using Hadoop, discusses higher-level analytic workflows,techniques such as machine learning, and their support in Hadoop. We takedocument analysis as an example to illustrate analytics using Pig on Hadoop.Appendix, Hadoop for Microsoft Windows, explores Microsoft Window OperatingSystem's native support for Hadoop that has been introduced in Hadoop 2.0. In thischapter, we look at how to build and deploy Hadoop on Microsoft Windows natively.What you need for this book?The following software suites are required to try out the examples in the book: Java Development Kit (JDK 1.7 or later): This is free software from Oraclethat provides a JRE (Java Runtime Environment) and additional toolsfor developers. It can be downloaded from loads/index.html. The IDE for editing Java code: IntelliJ IDEA is the IDE that has been usedto develop the examples. Any other IDE of your choice can also be used.The community edition of the IntelliJ IDE can be downloaded fromhttps://www.jetbrains.com/idea/download/. Maven: Maven is a build tool that has been used to build the samples inthe book. Maven can be used to automatically pull-build dependencies andspecify configurations via XML files. The code samples in the chapters canbe built into a JAR using two simple Maven commands:mvn compilemvn assembly:single[4]
PrefaceThese commands compile the code into a JAR file. These commands createa consolidated JAR with the program along with all its dependencies. It isimportant to change the mainClass references in the pom.xml to the driverclass name when building the consolidated JAR file.Hadoop-related consolidated JAR files can be run using the command:hadoop jar jar file argsThis command directly picks the driver program from the mainClass thatwas specified in the pom.xml. Maven can be downloaded and installed fromhttp://maven.apache.org/download.cgi. The Maven XML template fileused to build the samples in this book is as follows: ?xml version "1.0" encoding "UTF-8"? project xmlns "http://maven.apache.org/POM/4.0.0"xmlns:xsi emaLocation "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" modelVersion 4.0.0 /modelVersion groupId MasteringHadoop /groupId artifactId MasteringHadoop /artifactId version 1.0-SNAPSHOT /version build plugins plugin groupId org.apache.maven.plugins /groupId artifactId maven-compiler-plugin /artifactId version 3.0 /version configuration source 1.7 /source target 1.7 /target /configuration /plugin plugin version 3.1 /version groupId org.apache.maven.plugins /groupId artifactId maven-jar-plugin /artifactId configuration archive manifest mainClass MasteringHadoop.MasteringHadoopTest /mainClass /manifest /archive /configuration /plugin [5]
plugin artifactId maven-assembly-plugin /artifactId configuration archive manifest mainClass MasteringHadoop.MasteringHadoopTest /mainClass /manifest /archive descriptorRefs descriptorRef jar-with-dependencies /descriptorRef /descriptorRefs /configuration /plugin /plugins pluginManagement plugins !--This plugin's configuration is used to store Eclipsem2e settingsonly. It has no influence on the Maven builditself. -- plugin groupId org.eclipse.m2e /groupId artifactId lifecycle-mapping /artifactId version 1.0.0 /version configuration lifecycleMappingMetadata pluginExecutions pluginExecution pluginExecutionFilter groupId org.apache.maven.plugins /groupId artifactId maven-dependency-plugin /artifactId versionRange [2.1,) /versionRange goals goal copy-dependencies /goal /goals /pluginExecutionFilter action ignore / /action /pluginExecution /pluginExecutions /lifecycleMappingMetadata /configuration
Preface /plugin /plugins /pluginManagement /build dependencies !-- Specify dependencies in this section -- /dependencies /project Hadoop 2.2.0: Apache Hadoop is required to try out the examples ingeneral. Appendix, Hadoop for Microsoft Windows, has the details on Hadoop'ssingle-node installation on a Microsoft Windows machine. The steps aresimilar and easier for other operating systems such as Linux or Mac, andthey can be found at -dist/hadoop-common/SingleNodeSetup.htmlWho this book is forThis book is meant for a gamut of readers. A novice user of Hadoop can use thisbook to upgrade his skill level in the technology. People with existing experiencein Hadoop can enhance their knowledge about Hadoop to solve challenging dataprocessing problems they might be encountering in their profession. People who areusing Hadoop, Pig, or Hive at their workplace can use the tips provided in this bookto help make their jobs faster and more efficient. A curious Big Data professionalcan use this book to understand the expanding horizons of Hadoop and how it isbroadening its scope by embracing other paradigms, not just MapReduce. Finally, aHadoop 1.X user can get insights into the repercussions of upgrading to Hadoop 2.X.The book assumes familiarity with Hadoop, but the reader need not be an expert.Access to a Hadoop installation, either in your organization, on the cloud, or on yourdesktop/notebook is recommended to try some of the concepts.ConventionsIn this book, you will find a number of styles of text that distinguish betweendifferent kinds of information. Here are some examples of these styles, an
Chapter 1: Hadoop 2.X 11 The inception of Hadoop 12 The evolution of Hadoop 13 Hadoop's genealogy 14 Hadoop-.20-append15 Hadoop-.20-security16 Hadoop's timeline 16 Hadoop 2.X 17 Yet Another Resource Negotiator (YARN) 18 Architecture overview 19 Storage layer enhancements 20 High availability 20 HDFS Federation 22 HDFS snapshots 22
