WIXLIB

on editi datEd ph &U4t s e dReviHadoop: The Definitive GuideUsing Hadoop 2 exclusively, author Tom White presents new chapterson YARN and several Hadoop-related projects such as Parquet, Flume,Crunch, and Spark. You’ll learn about recent changes to Hadoop, andexplore new case studies on Hadoop’s role in healthcare systems andgenomics data processing. Learn fundamental components such as MapReduce, HDFS,and YARN Explore MapReduce in depth, including steps for developingapplications with it Set up and maintain a Hadoop cluster running HDFS andMapReduce on YARN Learn two data formats: Avro for data serialization and Parquetfor nested data Use data ingestion tools such as Flume (for streaming data) andSqoop (for bulk data transfer) Understand how high-level data processing tools like Pig, Hive,Crunch, and Spark work with Hadoop Learn the HBase distributed database and the ZooKeeperdistributed configuration serviceyou have the “Now opportunityto learnabout Hadoop from amaster—not only of thetechnology, but also of common sense andplain talk.”—Doug CuttingClouderaTom White, an engineer at Cloudera and member of the Apache SoftwareFoundation, has been an Apache Hadoop committer since 2007. He has writtennumerous articles for oreilly.com, java.net, and IBM’s developerWorks, and speaksregularly about Hadoop at industry conferences.US 49.99Twitter: @oreillymediafacebook.com/oreillyWhitePROGR AMMING L ANGUAGES/HADOOPFOURTH EDITIONHadoop:The Definitive GuideGet ready to unlock the power of your data. With the fourth edition ofthis comprehensive guide, you’ll learn how to build and maintain reliable,scalable, distributed systems with Apache Hadoop. This book is ideal forprogrammers looking to analyze datasets of any size, and for administratorswho want to set up and run Hadoop clusters.HadoopThe Definitive GuideSTORAGE AND ANALYSIS AT INTERNET SCALECAN 57.99ISBN: 978-1-491-90163-2Tom White

on editi datEd ph &U4t s e dReviHadoop: The Definitive GuideUsing Hadoop 2 exclusively, author Tom White presents new chapterson YARN and several Hadoop-related projects such as Parquet, Flume,Crunch, and Spark. You’ll learn about recent changes to Hadoop, andexplore new case studies on Hadoop’s role in healthcare systems andgenomics data processing. Learn fundamental components such as MapReduce, HDFS,and YARN Explore MapReduce in depth, including steps for developingapplications with it Set up and maintain a Hadoop cluster running HDFS andMapReduce on YARN Learn two data formats: Avro for data serialization and Parquetfor nested data Use data ingestion tools such as Flume (for streaming data) andSqoop (for bulk data transfer) Understand how high-level data processing tools like Pig, Hive,Crunch, and Spark work with Hadoop Learn the HBase distributed database and the ZooKeeperdistributed configuration serviceyou have the “Now opportunityto learnabout Hadoop from amaster—not only of thetechnology, but also of common sense andplain talk.”—Doug CuttingClouderaTom White, an engineer at Cloudera and member of the Apache SoftwareFoundation, has been an Apache Hadoop committer since 2007. He has writtennumerous articles for oreilly.com, java.net, and IBM’s developerWorks, and speaksregularly about Hadoop at industry conferences.US 49.99Twitter: @oreillymediafacebook.com/oreillyWhitePROGR AMMING L ANGUAGES/HADOOPFOURTH EDITIONHadoop:The Definitive GuideGet ready to unlock the power of your data. With the fourth edition ofthis comprehensive guide, you’ll learn how to build and maintain reliable,scalable, distributed systems with Apache Hadoop. This book is ideal forprogrammers looking to analyze datasets of any size, and for administratorswho want to set up and run Hadoop clusters.HadoopThe Definitive GuideSTORAGE AND ANALYSIS AT INTERNET SCALECAN 57.99ISBN: 978-1-491-90163-2Tom White

FOURTH EDITIONHadoop: The Definitive GuideTom White

Hadoop: The Definitive Guide, Fourth Editionby Tom WhiteCopyright 2015 Tom White. All rights reserved.Printed in the United States of America.Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions arealso available for most titles (http://safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.Editors: Mike Loukides and Meghan BlanchetteProduction Editor: Matthew HackerCopyeditor: Jasmine KwitynProofreader: Rachel HeadJune 2009:First EditionOctober 2010:Second EditionMay 2012:Third EditionApril 2015:Fourth EditionIndexer: Lucie HaskinsCover Designer: Ellie VolckhausenInterior Designer: David FutatoIllustrator: Rebecca DemarestRevision History for the Fourth Edition:2015-03-19:First release2015-04-17:Second releaseSee http://oreilly.com/catalog/errata.csp?isbn 9781491901632 for release details.The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Hadoop: The Definitive Guide, the coverimage of an African elephant, and related trade dress are trademarks of O’Reilly Media, Inc.Many of the designations used by manufacturers and sellers to distinguish their products are claimed astrademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademarkclaim, the designations have been printed in caps or initial caps.While the publisher and the author have used good faith efforts to ensure that the information and instruc‐tions contained in this work are accurate, the publisher and the author disclaim all responsibility for errorsor omissions, including without limitation responsibility for damages resulting from the use of or relianceon this work. Use of the information and instructions contained in this work is at your own risk. If any codesamples or other technology this work contains or describes is subject to open source licenses or the intel‐lectual property rights of others, it is your responsibility to ensure that your use thereof complies with suchlicenses and/or rights.ISBN: 978-1-491-90163-2[LSI]

For Eliane, Emilia, and Lottie

Table of ContentsForeword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiPreface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xixPart I.Hadoop Fundamentals1. Meet Hadoop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Data!Data Storage and AnalysisQuerying All Your DataBeyond BatchComparison with Other SystemsRelational Database Management SystemsGrid ComputingVolunteer ComputingA Brief History of Apache HadoopWhat’s in This Book?356688101112152. MapReduce. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19A Weather DatasetData FormatAnalyzing the Data with Unix ToolsAnalyzing the Data with HadoopMap and ReduceJava MapReduceScaling OutData FlowCombiner FunctionsRunning a Distributed MapReduce JobHadoop Streaming1919212222243030343737v

RubyPython37403. The Hadoop Distributed Filesystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43The Design of HDFSHDFS ConceptsBlocksNamenodes and DatanodesBlock CachingHDFS FederationHDFS High AvailabilityThe Command-Line InterfaceBasic Filesystem OperationsHadoop FilesystemsInterfacesThe Java InterfaceReading Data from a Hadoop URLReading Data Using the FileSystem APIWriting DataDirectoriesQuerying the FilesystemDeleting DataData FlowAnatomy of a File ReadAnatomy of a File WriteCoherency ModelParallel Copying with distcpKeeping an HDFS Cluster 7476774. YARN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Anatomy of a YARN Application RunResource RequestsApplication LifespanBuilding YARN ApplicationsYARN Compared to MapReduce 1Scheduling in YARNScheduler OptionsCapacity Scheduler ConfigurationFair Scheduler ConfigurationDelay SchedulingDominant Resource FairnessFurther Readingvi Table of Contents808182828385868890949596

5. Hadoop I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97Data IntegrityData Integrity in decsCompression and Input SplitsUsing Compression in MapReduceSerializationThe Writable InterfaceWritable ClassesImplementing a Custom WritableSerialization FrameworksFile-Based Data StructuresSequenceFileMapFileOther File Formats and Column-Oriented FormatsPart MapReduce6. Developing a MapReduce Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141The Configuration APICombining ResourcesVariable ExpansionSetting Up the Development EnvironmentManaging ConfigurationGenericOptionsParser, Tool, and ToolRunnerWriting a Unit Test with MRUnitMapperReducerRunning Locally on Test DataRunning a Job in a Local Job RunnerTesting the DriverRunning on a ClusterPackaging a JobLaunching a JobThe MapReduce Web UIRetrieving the ResultsDebugging a JobHadoop 65167168172Table of Contents vii

Remote DebuggingTuning a JobProfiling TasksMapReduce WorkflowsDecomposing a Problem into MapReduce JobsJobControlApache Oozie1741751751771771781797. How MapReduce Works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185Anatomy of a MapReduce Job RunJob SubmissionJob InitializationTask AssignmentTask ExecutionProgress and Status UpdatesJob CompletionFailuresTask FailureApplication Master FailureNode Manager FailureResource Manager FailureShuffle and SortThe Map SideThe Reduce SideConfiguration TuningTask ExecutionThe Task Execution EnvironmentSpeculative ExecutionOutput 971982012032032042068. MapReduce Types and Formats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209MapReduce TypesThe Default MapReduce JobInput FormatsInput Splits and RecordsText InputBinary InputMultiple InputsDatabase Input (and Output)Output FormatsText OutputBinary Outputviii Table of Contents209214220220232236237238238239239

Multiple OutputsLazy OutputDatabase Output2402452459. MapReduce Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247CountersBuilt-in CountersUser-Defined Java CountersUser-Defined Streaming CountersSortingPreparationPartial SortTotal SortSecondary SortJoinsMap-Side JoinsReduce-Side JoinsSide Data DistributionUsing the Job ConfigurationDistributed CacheMapReduce Library ClassesPart 79Hadoop Operations10. Setting Up a Hadoop Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283Cluster SpecificationCluster SizingNetwork TopologyCluster Setup and InstallationInstalling JavaCreating Unix User AccountsInstalling HadoopConfiguring SSHConfiguring HadoopFormatting the HDFS FilesystemStarting and Stopping the DaemonsCreating User DirectoriesHadoop ConfigurationConfiguration ManagementEnvironment SettingsImportant Hadoop Daemon 93294296Table of Contents ix