WIXLIB

Practical HiveA Guide to Hadoop’sData Warehouse SystemScott ShawAndreas François VermeulenAnkur GuptaDavid Kjerrumgaard

Practical Hive: A Guide to Hadoop's Data Warehouse SystemScott ShawSaint Louis, Missouri, USAAndreas François VermeulenWest Kilbride North Ayrshire, United KingdomAnkur GuptaUxbridge, United KingdomDavid KjerrumgaardHenderson, Nevada, USAISBN-13 (pbk): 978-1-4842-0272-2DOI 10.1007/978-1-4842-0271-5ISBN-13 (electronic): 978-1-4842-0271-5Library of Congress Control Number: 2016951940Copyright 2016 by Scott Shaw, Andreas François Vermeulen, Ankur Gupta, David KjerrumgaardThis work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the materialis concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting,reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval,electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafterdeveloped. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarlyanalysis or material supplied specifically for the purpose of being entered and executed on a computer system,for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted onlyunder the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for usemust always be obtained from Springer. Permissions for use may be obtained through RightsLink at the CopyrightClearance Center. Violations are liable to prosecution under the respective Copyright Law.Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with everyoccurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashionand to the benefit of the trademark owner, with no intention of infringement of the trademark.The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identifiedas such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.While the advice and information in this book are believed to be true and accurate at the date of publication, neitherthe authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that maybe made. The publisher makes no warranty, express or implied, with respect to the material contained herein.Managing Director: Welmoed SpahrAcquisitions Editor: Robert HutchinsonDevelopmental Editor: Matt MoodieTechnical Reviewer: Ancil McBarnett, Chris HillmanEditorial Board: Steve Anglin, Pramila Balen, Laura Berendson, Aaron Black, Louise Corrigan,Jonathan Gennick, Robert Hutchinson, Celestin Suresh John, Nikhil Karkal, James Markham,Susan McDermott, Matthew Moodie, Natalie Pao, Gwenan SpearingCoordinating Editor: Rita FernandoCopy Editor: Kezia EndsleyCompositor: SPi GlobalIndexer: SPi GlobalCover Image: Designed by FreePikDistributed to the book trade worldwide by Springer Science Business Media New York, 233 Spring Street,6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com,or visit www.springer.com. Apress Media, LLC is a California LLC and the sole member (owner) is SpringerScience Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.For information on translations, please e-mail rights@apress.com, or visit www.apress.com.Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use.eBook versions and licenses are also available for most titles. For more information, reference our Special BulkSales–eBook Licensing web page at www.apress.com/bulk-sales.Any source code or other supplementary materials referenced by the author in this text is available to readers atwww.apress.com. For detailed information about how to locate your book’s source code, go towww.apress.com/source-code/.Printed on acid-free paper

I dedicate this book to my family. They put up with me being on the computer everydayand yet they have no idea what I do for a living. Love you!—Scott ShawI dedicate this book to my family and wise mentors for their support. Special thanksto Denise and Laurence.—Andreas François VermeulenI would like to express my gratitude to the many people who saw me through this book.Above all I want to thank my wife, Jasveen, and the rest of my family, who supported andencouraged me in spite of all the time it took me away from them.—Ankur Gupta“By perseverance, study, and eternal desire, any man can become great.” —George S. Patton—David Kjerrumgaard

Contents at a GlanceAbout the Authors.xvAbout the Technical Reviewers .xviiAcknowledgments .xixIntroduction .xxi Chapter 1: Setting the Stage for Hive: Hadoop . 1 Chapter 2: Introducing Hive . 23 Chapter 3: Hive Architecture . 37 Chapter 4: Hive Tables DDL . 49 Chapter 5: Data Manipulation Language (DML) . 77 Chapter 6: Loading Data into Hive . 99 Chapter 7: Querying Semi-Structured Data . 115 Chapter 8: Hive Analytics . 133 Chapter 9: Performance Tuning: Hive . 219 Chapter 10: Hive Security . 233 Chapter 11: The Future of Hive . 245 Appendix A: Building a Big Data Team . 249 Appendix B: Hive Functions . 253Index . 263v

ContentsAbout the Authors.xvAbout the Technical Reviewers .xviiAcknowledgments .xixIntroduction .xxi Chapter 1: Setting the Stage for Hive: Hadoop . 1An Elephant Is Born . 2Hadoop Mechanics . 3Data Redundancy . 6Traditional High Availability. 6Hadoop High Availability . 8Processing with MapReduce . 11Beyond MapReduce . 16YARN and the Modern Data Architecture . 17Hadoop and the Open Source Community . 18Where Are We Now . 22 Chapter 2: Introducing Hive . 23Hadoop Distributions . 24Cluster Architecture. 26Hive Installation . 29Finding Your Way Around . 31Hive CLI . 34vii

CONTENTS Chapter 3: Hive Architecture . 37Hive Components . 37HCatalog . 38Hiveserver2 . 41Client Tools . 43Execution Engine: Tez . 46 Chapter 4: Hive Tables DDL . 49Schema-on-Read . 49Hive Data Model . 50Schemas/Databases . 50Why Use Multiple Schemas/Databases . 50Creating Databases . 50Altering Databases . 51Dropping Databases . 51List Databases . 52Data Types in Hive . 52Primitive Data Types . 52Choosing Data Types. 52Complex Data Types . 53Tables . 54Creating Tables . 55Listing Tables . 55Internal/External Tables . 56Internal or Managed Tables . 56External/Internal Table Example . 57Table Properties . 61Generating a Create Table Command for Existing Tables . 62Partitioning and Bucketing . 62Partitioning Considerations . 64Efficiently Partitioning on Date Columns . 65viii

CONTENTSBucketing Considerations . 66Altering Tables . 68ORC File Format . 69Altering Table Partitions. 70Modifying Columns . 74Dropping Tables/Partitions. 74Protecting Tables/Partitions . 75Other Create Table Command Options . 75 Chapter 5: Data Manipulation Language (DML) . 77Loading Data into Tables . 77Loading Data Using Files Stored on the Hadoop Distributed File System . 78Loading Data Using Queries . 80Writing Data into the File System from Queries . 83Inserting Values Directly into Tables . 85Updating Data Directly in Tables . 86Deleting Data Directly in Tables . 88Creating a Table with the Same Structure . 89Joins . 90Using Equality Joins to Combine Tables . 90Using Outer Joins . 91Using Left Semi-Joins . 94Using Join with Single MapReduce . 95Using Largest Table Last . 96Transactions . 97What Is ACID and Why Use It? . 97Hive Configuration . 97 Chapter 6: Loading Data into Hive . 99Design Considerations Before Loading Data . 99Loading Data into HDFS. 100Ambari Files View . 100Hadoop Command Line. 102ix

CONTENTSHDFS NFS Gateway. 102Sqoop. 103Apache Nifi . 106Accessing the Data in Hive . 111External Tables . 111Load Data Statement . 112Loading Incremental Changes in Hive . 113Hive Streaming . 113Summary . 114 Chapter 7: Querying Semi-Structured Data . 115Clickstream Data . 117Ingesting Data . 119Creating a Schema . 122Loading Data. 123Querying the Data . 123Ingesting JSON Data . 126Querying JSON with a UDF . 128Accessing JSON Using a SerDe . 129 Chapter 8: Hive Analytics . 133Building an Analytic Model . 133Getting Requirements Using Sun Models . 133Converting Sun Models to Star Schemas . 138Building the Data Warehouse . 149Assessing an Analytic Model . 152Assess the Sun Models. 152Assess the Aggregations . 155Assess the Data Marts . 155Master Data Warehouse Management . 156Prerequisites. 157Retrieve Database . 157x

CONTENTSAssess Database . 160Process Database . 175Transform Database . 201What Have You Mastered . 209Organize Database . 209Report Database . 213Example Reports. 214Advanced Analytics . 216What’s Next? . 217 Chapter 9: Performance Tuning: Hive . 219Hive Performance Checklist . 219Execution Engines . 220MapReduce. 220Tez . 220Storage Formats . 222The Optimized Row Columnar (ORC) Format . 222The Parquet Format . 224Vectorized Query Execution. 225Query Execution Plan . 226Cost-Based Optimization . 227The Execution Plan . 230Performance Checklist Summary . 232 Chapter 10: Hive Security . 233Data Security Aspects . 233Authentication . 234Authorization . 234Administration . 234Auditing . 234Data Protection . 234xi

CONTENTSHadoop Security . 235Hive Security . 235Default Authorization Mode . 235Storage-Based Authorization Mode . 236SQL Standards-Based Authorization Mode . 237Managing Access through SQL . 238Hive Authorization Using Apache Ranger . 239Accessing the Ranger UI. 240Creating Ranger Policies . 240Auditing Using Apache Ranger . 243 Chapter 11: The Future of Hive . 245LLAP (Live Long and Process) . 245Hive-onSpark. 246Hive: ACID and MERGE. 246Tunable Isolation Levels . 246ROLAP/Cube-Based Analytics. 247HiveServer2 Development . 247Multiple HiveServer2 Instances for Different Workloads . 247 Appendix A: Building a Big Data Team . 249Minimum Team . 249Executive Team . 249Business Team . 249Technical Team . 250Expanded Team . 250Business Team . 250Technical Team . 251Work Lifecycle for the Team . 252xii

CONTENTS Appendix B: Hive Functions . 253Built-In Functions . 253Mathematical Functions .

Practical Hive: A Guide to Hadoop's Data Warehouse System Scott Shaw Andreas François Vermeulen Saint Louis, Missouri, USA West Kilbride North Ayrshire, United Kingdom Ankur Gupta David Kjerrumgaard Uxbridge, United Kingdom Henderson, Nevada, USA ISBN-13 (pbk): 978-1-4842-0272-2 ISBN-13 (electronic): 978-1-4842-0271-5