WIXLIB

1SQL-on-Hadoop TutorialVLDB 2015SQL-on-Hadoop Tutorial16-09-14

Presenters2Fatma ÖzcanIBM ResearchDaniel AbadiYale University andTeradataIBM Big SQLSQL-on-Hadoop TutorialHadoopDB/HadaptIppokratis PandisClouderaShivnath BabuDuke UniversityCloudera ImpalaStarfish16-09-14

3Why SQL-on-Hadoop? People need to process data in parallel Hadoop is by far the leading open source parallel dataprocessing platform Low costs of HDFS results in heavy usageLots of data in Hadoop with appetite to process itSQL-on-Hadoop Tutorial16-09-14

MapReduce is not the answer4 MapReduce is a powerful primitive to do many kinds ofparallel data processing BUT Little control of data flow Fault tolerance guarantees not always necessary Simplicity leads to inefficiencies Does not interface with existing analysis software Industry has existing training in SQLSQL interface for Hadoop critical for mass adoptionSQL-on-Hadoop Tutorial16-09-14

5The database community knows howto process data Decades of research in parallel database systems Efficient data flow Load balancing in the face of skew Query optimization Vectorized processing Dynamic compilation of query operators Co-processing of queriesMassive talent war between SQL-on-Hadoopcompanies for members of database communitySQL-on-Hadoop Tutorial16-09-14

6SQL-on-Hadoop is not a directimplementation of parallel DBMSs Little control of storage Most deployments must be over HDFS Append-only file system Must support many different storage formats Avro, Parquet, RCFiles, ORC, Sequence Files Little control of metadata management Optimizer may have limited access to statistics Little control of resource management YARN still in its infancySQL-on-Hadoop Tutorial16-09-14

7SQL-on-Hadoop is not a directimplementation of parallel DBMSs Hadoop often used a data dump (swamp?) Data often unclean, irregular, and unreliable Data not necessarily relational HDFS does not enforce structure in the data Nested data stored as JSON extremely popular Scale larger than previous generation parallel databasesystems Fault tolerance vs. query performance Most Hadoop components written in Java Want to play nicely with the entire Hadoop ecosystemSQL-on-Hadoop Tutorial16-09-14

Outline of Tutorial8 This session [13:30-15:00] SQL-on-Hadoop Technologies Second Session [15:30-17:00] SQL-on-Hadoop examples Storage HadoopDB/Hadapt Run-time engine Presto Query optimization Impala Q&A BigSQL SparkSQL Phoenix/Spice Machine Research directionsSQL-on-Hadoop Tutorial Q&A16-09-14

9StorageSQL-on-Hadoop Tutorial16-09-14

10Quick Look at HDFSNameNodeDataNodeSQL-on-Hadoop TutorialDataNode DataNode16-09-14

11HDFS is Good for Storing large files Write once and read many times “Cheap” commodity hardware Not good for Low-latency reads Short-circuit reads and HDFS caching help Large amounts of small files Multiple writersSQL-on-Hadoop Tutorial16-09-14

12In-situ Data Processing HDFS as the data dump Store the data first, figure out what to do later Most data arrive in text format Transform, cleanse the data Create data marts in columnar formats Lost of nested, JSON data Some SQL in data transformations, but mostly otherlanguages, such as Pig, Cascading, etc. Columnar formats are good for analyticsSQL-on-Hadoop Tutorial16-09-14

13SQL-on-Hadoop according to storage formats Most SQL-on-Hadoop systems do not control or own the data Hive, Impala, Presto, Big SQL, Spark SQL, Drill Other SQL-on-Hadoop systems tolerate HDFS data, but workbetter with their own proprietary storage HadoopDB/Hadapt HAWQ, Actian Vortex, and HP VerticaSQL-on-Hadoop Tutorial16-09-14

14Query Processors with HDFS Native Formats Only support native Hadoop formats with opensource reader/writers Any Hadoop tool can generate their data Pig, Cascading and other ETL tools They are more of a query processor than a database Indexing is a challenge !! No co-location of multiple tables Due to HDFSSQL-on-Hadoop Tutorial16-09-14

15Systems with Proprietary Formats Almost all exploit some existing database systems They store their own binary format on HDFS Hadapt stores the data in a single node database,like postgres Can exploit Postgres indexes HAWQ, Actian, HP Vertica, and Hadapt all controlhow tables are partitioned, and can support colocated joinsSQL-on-Hadoop Tutorial16-09-14

16HDFS Native Formats CSV files are most common for ETL-like workloads Lots of nested and complex data Arrays, structs, maps, collections Two major columnar formats ORCFile Parquet Data serialization JSON and Avro Protocol buffers and ThriftSQL-on-Hadoop Tutorial16-09-14

17Parquet§ PAX format, supporting nested data§ Idea came from the Google‘s Dremel System§ Major contributors: Twitter & Cloudera§ Provides dictionary encoding and several compressions§ Preffered format for Impala, IBM Big SQL, and Drill§ Can use Thrift or Avro to describe the schemaColumnar storageNested data§ Fast compression§ Schema projection§ Efficient encoding§ A natural schema§ Flexible§ Less duplication applyingdenormalizationSQL-on-Hadoop Tutorial1716-09-14

18Parquet, cont. A table with N columns is splitinto M row groups. The file metadata containsthe locations of all thecolumn metadata startlocations. Metadata is written after thedata to allow for single passwriting. There are three types ofmetadata: file metadata,column (chunk) metadataand page header metadata. Row group metadataincludes Min-max values for skippingSQL-on-Hadoop Tutorial16-09-14

19ORCFile Second generation, following RC file PAX formats with all data in a single file Hortonworks is the major contributor, together with Microsoft Preferred format for Hive, and Presto Supports Dictionary encoding Fast compression File, and stripe level metadata Stripe indexing for skipping Now metadata even includes bloom filters for point query lookupsSQL-on-Hadoop Tutorial16-09-14

20ORCFile LayoutSQL-on-Hadoop Tutorial16-09-14

21Handling Updates in HDFS No updates in HDFS Appends to HDFS files are supported,but not clear how much they areused in productiondelta1 Updates are collected in delta filesdelta2 At the time of read delta and mainfiles are merged Special inputFormats Lazy compaction to merge deltafiles and main files When delta files reach a certain sizeFile A deltan Scheduled intervalsSQL-on-Hadoop Tutorial16-09-14

22SQL on NoSQL! Put a NoSQL solution on top of HDFS For the record, you can avoid HDFS completely But, this is a SQL-on-Hadoop tutorial NoSQL solutions can provide CRUD at scale CRUD Create, Read, Update, Delete And, then run SQL on it? Sounds crazy? Well, lets seeSQL-on-Hadoop Tutorial16-09-14

23HBase: The Hadoop Database Not HadoopDB, which we will see later in the tutorial HBase is a data store built on top of HDFS based on Google Bigtable Data is logically organized into tables, rows, and columns Although, Key-Value storage principles are used at multiple points in the design Columns are organized into Column Families (CF) Supports record-level CRUD, record-level lookup, random updates Supports latency-sensitive operations

24HBase Architecture

25HBase ArchitectureHBase stores three types of fileson HDFS: WALs HFiles Links

26HBase Read and Write Paths

27HFile FormatHFile Structure Immutable Created on flush or compaction Sequential writes Read randomly or sequentially Data is in blocks HFile blocks are not HDFS blocks Default data block size 64K Default index block size 128K Default bloom filter block size 128K Use smaller block sizes forfaster random lookup Use larger block sizes for faster scans Compression is recommended Block encoding is recommended

28Run-time EngineSQL-on-Hadoop Tutorial16-09-14

29Design Decisions: Influencers Low Latency High Throughput Degree of tolerance to faults Scalability in data size Scalability in cluster size Resource elasticity Multi-tenancy Ease of installation in existing environmentsSQL-on-Hadoop Tutorial16-09-14

30Accepted across SQL-on-Hadoop Solutions Push computation to data Columnar data formats Vectorization Support for multiple data formats Support for UDFsSQL-on-Hadoop Tutorial16-09-14

31Differences across SQL-on-Hadoop Solutions What is the Lowest Common Execution Unit Use of Push Vs. Pull On the JVM or not Fault tolerance: Intra-query or inter-query Support for multi-tenancySQL-on-Hadoop Tutorial16-09-14

32SQL on MapReduce Hive TenzingSQL-on-Hadoop Tutorial16-09-14

33HiveSQL-on-Hadoop Tutorial16-09-14

34Example: Joins in MapReduceSQL-on-Hadoop Tutorial16-09-14

35Limitations Having a MapReduce Job as the Lowest Execution Unitquickly becomes restrictive Query execution plans become MapReduce workflowsSQL-on-Hadoop Tutorial16-09-14

MapReduce Workflows36D01D02J2J1DatasetsMapReduce JobsD2D1J3D3J4D4J5D5D6J7D7J6

37Research Done to Address these Limitations On efficient joins in the MapReduce paradigm On reducing the number of MapReduce jobs bypacking/collapsing the MapReduce workflow Horizontally Shared scans Vertically Making using of static and dynamic partitioning On efficient management of intermediate dataSQL-on-Hadoop Tutorial16-09-14

38From MapReduce to DAGs Dryad TezSQL-on-Hadoop Tutorial16-09-14

39Dryad: Dataflows as First-class CitizensSQL-on-Hadoop Tutorial16-09-14

40Smart DAG Execution in DryadSQL-on-Hadoop Tutorial16-09-14

41Tez: Inspired by Dryad and Powered by YARNSQL-on-Hadoop Tutorial16-09-14

42Quick Detour on YARN The Hadoop Community realized thatMapReduce cannot be the LowestExecution Unit for all data apps Separated out the resourcemanagement aspects fromapplication management YARN is best seen as an OperatingSystem for Data Processing Apps Recall the 80s: Databases andOperating Systems: Friends or Foes?SQL-on-Hadoop Tutorial16-09-14

43An Example of What Tez EnablesSQL-on-Hadoop Tutorial16-09-14

44A Tez Slide on TezSQL-on-Hadoop Tutorial16-09-14

Spark: A Different Way to Look at a aveAsTextFile

Spark: A Different Way to Look at a DataflowStage ).saveAsTextFile(outPath)Stage lterreduceBykeymap saveAsTextFile

Spark: A Different Way to Look at a DataflowStage ).saveAsTextFile(outPath)Stage pfilterreduceBykeysaveAsTextFile

Spark: A Different Way to Look at a rt1part1part1part1part1part1Exec2Stage 1part2part2part2part2part2part2Exec3Stage apsaveAsTextFile

49Spark: A Different Way to Look at a DataflowSQL-on-Hadoop Tutorial16-09-14

50Fault ToleranceSQL-on-Hadoop Tutorial16-09-14

51MapReduce Fault apSQL-on-Hadoop e16-09-14

52MapReduce Fault apSQL-on-Hadoop e16-09-14

53MapReduce Fault apSQL-on-Hadoop e16-09-14

54MapReduce Fault apSQL-on-Hadoop e16-09-14

55MapReduce Fault apSQL-on-Hadoop e16-09-14

56MapReduce Fault apSQL-on-Hadoop e16-09-14

Fault TolerancePercentage Slowdown57200180160140120100806040200 SELECT sourceIP,SUM(adRevenue)FROM UserVisitsGROUP BY sourceIPTraditionalDBMSMapReduce Node fails (or slows downby factor of 2) in themiddle of queryFault toleranceSQL-on-Hadoop TutorialSlowdown tolerance16-09-14

58Downsides of MapReduce FaultToleranceMapMap outputwritten to diskReduce outputwritten to -Hadoop e16-09-14

59Spark RDDs Stores intermediate results in memory rather than disk Advantage: Performance Disadvantage: Memory requirementsSQL-on-Hadoop Tutorial16-09-14

60Resource ManagementSQL-on-Hadoop Tutorial16-09-14

61Resource Management (At least) Two dimension problem:1. RM across different frameworks Usually not a dedicated cluster Shared across multiple frameworks ETL (MapReduce, Spark), Hbase SQL-on-Hadoop processing2. RM across concurrent queriesSQL-on-Hadoop Tutorial16-09-14

62RM -- Across frameworks YARN – Yet Another Resource Negotiator Centralized, cluster-wide resource managementsystem Allows frameworks to share resources withoutpartitioning between them Designed for batch-mostly processing Not mature Not good for interactive analytics Not meant for long running processes Approaches: Llama and SliderSQL-on-Hadoop Tutorial16-09-14