WIXLIB

Data Management for Data ScienceCorso di laurea magistrale in Data ScienceSapienza Università di Roma2015/2016An Introduction to Big DataDomenico LemboDipartimento di Ingegneria Informatica Automatica e Gestionale A. Ruberti

Availability of Massive Data Digital data are nowadays collected at an unprecedentscale and in very many formats in a variety of domains(e-commerce, social networks, sensor networks,astronomy, genomics, medical records, etc.) This is has been made possible by the incredible growthof the last years of the capacity of data storage tools, andof the computing power of electronic devices, and aswell as the advent of mobile and pervasive computing,cloud computing and cloud storage.Introduction to Big Data2

Exploitability of Massive Data How to transform available data into information, andhow to make organizations’ business to takeadvantages of such information are long-standingproblems in IT, and in particular ininformation managementand analysis. These issues have becomemore and more challengingand complex in the “Big Data” era At the same time, facing the challenge may be worthy,since the massive amount of data that is now availablemay allow for analytical results never achieved beforeIntroduction to Big Data3

but be careful! “Big data is a vague term for amassive phenomenon that hasrapidly become an obsession withentrepreneurs, scientists,governments and the media” (TimHarford, journalist and economist)** 00144feabdc0.html#axzz3EvSLWwbuMoore's Law for #BigData: The amount of nonsense packedinto the term "BigData" doubles approximately every twoyears (Mike Pluta, Data Architect on Twitter August 2014).Introduction to Big Data4

Thinking Big Data*"Big Data" has leapt rapidly into one of the most hyped terms in our industry,yet the hype should not blind people to the fact that this is a genuinelyimportant shift about the role of data in the world. The amount, speed, andvalue of data sources is rapidly increasing. Data management has to changein five broad areas: extraction of data from a wider range of sources, changesto the logistics of data management with new database and integrationapproaches, the use of agile principles in running analytics projects, anemphasis on techniques for data interpretation to separate signal from noise,and the importance of well-designed visualization to make that signal morecomprehensible. Summing up this means we don't need big analytics projects,instead we want the new data thinking to permeate our regular work.”Martin troduction to Big Data5

Thinking Big Data Thus, roughly, Big Data is data that exceeds the processing capacityof conventional database systems But also Big Data is understood as a capability that allowscompanies to extract value from large volumes of data but, notice, this does not mean only extremely large, massivedatabases Besides data dimension, what characterizes Big Data are also theheterogeneity in the way in which information is structured, thedynamicity with which data changes, is the ability of quicklyprocessing it This calls for new computing paradigms or frameworks, not onlyadvanced data storage mechanismsIntroduction to Big Data6

The Three VsTo characterize Big Data, three Vs are used,which are the Vs of– Volume– Velocity– VarietyIntroduction to Big Data7

Volume Big data applications are characterized of course by big amounts ofdata, where big means extremely large, e.g., more than a terabyte(TB) or petabyte (PB), or more. Some examples:– Walmart: 1 million transaction per hour (2010)1– eBay: data throughput reaches 100 petabytes per day (2013)2– Facebook: 40 billion photos (2010)1; 250PB data warehousewith 600TB added to the warehouse every day (2013)3– 500 millions of tweet per day (in 2013)– And very many other examples, as chatters from socialnetworks, web server logs, traffic flow sensors, satellite imagery,broadcast audio streams, banking transactions, GPS trails,financial market data, biological data, www.theregister.co.uk/2013/06/07/hey presto facebook reveals exabytescale query engine/Introduction to Big Data8

Volume How many data in the world?–––––800 Terabytes, 2000160 Exabytes, 2006 (1EB 1018B)500 Exabytes, 20092.7 Zettabytes, 2012 (1ZB 1021B)35 Zettabytes by 2020 90% of world’s data generated in the last two years.Introduction to Big Data9

Volume In a data integration context, the number of sourcesproviding information can be huge too, much higherthan the number considered in traditional dataintegration and virtualization systems The sheer volume of data is enough to defeat manylong-followed approaches to data management Traditional centralized database systems cannothandle many of the data volumes, forcing the use ofclustersIntroduction to Big Data10

Velocity Data velocity (i.e., the rate at which data is collectedand made available into an organization) hasfollowed a similar pattern to that of volume Many data sources accessed by organizations fortheir business are extremely dynamic Mobile devices increase the rate of data inflow: data“everywhere”, collected and consumed continuouslyIntroduction to Big Data11

Velocity Processing information as soon as it is available, thus speedingthe “feedback loop”, can provide competitive advantages As an example, consider online retails that are able to suggestadditional products to a customer at every new informationinserted during an on-line purchase Stream processing is a new challenging computing paradigm,where information is not stored for later batch processing, butis consumed on the fly This is particularly useful when data are too fast to store thementirely (for example because they need some processing tobe stored properly), as in scientific applications, or when theapplication requires an immediate answerIntroduction to Big Data12

Variety Data is extremely heterogeneous: e.g., in the format inwhich are represented, but also and in the way theyrepresent information, both at the intensional andextensional level E.g., text from social networks, sensor data, logs fromweb applications, databases, XML documents, RDF data,etc. Data format ranges therefore from structured (e.g,relational databases) to semistructured (e.g., XMLdocuments), to unstructured (e.g., text documents)Introduction to Big Data13

Variety As for unstructured data, for example, the challengeis to extract ordered meaning for consumption bothby humans or machines Entity resolution, which is the process that resolves(i.e., identifies) entities and detects relationships,then plays an important role In fact, these are well-known issues studied sinceseveral years in the fields of data integration, dataexchange and data quality. In the Big Data scenario,however, they become even more challengingIntroduction to Big Data14

A fourth V: Veracity* Data are of widely different quality Traditionally data is thought of as coming from well organizeddatabases with controlled schemas Instead, in “Big Data” there is often little or no schema tocontrol its structure The result is that there are serious problems with the qualityof the data* The literature often mentions only three Vs and does not include veracity.However some authors tend to include veracity as a core characteristc ofBig Data (in the othe cases, veracity is considered an aspect of variety)Introduction to Big Data15

Big Data:3V ValueBig Data can generate huge competitiveadvantages!Introduction to Big Data16

The value of Data for organizations Although it is difficult to get hard figures on the value of makingfull use of your data, much of the success of companies such asAmazon and Google is credited to their effective use of data1 Thus companies spend large amounts of money to reach thiseffective use: International Data Corporation (IDC) forecaststhat the worldwide Big Data technology and services marketwill grow at a 31.7% compound annual growth rate – aboutseven times the rate of the overall ICT market – with revenuesreaching 23.8 billion in 20162 Thus various Big Data solutions are now promoted by all majorvendors in data management dex.jspIntroduction to Big Data17

Potential valueIntroduction to Big Data18

Demand for new data management solutions* Despite the popularity and well understood nature ofrelational databases, it is not the case that they shouldalways be the destination for data Depending on the characteristic of data, certain classesof databases are more suited than others for theirmanagement XML documents are more versatile when stored indedicated XML store (e.g., MarkLogic) Social network relations are graph by nature and graphdatabases such as Neo4J can make operations on themsimpler and more efficient* From: Edd Dumbill. What is Big data. In Planning for Big Data. O’Reilly Radar TeamIntroduction to Big Data19

Demand for new data management solutions* A disadvantage of the relational database is the staticnature of its schema In an agile environment, the results of computationwill evolve with the detection and extraction of newinformation Semi-structure NoSQL databases meet this need forflexibility: they provide some structure to organizedata (enough for certain applications), but do notrequire the exact schema of the data before storing it* From: Edd Dumbill. What is Big data. In Planning for Big Data. O’Reilly Radar TeamIntroduction to Big Data20

NoSQL databases*Or better not only SQL The term "NoSQL" is very ill-defined. It's generallyapplied to a number of recent non-relational databasessuch as Cassandra, Mongo, Dynamo, Neo4J, Riak, andmany others They embrace schemaless data, run on clusters, and havethe ability to trade off traditional consistency for otheruseful properties Advocates of NoSQL databases claim that they can buildsystems that are more performant, scale much better,and are easier to program with.* From: Martin Fowler. NoSQL Distilled. Introduction to Big Data21

Graph databasesIntroduction to Big Data22

Key-values databasesIntroduction to Big Data23

Document databasesIntroduction to Big Data24

Column Family DatabasesIntroduction to Big Data25

NoSQL databases* Is this the first rattle of the death knell for relationaldatabases, or yet another pretender to the throne? Ouranswer to that is "neither” Relational databases are a powerful tool that we expect to beusing for many more decades, but we do see a profoundchange in that relational databases won't be the onlydatabases in use Our view is that we are entering a world of PolyglotPersistence where enterprises, and even individualapplications, use multiple technologies for data management* From: Martin Fowler. NoSQL Distilled. Introduction to Big Data26

Multiple technologies for data managementAs an exercise, let us ask google which is the databaseengine used by Facebook. We get the following tools1: MySQL as core database engine (in fact a customizedversion of MySQL, highly optimized and distributed) Cassandra (an Apache open source fault tolerantdistributed NoSQL DBMS, originally developed atFacebook itself) as database for the Inobx mail search Memcached, a memory caching system to speed updynamic database driven websites HayStack, for storage and management of photos Hive, an open source, peta-byte scale data warehousingframework based on Hadoop, for analytics, and alsoPresto, a recent exabyte scale odb.io/Introduction to Big Data27