WIXLIB

Big data technology patternsEmerging patterns we did not findWE HAVE SEEN EXAMPLES OF THESE THREE PATTERNS BUT DID NOT FIND PRODUCTIONEXAMPLES THAT MET OUR CRITERIA FOR THIS TOOLKITOur criteria for this research was that we could speak with a user that has a production solution.We may update this toolkit in the future with examples of these three immature patterns as wefind firms willing to talk with us.DescriptionTechnology considerationsStandalonepackageBuy a packaged big data analytic tool to meetdepartment needs rapidly. Uses are generallylimited to the capabilities of the tools. Most arefocused on customer intelligence andmarketing use cases.Examples of packaged big data applications: KXEN, nPario,and NGDataStreaming analyticsA streaming analytics package solution isdeployed to capture and analyze high-velocitydata  as  it  “streams”  through  the  system.Distributed data hub options: none initially, but may add later aspart of path to hub-and-spokeStreaming package examples: IBM InfoSphere Streams,SQLstream, Apache S4, and StormHub-and-spoke plusin-memoryVery early pattern emerging that extends huband-spoke with in-memory and with an elasticcaching or data grid technology to provide veryhigh performance, embedded, or interactiveanalytics without using a BI DBMSHub options are the same as for the hub-and-spoke pattern.In-memory data grid platforms examples: Platfora, ScaleOutSoftware, Tibco SoftwareNote: The June 12, 2013,  “Deliver  On  Big  Data  Potential  With  A  Hub-And-Spoke  Architecture”  Forrester reportdefines the seven patterns, but three of them are immature or nascent without production examples we could find.The four patterns on the previous slide are covered in detail in this toolkit. 2013 Forrester Research, Inc. Reproduction Prohibited14

Big data technology patternsBasic pattern building blocksWE IDENTIFIED SIX BUILDING BLOCKS IN THE PATTERNSBuilding blockDescriptionConsiderationsDistributed datahubThe center of the architecture;provides a low-cost data persistencecapability that meets minimumrequirements for availability, security,and recovery, while exposing data forlow-level transformation and analyticsTechnology choices: Hadoop, other NoSQL, open source, orvendor supported, use of advanced technologies such as inmemory data grids, incorporation of mainframe data,integrated unstructured and structured data platforms, loadingand disposal processes, harmonization standards, metadata,cloud versus dedicated optionsIncludes both contextual servicessuch as data quality, master datamanagement, metadata andmodeling, and delivery services suchas federation/virtualization,transformation, movement, andsecurity services that operate on thehub and on the spokesMaster data management strategy and technology choices,integration approach and technology choices, level of quality,service performance, service availability to hub or spokes,vertical and horizontal scaling, cloud service utilizationHigh-performance BI databaseappliances and/or homegrown datawarehouse solutions that areappropriately spokes. Provides highavailability, low latency SQLanalytics.Enterprise versus departmental implementations, data storagecost, analytics requirements for latency and user access, BItool integrations, loading technology and performance, skills ofusers, volume and velocity projects versus tool performancecharacteristics. Examples: Greenplum, Netezza, TeradataAster, and Exadata.HubData servicesSVCEnterprise datawarehouse/departmentalBI databasesDW/BI DBNote: These building blocks emerged from our assessment of the big data implementations. 2013 Forrester Research, Inc. Reproduction Prohibited15

Big data technology patternsBasic pattern building blocks (cont.)WE IDENTIFIED SIX BUILDING BLOCKS IN THE PATTERNSBuilding blockBig dataanalyticspackagesBI and analyticspackagesData ionConsiderationsPackages applications that providedata operations and analytic tools thatinteract directly with hub dataIntegration with NoSQL vendors, needs and skills of users,volume and velocity projects versus tool performancecharacteristics. Examples: Datameer, Pentaho*Traditional business intelligence andanalytics packages that do notaccess, analyze, or operate on datain the hub. Instead they accessprocessed data in an operational orstructured analytic data store.What packages to buy for the functionality needed, how it getssupported, how data is sourced into the package. Examplesinclude BusinessObjects, Cognos, Tableau Software,QlikView. As more vendors add Hadoop integration to theircapabilities, the distinction between these tools and big dataanalytics packages will blur.Tools used by data scientists toexplore, manage hub data, stage fordata mining, and model development,management, and deploymentType of operations, data requirements, departmental versusenterprise team, sandbox and staging area needs, model toapplication integration, model to BI DB integration, analyticand exploration tool needs, operating procedures, security.Example technology: SPSS, R, SAS, Mahout, MapReduce*Note:Pentaho can function as a big data analytics package ora BI and analytics package, depending on how it’s  employed.Note: These building blocks emerged from our assessment of the big data implementations. 2013 Forrester Research, Inc. Reproduction Prohibited16

KeyWE USE NUMBERED CIRCLES TO MAP HUB-AND-SPOKE COMPONENTS TO PATTERNS ANDEXAMPLES1 data hub2 data services3 enterprise data warehouse/departmental BI database4 big data aware analytics packages5 standalone BI and analytics packages6 data science workbenchHub 2013 Forrester Research, Inc. Reproduction ProhibitedSVCDW/BI DBBigdataStandaloneWorkbench17

Hub-and-spoke architectureTHE BUILDING BLOCKS CREATE A HUB-AND-SPOKE DATA MANAGEMENT ARCHITECTUREOperational systems (afew examples)6Data science workbench facilitates dataexploration and discoveryMany datawarehouses and BIdatabases moved outto spokes.Traditional extract,transfer, load (ETL) indata warehousessupports quality andstructure needs.53253153Extract-load-transform (notETL!) means data istransformed and loadedinto  “spokes”  wheneverappropriate.534Some datawarehouse and BI DBappliances have builtin Hadoop integration,so they may beconsidered part of thehub or a spoke.Big-data-aware BI packages can operateagainst schema-less data in the hub directly;standard BI packages operate against relationaldata warehouses and BI databases.In our June 12, 2013, “Deliver  On  Big  Data  Potential  With  A  Hub-And-Spoke  Architecture”  report, we present a more abstract picture of thehub-and-spoke. This diagram reduces that picture to a more concrete level.123456 2013 Forrester Research, Inc. Reproduction ProhibitedHubSVCDW/BI DBBigdataStandaloneWorkbench18

Pattern: enterprise data warehouse augmentationPattern: enterprise data warehouseaugmentationPrimary purpose: make existing data warehouse environment more cost effectiveSecondary purpose: add more data and conduct rapid analysis in the hubExamples:Pharmaceutical  company . 21-23Wealth  management  firm  (financial  services) .24-26 2013 Forrester Research, Inc. Reproduction Prohibited19

Pattern: enterprise data warehouse augmentationEnterprise data warehouse augmentation patternUsers employ the same BI tools they are used to. Note: some BItools have integrations with data hub platforms, others do not andneed an intermediary such as a data virtualization layerThe main feature of this pattern is that some datawarehouse loads are rerouted to use the big datahub’s  data  services.or452 or 413VirtualizationThe dirty DW or operational datastore (ODS) contains lightlyharmonized data that can bequeried using structured query orvia an API (e.g., HBase). 2013 Forrester Research, Inc. Reproduction ProhibitedCan be used as a dataservice to expediteintegration of data notsourced to the hub.Virtualization supportstraditional, non-big-dataaware, BI tools.123456HubSVCDW/BI DBBigdataStandaloneWorkbench20

Pattern: enterprise data warehouse augmentationExample: pharmaceutical companyEXAMPLE — ENTERPRISE DATA WAREHOUSE AUGMENTATION PATTERNNeed›››Regulations (e.g., HIPAA)require healthcare orgs to storeelectronic data interchange(EDI) data for extended periodsof timeTrouble meeting seven-yeardata retention requirementwhile processing millions ofclaims every dayExisting system was storingdata as character large objectsin clustered, high-availabilityrelational databasemanagement system (RDBMS)Solution›››››Implemented Hadoop(Cloudera) as an augmentationto existing data warehousesolutions for EDI data archivalImplemented customeringestion process thataggregates all EDI files for theday and loads HadoopResults››Ten times lower total cost ofownership (TCO) whileenabling analytics on storeddataImplemented Hadoop solutionfor about half the cost of otheroptionsParses files into HBase forfaster data accessAs part of ingestion processing,does some data enrichment tosupport downstream analyticsUsed Flume to ingest data fromother transactional systemsDevelopment: four months (initial)Note: The pharmaceutical company appears twice, illustrating a firm initially pursuing one pattern then evolving tohub-and-spoke. 2013 Forrester Research, Inc. Reproduction Prohibited21

Pattern: enterprise data warehouse augmentationPharmaceutical company — conceptual solutionarchitectureWrote custom loader toaggregate one day of data inAvrio file formatData operations andharmonization usingMapReduce and loading toHBaseThe combination of enterpriseanalytics and a data warehouse is stillused but augmented with Hadoop,initially for lower-cost data retention25SqoopCustom Inges onLEGACY FILE BASED SYSTEMS1Flume (1TB/day)SqoopPAYER DATA, TRANSACTIONALSYSTEMS, ETC.Chose to learn low-levelMapReduce coding versususing a packaged toolSource: pharmaceutical company 2013 Forrester Research, Inc. Reproduction ProhibitedORACLE,IBM NETEZZAThis part is addressed the inhub-and-spoke pattern, slides63 to 65.123456HUBSVCDW/BI DBBigdataStandaloneWorkbench22

Pattern: enterprise data warehouse augmentationVendor informationCLOUDERASource: Cloudera 2013 Forrester Research, Inc. Reproduction Prohibited23

Pattern: enterprise data warehouse augmentationExample: wealth management firm (financial services)EXAMPLE — ENTERPRISE DATA WAREHOUSE AUGMENTATIONNeed›››This capital investment andwealth management firm hadstrong business demand forrisk data from many differentsystems.IT was taking a month or two toproduce new reports.There was no way to get all theinformation. Capturing all thehistorical trade data would havecost millions using a“traditional”  approach.Solution›››››››Implement Apache Hadoopdistribution (pure open source);a  “do  it  yourself”  approachLoad historical trade dataIntegrate this data with othersystems via data virtualizationExpose to existing BI toolsChose this as opposed to usingits existing data warehouseConsidering changing over to asupported Hadoop distributionand adding more tools (likeHBase)Results›››››More than 100 million recordsin Hadoop todayImplemented at a fraction of thecost of a relational databaseapproachCan produce reports in daysBusiness can access big datain small chunks for self-serviceanalytics using SpotfireTremendous data growth,expect over 1 PB next yearImplemented with three fulltime equivalents (FTEs)internallyDevelopment: three FTEs 2013 Forrester Research, Inc. Reproduction Prohibited24

Wealth management firm — conceptual solutionarchitectureVisualization/analysis toolStatisticalmodeling5Custom analysisjobs6ViewViewExample of how avirtualization serviceexposes big data in smallviews to other toolsViewData virtualizationWebservice2Hadoop and extreme-scale ELTused as a cost-effective way toaugment the data warehousewith cost-effective datapersistence and ubSVCDW/BI DBBigdataStandaloneWorkbenchFor  detailed  discussion  of  the  impact  data  virtualization  is  having  on  firm’s  data  architectures,  see  the  June  15,  2011,“Data  Virtualization  Reaches  Critical  Mass”  Forrester  report. 2013 Forrester Research, Inc. Reproduction Prohibited25

Pattern: enterprise data warehouse augmentationVendor informationCOMPOSITE gementGovernance, risk,and complianceHuman capitalmanagementMergers andacquisitionsSingle view ofenterprise dataSupply chainmanagementSAP dataintegrationComposite data virtualization platformDevelopmentenvironmentRuntime agerComposite information serverStudioMonitorPerformance plusadaptersActive clusterXMLPackaged appsRDBMSExcel filesData warehouseOLAP cubesHadoop/big dataXML docsFlat filesWeb servicesSource: Composite Software 2013 Forrester Research, Inc. Reproduction Prohibited26

Pattern: enterprise data warehouse augmentationForrester’s  point  of  view› The enterprise data warehouse augmentation pattern is the easiest to fund when big data isperceived  as  an  “IT  thing.”› The benefits are tangible and immediately realized, the business impacts are manageable, andthe upside is huge.› We suggest: Start by doing a five-year TCO calculation on all data in your data warehouse or data mart environments.Include the cost of integrating all that data. Do an analysis of how much data in your data warehouse environment has no or low analytic usage. Determine if any of this cold data has retention requirements that drive its storage. Develop a five-year TCO for an open source distributed data hub. See if a business case can be made.› Your biggest technology strategic decisions are: How to enable analytics on data in the hub. Data virtualization and BI tools with big data tool integrationcapability can help. What distributed data hub technology to choose, and how to acquire the skills. The approach to data movement and the level to which hub data is harmonized. 2013 Forrester Research, Inc. Reproduction Prohibited27

Pattern: enterprise data warehouse augmentationLessons learned from users› Dealing with raw data is messy. This is a new way of thinking. You need a data harmonization andintegration approach that delivers a minimum quality level. Think minimum viable quality, notcompletely clean data. An enterprise data model is essential for semantic consistency, but thedata  doesn’t  have  to  conform  completely  to  the  model  to  be  useable  — this is one reason big datadelivers hyperflexibility.› Data access will be challenging, both politically and technically. Ensure your strategy andbusiness case is strong enough to overcome these challenges. Be sure you have support fromthe top to overcome parochial concerns. Define who owns the data once you have sourced andharmonized it.› Tool selection makes all the difference. The devil is in the details — understand  what  “integrateswith  Hadoop”  really  means  in  terms  of  specific  versions  of  Hadoop  components,  the  specificintegration functionality, and quality of community and vendor support available. 2013 Forrester Research, Inc. Reproduction Prohibited28

Pattern: data refinery plus DW / BI DBMSPattern: data refinery plus DW / BIDBMSPrimary purpose: lower the cost of data capture and operations while loading astructured business intelligence database for low latency structured analyticsExamples:edo  interactive . . .31-33Vestas  (manufacturing)   . . .34-36NK  (social  media)   . .37-39Opera  Solutions  (IT)   . 40-42Rubicon  (digital  marketing)   .43-45Razorfish  (digital  marketing)   .46-48 2013 Forrester Research, Inc. Reproduction Prohibited29

Pattern: data refinery plus DW / BI DBMSData refinery plus DW / BI DBMSData science work is primarily developingmodels and attributes for deployment toembedded analytics and BI DBMS.The primary feature of this pattern is theloading of an EDW or other BI DBMS froma distributed hub.6Distributed data hub anddata  services’  primarypurpose is extreme-scaledata operations.Sourcesystems5Raw data is typicallypurged after harmonizationand archival.2 or 4231Sourcesystems 2013 Forrester Research, Inc. Reproduction ProhibitedData may be archivedafter harmonization, butthe BI DBMS remains theprimary place foranalytics.123456HubSVCDW/BI DBBigdataStandaloneWorkbench30

Pattern: data refinery plus DW / BI DBMSExample: edo interactiveNeed››››B2B electronic marketing firmwas projecting rapid datagrowth, needed to rethinkinfrastructure. Currentlyproviding 120 million offers amonth and more than 25 milliontransactions a day, producingas much as 50 terabytes ofdata and growing.Wanted an affordable and easyto use extract, load, andtransform tool (not ETL) tohandle massive scaleaffordablySolution›››Implemented a Hadoop(Cloudera 3) data refineryplatform to load massiveamounts of data t

An enterprise data warehouse (EDW) remains the locus of analytic data architecture, but cold data is offloaded to hub. High-volume data that is not cost effective to move into a warehouse is added and analyzed using new or existing tools, but primary analytics remain in the data warehouse and marts. D