WIXLIB

MIT Technology Review InsightsPartnerperspectiveDatabricks perspective:The rise of the lakehouse effectEvery company feels the pull to becomea data company, and they are placingincreasing importance on AI to deliver onthe tremendous business potential it canoffer. But, as indicated in this report,only 13% of organizations today are succeeding atdelivering on their enterprise data strategy. Data andanalytics leaders attribute much of their successto having a solid handle on data management basics.So why do so many others struggle?The technology ecosystem across data warehousesand data lakes further complicates the architecture.It ends up being expensive and resource-intensiveto manage. That complexity impacts data teams.Data and organizational silos can accidentlyslow communication, hinder innovation and createdifferent goals amongst the teams. The resultis multiple copies of data, no consistent security/governance model, closed systems, and lessproductive data teams.The challenge starts with the data architecture.The research suggests organizations need tobuild four different stacks to handle all of their dataworkloads: business analytics, data engineering,streaming, and ML. All four of these stacks requirevery different technologies and, unfortunately, theysometimes don’t work well together.Meanwhile, ML remains an elusive goal. With theemergence of lakehouse architecture, organizationsare no longer bound by the confines and complexityof legacy architectures. By combining theperformance, reliability, and governance of datawarehouses with the scalability, low cost, andworkload flexibility of the data lake, lakehouseContinued, next pageowing to data duplication and poor quality. This creates agood deal of complexity when it comes to datainfrastructure. The cloud, for all its game-changing impact,can also increase complexity as organizations continue tostore their data with multiple providers to hedge vendorlock-in, meet regional needs, or optimize for best-of-breedsolutions. And data architectures have evolved in arelatively short space of time so that organizations maysimultaneously be using on-premise databases, datawarehouses, data lakes, or other emerging dataarchitectures along with different cloud-based toolsperforming configuration, governance, or other functions.“Architectures have gotten really complicated, but onlybecause we tend to over-complicate them,” saysSol Rashidi, chief analytics officer at The Estée LauderCompanies. “We do this because we lose sight of whatmatters most. We too often bring in the latest and greatestin technology and platforms, thinking they will solve theproblem. But unless the business is ready to leverage thetools, has the maturity to extract the insights, andprocesses and logic are agreed upon, we’re only adding tothe spaghetti architecture.”If organizations are unable to manage the complexity, theconsequences are usually a combination of missedopportunities (in the failure of ML use cases to deliverreturns, for example), higher costs (such as fromadministering and supporting multiple overlappingsystems), difficulty meeting the growing regulatoryrequirements on data, and, ultimately, considerableexposure to competition.Nevertheless, as our research makes clear, enthusiasmand optimism outweigh any sense of frustration amongdata and technology leaders when it comes to theirpresent and future ability to manage data effectively fortheir business.7

8  MIT Technology Review InsightsPartnerperspectivearchitecture provides a flexible, high-performancedesign for diverse data applications—includingreal-time streaming, batch processing, SQLanalytics, data science, and ML.At Databricks, we bring the lakehouse architectureto life through the Databricks Lakehouse Platform.The key enabler behind this innovation is Delta Lake.Delta Lake is at the core of the platform, and itcreates curated data lakes that add reliability,performance, and governance from datawarehouses directly to the existing data lake.Organizations get a better grasp on enterprise-widedata management.It’s collaborative: Data engineers, analysts,and data scientists can work together and moreefficiently.The cost savings, efficiencies, and productivity-gainsoffered by the Databricks Lakehouse Platform arealready making a bottom-line impact on enterprisesin every industry and geography. Freed from overlycomplex architecture, Databricks provides onecommon cloud-based data foundation for all dataand workloads across all major cloud providers.Data and analytics leaders can foster a data-drivenculture that focuses on adding value by relieving thedaily grind of planning and all its complexities, withpredictive maintenance.The Databricks Lakehouse Platform excels in 3 ways:It’s simple: Data only needs to exist once to supportall workloads on one common platform.It’s open: Based on open source and openstandards, it’s easy to work with existing tools andavoid proprietary formats.From video streaming analytics to customer lifetimevalue, and from disease prevention to finding life onMars, data is part of the solution. Understandingdata is the key that opens the doors.Late 1980’s20112020Data WarehouseData LakeLakehouseReportsBIData MartsETLExternal DataOperational DataData MachineScience LearningData Prepand ValidationReal-TimeDatabaseETLReportsBIData MartsStreaming Analytics Data ScienceBIMachine Learning01010110100101101001010010101 1 0 1 1 0 1 0 1 0 0101101011001001000011010110101100010101001101 1010Structured, Semi-Structured and Unstructured DataStructured, Semi-Structured and Unstructured DataSOURCE: DATABRICKS0

03MIT Technology Review InsightsAligning anddeliveringon strategyAmid a global economic downturn of a scalenot seen for nearly a century, businessesmight be expected to be reining in theirambitions and focusing on the bottom line.Many of those represented in our survey,however, appear growth oriented. When asked about themost important business objectives they have set fortheir enterprise data strategy over the next two years,more respondents stress top-line growth, in the formof expanded sales and service channels (cited by 45%)than those who point to improved efficiency (43%).Following closely (at 42%) is improving innovation andreducing time to market of new or improved products. A look at the surveyed firms’ principal data initiatives overthe next two years suggests a substantial degree ofalignment with a growth-oriented business strategy. It alsoreflects their recognition of the urgency of improving datamanagement in order to support those businessobjectives. The wider adoption of cloud-native platformswill underpin this and other initiatives.The most frequently cited priority is achieving better datamanagement by improving data quality and processing,mentioned by 48% of respondents. (That figure is 74%among those working in oil and gas companies and 67% inconsumer products firms.) Such efforts are critical toFigure 1: Companies’ most important business objectives for enterprise datastrategy over the next two years (top responses; % of respondents)Expand sales and services channels45%43%45%Improve operational efficiency43%46%44%Improve innovation and reduce time to market42%42%43%Improve maintenance of physical assets34%32%Enter new product or service markets33%32%Improve ESG33%31%TotalNorth AmericaMIT Technology Review Insights survey, 202135%28%37%46%38%43%34%39%31%EuropeAsia-Pacific9

10MIT Technology Review InsightsAchieving better data management by improving data qualityand processing is critical to enabling growth-oriented efforts,like those driven by ML, to move ahead at speed.enabling growth-oriented efforts, such as those driven byML, to move ahead at speed. For Hivery, an Australiabased retail technology firm whose products are poweredby artificial intelligence (AI), the quality of its customers’data matters even more than its ability to ingest largevolumes. In fact, says Andy McQuarrie, Hivery’s chieftechnology officer, the cleaner its customers’ data, thefewer ingestion problems it encounters.The other top data priorities of the surveyed firms—increasing the adoption of cloud platforms (cited by 43%),enhancing data analytics (43%), and expanding theapplication of ML (42%)—if met, will provide data teamswith additional capacity, power, and scale to, among otherthings, quickly tap new sales and service opportunitiesand support new data product development. They also, ofcourse, fully support the goal of improving operationalefficiency. Another priority (cited by 38%) is expanding theuse of streaming, unstructured, and other varieties of data.That data strategy should be closely aligned with theoverall business objectives seems self-evident today, butthe importance of alignment has not always been clear.According to Don Vu, chief data officer of US financialservices firm Northwestern Mutual, alignment of data andbusiness strategy has become much tighter at manycompanies as CDOs have exerted their influence, anddata responsibilities have been brought together instreamlined organizational structures. At his firm, saysVu, “people knew that alignment was important, but thatbecame crystallized as our teams dug deeper into howwe’re actually going to deliver on the various businessstrategy initiatives. The link to business strategy fromnotions such as trust in essential sources of truth, ordemocratizing the use of data, became much clearer.”Figure 2: Companies’ most important enterprise-wide data strategy initiatives over the nexttwo years (top responses; % of respondents)48%51%50%43%52%50%44%43%35%Improve data qualityand processingIncrease adoption ofcloud platformsMIT Technology Review Insights survey, 202143%41%36%Enhance dataanalyticsTotal42%39%42%44%38%Expand applicationof MLNorth America36%39% 39%Expand usage of alldata (e.g. streaming andunstructured data)EuropeAsia-Pacific

MIT Technology Review InsightsData high-achieversNot many large enterprises excel at data management.This is reflected in the survey, where only 13% ofrespondents rate their organization’s performancehighly when it comes to delivering on data strategy,scoring it at the top end (9-10) of a 1-10 scale. Thesedata “high-achievers” deliver with measurable businessimpact across multiple business units, say theirexecutives. They are contrasted with a similarly sizedgroup of “low-achievers” (12% of the sample), whosedata performance is rated at 6 or lower on the scale.Large gaps separate these two groups in certainattributes as well as intentions. For example, cloudfeatures more prominently in the data architecture ofFigure 3a: The extent to which organizations are successfully delivering on the enterprise data strategy(self-assessed rating on a 1-10 scale where 10 succeeding)RatingSucceeding chievers1%Low-achievers2%30%20%10%MIT Technology Review Insights survey, 2021Figure 3b: High-achievers: respondents rating their organizations 9 or 10 on their delivery ofenterprise data strategy, with measurable business impact across multiple business units(total, regions and selected industries)13%Total15%North America13%Asia-PacificEurope11%Financial services21%20%Government/public sectorLife sciences & health care17%Oil and gas17%Automotive & transportation16%Telecom16%MIT Technology Review Insights survey, 202111

12MIT Technology Review InsightsFigure 4: The main success factors enabling “high-achiever” organizations to deliver on theirdata strategy initiatives (top responses; % of respondents)47%Data duplication reducedEase of data access38%Fast processing of large amounts of data36%Data quality improved31%Easy collaboration across cross-functionalteams on all analytical use cases20%Ability to do analytics on all datawherever it resides20%MIT Technology Review Insights survey, 2021high-achievers: 74% of this group run at least half oftheir data services or infrastructure in a cloudenvironment, compared with 60% of low-achievers whodo the same. When it comes to data priorities, mostlow-achievers (59%) are focused on improving datamanagement (data quality and processing) over the nexttwo years, while high-achievers’ most frequently citedinitiative (by 53%) is expanding the application of ML.Duplication of data in large organizations happens atmultiple levels such as data warehouses, operationalsystems, reports, dashboards, and desktop tools. Thishas significant cost, risk management, and reliabilityimplications, says Ashwin Sinha, chief data and analyticsofficer at Macquarie Bank. Data duplication also impactsthe ability to scale and make effective use of machinelearning across the organization.Far from taking the basics for granted, the high-achieversattribute their success to their close attention to thefoundations of sound data management. These includethe reduction of data duplication, ease of data accessfor enterprise end-users—a hallmark of data“democratization”—and the processing of large datavolumes at high speeds.Asked what’s holding back their progress, the largestpercentage of low-achievers point to limited scalability oftheir data management platform. Other often-citedimpediments are slow processing of large data volumesand difficulties in facilitating collaboration. Achieving scale,speed, and collaboration, as we will see, are challenges fororganizations right across the span of data operations.Figure 5: The main challenges keeping “low-achiever” organizations from delivering on theirdata strategy initiatives (top responses; % of respondents)44%Data management platform does not easily scale39%Slow processing of large amounts of dataHard for cross-functional teams to collaborate on allanalytics use casesHigh data duplicationComplex and fragmented toolsfor MLMIT Technology Review Insights survey, 202129%22%20%

MIT Technology Review InsightsNielsen: data transformation for a data-reliant businessIt is hard to overestimate the importance ofsound data management to Nielsen, one of afew century-old organizations in which datahas been central to the business model fromday one. Nielsen’s panels tell consumer goodscompanies what products customers are buyingand how behaviors are changing. The panelsalso advise such companies on where and whenthey should place their television advertising.Now in his second year as the firm’s chief dataofficer and fifth as chief research officer, MainakMazumdar has presided over a transformationof its data management and infrastructure.“Just a few years ago,” he recalls, “we struggledwith fragmentation—lots of data in silos andtribal knowledge needed to access it—a lack ofmetadata and very little governance, all while datavolumes were growing by petabytes each day.”Mazumdar paints a different picture today: “Nowwe’re able to scale quickly from having 20-30specialists on a platform to 300-plus. We’re ona cloud platform with a data lake where data iscurated, labelled, defined, has metastores, and isconsolidated. We’ve built our own analytics engine.In fact, much of what was done by softwareengineers in the past is now done by my team,deploying directly into production.” The changes,says Mazumdar, have reduced his team’s cycletime by 50%. “The speed of our models is nowabout 50x. What used to take 20 minutes we cannow do in a minute or less. At the same time, weare ingesting and processing massive amounts ofdata, which are easily accessible to data science.It’s a huge change.”An example of how Nielsen has used thesecapabilities to enable growth is the roll-out of anew ratings product in the company’s roughly200 local markets in the US. Crunching largeamounts of data from TV set-top boxes, the “highrecognition deep-learning model” enables Nielsento predict for customers not only what a viewer islikely to watch at any given time, but also who ineach household is doing the watching, somethingnot previously possible. “We could not have rolledout this product without the changes made to howwe manage data, and we could not have ingestedsuch volumes of data,” says Mazumdar. “It’s withthat ingestion of ever greater volumes of datathat the models—and the product—get better andbetter.”13

14MIT Technology Review Insights04Scaling analyticsand machinelearningBusiness leaders know that their company’sability to keep pace with and anticipatedemand, to manage competitive pressures, toinnovate effectively, and to operate efficientlyis coming to rest on their mastery of analyticsand ML. Organizations in virtually every industry are busydeveloping analytics and ML use cases that will delivergreater business impact. For most large enterprises, awide portfolio of use cases in production and at scale isno longer a nice-to-have but a must-have. CDOs and theirteams are increasingly judged on their contributions todelivering such cases.Many organizations struggle with this, and particularlywith achieving the scale needed to generate a sizeableimpact. According to Sol Rashidi of Estée Lauder, onereason is over-ambition: “Too often companies want toskip crawling and walking with ML and go straight torunning, without having mastered the basics.” For otherCDOs, such as Don Vu of Northwestern Mutual, the keychallenges lie in selecting the right use cases to deployinto production. Without business user input, he says, theprobability rises of selecting cases that do not clearly mapto a business objective.Figure 6: The main difficulties companies encounter in scaling ML use cases(top responses; % of respondents)No central place to store and discoverML models55%Numerous types of deployments and error pronehand-offs between data science and production39%Lack of ML expertise39%32%A plethora of tools and frameworks28%Hard to explain and govern ML modelsOutdated models because of infrequentlyrefreshed dataAccess to relevant quality dataMIT Technology Review Insights survey, 202127%11%

MIT Technology Review InsightsA paradigm shift at CVS HealthPharmacies have always played anessential role in societies, but arguablynever more so than during the pastyear. Pharmacy chains such as CVS,America’s largest by revenue, are usingever more advanced data capabilities to ensurethat their customers are up to date on theirmedications and use of other health services, lestunderlying conditions lead to more serious healthconsequences. Bob Darin is leading many of thoseefforts as chief data officer of CVS Health andas chief analytics officer of its retail pharmacybusiness.The company has long used data systems to prodcustomers to stay current with their medications.This has involved, for example, patient outreachthrough phone calls and texts, prompts at thepharmacy counter, or recommendations for thepatient to talk with their health-care provider aboutspecific follow-up or medication reviews. In recentyears, those initiatives have become embed

cloud platforms: improving data management, enhancing data analytics and ML, and expanding the use of all types of enterprise data, including streaming and unstructured data. For "low-achievers—" organizations having difficulty delivering on data strategy—improving data management overshadows all other priorities, cited by 59% of this .