Transcription
Establish an Analytics Hub onLinux on IBM z Systems &LinuxONEWilhelm MildExecutive IT ArchitectIntegration Architecturesfor Mobile, Linux & IBM ZIBM Boeblingen Laboratory 2016 IBM Corporation
The Platform of Choiceintegrate transactional and analytic processingReduced data movement, reduced complexity, reduced configuration resources,more accurate data, more secure, more availableStreamlined decision processTransaction ProcessingFaster,Datamore Servingaccurate modelingFaster, more accurate scoresFastertransactionsMixed WorkloadsOperational EfficiencyTrusted and Secure ComputingReliable, Available, ResilientFaster, moreaccurateVirtuallyreportingLimitless ScaleEasier integration of other data sources1Fastertransformationof data
IBM Z point-of-viewBuilding a foundation to grow with business needsWhy z13 and z13s or z14?Cognitive Larger cache in-memory analytics for faster insight MASS libraries 2X to 10X improvement porting x86analytic workloads SIMD improvements 80% increase in CPLEX on z/OSmodeling CPU enhancements no performance impact to real-timescoring in transactionsSparkWhy IBM Z? Currency of dataReduce complexityEliminate data duplicationImprove synchronizationBring analytic function to the data
Evolution of Analyticsdata is the resource for competitive advantageAlerts & suggest next actionsIdentify & immediately actA businessmanagementexerciseStudying business data tounderstand trends andpredict the futureExpanding beyondbusiness data forbusiness opportunitieswhoAnalytics by spreadsheetAnalysis by smallteamsOwned by IT organizationsLines of businessWhatCognitiveOn historicalstructured dataCopy data to build datawarehouses & data martsCombine structured &unstructured dataHowPredictiveUsing utilities or rollyour own toolsTools build models to learnfrom historical dataDiscover & act usingopen source toolswhyDescriptiveInternal executivedecision supportIncrease businessopportunitiesEmbed analytics intobusiness processes
IT infrastructure matters not because of what you are doing nowBut because of what happens ysis onstatic data Currency of data becomes the driver of business value Embedding real-time analytics into the business transaction Privacy of client data and issues of ethical use Integration of structured and unstructured data: Bringing businesslevel QoS to a large scale with clustered and unstructured data Analytics evolving into Cognitive: Power of InferenceVirtualization Orchestration, automation, and control of all resources:On-premise and off-premise Splitting of application and data layers Hybrid Cloud and Multi-architecture integration Data privacy and security“Let’s talk tothe iPhone” “Remember the lessons of the PC”- Today's toy can run tomorrow's world Broader access unprecedented and unpredictabletransaction and data volumes fluid scale Unpredictable timing dictates 24x7 response and availability Security: Data integrity and privacy with broader transactionaccessCloudMobile4What Happens Next 2015 IBM Corporation
Imagine the possibility of leveraging all of your data assetsTraditional ransactionData“Here’s aquestion, what’sthe answer?”Internal AppDataMainframeDataStructuredRepeatableLinearOLTP SystemDataEmerging TechniqueCreativeHolistic thoughtIntuitionHadoopStreamsMultimediaWeb LogsSocial DataNew ideas,UnstructuredText Data:newemailsquestions, ExploratoryDynamic Sensor data:new answersimagesERPData“Here’s somedata, are cesTransformational benefit comes from integration of new datasources with traditional corporate data5 Copyright IBM Corporation 2015
IBM Systems with z ArchitectureIBM ZIBM LinuxONE SystemsIBM z13IBM LinuxONE EmperorThe world’sfastest processorMassive I/O throughputDedicated cryptographicprocessorsIBM z13sIBM LinuxONERockhopper
IBM LinuxONE SystemsLinux your way, open & without limitsAn open source community and ISV eco system to support a data serving ons7
(2) The Enterprise Analytics Hub on Linux on z SystemsBuild an end-to-end Analytics and Real-timefraud detection environmenton Linux on z SystemsLinux on z SystemsOff-prem /Cloud gnos BIz/OSSPSSdashDB local*Explore & AnalyzeDB2 z/OSSPARKIDAADB2 BLUMachineLearningAnalyticsWatson ExplorerUnstructured ContentCollaborationWebEmailContent CloudFileSystems ManagementMore Federation ServerVSAM,IMS data* - offering not yet availableSAGAdabasCADatacomSQL structured & no-SQL(DB2, Oracle, Postgres,MariaDB, MySQL, )(MongoDB, Cloudant, CouchDB)
The Enterprise Analytics Hub on LinuxONEProductSPARKFunctionAnalyze the contextof data in structured& unstructured dataUIMainAPIsAnalyticsresultsAnalyze the contextof data in structured& unstructured datacommands,Scala IDE orNotebooksJupyterNotebookSPARKSQL, ML,GraphXSPARK APIsSQL, ML,GraphXReports, scoring,Machine learningIn memory processingData sourcesdashDBlocal*Structured &unstructureddata, MESOS,Hadoop FS* - offering not yet availableDB2 BLUWarehouse &in memoryanalyticsStructured &unstructureddata, MESOS,HadoopCognosAnalyticsSPSSAnalyze current& historical dataPredictiveAnalyticsReport Studio forvisual, interactive, selfservice business usersSPSSdashboardJDBCInteractive reportsand dashboardsStructureddataJDBC,ODBCPredictive models,statistics, scoringStructureddataWatsonExplorerAnalyze and predictfrom the context ofdata using structured& unstructured dataExplorerdashboardVariousinterfaces i.eJDBC, JSONInteractive reports,Predictive, statisticsStructured-,unstructureddata andWatson cloudservices
What is Apache Spark?LanguagesJava / Python / Scala / RSpark SQLSpark MLlibSpark GraphXSpark ocessingReal-TimeStreamingSpark CoreSpark CoreGeneral Execution EngineYARNMESOSHDFS / Cassandra / HBase / Parquet / .Spark LibrariesStandaloneCluster ManagerData AbstractionDB2 Connect Driver (JDBC)
Get Started – Exploit Your Datadownload free software to start a project Federated, data-in-placeanalytics – reduce ETL Performance gains from colocation with data z Systems: SMT2, zEDC, SIMD,Large Pages, very high zIIP useStreamGraphXSQLCompressionFile SystemMLlibRDDcacheNetworkSchedulerSerialization Find insights from structured & unstructured data with Apache park/11
Spark Analytics on Linux on z and z/OSIBM z Systems provides an optimized platformto derive insights from all client data without moving itAccurate – Secure – Federated analysis in a hybrid cloud modelLinux on z SystemsSparkz/OSSparkSparkIBM P rkRDD*RDD*RDD*RDD*x86z/OS Optimized data LayerLeverage non-z dataType 2 / Type 4DB2 z/OSDB2Type 4IMSVSAMSMFand more Leverage Linux on zvirtualization benefitsLeverage z/OS data and transactions*RDD – Resilient Distributed Datasetis the Java of data analysis!
IBM Machine Learning for z/OS - OverviewLinux on zMachine Learning User InterfaceMachine Learning Applicationz/OSIBM z/OS Platform for Apache SparkData
IBM dashDB Local for Analytics*dashDB Local is the premier private cloud data warehouse optimized for analyticworkloads for Software Defined Environments (SDE) such as private clouds, virtual privateclouds and other infrastructures that support Docker container.Benefits of dashDB Technology with Fast Deployment into Private Cloud EnvironmentPrivate orVirtualizedPrivate CloudDockerContainerTechnologyFor apps that need: ElasticHighly flexibledatascalabilitywarehouse Optimized for fast and flexible deployment High availability Data model flexibility Data mobilityinto private or virtual private clouds Uses Dockercontainer Textsearch technology Built on top of dashDB technology, it Geospatialshares the benefits ofdashDBTechnologyMPP withAutomatic Scaling BLU Acceleration in-memory columnar technologyAvailable as:Massively Parallel Processing (MPP) with Fully managed DBaaSautomatedcapabilitiesincrease scalingOn-premisesprivate tocloud Hybridarchitectureinfrastructureefficiency* - offering not yet available
dashDB ks/https://www.youtube.com/watch?v mzOi45-KJN4Streaming data using the built-in Apache Spark infrastructurein dashDB Local runs in Docker containers UI using Jupyter NotebookIoT example* - offering not yet available15
DB2 w/ BLU Acceleration – inside dashDBSuper Simple. Super Fast.Solution DB2 with BLUAcceleration is thepreferred solution forcustomers who would liketo run analytics with zSystems & Linux data Satisfy requirement for acolumnar in-memory db Alternative of Linux on zOracle installations Enhanced for distributedconsolidations onto zSystems
Predictive Analytics: IBM SPSS on Linux on zlearn from historical data to make predictionsTechniques used to analyze data- Data mining- Statistics- Modeling- Machine learning- Artificial intelligenceExample use cases- Market Basket Analysis- Fraud Detection- Cross Sell – Up Sell OpportunitiesCommon Problems- Copies of data created for specific needs- Created complexity in managing data- Data Synchronization- Excessive CostsWhy IBM ZCurrency ofdataReduceComplexityEliminate dataduplicationImproveSynchronizationBring analyticfunction to thedata17
Predictive Analytics with Linux on z Systems technologyan open ecosystem for innovationCo-locate for Right-Time insights Reduced latency allowing for better insights Minimize cost & complexity Improve data governance & security Open the aperture on innovation Cost efficiencies through consolidation Efficient HiperSocket LPAR connectionsWhy IBM ZFreedom &AgilityStandardsBasedDeveloperProductivitySIMD delivers accelerated analytics processing for mathematical optimizationSMT delivers more throughput for Linux and zIIP-eligible workloadszEDC reduces data transfer time and storage cost by up to 75%16 Gbps FICON links reduce latency for workloads such as Db2Cache increased to enable faster in-memory insights10-32 TB of real memory even more critical with Linux18
Descriptive Analytics : IBM Cognos Analyticsthe analysis of historical dataWhy IBM Z1960s to early 1970s: Analyticsexpanded with the introduction ofcomputersSystem ofRecord fordataDecision Support Systemsprovided business data foranalysis.Late 1970s: Relationaldatabases created to eliminatedata redundancy / inconsistencyand improve structure of data- data was organized around records- relationships were enforced- indexing for hi-speed access- SQL was standardizedReporting evolved from DButilities to the creation ofspecific data marts for datamanipulation-To handlevolume andvelocity of data-To accuratelyreport thestate of thebusiness- creating cubes- providing dashboards19
Cognos AnalyticsIBM Cognos Analytics is an enterprise BI platform for governed data discovery and managed reporting that automates thecreation of reports and dashboards so users have the freedom to do it on their own. The user experience is designed forbusiness professionals so they can easily prepare, create and visualize content using the built-in intelligence to guide them.“We are excited about the new self service and visualisation capabilities of Cognos Analytics, it willenable our users to make more informed decisions.”Lizette RoblesBI Project LeaderBring your data to lifeUniversidad de GuadalajaraCognos Analytics is ready when you are with a unified experience that works the same on webor mobile devices, enabling you to quickly find, analyze, create and share insight. Intuitive interface lets all users quickly author content Dashboards created using drag and drop on mobile device or desktop Best visualizations automatically recommended Templates and styles to let you format reports instantly On demand menus for access to full capabilities over a clean workspace Single interface to create ad hoc or pixel perfect reports, frees up ITAnalytics when, where, and how you need them Simple intuitive interface Smart search works in context Personalized experience Scheduling and alerts Interactive content available on-line or off-lineAnalytics you can trust for confident actionConfident action comes from access to curated data that eliminate risk anddebate over numbers. Data protected with layers of permissions, authentication, and history Report integrity maintained regardless of range of inputs acrossbusiness Controls to protect data whether you’re creating one report for many ormany are creating one report Scheduling and ognos-analytics#othertab120
What is Cognitive Analytics ?Cognitive: Psychological processes involved in acquisition andunderstanding of knowledge, formation of beliefs and attitudes, anddecision making and problem solving.What is Cognitive computing?Data understanding& interpretationLearn &adaptDecision21
Cognitive Analytics: IBM Watson ExplorerSystems that learn, understand, reason & interactCognitive Business requires access tothe right data, a trusted system to holdthat data and the ability to gain meaningfulinsights in time to affect outcomes.Cognitive requires keeping up with thecustomer via mobile devices andconnecting to other sources ofinformation through the cloud to see thefull picture of what’s happening.Cognitive requires systems with analyticsintegrated into the business process sothat intelligence can be gained and actionstaken while they still matter.Why IBM ZCurrency ofdataReduceComplexityEliminate dataduplicationImproveSynchronizationExtreme speedand scaleCognitive brings all these qualities together22
Cognitive Analytics Digital Business Digital IntelligenceIBM Watson ExplorerAnalytics – Foundation for Digital BusinessWatson technology – the Digital Intelligence for Cognitive Computingz/OSIBM Watson ServicesLinux on zInterpretWatson ExplorerStructured DataQuestion AnswerRelationship ExtractionConcept ExpansionLatest versionincludesContent AnalyticsDatabases DataWarehousesUser ModelingLanguage IdentificationMachine TranslationMore Message Resonance more Web ServicesExplore & AnalyzeWatson Developer g.htmlUnstructured ContentCollaboration WebEmailContentFileCloudSystems ManagementMore
rer-content-analytics?lnk STW US MYIBME C2 BLC&lnk2 learn watson-explorer-content-analytics24
Analytics workloads run faster on the Linux on z platformTPC-E Database Aggregation Query Test using independent OLTP brokeragedatabase and analytical queries 1 master and 4 worker JVMs Oracle used for OLTP database,hosted on our platform 348 M rows of brokerage trade data Spark co-located on our platform drove upto 3x more throughput than Spark runningoff platform on x86LinuxONEx8612Million Rows/Minute Apache Spark with map-reduce scriptperforms fetch, query and aggregation14101.9x862.3x43.1x20124# Spark PartitionsTest ran one LinuxONE guest, RHEL 6.5 on LinuxONE & z/VM 6.3 with 32 vCPUs (16 cores with SMT) and 512GB memory, Spark1.5.0, IBM JDK1.7-SR3Test ran on unvirtualized x86 with 16 cores on Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz, 512GB memory running SLES 11.3,Spark 1.4.1, JDK1.8TPC-E 20K scale on Oracle database V12Scala map-reduce script (aggregation query) on TPC-E 20K scale Trade table using 1 master and 4 worker JVMsOut-of-box, default parameters, no tuning
Linux on z Systems Technology: Not a “box” but “Linux Your Way”“Linux Your Way” means acommitment to:An Open Source EcosystemUnmatched TCA withunmatched scaleThe best performance26NYC 2015 IBM Corporation150518 Offering Manageme . cts-v7
Increased retail sales revenue throughpoint-of-sale & suggest-sale insightBusiness ChallengeHow to improve customer service andsatisfaction in order to drive greaterrevenue.Technical ChallengeExisting analytic processes wereunable to manage the analysis ofhistoric and transaction data fromPetrol’s retail stores, service stationsand home oil/gas businesses.SolutionImplemented IBM DB2 AnalyticsAccelerator to support highperformance queries and IBM SPSSto make real time, point of saleproduct recommendations.“IBM provides us with tools that align withsmarter commerce, enabling us to deliverthe right message to the right person at theright time, to understand product affinitiesand intelligently drive the sale all in acustomer centric way”External Links: Case Study
Building an open & flexible strategicanalytics platform for the futureBusiness ChallengeSDV is a German Bank providing retail services forover 100 years. They needed to find the right platformthat will fulfill regulatory requirements and expand toprovide high-end, real-time analytic solutions.Technical ChallengeThe existing infrastructure was challenged in meetingthe fulfillment and regulatory requirement of BCBS239 and was not capable of supporting therequirements of new projects.SolutionSelecting the IBM LinuxONE System allows the bankto consolidate their analytic environment on oneplatform using DB2 on z Systems Linux. The platformprovides an innovative, flexible infrastructure toexpand their analytic capabilities to an opencommunity-driven ecosystem provided by Linux.28
Unleashing new growth and operationalefficiency with an infrastructure transformationBusiness ChallengeGrowing fast, Brazilian credit union system Sicoobmust cope with extra transactions at short notice.How could it combat the increased IT complexitythat resulted without impacting service quality.Technical ChallengeTo keep pace with their rapid business growthSicoob did not have the IT infrastructure to supportreliable 24/7 service and mobile access for theircustomers.SolutionSelecting IBM z Systems as the strategic platform.Sicoob migrated and consolidated memberdatabases to IBM DB2 LUW with BLUAcceleration. They deployed IBM InfoSphereDataStage and IBM Cognos running in a Linuxenvironment on the mainframe.“IBM Challenges and opportunities have ledus to restructure our technologyinfrastructure and adopt IBM System ztechnology, which guarantees greaterstability and performance for our productsand services. This facilitates our growth, bylowering the cost of maintenance andadministration in the productionenvironment, and by reducing powerconsumption in the data center. ”– Denio Rodrigues, IT ExecutiveExternal Links: Case Study2, YouTube Video1, YouTube Video229
IBM Z SOR SOI SOE in a BoxSystem Of Record (SOR), System Of Insight (SOI), System Of Engagement (SOE) DockerDockerDockerz/OSLinux guestLPAR 1LPAR 2 Compose high-performance scalableapplications Dynamically and seamlessly reallocate resources between microservices Provide right-time analytics Provide powerful engagementDockerHiperSockets LANHWhttps://www.youtube.com/watch?v VWBNoIwGEjo30
Putting it all together – Open Source running LinuxONE and IBM zSystems Demo: “Scalable Financial Trading Analysis & Insights”Input DataVisualization DashboardHistorical S&P 500 IndexNews FeedSentiment AnalysisTrade TransactionsGeospatial Analysishttps://www.youtube.com/watch?v VWBNoIwGEjo
Linux your Way - Greater flexibility and choiceincludes AnalyticsChoose the distribution, runtime, hypervisor, database and analytics – it’s the Linux youknow and love with the openness, flexibility and agility you need for you ManagementOtherDatabasevRealizeCloud ManagerCommunity VersionsDB2Analytics
Questions?Wilhelm MildIBM Executive IT ArchitectIBM Deutschland Research& Development GmbHSchönaicher Strasse 22071032 Böblingen, GermanyOffice: 49 (0)7031-16-3796wilhelm.mild@de.ibm.com
TrademarksThe following are trademarks of the International Business Machines Corporation in the United States and/o
IBM z13 IBM z13s IBM LinuxONE Rockhopper The world’s fastest processor Massive I/O throughput Dedicated cryptographic processors. 7 Database Analytics Languages Db2 IBM LinuxONE Systems Linux your way, open & without limits Distributions An open source community
