Transcription
SAP Data Hub:Data Integration with Enterprise ApplicationsDAT202PUBLIC
SpeakersLas VegasSeptember 24–27, 2019Marc HartzBarcelonaOctober 8-10, 2019Jens RannacherBangaloreNovember 13-15, 2019Lalitendu Samantray 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC2
Take the session survey.We want to hear from you!Complete the session evaluation for this sessionDAT202 on the SAP TechEd mobile app.Download the app fromiPhone App Store or Google Play. 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC3
DisclaimerThe information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP.Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or any other serviceor subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or any relateddocument, or to develop or release any functionality mentioned therein.This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms directions andfunctionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information in thispresentation is not a commitment, promise or legal obligation to deliver any material, code or functionality. This presentation is providedwithout a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for aparticular purpose, or non-infringement. This presentation is for informational purposes and may not be incorporated into a contract. SAPassumes no responsibility for errors or omissions in this presentation, except if such damages were caused by SAP’s intentional or grossnegligence.All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially fromexpectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates,and they should not be relied upon in making purchasing decisions. 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC4
AgendaSAP Data Hub Introduction Capability OverviewData Integration With SAP Applications With non-SAP Applications Leveraging SAP Cloud PlatformEnd-2-end DemoQ&A 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC5
SAP Data Hub Introduction
SAP Strategy – Deliver the Intelligent EnterpriseTHE INTELLIGENT ENTERPRISEfeatures 3 KEY COMPONENTS: 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC7
Digital Platform: Unlock data-driven intelligence and innovationUnified data management to capturereal-time value from different types of dataSAP Data ServicesBest-in-class digital platform for new appdevelopment, extensions, and integrationDigitalPlatformMeta Data and lifecycle managementMarketplaceSmart data IntegrationSAP CP Big Data ServicesIntegration servicesSAP Data HubObject Store (S3, Swift )SAP HANASAP HANA EnginesOrchestration and governanceThird party (Spark/Hadoop)DataManagementCloudPlatformSAP Cloud PlatformMobile ServicesSmart data accessBig Data ServicesData Lifecycle Management 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICUX ServicesLeonardo IoT ServicesAPI ManagementStreaming analyticsSAP HANA powersSAP applications asthe foundation ofhigh-performancedata warehousingand analyticsPortalSAP API Business HubAnalytics ServicesSAP Vora distributed enginesNext-generation datamanagement expandsSAP HANA in-memorydatabase to addressstructured andunstructured datause cases andexternal dataCollaboration ServicesSecurity ServicesIntegration ServicesLeonardo ML ServicesSAP Data Hubprovides dataorchestration andmetadatamanagement acrossheterogeneous datasourcesPlatform for extendingthe business processesof the Intelligent Suiteand enabling newinnovationsDelivering deepdata and processintegrationsthrough APIs andmicroservicesMarketplace forecosystem to buildnew innovationsleveraging APIs andbusiness services8
SAP Data Hub – Unified Data Integration for the Intelligent EnterpriseDistributed & ExternalData SystemsSAP ApplicationsSAPNetWeaver DMIS AddonSAP BW/4 HANASAP BWABAPIntegrationSAP Data HubBWIntegrationCloud DataIntegration*Data Discovery / Data Profiling / Metadata CatalogingConnection Management / Workflows / SchedulingData ServicesJobs(open & nativeprotocols)HANAFlowgraphsSAP Analytics Cloud(on-premise, cloud, multi cloud)WorkflowsSAC Push APISAP HANAIntegration 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICREST APIsPublic Clouds3rd PartyConnectors3rd Party ApplicationsSAP APIBusiness HubData Pipelining & ProcessingData ingestion / Data Processing / Data EnrichmentHadoop / HDFSDatabasesData Orchestration & Monitoring BW ProcessChainsCloud StoragesStandardConnectorsData GovernanceSAP C/4HANAStreaming (e.g. IoT)SAP CloudPlatformConnectorsSCI for processintegrationSAP OpenConnectors*This is the current state of planning and may be changed by SAP at any time without notice.9
SAP Data HubCapabilities Overview
SAP Data Hub Capabilities: Data GovernanceDistributed & ExternalData SystemsSAP ApplicationsSAPNetWeaver DMIS AddonSAP BW/4 HANASAP BWABAPIntegrationSAP Data HubBWIntegrationCloud DataIntegration*Data Discovery / Data Profiling / Metadata CatalogingConnection Management / Workflows / SchedulingData ServicesJobs(open & nativeprotocols)HANAFlowgraphsSAP Analytics Cloud(on-premise, cloud, multi cloud)WorkflowsSAC Push APISAP HANAIntegration 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICREST APIsPublic Clouds3rd PartyConnectors3rd Party ApplicationsSAP APIBusiness HubData Pipelining & ProcessingData ingestion / Data Processing / Data EnrichmentHadoop / HDFSDatabasesData Orchestration & Monitoring BW ProcessChainsCloud StoragesStandardConnectorsData GovernanceSAP C/4HANAStreaming (e.g. IoT)SAP CloudPlatformConnectorsSCI for processintegrationSAP OpenConnectors11
Data GovernanceMetadata ManagementBuild up catalog to get insight into your company’s metadata 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC12
Data GovernanceDAT162Metadata Management using the Meta Data Explorer Explore (Discover, Preview, Profile) data assets inconnected systems System-wide Metadata Catalog– Metadata Crawler: Automatic indexing undpublication of metadata– Manual Definition of metadata (datasets)SAP Data HubMetadata ExplorerDiscovery &ProfilingSearch Define and run Business Rules to understand thedata quality of your data Data Preparation allows business users to acquire,enrich, and enhance data in a spreadsheet like UI Tight integration with modeling environments for DataPipelines and Workflows 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICDataPreparationBusinessRulesMetadataCatalog Classification of data using labels (e.g. regardingquality and sensitivity) Data Lineage provides a clear end-to-endunderstanding of the data’s origins and usageLineageMetadata CrawlerManual DefinitionConnected SourcesSAP Data Hub Sources(SAP HANA, SAP BW, S3, HDFS,OData, Oracle, )*This is the current state of planning and may be changed by SAP at any time without notice.Other Repositories(SAP Information Steward, Atlas bySAP, Hive*)13
Data GovernanceData Preparation Self-service for Non-Technical UsersCapabilities Prepare the data based on a sample dataset Transform, shape, harmonize, curate, enrich thedata via simple click actions View, present and report the outcome immediately Apply the transformations to the full dataset at anytimeMain Use Cases End-to-end self-service data preparation Improve data quality to achieve data excellencedeletecombine 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICnew or adjusteddata set Create new data sets based for scenario andproject requirements14
SAP Data Hub Capabilities: Data Orchestration & MonitoringDistributed & ExternalData SystemsSAP ApplicationsSAPNetWeaver DMIS AddonSAP BW/4 HANASAP BWABAPIntegrationSAP Data HubBWIntegrationCloud DataIntegration*Data Discovery / Data Profiling / Metadata CatalogingConnection Management / Workflows / SchedulingData ServicesJobs(open & nativeprotocols)HANAFlowgraphsSAP Analytics Cloud(on-premise, cloud, multi cloud)WorkflowsSAC Push APISAP HANAIntegration 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICREST APIsPublic Clouds3rd PartyConnectors3rd Party ApplicationsSAP APIBusiness HubData Pipelining & ProcessingData ingestion / Data Processing / Data EnrichmentHadoop / HDFSDatabasesData Orchestration & Monitoring BW ProcessChainsCloud StoragesStandardConnectorsData GovernanceSAP C/4HANAStreaming (e.g. IoT)SAP CloudPlatformConnectorsSCI for processintegrationSAP OpenConnectors15
Data Orchestration and MonitoringConnect, orchestrate and monitor processes across systems 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC16
Data Orchestration and MonitoringConnection Management, Data Workflows, Scheduling Central management of all system connections– Connect to diverse systems natively and remotely– Support user access management Data Workflows for cross-system dataorchestration, e.g.–––––Trigger Execution of SAP BW process chainsTransfer data from SAP BW and SAP HANAExecute remote SAP Data Services jobsTrigger iFlows in SAP Cloud Platform IntegrationSubmit Spark jobs to Hadoop clusters Scheduling of Data Workflows Extensive Monitoring– Support view, control, and auditing of data operation inthe connected data landscape 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC17
SAP Data Hub Capabilities: Data Pipelining & ProcessingDistributed & ExternalData SystemsSAP ApplicationsSAPNetWeaver DMIS AddonSAP BW/4 HANASAP BWABAPIntegrationSAP Data HubBWIntegrationCloud DataIntegration*Data Discovery / Data Profiling / Metadata CatalogingConnection Management / Workflows / SchedulingData ServicesJobs(open & nativeprotocols)HANAFlowgraphsSAP Analytics Cloud(on-premise, cloud, multi cloud)WorkflowsSAC Push APISAP HANAIntegration 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICREST APIsPublic Clouds3rd PartyConnectors3rd Party ApplicationsSAP APIBusiness HubData Pipelining & ProcessingData ingestion / Data Processing / Data EnrichmentHadoop / HDFSDatabasesData Orchestration & Monitoring BW ProcessChainsCloud StoragesStandardConnectorsData GovernanceSAP C/4HANAStreaming (e.g. IoT)SAP CloudPlatformConnectorsSCI for processintegrationSAP OpenConnectors18
Data Pipelining & ProcessingBuild scalable and flexible flow-based applications to process,refine, and enrich data at the source. 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC19
Data Pipelining & ProcessingBuild Flow-based Applications using the Pipeline ModelerDAT363 Data Pipelines Flow-based applications– Operators (independent computation units)– Data (messages) flows between operators Extensible– Over 250 pre-defined operators (Connectivity,Processing, Data Quality, CV, ML, etc.)– Custom / Partner operators– Wrap any custom code Scalable– Containerized – Docker containers constitute theoperators’ execution environments– Distributed – Easy horizontal scaling Re-Usability– Create complex, multistep, reusable data pipelines andoperators 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC20
SAP Data HubData Integration with SAP Applications
SAP Data Hub: Data Integration with SAP ApplicationsDistributed & ExternalData SystemsSAP ApplicationsSAPNetWeaver DMIS AddonSAP BW/4 HANASAP BWABAPIntegrationSAP Data HubBWIntegrationCloud DataIntegration*Data Discovery / Data Profiling / Metadata CatalogingConnection Management / Workflows / SchedulingData ServicesJobs(open & nativeprotocols)HANAFlowgraphsSAP Analytics Cloud(on-premise, cloud, multi cloud)WorkflowsSAC Push APISAP HANAIntegration 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICREST APIsPublic Clouds3rd PartyConnectors3rd Party ApplicationsSAP APIBusiness HubData Pipelining & ProcessingData ingestion / Data Processing / Data EnrichmentHadoop / HDFSDatabasesData Orchestration & Monitoring BW ProcessChainsCloud StoragesStandardConnectorsData GovernanceSAP C/4HANAStreaming (e.g. IoT)SAP CloudPlatformConnectorsSCI for processintegrationSAP OpenConnectors22
Integration with ABAP-based SAP SystemsDAT108One model to consolidate all interaction scenarios between SAP Data Huband an ABAP-based SAP systems directional and bi-directionalCapabilitiesSAP S/4HANAProvide ABAP METADATA to theSAP Data Hub Metadata ExplorerSAP S/4HANA CloudABAP DATA PROVISIONING totransfer data into SAP Data HubBusiness WarehouseBusiness Suite 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICSAP Data HubABAP FUNCTIONAL EXECUTION thatis triggerable as a SAP Data Hub OperatorIntegration requires certain system level, planned at least SAP S/4HANA 1909, SAP S/4HANA cloud 1911, SAP NetWeaver 7.00with DMIS 2011/2018 Q4/2019 version. Certain functionality can only be made available for certain release levels.23
Integration with ABAP-based SAP SystemsGood to knowABAP CDSREADEROperator that supports initial load and deltareplication of ABAP CDS views.ABAP CDS Views are onlyavailable in SAP S/4HANACUSTOM ABAPOPERATOROperator to implement your ABAP customcode that will be executed as part of apipeline in the connected ABAP system.For example to call a function module.SAP S/4HANA cloud doesnot support custom codeSLTCONNECTOROperator to communicate with SAP LTReplication Server (SLT). Leverage existingreplication scenarios to bring data into aSAP Data Hub pipelines.Allows data consumption oflower SAP NetWeaver ( 7.00)and S/4HANA ( 1909) releases 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICIntegration requires certain system level, planned at least SAP S/4HANA 1909, SAP S/4HANA cloud 1911, SAP NetWeaver 7.00with DMIS 2011/2018 Q4/2019 version. Certain functionality can only be made available for certain release levels.24
Integration with ABAP-based SAP SystemsExample Data provisioning for SAP S/4HANA with ABAP CDS ReaderABAP CDS is the semantically rich data model in SAP S/4 HANA and allows theconsistent representation of a business object like a Business PartnerCDS replication from SAP S/4HANA to file storageABAP CDS LayerInitial Load &Application TablesInserts, Updates, DeletesDB triggersLoggingtableABAP Pipeline EngineSAP Data Hub 2.7CDC engineSAP S/4HANA 1909SAP S/4HANA CE 1911 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICThe ABAP CDS READER operator allows thereplication of SAP ABAP CDS Views and customerABAP CDS views in initial load and delta mode.25
Integration with SAP Cloud solutionsOne SAP Cloud Data Integration (CDI) for integration into all SAP Cloud solutionsMain Use Cases Holistic Data Management with SAP Data HubSAP Analytics Cloud Building Data Warehouse Analytics with SAP BW/4HANA Advanced Analytics and Planning with SAP Analytics CloudSAP BW/4HANASAP Data HubCapabilities Support both full and delta requestsCloud Data Integration API Seamless integration of data and metadata OData v4 as communication protocolSAP C/4HANA:Available TodaySAP Marketing CloudSAP Service CloudPlanned for Q3/Q4 2019 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Planned 2020SAP Cloud Data Integration (CDI) is acore concept for integrating all SAP solutions in thecloud using ONE API based on open standards26
Integration with SAP Cloud solutionsExample Cloud Data Integration scenario with SAP Fieldglass & SAP Data HubUnderstanding metadata of SAP Fieldglasswithin the SAP Data Hub Metadata ExplorerLoading Fieldglass Work Orders intoSAP Vora tables with data protection 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC27
Integration with SAP Business WarehouseAvailable Pipeline Operators: SAP BW Process Chain– Trigger execution of a process chain on a SAP BW system Data Transfer (SAP BW & SAP HANA)– Transfer data (query, infoprovider / tables, views) from SAP BW / SAP HANAinto big data stores or SAP Vora tables– Via INA interface or direct access to HANA (Calculation Views)Typical Scenarios: Load Data from Data Lake into BW Data Tiering from BW into Data Lake 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC28
Integration with SAP HANA and SAP Analytics CloudSAP Data Hub integrates with SAP HANA and SAP Analytics Cloud on multiple levelsSAP Analytics CloudStoryDatasetLiveConnectionSmart DataIntegrationViewsModelTablesVirtual TablesSmart DataAccessSAC PushOperatorHANAFlowgraphOperatorPipeline entSAP VoraEnginesRelationalMetadata ExplorerMetadataCatalogProfiling /DiscoveryVORAIngestorVORAClientFilesSAP Data Hub 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC29
SAP Data HubData Integration with Non-SAP Applications
SAP Data Hub: Non-SAP Data IntegrationDistributed & ExternalData SystemsSAP ApplicationsSAPNetWeaver DMIS AddonSAP BW/4 HANASAP BWABAPIntegrationSAP Data HubBWIntegrationCloud DataIntegration*Data Discovery / Data Profiling / Metadata CatalogingConnection Management / Workflows / SchedulingData ServicesJobs(open & nativeprotocols)HANAFlowgraphsSAP Analytics Cloud(on-premise, cloud, multi cloud)WorkflowsSAC Push APISAP HANAIntegration 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICREST APIsPublic Clouds3rd PartyConnectors3rd Party ApplicationsSAP APIBusiness HubData Pipelining & ProcessingData ingestion / Data Processing / Data EnrichmentHadoop / HDFSDatabasesData Orchestration & Monitoring BW ProcessChainsCloud StoragesStandardConnectorsData GovernanceSAP C/4HANAStreaming (e.g. IoT)SAP CloudPlatformConnectorsSCI for processintegrationSAP OpenConnectors31
Non-SAP Data IntegrationBuilt-in Standard ConnectorsConnectivity:- Azure Data Lake (ADL)- Google Cloud Storage (GCS)- HDFS- Amazon S3- Azure Storage Blob (WASB)- Local File System (file)- SAP Semantic Data Lake- WebHDFSSAP Vora:Connectivity (via Flowagent):Spark / Hadoop:- Spark- Spark SQL- PySpark- HiveData Quality: 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC32
Non-SAP Data IntegrationOperators for Data ProcessingProcess / Command Executors:Transformation Operators: Run a process within a pipeline and give contiguous stream to it Run on-the fly transformations and do event stream processingusing continous query language (CQL) on data within a pipeline Run a shell command for each arrival of a message within a pipelineScripting Operators: Write and run custom scripts for data manipulation within a pipeline Build re-usable operators in different programming languagesSubengines: Develop and compile new operators locally using SDK Register and run custom operators in available pipeline subengine**This is the current state of planning and may be changed by SAP at any time without notice. 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC33
SAP Data Hub: Extensibility for 3rd Party ConnectivityDistributed & ExternalData SystemsSAP ApplicationsSAPNetWeaver DMIS AddonSAP BW/4 HANASAP BWABAPIntegrationSAP Data HubBWIntegrationCloud DataIntegration*Data Discovery / Data Profiling / Metadata CatalogingConnection Management / Workflows / SchedulingData ServicesJobs(open & nativeprotocols)HANAFlowgraphsSAP Analytics Cloud(on-premise, cloud, multi cloud)WorkflowsSAC Push APISAP HANAIntegration 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLICREST APIsPublic Clouds3rd PartyConnectors3rd Party ApplicationsSAP APIBusine
– Trigger iFlows in SAP Cloud Platform Integration – Submit Spark jobs to Hadoop clusters . integration. SAP Open Connectors. SAP API Business Hub. REST APIs. SAP Cloud Platform Connectors. 3.
