Beyond ETL: Next Generation Data Integration Architectures .

Transcription

Insert Picture Here Beyond ETL: Next Generation Data Integration Architectures withOracle Data Integrator Enterprise EditionUgo PollioPrincipal Sales ConsultantOracle EMEA Data Integration Solutions Team

Data Integration: Agenda Obstacles to Unifying Enterprise InformationOracle’s Data Integration StrategyWhat’s new in ODI-EE 11g?ODI-EE Architectural AdvantagesWhy Data QualityOracle DQ offerings in DIDQ Lifecycle management Insert Picture Here Investigate Improve Govern Oracle Data Integrator in Action How to get started2

Unifying Information in a Mixed LandscapeMaximize the Value from BI/DW, SOA, and Business taMigrationData SilosDataWarehousingData MartsDataFederationData HubsScriptsData AccessSQLJavaCustomOLTP & ODSSystemsDataWarehouse, Data MartHigh Cost ofCustom CodingOracleFilesPeopleSoft, Siebel, SAPExcelCustom AppsXMLLack of clean,consistent dataOLAPMultiple standards,disciplines

Comprehensive Approach to Data IntegrationCombining Operational and Analytical Data SOAPlatformsComprehensive Data Integration SolutionSOA Abstraction LayerProcess ManagerService BusData ServicesData FederationOracle Data IntegratorOracle GoldenGateOracle Data QualityELT/ETLReal-time DataData ProfilingData TransformationLog-based CDCData ParsingBulk Data MovementReplicationData CleansingData LineageData VerificationMatch and MergeStorageData Warehouse/Data MartOLTPSystemOLAP CubeFlat FilesWeb 2.0Web and EventServices, SOA

Oracle Data Integrator 11g (ODI-EE)Fastest E-LT, Lowers TCO, Speeds Time-to-ValueLegacySourcesApplicationSourcesOLTP DBSourcesOracle Data Integrator Enterprise EditionAny DataWarehouseAnyPlanningSystem

Data Integration: Agenda Obstacles to Unifying Enterprise InformationOracle’s Data Integration StrategyWhat’s new in ODI-EE 11g?ODI-EE Architectural AdvantagesWhy Data QualityOracle DQ offerings in DIDQ Lifecycle management Insert Picture Here Investigate Improve Govern Oracle Data Integrator in Action How to get started6

What’s New in ODI-EE 11gComprehensive Solution Goes Beyond Traditional ETL

Enterprise DeploymentProvides Lowered TCO and Improved Efficiencies Run-time components deploy onOracle WebLogic Suite, Coherence Options for High Availabilitydeployments as standalone or JEEbased Connection pooling, load balancing Connection retry (Oracle RAC) Corporate Security Integration Lightweight E-LT agents for loweredTCOWebLogic 11g / Application ServerDesktopJVMODI StudioFMW ConsoleODI Plug-inDesignerJava EEApplicationODI SDKOperatorServlet ContainerTopologyWeb Service ContainerJava EEApplicationSecurityRuntime WSODI ConsoleODI SDKPublic WSJava EEAgentDataServicesData Sources Connection PoolRepositoriesODI MasterRepositorySources and TargetsJVMRuntime WSODI WorkRepositoryStandaloneAgentLegacyFiles / XMLDBMSApplicationsERP/CRM/PLM/SCMDW / BI / EPM

Enterprise DeploymentProvides Lowered TCO and Improved EfficienciesWebLogic 11g / Application ServerDesktopJVMODI StudioFMW ConsoleODI Plug-inDesignerJava EEApplicationODI SDKOperatorServlet ContainerTopologyWeb Service ContainerJava EEApplicationSecurityRuntime WSODI ConsoleODI SDKPublic WSJava EEAgentDataServicesData Sources Connection PoolRepositoriesODI MasterRepositorySources and TargetsJVMRuntime WSODI WorkRepositoryStandaloneAgentLegacyFiles / XMLDBMSApplicationsERP/CRM/PLM/SCMDW / BI / EPM

Enhanced ProductivityImproves ease-of-use, speeds deployment New IDE based on JDeveloper Interface Editor leveragesdiagramming framework andincludes Auto fixing Quick-Edit allows for fasterinterface design and massupdates Code Simulation

Flexible ConnectivityImproves ease-of-use, speeds deployment Application Adapters for Oracle EBusiness Suite, PeopleSoft, Siebel,JD Edwards, SAP ERP and SAPBW. Oracle OLAP, Oracle CDC Hyperion Planning, FinancialManagement and Essbase Real-Time CDC with GoldenGate Optimizations for Teradata andmulti-statements Oracle DB multi-table insert ADF-View Objects via OBI-EEOracle Data IntegratorEnterprise P, OLAPSources

Improved ManageabilityImproves ease-of-use, speeds deployment, lowers TCO Oracle Enterprise Manager forunified administration andmanagement Enhanced error management –precise error messages,extended notifications Oracle Data Integrator Consolemanages run-time operationsand browse run-time ,designtime artifacts, errors Enhanced Session Control stop immediate stale sessions

Real-time CDC IntegrationBest-in-class solution for real-time with Oracle GoldenGateTraditional ETL CDC Invasive Capture on OLTP systemsusing complex Adapters Transformations in ETL engine onexpensive middle tier servers Bulk load to the data warehouse withlarge nightly/daily batchODI Oracle GoldenGate Continuous feeds from operationalsystems Non-invasive data capture Thin middle tier with transformationson the database platform (target) Mini-batches throughout the day orbulk processing G ODIXformGG ODIXformTrickleHeterogeneous

Data Integration: Agenda Obstacles to Unifying Enterprise InformationOracle’s Data Integration StrategyWhat’s new in ODI-EE 11g?ODI-EE Architectural AdvantagesWhy Data QualityOracle DQ offerings in DIDQ Lifecycle management Insert Picture Here Investigate Improve Govern Oracle Data Integrator in Action How to get started14

Optimized Data Loading through E-LTThe key to improved performance and reduced costsConventional ETL ArchitectureExtractTransformLoad E-LT provides flexiblearchitecture for optimizedperformance Benefits:Next Generation Architecture“E-LT”ExtractTransform Leverage Set-basedtransformations Improved performance forloading, no network hop Takes advantage ofexisting hardwareLoadTransform15

ODI-EE Knowledge ModulesPluggable Architecture for Improved FlexibilityPluggable Knowledge Modules ArchitectureSample out-of-the-box Knowledge ModulesSAP/R3SiebelLog MinerSQL ServerTriggersDB2 JournalsOracleDBLinkDB2 Exp/ImpJMS QueuesOracleSQL*LoaderCheck MSExcelCheckSybase Key Architecture Benefits: Ease of maintenance, Flexibility, tailored to existing best practices, Reduces cost of ownershipTPump/MultiloadType II SCDOracle MergeSiebel EIMSchemaOracle WebServicesDB2 WebServices

Leverage Pre-Packaged Integration SolutionsDirect Pre-Built Integrations”- Data Integration made simple Oracle CRM On Demand Integration to Siebel CRM Siebel Life Sciences Integration for Oracle Adverse Event Reporting System Demantra Integration Pack for Siebel CRM Consumer Goods Oracle Transportation Management (Glog) Integration to E-Business SuiteCross Industry Process Integration Packs- Design, Sell, Plan and Execute Design to ReleaseAgile Product Lifecycle Management integration Pack for Oracle E-Business Suite Opportunity to QuoteOracle CRM On Demand Integration Pack for Oracle E-Business Suite Order to CashSiebel CRM Integration Pack for Oracle EBS Order Management Trade Promotion MgmtSiebel CRM Integration Pack for Trade Promotion ManagementIndustry Process Integration Packs- Transform key business processes Comms Order to BillSiebel CRM Integration Pack for Oracle Comms Billing and Revenue Management: Order to Bill Comms Customer careSiebel CRM Integration Pack for Oracle Comms Billing & Revenue Management: AgentAssisted Billing Care Comms Revenue AccntgOracle Comms Billing and Revenue Management Integration Pack for Oracle E-BusinessSuite: Revenue Accounting Banking Acct OriginationsSiebel CRM Integration Pack for i-flex FLEXCUBE Account Originations Banking Acct OriginationsSiebel CRM Integration Pack for Banking Account Originations

Light E-LT Architecture with ODI-EEHigh Performance, Flexible, Lightweight Architecture Key Architecture Benefits: 100% Java, Open APIs, fast E-LTPackagedApplicationBusiness Intelligence& Data WarehouseODIAgentJKMALKMIKMDBFileODI Agent may bedeployed in any partof the architectureC 0LKMI C 1E (Errors)IKMCRKMExtract-LoadTransformCKMCheck-Load

Best Data Warehouse MachineFastest E-LT ProcessingOLAP Massively parallel high volume hardware toquickly process vast amounts of data– Exadata runs data intensive processingdirectly in storage Most complete analytic capabilitiesELT– OLAP, Statistics, Spatial, Data Mining, Real-timetransactional ETL, Efficient point queries Powerful warehouse specific optimizationsData MiningNew– Flexible Partitioning, Bitmap Indexing, Join indexing,Materialized Views, Result Cache E-LT runs 20X faster only with Oracle

Service-based Data IntegrationBest of Breed Data Integration as a SOA ServiceOracle SOA SuiteBusiness ActivityMonitoringBPEL Process ManagerWeb ServicesManagerBusiness RulesEngineEnterpriseService BusOracle Data IntegratorODI Knowledge vicesODI Connectivity FrameworkOracle GoldenGateHigh PerformanceETL & ReplicationAny Data SourceData Warehouse& OLAP Decouple business apps fromunderlying data-centricapplications Recombine existing services andapp components instead of newdevelopment Orchestrate services to createcustom view. Benefit Enables efficiencies intransformation, data loading,data synchronization, all atimproved performance

OBI EE Suite and ODI-EEData Lineage and Data Quality provides trusted dataOracle BIEE Suite PlusInteractiveDashboardsAd & Publishing Support OLAP/R-OLAPsources and targetsCommon Enterprise Information Model Report-to-source lineageOracle Business Intelligence ServerDesign& DrillDataWarehouseDataFlow Integrated Data QualityOracle Data Integrator Enterprise EditionData Distribution & Delivery APIsMetadataLineageBulk/TrickleLoadingChanged DataCaptureMasterDataData Quality& Profiling Common management:administration, monitoring,scheduling, auditing,exceptionsODI Knowledge Module FrameworkBulk and Real-TimeData ProcessingInformationAssets Works with Oracle GoldenGatefor improved real-time dataaccess21

Data Integration: Agenda Obstacles to Unifying Enterprise InformationOracle’s Data Integration StrategyWhat’s new in ODI-EE 11g?ODI-EE Architectural AdvantagesWhy Data QualityOracle DQ offerings in DIDQ Lifecycle management Insert Picture Here Investigate Improve Govern Oracle Data Integrator in Action How to get started22

Example of Data Quality IssuesA Simple Customer Table SampleNon StandardformatsMatching RecordsNameAddressCityStateZipPhoneEmailBob Williams36 Jones AvenueNewtonMA02106617 555 000bob.williams@yahoo.comRobert Williams36 Jones Av.MA02106617555000Burkes, Mike and Ilda38 Jones av.NwetonMA02106617-532-9550mburkes@gmail.comJason Bourne,Bourne & Cie.76 East 51stNewtonMA617-536-54806175541329 Mis-fieldeddataMultiple NamesMixed businessand contactnamesTyposMissing Data

Two Facts about Data QualityClear view on data The Data Quality Challenge is an iceberg The biggest DQ threats are the ones we do not see.9 Data Profiling lowers the water line and draws a clear view ofthe quality issuesKnown DataIssues Risk manageableBusiness rules tractableExpectations clearHigh business user involvement Risk unmanageableBusiness rules unknowableMissed expectationsLittle business user involvementSuspectedData IssuesUnexpectedData Issues

Two Facts about Data QualityData quality decays Data value decays Data is an asset which value decays over time Business events can make this worse Quality is not a one shot process but a constant effort in theenterprise processes.9 Data Quality needs to be pervasive and continuous. Pervasive Oracle Data Integrator SuppliesStandard, Inline Data Quality and DataProfiling Capabilities with Every ETL andE-LT Job Continuous Oracle Data Profiling and Oracle DataQuality for Data Integrator supportIntegrated Workflow, Recycling, andSteward GUIs

Data Integration: Agenda Obstacles to Unifying Enterprise InformationOracle’s Data Integration StrategyWhat’s new in ODI-EE 11g?ODI-EE Architectural AdvantagesWhy Data QualityOracle DQ offerings in DIDQ Lifecycle management Insert Picture Here Investigate Improve Govern Oracle Data Integrator in Action How to get started26

Oracle Data Quality Offering Joint Development with Trillium Software Recognized Leader in Data Quality (1600 customers)More than 30 years experienceFull Breadth of Data Quality functionalitiesProven Scalability and PerformanceData Quality ProcessProductAdvanced Data ProfilingOracle Data Integrator Oracle Data ProfilingStandard Profiling and IntegrityOracle Data IntegratorAdvanced Data Quality Standardize, Match, EnrichOracle Data Integrator Oracle Data Quality for ODI

Best of Breed Data Quality & ProfilingIntroducing Two New Oracle ProductsData Governance [Auditing, Cleansing, Matching, Parsing andStandardization] for Enterprise DataBENEFITS1.2.3.4.5.GovernanceGlobal ReachPerformanceFlexibilityTrack RecordKEY DIFFERENTIATED FEATURESCore Cleansing & MatchingInternationalizationSingle-Pass ArchitectureWide Platform SupportProven Technology

Data Integration: Agenda Obstacles to Unifying Enterprise InformationOracle’s Data Integration StrategyWhat’s new in ODI-EE 11g?ODI-EE Architectural AdvantagesWhy Data QualityOracle DQ offerings in DIDQ Lifecycle management Insert Picture Here Investigate Improve Govern Oracle Data Integrator in Action How to get started29

First, investigate

Oracle Data Profiling FunctionOracle Data Quality Profiling?Data Stewards andBusiness AnalystsSourcesDiscovery ofSources Investigate the Data Data Structure Data Content values, statistics,frequencies, ranges Data Relationships dependencies, keys, joins Assess Data Compliance Create and validate data rules Report & Alerts Monitor Data Qua

ODI SDK WebLogic 11g / Application Server Data Sources C onnection Pool Web Service Container Public WS Data Services FMW Console ODI Plug-in Servlet Container ODI Console Java EE Application ODI SDK Runtime WS Java EE Agent JVM Runtime WS Standalone Agent. WLS Integration - By providing its run-time components as Java EE applications deployable in Oracle WebLogic Server,