What’s New In Cloud Data Integration In . - Informatica

Transcription

June 9 2020What’s New in Cloud DataIntegration in 2020 releasesMeenakshi Vasudevan, Product ManagementCloud Data Integration

Housekeeping Tips Today’s Webinar is scheduled for 1 hour The session will include a webcast and then your questions will be answered live at the end of the presentation All dial-in participants will be muted to enable the speakers to present without interruption Questions can be submitted to “All Panelists" via the Q&A option and we will respond at the end of the presentation The webinar is being recorded and will be available to view on our INFASupport YouTube channel and Success Portal.The link will be emailed as well. Please take time to complete the post-webinar survey and provide your feedback and suggestions for upcoming topics.2 Informatica. Proprietary and Confidential.

Success Portalhttps://success.informatica.comLearn. Adopt. Succeed.Bootstrap producttrial experienceEnriched Onboardingexperience Informatica. Proprietary and Confidential.FREE ProductLearning Pathsand weekly ExpertsessionsInformaticaConcierge withChatbot integrationsTailored training andcontentrecommendations

Safe HarborThe information being provided today is for informational purposes only. Thedevelopment, release, and timing of any Informatica product or functionalitydescribed today remain at the sole discretion of Informatica and should not berelied upon in making a purchasing decision.Statements made today are based on currently available information, which issubject to change. Such statements should not be relied upon as arepresentation, warranty or commitment to deliver specific products orfunctionality in the future.4 Informatica. Proprietary and Confidential.

Agenda51Data hts4Connectivity5ServerlessOffering6Demo Informatica. Proprietary and Confidential.

Blueprint for Cloud Data Warehouse/Data LakeStreaming26MachineDataAppsOn-PremisesMainframe Application DatabasesServersDocumentsData IngestionMobile3Data Integration & mentCloud StorageSpark ProcessingDRMData Catalog &Data GovernanceDiscovery Informatica. Proprietary and Confidential.EnterpriseZoneCloud DataWarehouseData ProvisioningSocial5Data IntegrationCloud Data LakeLog filesReal-timeAnalyticsStream Storage4Data ProvisioningIoTStreamProcessingLineageData Science/AIEnterpriseAnalyticsLine of Business /Self-ServiceAnalyticsBusinessUserDataAnalystLine ofBusinessDataEngineerDataScientistGlossary

Cloud Lakehouse Data Management T Codeless integration Data profiling Data discovery Mass ingestion for files, DB andstreaming Data quality rules End-to-end lineage Push down optimization Dictionaries to manage values lists Cleansing, standardization, parsing,verification and deduplication/consolidation processes Metadata – technical, business,operational, usage Serverless and elastic scaling Spark-based processing in the cloud Broad connectivity Stream processing Integrated into data integration Data quality analytics Connect and scan metadata –databases (DW, DL), apps, ETL,BI tools and others Common metadata foundation MLOpsAI-DRIVEN AUTOMATIONCLOUD NATIVEMulti cloud API driven Microservices Containerization Serverless architecture Minimal install and setup Auto-upgradesUsage-based pricing Trust certifications7 Informatica. Proprietary and Confidential.

Cloud Lakehouse Data Management T Codeless integration Data profiling Data discovery Mass ingestion for files, DB andstreaming Data quality rules End-to-end lineage Push down optimization Dictionaries to manage values lists Cleansing, standardization, parsing,verification and deduplication/consolidation processes Metadata – technical, business,operational, usage Serverless and elastic scaling Spark-based processing in the cloud Broad connectivity Stream processing Integrated into data integration Data quality analytics Connect and scan metadata –databases (DW, DL), apps, ETL,BI tools and others Common metadata foundation MLOpsAI-DRIVEN AUTOMATIONCLOUD NATIVEMulti cloud API driven Microservices Containerization Serverless architecture Minimal install and setup Auto-upgradesUsage-based pricing Trust certifications8 Informatica. Proprietary and Confidential.

Data low

Transformation Enhancements Support for Transaction ControlTransformation. Support for dynamic file namecreation for targets. Union transformation can nowsupport more than 2 input groups. Support for additional Lookupproperties for cached-lookup. Informatica. Proprietary and Confidential.

Transformation Enhancements System variables can be used in creatingexpressions. Join types extended for multi-object sources. Target update SQL override support forrelational targets like Oracle, mySQL, SQLServer and ODBC. Support for target load order (Flow run order)in the mapping designer. Informatica. Proprietary and Confidential.

Transformation EnhancementsDATA INTEGRATIONTransaction ControlTarget TX - Dynamic targetfile names, Target UpdateSQL overrideUnion Transformation tosupport more than 2 groups.DATA QUALITYSTRUCTURE PARSERParserDe-duplicateSupport for hierarchicaloutput types – JSON, XML,Avro, ParquetAlready Available:Rule SpecificationSupport for Real time source(Kafka)VerifierSupport pass through fieldsPersistent cache LookupExtend join types in sourcetransformation for relatedobjectsSystem variables inExpression TransformationAdditional licenses are required to use Data Quality andStructure Parser transformations.

Parameterization

Parameterization Enhancements Parameter files can now beaccessed from Cloud storageslike S3/ADLS. Generate parameter filetemplate easily. Support for pre/post SQLparameterization, Lookup SQLoverride. Request messageparameterization for Workday/Oracle Financials. Override Source/target FileDirectory, Session log filename in mapping tasks. Informatica. Proprietary and Confidential.

Task flows

Support for file ingestion tasks and filewatch step New Ingestion step toinclude File massingestion tasks in taskflow. File-watch step – Basedon a file listener taskand wait for file arrivalmid-stream in the taskflow. Informatica. Proprietary and Confidential.

Usability

Mid-stream data preview In the mapping designer, you can nowpreview the output of each transformationas you build the mapping. The mapping pipeline up-to the point ofpreview needs to be valid (not the entiremapping). Data preview is a ‘mini-job’ that is run usinga secure agent and can be monitored. By default first 100 rows are fetched. Thiscan be configured. The preview results can be downloaded. Supported only for CDI mappings in R34. Informatica. Proprietary and Confidential.

Session log Enhancements Session log file now includes Task name and IICS FRS ID. Agent Group name and Id Agent name and Id. Log file name when downloadedcontains the mapping task name(not just logX anymore!) Timezone information Transformation names forSources and Targets. Informatica. Proprietary and Confidential.

Operational Insights (Cloud Data Integration Jobs)

Operational Insights for Cloud Services Infrastructure monitoring – Secure agent health, resource consumption and ability to setalerts. Cloud Data Integration jobs monitoring - Aggregated analytics and visual dashboards withjob history for last 30 days. Cloud Application Integration – Preview in Spring 2020 release Availability - In all PODs with the Spring release. Included with the Data Integration/Cloud Application Integration subscription. Please reach out to your customer success manager or raise a support case if the service is not enabledon any of your Informatica orgs. Informatica. Proprietary and Confidential.

Jobs heat map and scheduled jobs Quickly identify the peak timewith the jobs heat map. Heat map helps you analyze theimpact on your integration jobsand optimize resources andschedules. View the number of upcomingscheduled jobs for the next 24hours. Informatica. Proprietary and Confidential.

Historical job runs Visualize the previous runs ofany integration job to spotanomalies in the task run. Historical job runs availablefor the last 30 days. Export the data to csv forfurther analytics and reportingneeds. Informatica. Proprietary and Confidential.

Project level and Secure Agent view of activity Dashboards that provideproject/folder level overviewof the data integration jobs. Quickly identify businessunits that have job failuresand need attention. Compare the load on yoursecure agents to optimizeresource and scheduling ofthe integration jobs. Informatica. Proprietary and Confidential.

Connectivity Highlights

Big Push for Pushdown OptimizationEcosystem PDO for AWS Push mapping logic to AWS to leverage COPY command To load data from S3 to Redshift Using NATIVE connectors Basic transformations between S3 and Redshift: Filter and Expression Roadmap – More transformations and connectivity!AWSS3RedshiftUsing INFA engineAWSS3RedshiftEcosystem PDO26 Informatica. Proprietary and Confidential.

Big Push for Pushdown OptimizationSnowflake Enhancements ( via ODBC) Sequence Generator Unconnected Lookups Multiple Targets/Target Instances Insert/Update/Upsert with targets Source Query Additional Functions, such as MD5 Create Target27 Informatica. Proprietary and Confidential.

Azure – R34 Enhancements Azure Synapse (SQL DW V3)- ADLS Gen2 as temp storage for Polybase- File Port support- Mass Ingestion from ADLS Gen2 to Synapse- Azure Gov Cloud- Unconnected Lookup- User Authenticated Proxy- Logging enhancements- Source/Target overrides with Parameter Files- Source/Target overrides with Parameter Files- New Data types for Parquet files(Date/Datetime/Decimal)- Partial Parameterization (database, schema,and table) support in PRE/POST SQL,SQL/LOOKUP Override- Performance enhancements CDM Folders (Tech Preview)- Support for CDM Schema v0.9- Integration with new MSFT CDM Folders SDK- Support for Unicode Characters28 ADLS Gen2 Informatica. Proprietary and Confidential.- Performance Enhancements- New Connector for CDI-E service (tech preview) Blob Storage (Blob V3)- Source/Target overrides with Parameter Files

Google – R34 Enhancements Google BigQuery- CDC as target support for Google BigQueryGBQV2- Merge support for Upsert operation for GBQ V2- -GBQ V2 – cached and uncached lookupsupport- GBQ V2 – SQL overide support29 Informatica. Proprietary and Confidential. Google Cloud storage V2- Folder and Subfolder support for Google Storageconnector.

Serverless Runtime for CDI –Preview in R34

Serverless Offering (New!)Informatica VPCCustomer VPCMetadataWeb browser(Build & manage)DataOutbound connectionHTTPS: 443MicroservicesVPNComputeData Pluggable micro-service based enginesAuto upgradeHigh Availability with Zero downtimeClustering - Agent groupsSecure - HIPAA, SOC2, PCIDatabasesDataWarehouseERP 31 Informatica. Proprietary and Confidential.Auto-ScalingMultiple AWS regionsResiliency and HATenant IsolationDMZ

Informatica Serverless in IICS R34 (Preview) Which IICS services supports serverless feature?- CDI and CDI-elastic only Which Cloud ecosystem is supported by serverless feature?- AWS only What sources and targets will be supported?- For CDI-elastic: S3, Redshift. JDBC.- For CDI: RedShift, RDS, Dynamo DB. Also any supported on-premise data source to which there is aconnection from the subnet used in serverless config.Interested in the Preview? - Please reach out to your customer success manager oraccount manager!32 Informatica. Proprietary and Confidential.

Connectivity Highlights

Big Push for Pushdown OptimizationEcosystem PDO for AWS Push mapping logic to AWS to leverage COPY command To load data from S3 to RedshiftAWSS3- Using NATIVE connectors- Basic transformations between S3 and Redshift: Filter and Expression- Roadmap – More transformations and connectivity!RedshiftUsing INFA engineAWSS3RedshiftEcosystem PDO

Big Push for Pushdown OptimizationSnowflake Enhancements ( via ODBC) Sequence Generator Unconnected Lookups Multiple Targets/Target Instances Insert/Update/Upsert with targets Source Query Additional Functions, such as MD5 Create Target

Azure – R34 Enhancements Azure Synapse (SQL DW V3)- ADLS Gen2 as temp storage for Polybase- File Port support- Mass Ingestion from ADLS Gen2 to Synapse- Azure Gov Cloud- Unconnected Lookup- User Authenticated Proxy- Logging enhancements- Source/Target overrides with Parameter Files- Source/Target overrides with Parameter Files- New Data types for Parquet files(Date/Datetime/Decimal)- Partial Parameterization (database, schema,and table) support in PRE/POST SQL,SQL/LOOKUP Override- Performance enhancements CDM Folders (Tech Preview)- Support for CDM Schema v0.9- Integration with new MSFT CDM Folders SDK- Support for Unicode Characters36 ADLS Gen2 Informatica. Proprietary and Confidential.- Performance Enhancements- New Connector for CDI-E service (tech preview) Blob Storage (Blob V3)- Source/Target overrides with Parameter Files

Google – R34 Enhancements Google BigQuery- CDC as target support for Google BigQueryGBQV2- Merge support for Upsert operation for GBQ V2- -GBQ V2 – cached and uncached lookupsupport- GBQ V2 – SQL overide support37 Informatica. Proprietary and Confidential. Google Cloud storage V2- Folder and Subfolder support for Google Storageconnector.

Demo

Q&A

Thank You

Parameter files can now be accessed from Cloud storages like S3/ADLS. Generate parameter file template easily. Support for pre/post SQL parameterization, Lookup SQL override. Request message parameterization for Workday/ Oracle Financials. Override Source/target File Directory, Session log file name in mapping tasks.