Transcription
Qlik Consulting ServicesQlik Application Architectural PatternsSajid Shaikh – Financial Services Practice LeadFeb 2018sajid.shaikh@qlik.comchenly.paz@qlik.com
Agenda Qlik Application & Data Architectural Patterns 3-Tier Framework Representative Case Studies Emerging Technologies FS Consulting Services2
Data Integration Design PatternsHigh Level Data & Application Integration Design PatternsBig DataMulti-Tenant Design Pattern-2Selection AppApply User FiltersApply SectionAccessDatasetSendFormattedConstraint toQVD LayerApp Integration LayerDaily Load ProcessData PreparationODAG Big Data Design Pattern-1Access/CreateClient SpecificFolderStructureQRS – APICreate /ReplaceCustomer AppRefresh / ReloadDashboardSend Data RequestReceive DataSelf Service Design Pattern-3Analyst Viewer (Consumer) Contributor Developer Team AdminDMAP Content AdminDatasetQlik SenseIncrementalProcessReload QS inmemoryDashbaordXORRoles In memoryData ModelETL / ELT Design Pattern-4Daily Load ProcessQlik Sense AppCreate SelfServiceDashboardsQRS – APIReload / RefreshCustomer AppExtranet CustomerApps / Mashup PerCustomerBDI (Big Data Index) Emerging Platform H2 2018 Leave data where it is – No in-memory loadsOne Qlik Servers per Hadoop NodeData Model, Master Items, LibrariesExtensions, TemplatesCreate / AccessMaster TemplateAppsQlik SenseRip/ReplaceProcessRoot AdminDirect DiscoveryBlend Live Data- or –IntermittentRefreshes
3-Tier Framework Data IngestionSource DataEDW /Big DataFilesTransformation LayerAssociative Data Model Data Integration LayerOpsSystemsExtract LayerStar SchemaSCD DimensionsGenerate QVDs,QVX, CSV, etcDashboardsRisk / Analytics,Performance,Compliance, etc.DashboardsEntityData GovernanceGlobal DataLayer Data tricsFactsProductMeasuresPreparationCalendar CustomerHierarchyData RefreshesTime / Event BasedIntraday RefreshesODAG - On-Demand AppCustom Streaming APIsBDI – Big Data Index Key Artifacts Data Ingestion Best Practices A Global Data Layer Framework Build Data Integration Framework for analytics Dashboarddevelopment and 3-Tier Architecture: Extract Layer, Transformed and AssociativeData Model Various connectors for Databases, Big Data, SFDC, SAP, CSV,etc. Data Governance guidelines & validations rules for dataprofiling and exception management Star Schema or Dimensional Data Model Extract, Transform load processes, Source data ingestions guidelines: Daily, Event or Scheduled,incremental, Full Rip & Replace, Blended in-memory and real-time(Direct Discovery), & ODAG (Big Data) Dashboards to Binary load final data model
Business BenefitsWorking with Qlik to simplify and enhance data management practices will help transform the usability and agilityof data. The following are just some of the core benefits delivered through the 3-Tier Framework; Greater trust and confidence in data quality through governed operational analytics The ability to validate data assumptions through analytics A single source of truth for analytical dashboard development Increased agility and speed when interpreting data and generating reports
VLDB Handling Big Data (ODAG)High level Design Approach for Large DatasetsData MartData Integration LayerEntitlements SAData Integration LayerData MartGenerateDimensionalLevel Aggrfor eachCountryData MartSection AccessQVD LayerAggrDim.QVDSelection AppApply User FiltersApply SectionAccessPros (Design Approach)ConsAvailable functionality in ViewSystem overheadSeparation of Apps/workloadApp per countryView functionality onlyDocument Chain DetailDetailTransactionData for allCountriesGenerateDimensionalLevel Aggr foreach CountryQVD LayerCountry DetailMex YYYYMM.QVDQVD LayerAggrDim.QVDDetail App AllCountriesSelection AppApply User FiltersApply SectionAccessDocument Chain DetailDetailTransactionData for allCountriesSection AccessData Integration LayerOpt-3: ODAG Qlik SenseFinancial Analysis,Ad hoc Data andReport generations &Big Data AnalysisOpt-2: Segmentation by CountryOpt-1: Selection App Doc Chain (Detail)PhaseGenerateDimensionalLevel Aggr foreach CountryQVD LayerCountry DetailMex YYYYMM.QVDQVD LayerAggrDim.QVDLoop &Reduce byCountrySelection AppApply User FiltersApply SectionAccessPros (Design Approach)ConsAvailable functionality in ViewSystem overheadSeparation of Apps/workloadApp per countryTargeted Performance Improvementnon-consolidated AppsView functionality onlyDetail App aSpecific CountryReload CountryData into TargetODAG AppPros (Sense vs. View)Cons for ViewFlexible User Driven SelectionComplex ODAG Macro codeCan Pull data directly from Martlimited self-serviceImproved Self Service VisualizationsDetailTransactionData for allCountriesQVD LayerCountry DetailMex YYYYMM.QVDSendFormattedConstraint toQVD LayerUse Sense ODAG built in-functionality
ODAG Built-In NavigationODBC or otherconnectivity1. User logs into the Selection App which ispopulated with dimensional data and high levelaggregated charts on a schedule. The userhas been granted the relevant security rules2. User selects dimensional criteria in theselection app from filter boxes/charts and themap. After the governed limit is reached thenavigation button on the toolbar becomesactive with a green indicator. The user canthen choose to generate a new application bydynamically reloading data from the sourcedatabases or choose an existing app.Selections andaggregationsNavigation OptionSecuritisedGovernedConsumer/creator optionsQlik SenseServerDetailsODBC or otherconnectivity3. Analysis app updated in the chosen streamwith details of the energy consumption overtime.Other sourcesBig DataSource
Qlik and Big DataFull details for all records available all of the timeAggregates to make selectionsDetails for each selectionWith On-Demand App Generation Qlik surfacesaggregate data for a selection of a slice of detailrecords to be loaded on-demand from the source systemNow: ODAGSetup at Qonnections 8 x i3.xlarge AWS EC2 instances with 16 vCPUs Data distributed along 1424 workers in multiplegeographies Memory footprint about 22GB per EC2 instanceWith distributed indexing and calculation, Qlik scales outto make all the data available even for high data volumesand velocity while leaving the source data in placeUnder development: BDI
Qlik Financial Services Consulting
Why does Consulting help improve Customer Success?PoorAdoptionToo manyrisks(e.g. skills)LimitedROIPoorinsightsToo relianton IT
Qlik Consulting Project PortfolioInvestment Banks, Commercial Banks, Wealth & Asset Mgt, Capital Markets, Fund Managers, InsuranceInsurance Claims tracking and cyber index trackingClient Reporting exposure positions, and lot level detail reportingLoan portfolio analysis, production vs. pipeline, Relationship Manager score cardTrade Reporting, FINRA OATS, TRACE, MiFID 2, Reg NMS, and Post Trade AnalysisRegulatory reporting, Basel 3, CCAR, Volcker Rule, PRA, FFIEC & FR Y8/9C for Fed ReportingGlobal Institutional Sales Commissions & Revenue Reporting Dashboards and N Printing enablementFixed Income Analytics and revenue reporting including pricing and Duration based models comparisonsWealth Management in-flows / outflows, Client & Product insight, NNA, CBV & FTE Analysis, P&L vs. CostAsset Management Benchmark & Portfolio Risk/Returns, Factor based smoothing, Morningstar peer groupEmbedded analytics & Mashups OEM Loan Application (SaaS) provider in a Multi-tenant environmentBenchmark Index Plotter, TCA, Embedded Analytics, APIs, integration of legacy new EDW dataSales Pipeline & Production reporting, track coverage, contact, campaign, expenses, P&L & RWAExternal Client Facing Reporting Dashboard. Supported Custody, FX, Account Services, etc.Financial Analysts Reporting, a self-service guided Dashboard reporting for field repsPEP (Politically Exposed People) tracking and KYC (Know Your Customer) analysisLegacy Migration of OBIEE, Cognos, BO users to Qlik, SAP / BW Integration
PortfolioHybridOn-PremiseCloudTechnicalAnalytical ServicesVisualAnalyticServicesDataServicesPlatform &InfrastructureServicesBusinessAnalytical nDevelopmentData Integration &ModelingServer ArchitectureServicesBenefits RealizationWeb & MobileDevelopmentData Quality &ManagementPerformanceServicesCenter ofExcellenceSolutionAcceleratorsAdvanced UI/UXDesignBig Data ServicesSecurity Set-upDeploymentServicesSolution WorkflowServicesQlik Project Methodology (QPM)Industry ncedPackagesPartnerPackages
Americas Tech Services DisciplinesGeo AnalyticsNprintingUX DesignWeb DevelopmentPredictive AnalyticsDev Methodology Geo Coding Geographic Analysis Geometric analysis Template Development Reporting on Demand Data Strategies for Reports Information Design Visual Design Application Prototyping Custom Web Application Advanced Visualizations Embedded Analytics R, Python, MATLAB Server Side Extensions Machine Learning Standardization Leading Practices Frameworks (SEEQ)Visual Analytics Services DisciplinesBig Data On Demand App Gen App on Demand (Session) Direct QueryGovernanceCloudPerformance Foundational Security Rules Data Governance Platform Governance Qlik Cloud AWS Private Cloud Scalability Testing Platform Tuning Automated TestingData Services DisciplinesSecurity ns Custom Security Enterprise SecurityIntegration OEM Solutions Deployment Automation Platform Management Hybrid Cloud Model Global Deployments Geo Analytics, NprintingPlatform & Infrastructure Services Disciplines13
Industry (Vertical) TrendsManufacturing & High TechPublic Sector Demand Forecasting Quality & Machine Breakdown Warranty Analytics Crime Prevention Transportation Optimization RecidivismHealthcareRetail & Consumer Patient Admission Forecasting Readmission Analysis Clinical Outcomes Market Basket Analysis Retail Store Selection Product RecommendationCommunicationsLife Sciences Customer Churn Quality/Process Control Next Best Offer/Action Territory Alignment Campaign Management Drug Discovery & EffectivenessFinancial ServicesEnergy & Utilities Credit Risk Analysis Loan Default Fraud Detection Customer Churn Proactive Failure Peak Demand14
Qlik Consulting RolesThe following roles are potentially relevant to assist in achieving business objectives.Technical ArchitectData ArchitectDeveloperUI / UX Designer Platform and security Data Center Capacity Data Extraction Wireframes Long range capacityplanning Shared data layer Complex Expressions Low Fidelity Prototyping Source connectivity Data Modeling Responsive Designs Performance andAvailability Data certification Workflow Graphic Assets andLibraries Governance App Certification Information Architecture Onsite or Near-shore VM/HardwareManagement User Interaction DesignProject Manager Manages development anddelivery of theSolutions to agreedscope, timelines andbudget First point of contactQlik AdvisoryDeploymentStrategy andChangeManagementBenefitsRealization,Business Case,Adoption Plans
Engagement OptionsQlik Consulting’s world class team ensures rapid time to value and returnon investment, combined with knowledge transfer and enablementQlik LeadsCollaborationCustomer LeadsImplementationEnablementCoaching Jointly determine vision,goals and success criteria Co-development ofadditional app(s) Customer lead development,with Qlik assistance Qlik defines infrastructure,data & application foundation Qlik knowledge transfer andenablement Qlik provides architectureand development assistance Qlik installs & configures Qlikcomponents Joint enterprise scalabilityplanning Collaborative planning forQlikView competency center Qlik lead agile developmentof initial app(s) Define an informationarchitecture plan Qlik provides periodic appand architecture reviews
Thankyou
Legacy Migration of OBIEE, Cognos, BO users to Qlik, SAP / BW Integration . Portfolio Cloud Qlik Project Methodology (QPM) Services Packages Start-up Packages Advanced Packages Partner Packages Business Analytical Services Advisory Services Solution Services Benefits Realization Center of Excellence Deployment Services
