Algorithmic Operations: Business PDF Free Download

2y ago

61 Views

1 Downloads

3.35 MB

69 Pages

Report/dmca

Download PDF

Transcription

Algorithmic Operations:Integrating Real-TimeAnalytics into yourBusinessRichard TibbettsCTO, TIBCO Event ProcessingQCon San Francisco 2014

Macro Trend: Digital Business veFull StackStartupsInternetConnectedDevicesMachine Learningand PredictiveAnalyticsEmpoweredSuper UsersTechnologyInitiativesOutside ITMobile Devicesand AppsAPI EconomyHow will you transform into a digital business? Copyright 2000-2014 TIBCO Software Inc.3

To automate your business with analytics,you must give the businessvisibility and controlUnderstandControlAutomate4

AgendaDigital Business: From Big Data to Fast DataOperational Intelligence and Algorithmic OperationsLessons Learned from Key Fast Data Use CasesTIBCO Event Processing Copyright 2000-2014 TIBCO Software Inc.5

Big Data: There can be an overwhelming volume

Most solutions focus on accumulating and analyzing

Especially analyzing things which already happened

But how will you access the data when you needit?

Fast DataFAST DATA IS PROCESSING BIG DATAIN REAL-TIME TO GAIN INSTANTAWARENESS AND INSTANT ACTION

To automate your business with analytics,you must give the businessvisibility and controlUnderstandControlAutomate11

The Problem: 40 Years of “The Too Late Architecture”AppTXNEventsBusiness DataDataDiscoveryAnalytics“Who cares if we find out we lost a customer after he left?”

The Algorithmic Enterprise: Fast Data at WorkEnterpriseIntegrationIntegration BusExternal EventsEvent ProcessingAlerting andVisualizationBusinessProcessesInternet of ThingsAutomated ReactionEventsStreaming Analytics & Event Driven Rules

Controlling Algorithmic Operations: Business RulesEnterpriseIntegrationIntegration BusExternal EventsEvent ProcessingAlerting andVisualizationBusinessProcessesInternet of ThingsAutomated DecisionsEventsEvent Rules

The LiveView Datamart – Operational IntelligenceIntegration BusExternal EventsInternet of ThingsEventsLive ng andVisualizationBusinessProcessesAutomated Decisions

Beyond Big Data: Algorithmic OperationsEvent Analytics(LiveView)SensorsSocialFast DataAppLive DatamartStreamingAnalyticsBig istoricalAnalytics(Spotfire)

Operational IntelligenceAlgorithmic OperationsHuman decisions in real timeinformed by up to date informationAutomated action based on analyticmodels of history combined withlive context and business rulesThe Challenge: The rightinformation, ready to be used bythe right people at the right time.The Challenge: Empowering businessstakeholders to understand andcontrol the selection of models 18 Copyright 2000-2014 TIBCO Software Inc.18

To automate your business with analytics,you must give the businessvisibility and controlUnderstandControlAutomate19

Key Fast Data Use Cases Trading: Execution Optimization Telco: Customer Churn Prevention Banking: Real-Time Compliance Marketing: Real-Time Campaign Attribution Dealing: Counterparty Behavioral Analysis Manufacturing: Yield Improvement and Cost Reduction

Execution Performance Analytics Situation: Major asset manager needs more insight– Problem: How to get better price discovery?– Need real-time analytics using market data, interface toOMS, EMS (Portware), and internal systemsSolution: StreamBase, LiveView, and Spotfire– Which trades are slipping? How close to target are we forparticipation rates? How can we adjust our trading in themarket now?Developed a live visualization system with StreamBase CEP,LiveView, a custom .NET visualization front end for traders &PMs, and Spotfire for historical analyticsImpact: Improved trading performance–Solution allows traders and PM’s to understand where theirtrade performance is going, and adjust trading strategiesduring the day

Asset Management ArchitectureFIXStreamBase CEP.NET GUITrading AnalyticsProprietary .NET UserInterfaceFIXMQDBMSStreamBase LiveViewTraders &PortfolioManagersLiveView AggregationAlertsOMSEMSSpotfire

Lesson #1Identify opportunities being missed by operationalstaff because they are too small or too hard tooptimize, and automate those first for quick wins.UnderstandControlAutomate26 Copyright 2000-2013 TIBCO Software Inc.26

“If I can see network issues in the moment, I can fix them before mycustomer notices, and automatically improve call center efficiency by lettingcustomers proactively know we’re working on the problem.CIO, major telecom28

TIBCO EP for Telco Customer Churn Prevention The Situation Before: Customer Churn Projections toCost 28M / Year– Negative Consequences: Call Centers Can’t React toProblems in the Network as They Happen– Systems don’t talk to each other in any way other than batch, sothe call center handles calls for network issues that are alreadyidentifiedPositive Business Outcome: Predictive Real-TimeCustomer Service–– Telecom flying blind in real-time with respect to network outagesthat can effect valuable customers and social influencersCall center systems proactively inform effected customers effectedby network issuesNetwork systems proactively re-route high-value customersaround effected network areas in real-timeHow We Do It: TIBCO Event Processing for Real-TimeAnalytics in Telco Networks and Call Center Action–High-speed network and user event analytics, visualization,alerting, and action to optimize customer call center behavior

Key Metrics of Customer Churn One Telecommunications Firm Customer Churn Study –––– Studied 8M customers, 7 billion service calls for a 3 week sampleFound 1M clusters of callers, 120K “Dropped Call Watch List” clustersIdentified 40K network influencer “signatures”Found 4000 watch list customers who already cancelled service and can influenceothers; impacted an additional 18K customers when they cancelledImpaction: Projected 28M in Lost Annual Revenue– Customer loss directly linked to network outages whose impact might have beenmitigated if real-time visibility was available– By providing influencers better service in real-time, and optimizing call centers tosupport high-value customers, revenue loss can be dramatically reduced– By predicting which customers will call due to service outage and intercepting calls,customers will know immediately that the Telecom is on top of the issue. Buildstrust. Copyright 2000-2013 TIBCO Software Inc.30

Call Center Analytics, Driven By Network InsightHigh priority alerts forNetwork Operators. e.g.,“These 100 high valuecustomers are beingeffected by these networkevents. Give them highpriority access and fix thenetwork.”Live aggregation and sliceand dice of call hand backs,dropped calls, and PingPong events – all signs ofproblemsReal-time analytic allows operators to view in real-time thedropped calls, hand overs between towers – these is criticaldata to resolve network issues in the moment, in theNetwork Operations Center.31 Copyright 2000-2013 TIBCO Software Inc.

“In December 2012, Knight Capital lost 460Min under 40 minutes. That changed everything.Now, it’s no longer acceptable to run our businessbased on end-of-day reports.”- Head of Risk Management, top 3 bank Copyright 2000-2013 TIBCO Software Inc.

Real-Time Correlation and Time-Based Windowing“WHEN any security has SIMULTANEOUS TXNS with THE SAME PRICE, within 30SECONDS, flag the TXN as a potential WASH TRADE”Trade OrdersTrade ExecutionsALERTTrade Cancels9:25 9:29:59Average DailyVolume

Telco Marketing Campaign Attribution Aleading cellular operator in Asia, which serves more than130 million customers. The operator has experienced doubledigit growth over the last 3 years in prepaid, postpaid, andvalue added servicesThe Problem: Bridging the gap between marketingcampaigns launched and potential revenue that is generated– Complexity of data correlation between currentcampaign system and surrounding environment meantmanually scripting and calculating the potential revenueThe Solution: Automated Campaign Tracking– Integrate data from four systems– Campaign provision report– Campaign spikes: Alert on spikes in offer provisioning Copyright 2000-2014 TIBCO Software Inc.39

Automated Campaign Tracking: The Outcome Replacing manual work with automated processCampaign revenue visualization by mapping campaign data to multiple sourcesStreambase provides a expandable platform for correlating different streamsLive view provides source for tracking data movement in real timeSpotfire can be used for war room reportsWhy TIBCO?– TIBCO solution will help view campaign flowing data (LiveView) and also correlate multiplestreams in StreamBase (4 sources of data) and view revenue reports in Spotfire (Data to bemade available in DB)– The existing campaign solution cannot provide reports. TIBCO can handle both the live view andhistorical – which will help Telkomsel understand data and spikes in a much better fashion– Live view will also help in raising alerts whenever campaign systems breach the threshold ofbonus provision (This can be extended to multiple sources) Copyright 2000-2014 TIBCO Software Inc.40

Campaign Tracking Deployment ArchitectureBusiness Analysis argeBillingNetworkShared StoragePoll Read FileCampaignPoll Read FilePromotionsEvent DrivenRulesBusiness Analysis (LIVE)NetworkCDRSplunkAgentSplunkSplunk AgentAgentBusinessUserNW Ops Analysis (BATCH)Splunk(Data Source)CDR Copyright 2000-2013 TIBCO Software Inc.NW OpsUser

Toxic Flow Identify customers actingon superior information Route trade flow elsewhereAt 8:31:21 AM, client sells 7 times in 10seconds. At this moment, the firm mademoney35 seconds later, the firm has lost over 10,000 on these trades because the clienthad better market data

Customer Behavior Analysis Reference ArchitectureHadoopInternet /CustomersLog FilesHistorical dataMetricsMachine LearningHDFSMachine Generated DataAnalystsMap ReduceAnalyticTERRAggregation,LoggingBrick & Motor / SensorLive AnalysisEvent ProcessingAutomated ResponseCorrelateAggregateStreamBase LiveViewTERRAutomateReal-Time ApplicationsAlertActive SpacesOperational StaffFraud & Risk ManagementRisk Analysts

Lesson #5Understand your data analytics pipeline and makethe same data available to operational intelligenceand algorithmic operationsUnderstandControlAutomate47 Copyright 2000-2013 TIBCO Software Inc.47

Manufacturing Yield Management & Optimization About The Customer–– Solar Panel ManufacturerOne of many manufacturingfacilitiesManufacturing YieldManagement and Optimization–––Spotfire & LiveView formanufacturing process yieldmanagement and optimizationMonitor temperature, moisturereading sensors in real-time withLiveViewReal-time predictive detection ofmanufacturing problems Copyright 2000-2014 TIBCO Software Inc.

Stream Correlation for Predictive Warning SystemsTemperatureStreamBase CEPPressureResistivityLiveViewData MartRULESLiveView DesktopStreaming DataIF avg (temperature) in the LAST 5 MINUTES 100CAND avg (pressure) in the LAST 10 MINUTES 1000 PSITHEN RAISE ALERT (“DANGER: Maximum Operating Parameters Exceeded”) Copyright 2000-2013 TIBCO Software Inc.

StreamBase, TERR, and Spotfire Working TogetherEvent ProcessingIncomingDataIncomingRecordsto scoreStreamProcessingAggregatorAlertsExample:At POCL stage we can predictprobability of the finalproduct RRPredictive Scoring ModelsQuality / Yield predictionAlerts withContextDataEmail withAnalysis snapshotLive View StatusLiveView DesktopTERR

Real Time Close Loop: Understand – Anticipate – ActCORRELATE: Real-time factorybehavior compared to the modelUNDERSTAND: Normal patternsof effective manufacturingprocesses analyzedACT: Shut down equipment,schedule additionalmeasurements,Event Processing Copyright 2000-2014 TIBCO Software Inc.CorrelateAggregateAutomateAlert52

Lesson #6Follow up on automated problem detection with notonly automated action, but also root cause analysisusing historical data toolsUnderstandControlAutomate53 Copyright 2000-2013 TIBCO Software Inc.53

Lessons learned from Fast Data Deployments1. Optimize and automate small decisions first, for quick wins2. Resolve problems while they are fresh to prevent snowballs3. Understand your analytics pipeline and have the same data in real time4. Use event processing to correlate events from multiple systems5. Empower staff with decision making authority and information, andalso provide recommended action6. Follow up not only with automated action, but root cause analysis Copyright 2000-2014 TIBCO Software Inc.54

To automate your business with analytics,you must give the businessvisibility and controlUnderstandControlAutomate55

TIBCO Event ProcessingStreaming Analytics & Event RulesEvent-driven rules and streaming analytics for automated,event-driven business decisions and processesUsers: Developers, IT and Data ScientistsDiscovery & Live InsightLiveView: Discover new patterns via human expertise, graphing, realtime stream aggregation, and real-time alertsUsers: Operations (network & business), Customer Support, Network Operations Center,Marketing operations

TIBCO Event Processing PlatformStreamBaseBusiness EventsLive DatamartInputsMessageBusOutputsBPMLiveView DatamartActiveSpacesSHARDINGUSER AUTHStreaming AnalyticsRULESAGGREGATIONDistributed Transactional ionCONTINUOUS QUERY Analytics

Industry Leading Breadth and Depth of EP Platforms – Data grid (ActiveSpaces), distributed transactionalmemory Social Media Integration–Twitter, GNIPStreamBase Component Exchange, certificationLanguage Integration– Test recording, Junit, Fitness, Continuous IntegrationDeveloper Community–Hadoop (Flume), Amazon Kinesis, Twitter StormTrace and break-point debuggerEnterprise Testing– Replay / testingVisual Debugging–R, TERR, MatlabBig Data Integration– With AMX-BPMSimulation and Backtesting–StreamBaseBig Data Processing– TIBCO Silver FabricEvent-Driven BPM–Business EventsIntegrated Analytics– Managed clusters, continuous availabilityCloud Deployment–150 adapters, Event-driven database integrationStreaming Analytics– Event-Driven Rules–Fault Tolerance–Visual programmingStream & Database Connectivity– Visual ProgrammingJava, .NET, HTML5Management Tools– Inc.GUI, Copyright 2000-2013 TIBCO Softwarecommand line, API, JMX59

“Business value doesn’t come frombuilding faster applications; it comesfrom building applications faster.”60

Event Processing Connectivity 2014: 150 Streaming Source & Sinks Enterprise Messaging Data Grid / Cache Hadoop (Flume)Amazon KinesisTwitter StormSpark ActiveSpacesTIBCO Distributed Transactional MemoryMongoIntegrationBusinessWorksInternet of Things TIBCO RendezvousTIBCO EMSTIBCO FTLIBM MQSeriesInformatica 29WestSolaceTervelaJMS (publish / subscribe)TCP/IPBig Data OSI PiMQTT (MQ Telemetry Transport)InsteonX10XBee,Google GlassLive cameras for face detectionBeaconsSDR (software defined radio) for planetrackingAR Drone quadcopterOBD-II car on-board computerdiagnostic adapterXbox Database TechnologySocial VerticaOracleSQL ServerMySQLIBM DB2Sybase ASE / IQ / RAPActive Oracle log readerActive SQL Server log readerTwitter150 adapters to Streaming,GNIPSocial, Hadoop , IoT,TIBBRIntegration, Technology dataJabber / IRC sourcesRSSThis document (including, without limitation, any product roadmap or statement of direction data) illustrates the planned testing, release and availability dates for TIBCO products and services. It is for informational purposes only andits contents are subject to change without notice. Copyright 2000-2014 TIBCO Software Inc. All rights reserved. TIBCO Confidential & Proprietary Information. Binary file (read and write) CSV files and sockets (read andwrite) Email (read and writer) IP packet capture (HTTP / IRC /POP3 / SMTP) Log files Regular expression file and socket

“The LiveView Datamart is like a datawarehouse for data that’s constantly inmotion, and changing all the time.”62

The Birth of the Live HE END OF THEEND OF DAY Network sOperationsCenterContinuousComplianceMonitoring63

TIBCO Live DatamartEMSFTLBusinessEventsLive DatamartAd Hoc QueryActiveSpacesBusinessWorksContinuous Query ProcessorSocial Media DataJDBCActiveSpacesMarket DataSensor DataContinuous QueryHistoricalDataEnterprisedataAlerts

Fast Geo-Spatial Data65

Live Datamart Client APIs Open Client Support–––– HTML5– D3, jQuery, ExtJS,Google Charts, Bing,AngularJS.NET– For custom GUIdevelopmentAllows open partnerdevelopmentAd-hoc queryNotificationsFor custom .NETdevelopmentJava–For custom Java GUIdevelopment

To automate your business with analytics,you must give the businessvisibility and controlUnderstandControlAutomate68

Questions?How will you automate with analytics?How will you give visibility and control?UnderstandControlAutomate70

More info at tibco.comOr at our booth hereFollow me @tibbettsFollow up with tib

Lessons Learned from Key Fast Data Use Cases TIBCO Event Processing Agenda. Big Data: There can be an overwhelming volume . But how will you access the data when you need it? Fast Data FAST DATA IS PROCESSING BIG DATA IN REAL-TIME TO GAIN INSTANT AWARENESS AND INSTANT ACTION. 11 To automate