WIXLIB

StreamInsight를 통한실시간 이벤트 데이터 처리홍세환 인브레인

발표자 소개홍세환 ��팅사업부 부장개발 PMC#

목차 StreamInsight 소개– Complex Event Processing– StreamInsight Architecture Application Component––––––Event (Flow, Control and Definition)LINQ & WindowAdaptersDevelopment ModelDeployment ModelMonitoring & Event Flow Debugger Usage Example

StreamInsight 소개(Complex Event Processing)Event Driven Architecture (Event Processing Styles) – Wikipedia Simple event processing– Simple events can be created by a sensor detecting changes intire pressures or ambient temperature. Event stream processing– Stream event processing is commonly used to drive the real-timeflow of information in and around the enterprise, which enablesin-time decision making. Complex event processing– CEP is commonly used to detect and respond to businessanomalies, threats, and opportunities.Brenda M. Michelson, Event-Driven Architecture Overview, Patricia Seybold Group,February 2, 2006

StreamInsight 소개(Complex Event Processing)Complex Event Processing Complex Event Processing (CEP) is the continuous andincremental processing of event streams from multiple sourcesbased on declarative query and pattern specifications withnear-zero latencyThe Goals of CEP Identify and detect from seemingly unrelated multiple events:– Meaningful patterns, Relationships– Trends, Gaps (expected events that did not occur)– Abstractions– Exceptions, Opportunities Analyze data without storing it first Trigger immediate response actions Mine events for new business KPIs

StreamInsight 소개(Complex Event Processing)The Value of Timely Analytics value ofanalyticsWeb Analytics – Ad placement,Financial Services, Smart Grids,Monitoring – Systems mgmt,Health Care, Manufacturing, etcForecasting in EnterprisesHistorical Trend AnalysisyearsmonthsdaysTime of interesthrsminsecPresent

StreamInsight 소개(Complex Event Processing)Web Analytics Demo(Package StreamInsightSolution)

StreamInsight 소개(Complex Event Processing)Web Analytics Demo(Package StreamInsightSolution)

StreamInsight 소개(Complex Event Processing)CEP Sweet SpotMonthsCEP Target ScenariosDaysRelational Database ApplicationsHoursOperational AnalyticsApplications (e.g., Logistics)Data WarehousingApplicationsWeb Analytics ApplicationsMinutesSeconds100 msMonitoringApplicationsManufacturing Applications 1 ms010100100010000Aggregate Data Rate (Events/sec)Financial TradingApplications100000higher

StreamInsight 소개(StreamInsight Architecture)StreamInsightApplication DevelopmentEvent sourcesDevices, SensorsStreamInsight Application at RuntimeInputAdaptersStreamInsight EngineOutputAdaptersEvent targetsPagers &Monitoring devicesStanding QueriesWeb servers QueryLogicKPI Dashboards,SharePoint UIQueryLogicTrading stationsEvent stores & DatabasesStock ticker, news feedsQueryLogicEvent stores & Databases

StreamInsight 소개(StreamInsight Architecture)StreamInsight VS. DBMSAnalytical results need to reflect important changes in business realityimmediately and enable responses to them with minimal latencyDatabaseCEPQueriesAd hoc on stored dataContinuous standing queriesLatencySecondsMillisecondsData RateHundreds per SecondTens of thousands per secondQuerySemanticsDeclarative relational analyticsDeclarative relational and tputstream

StreamInsight 소개(StreamInsight Architecture)SQL Server Capabilities by EditionStandardCustom/Packaged OLTPApps4 procs,64GB RAM,BackupCompression8 procs,2TB RAM,Adv. Security,Backup Compression 8 procs,OS Max,Adv. Security,Backup CompressionN/A1 VM/license4 VMs/license,Resource GovernorApp & Multi-Server Mgmt(up to 25 instances)Unlimited Virtualization,Resource Governor, App &Multi-Server Mgmt ( 25instances)N/AScale-Up DW,Data CompressionScale-Up DW,Data CompressionScale-Out DW10s of TBs, Up to 30 TBwith FastTrack10s of TBs10s - 100s of TBsEnterprise-Scale BI,Master Data Services,PowerPivot MgmtEnterprise-Scale BI, MasterData Services, PowerPivotMgmt 5000events/sec& 5 s latency 5000events/sec& 5 s latencyServer ConsolidationData WarehousingBusiness IntelligenceDept/Team BI 5000StreamInsightevents/sec& 5 seclatencyEnterpriseDatacenterParallel DataWarehouseWorkloadIntegrated with SSIS,SSAS and SSRSFuturecoverage

Application Component Event Stream Concept.NET 3.5 SP1, 4.0 and CLR for the runtimeVisual Studio for developer productivity (IntelliSense)C# for apps development––––Adapter, Object ModelDiagnostic APIsIEnumerable – Pull ModelIObservable – Push Model LINQ for Query Language surface– Anonymous Method– Lambda Expression– SQOs (Standard Query Operators) Event Flow DebuggerMicrosoft StreamInsightTMEvent Flow Debugger

Application ComponentEvent (Flow, Control and Definition)Streams All data organized into Streams, Potentially unending Data may change over time Often represent the same value over timeEvents Data in streams is packaged into events Two parts to an event– Header (kind of event and timestamps)– Payload (.NET data structure) Timestamps are DateTimeOffset data type– All times normalized to UTC in serverTimestamps/MetadataC LongpumpID StringType StringLocation Doubleflow Doublepressure

Application ComponentEvent (Flow, Control and Definition)Event Model (Event Shapes) - Point -2702:04:00.2132009-12-2702:04:00.213 02:04:04.329 02:04:04.234 cEU-23423-18

Application ComponentEvent (Flow, Control and Definition)Event Model (Event Shapes) - Interval 04:04.523EU-23423-18

Application ComponentEvent (Flow, Control and Definition)Event Model (Event Shapes) - Edge TStart2009-12-2702:04:00.213 4:04.234 2702:04:08.238EU-23423-18

Application ComponentEvent (Flow, Control and Definition)Event Kind Two event kinds– INSERT (new data for the stream)– CTI (Current Time Increment) CTI– Added to input stream Typically added by input adapter Can be declaratively added in query binding– Processing out of order data– Responsive stream Update – in Edge Event but Insert EndTime Delete – no concept of deleting, ReleaseEvent

Application ComponentEvent (Flow, Control and Definition)Stream Liveliness?01234567Time

Application ComponentLINQ & WindowLINQ UPD05-HOL-01\Source\Assets\DisplayQueryVersion)

Application ComponentLINQ & WindowLINQ UPD05-HOL-01\Source\Assets\DisplayQueryVersion)

Application ComponentLINQ & WindowLINQ UPD05-HOL-01\Source\Assets\DisplayQueryVersion)

Application ComponentLINQ & WindowLINQ Query ExampleLINQ Example – JOIN, PROJECT, FILTER:from e1 in MyStream1join e2 in MyStream2on e1.ID equals e2.IDwhere e1.f2 “foo”select new { e1.f1, e2.f4 };JoinFilterProjectLINQ Example – GROUP&APPLY, WINDOW:from e3 in MyStream3group e3 by e3.i into SubStreamsfrom s4 in SubStreamsfrom e4 in s4.HoppingWindow(FiveMinutes,ThreeSeconds)select new {pl new MyNewPayload(e4.i, e4.f)};GroupingWindow

Application ComponentLINQ & WindowCEP Query Features Operators over LTERGROUP & APPLYSUM, COUNT, TOP-KTemporal operations - window Extensibility – to add new domain-specific operators Queries are written over specific event types Support for streaming data, reference data (lookup), andhistorical data (replay)

Application ComponentLINQ & WindowQuery Elements : Event WindowsHopping WindowTumbling �창(Window Size 4Hop Size 1)e1e2시간e1e2ResultEventse1e2e1e2Count Window입력이벤트Snapshot Window입력이벤트e1e2e3시간시간e1e2개수 (N 2)e2e3스냅숏 창

Application ComponentLINQ & WindowQuery Elements : Event Window Demo

Application ComponentLINQ & WindowQuery Elements : Event Window Demo

Application ComponentLINQ & WindowQuery BindingAnalytics & QueriesData SourcesQT1CEP EngineAT1EventStanding QueriesEventQ1EventQ1’EventQ1’’EventOutput AdaptersAT3Input AdaptersAT2Event

Application ComponentAdaptersEvent Stream Adapters : Push vs. Pull Model Interaction points support Push and Pull Consumer can throttle input범례Event PushEvent PullThrottle (process overflow)READENQUEUE시장 피드 컴퓨터가생성됨낮음InputAdapter1지연 스로틀기능InputAdapter2높음DB, 파일DEQUEUEWRITE높음CEP serverOutputAdapter높은 처리량을처리할 수 있는 기능낮음

Application ComponentAdaptersAdapter Development TasksDetermine Event Type, Event Model, andEvent KindChoose Appropriate Adapter Base ClassDesign Flow Control Logic for READ andWRITEDesign ENQUEUE or DEQUEUE FlowControl InteractionDesign AdapterFactory Object

Application ComponentDevelopment ModelStreamInsight Development Two ModelsImplicit ServerExplicit ServerEasiestMost FlexibleHides most complexityProvide complete control ofStreamInsight application,Development EnvironmentAllows developers to focus onquery logicAllows for reuse Queries, Adapters,Event Types, Third-party Query TemplatesQuery is automatically hostedCode creates all objects and registersthem into serverStored in memory, not on stablestorage (disk)Server can store metadata in memory orpersist to diskServer can run locally or remotely viaWeb service

Application ComponentDeployment ModelCEP Deployment ScenariosScenario 1: Custom CEP Application Dev Scenario 2: Embed CEP in Application.NET, C#, LINQCustom CEP ApplicationISV Application with CEP EngineCEP EngineCEP EngineReference dataScenario 3: CEP Enabled DeviceReference dataScenario 4: Operational Intelligence w/ CEPKPIsKPI miningDevice with Embedded CEP EngineCEP EngineCEP EngineETL Pipeline with CEP engineCEPCEPCEPCEPMadison

Application ComponentDeployment ModelCEP Deployment alternativesWeb serversData SourcesCEPSensorsCEPFeedsDevicesCEPAggregation &CorrelationCEPCEPEvent processing engines are deployedat multiple places on different scalesCEP At the edge – close to the data source In the mid-tier – consolidate relateddata sources In the data center – historical archive,mining, large scale correlation.CEPCEPCEPComplex Analytics &MiningCEPCEP for lightweight processing and filteringCEPCEP for aggregation and correlation ofin-flight eventsCEPCEP for complex analytics includinghistorical data

Application ComponentMonitoring & Event Flow DebuggerMonitoring StreamInsight Need to track : Overall health of system, Query performance Diagnostic views using ManagementService API– Use URIs for resource naming– GetDiagnosticView(), SetDiagnosticSettings(), ClearDiagnosticSettings()– Can be accessed via PowerShellQuery Monitoring PointsOutgoing latencyProduced latencyConsumer latencyInputAdapter 1CEP serverOutputAdapterInputAdapter 2IncomingConsumedProducedOutgoing

Application ComponentMonitoring & Event Flow DebuggerEvent Flow Debugger Demo(TechEdUS2010)

Application ComponentMonitoring & Event Flow DebuggerEvent Flow Debugger Demo(TechEdUS2010)

Usage ExampleCapital MarketsDemo(TechEdUS2010)

Usage ExampleCapital MarketsDemo(TechEdUS2010)

Usage ExampleCapital MarketsDemo(TechEdUS2010)

Usage ExampleManufacturing:Web Analytics:Financial Services: Sensor on plant Click-stream data Stock & news feedsfloor Online customer Algorithmic trading React throughbehavior Patterns over timedevice controllers Page layout Super-low latency Aggregated data 100,000 events /sec 100,000 events /sec 10,000 events/secPower, Utilities: Energyconsumption Outages Smart grids 100,000 events/secStream DataStore & ArchiveVisual trend-line and KPI monitoringBatch & product managementAutomated anomaly detectionReal-time customer segmentationAlgorithmic tradingProactive condition-based maintenanceAsset Specs &ParametersData StreamData StreamAsset Instrumentation for Data Acquisition, Subscriptions to Data FeedsEvent Processing EngineLookup Threshold queries Event correlation frommultiple sources Pattern queries

Competitive LandscapeProgress – ApamaIBM – Open ESBWest Global – Total InsightCoral8Bristol Technology – SenActiveGmbH – Real Time MonitoringStreambaseAgent LogicInetcoTibco - BusinessEventsOracle – Fusion MiddlewareEvent ZeroSynderaRulecareLG CNS Event ProSybase Aleri Event Stream ProcessorIndustry Forum: http://complexevents.com