Transcription
Advanced NetFlow for Service ProvidersAamer Akhter (aa@cisco.com)Benoit Claise (bclaise@cisco.com)1
Agenda IntroductionNetFlow Version 9Interesting Features on Traditional NetFlowFlexible NetFlowNetFlow for SecurityNetFlow for Application VisibilityNetFlow Performance2
NetFlow – What is it? Developed and patented at Cisco Systems in 1996 NetFlow is a standard for acquiringIP operational data Provides network and securitymonitoring, network planning,traffic analysis, and IP accounting IETF’s IPFIX (RFC 5101) basedon NetFlow v9 (with changes)Network World Article – NetFlow Adoption on the 005/0314nsm1.html3
Key Concept — *Flow Scalability Packet capture is like a wiretap *Flow is like a phone bill This level of granularity allows *Flow to scale for verylarge amounts of trafficWe can learn a lot from studying the phone billWho’s talking to whom, over what protocols and ports, for howlong, at what speed, for whatduration, etc.*Flow is a form of telemetry pushed from the routers/switches —each one can be a sensor4
Key Concept — *Flow Follows the Topology Network traffic is often asymmetrical, even on smallnetworks Probes typically require engineered symmetry This means that with *Flow, there’s no need to engineerthe network around the instrumentationWe can follow traffic through the network over its natural pathWe can see pps, bps, packet-size, QoS markings, TCP flags,etc. for specific apps/services at each point in the networkWe can validate traffic engineering, policy enforcement, etc. atany point in the topology, as long as *Flow is enabled5
Why *Flow?Network Operator Benefits UnderstandProductivity and utilization of assets in the networkApplication and network usageImpact of network changes and servicesNetFlow answers the who, what, when, where, and how network traffic is flowing Detect and classify security incidents with proven threat defence Improve network usage and application performance6
Principal *Flow UsesService ProviderEnterprisePeering ArrangementsInternet access monitoring (protocoldistribution, where traffic isgoing/coming)Network PlanningUser MonitoringTraffic EngineeringApplication MonitoringAccounting and BillingChargeback billing for departmentsSecurity MonitoringSecurity MonitoringPerformance MonitoringPerformance Monitoring7
Examples of useCustomer ChallengeSecurityTraffic AnalysisTraffic AnalysisDescriptionProblem SituationNetFlow ResolutionDetect SQL Slammer on dayoneDetrimental incapacity ofserversNetFlow day-zero anomalydetectionBandwidth HogFull Circuit- Sluggish networkperformance- Single user applicationmonopolizing networkCircuit 100% utilized- More servers andbandwidth addedCapacity PlanningSlow network performance- Users still complained- Rented RMON probes didn’t workCapacity PlanningPoor network performance –low bandwidthWe need more bandwidthCost savings of 7k in laborcostsQuickly tracked problem andsaved 300 hours 34k inlabor costsCost savings of 126k inprobe costsTracked point of slowdown –saved 36k per yr. circuits8
NetFlow—nfdump and nfsenSource: http://nfsen.sourceforge.net9
NetFlow—StagerSource: UNINETT10
NetFlow—Stager (Cont.)Source: UNINETT11
NetFlow—Stager (Cont.)Source: UNINETT12
Arbor Peakflow SP — ApplicationDistribution13
Example—SQL Slammer14
*Flow and SNMP All the *flow protocols are essentially pushtechnologies:Information is sent asynchronously from measurement nodePost-processing (aggregation) might be done onrouter/switchInformation is exported and usually immediately expiredNMS does not decide on rate of information SNMP is a pull technologyNMS needs to decide when and how often to poll deviceDevice may not retain information when polledDevice does not decide on rate of informationCorrelated information may require multiple transactions15
BGP Accounting BGP accounting provides a way of getting prefix trafficinformation BGP prefixes are colored in one of X colors Each color has counters (byte/packet) associated When traffic from/to prefix is received/sent counters areincremented. Easy form of aggregation – usually limited 8 buckets Information organized only around buckets, noadditional information provided.16
sFlow Created by InMon (sells sFlow collectors) sFlow v2, v4 and v5 (v2 and v4 deprecated) sFlow somewhere in between NFv8 andNFv9/IPFIXExtendable set of fields – called structures (fixed templates)sampling (sampling is required part of sFlow) Supported by (generally switch vendors) Alcatel,Extreme, Force10, HP, Hitachi (*) Incompatible with NetFlow V9/IPFIX, but somecollectors support both NFv9 and sFlow* Complete list at sFlow.org17
J-Flow / cflow J-Flow and cflowd is essentially NetFlow NetFlow collectors will support J-Flow/cflow output J-Flow and cflow are terms used by Juniper(v5, v8,v9) cflow term used by Alcatel (v5, v8, v9) Implementations do not support flexible templates18
NetStream NetStream comes in three formats: v5, v8 and v9Essentially mirroring NetFlow v5, v8 and v9 Generally easily supported by NetFlow collectorsHowever differences exist between NetStream v9 and NFv9eg: NetFlow represents interfaces using ifIndex(standard MIB), NetStream represents using proprietaryinterface MIBCollector needs to be explicitly told record is NetStream NetStream is supported by 3COM and Huawei Implementations do not support flexible templates19
NetFlow V5, v8 and v9 export formats exist IETF standard (IPFIX) is based on v9 NetFlow v9 Documented in RFC3954 (informational)Lack of regulation has lead to minor (and corrected) collisions in field(nProbe) identifiers Enjoys wide collector support. NFv9 collectors generally need minor tweaks to support IPFIX Supported by cisco, Alcatel, Juniper (as J-Flow/cflow),Packeteer (v5), 3COM/ Huawei (sort of), Riverbed, Adtran,Enterasys, wide open-source support Cisco implementations support flexible templatesProviding flexible reports down the field level20
IETF: IP Flow Information ExportWG (IPFIX) IPFIX protocol specificationsChanges in terminology but same NetFlow Version 9 principles(IPFIX version field says ‘10’) Improvements vs. NetFlow v9: SCTP-PR, security, variablelength information element, IANA registration, etc. Generic streaming protocol, not flow-centric anymore Security:Threat: confidentiality, integrity, authorizationSolution: DTLS on SCTP-PRAnonymization draft IPFIX information modelMost NetFlow v9 information elements ID are keptProprietary information element specification21
IETF: IP Flow Information ExportWG (IPFIX) RFC3954 Cisco Systems NetFlow Services ExportVersion 9 RFC3917 Requirements for IP Flow Information ExportGathers all IPFIX requirements for the IPFIX evaluation process RFC3955 Evaluation of Candidate Protocols for IPFIX RFC5101 Specification of the IPFIX Protocol for theExchange of IP Traffic Flow Information RFC5102 Information Model for IP Flow InformationExport RFC5103 “Bidirectional Flow Export using IP FlowInformation Export (IPFIX)”22
IPFIX: Interesting Drafts Export of Application Information in IPFIXdraft-claise-export-application-info-in-ipfix Exporting MIB variables using the IPFIX Protocoldraft-johnson-ipfix-mib-variable-export Export of Structured Data in IPFIXdraft-ietf-ipfix-structured-data IP Flow Anonymisation Supportdraft-ietf-ipfix-anon Information Elements for Flow PerformanceMeasurementdraft-akhter-ipfix-perfmon23
IETF: Packet Sampling WG (PSAMP) PSAMP is an effort to:Specify a set of selection operations by which packets aresampled, and describe protocols by which information on sampledpackets is reported to applications Sampling and filtering techniques for IP packet selectionTo be compliant with PSAMP, we must implement at least one ofthe mechanisms: sampled NetFlow, NetFlow input filters arealready implemented PSAMP protocol specificationsAgreed to use IPFIX for export protocol Information model for packet sampling exportExtension of the IPFIX information model24
In the olden times Flow was Defined By Seven Unique KeysTraffic Source IP address Destination IP addressEnable NetFlowNew SNMPMIBInterface Source port Destination port Layer 3 protocol type TOS byte (DSCP) Input logical interface(ifIndex)NetFlowExport PacketsTraditional Export &CollectorGUI 2004 Cisco Systems, Inc. All rights reserved.SNMP Poller2525
NetFlow Processing OrderPreProcessing Packet Sampling FilteringFeaturesandServices IPMulticastMPLSIPv6PostProcessing Aggregationschemes Non-key fieldslookup Export26
KeyfieldsNetFlow Cache Example1. Create and update flows in NetFlow 510.0.23.2104024.514 Inactive timer expired (15 sec is default) Active timer expired (30 min is default) NetFlow cache is full (oldest flows are expired) RST or FIN TCP flag2. 1180101100000A2/24500A2/241510.0.23.21528180043. Aggregation4. Export version5. Transport protocol(UDP, SCTP)ExportPacketHeaderNon-aggregated flows—export version 5 or 9Payload(Flows)E.g., Protocol-Port AggregationScheme 0A200A21528Aggregated Flows—Export Version 8 or 927
Extensibility and Flexibility RequirementsPhases Approach Traditional NetFlow with the v5 or v8 NetFlow exportReally needed something flexible and extensible Phase One: NetFlow Version 9Advantages: extensibilityIntegrate new technologies/data types quicker(MPLS, IPv6, BGP next hop, etc.)ExportingProcessIntegrate new aggregations quicker Phase Two: Flexible NetFlowAdvantages: cache and export content flexibilityUser selection of flow keysUser definition of the recordsMeteringProcess28
NetFlow Open Source ToolsProduct NamePrimary UseCommentOSCflowdTraffic AnalysisNo longer supportedv5, v8UNIXFlow-toolsCollector Devicev5, v8, v9 (only old fields)UNIXFlowdCollector DeviceV5, v7, and v9BSD, LinuxFlowScanReporting for Flow-Tools-UNIXIPFlowTraffic AnalysisSupport V9, IPv4, IPv6,MPLS, SCTP, etc.Linux, FreeBSD,SolarisNetFlow GuideReporting ToolsNetFlow MonitorTraffic AnalysisSupports V9UNIXNetmetCollector Devicev5, support v9LinuxNTOPSecurity Monitoringv9UNIXStagerReporting for Flow-ToolsNfdump/nfsenTraffic AnalysisDifferent costs: implementation and customizationBSD, LinuxUNIXV5, v7, v9UNIX29
NetFlow v930
NetFlow Version 9 Export PacketTemplate FlowSetHEADERTemplateRecordTemplateID #1TemplateRecordTemplateID #2(SpecificFieldTypes andLengths)(SpecificFieldTypes andLengths)Template 1Template 2Data FlowSetFlowSet ID #1Data FlowSetFlowSet ID #1FlowSetID dValues)(FieldValues)31
NetFlow Version 9 Export PacketOptions Template FlowSet Specifies the Scope: Cache, System,Template, etc.Template 3HEADEROptionsTemplate FlowSetData FlowSetFlowSet ID #3OptionTemplate RecordTemplateID #3OptionDataRecordOptionDataRecord(Specific Scope,Field Typesand Lengths)(FieldValues)(FieldValues)32
Interesting Features on TraditionalNetFlow33
Multicast & NetFlow Multicast NetFlow ingressSees incoming mcast flowfan out is not represented as there are multiple interfacesByte counts do not include replication Multicast NetFlow egressSees outgoing multicast packetsfan out is represented by multiple cache entries (one per output interface) New fields that represent the size of OIL (output interface list) Display the multicast data that fails the Reverse Path Forwarding(RPF) check No NetFlow export over multicast34
IPv6 and NetFlow New NetFlow fields represent IPv6 header fieldsNeeds NFv9 to exportLack of IPv6 capable NetFlow collectors (chicken or egg situation)Currently need it export records about IPv6 via IPv4 A flow is either IPv4 or IPv6!Separate metering and export for v4 vs. v6, otherwise waste of exportbandwidth.35
NetFlow Reliable Export with SCTP SCTP: stream control transport protocol (RFC4960)Reliable data transferCongestion control and avoidanceMultihoming supportOne association support for multi-streams SCTP-PR: SCTP partially reliable (RFC3578)Three modes of reliability: reliable, partial reliable, unreliable Advantages: (Options) templates sent reliably Backup Options:Fail-over mode: open the backup connection when the primary failsRedundant mode: open the backup connection in advance, andalready send the templatesNote: “An Introduction to SCTP”, RFC 328636
NetFlow Reliable Export with SCTPSecurity/MonitoringSCTP Backup:Fail-Over iableSCTP Backup:Redundant ModeDestinationPrefix Aggr.37
Flexible NetFlow38
Typical NetFlow DeploymentNetFlow for MonitoringNetFlow for SecurityNetFlow for CoreTraffic MatrixInteresting Features on Traditional NetFlowManaged Services:Application VisibilityISPNetFlow for Peering39
Flexible NetFlowHigh-Level Concepts and Advantages Flexible NetFlow feature allows user configurableNetFlow record formats, selecting from a collectionof fields:Key, non-key, counter, timestamp Advantages:Tailor a cache for specific applications, not covered byexisting 21 NetFlow features in traditional NetFlowDifferent NetFlow caches: per subinterface, per direction(ingress, egress), per sampler, per Better scalability since flow record customization forparticular application reduces number of flows to monitor40
Flexible NetFlowMultiple Monitors with Unique Key FieldsTrafficFlowMonitor1FlowMonitor2Key FieldsPacket 1Non-Key FieldsKey FieldsPacket 1Non-Key FieldsSource IP3.3.3.3PacketsSource IP3.3.3.3PacketsDestination IP2.2.2.2BytesDest IP2.2.2.2TimestampsSource Port23TimestampsInput InterfaceEthernet 0Destination Oort22078Next Hop AddressSYN Flag0Layer 3 ProtocolTCP - 6TOS Byte0Input InterfaceEthernet 0Traffic Analysis CacheSecurity Analysis InputI/F Pkts3.3.3.32.2.2.2232207860E0 1100Source IPDest. IPInput I/FFlag Pkts3.3.3.32.2.2.2E00 1100041
Flexible NetFlow ModelInterfaceMonitor “C”Monitor “A”Exporter “M”Monitor “B”Record “X”Record “Z”Exporter “M”Exporter “N”Record “Y” A single record per monitor Potentially multiple monitors per interface Potentially multiple exporters per monitor42
Flexible Flow Record: Key FieldsFlowIPv4SamplerIDorIP (SourceDestination)DirectionPrefix (Source orDestination)InterfaceMask (Source r2Destination)SourceVLANProtocolIPv6Payload SizePacket Section(Header)Packet tationVLANVersionFlagsFragmentationSource MAC onHeaderLengthTOSMAC addressTotal LengthIP (Source orDestination)Payload SizePrefix (Source orDestination)Packet Section(Header)Mask (Source orDestination)Packet Section(Payload)Minimum-Mask(Source orDestination)DSCPProtocolExtension HeadersTraffic ClassHop-LimitFlow LabelLengthOption HeaderNext-headerHeader LengthVersionPayload Length43
Flexible Flow Record: Key FieldsRoutingTransportsrc or dest ASDestination PortTCP Flag: ACKPeer ASSource PortTCP Flag: CWRTraffic IndexICMP CodeTCP Flag: ECEForwardingStatusICMP TypeTCP Flag: FINIGMP Type*TCP Flag: PSHTCP ACK NumberTCP Flag: RSTTCP Header LengthTCP Flag: SYNTCP Sequence NumberTCP Flag: URGTCP Window-SizeUDP Message LengthTCP Source PortUDP Source PortTCP Destination PortUDP Destination PortIGP Next HopBGP Next HopInput VRFNameNEWTCP Urgent PointerNEWApplicationApplication ID*MulticastReplicationFactor*RPF CheckDrop*Is-Multicast*: IPv4 Flow only44
Flexible Flow Record: Non-Key FieldsCountersTimestampIPv4IPv4 and IPv6BytessysUpTime FirstPacketTotal LengthMinimum (*)Total LengthMinimum (**)Bytes LongsysUpTime FirstPacketTotal LengthMaximum (*)Total LengthMaximum (**)Bytes Square SumTTL MinimumBytes Square Sum LongTTL MaximumPacketsPackets Long Plus any of the potential “key” fields: will be the value fromthe first packet in the flow(*) IPV4 TOTAL LEN MIN, IPV4 TOTAL LEN MAX(**)IP LENGTH TOTAL MIN, IP LENGTH TOTAL MAX45
Three Types of NetFlow Caches Normal cache (traditional NetFlow)More flexible active and inactive timers: one second minimum Immediate cacheFlow accounts for a single packetDesirable for real-time traffic monitoring, DDoS detection, loggingDesirable when only very small flows are expected (ex: sampling)Caution: may result in a large amount of export data Permanent cacheTo track a set of flows without expiring the flows from the cacheEntire cache is periodically exported (update timer)After the cache is full (size configurable), new flows will notbe monitoredUses update counters rather than delta counters46
NetFlow Deployment ScenariosNetFlow for CoreTraffic MatrixSecurity Flow Monitor Protocol Ports IP addresses TCP flagsManaged ServiceApplicationVisibility IP addresses Application DSCPServer Flow Monitor Standard seven keys Source/destination AS IP addresses (src/dest) BGP next hop Protocols DSCPISPPeering Flow Monitor Destination AS Source traffic index BGP next hop DSCP47
NetFlow and Capacity Planning48
The Core Traffic Matrix TrafficEngineering and Capacity PlanningParis POPRome POPISP-1SLASourceISP-2BusinessCritical TrafficDestinationBest EffortTrafficBest EffortLondon POPMunich POPRome Exit PointParis Exit PointLondon Exit PointMunich Exit PointRome Entry PointNA (*) Mb/s Mb/s Mb/sParis Entry Point Mb/sNA (*) Mb/s Mb/sLondon Exit Point Mb/s Mb/sNA (*) Mb/sMunich Exit Point Mb/s Mb/s Mb/sNA (*)(*) Potentially Local Exchange Traffic49
Core Capacity Planning the Big Picture1. The ability to offer SLAs is dependent upon ensuringthat core network bandwidth is adequately provisioned2. Adequate provisioning (without gross overprovisioning) is dependent upon accurate corecapacity planning3. Accurate core capacity planning is dependent uponunderstanding the core traffic matrix and flows andmapping these to the underlying topology4. A tool for what if scenarios50
BGP Next Hop TOS AggregationTypical ExampleAS1CustomersAS2AS3AS4AS5PEPEMPLS CoreorIP Core with BGP Routes OnlyPEPEPEPEPoPPoPServer Farm 1CustomersServer Farm 2Internal Traffic: PE to PEExternal Traffic Matrix PE to BGP AS51
NetFlow BGP Next Hop TOSAggregation Flow KeysKey Fields (UniquelyIdentifies the Flow)Additional Export Fields Origin AS Flows Destination AS Packets Inbound Interface Bytes Output Interface First SysUptime ToS/DSCP (*) Last SysUptime BGP Next Hop(*) Before Any Recoloring52
Core Traffic Matrix with Flexible NetFlowKey Fields (UniquelyIdentifies the Flow)Additional Export Fields Origin AS Flows Destination AS Packets Inbound Interface Bytes Output Interface First SysUptime ToS/DSCP (*) Last SysUptime BGP Next Hop Less flow records, less CPU impact Potentially choose higher sampling rate for a better accuracy(*) Before Any Recoloring53
NetFlow and Security Analysis54
What Does a DoS Attack Look Like?Router# show ip cache flow SrcIf SrcIPaddress SrcP SrcAS29192.1.6.6977aaa29192.1.6.222 1243aaa29192.1.6.108 29192.1.6.291209aaaDstIf 1618941600DstASbbbbbbbbbbbbbbbbbbbbbbbbbbbPr Pkts B/Pk614061406140614061406140614061406140 Typical DoS attacks have the
ð§sFlow v2, v4 and v5 (v2 and v4 deprecated) ð§sFlow somewhere in between NFv8 and NFv9/IPFIX Extendable set of fields called structures (fixed templates) sampling (sampling is required part of sFlow) ð§Supported by (generally switch vendors) Alcatel, Extreme, Force10, HP, H
