Transcription
Flow-Based Monitoring, Troubleshootingand Security using nProbeLuca Deri deri@ntop.org @lucaderi 2017 - ntop.org
Packets, Flows, Activities [1/3]For years monitoring tools focused on standards oftenfostered by vendors: NetFlow vs sFlow vs SNMP, Cisco vsJuniper This has plagued the market by creating tools morevendor oriented, than result oriented. Fortunately recent advances in computing and inparticular the big data movement, have pushedcompanies to overcome the market/vendor fragmentationand produce tools able to produce data on a standardformat (often JSON) that could be consumed even bynon-monitoring tools (e.g. Hadoop, ElasticSearch). 2017 - ntop.org2
Packets, Flows, Activities [2/3] Asdata increases and people demand feature richmonitoring tools, it has become necessary to ‘compress’monitoring data. Network packets are still important for providingevidence or troubleshooting problems (packets or itdidn't happen!) but they are “too raw” and take toomuch storage space, so limiting them to specificsituations is a good idea. Network flow analysis is a good way to “compresspackets” into events: sFlow do it with sampling, NetFlowwith stateful connection-based packet classification. 2017 - ntop.org3
Packets, Flows, Activities [3/3] These days, saving flows on a big data system is a commonpractice but it still plagued by the visibility issue: What flows are “more relevant” than others? Can we use flows for more than just host/protocol/applicationtraffic accounting ? How can a network administrator look for a needle in a haystackwhen the monitoring platform is emitting tenth of thousand flows/second? We need yet another level of abstraction on top of flowsable to identify activities on top of flows (e.g. these 20HTTPS connections and 5 DNS queries mean that host Xjust open the landing page of newspaper corriere.it). 2017 - ntop.org4
Flow Generation [1/2]Unfortunately there are still too many “NetFlowdialects” (e.g. Cisco ASA or Barracuda Networks flows)available that make interoperability not that simple sFlow is even simpler than NetFlow/IPFIX to implementand available in most switches deployed today (Ciscofeatures a sFlow-like protocol named NetFlow Lite). With the baseline bytes/packets of traffic flow fromthese flow protocols, we can do a lot with goodanalytics. This including congestion, cost analysis, DDoSdetection, security and forensics. 2017 - ntop.org5
Flow Generation [2/2]Ideally, we want to gather rich measurement metrics,from everywhere possible. For the above goals 5-tuples (IPs, Ports, Protocol) andand bytes/packets are not enough as we expect atleast: Latency, Packet Drops, Retransmissions. QoE (e.g. HTTP service time). Application visibility (DPI, URLs, DNS responses). And with those metrics per flow, we can provide evenmore actionable insights into performance and securityissues. 2017 - ntop.org6
Do We Need Custom Probes in 2017? [1/2]Flow-based traffic analysis relies on network probesthat are usually implemented in network devicessuch as routers and firewalls. Unfortunately being probe development driven byvendors, probes have not evolved much over thepast 10 years, and where it happened, it as not beenin a standard way. Most collectors complexity is due to the support ofvarious IPFIX/NF dialects rather on using thecollected data. 2017 - ntop.org7
Do We Need Custom Probes in 2017? [2/2]More than 10 years ago, (at ntop) we realisedthat improvements in network monitoring werelimited by the ability of network probes togenerate rich monitoring data. In essence we cannot expect to move forwardwaiting the Cisco, Juniper to embed in routersa next-generation probe. This was the motivation to create our ownsoftware probe, named nProbe. 2017 - ntop.org8
nProbe [1/4] nProbe is a high-speed (multi 10G) traffic probe/collector for Linux and Windows platforms.DiskDatabaseMySQL/SQLiteRaw FilesRedisnLog ManagersFlowExportnProbeJSONIncoming Packets(Raw Traffic)JSONetFlow/IPFIXsFlow 2017 - ntop.orgNntopng9
nProbe [2/4] Originally designed as a drop-in replacement of aphysical NetFlow probe, currently it can: Convert flow format (sFlow-to-NetFlow/IPFIX) orversion (e.g. v5 to v9). High-speed packet-to-flow processing. Leverage on in-memory-databases to maintain flowstate coherency (e.g. SIP/RTP, Radius/Diameter or GTPtraffic). Ability to pre-compute data for realtime trafficaggregation. 2017 - ntop.org10
nProbe [3/4] It has an open architecture extensible by meansof plugins that include: GTP (v0, v1, v2) plugins. VoIP (SIP and RTP) plugins for analysing voice signalling(who’s calling who/when) and voice quality (Jitter andpseudo-MOS/R-Factor). HTTP(S), Email (SMTP, IMAP, POP3), Radius, Database(Oracle and MySQL), FTP, DHCP, and BGP. JSON export (TCP, Kafka and ElasticSearch) 2017 - ntop.org11
nProbe [4/4] nProbesupports flexible NetFlow, that allows dataexport format to be customised at runtime. nProbe allow users to define a template on thecommand line. In addition to the standard fields (IP, port ), nProbecan export many other fields such as packet stats(TTL and size distribution), network/applicationlatency, geolocation, packets retransmitted/out-oforder, tunnel information, and DPI (Deep PacketInspection). 2017 - ntop.org12
nProbe vs YAFAbility to dissect application protocols (e.g.VoIP, Radius,HTTP) and export them in standard flow records (withntop PEN). Network (latency, retransmission, OOO, TCP Window/TTL stats ) and application metrics (response time). Application metadata (URL, BitTorrent Hash, SSLcertification information, OperatingSystem). Flow user info (Radius/GTP/Diameter) in-probe flowcorrelation. Probe scriptability via Lua to extend flow information orto manipulate flow export policy. 2017 - ntop.org13
DNS Traffic MonitoringUsing nProbe it is possible to generate flowrecords containing DNS req/reply and dumpthem to disk Malformed DNS packets can be saved in pcapformat for later analysis and troubleshooting. ## When[epoch]DNS Client[ascii:32]AS[uint]ClientCountry[ascii:32] ClientCity[ascii:32]DNS Server[ascii:32]Query[ascii:64] NumRetCode[uint]RetCode[ascii:16]NumAnswer[uint] t]#1481220355192.168.1.708.8.8.8 bramp.github.io 0NOERROR ME02551165.628 121481220356192.168.1.708.8.8.8 edition.cnn.com 0NOERROR 20357192.168.1.708.8.8.8 tpc.googlesyndication.com0NOERROR e.com/CNAME02551146.898 2991481220357192.168.1.708.8.8.8 pagead2.googlesyndication.com0NOERROR /CNAME02551149.280 2991481220357192.168.1.708.8.8.8 www.google.com 0NOERROR 11A33128216.58.205.196/A02551147.855 2921481220357192.168.1.708.8.8.8 aax.amazon-adsystem.com 0NOERROR 974 29 2017 - ntop.org14
VoIP Traffic Monitoring [1/3]nProbe can monitor both SIP and RTP, andcorrelate signalling to traffic via the redis cache. It can: Generate “legacy” CDR (Call Data Records) containingcaller/called/duration information on SIP traffic. Dissect RTP traffic and generate call quality metricsincluding jitter, packet lost, max inter-arrival time,pseudo-MOS (Mean Opinion Score) and R-Factor. A low-end x86 PC with nProbe can monitorthousand of simultaneous VoIP calls. 2017 - ntop.org15
VoIP Traffic Monitoring [2/3] 2017 - ntop.org16
VoIP Traffic Monitoring [3/3] 2017 - ntop.org17
BitTorrent Traffic Monitoring [1/3]BitTorrent traffic is encrypted and so its contentcannot be inspected. However sometimes it isnecessary to track what users are downloading tocheck if the what they are doing is legitimate. Fortunately BitTorrent traffic tracks resources bymeans of a hashId that it can be dissected bynProbe. Using a search engine it is possible to bind a hashIdto a downloaded resource and thus decide if thedownloaded files are legitimate. 2017 - ntop.org18
BitTorrent Traffic Monitoring [2/3] 2017 - ntop.org19
BitTorrent Traffic Monitoring [3/3] 2017 - ntop.org20
HTTP Traffic Analysis [1/2]The HTTP plugin is used by many users to bothanalyse user traffic and detect malware. For thisreason it produces a rich set of metrics andmetadata to make detection possible. As flow-based probes usually emit flows aftersome timeouts are past (e.g. flow duration), inorder to promptly trigger detection, nProbeplugins can emit events immediately (e.g. as soonas the HTTP headers have been observed). 2017 - ntop.org21
HTTP Traffic Analysis [2/2] The plugin supports both HTTP 1.0 and 1.1(multi-requests per TCP connection).## ethod[ascii:8] URL[ascii:255] i:4]ApplLatency(ms)[uint]ClientLatency(ms)[uint] ServerLatency(ms)[uint] nt2Server TEID[ascii:8]Server2Client TEID[ascii:8]FlowUserName (Useror scii:32]Pkts Cli2Srv[uint]Pkts Svr2Cli[uint]Bytes Cli2Srv[uint]Bytes Svr2Cli[uint]OOO Cli2Srv[uint]OOO Svr2Cli[uint]Retr Cli2Srv[uint]Retr Svr2Cli[uint]Duration Cli2Srv(ms)[uint]Duration :255]Via[ascii:255] pGET/blog/?feed rss2301www.ntop.org/blog/feed www.ntop.org/blog/?feed rss2SimplePie/1.4-dev (Feed Parser; http://simplepie.org; Allow like Gecko) /?feed rss2SimplePie/1.4-dev(Feed Parser; http://simplepie.org; Allow like Gecko) 1.101 version.ntop.orghttpGET/version.xml200ntop/5.0.2 host/x86 64-2.6.32-279.5.2.el6.x86 64-linux-gnu distro/centos release/6.3 kernrlse/2.6.32-279.14.1.el6.x86 64 GCC/4.4.6 config() run(u;W; a; F; d) gdbm/1.8.0 openssl/1.0.0-fips zlib/1.2.3 access/https interfaces(eth1)application/xml 1680.1415 2017 - ntop.org22
MySQL/Oracle Latency Analysis Same as for HTTP, DNS, DHCP, FTP nProbe isalso able to analyse applications protocols (whennot used with encryption) and extract relevantmetadata that is used to troubleshoot bothnetwork and coding issues.## Client[ascii:32]Server[ascii:32]User[ascii:32] i:32]Bytes[uint]BeginTimeepoch[EndTime] int] ELECT msisdn FROM big white list WHERE authid 4591403nodata 4.14110.96.4.28rootSELECT msisdn FROM big white list WHERE authid 1591403nodata 4.14110.96.4.28rootSELECT msisdn FROM big white list WHERE authid 7591403nodata 4.14110.96.4.28rootSELECT msisdn FROM big white list WHERE authid 5591403nodata found6527148131417714813141770.0010.0000.001 2017 - ntop.org23
Lua Scripting [1/2]During flow export, it is possible to instrumentnProbe to execute some actions using Luascripts. nProbe embeds a LuaJIT interpreter thatexecutes a function when a flow is exported. Network administrators can use scripts to Execute actions when specific flow values are observed(e.g. when a malware URL is reported). Selectively prevent flow export (e.g. unidirectionalflows) 2017 - ntop.org24
Lua Scripting [2/2]function checkFlow(label, flow, rule)for i 1,#rule dolocal id rule[i].idlocal filter rule[i].filterlocal rsp true -- be optimisticif(debug true) then io.write("Checking rule ".id."\n") endfor j 1,#filter do .endif(debug true) then io.write("Result for rule ".id." ".tostring(rsp)."\n") endif(rsp true) thenexecMatchCommand(label, flow, id, r)endendend 2017 - ntop.org25
Final RemarksnProbe is a modern network probe: Flow generation and collection, 10 Gbit capable. Able to produce “rich” monitoring metrics (augmentedflows). Extensible by means of plugins (SDK available). Scriptable using Lua. Available for Linux, Windows and embedded systems. BigData friendly (JSON export, Syslog, Kafka, ELK). Free for no-profit, research, and education. 2017 - ntop.org26
fostered by vendors: NetFlow vs sFlow vs SNMP, Cisco vs Juniper This has plagued the market by creating tools more vendor oriented, than result oriented. Fortunately recent advances in computing and in particular the b
