WIXLIB

ESnet WAN Security UpdatesScott CampbellESnet Network Security2018 Technology ExchangeOctober 16, 2018

Intro to ESnet and TeamOur MissionESnet provides the high-bandwidth, reliable connections that link scientists at national laboratories, universities, and other researchinstitutions, enabling them to collaborate on some of the world's most important scientific challenges including energy, climate science, andthe origins of the universe. Funded by the DOE Office of Science, ESnet is managed and operated by the Scientific Networking Division atLawrence Berkeley National Laboratory. As a nationwide infrastructure and DOE User Facility, ESnet provides scientists with access to uniqueDOE research facilities and computing resources.ESnet has a strong ethos of leadership in networking innovation. Our team is constantly improving the services to create a more versatile androbust network to serve the emerging needs of scientific researchers. We work closely with the international technical community to developopen-source software and collaborative technical projects. Our programs, as well as our collaborations with other research and educationnetworks around the world, demonstrate our commitment to push network research forward so we will be able to anticipate and flawlesslyserve the needs of DOE scientists.As computational tools such as simulation and visualization become more fine-tuned and sophisticated, they also require the processing andmanagement of greater amounts of data. The sheer volume of data generated by international scientific collaborations and the increasinglydistributed nature of large data flows, as well as the development of real time cloud computing, is creating a staggering volume of networktraffic. Since 1990, ESnet's average traffic has grown by a factor of 10 every 47 months. This growth trend is only accelerating as internationalscientific collaborations such as the Large Hadron Collider (LHC), Coupled Model Intercomparison Project (CMIP), and InternationalThermonuclear Experimental Reactor (ITER), generate and exchange massive amounts of data. ESnet provides a scalable and efficientnetwork infrastructure that enables scientists to optimize their resources and conduct ambitious, world-class research.ESnet is supervised by the Office of Science, which oversees 10 national laboratories that carry out the missions of its science programs, aswell as underwrites research and development projects conducted at additional laboratories overseen by other DOE offices.

Agenda Introduction The larger problem Our Approach Security AssistBro on the WANCEASE(D)DOS Futures ESnet6

Statement of ProblemPrologue: The foreboding bit in the beginning of a movie thatprovides enough backstory for the plot to make sense .It has come to our realization that our ability to observe andrespond to attacks in/on our control plane network was limited.As well, defining what exactly constitutes normal in a long termperspective for our sites and peering points was not wellorganized. This is not about getting more data, just having theinfrastructure in place to digest this data and provide an ideaabout what has changed off norm.

Our ApproachFor modern scientific research at scale, we see the Network asan indispensable tool/instrument for critical data andcommunications which means that it needs to be secure,performant and reliable. Most of this work is experimental: iterative design changesas we use what we learn from stage 1 to stage 2 and beyond. Listen to the data - informs project direction. Be prepared to junk mental models. Complexity in design is bad. Software discipline is good - testing and automation is asmall investment in sanity and longevity.

Agenda Introduction The larger problem Our Approach Security AssistBro on the WANCEASE(D)DOS Futures ESnet6

Security Assist Security-Assist will be availableto anyone connected to ESnet Designed to augment, notreplace, Site security It is most important that Sitesremain in full control of theirSite security

Security Assist : PurposeThe Security Assist project is an effort to take advantage of the the unique perspectiveprovided by ESnet as a service provider to: Provide tools to remediate (upstream) trafficLeverage our perspective to give information otherwise too costly or complexto obtainAssist smaller sites constrained by personnel or budget issues in a finite welldefined wayThe purpose expressly is not to interfere with local sites security policy, issues orpractices.The tools and data we are providing/will provide is evolving as we better understandthe technology, ecosystem and needs of both the end sites and the underlyingnetwork.

Security Assist : BGP Monitoring and Alerting(BGPmon)XBGP Community triggered BHRWAN SME AnalysisXXSite Black Hole Routing (BHR)XExternal Vulnerability ScannerXInfrastructure Filtering (againstESnet)Upstream Peer FilteringAutoXXShare Custom Bro PoliciesXFiltering (5-tuple)XFull Packet Capture On-DemandX

Security Assist : TimelineServicePhase 1BGP Monitoring and Alerting (BGPmon)BGP Community triggered BHRWAN SME AnalysisSite Black Hole Routing (BHR)External Vulnerability ScannerPhase 2Infrastructure Filtering (against ESnet)Upstream Peer FilteringShare Custom Bro PoliciesPhase 3Filtering (5-tuple)Full Packet Capture On-Demand

Agenda Introduction The larger problem Our Approach Security AssistBro on the WANCEASE(D)DOS Futures ESnet6

BackgroundESnet has loads of metrics, logs, and sampled netflow data across the entireWAN.Goals: Increase visibility into the traffic traversing our WAN and our controlplane via deep packet inspection Be better prepared to assist Site security teams by understanding thekinds of attacks they see.Implementation: Due to space and budget limitations, it's unrealistic to install amulti-100G Bro cluster at several WAN locations. Be particular about filtering traffic to Bro that has the highest likelihoodof being interesting

Phase I: Initial Data CollectionUse spanning port to forward traffic destined to thestub network on each of the routers used forcollection.Logs toSplunkESnetHumansBOWACL/Port MirrorR2R1StubNetworkPeeringTraffic

Single Instance Architecture

Current Pilot Deployment**Plus LBL Datacenter

Multiple Instance Architecture

Potential Future Architecture

Early Success & Lessons Learned Success– Control plane traffic really is generally as boring as it should be Anomalies should be easy to spot– DoS reconnaissance traffic– Configuration mistakes Including connection attempts to hosts that no longer exist. Lessons Learned– ALU/Nokia limitations Expansion plans– PNWG (Seattle), LOND, AMST

Agenda Introduction The larger problem Our Approach Security AssistBro on the WANCEASE(D)DOS Futures ESnet6

Agenda Introduction The larger problem Our Approach Security AssistBro on the WANCEASE(D)DOS Futures ESnet6BackgroundGoalsMethodArchitectureResults

CEASEESnet has limited ability to mitigate malicious traffic on the WAN and itrequires manual investigation and input of ACLs on the routers. CEASE(Correlation Evaluation and Security Enforcement) is a framework thatencompases both data analytics as well as infrastructure designed to mitigateattacks. The goal with CEASE is to make automated decisions on what ismalicious and, as a later stage, stop that traffic as it comes in from ourpeering points.Implementation will be in a series of small steps to avoid the unrelentinghilarity that comes from automating defensive measures.

CEASE : GoalsCEASE v.1 is designed to prototype passive monitoring and detection analysis.Focus on identifying network characteristics that are stable enough to be useful,systematically measuring and modeling these characteristics to identify unusualbehavior.This is expressed as a set of goals:G1: Identify attacks on network control plane focusing on peering pointtraffic.G2: Identify typical characteristics of network data volume/characteristicsfor a small number of known sites.G3: Develop simple baselines for traffic characteristics at instrumentedpeering points.G4: Topology/BGP Monitoring and Alerting.

CEASE : Goals, Organic GrowthG1: Identify attacks on network control plane focusing on peering pointtraffic.G1.1 BOW for DPI analysis of desired dataG2: Identify typical characteristics of network data volume/characteristicsfor a small number of known sites.G2.1 Site SNMP producer/consumer ratioG2.2 Site Netflow protocol ratiosG2.3 Site Netflow Port DistributionsG2.4 Site Netflow Data DistributionsG3: Develop simple baselines for traffic characteristics at instrumentedpeering points.G3.1 Peering Netflow protocol ratiosG4: Topology/BGP Monitoring and Alerting.

CEASE : Goals, Organic Growth Pruning GoalsG1: Identify attacks on network control plane focusing on peering pointtraffic.G1.1 BOW for DPI analysis of desired data BOW get its own section aswell our netflow is 1:1000 sampled and control plane deserves better .G2: Identify typical characteristics of network data volume/characteristicsfor a small number of known sites.G2.1 Site SNMP producer/consumer ratio (redundant data, theory worksbut parallel infrastructure not good use of time)G2.2 Site Netflow protocol ratiosG2.3 Site Netflow Port DistributionsG2.4 Site Netflow Data DistributionsG3: Develop simple baselines for traffic characteristics at instrumentedpeering points.G3.1 Peering Netflow protocol ratiosG4: Topology/BGP Monitoring and Alerting not completely done - see notesat end

CEASE : End Result GoalsIdentify typical characteristics of network data volume/characteristics for asmall number of known sites. G1 Site Netflow producer/consumer ratio testG2 Site Netflow protocol ratiosG3 Site Netflow Port DistributionsG4 Site Netflow Data DistributionsDevelop simple baselines for traffic characteristics at instrumented peeringpoints. G5 Peering Netflow protocol ratios

CEASE Background: Multiple SourcesFeederRoutersSite

Data Flow OverviewStart with one or more raw nfcapd files: test dataConvert to minimum data fields - raw time seriesSummarize into 1 sec data units - discrete time seriesStore data units in DatabaseWhen all routers present, summarize and look up model data setsBuild modelRun test

Data Flow Overview: Raw FilesStart with one or more raw nfcapd files: test dataEachroutertohasa nfcapddatafile per5 mintimetimewindow.Each site has one or moreConvertminimumfields- rawseriesso wait for all of them to get dumped :Summarize into 1 sec data units - discrete time series-rw-r--r-- 1 user group 3085294 Sep 20 00:05 sunn-cr5/2018/09/20/nfcapd.201809200000-rw-r--r-- 1 user group 1999656 Sep 20 00:05 sacr-cr5/2018/09/20/nfcapd.201809200000-rw-r--r-- 1 user group 14753 Sep 20 00:05 nersc-mr2//2018/09/20/nfcapd.201809200000Store data units in DatabaseWhen all present, summarize and look up model data setsBuild modelRun test

Data Flow Overview: Extract Desired FieldsStart with one or more raw nfcapd files: test dataConvert to minimum data fields - raw time seriesnfdump -r file -B -q -o "fmt:%ts,%pr" "as 2936"Summarize into 1 sec data units - discrete time series2018-09-19 23:59:52.880,TCP2018-09-19 23:59:52.900,TCPStore dataunits in23:59:53.990,TCPDatabase2018-09-192018-09-19 23:59:54.650,TCP2018-09-19 23:59:54.870,UDPWhen allpresent, Lookup appropriate2018-09-1923:59:54.920,TCP2018-09-19 23:59:54.920,TCP2018-09-19 23:59:54.920,UDPBuild model2018-09-19 23:59:54.990,TCP2018-09-19 23:59:55.910,TCP2018-09-19 23:59:55.980,ICMPRun test2018-09-19 23:59:55.990,UDP CEASE SCRATCH/ RAND.datamodel data sets

Data Flow OverviewStart with one or more raw nfcapd files: test dataConvert to minimum data fields - raw time seriesSummarize into 1 sec data units - discrete time seriesStore data units in DatabaseWhen all present, summarize and look up model data setsBuild modelRun test