Transcription
ISP Security – Real WorldTechniques IIThe Threat from Violated CPE RoutersNANOG26, Eugene, ORVersion 1.1Kevin Houle [kjh@cert.org]Barry Raveendran Greene [bgreene@cisco.com]
Where to get Slides and cure.htmlISP Essentials Archivenhttp://www.ispbook.com/security/NANOG 26 – ISP Security
ISP Security Treats“The wonderful thing about the Internet isthat you’re connected to everyone else.The terrible thing about the Internet isthat you’re connected to everyone else.”Vint CerfNANOG 26 – ISP Security
Role of Service Providers Deliver service in the face ofmistakes, failures, and attacksData PlaneOther ISPsCustomerControl PlaneProtect ISP infrastructure from customers and the Internet.Protect the data plane and control plan from each other.NANOG 26 – ISP Security
Role of Service Providers Help protect other peersOther ISPsCustomerData PlaneControl PlaneCustomers are unwitting victimsNANOG 26 – ISP Security
Role of Service Providers Protect customers from attacks comingfrom the infrastructure or other customersOther ISPsCustomerData PlaneControl PlaneCustomers are targetsNANOG 26 – ISP Security
Focus of the TutorialnOur focus is on the ISP – Customer Edgewith specific focus on the Customer’s CPE.nWhy? Cause CPE configs are something thatthe ISP could feasible tworkISPBackboneUpstream ISPNANOG 26 – ISP Security
ISP Security – Real WorldTechniques IInnnnnIntruders Compromising Customer CPEsMalicious Configuration AlterationMalicious Route InjectionAlteration of Registry InformationDenial-Of-Service Attacks Directed at CPEsNANOG 26 – ISP Security
Intruders Compromising CPE
Intruders Compromising CPEPerceived Threat & RealitynPerceived Threat:nnSomeone gaining control of a customer’s CPE can dosome nasty things.Reality:nIntruders are actively scanning for and compromisingCPE devicesnnnBroadband devicesCustomer premise routersAutomated tools exist for scanning, compromise, anduse of compromised devicesNANOG 26 – ISP Security
Intruders Compromising CPERealitynnnIntruder-developed texts exist to teachothersLists of compromised CPE are traded in theundergroundCERT/CC aware of incidents involvingthousands of impacted devicesNANOG 26 – ISP Security
Intruders Compromising CPEAttack MethodsnnFingerprint scanning and traceroute toidentify targetsTargets compromisednnnDefault passwords (most common)Weak and well-known passwordsStolen authentication credentialsnnnSniffing network trafficSocial engineeringInsider attackNANOG 26 – ISP Security
Intruders Compromising CPEImpactnnnnSometimes no impact – just for fun.Intruder Proxy / Bounce Point / GRE TunnelPointDenial-of-service for customer(s)Attacks against other sitesnnDDoS via automated toolsTrust-based attacksNANOG 26 – ISP Security
Intruders Compromising CPEWhat can ISPs Do?nAssume the Worse!nnnAlways assume the customer’s CPE is notsecure. So take measures to protect yournetwork.BCP38 – Ingress Source filtering. Severaltechniques today (ACLs, uRPF Strict Mode,Radius Per-User ACLs, Cable source-verify).BGP Ingress Route Filtering – if customer is aBGP speaking Router.NANOG 26 – ISP Security
Intruders Compromising CPEWhat can ISPs Do?nProvide your customer the tools to take careof themselves.nn#1 – Customer Service Web Page on Security.Links with procedures, vendor security pages,recommendations, and other BCPs.Customer “security” alias – allow customers tosign on and get news and alerts.NANOG 26 – ISP Security
Intruders Compromising CPEWhat can ISPs Do?nPolicies, Preparation, and Practice!nCreate and Publish your security Policies.nnCreating policies on the fly in the middle of a securityincident is not advisable.Prepare your Identification, Classification,Traceback, and Reaction Tools.nnnClassification ACLsSink HolesBackscatter Traceback – works for customeraggregation routers as well as ISP – ISP peeringpoints.NANOG 26 – ISP Security
What can ISPs Do?Monitoring Scan Rates & WormsPlace various /32Infrastructureaddresses hereTo ISP BackboneSink HoleGatewayTarget RouterTo ISPBackboneSniffers andAnalyzersTo ISP BackbonennnSelect /32 address from different block of your addressspace. Advertise them out the Sink HoleAssign them to a workstation built to monitor and logscans.Find or create a Dark IP Application that automaticallymonitor scan rates and worms . Providing list ofviolated customers.NANOG 26 – ISP Security
What can ISPs Do?Monitoring Scan Rates & WormsOperator instantlynotified of Worminfection.System automaticallygenerates a list ofinfected hosts forquarantine andclean-up.NANOG 26 – ISP Security
Intruders Compromising CPEWhat can ISPs Do?nMonitor Customer BandwidthnnNeed to do it for traffic engineering.Important for detecting attacks.NANOG 26 – ISP Security
Intruders Compromising CPEWhat can ISPs Do?nUse Strong Authentication for CPEManagementnnnnPublic key cryptography (e.g., ssh)Good password policies (change defaults!)Do not authenticate in the clear across untrustednetworks.Critical for managed CPE services. How manyrouters do you you really have control?NANOG 26 – ISP Security
Intruders Compromising CPEWhat are ISPs Doing?nNot much! Based on the observational evidence,ISPs are not doing much.nExample from Barry’s home.nnnnnnTwo DSL links and one Cable link.Barry has control over the CPEs for each of the three providers.Two provided “security best practices web pages”.All three allow spoofed source addresses (I can create niceasymmetrical flows going out one and back in the other).No messages from any of the three providers about softwareupdates or security alerts (i.e. remember the SNMP fun).One of the three types of CPEs provide an easy way toshutdown external access to service ports.NANOG 26 – ISP Security
Malicious Configuration Alteration
Malicious Configuration AlterationPerceive Threat & RealitynWhat fun can you have once you havebroken into a router?nnIntruders continue to develop and sharetechniques for altering router configurationsonce compromisedHOWTO texts are publicly available for multipleplatformsNANOG 26 – ISP Security
Malicious Configuration AlterationAttack MethodsnnDirect privileged access into the ISP orcustomer’s network via compromised routerUnprotected remote management interfacesnnHTTPSNMPnSame community string used everywhereNANOG 26 – ISP Security
Malicious Configuration AlterationImpactnAdministrative lockoutnnAlteration of security policiesnnnnIntruder changes access/privilege passwordsRemoval/alteration of ACLsEnabling/disabling servicesBroadens exposure to further attacksThe CPE turns into a bridge into thecustomer’s internal trust domains (andpossibly the ISP’s)NANOG 26 – ISP Security
Malicious Configuration AlterationImpactnTrash the CPEnnAlteration of layer 2 configurationnn“Write Erase” or delete the software imageInterfaces may be disabled causing outages (shutdown).Alteration of layer 3 configurationnnnnRouting protocols and policiesDenial-of-serviceTraffic redirection / interception (Cisco Sniffer)Prefix hijackingNANOG 26 – ISP Security
Malicious Configuration AlterationWhat can Customers and ISPs Do?nProtect Routers from CompromisennnnDisable unneeded servicesRestrict traffic to needed servicesMonitor traffic with src/dst routersUse strong authentication for managementnnnAt least use non-default passwords!Out-of-band management pathAuthenticate and backup configurationsNANOG 26 – ISP Security
Malicious Route Injection
Malicious Route InjectionPerceive ThreatnnnnBad Routing Information does leak out. This hasbeen from mistakes, failures, bugs, and intentional.Intruders are beginning to understand thatprivileged access to a router means route tablescan be alteredCERT/CC is aware of a small number of incidentsinvolving malicious use of routing informationPerceived Threat is that this will be a growth areafor attackers.NANOG 26 – ISP Security
Malicious Route InjectionReality – an ExamplennnAS 7007 incident used as an attack.Multihomed CPE router is violated and usedto “de-aggregate” large blocks of theInternet.Evidence collected by several CERTs thathundreds of CPEs are violated.NANOG 26 – ISP Security
Malicious Route InjectionReality – an ExampleI accept the entireInternet with /24 morespecifics and sentthem on.XAS 500EAS 400DLets advertise theentire Internetwith /24 morespecificsAS XYZNAS 300CAS 100ABAS 200I accept the entire Internet with /24more specifics and sent them on.NANOG 26 – ISP Security
Malicious Route InjectionReality – an ExampleDURESSXThe restof theInternetUnstableEAS 500DLets advertisethe entireInternet with /24more specificsAS XYZNDURESSAS 300CAAS 100DURESSNANOG 26 – ISP SecurityBThe rest oftheInternetUnstable
Malicious Route InjectionReality – an ExamplennnnGarbage in – Garbageout does happen on theNetAS 7007 Incident (1997)was the most visible caseof this problem.Key damage are to thoseISPs who pass on thegarbage.Disruption, Duress, andInstability has been anInternet wide effect ofGarbage in – Garbageout.DURESSXThe restUnstableof theEAS 500InternetDDURESSAS300Lets advertisethe entireInternet with /24more specificsCNAS XYZNANOG 26 – ISP SecurityAAS 100DURESSBThe restof theUnstableInternet
Malicious Route InjectionAttack MethodsnnnnnnGood News – Risk is mainly to BGP speakingRouters.Bad News – Multihomed BGP Speaking customersare increasing!Really Bad News – Many of these routers have nopasswords!Local layer 3 configuration alteration oncompromised routerIntra-AS propagation of bad routing informationInter-AS propagation of bad routing informationNANOG 26 – ISP Security
Malicious Route InjectionImpactnnnnDenial-Of-Service to Customer(s), ISP(s),and the Internet.Traffic Redirection / InterceptionPrefix HijackingAS HijackingNANOG 26 – ISP Security
Malicious Route InjectionWhat can ISPs Do?nnnCustomer Ingress Prefix Filtering!ISPs should only accept customer prefixeswhich have been assigned or allocated totheir downstream customers.For examplennnDownstream customer has 220.50.0.0/20 block.Customer should only announce this to peers.Upstream peers should only accept this prefix.NANOG 26 – ISP Security
Malicious Route InjectionWhat can ISPs Do?nCisco Configuration Example on Upstreamrouter bgp 100neighbor 222.222.10.1 remote-as 101neighbor 222.222.10.1 prefix-list customer in!ip prefix-list customer permit 220.50.0.0/20ip prefix-list customer deny 0.0.0.0/0 le iseNetworkPrefixesNANOG 26 – ISP SecurityISPBackboneUpstream ISP
Malicious Route InjectionWhat can ISPs Do?nContainment Filters!nnnnDesign your network with the principles of ofsurvivability.Murphy’s Law of Networking implies that thecustomer ingress prefix filter will fail.Remember 70% to 80% of ISP problems aremaintenance injected trouble (MIT).Place Egress Prefix Filters on the Network tocontain prefix leaks.NANOG 26 – ISP Security
What can ISPs Do?Containment Egress Prefix FiltersnnnCould place themon thePOP/RegionalInterconnects.Could place themon the border tothe core.Should placethem on the ISPpeering links.ISP #1ISP #2ISP #3CORENANOG 26 – ISP Security
What can ISPs Do?Containment Egress Prefix FiltersnnnIt is not rocket science!Just create a hard list of your RIR allocatedprefixes.Cisco Configuration Examplerouter bgp 100network 221.10.0.0 mask 255.255.224.0neighbor 222.222.10.1 remote-as 101neighbor 222.222.10.1 prefix-list out-filter out!ip route 221.10.0.0 255.255.224.0 null0!ip prefix-list out-filter permit 221.10.0.0/19ip prefix-list out-filter deny 0.0.0.0/0 le 32NANOG 26 – ISP Security
What can ISPs Do?Containment Egress Prefix FiltersnnWhat about all my multihomed customerswith prefixes from other ISPs?Add them to the customer ingress prefixfilter.nnYou should know what you will accept.Add them to the master egress prefix-filter.nnYou should know what your advertising toeveryone else.Bigness is not an excuse.NANOG 26 – ISP Security
Malicious Route InjectionWhat can ISPs Do?nnnCustomer Ingress Prefix FilteringPrefix filtering between intra-AS trust zonesRoute table monitoring to detect alterationof critical route pathsNANOG 26 – ISP Security
Alteration of Registry Information
Alteration of Registry InformationPerceived Threat & RealitynnMalicious People can change the RIR informationfor a target.RealitynIP and domain registries historically have not providedstrong authentication for client transactions.nnnMAIL-FROMEven when strong authentication is available at the RIR,it is commonly not used.RIRs are commonly referenced to determine ownershipof IP/domain assets.NANOG 26 – ISP Security
Alteration of Registry InformationRealitynnnRegistry transactions are often the key to alteringDNS delegations for IN-ADDR.ARPA and domainnamespace.CERT/CC is aware of numerous incidents based onthe attacker modifying registry informationhttp://www.cert.org/vul notes/VN-99-01.htmlNANOG 26 – ISP Security
Alteration of Registry InformationAttack MethodsnSocial engineeringnnSomeone calls the NOC to change their routingpolicy. How do you know the person is anauthorized to make the change?Defeating weak authentication methodsnMAIL-FROMNANOG 26 – ISP Security
Alteration of Registry InformationImpactnAlteration of DNS glue records in top-level zonesnnDenial-of-serviceAlteration of delegated nameserversnnnnDenial-of-serviceTraffic redirection via malicious RR’sBypass of DNS-based access controlsAlteration of information recorded by DNS-based loggingmechanismsNANOG 26 – ISP Security
Alteration of Registry InformationImpactnAlteration of contact informationnnnIncludes domains, netblocks, and AS numbersEnables social engineering attacksCERT/CC is aware of this technique beingused to social engineer an ISP into routing ahijacked /8 prefix using a hijacked ASnumberNANOG 26 – ISP Security
Alteration of Registry InformationWhat can ISPs and Customers Do?nnnDemand and use strong transactionauthentication methods to protect registryobjects from malicious changesVerify critical registry records on a regularbasisRequest read-only ‘freeze’ for critical recordsNANOG 26 – ISP Security
Denial-of-service AttacksDirected at Customer’s CPERouters
DOS the CPEPerceived ThreatnnIntruders understand that packet floodingattacks directed at routers can have broaderimpact than attacks directed at hostsThe IP stack code path may be moreexpensive for packets directed at a routervs. packets transiting a routerNANOG 26 – ISP Security
DOS the CPEBig Sites Before Feb’00Co-loISP-2Attack theWeb ServersGigabitFast EthernetBackup LinksNANOG 26 – ISP Security
DOS the CPEBig Sites after Feb’00Co-loISP-2Attacks shift toWeb Site’ssupportingNetworkInfrastructureHyper EngineeredWeb ServerCapacity Ridesout DOS/DDOSAttacksGigabitFast EthernetBackup LinksNANOG 26 – ISP Security
DOS the CPEReality – Attacks hit the ISPPeer AIXP-WPeer BIXP-EUpstream AUpstream AUpstream BUpstream BTargetDDoS againstthe ISP to takeout the target.POPNANOG 26 – ISP Security
DOS the CPEReality – Miscreant Wars!JP Biz Protection 2002 2002, Cisco Systems, Inc. All rights reserved.NANOG26 – ISP Security
DOS the CPEAttack MethodsnTraceroute to discover attack targetnnnLooking for something just upstream fromintended victimSome attacks target each router in the pathPacket rate attacksnnResource consumption attack against the routerThe ol’ stack code path issue may applynRouter service ports (e.g., BGP, telnet, ssh, etc)NANOG 26 – ISP Security
DOS the CPEAttack MethodsnPacket size attacksnnBandwidth denial-of-service against layer 1Resource consumption if router has morebandwidth attached at layer 1 than it’sresources can handleNANOG 26 – ISP Security
DOS the CPEImpactnDenial-of-serviceNANOG 26 – ISP Security
DOS the CPEWhat is Co-Lateral Damage?nnnnCo-Lateral Damage hurts others around thetarget of attack.Some attackers work very hard to minimize colateral damage (cruse missile strike).Others do not care (use a tank to swat amosquito).Co-Lateral Damage is core reason why ISPs mustrespond to their customer’s DOS attacks.NANOG 26 – ISP Security
DOS the CPEWhat is Co-Lateral Damage?nnInternetIPetckaPIt is all about the packet .Once a packet gets into theInternet, someone,somewhere has to do one oftwo things:Deliver the PacketnDrop the PacketnnIn the context of a DOSattack, the question is whoand where will that drop thatpacket.NANOG 26 – ISP Security
DOS the CPEWhat is Co-Lateral Damage?nnnSingle HomedCustomer’s CircuitSaturates from a DOSAttack.Which router has thestatic route?Which router has theaggregate route?NANOG 26 – ISP Security
DOS the CPEDOS Funnel and Collateral DamageIXP-WAPeer APeer rsGF POPNANOG 26 – ISP SecurityNOCUpstreamB
DOS the CPERisk Increases with DensityPOP BorderPOP BorderPOP BorderOC48OC12121534BigAggregationBoxPOP BorderOC12BigAggregationBox145678910111213Lots of Aggregations Routerswith 10s to 100s of customersper router.NineChOC12Few Aggregations Routerswith 100s to 1000s ofcustomers per router.It is all about # of Customers per RUNANOG 26 – ISP Security
DOS the CPEDOS FlappingprimaryCAAS 109HackernEAS 65534BDbackupMultihomed Customer’s Primary Link get saturated?Link saturation causes BGP to dropnBGP drop on the primary means that the back-up is usednWho drops the packets during convergence?nBack-up path saturates, dropping BGP, then what? Back toprimary?nNANOG 26 – ISP Security
DOS the CPEDOS FlappingHackerAS110AAS 109AS 108CDAS 107AnBMultihomed Customer to two ISPs gets hit.nLine saturates, BGP drops, attack shifts OR attackaggregates!NANOG 26 – ISP Security
DOS the CPECo-Lateral Damage is RealnnnnnCo-Lateral Damage is Real. If you have not yetexperienced it, you will.How you architect your network, your routing, and yourprovisioning effects the extent of co-lateral damage.All those “VPN Tunneling Solutions” are just asvulnerable to co-lateral damage.What tools and techniques you prepare affects how youcan mitigate the effects of co-lateral damage.Do nothing and you may find that a simple DOS attacksagainst one customer turns into a network nightmare.NANOG 26 – ISP Security
DOS the CPEWhat can ISPs Do?nnPolicies, Preparation, and Practice!Prepare your Identification, Classification,Traceback, and Reaction Tools.nnnClassification ACLsSink HolesBackscatter Traceback – works for customeraggregation routers as well as ISP – ISP peeringpoints.NANOG 26 – ISP Security
What can ISPs Do?Remote-Triggered Black HolennnnWe use BGP to trigger a network wide response toan attack flow.Push the packet drop to the edge of the network.A simple static route and BGP will allow an ISP totrigger network wide black holes as fast as iBGPcan update the network.This provides ISPs a tool that can be used torespond to security related events or used forDOS/DDOS Backscatter Tracebacks.NANOG 26 – ISP Security
What can ISPs Do?Remote-Triggered Black HoleIXP-WAPeer APeer BIXP-EUpstreamAUpstreamABDCUpstreamBETargetF POPGTarget is takenoutNANOG 26 – ISP SecurityNOCUpstreamB
What can ISPs Do?Remote-Triggered Black HoleIXP-WAPeer APeer rsF POPAttack causesCo-LateralNANOG Damage26 – ISP SecurityGNOCUpstreamB
What can ISPs Do?Remote-Triggered Black HoleIXP-WAPeer APeer BIXP-EUpstreamAUpstreamABDCUpstreamBETargetGF POPNANOG 26 – ISP SecurityNOCUpstreamBiBGPAdvertisesList ofBlack HoledPrefixes
What can ISPs Do?Remote-Triggered Black HolennnRemote Triggered Black Hole filtering is thefoundation for a whole series of techniquesto traceback and react to DOS/DDOS attackson an ISP’s network.Preparation does not effect ISP operations orperformance.It does adds the option, providing a valuableISP’s security toolkit.NANOG 26 – ISP Security
What can ISPs Do?Sink Hole Routers/NetworksnSink Holes are versatile security tools.nnnBGP speaking Router or Workstation that built tosuck in attacks.Used to redirect attacks away from the customer– working the attack on a router built towithstand the attack.Used to monitor attack noise, scans, and otheractivity (via the advertisement of default)NANOG 26 – ISP Security
What can ISPs Do?Sink Hole Routers/NetworksSink Hole NetworkTarget ofAttack172.168.20.0/24 – target’s network172.168.20.1 is attackedNANOG 26 – ISP Security
What can ISPs Do?Sink Hole Routers/NetworksRouter advertises172.168.20.1/32Sink Hole NetworkTarget ofAttack172.168.20.0/24 – target’s network172.168.20.1 is attackedNANOG 26 – ISP Security
What can ISPs Do?Sink Hole Routers/NetworksnnnAttack is pulled offcustomer and youraggregation router.Can now safely runclassification ACLs, FlowAnalysis, Sniffer Capture,Traceback, etc.Objective is to minimizethe risk to the networkwhile working the attackincident.NANOG 26 – ISP SecurityRouter advertises172.168.20.1/32Sink Hole NetworkTarget ofAttack172.168.20.0/24 – target’s network172.168.20.1 is attacked
DOS the CPEWhat can Customers Do?nnAssume that one day you will be attacked.Prepare!nnHave the security contacts for each of yourupstream ISPs.Be prepared to switch IP addresses for criticalservices that are under attack.nnMove the service under attack to a new /32 as theoriginal /32 is black holed by the ISP.Be on all your vendor’s security vulnerabilitymailing list and PATH your software.NANOG 26 – ISP Security
Summary
SummarynnnIntruders are actively pursuing attacks againstrouting infrastructureISPs have a vested interest to do something toprotect themselves from violated CPEs.Next StepsnnNANOG Security BOF – Monday NightNsp-security Forum. Peers in the NSP/ISP Operationscommunity actively working together to combat psecurityNANOG 26 – ISP Security
Q&A
More Information
More InformationnDenial of Service information pagennIOS Essentials—Features every ISP should book.comRFC 2827 “Network Ingress Filtering: Defeating Denial of ServiceAttacks which Employ IP Source Address nDistributed systems intruder tools workshop reportnnCERT advisoriesnnhttp://www.cert.org/reports/dsit irst.org/NANOG 26 – ISP Security
More Information “Tackling Network DoS on Transit Networks”: David Harmelin,DANTE, March 2001 (describes a detection method based l] “Inferring Internet Denial-of-Service Activity”: David Moore et al,May 2001; (described a new method to detect DoS attacks,based on the return traffic from the victims, analysed on a /8network; very interesting scatter/index.xml] “The spread of the code red worm”: David Moore, CAIDA, July2001 (using the above to detect how this worm spread acrossthe Internet) NANOG 26 – ISP Security
More InformationDoS Tracing: “Tracing Spoofed IP Addresses”: Rob Thomas, Feb 2001;(good technical description of using netflow to trace back aflow)[http://www.enteract.com/ robt/Docs/Articles/trackingspoofed.html]Honeypots and Honeynets: Honeypots: Tracking Hackershttp://www.tracking-hackers.com/IETF RFCs: RFC 2179 Network Security For Trade Shows. A. Gwinn.July 1997. (Format: TXT 20690NANOG 26 – ISP Security bytes)
More Information“DoS attacks against GRC.com”: Steve Gibson, GRC, June 2001 (areal life description of attacks from the victim side; somewhatdisputed, but fun to read!)http://grc.com/dos/grcdos.htmImproving Security on Cisco lIncreasing Security on IP cie/ndcs798/nd2016.htmPaper from S. Fluhrer (Cisco Systems), I. Mantin and A. Shamir(Weizmann Institute)www.crypto.com/papers/others/rc4 ksaproc.psOther security toolswww.insecure.org/tools.htmlNANOG 26 – ISP Security
More InformationnCAIDA paper “Inferring Internet G 26 – ISP Security
n Design your network with the principles of of survivability. n Murphy's Law of Networking implies that the customer ingress prefix filter will fail. n Remember 70% to 80% of ISP problems are maintenance injected trouble (MIT). n Place Egress Prefix Filters on the Network to contain prefix leaks.
