Transcription
ISP & IXP Design30thPhilip SmithMENOG 11AmmanSeptember – 9th October 20121
ISP & IXP Network DesignPoP Topologies and Designp Backbone Designp Upstream Connectivity & Peeringp Addressingp Routing Protocolsp Out of Band Managementp Operational Considerationsp Internet Exchange Pointsp 2
Point of PresenceTopologies3
PoP TopologiesCore routers – high speed trunkconnectionsp Distribution routers and Access routers –high port densityp Border routers – connections to otherprovidersp Service routers – hosting and serversp Some functions might be handled by asingle routerp 4
PoP DesignModular Designp Aggregation Services separated accordingtop n n n n connection speedcustomer servicecontention ratiosecurity considerations5
Modular PoP DesignOther ISPsISP Services(DNS, Mail, News,FTP, WWW)Web CacheHosted Services &DatacentreBackbone linkto another PoPBackbone linkto another PoPNetworkCoreConsumer cable,xDSL andwireless AccessConsumerDial AccessBusiness customeraggregation layerMetroE customeraggregation layerNetworkOperationsCentreChannelised circuitsfor leased line circuit deliveryGigE fibre trunksfor MetroE circuit delivery6
Modular Routing Protocol Designp Modular IGP implementationn n n p IGP “area” per PoPCore routers in backbone area (Area 0/L2)Aggregation/summarisation where possibleinto the coreModular iBGP implementationn n n BGP route reflector clusterCore routers are the route-reflectorsRemaining routers are clients & peer withroute-reflectors only7
Point of Presence Design8
PoP Modulesp Low Speed customer connectionsn n n p PSTN/ISDN dialupLow bandwidth needsLow revenue, large numbersLeased line customer connectionsn n n n E1/T1 speed rangeDelivery over channelised mediaMedium bandwidth needsMedium revenue, medium numbers9
PoP Modulesp Broad Band customer connectionsn n n p xDSL, Cable and WirelessHigh bandwidth needsLow revenue, large numbersMetroE & Highband customer connectionsn n n n Trunk onto GigE or 10GigE of 10Mbps andhigherChannelised OC3/12 delivery of E3/T3 andhigherHigh bandwidth needsHigh revenue, low numbers10
PoP Modulesp PoP Coren n n n p Two dedicated routersHigh Speed interconnectBackbone Links ONLYDo not touch them!Border Networkn n n n Dedicated border router to other ISPsThe ISP’s “front” doorTransparent web caching?Two in backbone is minimum guarantee forredundancy11
PoP Modulesp ISP Servicesn n n n p DNS (cache, secondary)News (still relevant?)Mail (POP3, Relay, Anti-virus/anti-spam)WWW (server, proxy, cache)Hosted Services/DataCentresn n n Virtual Web, WWW (server, proxy, cache)Information/Content ServicesElectronic Commerce12
PoP Modulesp Network Operations Centren n n n p Consider primary and backup locationsNetwork monitoringStatistics and log gatheringDirect but secure accessOut of Band Management Networkn The ISP Network “Safety Belt”13
Low Speed Access ModuleWeb CacheAccess NetworkGateway RoutersPrimary RateT1/E1Access ServersPSTN lines tomodem bankTo Core RoutersPSTN lines tobuilt-in modemsTACACS /Radiusproxy, DNS resolver,Content14
Medium Speed Access ModuleAggregation EdgeChannelised T1/E164K and nx64KcircuitsTo Core RoutersMixture of channelisedT1/E1, 56/64K andnx64K circuits15
High Speed Access ModuleAggregation EdgeMetro EthernetChannelised T3/E3To Core RoutersChannelised OC3/OC1216
Broadband Access ModuleWeb CacheTelephone NetworkDSLAMBRASIP, ATMCable RASAccess NetworkGateway RoutersTo Core RoutersThe cable systemSSG, DHCP, TACACS or Radius Servers/Proxies,DNS resolver, Content17
ISP Services ModuleTo core routersService NetworkGateway he18
Hosted Services ModuleTo core routersHosted NetworkGateway Routersvlan11vlan12 vlan13vlan14vlan15vlan16vlan17Customer 1Customer 3Customer 5Customer 7Customer 2Customer 4Customer 619
Border ModuleTo local IXPNB: router has no default route local AS routing table onlyISP1ISP2NetworkBorder RoutersTo core routers20
NOC ModuleCritical ServicesModuleTo core routersCorporate LANOut of BandHosted NetworkGateway RoutersManagement NetworkFirewall2811/32asyncNetFlow TACACS SYSLOG Primary DNSAnalyserserverserverNetwork Operations Centre StaffBilling, Databaseand AccountingSystems21
Out of Band NetworkOut of BandManagement NetworkRouterconsolesTerminal serverTo the NOCNetFlowenabledroutersNetFlowCollectorOut of Band Ethernet22
Backbone NetworkDesign23
Backbone DesignRouted Backbonep Switched Backbonep n n p Point-to-point circuitsn p ATM/Frame Relay core networkNow obsoletenx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48,10GigE, OC192, OC768ATM/Frame Relay service from telcon n n T3, OC3, OC12, deliveryEasily upgradeable bandwidth (CIR)Almost vanished in availability now24
Distributed Network Designp PoP design “standardised”n operational scalability and simplicityISP essential services distributed aroundbackbonep NOC and “backup” NOCp Redundant backbone linksp 25
Distributed Network DesignCustomerconnectionsISP ServicesBackupOperations CentrePOP TwoCustomerconnectionsCustomerconnectionsISP ServicesPOP OnePOP ThreeISP ServicesExternalconnectionsOperations CentreExternalconnections26
Backbone Linksp ATM/Frame Relayn n p Virtually disappeared due to overhead, extraequipment, and shared with other customersof the telcoMPLS has replaced ATM & FR as the telcofavouriteLeased Line/Circuitn n Most popular with backbone providersIP over Optics and Metro Ethernet verycommon in many parts of the world27
Long Distance Backbone LinksThese usually cost morep Important to plan for the futurep n n n This means at least two years aheadStay in budget, stay realisticUnplanned “emergency” upgrades will bedisruptive without redundancy in the networkinfrastructure28
Long Distance Backbone Linksp Allow sufficient capacity on alternativepaths for failure situationsn n n n Sufficient can depend on the business strategySufficient can be as little as 20%Sufficient is usually over 50% as this offers“business continuity” for customers in the caseof link failureSome businesses choose 0%p Very short sighted, meaning they have no sparecapacity at all!!29
Long Distance LinksPOP TwoLong distance linkPOP OnePOP ThreeAlternative/Backup Path30
Metropolitan Area Backbone Linksp Tend to be cheapern n p Circuit concentrationChoose from multiple suppliersThink bign n n More redundancyLess impact of upgradesLess impact of failures31
Metropolitan Area Backbone LinksPOP TwoMetropolitan LinksPOP OnePOP ThreeMetropolitan LinksTraditional Point to Point Links32
Upstream Connectivityand Peering33
Transitsp Transit provider is another autonomous systemwhich is used to provide the local network withaccess to other networksn n p Might be local or regional onlyBut more usually the whole InternetTransit providers need to be chosen wisely:n Only onep n Too manyp p p p no redundancymore difficult to load balanceno economy of scale (costs more per Mbps)hard to provide service qualityRecommendation: at least two, no morethan three
Common Mistakesp ISPs sign up with too many transit providersn n n p Lots of small circuits (cost more per Mbps than largerones)Transit rates per Mbps reduce with increasing transitbandwidth purchasedHard to implement reliable traffic engineering thatdoesn’t need daily fine tuning depending on customeractivitiesNo diversityn n Chosen transit providers all reached over same satelliteor same submarine cableChosen transit providers have poor onward transit andpeering
Peersp p A peer is another autonomous system with whichthe local network has agreed to exchange locallysourced routes and trafficPrivate peern p Public peern p Private link between two providers for the purpose ofinterconnectingInternet Exchange Point, where providers meet andfreely decide who they will interconnect withRecommendation: peer as much as possible!
Common MistakesMistaking a transit provider’s “Exchange”business for a no-cost public peering pointp Not working hard to get as much peeringas possiblep n n p Physically near a peering point (IXP) but notpresent at it(Transit is rarely cheaper than peering!!)Ignoring/avoiding competitors becausethey are competitionn Even though potentially valuable peeringpartner to give customers a better experience
Private Interconnectionp Two service providers agree tointerconnect their networksn n They exchange prefixes they originate into therouting system (usually their aggregatedaddress blocks)They share the cost of the infrastructure tointerconnectp p n Typically each paying half the cost of the link (be itcircuit, satellite, microwave, fibre, )Connected to their respective peering routersPeering routers only carry domestic prefixes38
Private InterconnectionUpstreamUpstreamISP2PRPRISP1p PR peering routern n n n p Runs iBGP (internal) and eBGP (with peer)No default routeNo “full BGP table”Domestic prefixes onlyPeering router used for all private interconnects39
Public Interconnectionp Service provider participates in anInternet Exchange Pointn n It exchanges prefixes it originates into therouting system with the participants of the IXPIt chooses who to peer with at the IXPp p n n Bi-lateral peering (like private interconnect)Multi-lateral peering (via IXP’s route server)It provides the router at the IXP and providesthe connectivity from their PoP to the IXPThe IXP router carries only domestic prefixes40
Public 3-PRp ISP1ISP2-PRISP1-PR peering router of our ISPn n n n p ISP1-PRRuns iBGP (internal) and eBGP (with IXP peers)No default routeNo “full BGP table”Domestic prefixes onlyPhysically located at the IXP41
Public Interconnectionp The ISP’s router IXP peering router needs carefulconfiguration:n n n n It is remote from the domestic backboneShould not originate any domestic prefixes(As well as no default route, no full BGP table)Filtering of BGP announcements from IXP peers (in andout)42
Upstream/Transit Connectionp Two scenarios:n Transit provider is in the localityp n Transit provider is a long distance awayp p Which means bandwidth is cheap, plentiful, easy toprovision, and easily upgradedOver undersea cable, satellite, long-haul crosscountry fibre, etcBoth scenarios have differentrequirements which need to be considered43
Local Transit ProviderISP1ARBRTransitp BR ISP’s Border Routern n n n Runs iBGP (internal) and eBGP (with transit)Either receives default route or the full BGP table fromupstreamBGP policies are implemented here (depending onconnectivity)Packet filtering is implemented here (as required)44
Distant Transit ProviderAR1TransitBRISP1AR2p BR ISP’s Border Routern n n n n Co-located in a co-lo centre (typical) or in the upstream provider’spremisesRuns iBGP with rest of ISP1 backboneRuns eBGP with transit provider router(s)Implements BGP policies, packet filtering, etcDoes not originate any domestic prefixes45
Distant Transit Providerp Positioning a router close to the TransitProvider’s infrastructure is stronglyencouraged:n n n Long haul circuits are expensive, so the routerallows the ISP to implement appropriatefiltering firstMoves the buffering problem away from theTransit providerRemote co-lo allows the ISP to choose anothertransit provider and migrate connections withminimum downtime46
Distant Transit Providerp Other points to consider:n n n n Does require remote hands support(Remote hands would plug or unplug cables,power cycle equipment, replace equipment, etcas instructed)Appropriate support contract from equipmentvendor(s)Sensible to consider two routers and two longhaul links for redundancy47
Summaryp Design considerations for:n Private interconnectsp n Public interconnectsp n Router co-lo at an IXPLocal transit providerp n Simple private peeringSimple upstream interconnectLong distance transit providerp Router remote co-lo at datacentre or Transitpremises48
Addressing49
Getting IPv4 & IPv6 address spacep Take part of upstream ISP’s PA spaceorp Become a member of your Regional InternetRegistry and get your own allocationn n p Require a plan for a year aheadGeneral policies are outlined in RFC2050, morespecific details are on the individual RIR websiteThere is no more IPv4 address space at IANAn n n APNIC & RIPE NCC are now in their “final /8” IPv4delegation policy phaseLimited IPv4 availableIPv6 allocations are simple to get in most RIR regions50
What about RFC1918 addressing?p RFC1918 defines IPv4 addresses reserved forprivate Internetsn n p Commonly used within end-user networksn n p Not to be used on Internet backboneshttp://www.ietf.org/rfc/rfc1918.txtNAT used to translate from private internal to publicexternal addressingAllows the end-user network to migrate ISPs without amajor internal renumbering exerciseISPs must filter RFC1918 addressing at theirnetwork edgen http://www.cymru.com/Documents/bogonlist.html51
What about RFC1918 addressing?p There is a long list of well known problems:n p :n n n n n n n n n False belief it conserves address spaceAdverse effects on TracerouteEffects on Path MTU DiscoveryUnexpected interactions with some NAT implementationsInteractions with edge anti-spoofing techniquesPeering using loopbacksAdverse DNS InteractionSerious Operational and Troubleshooting issuesSecurity Issuesp false sense of security, defeating existing securitytechniques52
What about RFC1918 addressing?p Infrastructure Security: not improved by usingprivate addressingn p Troubleshooting: made an order of magnitudehardern n p Still can be attacked from inside, or from customers, orby reflection techniques from the outsideNo Internet view from routersOther ISPs cannot distinguish between down and brokenSummary:n ALWAYS use globally routable IP addressing for ISPInfrastructure53
Addressing Plans – ISPInfrastructurep p Address block for router loop-back interfacesAddress block for infrastructuren n n p Per PoP or whole backboneSummarise between sites if it makes senseAllocate according to genuine requirements, not historicclassful boundariesSimilar allocation policies should be used for IPv6as welln ISPs just get a substantially larger block (relatively) soassignments within the backbone are easier to make54
Addressing Plans – CustomerCustomers are assigned address spaceaccording to needp Should not be reserved or assigned on aper PoP basisp n n ISP iBGP carries customer netsAggregation not required and usually notdesirable55
Addressing Plans (contd)p Document infrastructure allocationn p Eases operation, debugging and managementDocument customer allocationn n n Contained in iBGPEases operation, debugging and managementSubmit network object to RIR Database56
Routing Protocols57
Routing Protocolsp IGP – Interior Gateway Protocoln n p EGP – Exterior Gateway Protocoln n p Carries infrastructure addresses, point-to-pointlinksExamples are OSPF, ISIS,.Carries customer prefixes and Internet routesCurrent EGP is BGP version 4No connection between IGP and EGP58
Why Do We Need an IGP?p ISP backbone scalingn n n n HierarchyModular infrastructure constructionLimiting scope of failureHealing of infrastructure faults using dynamicrouting with fast convergence59
Why Do We Need an EGP?p Scaling to large networkn n p HierarchyLimit scope of failurePolicyn n n Control reachability to prefixesMerge separate organizationsConnect multiple IGPs60
Interior versus Exterior RoutingProtocolsp Interiorn n n n Automatic neighbourdiscoveryGenerally trust your IGProutersPrefixes go to all IGProutersBinds routers in one AStogetherp Exteriorn n n n Specifically configuredpeersConnecting with outsidenetworksSet administrativeboundariesBinds AS’s together61
Interior versus Exterior RoutingProtocolsp Interiorn n Carries ISPinfrastructure addressesonlyISPs aim to keep theIGP small for efficiencyand scalabilityp Exteriorn n n Carries customerprefixesCarries InternetprefixesEGPs are independentof ISP network topology62
Hierarchy of Routing ProtocolsOther ISPsBGP4BGP4and OSPF/ISISBGP4IXPStatic/BGP4Customers63
Routing Protocols:Choosing an IGPp p OSPF and ISIS have very similar propertiesWhich to choose?n n n n n n Choose which is appropriate for your operators’experienceIn most vendor releases, both OSPF and ISIS havesufficient “nerd knobs” to tweak the IGP’s behaviourOSPF runs on IPISIS runs on infrastructure, alongside IPISIS supports both IPv4 and IPv6OSPFv2 (IPv4) plus OSPFv3 (IPv6)64
Routing Protocols:IGP Recommendationsp Keep the IGP routing table as small as possiblen p If you can count the routers and the point-to-point linksin the backbone, that total is the number of IGP entriesyou should seeIGP details:n n n Should only have router loopbacks, backbone WANpoint-to-point link addresses, and network addresses ofany LANs having an IGP running on themStrongly recommended to use inter-routerauthenticationUse inter-area summarisation if possible65
Routing Protocols:More IGP recommendationsp To fine tune IGP table size more, consider:n Using “ip unnumbered” on customer point-topoint links – saves carrying that /30 in IGPp n n n (If customer point-to-point /30 is required formonitoring purposes, then put this in iBGP)Use contiguous addresses for backbone WANlinks in each area – then summarise intobackbone areaDon’t summarise router loopback addresses –as iBGP needs those (for next-hop)Use iBGP for carrying anything which does notcontribute to the IGP Routing process66
Routing Protocols:iBGP Recommendationsp iBGP should carry everything whichdoesn’t contribute to the IGP routingprocessn n n n Internet routing tableCustomer assigned addressesCustomer point-to-point linksAccess network dynamic address pools,passive LANs, etc67
Routing Protocols:More iBGP Recommendationsp Scalable iBGP features:n n n n Use neighbour authenticationUse peer-groups to speed update process andfor configuration efficiencyUse communities for ease of filteringUse route-reflector hierarchyp Route reflector pair per PoP (overlaid clusters)68
Security69
Securityp p p p ISP Infrastructure securityISP Network securitySecurity is not optional!ISPs need to:n n n p Protect themselvesHelp protect their customers from the InternetProtect the Internet from their customersThe following slides are general recommendationsn Do more research on security before deploying anynetwork70
ISP Infrastructure Securityp Router & Switch Securityn Use Secure Shell (SSH) for device access &managementp n Device management access filters should onlyallow NOC and device-to-device accessp n Do NOT use TelnetDo NOT allow external accessUse TACACS for user authentication andauthorisationp Do NOT create user accounts on routers/switches71
ISP Infrastructure Securityp Remote accessn n For Operations Engineers who need accesswhile not in the NOCCreate an SSH server host (this is all it does)p n Or a Secure VPN access serverOps Engineers connect here, and then they canaccess the NOC and network devices72
ISP Infrastructure Securityp Other network devices?n n p Servers and Services?n n p These probably do not have sophisticated securitytechniques like routers or switches doProtect them at the LAN or point-to-point ingress (onrouter)Protect servers on the LAN interface on the routerConsider using iptables &c on the servers tooSNMPn n Apply access-list to the SNMP portsShould only be accessible by management system, notthe world73
ISP Infrastructure Securityp General Advice:n n Routers, Switches and other network devicesshould not be contactable from outside the ASAchieved by blocking typical managementaccess protocols for the infrastructure addressblock at the network perimeterp n Use the ICSI Netalyser to check access levels:p n E.g. ssh, telnet, http, snmp, http://netalyzr.icsi.berkeley.eduDon’t block everything: BGP, tra
ISP & IXP Network Design ! PoP Topologies and Design ! Backbone Design ! Upstream Connectivity & Peering ! Addressing ! Routing Protocols ! Out of Band Management ! Operational Considerations ! Internet Exchange Points 2 . Point of Presence Topologies 3 . PoP Topologies ! .
