Transcription
ISP Network DesignISP WorkshopsLast updated 16 September 20131
ISP Network DesignPoP Topologies and Designp Backbone Designp Upstream Connectivity & Peeringp Addressingp Routing Protocolsp Securityp Out of Band Managementp Operational Considerationsp 2
Point of PresenceTopologies3
PoP TopologiesCore routers – high speed trunkconnectionsp Distribution routers and Access routers –high port densityp Border routers – connections to otherprovidersp Service routers – hosting and serversp Some functions might be handled by asingle routerp 4
PoP DesignModular Designp Aggregation Services separated accordingtop n n n n connection speedcustomer servicecontention ratiosecurity considerations5
Modular PoP DesignOther ISPsISP Services(DNS, Mail, News,FTP, WWW)Web CacheHosted Services &DatacentreBackbone linkto another PoPBackbone linkto another PoPNetworkCoreConsumer cable,xDSL andwireless AccessConsumerDial AccessBusiness customeraggregation layerMetroE customeraggregation layerNetworkOperationsCentreChannelised circuitsfor leased line circuit deliveryGigE fibre trunksfor MetroE circuit delivery6
Modular Routing Protocol Designp Modular IGP implementationn n n p IGP “area” per PoPCore routers in backbone area (Area 0/L2)Aggregation/summarisation where possibleinto the coreModular iBGP implementationn n n BGP route reflector clusterCore routers are the route-reflectorsRemaining routers are clients & peer withroute-reflectors only7
Point of Presence Design8
PoP Modulesp Low Speed customer connectionsn n n p PSTN/ISDN dialupLow bandwidth needsLow revenue, large numbersLeased line customer connectionsn n n n E1/T1 speed rangeDelivery over channelised mediaMedium bandwidth needsMedium revenue, medium numbers9
PoP Modulesp Broad Band customer connectionsn n n p xDSL, Cable and WirelessHigh bandwidth needsLow revenue, large numbersMetroE & Highband customer connectionsn n n n Trunk onto GigE or 10GigE of 10Mbps andhigherChannelised OC3/12 delivery of E3/T3 andhigherHigh bandwidth needsHigh revenue, low numbers10
PoP Modulesp PoP Coren n n n p Two dedicated routersHigh Speed interconnectBackbone Links ONLYDo not touch them!Border Networkn n n n Dedicated border router to other ISPsThe ISP’s “front” doorTransparent web caching?Two in backbone is minimum guarantee forredundancy11
PoP Modulesp ISP Servicesn n n n p DNS (cache, secondary)News (still relevant?)Mail (POP3, Relay, Anti-virus/anti-spam)WWW (server, proxy, cache)Hosted Services/DataCentresn n n Virtual Web, WWW (server, proxy, cache)Information/Content ServicesElectronic Commerce12
PoP Modulesp Network Operations Centren n n n p Consider primary and backup locationsNetwork monitoringStatistics and log gatheringDirect but secure accessOut of Band Management Networkn The ISP Network “Safety Belt”13
Low Speed Access ModuleWeb CacheAccess NetworkGateway RoutersPrimary RateT1/E1Access ServersPSTN lines tomodem bankTo Core RoutersPSTN lines tobuilt-in modemsTACACS /Radiusproxy, DNS resolver,Content14
Medium Speed Access ModuleAggregation EdgeChannelised T1/E164K and nx64KcircuitsTo Core RoutersMixture of channelisedT1/E1, 56/64K andnx64K circuits15
High Speed Access ModuleAggregation EdgeMetro EthernetChannelised T3/E3To Core RoutersChannelised OC3/OC1216
Broadband Access ModuleWeb CacheTelephone NetworkDSLAMBRASIP, ATMCable RASAccess NetworkGateway RoutersTo Core RoutersThe cable systemSSG, DHCP, TACACS or Radius Servers/Proxies,DNS resolver, Content17
ISP Services ModuleTo core routersService NetworkGateway he18
Hosted Services ModuleTo core routersHosted NetworkGateway Routersvlan11vlan12 vlan13vlan14vlan15vlan16vlan17Customer 1Customer 3Customer 5Customer 7Customer 2Customer 4Customer 619
Border ModuleTo local IXPNB: router has no default route local AS routing table onlyISP1ISP2NetworkBorder RoutersTo core routers20
NOC ModuleCritical ServicesModuleTo core routersCorporate LANOut of BandHosted NetworkGateway RoutersManagement NetworkFirewallasync terminal serverNetFlow TACACS SYSLOG Primary DNSAnalyserserverserverNetwork Operations Centre StaffBilling, Databaseand AccountingSystems21
Out of Band NetworkOut of BandManagement NetworkRouterconsolesTerminal serverTo the NOCNetFlowenabledroutersNetFlowCollectorOut of Band Ethernet22
Backbone NetworkDesign23
Backbone DesignRouted Backbonep Switched Backbonep n n p Point-to-point circuitsn p ATM/Frame Relay core networkNow obsoletenx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48,10GigE, OC192, OC768, 100GEATM/Frame Relay service from telcon n n T3, OC3, OC12, deliveryEasily upgradeable bandwidth (CIR)Almost vanished in availability now24
Distributed Network Designp PoP design “standardised”n operational scalability and simplicityISP essential services distributed aroundbackbonep NOC and “backup” NOCp Redundant backbone linksp 25
Distributed Network DesignCustomerconnectionsISP ServicesBackupOperations CentrePOP TwoCustomerconnectionsCustomerconnectionsISP ServicesPOP OnePOP ThreeISP ServicesExternalconnectionsOperations CentreExternalconnections26
Backbone Linksp ATM/Frame Relayn n p Virtually disappeared due to overhead, extraequipment, and shared with other customersof the telcoMPLS has replaced ATM & FR as the telcofavouriteLeased Line/Circuitn n Most popular with backbone providersIP over Optics and Metro Ethernet verycommon in many parts of the world27
Long Distance Backbone LinksThese usually cost morep Important to plan for the futurep n n n This means at least two years aheadStay in budget, stay realisticUnplanned “emergency” upgrades will bedisruptive without redundancy in the networkinfrastructure28
Long Distance Backbone Linksp Allow sufficient capacity on alternativepaths for failure situationsn n n n Sufficient can depend on the business strategySufficient can be as little as 20%Sufficient is usually over 50% as this offers“business continuity” for customers in the caseof link failureSome businesses choose 0%p Very short sighted, meaning they have no sparecapacity at all!!29
Long Distance LinksPOP TwoLong distance linkPOP OnePOP ThreeAlternative/Backup Path30
Metropolitan Area Backbone Linksp Tend to be cheapern n p Circuit concentrationChoose from multiple suppliersThink bign n n More redundancyLess impact of upgradesLess impact of failures31
Metropolitan Area Backbone LinksPOP TwoMetropolitan LinksPOP OnePOP ThreeMetropolitan LinksTraditional Point to Point Links32
Upstream Connectivityand Peering33
Transitsp Transit provider is another autonomous systemwhich is used to provide the local network withaccess to other networksn n p Might be local or regional onlyBut more usually the whole InternetTransit providers need to be chosen wisely:n Only onep n Too manyp p p p no redundancymore difficult to load balanceno economy of scale (costs more per Mbps)hard to provide service qualityRecommendation: at least two, no morethan three
Common Mistakesp ISPs sign up with too many transit providersn n n p Lots of small circuits (cost more per Mbps than largerones)Transit rates per Mbps reduce with increasing transitbandwidth purchasedHard to implement reliable traffic engineering thatdoesn’t need daily fine tuning depending on customeractivitiesNo diversityn n Chosen transit providers all reached over same satelliteor same submarine cableChosen transit providers have poor onward transit andpeering
Peersp p A peer is another autonomous system with whichthe local network has agreed to exchange locallysourced routes and trafficPrivate peern p Public peern p Private link between two providers for the purpose ofinterconnectingInternet Exchange Point, where providers meet andfreely decide who they will interconnect withRecommendation: peer as much as possible!
Common MistakesMistaking a transit provider’s “Exchange”business for a no-cost public peering pointp Not working hard to get as much peeringas possiblep n n p Physically near a peering point (IXP) but notpresent at it(Transit sometimes is cheaper than peering!!)Ignoring/avoiding competitors becausethey are competitionn Even though potentially valuable peeringpartner to give customers a better experience
Private Interconnectionp Two service providers agree tointerconnect their networksn n They exchange prefixes they originate into therouting system (usually their aggregatedaddress blocks)They share the cost of the infrastructure tointerconnectp p n Typically each paying half the cost of the link (be itcircuit, satellite, microwave, fibre, )Connected to their respective peering routersPeering routers only carry domestic prefixes38
Private InterconnectionUpstreamUpstreamISP2PRPRISP1p PR peering routern n n n p Runs iBGP (internal) and eBGP (with peer)No default routeNo “full BGP table”Domestic prefixes onlyPeering router used for all private interconnects39
Public Interconnectionp Service provider participates in anInternet Exchange Pointn n It exchanges prefixes it originates into therouting system with the participants of the IXPIt chooses who to peer with at the IXPp p n n Bi-lateral peering (like private interconnect)Multi-lateral peering (via IXP’s route server)It provides the router at the IXP and providesthe connectivity from their PoP to the IXPThe IXP router carries only domestic prefixes40
Public 3-PRp ISP1ISP2-PRISP1-PR peering router of our ISPn n n n p ISP1-PRRuns iBGP (internal) and eBGP (with IXP peers)No default routeNo “full BGP table”Domestic prefixes onlyPhysically located at the IXP41
Public Interconnectionp The ISP’s router IXP peering router needs carefulconfiguration:n n n n p It is remote from the domestic backboneShould not originate any domestic prefixes(As well as no default route, no full BGP table)Filtering of BGP announcements from IXP peers (in andout)Provision of a second link to the IXP:n n (for redundancy or extra capacity)Usually means installing a second routerp p Connected to a second switch (if the IXP has two more moreswitches)Interconnected with the original router (and part of iBGP mesh)42
Public P4-PRISP3-PRp ISP1-PR1ISP1ISP2-PRProvision of a second link to the IXP meansconsidering redundancy in the SP’s backbonen n n Two routersTwo independent linksSeparate switches (if IXP has two or more switches)43
Upstream/Transit Connectionp Two scenarios:n Transit provider is in the localityp n Transit provider is a long distance awayp p Which means bandwidth is cheap, plentiful, easy toprovision, and easily upgradedOver undersea cable, satellite, long-haul crosscountry fibre, etcEach scenario has different considerationswhich need to be accounted for44
Local Transit ProviderISP1ARBRTransitp BR ISP’s Border Routern n n n Runs iBGP (internal) and eBGP (with transit)Either receives default route or the full BGP table fromupstreamBGP policies are implemented here (depending onconnectivity)Packet filtering is implemented here (as required)45
Distant Transit ProviderAR1TransitBRISP1AR2p BR ISP’s Border Routern n n n n Co-located in a co-lo centre (typical) or in the upstream provider’spremisesRuns iBGP with rest of ISP1 backboneRuns eBGP with transit provider router(s)Implements BGP policies, packet filtering, etcDoes not originate any domestic prefixes46
Distant Transit Providerp Positioning a router close to the TransitProvider’s infrastructure is stronglyencouraged:n n n Long haul circuits are expensive, so the routerallows the ISP to implement appropriatefiltering firstMoves the buffering problem away from theTransit providerRemote co-lo allows the ISP to choose anothertransit provider and migrate connections withminimum downtime47
Distant Transit Providerp Other points to consider:n n n n Does require remote hands support(Remote hands would plug or unplug cables,power cycle equipment, replace equipment, etcas instructed)Appropriate support contract from equipmentvendor(s)Sensible to consider two routers and two longhaul links for redundancy48
Distant Transit ProviderAR1TransitBR1AR2p ISP1BR2Upgrade scenario:n n n Provision two routersTwo independent circuitsConsider second transit provider and/or turning up atan IXP49
Summaryp Design considerations for:n Private interconnectsp n Public interconnectsp n Router co-lo at an IXPLocal transit providerp n Simple private peeringSimple upstream interconnectLong distance transit providerp Router remote co-lo at datacentre or Transitpremises50
Upstream Connectivityand Peering Case StudyHow Seacom chose theirinternational peering locationsand transit providers51
ObjectiveObtain high grade Internet connectivity forthe wholesale market in Africa to the restof the worldp Emphasis on:p n n n ReliabilityInterconnectivity densityScalability52
Metrics Needed in DeterminingSolution (1)p Focusing on operators that cover the destinationsmostly required by African p p i.e., English-speaking (Europe, North America)Include providers with good connectivity intoSouth America and the Asia Pacific.Little need for providers who are strong in theMiddle East, as demand from Africa for thoseregions is very, very low.53
Metrics Needed in DeterminingSolution (2)p Split the operators between Marseille (where theSEACOM cable lands) and London (where there isgood Internet density)n n p p To avoid outages due to backhaul failure across EuropeAnd still maintain good access to the InternetLook at providers who are of similar size so asnot to fidget too much (or at all) with BGP tuning.The providers needed to support:n n n 10Gbps portsBursting bandwidth/billingFuture support for 100Gbps or N x 10Gbps54
Metrics Needed in DeterminingSolution (3)p Implement peering at major exchange points inEuropen To off-set long term operating costs re: upstreamproviders.55
Implementing Solutionp p p p p p p Connected to Level(3) and GT-T (formerlyInteliquent, formerly Tinet) in MarseilleConnected to NTT and TeliaSonera in LondonPeered in London (LINX)Peered in Amsterdam (AMS-IX)BGP setup to prefer traffic being exchanged atLINX and AMS-IXBGP setup to prefer traffic over the upstreamsthat we could not peer awayNo additional tuning done on either peered ortransit traffic, i.e., no prepending, no deaggregation, etc. All traffic setup to flow naturally56
End Resultp p p p p p 50% of traffic peered away in less than 2xmonths of peering at LINX and AMS-IX50% of traffic handled by upstream providersEqual traffic being handled by Level(3) and GT-Tin MarseilleEqual traffic being handled by TeliaSonera andNTT in LondonTraffic distribution ratios across all the transitproviders is some 1:1:0.9:0.9This has been steady state for the last 12xmonthsn No BGP tuning has been done at all57
Addressing58
Where to get IP addresses and ASnumbersp p Your upstream ISPAfrican p Asia and the Pacificn p ARIN – http://www.arin.netLatin America and the Caribbeann p APNIC – http://www.apnic.netNorth American p AfriNIC – http://www.afrinic.netLACNIC – http://www.lacnic.netEurope and Middle Eastn RIPE NCC – http://www.ripe.net/info/ncc59
Internet Registry Regions60
Getting IP address spacep Take part of upstream ISP’s PA spaceorp Become a member of your Regional InternetRegistry and get your own allocationn n p Require a plan for a year aheadGeneral policies are outlined in RFC2050, morespecific details are on the individual RIR websiteThere is no more IPv4 address space at IANAn n n APNIC and RIPE NCC are now in their “final /8” IPv4delegation policy phaseLimited IPv4 availableIPv6 allocations are simple to get in most RIR regions61
What about RFC1918 addressing?p RFC1918 defines IPv4 addresses reserved forprivate Internetsn n p Commonly used within end-user networksn n p Not to be used on Internet backboneshttp://www.ietf.org/rfc/rfc1918.txtNAT used to translate from private internal to publicexternal addressingAllows the end-user network to migrate ISPs without amajor internal renumbering exerciseISPs must filter RFC1918 addressing at theirnetwork edgen http://www.cymru.com/Documents/bogonlist.html62
What about RFC1918 addressing?p There is a long list of well known problems:n p :n n n n n n n n n False belief it conserves address spaceAdverse effects on TracerouteEffects on Path MTU DiscoveryUnexpected interactions with some NAT implementationsInteractions with edge anti-spoofing techniquesPeering using loopbacksAdverse DNS InteractionSerious Operational and Troubleshooting issuesSecurity Issuesp false sense of security, defeating existing securitytechniques63
Private versus Globally Routable IPAddressingp Infrastructure Security: not improved by usingprivate addressingn p Troubleshooting: made an order of magnitudehardern n p Still can be attacked from inside, or from customers, orby reflection techniques from the outsideNo Internet view from routersOther ISPs cannot distinguish between down and brokenSummary:n ALWAYS use globally routable IP addressing for ISPInfrastructure64
Addressing Plans – ISPInfrastructurep p Address block for router loop-back interfacesAddress block for infrastructuren n n p Per PoP or whole backboneSummarise between sites if it makes senseAllocate according to genuine requirements, not historicclassful boundariesSimilar allocation policies should be used for IPv6as welln ISPs just get a substantially larger block (relatively) soassignments within the backbone are easier to make65
Addressing Plans – CustomerCustomers are assigned address spaceaccording to needp Should not be reserved or assigned on aper PoP basisp n n ISP iBGP carries customer netsAggregation not required and usually notdesirable66
Addressing Plans – ISP Infrastructurep Phase One223.10.0.0/21223.10.0.1223.10.6.255 /24Infrastructure LoopbacksCustomer assignmentsp Phase al assignments223.10.15.255New Assignments67
Addressing PlansPlanningp Registries will usually allocate the nextblock to be contiguous with the firstallocationn n n Minimum allocation could be /21Very likely that subsequent allocation willmake this up to a /20So plan accordingly68
Addressing Plans (contd)p Document infrastructure allocationn p Eases operation, debugging and managementDocument customer allocationn n n Contained in iBGPEases operation, debugging and managementSubmit network object to RIR Database69
Routing Protocols70
Routing Protocolsp IGP – Interior Gateway Protocoln n p EGP – Exterior Gateway Protocoln n p Carries infrastructure addresses, point-to-pointlinksExamples are OSPF, ISIS,.Carries customer prefixes and Internet routesCurrent EGP is BGP version 4No connection between IGP and EGP71
Why Do We Need an IGP?p ISP backbone scalingn n n n HierarchyModular infrastructure constructionLimiting scope of failureHealing of infrastructure faults using dynamicrouting with fast convergence72
Why Do We Need an EGP?p Scaling to large networkn n p HierarchyLimit scope of failurePolicyn n n Control reachability to prefixesMerge separate organizationsConnect multiple IGPs73
Interior versus Exterior RoutingProtocolsp Interiorn n n n Automatic neighbourdiscoveryGenerally trust your IGProutersPrefixes go to all IGProutersBinds routers in one AStogetherp Exteriorn n n n Specifically configuredpeersConnecting with outsidenetworksSet administrativeboundariesBinds AS’s together74
Interior versus Exterior RoutingProtocolsp Interiorn n Carries ISPinfrastructure addressesonlyISPs aim to keep theIGP small for efficiencyand scalabilityp Exteriorn n n Carries customerprefixesCarries InternetprefixesEGPs are independentof ISP network topology75
Hierarchy of Routing ProtocolsOther ISPsBGP4BGP4and OSP
ISP Network Design ISP Workshops Last updated 16 September 2013 1 . ISP Network Design ! PoP Topologies and Design ! Backbone Design ! Upstream Connectivity & Peering ! Addressing ! Routing Protocols ! Security ! Out of Band Management ! Operational Considerations 2 . Point of Presence .
