Transcription
ISP Network Design PoP Topologies and Design Backbone Design ISP Systems DesignISP Network Design Addressing Routing ProtocolsISP/IXP Workshops Security Out of Band Management Operational ConsiderationsCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.1Cisco ISPWorkshops2 2005, Cisco Systems, Inc. All rights reserved.PoP Topologies Core routers – high speed trunk connections Distribution routers and Access routers – highport densityPoint of Presence Topologies Border routers – connections to other providers Service routers – hosting and servers Some functions might be handled by a singlerouterCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.3PoP DesignCisco ISPWorkshops4 2005, Cisco Systems, Inc. All rights reserved.Modular PoP DesignOther ISPsISP Services(DNS, Mail, News,FTP, WWW) Modular DesignWeb CacheHosted ServicesBackbone linkto another PoP Aggregation Services separated according toBackbone linkto another PoPNetworkCoreconnection speedcustomer serviceConsumer cable,xDSL andwireless AccessConsumerDIAL Accesscontention ratiosecurity considerationsNx64 customeraggregation layerNxT1/E1 customeraggregation layerNetworkOperationsCentreChannelised T1/E1 circuitsNx64 leased line circuit deliveryCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.5Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.Channelised T3/E3 circuitsT1/E1 leased line circuit delivery61
Modular Routing Protocol Design Modular IGP implementationIGP “area” per moduleaggregation/summarisation where possible into the corePoint of Presence Design Modular iBGP implementationBGP route reflector cluster per modulecore routers are route-reflectorsclients peer with core onlyCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.7PoP ModulesCisco ISPWorkshops High Speed customer connectionsPSTN/ISDN dialupE1 speedslow bandwidth needsmedium bandwidth needslow revenue, large numbershigh revenue, low numbers Medium Speed customer connections Broad Band customer connections56/64K to sub-T1/E1 speedsxDSL, Cable and Wirelesslow bandwidth needshigh bandwidth needsmedium revenue, medium numberslow revenue, large numbers 2005, Cisco Systems, Inc. All rights reserved.9PoP ModulesCisco ISPWorkshops10 ISP ServicesTwo dedicated routersDNS (cache, secondary)High Speed interconnectNews, Mail (POP3, Relay)Backbone Links ONLYWWW (server, proxy, cache)Do not touch them! Hosted Services Border NetworkVirtual Web, WWW (server, proxy, cache)dedicated border router to other ISPsInformation/Content Servicesthe ISP’s “front” doorElectronic Commercetransparent web caching 2005, Cisco Systems, Inc. All rights reserved. 2005, Cisco Systems, Inc. All rights reserved.PoP Modules PoP CoreCisco ISPWorkshops8PoP Modules Low Speed customer connectionsCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.11Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.122
PoP ModulesLow Speed Access Module Network Operations CentreWeb Cacheprimary and backup locationsAS5300network monitoringAccess NetworkGateway RoutersPrimary Rate T1/E1statistics and log gatheringAS2511PSTN lines tomodem bankdirect but secure access Out of Band Management NetworkTo Core Routers2600/3600PSTN lines tobuilt-in modemsThe ISP Network “Safety Belt”TACACS /Radiusproxy, DNS resolver,ContentCisco ISPWorkshops13 2005, Cisco Systems, Inc. All rights reserved.Medium Speed Access ModuleCisco ISPWorkshopsHigh Speed Access Module3800/7206/76007200/7600Channelised T1/E1Channelised T3/E364K and nx64K circuitsT1 and E1 circuitsTo Core RoutersMixture of channelisedT1/E1, 56/64K andnx64K circuitsCisco ISPWorkshops14 2005, Cisco Systems, Inc. All rights reserved.To Core RoutersMixture of channelisedT3/E3 and T1/E1 circuits15 2005, Cisco Systems, Inc. All rights reserved.Broad Band Access ModuleCisco ISPWorkshops16 2005, Cisco Systems, Inc. All rights reserved.ISP Services ModuleTo core routersWeb CacheTelephone Network61xx6400IP, ATMuBR7246Service NetworkGateway RoutersAccess NetworkGateway RoutersTo Core RoutersThe cable systemWWWcacheCisco ISPWorkshopsSSG, DHCP, TACACS or Radius Servers/Proxies,DNS resolver, Content 2005, Cisco Systems, Inc. All rights reserved.17Cisco ISPWorkshopsDNSPOP3secondary 2005, Cisco Systems, Inc. All rights reserved.MailRelayNEWSDNScache183
Hosted Services ModuleBorder ModuleTo core routersHosted NetworkGateway RoutersISP1To local IXP NB - no default route local AS routing table onlyISP2NetworkBorder RoutersCustomer 1Customer 3Customer 5Customer 7Customer 2Customer 4Customer 6Cisco ISPWorkshopsTo core routers19 2005, Cisco Systems, Inc. All rights reserved.NOC ModuleCisco ISPWorkshops20 2005, Cisco Systems, Inc. All rights reserved.Out of Band NetworkCritical ServicesModuleTo core routersCorporate LANOut of BandHosted NetworkGateway RoutersManagement NetworkFirewallOut of BandManagement NetworkRouterconsoles2620/32asyncTo the NOC2620/32asyncNetFlowenabledroutersNetFlowNetFlow TACACS SYSLOGAnalyserserverserverPrimary DNSCollectorBilling, Databaseand AccountingSystemsOut of Band EthernetNetwork Operations Centre StaffCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.21Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.22Backbone Design Routed Backbone Switched Backbone Leased point-to-point circuitsBackbone Network Designnx64K, T1/E1, T3/E3, OC3, OC12,. ATM/Frame Relay service from telcoT3, OC3, OC12, deliveryeasily upgradeable bandwidth (CIR)Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.23Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.244
Distributed Network DesignDistributed Network DesignCustomerconnectionsISP Services PoP design “standardised”BackupOperations Centreoperational scalability and simplicity ISP essential services distributed aroundbackbonePOP TwoCustomerconnectionsCustomerconnections NOC and “backup” NOCISP ServicesPOP OnePOP Three Redundant backbone linksISP ServicesCisco ISPWorkshops25 2005, Cisco Systems, Inc. All rights reserved.Backbone LinksCisco ISPWorkshopsExternalconnectionsOperations CentreExternalconnections 2005, Cisco Systems, Inc. All rights reserved.26Long Distance Backbone Links Tend to cost more Plan for the future (at least two years ahead)but stay in budget ATM/Frame Relaynow less popular due to overhead, extra equipment,and shared with other customers of the telcoUnplanned “emergency” upgrades can be disruptivewithout redundancy Leased Line Allow sufficient capacity on alternative pathsfor failure situationsmore popular with backbone providerssufficient can be 20% to 50%IP over Optics and MPLS coming into the mainstreamCisco ISPWorkshops27 2005, Cisco Systems, Inc. All rights reserved.Long Distance LinksPOP TwoCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.28Metropolitan Area Backbone Links Tend to be cheaperLong distance linkCircuit concentrationChoose from multiple suppliers Think bigMore redundancyLess impact of upgradesPOP OnePOP ThreeLess impact of failuresAlternative/Backup PathCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.29Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.305
Metropolitan Area Backbone LinksPOP TwoMetropolitan LinksISP ServicesPOP OnePOP ThreeDNS, Mail, Newsdesign and locationMetropolitan LinksCisco ISPWorkshopsTraditional Point to Point Links 2005, Cisco Systems, Inc. All rights reserved.31ISP Services:DNSCisco ISPWorkshopsISP Services:DNS Domain Name System Primary nameserverHolds ISP zone filesProvides name and address resolutionforward zone (list of name to address mappings) for allISP’s and any customer zonesServers need to be differentiated, properlylocated and specifiedreverse zone (list of address to name mappings) for allISP’s address spacePrimary nameserverCisco ISPWorkshopsSecondary nameserverOne Unix server, fast I/O, reasonable amount ofmemory (512Mbytes), reasonable diskCaching nameserver – resolverLocated in secure part of net, e.g. NOC LAN 2005, Cisco Systems, Inc. All rights reserved.33ISP Services:DNSCisco ISPWorkshops apnic.net zoneprimary DNS in Brisbanesecondary DNS around the worldHolds copies of ISP zone filesAt least two are required, more is betterUnix server, fast I/O, reasonable amount of memory(512Mbytes), reasonable diskShould be geographically separate from each otherand the primary DNSAt different PoPsOn a different continent e.g. www.secondary.comAt another ISP 2005, Cisco Systems, Inc. All rights reserved.34 2005, Cisco Systems, Inc. All rights reserved.ISP Services:Secondary DNS Example Secondary nameserverCisco ISPWorkshops32 2005, Cisco Systems, Inc. All rights reserved.35 dig apnic.net ns;; ANSWER m44s50m44s50m44s50m44s;; ADDITIONAL m16sCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights erdamWashingtonBrisbane366
ISP Services:Secondary DNS ExampleISP Services:DNS apnic.net zone Caching nameserverprimary DNS in Brisbane (ns.apnic.net)This is the resolver – it is the DNS cachesecondary DNS run by APNIC in Tokyo(svc00.apnic.net)Your customers use this as resolver, NOT your primaryor secondary DNSzone secondaried byRIPE NCC in AmsterdamProvides very fast lookupsARIN in WashingtonDoes NOT secondary any zonesOne, or preferably two per PoP (redundancy)Geographical and service provider redundancy – thisis the perfect example!Cisco ISPWorkshopsUnix server, fast I/O, large amount of memory(512Mbytes depending on number of zones)37 2005, Cisco Systems, Inc. All rights reserved.ISP Services:Caching NameserverCisco ISPWorkshops38 2005, Cisco Systems, Inc. All rights reserved.ISP Services:Anycasting the Caching NameserverWeb Cache One trick of the tradeDIAL networkGeekAlertassign two unique IP addresses to befor the two DNS resolver systemsTo Core Routersuse these two IP addresses in every PoProute the two /32s across your backboneSwitch redundancyRouter redundancyDNS Cache redundancyDNS Cacheeven if the two resolver systems in the local PoP aredown, the IGP will ensure that the next nearestresolvers will be reachableRadius proxyDNS CacheKnown as IP AnycastDIAL users automatically given the IP addressesof DNS caches when they dial inCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.39ISP Services:DNSCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.40ISP Services:DNS Efficient and resilient design SoftwarePrimary DNS – keep it secureSecondary DNS – geographical and providerredundancyMake sure that the BIND distribution on the Unix systemis up to dateDon’t ever put them on the same LAN, switched orotherwisethe vendor’s distribution is rarely currentPay attention to bug reports, security issuesDon’t put them in the same PoPReboot the DNS cache on a regular (e.g. monthly) basisCaching DNS – one or two per PoPclears out the cachereduces DNS traffic across backbonereleases any lost RAMmore efficient, spreads the loadCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.accepted good practice by system administrators41Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.427
ISP Services:DNSISP Services:Mail Implementation Must have at least two mail hosts (MX records) forall supported domainsPut all your hosts, point-to-point links and loopbacksinto the DNSgeographical separation helpsunder your ISP’s domain name POP3 server dedicated to that functionuse sensible/meaningful namesDIAL users get mail from herePut all your hosts, point-to-point links and loopbacksinto the REVERSE DNS also SMTP gateway dedicated to that functiondon’t forget about in-addr.arpa – many ISPs doDIAL users send mail via heresome systems demand forward/reverse DNS mappingbefore allowing accessCisco ISPWorkshops Mail relay open to CUSTOMERS only!43 2005, Cisco Systems, Inc. All rights reserved.ISP Services:Mail ExampleCisco ISPWorkshopsISP Services:Mail telstra.net mail (MX records) Softwareprimary MX is mako1backup MX is postoffice – two addressesbackup MX used if primary unavailableMake sure that the MAIL and POP3 distributionson the Unix system are up to datethe vendor’s distribution are rarely currentPay attention to bug reports, security issues,unsolicited junk mail complaints dig telstra.net mx;; ANSWER SECTION:telstra.net.telstra.net.1H IN MX1H IN MX;; ADDITIONAL net.mako1.telstra.net.Cisco ISPWorkshops44 2005, Cisco Systems, Inc. All rights reserved.1H IN A1H IN A1H IN A10 postoffice.telstra.net.5 mako1.telstra.net.IMPORTANT: Do NOT allow non-customersto use your mail system as a relay139.130.4.7203.50.1.76203.50.0.28 2005, Cisco Systems, Inc. All rights reserved.45ISP Services:NewsCisco ISPWorkshops46 2005, Cisco Systems, Inc. All rights reserved.ISP Services:News System PlacementCustomerconnectionsNews Feeder News servers provide a Usenet news feed tocustomersPOP Two Distributed design requiredIncoming newsfeed to one large buted to feed servers in each PoPFeed servers provide news feed to customersPOP ThreePOP OneNews FeederOutgoing news goes to another serverSeparate reading news systemNews FeederSeparate posting news systemExternalconnectionsNews CollectorExternalconnectionsNews DistributorCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.47Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.488
ISP Services:News System PlacementISP Services:NewsCustomerconnectionsNews Feeder SoftwareMake sure that the Internet News distribution onthe Unix system is up to datePOP Twothe vendor’s distribution is rarely currentCustomerconnectionsCustomerconnectionsPOP ThreeNews FeederPOP OneExternalconnectionsPay attention to bug reports, security issues,unsolicited junk posting complaintsNews FeederExternalconnectionsNews CollectorIMPORTANT: Do NOT allow non-customersto use your news system for posting messagesNews DistributorCisco ISPWorkshops49 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.50Where to get IP addresses and AS numbers Your upstream ISP AfricaAfriNIC – http://www.afrinic.net Asia and the PacificAPNIC – http://www.apnic.netAddressing North AmericaARIN – http://www.arin.net Latin America and the CaribbeanLACNIC – http://www.lacnic.net Europe and Middle EastRIPE NCC – http://www.ripe.netCisco ISPWorkshops51 2005, Cisco Systems, Inc. All rights reserved.Internet Registry RegionsCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.52Getting IP address space Take part of upstream ISP’s PA spaceorARIN Become a member of your Regional InternetRegistry and get your own allocationRequire a plan for a year aheadGeneral policies are outlined in RFC2050, more specificdetails are on the individual RIR websiteLACNIC There is plenty of IPv4 address spaceregistries require high quality documentationCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.53Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.549
Addressing Plans – ISP InfrastructureAddressing Plans – Customer Address block for router loop-backinterfaces Customers assigned address spaceaccording to need Address block for infrastructure Should not be reserved or assigned on aper PoP basisper PoP or whole backbonesummarise between sites if it makes senseISP iBGP carries customer netsallocate according to genuine requirements,not historic classful boundariesaggregation not required and usually notdesirableCisco ISPWorkshops55 2005, Cisco Systems, Inc. All rights reserved.Phase One/24Minimum allocation is /21Instrastructure LoopbacksCustomer assignmentsVery likely that subsequent allocation willmake this up to a /20Phase al assignmentsCisco ISPWorkshops56 Registries will usually allocate the nextblock to be contiguous with the firstallocation220.10.0.0/21220.10.6.255 2005, Cisco Systems, Inc. All rights reserved.Addressing PlansPlanningAddressing Plans – ISP Infrastructure220.10.0.1Cisco ISPWorkshopsSo plan accordingly220.10.15.255New Assignments 2005, Cisco Systems, Inc. All rights reserved.57Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.58Addressing Plans (contd) Document infrastructure allocationeases operation, debugging and management Document customer allocationRouting Protocolscontained in iBGPeases operation, debugging and managementsubmit network object to RIR DatabaseCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.59Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.6010
Routing ProtocolsWhy Do We Need an IGP? ISP backbone scaling IGP – Interior Gateway Protocolcarries infrastructure addresses, point-to-point linksHierarchyexamples are OSPF, ISIS, EIGRP.Modular infrastructure construction EGP – Exterior Gateway ProtocolLimiting scope of failurecarries customer prefixes and Internet routesHealing of infrastructure faults using dynamicrouting with fast convergencecurrent EGP is BGP version 4 No link between IGP and EGPCisco ISPWorkshops61 2005, Cisco Systems, Inc. All rights reserved.Why Do We Need an EGP?Cisco ISPWorkshopsInterior versus Exterior Routing Protocols Scaling to large network InteriorHierarchyLimit scope of failure PolicyControl reachability to prefixesMerge separate organizationsConnect multiple IGPsCisco ISPWorkshops63 2005, Cisco Systems, Inc. All rights reserved.Interior versus Exterior Routing Protocols InteriorCarries ISPinfrastructureaddresses onlyISPs aim to keep theIGP small forefficiency andscalabilityCisco ISPWorkshops Exteriorautomatic neighbourdiscoveryspecifically configuredpeersgenerally trust your IGProutersconnecting withoutside networksprefixes go to all IGProutersset administrativeboundariesbinds routers in one AStogetherbinds AS’s togetherHierarchy of Routing ProtocolsOther ISPs ExteriorBGP4Carries customerprefixesCarries Internet prefixesBGP4and OSPF/ISISEGPs are independentof ISP network topologyBGP4 2005, Cisco Systems, Inc. All rights reserved.64 2005, Cisco Systems, Inc. All rights reserved.FDDICisco ISPWorkshops62 2005, Cisco Systems, Inc. All rights reserved.65Cisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.Static/BGP4LocalIXPCustomers6611
Routing Protocols:Choosing an IGPRouting Protocols:IGP Recommendations Keep the IGP routing table as small as possible Review the “Introduction to Link StateProtocols” presentationIf you can count the routers and the point to point linksin the backbone, that total is the number of IGP entriesyou should seei.e. – OSPF and ISIS have very similar properties IGP details: ISP usually chooses between OSPF and ISISChoose which is appropriate for your operators’experienceShould only have router loopbacks, backbone WANpoint-to-point link addresses, and network addressesof any LANs having an IGP running on themIn IOS, both OSPF and ISIS have sufficient “nerdknobs” to tweak the IGP’s behaviourStrongly recommended to use inter-routerauthenticationUse inter-area summarisation if possibleCisco ISPWorkshops 2005, Cisco Systems, Inc. All rights reserved.67Routing Protocols:More IGP recommendationsCisco ISPWorkshops iBGP should carry everything whichdoesn’t contribute to the IGP routingprocessUsing “ip unnumbered” on customer point-to-pointlinks – saves carrying that /30 in IGP(If customer point-to-point /30 is required formonitoring purposes, then put this in iBGP)Internet routing tableUse contiguous addresses for backbone WAN links ineach area – can then summarise into backbone areaCustomer assigned addressesDon’t summar
ISP Network Design ISP/IXP Workshops 205 ,C i scoSytem In .Al rght vd 2 CiscoISP Workshops ISP Network Design PoP Topologies and Design Backbone Design ISP Systems Design Addressing Routing Protocols Security Out of Band Management Operational Considerations 205 ,C i scoSytem In .Al rght vd 3 CiscoISP Workshops Point .
