Internet Exchange Point Design - PacNOG PDF Free Download

2y ago

34 Views

1 Downloads

936.02 KB

81 Pages

Report/dmca

Download PDF

Transcription

IXP Design Background Why set up an IXP? Layer 2 Exchange Point Layer 3 Exchange Point Design Considerations Route Collectors & Servers What can go wrong?ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.2

A Bit of History End of NSFnet – one major backbone move towards commercial Internetprivate companies selling their bandwidth need for coordination of routing exchange betweenprovidersTraffic from ISP A needs to get to ISP B Routing Arbiter project created to facilitate thisISP Workshops 2008 Cisco Systems, Inc. All rights reserved.4

What is an Exchange Point Network Access Points (NAPs) established at end ofNSFnetThe original “exchange points” Major providers connect their networks and exchangetraffic High-speed network or ethernet switch Simple concept – any place where providers cometogether to exchange trafficISP Workshops 2008 Cisco Systems, Inc. All rights reserved.5

Internet Exchange Points Layer 2 exchange pointEthernet (1000/100Mbps)Older technologies include ATM, Frame Relay, SRP, FDDI andSMDS Layer 3 exchange pointRouter basedHistorical statusISP Workshops 2008 Cisco Systems, Inc. All rights reserved.6

Internet Exchange PointWhy peer? Consider a region with one ISPThey provide internet connectivity to their customersThey have one or two international connections Internet grows, another ISP sets up in competitionThey provide internet connectivity to their customersThey have one or two international connections How does traffic from customer of one ISP get tocustomer of the other ISP?Via the international connectionsISP Workshops 2008 Cisco Systems, Inc. All rights reserved.8

Internet Exchange PointWhy peer? Yes, International Connections If satellite, RTT is around 550ms per hopSo local traffic takes over 1s round trip International bandwidthCosts significantly more than domestic bandwidthCongested with local trafficWastes money, harms performanceISP Workshops 2008 Cisco Systems, Inc. All rights reserved.9

Internet Exchange PointWhy peer? Solution:Two competing ISPs peer with each other Result:Both save moneyLocal traffic stays localBetter network performance, better QoS, More international bandwidth for expensive international trafficEveryone is happyISP Workshops 2008 Cisco Systems, Inc. All rights reserved.10

Internet Exchange PointWhy peer? A third ISP enters the equationBecomes a significant player in the regionLocal and international traffic goes over their internationalconnections They agree to peer with the two other ISPsTo save moneyTo keep local traffic localTo improve network performance, QoS, ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.11

Internet Exchange PointWhy peer? Peering means that the three ISPs have to buy circuitsbetween each otherWorks for three ISPs, but adding a fourth or a fifth means thisdoes not scale Solution:Internet Exchange PointISP Workshops 2008 Cisco Systems, Inc. All rights reserved.12

Internet Exchange Point Every participant has to buy just one whole circuitFrom their premises to the IXP Rather than N-1 half circuits to connect to the N-1 otherISPs5 ISPs have to buy 4 half circuits 2 whole circuits alreadytwice the cost of the IXP connectionISP Workshops 2008 Cisco Systems, Inc. All rights reserved.13

Internet Exchange Point SolutionEvery ISP participates in the IXPCost is minimal – one local circuit covers all domestic trafficInternational circuits are used for just international traffic – andbacking up domestic links in case the IXP fails Result:Local traffic stays localQoS considerations for local traffic is not an issueRTTs are typically sub 10msCustomers enjoy the Internet experienceLocal Internet economy grows rapidlyISP Workshops 2008 Cisco Systems, Inc. All rights reserved.14

Layer 2 ExchangeISP 6ISP 5ISP 4IXP Services:TLD DNS,Routing RegistryEthernet SwitchLooking Glass,IXPManagementNetworknews, etcISP 1ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.ISP 2ISP 316

Layer 2 ExchangeISP 6ISP 5ISP 4IXP Services:IXPManagementNetworkTLD DNS,Routing RegistryLooking Glass,Ethernet Switchesnews, etcISP 1ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.ISP 2ISP 317

Layer 2 Exchange Two switches for redundancy ISPs use dual routers for redundancy or loadsharing Offer services for the “common good”Internet portals and search enginesDNS TLD, News, NTP serversRouting Registry and Looking GlassISP Workshops 2008 Cisco Systems, Inc. All rights reserved.18

Layer 2 Exchange Requires neutral IXP managementusually funded equally by IXP participants24x7 cover, support, value add services Secure and neutral location Configurationprivate address space if non-transit and no value add servicesISPs require AS, basic IXP does notISP Workshops 2008 Cisco Systems, Inc. All rights reserved.19

Layer 2 Exchange Network Security ConsiderationsLAN switch needs to be securely configuredManagement routers require TACACS authentication, vtysecurityIXP services must be behind router(s) with strong filtersISP Workshops 2008 Cisco Systems, Inc. All rights reserved.20

Layer 3 Exchange/Wholesale Transit ISPISP 6ISP 5ISP 4IXP Services:TLD DNS,Routing RegistryIXP RouterLooking Glass,IXPManagementNetworknews, etcISP 1ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.ISP 2ISP 322

Layer 3 Exchange/Wholesale Transit ISPISP 6ISP 5ISP 4IXP Services:IXPManagementNetworkTLD DNS,Routing RegistryIXP RoutersLooking Glass,news, etcISP 1ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.ISP 2ISP 323

Layer 3 Exchange/Wholesale Transit ISP Two routers for redundancy ISPs use dual routers for redundancy or loadsharing Offer services for the “common good”Internet portals and search enginesDNS TLD, News, NTP serversRouting Registry and Looking GlassISP Workshops 2008 Cisco Systems, Inc. All rights reserved.24

Layer 3 Exchange/Wholesale Transit ISP Requires neutral managementUsually funded equally by participants24x7 cover, support, value add servicesBGP configuration skills essential Secure and neutral location Configurationprivate address space if non-transit and no value add servicesISPs and IXP require ASISP Workshops 2008 Cisco Systems, Inc. All rights reserved.25

Layer 3 Exchange/Wholesale Transit ISP Network Security ConsiderationsCore IXP router(s) require strong security, preferably with BGPneighbour authenticationManagement routers require TACACS authentication, vtysecurityIXP services must be behind router(s) with strong filtersISP Workshops 2008 Cisco Systems, Inc. All rights reserved.26

Transit IXPs/Wholesale Transit ISP Provides local Internet exchange facility to members Also provides transit to Internet or upstream ISP Usually operated as a commercial service Usually layer 3 designISP Workshops 2008 Cisco Systems, Inc. All rights reserved.27

Layer 3 Transit Exchange/Transit ISPISP 5ISP 4Transit RoutersInternetISPIXP Services:IXPManagementNetworkTLD DNS,Routing RegistryIXP RoutersLooking Glass,news, etcISP 1ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.ISP 2ISP 328

Layer 2 versus Layer 3 Layer 3IXP team requires good BGP knowledgeRely on 3rd party for BGP configurationLess freedom on who peers with whomUsually competes with IXP membershipTends to be distributed over wide areaISP Workshops 2008 Cisco Systems, Inc. All rights reserved.29

Layer 2 versus Layer 3 Layer 2IXP team does not need routing knowledgeEasy to get startedMore complicated to distribute over wide areaISPs free to set up peering agreements with each other as theywishISP Workshops 2008 Cisco Systems, Inc. All rights reserved.30

Exchange Point Design The IXP Core is an Ethernet switch Has superseded all other types of network devices foran IXPFrom the cheapest and smallest 12 or 24 port 10/100 switchTo the largest 32 port 10GigEthernet switchISP Workshops 2008 Cisco Systems, Inc. All rights reserved.33

Exchange Point Design Each ISP participating in the IXP brings a router to theIXP location Router needs:One Ethernet port to connect to IXP switchOne WAN port to connect to the WAN media leading back tothe ISP backboneTo be able to run BGPISP Workshops 2008 Cisco Systems, Inc. All rights reserved.34

Exchange Point Design IXP switch located in one equipment rack dedicated toIXPAlso includes other IXP operational equipment Routers from participant ISPs located inneighbouring/adjacent rack(s) Copper (UTP) connections made for 10Mbps, 100Mbpsor 1Gbps connections Fibre used for 10Gbps and 40GbpsISP Workshops 2008 Cisco Systems, Inc. All rights reserved.35

Peering Each participant needs to run BGPThey need their own AS numberPublic ASN, NOT private ASN Each participant configures external BGP directly withthe other participants in the IXPPeering with all participantsorPeering with a subset of participantsISP Workshops 2008 Cisco Systems, Inc. All rights reserved.36

Peering (more) Mandatory Multi-Lateral Peering (MMLP)Each participant is required to peer with every other participantas part of their IXP membershipHas no history of success — the practice is discouraged Multi-Lateral Peering (MLP)Each participant peers with every other participant Bi-Lateral PeeringParticipants set up peering with each other according to theirown requirements and business relationshipsThis is the most common situation at IXPs todayISP Workshops 2008 Cisco Systems, Inc. All rights reserved.37

Routing ISP border routers at the IXP generally should NOT beconfigured with a default route or carry the full Internetrouting tableCarrying default or full table means that this router and the ISPnetwork is open to abuse by non-peering IXP membersCorrect configuration is only to carry routes offered to IXP peerson the IXP peering router Note: Some ISPs offer transit across IX fabricsThey do so at their own risk – see aboveISP Workshops 2008 Cisco Systems, Inc. All rights reserved.38

Routing (more) ISP border routers at the IXP should not be configuredto carry the IXP LAN network within the IGP or iBGPUse next-hop-self BGP concept Don’t generate ISP prefix aggregates on IXP peeringrouterIf connection from backbone to IXP router goes down, normalBGP failover will then be successfulISP Workshops 2008 Cisco Systems, Inc. All rights reserved.39

Address Space Some IXPs use private addresses for the IX LANPublic address space means IXP network could be leaked toInternet which may be undesirableBecause most ISPs filter RFC1918 address space, this avoidsthe problem Some IXPs use public addresses for the IX LANAddress space available from the RIRsIXP terms of participation often forbid the IX LAN to be carriedin the ISP member backboneISP Workshops 2008 Cisco Systems, Inc. All rights reserved.40

Hardware Try not to mix port speedsif 10Mbps and 100Mbps connections available, terminate ondifferent switches (L2 IXP) Don’t mix transportsif terminating ATM PVCs and G/F/Ethernet, terminate ondifferent devices Insist that IXP participants bring their own routermoves buffering problem off the IXPsecurity is responsibility of the ISP, not the IXPISP Workshops 2008 Cisco Systems, Inc. All rights reserved.41

Services Offered Services offered should not compete with member ISPs(basic IXP)e.g. web hosting at an IXP is a bad idea unless all membersagree to it IXP operations should make performance andthroughput statistics available to membersUse tools such as MRTG to produce IX throughput graphs formember (or public) informationISP Workshops 2008 Cisco Systems, Inc. All rights reserved.42

Services to Offer ccTLD DNSthe country IXP could host the country’s top level DNSe.g. “SE.” TLD is hosted at Netnod IXes in SwedenOffer back up of other country ccTLD DNS Root serverAnycast instances of I.root-servers.net, F.root-servers.net etcare present at many IXes Usenet NewsUsenet News is high volumecould save bandwidth to all IXP membersISP Workshops 2008 Cisco Systems, Inc. All rights reserved.43

Services to Offer Route CollectorRoute collector shows the reachability information available atthe exchangeTechnical detail covered later on Looking GlassOne way of making the Route Collector routes available forglobal view (e.g. www.traceroute.org)Public or members only accessISP Workshops 2008 Cisco Systems, Inc. All rights reserved.44

Services to Offer Content Redistribution/CachingFor example, Akamised update distribution service Network Time ProtocolLocate a stratum 1 time source (GPS receiver, atomic clock,etc) at IXP Routing RegistryUsed to register the routing policy of the IXP membership (morelater)ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.45

What is a Route Collector? Usually a router or Unix system running BGP Gathers routing information from service providerrouters at an IXPPeers with each ISP using BGP Does not forward packets Does not announce any prefixes to ISPsISP Workshops 2008 Cisco Systems, Inc. All rights reserved.47

Purpose of a Route Collector To provide a public view of the Routing Informationavailable at the IXPUseful for existing members to check functionality of BGP filtersUseful for prospective members to check value of joining theIXPUseful for the Internet Operations community fortroubleshooting purposesE.g. www.traceroute.orgISP Workshops 2008 Cisco Systems, Inc. All rights reserved.48

Route Collector Requirements Router or Unix system running BGPMinimal memory requirements – only holds IXP routesMinimal packet forwarding requirements – doesn’t forward anypackets Peers eBGP with every IXP memberAccepts everything; Gives nothingUses a private ASNConnects to IXP Transit LAN “Back end” connectionSecond Ethernet globally routedConnection to IXP Website for public accessISP Workshops 2008 Cisco Systems, Inc. All rights reserved.50

Route Collector Implementation Most IXPs now implement some form of RouteCollector Benefits already mentioned Great public relations tool Unsophisticated requirementsJust runs BGPISP Workshops 2008 Cisco Systems, Inc. All rights reserved.51

What is a Route Server? Has all the features of a Route Collector But also:Announces routes to participating IXP members according totheir routing policy definitions Implemented using the same specification as for aRoute CollectorISP Workshops 2008 Cisco Systems, Inc. All rights reserved.53

Features of a Route Server Helps scale routing for large IXPs Simplifies Routing Processes on ISP Routers Optional participationProvided as service, is NOT mandatory Does result in insertion of RS Autonomous SystemNumber in the Routing Path Optionally uses Policy registered in IRRISP Workshops 2008 Cisco Systems, Inc. All rights reserved.54

Diagram of N-squared Peering Mesh For large IXPs (dozens for participants) maintaining alarger peering mesh becomes cumbersome and oftentoo hardISP Workshops 2008 Cisco Systems, Inc. All rights reserved.55

Advantages of Using a Route Server Helps scale Routing for very large IXPs Separation of Routing and Forwarding Simplify Routing Configuration Management on ISPsroutersISP Workshops 2008 Cisco Systems, Inc. All rights reserved.58

Disadvantages of using a Route Server ISPs can lose direct policy controlIf RS is only peer, ISPs have no control over who their prefixesare distributed to Completely dependent on 3rd partyConfiguration, troubleshooting, etc Insertion of RS ASN into routing pathTraffic engineering/multihoming needs more care These are major disadvantagesUsually out-weigh the advantagesISP Workshops 2008 Cisco Systems, Inc. All rights reserved.59

Typical usage of a Route Server Route Servers may be provided as an OPTIONALserviceMost common at large IXPs ( 50 participants)Examples: TorIX, AMS-IX, etc ISPs peer:Directly with significant peersWith Route Server for the restISP Workshops 2008 Cisco Systems, Inc. All rights reserved.60

Things to think about. Would using a route server benefit you?Helpful when BGP knowledge is limited (but is NOT an excusenot to learn BGP)Avoids having to maintain a large number of eBGP peersBut can you afford to lose policy control? (An ISP not in controlof their routing policy is what?)ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.61

What can go wrong?Concept Some Service Providers attempt to cash in on thereputation of IXPs Market Internet transit services as “Internet ExchangePoint”“We are exchanging packets with other ISPs, so we are anInternet Exchange Point!”So-called Layer-3 Exchanges — really Internet TransitProvidersRouter used rather than a SwitchMost famous example: SingTelIXISP Workshops 2008 Cisco Systems, Inc. All rights reserved.63

What can go wrong?Competition Too many exchange points in one localeCompeting exchanges defeats the purpose Becomes expensive for ISPs to connect to all of them An IXP:is NOT a competitionis NOT a profit making businessISP Workshops 2008 Cisco Systems, Inc. All rights reserved.64

What can go wrong?Rules and Restrictions IXPs try to compete with their membershipOffering services that ISPs would/do offer their customers IXPs run as a closed privileged club e.g.:Restrictive membership criteria (closed shop) IXPs providing access to end users rather than justService Providers IXPs interfering with ISP business decisions e.g.Mandatory Multi-Lateral PeeringISP Workshops 2008 Cisco Systems, Inc. All rights reserved.65

What can go wrong?Technical Design Errors Interconnected IXPsIXP in one location believes it should connect directly to the IXPin another locationWho pays for the interconnect?How is traffic metered?Competes with the ISPs who already provide transit betweenthe two locations (who then refuse to join IX, harming theviability of the IX)Metro interconnections work ok (e.g. LINX)ISP Workshops 2008 Cisco Systems, Inc. All rights reserved.66

What can go wrong?Technical Design Errors ISPs bridge the IXP LAN back to their offices“We are poor, we can’t afford a router”Financial benefits of connecting to an IXP far outweigh the costof a routerIn reality it allows the

ISP border routers at the IXP generally should NOT be configured with a default route or carry the full Internet routing table Carrying default or full table means that this router and the ISP network is open to abuse by non-peering IXP members Correct configuration is only to carry routes offered to IXP peers on the IXP peering router