Transcription
The Challenges of Scaling WISPsShaddi HasanYahel Ben-DavidMax BittmanBarath RaghavanUC Berkeleyshaddi@cs.berkeley.eduDe Novo/UC Berkeleyyahel@eecs.berkeley.eduSudo BSTRACTlow-cost wireless equipment to build networks in areas often lacking any other Internet infrastructure. These WISPs are small network operators, leveraging low-cost fixed wireless infrastructure tobuild their networks.While Internet access, speeds, and the diversity of applicationsavailable via that access has, in densely populated areas around theworld, rapidly increased in the past decade, rural access has notimproved at the same rate. Speeds delivered by many rural WISPshave not scaled with the increasing bandwidth needs of modernapplications; some WISPs we became familiar with in the courseof this work have neither substantially increased the bandwidth theydeliver to subscribers nor decreased prices in a decade.In this paper, we set out to understand why this situation exists,drawing on our own experiences and those of others, and to use thatunderstanding to develop new systems to help WISPs operate moreeffectively. We contribute three insights into the problems faced indeveloping rural Internet access. First, we describe the context ofrural Internet access through two studies of WISPs that we conducted. Second, we detail how assumptions about scaling networksand growing Internet access must be rethought to address the problems that WISPs face. Third, we describe a new software system,Celerate, that we are building to meet the needs of new WISP operators; we have actively been using this system for over a year tomanage a new WISP we have built in a rural area of North America.To calibrate our thinking on rural Internet access in generaland WISPs in particular, we conducted two studies on WISPs, collectively representing responses from 83 WISPs operating acrossNorth America. We found that North American WISPs ability tosucceed was driven largely by financial, regulatory, and businessfactors, rather than shortcomings in network-level tools.The results of these studies guide our work. First, our findingssuggest that there may be natural limits to the size of rural WISPs,and thus the approach to scaling Internet access for the last billionusers will require scaling not merely the size of networks but thenumber of distinct WISP operators. To do so, we concluded that anew architecture for WISP operation is needed to help these newoperators build and manage their networks, thereby lowering thebarrier for starting and sustaining a WISP. Second, at the outset ofour work, we hypothesized that SDN, in some form, could helpWISPs and thus help address the problems of rural Internet access.In concept SDN has much to offer to simplify network managementand implement complex network policies, but in practice we foundmore mundane functionality, such as integrating billing with subscriber data, yields more immediate benefits for WISPs. Thus, ourdevelopment effort focuses on mundane but ultimately more useful management functionality at the expense of sophisticated butunnecessary SDN functionality.Wireless ISPs (WISPs) are one of the primary means of delivering broadband Internet access to rural and underserved areas ofthe world. However, WISP operators often struggle to maintain letalone grow their operations. We set out both to understand whatchallenges WISP operators face and to develop new approaches tohelp them in managing their networks. First, we present a studywe conducted of operating WISPs to understand what barriers theyface. Second, we describe our experiences building and using asoftware system for WISP management, and a new WISP we havebuilt from the ground up using that system. We found that WISPsappear to reach natural scaling limits, and that despite excitement inthe networking community about the promise of Software DefinedNetworking (SDN) in new environments, more mundane functionality like subscriber management provides much of the actual benefit to WISPs.Categories and Subject DescriptorsC.2.3 [Computer Communication Networks]: Network managementGeneral TermsDesign, Economics, ManagementKeywordsWireless, Rural, Network Operations1.INTRODUCTIONOver the last few years, connecting the vast globally disconnected population of potential users—the next billion—to the Internet has attracted significant attention both in the research community (e.g. via GAIA) and among large industrial players suchas Google and Facebook. Rural users are most difficult to connectdue to lower subscriber density making profitable operation challenging. The task of connecting these last billions of rural users hasbeen taken up by small Wireless ISPs (WISPs), taking advantage ofPermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.ACM DEV 2015, December 1–2, 2015, London, United Kingdom.c 2015 ACM. ISBN 978-1-4503-3490-7/15/11 . 15.00.DOI: http://dx.doi.org/10.1145/2830629.2830637.3
SizeNumberPercentileTo validate our findings and to have a testbed for our workon our new architecture, we built and continue to grow an operational WISP to provide broadband Internet to a rural communityand indigenous tribe in North America. We describe our experiences in building the network and how we have leveraged it in testing new approaches to be replicated at WISPs elsewhere. To manage our WISP network, we are building Celerate, an architecture forWISP network management. Our aim is to provide a structure forall aspects of starting, operating, and growing a WISP in rural areas. Some of the components of Celerate build upon conventionalSDN designs, while some relate to aspects that are either uniqueto WISPs or are simply ignored in traditional SDN deployments.Architecturally Celerate differs from many SDN systems in that itmakes explicit two additional planes—the management subsystemand the operations subsystem—that are typically viewed as outsidethe scope of the network’s design. These planes are essential tomanaging many networks, especially WISPs, and by making themexplicit we clearly define the ways in which the human operatorsinteract with the system. Our aim is to create a system that meetsthe needs of small WISPs anywhere in the world, and as such wehave incorporated the feedback of numerous WISPs into our designs. Our system is open source and is in the early stages of usebeyond our primary deployment in a partner WISP who plans toindependently leverage it 293.1% 50005100%Table 1: WISP size (subscriber base) demographics.After completing the survey, participants were invited to participatein a follow-up semi-structured phone interview. We had a total of75 responses to our survey; 13 of those participated in a follow-upinterview. Twelve of those interviewed were active WISP operators,and of those ten operated networks in rural areas.We recruited participants via convenience sampling by distributing announcements on three WISP-focused email lists: theWireless Internet Service Providers Association’s (WISPA) public and members-only lists and the Animal Farm Microwave UsersGroup (AFMUG) list.1 The WISPA and AFMUG mailing lists havesignificant overlap, though the latter focuses primarily on users ofa particular manufacturer’s equipment (Cambium).The participation rate is difficult to calculate since the membership lists of each list are private. Based on public archiverecords, 434 unique users posted to the WISPA mailing list in thetwo years preceding our study; of course, this only captures activelist participants on a single mailing list (though likely the largest ofthe three we contacted). The AFMUG mailing list claims that “listmembership exceeds 450 members.” Nevertheless, our respondentpool represents a wide cross-section of the WISP industry. The vastmajority of our survey respondents were involved in the day-to-dayoperation of a WISP. Other respondents had operated WISPs in thepast, but now served primarily in a management role, often as aresult of growing their company through acquisition.Our second study, conducted in 2014, was a more informalsurvey of WISPs in the neighboring regions to our target deployment. We talked to 8 WISPs in the region; for all of these, wedirectly contacted each and spoke with them about their currentoperations and their potential interest in using Celerate during development. We did not interview any of these WISPs in our 2012study, though we have no way of knowing if they participated inthe web-based survey component.THE CHALLENGES OF WISPSIn mid-2012 we conducted a systematic survey of WISPsthroughout the United States, and in 2014 conducted a morefocused but informal survey of WISPs in a specific region inNorth America. WISPs provide service to millions of subscribersthroughout the US. Those who are unfamiliar with the WISP industry are often surprised by its size: over two million subscribersare served by WISPs in the US alone, and in rural areas, WISPscan be the only source of broadband Internet access besides satellite [3]. WISPs have an even larger impact outside the US. Ubiquiti,a leading hardware vendor for WISPs, reports that “the substantialmajority of [their] sales occur” outside the US, and sees emerging markets as a major opportunity for their growth [28]. Thesefacts are unsurprising when one considers the fundamental reduction in capital expenditure required to build a wireless ISP networkcompared to a traditional wired one. Falling costs and rising performance of commodity wireless equipment, driven by the popularity of WiFi, have allowed the industry to grow as availabilityof unlicensed spectrum has increased globally. Early research fromthe academic community demonstrated that the same chipsets usedin laptops and phones could be used to build long-distance WiFilinks [17]; since then, similar technologies have been commercialized and are widely available. Radio equipment for a 50km linkproviding more than 50Mbps of throughput can be had for under 200, with each radio consuming under 5W.Despite the importance of WISPs for providing economicalbroadband service to rural areas, there has been little study by theacademic community of the WISP industry. We set out to rectifythat situation by answering the following questions: a) what are thedemographics of rural WISPs, b) what are the key operational challenges WISPs face, and c) what policy support do WISPs requireto effectively provide service?2.11–9945.5%2.2DemographicsThe WISPs we surveyed were small, by almost any metricone considers. While only 5% of those surveyed had fewer than100 customers, half had less than 1000, and almost all of those surveyed had fewer than 5000 customers (Table 1). In terms of trafficload, 40% of WISPs surveyed saw a peak traffic demand of under100Mbps, and 80% had a peak demand of under 500Mbps.One of the most consistent and striking characteristics of manyof our survey respondents was that they often played multiple roleswithin their WISP’s operations. Most survey respondents reportedthat they filled a combination of business management, technicalmanagement, and marketing roles. This is unsurprising, given almost half of respondents had fewer than 5 employees; 90% hadless than 25 employees. For the smallest WISPs we talked to, onlyone or two people were responsible for the entire operation, thoughhiring part-time or contract workers for specialized tasks such astower climbing was common.2.3FindingsThe goal of this study was to develop an understanding of thetechnical challenges that impacted WISPs, with the hopes of motivating further research on systems to help WISPs operate more efficiently. After collecting data and conducting interviews, we cameMethodologyTo investigate the operation and characteristics of WISPs inour initial 2012 study, we developed a web-based survey. The survey consisted of 20 questions covering the size of the WISP andits network, budgeting, network failures, and network management.14WISPA is the industry association for WISPs in the United States.
to realize that some of the assumptions we made were misguided.We initially expected WISPs in the US to face struggles similarto those described in the literature about wireless networks in thedeveloping world [25]: flaky hardware, challenging fault diagnosis, and poor local IT expertise. This turned out not to be the case.In contrast to the “hacked together” wireless systems of the mid2000s, technology, especially commodity wireless hardware, usedby WISPs for building their networks has matured sufficiently thatoperators focus more of their effort on business development thantechnical issues. One WISP we spoke with in Colorado providedservice over a 45,000 sq. mi. area with only 10 employees.A common concern among WISPs in our study was spectrum scarcity. Of 43 respondents to the free-response question inour web survey “What is the biggest challenge your organizationfaces?”, 22 (51%) expressed concerns relating to spectrum. Thenext most common group of concerns was around business development (23%), followed by affordability of upstream bandwidthand backhaul (16%). This was an unexpected result: we had anticipated issues around configuration and manageability of WISPnetworks to be a major concern, but this was not the case.Spectrum. We asked about spectrum usage in our interviews. Allof the rural WISP operators we interviewed operated in unlicensedspectrum, but many used some licensed spectrum as well. In particular, 7 of the 10 rural WISPs used the 3650MHz “lightly licensed”band, which has been very popular for WISP operators due to therelative quiet of the band compared to unlicensed ones. Multipleinterview subjects expressed a desire to have spectrum set aside forWISPs due to overcrowding in the unlicensed bands. Accordingto one operator, “basically, we need to use lots of bands becausethings are so crowded. [The 3.65GHz band] will never have homerouters in it. So we can use, especially for backhaul, a relativelyobscure chunk of spectrum.” Higher frequency licensed spectrum(specifically, the 11GHz band) was used in some capacity by threeof the respondents, primarily for high-capacity backhaul links.On the surface, it surprising that spectrum scarcity would bean issue in areas that are largely underserved, but several factorsmake this an issue for WISPs. The WISPs we spoke with all hada limited number of tower sites for access points to connect customers to their networks. To reduce capital cost of expansion, theseWISPs would take advantage of geographic features, re-use existing towers, or re-purpose other tall structures (e.g., grain silos) toavoid building new towers from scratch. Adding more subscribersthus meant co-locating more access points on each tower, leadingto interference at the tower site. The second driver was foliage—WISPs serving forested areas reported heavy usage of 900MHzspectrum due to improved foliage penetration. The 900MHz bandis the lowest-frequency band commonly available to WISPs, andis only 28MHz wide (compared to over 150MHz for the morecommonly-used 5GHz band) yet is shared with a variety of nonWISP users, such as cordless phone systems and smart meters.Business development and Financing. An unexpected theme thatemerged was the difficulty of obtaining financing to expand network growth and meeting demand for service. Particularly for rural WISPs, the cost of adding a customer to the network is high,with an installation requiring a site survey, a trained technician,customer premises equipment, and physical installation of the hardware. Specifically, in our 2014 survey, we found that WISP growthwas often constrained by financing once the natural (often geographic, though sometimes market, cultural, or political) boundaries of the WISP’s growth had been reached, as the cost to growbeyond the boundary was a step function. Indeed, we found thatthis natural boundary to WISP growth seemed to have the sameorigin for many WISPs even if it manifested differently. For someWISPs, it appeared as though the WISP operators no longer had aninterest in growing their network and reaching new users, but whenpressed, this was because the cost (in time, effort, and money) wastoo great to take a step beyond the current size. For other WISPsthis boundary manifested as a more straightforward financial limitto expansion—while the area in which they operated may have beenprofitable enough to sustain their current size, it was not able to produce enough profit to finance expansion into new markets.A humorous but sobering comment from one WISP was thatthey switched to using minivans instead of trucks for service callsdue to better fuel economy, yet still spend 2 per user per monthon fuel. In an industry with an estimated ARPU of 30 [3], thisrepresents almost 7% of gross revenue. One of our interview subjects stated that their ability to buy used equipment from a bankruptcompetitor at a fraction of retail price was instrumental in allowingthem to grow their revenue to sustainability.Another unexpected theme in this area was the relationshipsamong neighboring WISPs. Several of the WISPs we spoke withhad cooperative, often informal, relationships with neighboringWISPs. These relationships included infrastructure and backhaulsharing, referring customers near the edge of one’s service areato competitors, agreements to not expand into each other’s serviceareas, and in one case even a co-op of several WISPs that madebulk equipment purchases and shared a customer support call center. Another common practice was WISPs buying out neighboringWISPs; three WISPs we spoke with reported having done this.2.4DiscussionMost WISPs we spoke with did not need network management tools to grow larger; their challenges were based on financialor regulatory issues. We argue that this is because they have typically already matured to natural size limits (or have failed). Atthose limits, even if the network is profitable it is likely unable toexpand, constrained by inefficient network management and business processes, and unable to finance improving network performance. The high rate of startup failure and the low performanceoffered by many WISPs suggests that tools for facilitating the creation of new high-performance WISPs would be valuable. One ofour 2012 interview subjects who both runs a WISP and consults fornew WISPs went so far as to say, “I tell a lot of people that [running a WISP] becomes a lifestyle until you have people, becauseyou have to babysit the network 24/7.” Their advice to new WISPsstarting out was to focus on automating and integrating as much oftheir backend processes (billing, subscriber authentication, etc) aspossible to improve the odds the WISP would be able to sustain itself and grow. This insight was reflected throughout our interviews,would be reflected in our own experiences starting a WISP, and succinctly motivates the design of Celerate.3.ASSUMPTIONSWith these results in mind, we now turn to the design of Celerate. We frame the discussion with a collection of assumptions thatwe brought to the problem. These assumptions are instructive: ourexploration of them over the past decade has led us to hone the listof challenges to be addressed.Target Rural Access. In urban areas density and existinginfrastructure—e.g., right of ways for electricity, telephony, orsewage—make building infrastructure for Internet service delivery easier. Even where existing infrastructu
the scope of the network’s design. These planes are essential to managing many networks, especially WISPs, and by making them . tion in capital expenditure required to build a wireless ISP network compared to a traditional wired one. Falling costs and rising per- . its network, budgeting, network failures, and network management.
