WIXLIB

SYSADMINZeus Load BalancerExploring the Linux-based Zeus load balancerOLYMPIANOn today’s networks, distributing requests in a cluster of web servers requires more than just assigning therequests in a round robin. The Zeus ZXTM 7400 appliance demonstrates the technical finesse necessary tokeep busy websites running. BY JÖRG FRITSCHEven the most powerful webserver eventually reaches itslimit. Getting help isn’t a problem; the load redistribution can be complicated. Each server cluster needs todistribute requests intelligently to useresources in a meaningful way, and theclient should not notice what is going onbehind the scenes. One way of achievingthis is the ZXTM 7400 appliance by ZeusTechnology [1], which I recently tested.TechniquesTHE AUTHORLoad balancers distinguish betweenphysical servers and virtual IPs (VIPs).In this case, the physical servers are web68Jörg Fritsch has a degree in chemistry and works in the field of softwaredevelopment and IT security. He hasworked in his current position as theEngineer for Communication andInformation Security for the Nato C3Agency since 2003. Jörg has alsopublished a variety of work on loadbalancing, TCP/IP, and security.ISSUE 80 JULY 2007servers. Each web server has a unique IP(real IP, or RIP). The VIP is only configured on the load balancer.Web clients only see the IP address belonging to the website, and they connectto this address without realizing they’retalking to a load balancer that uses ascheduling algorithm to assign clientrequests to servers.A fairly simple implementation of thisprinciple works like destination networkaddress translation (NAT), modifying thetarget address in the request (from VIPto RIP) and modifying responses tomatch (from RIP to VIP).This variant is typically found in application-specific integrated circuit (ASIC)based devices, which use highly specialized hardware to manipulate packets atextremely high speeds. In the early daysof web load balancing (see the box titled“Seven Years of Load Balancing”), ASICswere thought to virtually guarantee sufficient performance.If you need more performance, algorithms quickly become too complex forW W W. L I N U X - M A G A Z I N E . C O MASICs. Fortunately, the CPU performance of today’s server and PC hardware is sufficient to cope with demanding tasks, as the appliance that I testedZeus ZXTM 7400Task: Load balancing and applicationoptimization for web applicationsTechnology: PC-based appliance withLinux and proprietary software by ZeusVersion tested: 4.1r1 on ZXTM appliance 7400Price: Starts at around EUR 15,000 (US 20,000) for the entry-level device, ZXTM2000 LB, up to around EUR 57,000 (US 78,000) for the high-end ZXTM 7400 appliance including all software optionsfor the device tested here); the softwarewithout appliance costs between EUR5,500 and 28,400 (US 7,500 and39,600).

Zeus Load BalancerSYSADMINjust goes to prove. The ZXTM 7400works more like a reverse proxy thatdoesn’t convert IP addresses but,instead, terminates the client’s TCPconnection and opens up a new TCPconnection to the physical server.This approach helps the applianceretain control over the connection,with the ability to manipulate the datastream. The physical server sees the loadbalancer’s source IP address and sendsthe response to the appliance, which inturn sends it to the client.Single Process Per CoreTo keep pace with ASICs, PCs need someclever programming. The UK-basedmanufacturer Zeus has put much effortinto developing optimized network software, thanks to its own web server.On this basis, programmers deducedthat a legacy multi-processing or multithreading model would be insufficientbecause it would lose too much time oncontext changes.The Zeus approach also reflects DanKegel’s recommendations [2] for fastnetwork software. The program uses theepoll mechanism to search for data in allopen connections without blocking andwithout context changes. The developershave used nonblocking functions for allprocessing steps.Figure 1: It is easy to create a simple pool with three physical web servers in the Zeus GUI.The load balancer selects one of these servers for each request.To do so, the appliance analyzes theXML content of the messages.ZXTM 7400tings (network, default gateway). Because the appliance runs on Linux (amodified Debian Sarge with an Ubuntukernel), Linux experts will soon feel athome at the command line.The remaining configuration tasks areeasy and intuitive: define a pool of physical servers (Figure 1), select Healthcheck, set up routing between physicalservers and the load balancer, select VIPand scheduling algorithms (see Figure 2Health StatusZeus supplies the appliance with fivenetwork interfaces (see the “Hardware”box). One of these interfaces is mainlyused for out-of-band management(OOB) – that is, for administrative accessvia a separate cable. A web GUI or aserial console are available for basic set-One of the things that load balancers dois check the load and availability of thephysical servers and evaluate these parameters. The scheduler uses this information to decide to which server to assign which requests.At the same time, the scheduler has tokeep sessions persistent on the serverson which they are running: In many webapplications, the server stores information about the client status when userslog in or fill virtual shopping carts, forexample.The load balancer needs to take thisinto consideration to avoid tripping upthe application. More advanced loadbalancers implement a variety oftechniques to discover which requestsbelong to the shared session.For example, the Zeus appliance investigates cookies, adding its own cookiesif needed, or uses many other techniques. Thus, it can even accelerateSOAP applications by load balancing.Figure 2: The load balancer needs a separate IP address to which clients connect. In our lab,the VIP was 192.168.81.72. The load balancer forwards requests just like a reverse proxy.W W W. L I N U X - M A G A Z I N E . C O MISSUE 80 JULY 200769

SYSADMINZeus Load Balancerand Figure 3), decide whether some sessions need to be persistent on the server,and set up routing between the VIP andthe clients.To generate load for the web serversand the appliance in our lab, I used theApache benchmarking program, ab. It isincluded with the basic installation ofmany Linux distributions. Interestingly,Adam Twiss programmed the first version of ab in 1996, and he is one of thetwo founders of Zeus. Since then, theApache Software Foundation has maintained the tool. Additionally, Twiss hasnot worked for Zeus for many years.Thus, you can be certain that Zeus doesnot manipulate the measured values.The real point of performing the testwas not to discover the response time forevery single request. Users aren’t goingto notice whether the virtual server takes0.4 or 0.2 milliseconds to respond,which are just normal delays on the Internet. What’s more important is linearity of the measured values – that is, thatthe virtual web server just takes twice aslong to respond to 200 simultaneous requests as it does to respond to 100, andnot four times as long. We also wantedto find out whether it is possible to reproduce the effective connection man-Figure 3: Zeus can handle basic load balancing tasks, offering six algorithms for selecting thephysical server. Besides Layer 4 switching, the appliance also offers Layer 7 technology.agement by means of keepalives, HTTP1.1 compression and intelligent caching.Excellent Measured ValuesThe measurements in Figure 4 were generated by requesting a 4KB web pagefrom a cluster with two Apache serversSeven Years of Load BalancingThe idea of using load balancing to runmultiple parallel web servers has beenthrough several reincarnations over thelast seven years. When the dotcom bubble was at its peak (in 1999 to 2001), loadbalancing was justified because, in thetough world of e-commerce, traders withthe fastest systems would make deals.This form of load balancing typicallytook place in OSI Layer 4, the transportlayer.tures that offered PC-based systems atthe time are still in business today. Twocompanies that survived are F5 Networks and Arraynetworks.The Long Way UpAfter the events of September 11, the arguments started to change. Flash Events(FEs) or Flash Crowds generated suddenpeak loads that could bring an underequipped server to its knees. An FEcould be anything from a successful advertising campaigns to a popular newsstory.A modern Layer 7 load balancer, like theZeus ZXTM 7400 appliance tested here,offers more performance and stabilitythan the sum of its individual components. If you have four web servers andconnect into a modern load balancer,you get more than four times the performance – at least, that’s what the manufacturers promise. The new systems usea variety of approaches to achieve theirgoals. For example, they harmonize thehandling of TCP connections and useLayer 7 technologies such as compression and intelligent content caching.As a potential bottleneck, a load balancercouldn’t afford to be slow, which led tovendors opting for special ASIC-basedhardware for their load balancers andapplication switches. Sophisticatedscheduling helped them achieve maximum performance with the existing webserver hardware. Few of the manufac-In addition, manufactures of Layer 7 loadbalancers can rely on most web presentations not being perfectly programmedand thus offering some scope for optimization. Because tricks of this kind requirecomplex logic, many of today’s vendorsdo without ASICs and use PC-basedserver hardware instead.70ISSUE 80 JULY 2007W W W. L I N U X - M A G A Z I N E . C O M(Figure 6) running on normal PC hardware. Load grew from 100 to 1000 concurrent HTTP requests. Of course, ourlab conditions are fairly trivial comparedwith production use on e-commercewebsites. An enterprise-level productlike the Zeus ZXTM 7400 should be ableto handle 50,000 to 100,000 HTTP requests per second.According to the vendor, the appliancecan handle up to 92,000 HTTP requestsper second. Although I was unable toreproduce these results in our lab, Inoted an interesting effect below the1000 request level. Many CGI scripts andweb applications start to slow downwhen faced with 70 or 100 simultaneousrequests, instead demonstrating lineargrowth in response times.The measured values show that keepalives do not offer any real gains if theload balancer only uses them in connections to the servers. Keepalives need tobe enabled client-side to show any positive effect. In this case, the appliancesimply opens a couple of connectionsto each physical server and routes anyrequests over the active connections.The content cache also led to considerably improved measured values. Incontrast, I could not detect any noticeable gains after enabling HTTP 1.1 compression, probably because of the HTTPimplementation in ab and because I hadnetwork bandwidth to spare.