Transcription
Zeus ZXTM and ApacheWeb ServerA Broadband-Testing & Zeus Performance Study
First published September 2006 (V1.0)Broadband-TestingLa Calade,11700 Moux,FranceTel : 33 (0)4 68 43 99 70Fax : 33 (0)4 68 43 99 71E-mail: info@broadband-testing.co.ukWeb: http://www.broadband-testing.co.uk/Zeus TechnologyLimitedThe Jeffreys BuildingCowley RoadCambridge CB4 0WSUnited KingdomSales: 44 (0)1223 568555Main: 44 (0)1223 525000Fax: 44 (0)1223 525100Email: info@zeus.comWeb: http://www.zeus.com/All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without theexpress written consent of the authors.Please note that access to or use of this Report is conditioned on the following:21.The information in this Report is subject to change by Broadband-Testing without notice.2.The information in this Report, at publication date, is believed by Broadband-Testing to be accurate and reliable, but is notguaranteed. All use of and reliance on this Report are at your sole risk. Broadband-Testing is not liable or responsible for anydamages, losses or expenses arising from any error or omission in this Report.3.NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY Broadband-Testing. ALL IMPLIED WARRANTIES, INCLUDING IMPLIEDWARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT ARE DISCLAIMED ANDEXCLUDED BY Broadband-Testing. IN NO EVENT SHALL Broadband-Testing BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL ORINDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IFADVISED OF THE POSSIBILITY THEREOF.4.This Report does not constitute an endorsement, recommendation or guarantee of any of the products (hardware or software)tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors ordefects in the products, or that the products will meet your expectations, requirements, needs or specifications, or that they willoperate without interruption.5.This Report does not imply any endorsement, sponsorship, affiliation or verification by or with any companies mentioned in thisreport.6.All trademarks, service marks, and trade names used in this Report are the trademarks, service marks, and trade names of theirrespective owners, and no endorsement of, sponsorship of, affiliation with, or involvement in, any of the testing, this Report orBroadband-Testing is implied, nor should it be inferred. ZEUS TECHNOLOGY LIMITEDand BROADBAND-TESTING 1995-2006
ZXTM and Apache Web ServerImproving Apache PerformanceThe Apache Web Server is the Internet’s most popular web server software, as a result of itsmaturity, free distribution and the enormous range of community-supported modules. It isdeeply embedded into many organisations’ internet hosting infrastructures.However,Apache has several performance limitations that can result in poor and uneven end-userservice, inefficient resource utilisation and higher hardware and administration costs.This report investigates the performance limitations in the Apache Web Server anddescribes, with detailed benchmarks, how Zeus’ ZXTM Traffic Management software canovercome these deficiencies. Key findings include: Apache’s Keepalive Implementation gives very inconsistent levels of service when underload. ZXTM can manage Keepalives on Apache’s behalf to give even and consistentlevels of service. Apache’s performance when not using Keepalives is poor, with large error rates and lowtransaction rates. Using ZXTM with or without Keepalives totally eliminates errors, andresults in an increase of up to 18-times the sustained transaction rate. Apache’s SSL performance is sub optimal, with slow transaction times, limited capacityand connection errors under load.Using ZXTM to decrypt SSL traffic provides up to20-times the transaction rate and 20-times faster transactions, with noconnection errors. Apache performs very poorly on real-world high-latency networks. ZXTM almost totallyeliminates the high-latency effects, giving up to 40-times better utilisation, and 8times faster transaction times.What is ZXTM?Zeus’ ZXTM (Zeus Extensible Traffic Manager) operates at both Layer 4 (L4) load-balancing andLayer 7 (L7) intelligent traffic management levels and it is Ethernet-based but it is not a switch, orreally any kind of “Ethernet device” per se, but effectively a server-based network appliancewhich is sold as software or an appliance.It therefore typically sits in front of the server farm, behind the Internet gateway, from where itconducts traffic management in a wide number of different ways, none of which simply involvethrowing raw bandwidth at it.Being an appliance, rather than a switch, this means ZXTM works on a simple gateway principle– one way in, one way out (though in practise this is likely to be multiple, trunked Gigabit NICconnections) sharing Gigabit Ethernet switch capacity with the server farm. With its multi-facetedredundancy configurations, it also means that huge clusters of distributed ZXTM devices can becreated offering both extreme levels of performance and extreme levels of resilience (see later).ZXTMs feature set is extensive, covering intelligent load-balancing and every aspect of L7 trafficmanagement: throughput, compression, data manipulation, security – such as DoS protection –server and application optimisation, migration tools The company has unashamedly looked atF5 – as the market leader – and sought to equal or better every element of its own products. Theresult is what would be a very comprehensive set of capabilities for a mature product, let alone arelatively new kid on the block.One excellent example of this attention to detail lies in ZXTMs TrafficScript feature for deeppacket inspection and manipulation. This is quite simply the most comprehensive, rules-basedmethodology for traffic control available on anything we’ve seen.So what ZXTM is all about is not throwing more bandwidth at the problem but, instead, throwingintelligence at it. Never mind the width, feel the quality as you might say ZEUS TECHNOLOGY LIMITEDand BROADBAND-TESTING 1995-20063
ZXTM and Apache Web ServerTest BackgroundBroadband-Testing conducted a series of tests to investigate the performance profile of theApache server, and the effect of using a ZXTM 7000 Appliance to accelerate transactions onthe server.The tests evaluated Apache 2.0.56 running on RedHat Enterprise Linux release 4.0 onSunfire v20Z servers with 2Gb memory and Opteron 244 processors.Tests conducted by Broadband-Testing used Spirent Avalanche test equipment and a singleprocessor v20Z server. Other tests in this report were conducted by Zeus Technology, usingapachebench or zeusbench (an apachebench equivalent), with a dual-processor v20Z server.Zeus gratefully acknowledges the assistance and expertise of the Broadband-Testing teamwho oversaw and validated many of the tests in this report.4 ZEUS TECHNOLOGY LIMITED 1995-2006and BROADBAND-TESTING 1995-2006
ZXTM and Apache Web ServerKey Finding 1: Uneven levels of Service under loadThe Apache server gives very uneven levels of service under load. The following test usedapachebench to load up an Apache server with varying numbers of users, each withkeepalive connections. A new user visited the site, and the transaction time and transactionrate the new user achieved was measured.New User Transaction TimeNew User Transaction Rate300Transaction Time (ms)Transactions per pache1005004000100Current Users200300Current UsersOnce over 250 concurrency slots are occupied by site users, the service for additionalvisitors degrades very rapidly. Existing site users hog all the available resources.A ZXTM 7000 Appliance was used to transparently manage the client keepalive connectionsand marshal them into a far smaller number of keepalive connections to the Apache server.New User Transaction TimeNew User Transaction Rate300Transaction Time (ms)Transactions per Second300250200ApacheZXTM 70001501005000200400600800250200ApacheZXTM 700015010050010000200Current Users4006008001000Current UsersZXTM ensured that all site users obtained consistent performance and many moresimultaneous site visitors could be sustained. It’s possible to calculate the deviation fromthe average transaction rate that the new user experiences:Deviation from Average Transaction Rate100.00%Deviation50.00%ApacheZXTM 70000.00%-50.00%-100.00%02004006008001000Current UsersApache’s uneven distribution of service comes as a result of its limited per-thread or perprocess concurrency model.Many sites overcome it by disabling or restricting HTTPKeepalives, although this results in poorer overall performance. ZEUS TECHNOLOGY LIMITEDand BROADBAND-TESTING 1995-20065
ZXTM and Apache Web ServerKey Finding 2: Apache’s performance improved with ZXTMDisabling keepalives on Apache results in poorer overall performance.Broadband-Testingused Spirent Avalanche clients to simulate varying numbers of users accessing the Apacheserver.Error Rate (connect and response timeouts)60025%50020%400300Apache200Error RateTransactions per secondSustained Transaction Rate (no 007501000Simultaneous UsersSimultaneous UsersA ZXTM 7000 Appliance was then used to manage and marshal the HTTP requests to theApache server:Sustained Transaction Rate (no keepalives)Error Rate (connect and response timeouts)25%350020%30002500ApacheZXTM 7000200015001000Error RateTransactions per second400015%ApacheZXTM 700010%5%50000%025050075010000250Simultaneous Users5007501000Simultaneous UsersWithout keepalives, ZXTM lifts the performance of the Apache server dramatically.The previous finding indicated that it is safe to enable Keepalives with ZXTM, but not withApache. Nevertheless, the above test was repeated using Keepalives on both systems:Transaction Rate (using Keepalives)Transactions per Second45004000350030002500ApacheZXTM 700020001500100050005001000Simultaneous UsersUsing Keepalives with ZXTM adds a further performance boost. Note that the Apache serverwas the bottleneck in all these tests, running at between 90% and 100% CPU utilization.The ZXTM had ample spare capacity to load-balance traffic across a number of Apacheservers.6 ZEUS TECHNOLOGY LIMITED 1995-2006and BROADBAND-TESTING 1995-2006
ZXTM and Apache Web ServerKey Finding 3: Accelerating SSL transactions on Apache serverSSL is very processor intensive and puts a limit on the capacity of a web site or service.Broadband-Testing investigated Apache’s SSL performance, measuring the performance ofSSL transactions by requesting 16 byte files with no SSL session reuse.The Apache server was tested directly. CPU utilization consistently reached 100% in thesetests, but the server became overloaded with more than 500 simultaneous users, processingtransactions increasingly slowly.Average Transaction Time - SSL2501200020010000Transaction Time (ms)SSL Transactions persecondSustained Transaction Rate - 080010000200Simultaneous Users4006008001000Simultaneous UsersThen a ZXTM 7000 Appliance was used to transparently decrypt SSL traffic on behalf of theApache server, so that it processed unencrypted traffic only.Average Transaction Time - SSL1800120001600140010000Transaction Time (ms)SSL Transactions persecondSustained Transaction Rate - SSL12001000ApacheZXTM 700080060040020008000ApacheZXTM 70006000400020000020040060080010000Simultaneous Users2004006008001000Simultaneous UsersDuring the tests, many of the connections to the Apache server failed due to timeouts.When ZXTM was used to decrypt the connections, no failures were observed:Error Rate - SSLError Rate45%40%35%30%25%ApacheZXTM 700020%15%10%5%0%02004006008001000Simultaneous Users ZEUS TECHNOLOGY LIMITEDand BROADBAND-TESTING 1995-20067
ZXTM and Apache Web ServerKey Finding 4: Poor performance on high-latency networksThe majority of benchmarks are conducted on fast, local networks, and can concealproblems apparent to any user accessing a site over a remote, high-latency network. Thistest used ‘zeusbench’ and an intermediate device1 to simulate different network conditions,and investigated the performance of Apache (with and without keepalives):ClientNetwork GatewayZXTM (optional)Apache ServerApache performs poorly on high-latency networks:Transactions per SecondTransactions per Second for various network latenciesNetwork Round Trip Times80006000400020000 msLocal Area NetworkApache(no KA)50 msLocal Regional NetworkApache(with KA)100 msNational Network200 msInternational Network400 msIntercontinental Network00100200300400Network Round Trip Time (ms)At high latency values, the Apache server ran at no more than 8% CPU. Even though 5000concurrent connections were competing for service and the Apache server had plenty ofcapacity, the latency effects meant that Apache was unable to serve them.A ZXTM Appliance can manage the client connections on behalf of Apache. In this case, theApache server behaves as if it were communicating on a fast, low-latency network.Transactions per SecondTransactions per Second for various network latencies80006000ZXTM 70004000Apache(no KA)2000Apache(with KA)00100200300400Network Round Trip Time (ms)ZXTM can give consistent performance and can eliminate the latency effects that cause poorperformance in Apache.18A Linux gateway server running the Netem network emulation module. ZEUS TECHNOLOGY LIMITED 1995-2006and BROADBAND-TESTING 1995-2006
ZXTM and Apache Web ServerAnalysis: How do Keepalives affect Levels of Service?HTTP Keepalives produce a much better end-user browsing experience, but the Apache WebServer disables them by default because they cause performance problems.Why use HTTP Keepalives?The HTTP protocol sends a request using a TCP connection. HTTP Keepalives allows a webbrowser to reuse a TCP connection, sending a number of HTTP requests (for web pages,images, stylesheets and other resources) down the same connection. TCP connections taketime to set up and tear down, and without Keepalives, a new TCP connection must becreated for each resource on a web page.All modern web browsers support keepalives (and some also support pipelines withinkeepalives). The page load time and bandwidth usage benefits of Keepalives and Pipelinesare well known2,3,4,5.Analogy: A telephone conversation uses the equivalent of keepalives. It takes severalseconds to set up a call – dialling the number, waiting for the network and for the otherparty to pick up – and a second or two to close the call.Without Keepalives, you could only speak once on your phone call.The other partywould reply and then you would have to hang up. If you wanted to conduct a detailedtransaction, or even just a conversation, it would be a slow and laborious process as youwould have to hang up and redial between sentences.With Keepalives, the phone call can last as long as you need. You can speak as manytimes as you wish, and conversation (or transactions) are conducted with ease.Pipelines are a further development of keepalives, whereby the web client can issue severalrequests without having to wait for each reply in turn – just as in a phone conversation, youmay ask several questions in one sentence.Why not use Keepalives with Apache?The Apache server is not designed to handle Keepalives efficiently.The problem arisesbecause the Apache Server has a relatively low limit on the number of TCP connections(‘slots’) that it can manage at any one time.Each Keepalive connection occupies one ofthese slots for up to 15 seconds (the default timeout).A slot corresponds to a process or thread, depending on whether Apache uses its prefork(process) or worker (thread) multiprocessing module. The typical limit is 256 slots forprefork and 150 for worker, and it is generally unwise to configure a larger limit forstability reasons.All tests in this report used prefork because this is the commondefault and is the most stable and supported b5b10a-5eac485f-80f0-0e04eaf6c3ba.mspx?mfr ssues/2005/11/PumpUpPerformance/default.aspx3 ZEUS TECHNOLOGY LIMITEDand BROADBAND-TESTING 1995-20069
ZXTM and Apache Web Server1. Keepalives reduce Apache performanceAn occupied slot takes up resources on the server machine, even if it is occupied by anidle Keepalive connection. In a live deployment, it is more efficient to disable Keepalivesand take the hit of accepting and closing many TCP connections rather then to acceptthe cost of maintaining the many idle keepalive connections.2. Keepalives give very uneven levels of service in ApacheIf all of the available ‘slots’ are occupied by active users or idle keepalive connections,there are no free slots for additional users.A new user’s connection will be queued in the operating system’s listen queue, and willnot be serviced until an existing idle keepalive connection times out. During times ofhigh load, there is a lot of contention for free slots as they become available. Once thelisten queue fills, clients will receive ‘connection refused’ errors, and connections in thelisten queue may time out of they are not serviced quickly.New users will get very poor service, while users with existing keepalive connections willbe able to reuse them and get good service.For this reason, Keepalives are commonly disabled by default in Apache.The ApacheSoftware Foundation has implemented an experimental ‘event’ MPM in Apache 2.2 to copewith the ‘keep alive problem’6; at the time of writing, this module is experimental, notavailable on Apache 2.0.x, and incompatible with other Apache features including SSLsupport.Keepalives may give Misleading BenchmarksKeepalives appear to dramatically improve the performance of an Apache server. Using theapachebench tool against an Apache Server on a two-processor SunFire v20Z server, it’seasy to fully load up the web server:Command LineTransactions/secondCPU utilisationWith Keepalivesab –c 300 –t 10 –k7203100%Without Keepalivesab –c 300 –t 10505670%Misleading benchmarks indicate good Keepalive performanceThese benchmark figures are misleading because they only report completed connections.Once the concurrency figure exceeds the number of slots, additional requests will never beserved, so additional visitors to the site will have to wait in a queue until sufficient keepaliveconnections are closed. Apachebench does not reveal this poor “client experience”.The first benchmark test in this report is designed to identify and measure this /event.html ZEUS TECHNOLOGY LIMITED 1995-2006and BROADBAND-TESTING 1995-2006
ZXTM and Apache Web ServerBenchmarking ApacheThe default apache configuration on a RedHat Enterprise Linux 4 system was used, with a10kB response file. The default prefork MPM was used, with default d4000Existing users with Keepalive connections were simulated by one apachebench run usingvarious degrees of concurrency:ab –c num users –n 10000000 –k http://10.2.2.2/After 15 seconds, a new user attempted to download content from the new server using anew Keepalive connection:ab –c 1 –t 60 –k http://10.2.2.2/The test measured the transaction rate (transactions per second) and transaction time thatthe user achieved over a 60-second .70Time 061430New User Transaction TimeNew User Transaction Rate300250200150Apache1005000100200300Transaction Time (ms)Transactions per Second300250200150Apache1005
Zeus’ ZXTM (Zeus Extensible Traffic Manager) operates at both Layer 4 (L4) load-balancing and Layer 7 (L7) intelligent traffic management levels and it is Ethernet-based but it is not a switch, or really any kind of “Ethernet device” per se, but effectively a server-based net
