Transcription
International Journal of Computer Applications (0975 – 8887)Volume 80 – No 13, October 2013Performance of Eucalyptus and OpenStack Clouds onFutureGridSharath SScholar, CSE Department,East Point College of Engineering and Technology,Bengaluru 560068, Karnataka, India.ABSTRACTWith expanding demand for Cloud computing, numerousCloud providers are offering services. Benchmarking isextremely important to compare their performance. In thispaper, we study the performance of two IaaS cloudsEucalyptus 3 and OpenStack (version Essex) implemented onFutureGrid [1, 2], a test bed that allows researchers to conductexperiments to understand the behavior and utility of cloudcomputing services. Phoronix Test Suite [3] is utilised as abenchmark program to conduct diverse performancemeasurements on both the clouds. Besides the programs inPhoronix Test suite, time to launch instances, time to registerimages using scripts were also matched. The results arediscussed in this paper.Anirban BasuProfessor and Head, CSE Department,East Point College of Engineering and Technology,Bengaluru 560068, Karnataka, India.A significant feature of cloud computing is virtualization ofphysical resources which enables the execution of multiplejobs on the same, shared physical environment by creatingVirtual Machines (VMs). This make a cloud based serviceeconomical to both cloud service providers and cloud users.Cloud service providers can reduce their costs of servicedelivery by resource consolidation through virtualization.Benchmarking plays a critical role in evaluating theperformance of systems that are ready for operation. With theincreasing use of cloud computing, more and more users areseeking answers to questions on which type of cloud theyshould use for services at an economical cost.KeywordsWhile work is going on designing appropriate benchmarks, sofar Phoronix Test Suite [3] is broadly used. In this paper wediscuss the results obtained by running the Phoronix Test suiteon Eucalyptus and OpenStack on FutureGrid. For measuringthe time for registration and launch of VMs on the clouds,scripts had to be written.FutureGrid, Eucalyptus, OpenStack, Phoronix Test Suite,Image registration time.2. RELATED WORKGeneral TermsVirtual Machines, Benchmark, Cloud computing.1. INTRODUCTIONA Cloud computing platform has a large pool of resourceswhich can be accessed by users on demand. The resourcesshared bya number of users are allocated for betterproduction and for optimum resource utilization.Users submittheir jobs (or demands for resources such as CPU, RAM, disk,application, etc.) over the network. Depending upon theservice requirement a Cloud computing platform configuresits resources dynamically. Cloud computing platforms offerthree types services which are: Infrastructure-as-a-Service (IaaS) Platform-as-a-Service (PaaS) Software-as-a-Service (SaaS).Examples of IaaS cloud include Amazon EC2, IBM Cloud.Microsoft Azure and Google AppEngine are examples ofPaaS cloud. Examples of SaaS include Gmail, Google Docs.In this paper we focus on the performance of open source IaaStype of cloud computing platforms. Two popular open sourceIaaS Cloud Computing environments are: Eucalyptus [4, 5]and OpenStack [6]. These environments are widelyestablished in private and public clouds and FutureGriddeploys both of these. FutureGrid encompasses ageographically accessible set of heterogeneous computingsystems, a data administration system, and a library ofsoftware images. FutureGrid runs on a network accessible tousers in several countries allowing researchers to conductexperiments.Benchmarking is necessary because of wide-spread adoptionof cloud across world. Alexander Lenk, Michael Menzel,Johannes Lipsky, Stefan Tai and Philipp Offermann havedesigned performance measuring procedure for Infrastructureas-a-Service and evaluated Amazon EC2 platform, Flexiscaleplatform and Rackspace platform to validate its utility [7].Taxonomy to standardize performance evaluation ofcommercial Cloud services is provided in [8] and distinctcharacteristics that are to be considered while evaluatingperformance of Cloud are discussed. In [9] comparison of fourcommercial cloud providers Amazon EC2, Windows Azure,Google AppEngine and Rackspace CloudServers has beendone including several components of computing, network,database and storage. The startup time of cloud VMs acrossAmazon EC2, Windows Azure and Rackspace and differentfactors that affect startup time are studied in [10].3. CLOUD FRAMEWORKS USED3.1 FutureGridFutureGrid[2] is a distributed, high-performance test-bed thatpermits scientists to collaboratively work and designinnovative approaches to parallel, grid, and cloud computing.The FutureGrid project is funded by the National ScienceFoundation (NSF) and is led by IndianaUniversity,University of Chicago, University of Florida, San DiegoSupercomputing Center, Texas Advanced Computing Center,University of Virginia, University of Tennessee, University ofSouthern California, Dresden, Purdue University, and Grid5000.FutureGrid permits researchers to conduct experiments incomputer science related to grid and cloud computing.31
International Journal of Computer Applications (0975 – 8887)Volume 80 – No 13, October 2013FutureGrid provides IaaS and PaaS frameworks and supportsthree cloud services: Eucalyptus, OpenStack and Nimbus.Table 1: OpenStack VM types in FutureGridVM type3.2 EucalyptusEucalyptus is a software platform that applies IaaS type cloudcomputing [4, 5]. Eucalyptus presents Amazon EC2 basedweb service interface for cloud service. Eucalyptus onFutureGrid is comprised of five components: Cloud Controller is the entry point into the cloud. Itinteracts with other constituents and organizesunderlying virtualized resources (servers, networkand storage).Walrus [4] allows users to store persistent data asbuckets and objects.Storage Controller provides functionality to createElastic Block Storage (EBS) volumes. EBS volumesstore continual data of virtual machines.Node Controller controls virtual machine activitiesincluding execution, examination and termination ofVM instances.3.3 OpenStackOpenStack is a collection of open source components todeliver public and private clouds that supply IaaS [6].OpenStack on FutureGrid includes following constituents: Nova is designed to provision and manage largenetworks of virtual machines. Swift provides object storage to store or retrievefiles. Glance is responsible for discovery, registration anddelivers services for virtual disk images.4. PHORONIX TEST SUITEThe Phoronix Test Suite [3, 4] is the most comprehensivebenchmarking suite of programs available to effectively carryout performance evaluation and is also very simple to use.RAMDISK 096102m1.large8192204m1.xlarge16384408As m1.tiny is rarely used this case is not considered. Whenm1.small type virtual machines are created 10 GB disk spaceand one virtual CPU is allocated.Eucalyptus in FutureGrid has following types of VirtualMachines (see Table 2).Table 2: Eucalyptus VM types in FutureGridVM typeRAMDISK 120102m1.xlarge6000152c1.xlarge9216204We represent OpenStack instance types with prefix “s” andEucalyptus instances with prefix “e”. Results obtained arepresented here.5.1 Disk PerformanceThe Phoronix Test Suite is utilised for determiningcomputer’s performance for internal quality assurancepurposes, and hardware validation. It runs on platformsranging from smartphones and personal computers to multicore workstations and cloud computing infrastructures. It isdeveloped for Linux, OpenSolaris, Apple Mac OS X,Microsoft Windows, and BSD operating systems.5.1.1 AIO-Stress:5. IMPLEMENTATION AND RESULTS5.2.1 Build Linux Kernel:We created new FutureGrid portal account and used IaaSservices to create necessary virtual machines. Phoronix TestSuite software was installed in different types of instances ofboth clouds. Benchmark test in this suite were run to findperformance of instances. Shell scripts were written forregistering images and computing the time taken forregistering. Launch time of instances furthermore discoveredusing shell scripts.This test measures the time taken to compile and build Linuxkernel. As shown in Figure 2 even though smaller instancetypes of Eucalyptus take larger time compared to OpenStack,larger instances take less time.Phoronix Test Suite is a collection of test profiles with agranted set of test options. Phoronix Test Suite is used to findperformance of virtual machines related to aiostress,ramspeed, and compress-gzip and so on.This is asynchronous I/O benchmark using 2048 MB test fileand 64 KB record size. OpenStack virtual machines havelower I/O stress compared to Eucalyptus virtual machines. Asshown in Figure 1, I/O stress increases with size. Eucalyptusinstances have higher I/O stress.5.2 Processor Performance5.2.2Byte:5.2.2.1 DhrystoneThis standard is used to measure and compare theperformance of Virtual Machines. The test focuses on stringhandling and is heavily leveraged by hardware and softwaredesign, compiler and linker options, code optimization, cachememory, wait states, and integer data types[. (See figure 3)Openstack in FutureGrid has following types of VirtualMachines (see Table 1).32
International Journal of Computer Applications (0975 – 8887)Volume 80 – No 13, October 20135.2.2.2 Floating-Point Arithmetic:5.2.3 Gzip Compression:This test measures the speed and effectiveness of floatingpoint operations. A wide range of C functions including sin,cos, sqrt, exp, and log are used as well as integer and floatingpoint math operations, array accesses, conditional branches,and procedure calls (See figure 4).This test measures the time needed to compress a file usingGzip compression. Eucalyptus instance types almost takesame amount of time around 26 seconds. All OpenStackinstance types take less time compared to Eucalyptus exceptm1.small because of high processor speed (See figure5).Figure 1: AIO-StressFigure 2: Build Linux Kernel33
International Journal of Computer Applications (0975 – 8887)Volume 80 – No 13, October 2013Figure 3: DhrystoneFigure 4: Floating-Point ArithmeticFigure 5: Gzip Compression34
International Journal of Computer Applications (0975 – 8887)Volume 80 – No 13, October 2013Figure 6: CacheBenchFigure 7: RAMspeed SMP (System Memory Performance)Figure 8: Stream35
International Journal of Computer Applications (0975 – 8887)Volume 80 – No 13, October re 9: Virtual machines launch time5.3 Memory Performance5.5 Image registration time5.3.1 CacheBench:This includes time taken to bundle, upload and register animage. The supported kernel and ramdisk images available inFutureGrid should be chosen for registering custom images.Two images of 1.5 GB and 3 GB are registered to Eucalyptusand OpenStack (See figure 10).CacheBench is a standard designed to assess the performanceof the memory hierarchy of computer systems. This performsrecurring access to data items on varying vector lengths.Timings are taken for each vector length over a number ofiterations. Computing the product of iterations and vectorlength presents us the total amount of data accessed in bytes.This total is then divided by the total time to computebandwidth. Results in Figure 6 show that even thoughOpenStack processors are faster Eucalyptus instances performbetter in this test.5.3.2 RAMspeed:This benchmark tests the system memory (RAM)performance. RAMspeed tests how fast are cache andmemory subsystems by assigning certain memory space andstart either writing to or reading from it using continuousblocks sized in power of 2 from 1Kb up to the array boundary.They function with sequential data streams passed throughALU, FPU units respectively. OpenStack virtual machines arefound to have more bandwidth compared to Eucalyptus virtualmachines (See figure 7).5.3.3 Stream:The stream benchmark is a simple synthetic benchmarkprogram that measures sustainable memory bandwidth and thecorresponding computation rate for simple vector kernels. Thestandard is specifically designed to work with datasets muchlarger than the available cache on any allocated system.Eucalyptus c1.xlarge performs better than all other kinds, butoverall OpenStack instances have good bandwidth (See figure8).5.4 Virtual machines launch timeShell scripts were written to run virtual machines sequentially.Time taken by virtual machines to change from pending torunning state is computed which gives the time taken toacquire resources (start time or launch time). Eucalyptusvirtual machines take less time contrasted to OpenStackvirtual machines (See figure 9).Figure 10: Image registration time6. CONCLUSIONPhoronix Test Suite was used to benchmark both Eucalyptusand OpenStack Cloud Computing platforms. Outcomes showthat Openstack instances have lower IO/stress and higherprocessing speed. Eucalyptus instances have higher cache andRAM memory bandwidths. Eucalyptus is apt forcache/memory intensive applications and OpenStack for CPUintensive applications. Using shell-scripts we conclude thatEucalyptus instances take slightly lesser time to start runningcompared to OpenStack types. OpenStack registers imagesmore quickly than Eucalyptus. Thus we conclude in36
International Journal of Computer Applications (0975 – 8887)Volume 80 – No 13, October 2013FutureGrid Openstack Essex outperforms Eucalyptus 3 inmost cases.[6] OpenStack web page, http://www.openstack.org, July2013.7. REFERENCES[7] Alexander Lenk, Michael Menzel, Johannes Lipsky,Stefan Tai, Philipp Offermann, “What are you payingfor? Performance benchmarking for Infrastructure-as-aService offerings”, 4th International Conference onCloud Computing, IEEE 2011.[1] FutureGrid web page, https://portal.futuregrid.org/, July2013[2] Gregor von Laszewski, Javier Diaz, Fugang Wang,Geoffrey C. Fox, “Comparison of Multiple CloudFrameworks”, 5th International Conference on CloudComputing, IEEE 2012[3] Phoronix Test Suite web page, https://www.phoronixtest-suite.com, July 2013[4] Eucalyptus web page, http://open.eucalyptus.com/, July2013.[5] Daniel Nurmi, Rich Wolski, Chris Grzegorczyk,Graziano Obertelli, Sunil Soman, Lamia Youseff, DmitriiZagorodnov, “The Eucalyptus Open source Cloudcomputing System” In Proceedings of the 9thIEEE/ACM International Symposium on ClusterComputing and the Grid, CCGrid, China, 2009.IJCATM : www.ijcaonline.org[8] Zheng Li, Liam O'Brien, Rainbow Cai, He Zhang,“Towards a Taxonomy of Performance Evaluation ofCommercial Cloud Services”, 5th InternationalConference on Cloud Computing, IEEE 2012[9] A. Li, X. Yang, S. Kandula, and M. Zhang, “Cloudcmp:comparing public cloud providers,” in Proceedings of the10th annual conference on Internet measurement, ACM2010.[10] Ming Mao, Marty Humphrey, “A Performance Study onthe VM Startup Time in the Cloud”, 5th InternationalConference on Cloud Computing, IEEE 201237
We represent OpenStack instance types with prefix "s" and presented here. 5.1 Disk Performance 5.1.1 AIO-Stress: This is asynchronous I/O benchmark using 2048 MB test file and 64 KB record size. OpenStack virtual machines have lower I/O stress compared to Eucalyptus virtual machines. As shown in Figure 1, I/O stress increases with size.
