Transcription
(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 5, No. 5, 2014Performance Evaluation of Private Clouds Eucalyptusversus CloudStackMumtaz M.Ali AL-MukhtarAsraa Abdulrazak Ali MardanDepartement of Internet EngineeringAL-Nahrain UniversityBaghdad, IraqDepartement of Networks EngineeringAL-Nahrain UniversityBaghdad, IraqAbstract—the number of open source cloud managementplatforms is increasing day-by-day. The features of thesesoftware vary significantly and this creates a difficulty for thecloud consumers to choose the software based on their businessand scientific requirements. This paper evaluates Eucalyptus andCloudStack, the two most popular open source platforms used tobuild private Infrastructure as a service (IaaS) clouds. Theperformance of virtual machines (VMs) initiated and managedby Eucalyptus and CloudStack are evaluated in terms of CPUutilization, memory bandwidth, disk I/O access speed, andnetwork performance using suitable benchmarks. Different VMmanagement operations such as add, delete and live migrationare also assessed to determine which cloud solution is moresuitable than other to be adopted as a private cloud solution. As afurther performance testing, a simple web application has beenimplemented on the both clouds to evaluate their suitability inweb application hosting.bound together by a standardized or proprietary technology[4].Keywords—Cloud Computing; CloudStack; Eucalyptus;IaaS; Virtual Machine; Performance EvaluationI.INTRODUCTIONCloud computing as a new Internet service concept hasbecome popular to provide a variety of services to users. It is acombination of technologies that have been developed overthe last several decades, which includes virtualization,dynamic provisioning, internet delivery of services, gridcomputing, cluster computing and utility computing [1][2].According to NIST (National Institute of Standards andTechnology), “Cloud Computing is a model for enablingubiquitous, convenient, on-demand network access to a sharedpool of configurable computing resources (e.g., networks,servers, storage, applications, and services) that can be rapidlyprovisioned and released with minimal management effort orservice provider interaction" [3].There are three deployment models by which Cloudcomputing services are delivered: public, private, and hybrid.Public Cloud is a cloud that is made available as ―pay-asyou-go and accessible to the general public such as AmazonWeb Services. Private Cloud refers to a cloud infrastructurethat is internal to an organization and is not available to thegeneral public. A private cloud’s data centers can be onpremise and the physical infrastructure is owned and managedby the organization that owns it [4]. Hybrid cloud is acomposition of two or more cloud deployment models that areThere are certain legal, political, socio-organizationalreasons that may discourage an organization from using publiccloud infrastructure for certain kinds of activities, for exampleprocessing and storing citizens’ private data. There is also theissue of privacy, security, location and ownership of data [4].Many companies hesitate to use public cloud in whichcomputing resource are shared with other companies. Thesecompanies do not have any knowledge of where theirapplications are run and their data are stored or control accessto them [1]. Hence, private cloud infrastructure is consideredan appropriate alternative.Another big reason to increase the interest in setting up andmanaging private cloud is the SLA. The public cloudproviders nowadays provide guarantees on their service levelsand when service failures occur, they only offer to refund theircustomers regarding the infrastructure outages. However,service providers are not inclined to pay penalties of lowperformance level that would refund customers for loss ofbusiness revenue. Cloud providers are not only required tosupply correct services but, also, to meet their expectations inthe context of performance [5]. Also some software systemsand applications require different performance levels, qualityof services, reliability, and security, which are generally notguaranteed by a public cloud. Private cloud is an alternative tocompanies or researchers that need more control over that data[1] [6].There are many commercial and open source cloudmanagement platforms that are used to build Infrastructure asa service (IaaS) private cloud solution such as Eucalyptus,OpenNebula, and Vmware cloud. However the open sourcesolutions are gaining a lot of popularity and momentum withtheir features, rapid developing with low investment costwhich present a viable option for academic and scientificworlds [7], and enterprises who want first to test the cloudcomputing solution suitability to their business environmentbefore purchase the thousands dollars commercial solution.The number of cloud platforms related to a private IaaScloud is increasing day-by-day. The features of cloudmanagement software vary significantly and this creates adifficulty for cloud consumers to choose the software based ontheir business requirements.108 P a g ewww.ijacsa.thesai.org
(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 5, No. 5, 2014An example for this problem is choosing platform muchsuitable for hosting web applications or running highperformance computing (HPC) applications, or meetingspecific user usage way like users that demand a few virtualmachines (VMs) but want to run them for a long period oftime with guarantees on high-availability, or scientistsrequiring a large number of resources to conduct actualcalculations and analyses of data. The advent of several OpenSource Cloud platforms guarantees the performance anduptime. It is not easy for non-expert users to choose from thedifferent platforms without comprehending the characteristicsand advantages of each of this platform [6].As a consequence, performance evaluation of cloudcomputing platforms has been receiving considerable attentionby both the users and service providers as a prominent activityfor exploring the limitations of the cloud platforms andimproving service quality, infrastructure planning, and makinga wiser selection of the platforms. In addition cloudmanagement software vendors can develop and includeadditional features to their software by fixing the platformsbugs and including the missing features.The rest of this paper is organized as follows: Section IIpresents related work. Section III describes the testenvironment and methodology. Section IV covers theperformance evaluation of Eucalyptus and CloudStack VMs.Section V assesses VMs startup and release time. Section VIevaluates live migration of VMs. Section VII presentsresponse time of web application in the both clouds. Finallyconclusions are drawn in the last section.II.RELATED WORKMany studies have been conducted to evaluateperformance of open source cloud platforms such asEucalyptus, Opennebula and Nimbus. However these researchpapers did not perform a complete performance analysis of thecloud platform, and compare only the architectures andfeatures of the cloud management platforms. Nevertheless alittle work has been done yet to evaluate CloudStack due tothe fact that it is relatively new.performance of VMs are compared with regard to bare-metalor traditional IT infrastructure.III.TEST ENVIRONMENT AND METHODOLOGYCloudStack 4.1 cloud with one zone, pod and cluster hasbeen deployed using 3 identical physical servers. One server isused as a management server including primary and secondarystorage and the other two servers are used as host machines.Eucalyptus 3.2 cloud with one cluster has been deployed using3 identical physical servers each. One server is used as a cloudcontroller (CLC) including cluster controller (CC) and Walrusstorage and the other two servers are used as node controllers(NCs). Our servers are IntelR CoreTM i5-2410M CPU 2.3GHz,4GB RAM, 500GB SATA Hard Disk and 100MB Ethernetinterface. Centos 6.3 (final) is installed on each server asnative OS. CloudStack with NFS storage configuration isdeployed while Eucalyptus is deployed with local storageconfiguration. Each host in both clouds is configured withkernel-based virtual machine (KVM) as a hypervisor.In order to evaluate and analyze VMs performance of bothclouds, we have employed a number of benchmarks each fordifferent evaluation purpose. Table I shows the selectedbenchmarks.A customized CloudStack template (image used toestablish VM) and Eucalyptus VM image have been created inwhich all benchmarks are installed and configured to savetime and ease of work.Each benchmark test is repeated five times consequentlyand the average of results is considered. Different numbersand types of VM are regarded in the performance evaluation.In each cloud the same VM type is used and the same OS isrun which it Centos 6.3. Moreover, each cloud is built withsimilar hardware and uses the same hypervisor (KVM) toachieve a fair comparison between Eucalyptus and CloudStackand eliminate virtualization and hardware differences that mayaffect evaluation. Table II shows types of VMs that areprovided by Eucalyptus and CloudStack.TABLE I.De Sousa et al. [1] evaluated Eucalyptus VMs consideredprocessing and disk I/O performance only while in [6, 7, 8, 9,10, 11] authors brought out an overview of architectures ofopen source platforms and comparison of their generalfeatures and Characteristics. Mao and Humphery [12]investigated the performance of VM startup and release timeof public clouds. However, D. Steinmetz, et al. [13] evaluatedperformance and studied VM launch time of Eucalyptus andOpenStack but performance benchmarking was not specificand gave a general view of performance. While Folgar, et al.[14] evaluated performance of CloudStack primary storagedisk I/O only.Differently from previous works, this paper evaluatesperformance of Eucalyptus and CloudStack clouds VMscovering versatile parameters including performance of cloudmanagement platform considering add, delete and livemigration of VMs. Performance of VMs in term of CPUutilization ,memory bandwidth, disk I/O speed and networkingperformance is rated as key point of our evaluation. Also theBENCHMARKS DEPLOYED FOR VM TESTINGNo.Benchmarks1LINPACKProcessor2Bonnie Disk I/O3STREAMMemory Overall SystemTABLE II.Testing ResourceOFFERING VM TYPESTypeRAMCPU e4G460109 P a g ewww.ijacsa.thesai.org
(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 5, No. 5, 2014IV.PERFORMANCE E VALUATION OF VMSVMs of both clouds have been evaluated using selectedbenchmarks considering different relating metrics. The VMperformance has not been compared just between Eucalyptusand CloudStack, but it also has been compared in regard tobare-metal or traditional IT infrastructure.A. Comparison with Traditional IT infrastructureThe first question that comes in the mind of the cloud usersor organization that plan to adopt the cloud computingsolution is that "does the cloud virtual machine performance isthe same as traditional physical machine? ". To answer thisquestion, the performance of both machine with samehardware and software is tested using the same benchmarkthat mimic the real workload. UnixBench benchmark has beenrun with the traditional hardware stack on the host server ofboth cloud, then is run on both Eucalyptus and CloudStackCloud on a single VM utilizing the whole host serverresources.performance Computing) applications. LINPACK is abenchmark that measures a computer’s floating-point rate ofexecution by solving a dense n by n system of linear equationsin double precision. Gflop/s is the rate of execution; it refers tobillions of floating point operations per second.In this test three scenarios have been applied. First twotypes of VMs (small and large) are evaluated as VMcomputing power varies according to its type. The number oflinear equations is set to n 7000 in small VM and n 10000in large one. In the Second scenario, performance of VM isevaluated when there are different numbers of VMs arerunning the LINPACK simultaneously in order to test CPUisolation of VMs and check if there is any interference amongthem because of resource sharing. In this scenario a mediumtype VM with n 7000 has been used.As shown in figures 1 and 2, the performance ofEucalyptus cloud VM is nearly the same as physical one whilethere is a 7% gain in performance of the CloudStack VM. Thisresult suggests that the cloud computing management systemexploits or utilizes the computing resources on the samehardware stack better than the bare-metal or traditional ITsystem.Fig. 3. CPU Performance of VMsFig. 1. Performance of CloudStack vs. Bare-metal systemFig. 4. CPU IsolationFigure 3 shows the performance of VMs types. Eucalyptusand CloudStack VMs get a similar score. The floating pointexecution rate is considered very good with 7.7 Gflop/s and13.8 Gflop/s for small and large VMs respectively inEucalyptus, and 13.7Gflop/s and 7.6 Gflop/s in CloudStack ascompared to values with performance of physical machineswith similar hardware specifications as in [15]. Figure 4represents the performance when the benchmark is running ona multiple VMs. The figure reveals that CloudStack provides aslightly better VMs CPU isolation than Eucalyptus. In thisscenario the VMs have been assigned the entire physical coresof host server.Fig. 2. Eucalyptus VM vs. Bare-metal systemB. Processing PerformanceThe Eucalyptus and CloudStack VM computing power hasbeen assessed to test its ability in running a HPC (HighThe third scenario tests the performance when CPUsovercommitting is implemented. CPU overcommit is theprocess of allocating more virtualized CPUs (vCPU) to VM110 P a g ewww.ijacsa.thesai.org
(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 5, No. 5, 2014than actual physical CPUs of system. This requires underlyinghardware and hypervisor support, and this is one of reasonwhy KVM has been chosen in the clouds deployment. Itallows resource utilization and running fewer CPU coreswhich saves power and money. After testing and customizingthe overcommit ratio in our clouds, it has been set to two timesthe number of physical CPUs in the system.To evaluate and compare the performance of both cloudsVM disk I/O, the Bonnie benchmark is adopted in this test.Bonnie is a well-known Disk I/O performancebenchmark suite that uses a series of tests including data readand write speeds, maximum number of seeks per second,maximum number of file creations, and deletion or gatheringof file information per second.Two scenarios are implemented on both clouds. First, DiskI/O of two types of VMs, small and large are evaluated.Bonnie documentation recommend that file size should bedouble RAM size, therefore files with 1GB and 4GB sizes forsmall and large VM respectively were considered. Second,performance of VM when there is another VM performingintensive disk I/O operation is inspected.This is done to test isolation between VMs and check ifthere is any interference.Fig. 5. CloudStack CPU Overcommitted PerformanceFig. 7. Disk Access Speed in CloudStackFig. 6. Eucalyptus CPU Overcommitted PerformanceFigures 5 and 6 represent performance of VMs with vCPU.LINPACK is run on medium VM with N 7000. Then the testwas repeated when there are other VMs running with 90%CPU utilization to test the effects of processor interference dueto overcommiting. Lookbusy has been used to generate a highCPU utilization in VMs; it is an application for generatingsynthetic load on a system by generating fixed, predictableloads on CPUs, keeping chosen amounts of memory active,and generating disk traffic.Figures reveal that assigning VM a vCPUs is appropriateand works as expected, as there is no effects from other VMson the tested VM that run Linpack. Floating-point rate andtime of execution are nearly the same as number of VMs withhigh utilization increased in each case on the both cloudplatforms. This scenario revealed that the cloud vCPU solutionis better that using normal CPU core in performance andisolation; this is due to CPU job scheduling and fair sharingtechniques implementations of CPU overcommit.C. Disk 1/O PerformanceAs previously mentioned, Eucalyptus uses host local diskfor VM, while CloudStack uses primary storage that access viaNFS for VMs disks.Fig. 8. Disk Access Speed in EucalyptusFigures 7 and 8 show the performance of VM types of bothclouds. Sequential Output shows the speed in KB/s in whichthe data has been written. Sequential Input is the speed thedata has been read, Sequential and Random create refer to thenumber of files created per sec.Eucalyptus has a better overall performance thanCloudStack; this is due to using of local disk configuration forVMs in Eucalyptus so VM access the host disk locally, whilein CloudStack it accesses shared disk of primary storage overthe network via NFS which declines disk I/O speed andperformance.111 P a g ewww.ijacsa.thesai.org
(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 5, No. 5, 2014cache on any given system, so that the results are moreindicative of the performance of very large, vector styleapplications.Figure 11 indicates the results of memory performance ofsmall and large VMs in MB/s of both clouds. The array sizeapplied in the benchmarking is 10,000,000 elements for smallVM and 70,000,000 elements for large VM.Fig. 9. Disk Isolation in CloudStackFig. 11. Memory PerformanceFig. 10. Disk Isolation in EucalyptusFigures 9 and 10 illustrate the performance when two VMsare carrying intensive read write file operations concurrently.In this scenario medium type VM with 1GB file size is dealtwith. It reveals that disk I/O performance of VM disk isimpacted by the other VM as its performance drops in the bothclouds. In CloudStack NFS configuring, this is expected dueto primary storage disk sharing and available networkbandwidth of VM.In Eucalyptus, the NC’s disk capacity and bandwidth istypically shared between VMs. The capacity is shared in astraightforward way: each virtual machine has a virtual diskimage of a determined size that is allocated at the VM startingtime. It does not change until the termination of the VMexecution. On the other hand, the bandwidth of the disk isshared between all the resident VMs and there is currently nomethod of dividing this bandwidth or imposing limitations onits consumption by VMs. Therefore, the disk I/O performanceof one user would be interfered by another user’s VM withintensive disk I/O behavior.Figures 12 and 13 show the memory isolation betweenVMs, residing on the same host server. In this scenarioSTREAM benchmark is run on multiple VMs simultaneously.The tests demonstrates that with only one VM provisioned,there are plenty of rooms for further utilization of memory butas the number of VM increase the bandwidth available to eachdrops. Hence it requires a scheduler to avoid such effects.Despite that the memory isolation problem is existed in theboth cloud platforms; CloudStack shows better memoryperformance than Eucalyptus.Fig. 12. Memory Isolation in CloudStackDespite that the interference problem is existed in bothcloud platforms; Eucalyptus has a better disk performancethan CloudStack. This is due to local storage configurationwhere the VM disk is accessed locally (within host server) notover the network via NFS.D. Memory PerformanceThe memory performance stress test is based upon abandwidth test, as this is what distinguishes between types ofmemories. To measure the memory bandwidth the STREAMbenchmark is used. It is a synthetic benchmark tool thatmeasures memory bandwidth (in MB/s). It is specificallydesigned to work with datasets much larger than the availableFig. 13. Memory Isolation in Eucalyptus112 P a g ewww.ijacsa.thesai.org
(IJACSA) International Journal of Adv
OpenStack but performance benchmarking was not specific and gave a general view of performance. While Folgar, et al. [14] evaluated performance of CloudStack primary storage disk I/O only. Differently from previous works, this paper evaluates performance
