WIXLIB

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCFigure 1: Architecture of Test EnvironmentI/O Control SetupConfiguration OverviewVMware vSphere data centerssupporting clusters of virtualized hostscan use shared storage infrastructure.Applications in those virtualized hoststhat share the storage infrastructurecan be I/O constrained without propercontrol of the shared resources. This iswhat Storage I/O Control (SIOC) provides.SIOC is the ability to set threshold andshare adjustments within the VMwareESXi environment. Setting rate-limitsand triggers allow the administratorsto set a pre-programmed response tostorage resources avoiding congestionand ensuring performance for critical VMs.There are three SIOC settings used forthis experiment; Share Value, Limit-IOPs,and Congestion Threshold.Share Value and Limit-IOPs are both seton a per VM basis. Share Value sets apercentage value to the VM in order togive it more or less priority to the datastore. The changes made here will impactthe VMs on a single host regardless ofwhether SIOC in enabled or not. EnablingSIOC comes into play when there is morethan one host. As part of this experiment,a VM is migrated from one host to another.With SIOC enabled, settings made toShare Value or Limit-IOPs will be appliedduring the migration and on the host theVM is migrated. Shares can be changed toHigh (2000), Normal (1000), Low (500), orCustom; custom being in the range of 200to 4000. Whereas Limit-IOPs will set theupper bound for storage resources in therange of 16 to 2,147,483,647.Congestion Threshold is a setting inthe target properties. It is a parameteronly available with SIOC enabled.Congestion Threshold is a trigger that,when latency is exceeded, will throttlethe I/O throughput. A lower value will5

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCresult in lower device latency and stronger virtual machine I/O performance isolation whereas a higher value typically results in higheraggregate throughput and weaker isolation. Congestion Threshold can be set in the range of 5ms to 100ms.The steps to configure Share Value, Limit-IOPs, Congestion Threshold, and enable SIOC are as follows:1.From the VMware vSphere Console home page navigate to Inventory Datastores and Datastore Clusters page.2.Select the LUN on which testing will be performed to see the available option and views. Navigate to the Virtual Machines tab tobegin the process. Note that both VMs are in the default settings for Limit–IOPs and Data-store % Shares. To make changes tothese options, Right Click on the VM to be changed and select Edit Settings.6

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOC3.Navigate to the Resources tab and then choose Disk. There are two primary items that can be changed: Shares and Limit–IOPs.Shares adjustment allows a VM to be given a higher or lower percentage of the host’s resources. Shares can be changed to High(2000), Normal (1000), Low (500), or Custom; custom being in the range of 200 to 4000. Whereas Limit-IOPs will set the upperbound for storage resources in the range of 16 to 2,147,483,647.4.A “Share” example of 1:1 or Normal-Normal shows a Share Value of 1000 and 50% Share per VM.5.A “Share” example of 2:1 or Normal-Low shows a Share Value of 1000 and 500 with 66% and 33% Share per VM respectively.6.A “Share” example of 1:1 or Normal-Normal and a Limit-IOPs value of 62 (500Mbps) shows a Share Value of 1000 and 50% Shareper VM plus the rate-limiting of the App05 VM to 62.7

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOC7.Navigate to the Configuration tab and select the storage volume or LUN to be modified. On this page there are a couple of notableitems, the first being a list of servers that have access to the storage and the second is the state of SIOC. Next. By clicking onproperties, the storage properties dialog box is displayed. For this papers use-cases, there are two options: enable/disable SIOCand edit the Congestion Threshold which is found by clicking the advanced button.8.Changing the Congestion Threshold is only possible with SIOC enabled.Usage Model Test Results andAnalysis:Usage Model #1: Host Level Storage I/OResource Allocation for VMa. Disk shares ratio of 1:1 (with SIOCdisabled)This usage model talks about the storageI/O resource allocation at the host level.The setup consists of a single VMwareESXi 5.0 server is hosting two virtualmachines (app04 and app05) and the diskI/O shares for the virtual disks (VMDKs)of these virtual machines are varied withSIOC feature enabled and disabled. As ab. Disk shares ratio of 1:1 (with SIOCenabled)Note: Please refer to the section“Environment Setup” as to how to setshares on virtual disks and how toenable/disable SIOC. Also note that, theCongestion Threshold was set to 30ms(default) in all the scenarios when SIOC isenabled.c. Disk shares ratio of 2:1 (with SIOCdisabled)Disk shares ratio of 1:1 (with SIOCdisabled)8result there are four such scenarios thatwill be discussed below. The scenarios are:d. Disk shares ratio of 2:1 (with SIOCenabled)In this sub-case the disk I/O shares forthe virtual disks on both VMs app04 andapp05 are set to “Normal” or 1000 shares

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCas shown in the below figure. Hence the disk share ratio is 1:1 for both the VMs. Also note that SIOC is disabled.Once the disk shares are set, the application Iometer (1MB 100% read test) was run on the both the VMs. The throughput (MBps) andIOPs are captured from the Iometer tool running on the VMs and the total aggregate throughput was captured from the VMware ESXihost by running the tool “esxtop” as shown in the below figure.The throughput on app04 and app05 VMs was observed to be 328MBps and 329MBps.9

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCDisk shares ratio of 1:1 (with SIOC enabled)With the same disk share ratio as in sub-case (a), SIOC was enabled and the Iometer test was re-run on the VMs and the throughput is 329 MBps on both app04 and app05 as shown in the below figure.Disk shares ratio of 2:1 (with SIOC disabled)In this sub-case the disk shares were set to “Normal or 1000 shares” on app04 and “High or 2000 shares” on app05 VM. As a result wehave a disk share ratio of 2:1 as shown in the below figure.10

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCWith the disk shares set, Iometer was ran on both VMs and the throughput of about 430 MBps and 215 MBps was observed onapp05 and app04 respectively as shown in the below figure. The throughput numbers were proportional to the disk shares of 2:1.Disk shares ratio of 2:1 (with SIOC enabled)With SIOC enabled the above test was re-run and the throughput from the Iometer tool was observed to be 440MBps and 220 MBpsfor app05 and app04 respectively as shown in the below screenshot. These throughput numbers are again proportional to the diskshares of 2:1.11

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCThe above four scenarios can be summarized in a simple table as follows:SIOCDisk shares ratioApp05 VMApp04 VMDisabled1:1329 MBps328 MBpsEnabled1:1328 MBps328MBpsDisabled2:1430 MBps215 MBpsEnabled2:1440 MBps220 MBpsFrom the above table, it can be said that when the VMs are on the same host irrespective of whether SIOC is enabled or disabled,the performance of the VMs are proportional to the distribution of disk shares. This is due to the fact that the local host level diskscheduler is enforcing the shares.Usage Model #2: Data-store Array LevelStorage I/O Allocation, Congestion andLatency Managementdisabled. As a result there are four suchscenarios that will be discussed below.The scenarios are:This usage model discusses about theimplementation of resource allocation atthe shared data-store level. The setupconsists of two VMware ESXi 5.0 serverswhich are part of a cluster and the virtualmachines app04 and app05 are hosted oneach of the server and the disk I/O sharesfor the virtual disks (VMDKs) of thesevirtual machines are varied with StorageI/O Control (SIOC) feature enabled anda. Disk shares ratio of 1:1 (with SIOCdisabled)b. Disk shares ratio of 1:1 (with SIOCenabled)Note: Please refer to the section“Environment Setup” as to how to setshares on virtual disks and how toenable/disable SIOC. Also note that, thecongestion threshold was set to 30ms(default) in all the scenarios when SIOC isenabled.c. Disk shares ratio of 2:1 (with SIOCdisabled)d. Disk shares ratio of 2:1 (with SIOCenabled)Disk shares ratio of 1:1 (with SIOC disabled)To start, the disk I/O shares for the virtual disks on both virtual machines app04 and app05 are set to “Normal” or 1000 shares asshown in the below figure. Hence the disk share ratio is 1:1 for both the VMs.12

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCWith disk shares ratio of 1:1, the workload i.e. Iometer (1MB 100% read test) was run on the both the VMs. The throughput (MBps) andIOPs are captured from the Iometer tool running on the VMs. Throughput was also captured by running esxtop on the two servers.Both these screenshots are shown below.The throughput on app04 and app05 VMs was observed to be 395MBps and 393MBps.13

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCDisk shares ratio of 1:1 (with SIOC enabled)With the same disk share ratio as in above sub-case, SIOC was enabled and the Iometer test was re-run on the VMs and thescreenshots for Iometer and esxtop were captured as shown below.The throughput on app04 and app05 VMs was observed to be 394MBps and 393MBps.14

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCDisk shares ratio of 2:1 (with SIOC disabled)In this sub-case the disk shares were set to “Normal or 1000 shares” on app04 and “High or 2000 shares” on app05 VM. As a result wehave a disk share ratio of 2:1 as shown in the below figure.With the disk shares set, Iometer was ran on both VMs and the throughput of about 393 MBps and 394 MBps was observed onapp05 and app04 respectively as shown in the following captured screenshots.15

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCDisk shares ratio of 2:1 (with SIOC enabled)With SIOC enabled the above test was re-run and the throughput from the Iometer tool was observed to be 504 MBps and 276MBps for app05 and app04 respectively as shown in the below screenshots. With SIOC enabled on the data-store the throughputnumbers are proportional to the disk shares of 2:1.For the usage model #2, the above four scenarios can be summarized in a simple table as follows:SIOCDisk shares ratioApp05 VMApp04 VMDisabled1:1393 MBps395 MBpsEnabled1:1393 MBps394 MBpsDisabled2:1393 MBps394 MBpsEnabled2:1504 MBps276 MBps16

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCThe key take away from this usage modelis that, when the VMs are on multiple/different VMware ESXi hosts, the diskshares distribution is enforced onlywhen SIOC is enabled on the commonshared data-store. This is due to the factthat SIOC provides data-store-wide diskscheduling that responds to congestionat the storage array, not just on thehost side HBA. This provides an ability tomonitor storage traffic and the prioritiesof all the virtual machines accessing theshared data-store. This is accomplishedby way of a data-store-wide distributeddisk scheduler that uses I/O shares pervirtual machine to allow a higher-priorityworkload to get better performance. Thedata-store-wide disk scheduler totals upthe disk shares for all the VMDKs that avirtual machine has on the given datastore. The scheduler then calculateswhat percentage of the shares thevirtual machine has compared to thetotal number of shares of all the virtualmachines running on the data-store.Test MatrixUsage Model #3: Enforcing Limit onPeak I/O Resource UsageThis usage model demonstratesperformance impact from IO interferencesuch as one seen from VM migration andhow I/O control can be used to manageinterference. The setup consists of asingle VMware ESXi 5.0 server hostingtwo virtual machines (app04 and app05)and then measuring the impact of disk I/Oof these virtual machines with the SIOCfeature enabled and disabled during themigration of one of the VMs to a secondhost. All tests are the two VMs (App04 &App05), both on the single VMware ESXi5.0 host and running Iometer. VMs weretested at 20GB VMDKs, 4 vCPUs, and10GB memory. Iometer is running a 1MBpayload size and 32 outstanding I/O pertarget. Then while App05 VM is migratedto the second host, IOPs are measured onApp04 before and during each test.As mentioned, adjustments to this SIOCexperiment were Share Value, Limit-IOPs, and Congestion Threshold. Eachadjustment consisted in four tests: SIOCenabled and disabled, and VM migrationoutbound (Tx) and inbound (Rx). Abaseline test was completed before anyadjustments were made to capture thedefault measurements. For each test,the IOPs were recorded prior to the VMmigration. This is the “Starting State”column of Test Matrix. While the VM wasmigrated, the lowest value or impactingvalue was recorded in order to show theimpact to existing running VMs. This valueis the “Impacted State” column of TestMatrix. Five adjustments plus the baselinewere recorded in this experiment: Share Value of 2:1 (1000:500) Share Value of 4:1 (2000:500) Share Value of 1:1 (1000:1000); LimitIOPs to 200 on migrating VM Share Value of 1:1 (1000:1000); LimitIOPs to 62 on migrating VM Share Value of 2:1 (1000:500); LimitIOPs to 62 on migrating VM17

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCSeveral key takeaways are evident fromthe above table. When the VMs are on thesame VMware ESXi host, the disk sharesdistribution is enforced whether SIOC isenabled or not on the shared data-store.When a VM is migrated, the storageworkload was impacted to the extent ofits share value setting. The more diversethe share setting, the more cycles aretaken from VM (App04) under test. Theadjustment of Limit-IOPs had the bestoverall impact of all tests; its rate-limitfunction was completely adjustable. Asnoted in the previous tests, VMs are onmultiple/different VMware ESXi hostsshare the disk distribution enforcementwhen SIOC is enabled. Additionally, as thedisk scheduler responds to congestionat the storage array it allows theVM a higher-priority and thus betterperformance.ConclusionsI/O control capabilities available on Intelserver platforms from VMware vSphereSIOC offers I/O prioritization to virtualmachines accessing shared storageresources. Administrators now have anew tool available to help them increasethe consolidation density, detects andmanage bottlenecks at the array onlywhen congestion exists and providesperformance protection while ensuringthat they will have peace of mind, knowingthat during periodic periods of peak I/Oactivity there will be a prioritization andproportional fairness enforced acrossall the virtual machines accessing thatshared resource. It allows administratorsto implement quality of service expressedfor storage workloads. This is a stepforward in the journey towards anautomated and policy-based managementof shared storage resources.More Intel/VMware referencearchitectures are available at the IntelCloud Builders Web site at www.intel.com/cloudbuilders.18Additional Usage Models UnderDevelopment:Flexible partition and service assuranceusing VMware ESXi traffic types andflows on a vNetwork Distributed Switch(vDS)NetIOC to provide users with thefollowing features: Isolation: ensure traffic isolation so thata given flow will never be allowed todominate over others, thus preventingdrops and undesired jitter Shares: allow flexible networkingcapacity partitioning to help users todeal with overcommitment when flowscompete aggressively for the sameresources Limits: enforce traffic bandwidth limiton the overall vDS set of dvUplinks Load-Based Teaming: efficiently use avDS set of dvUplinks for networkingcapacity

Intel Cloud Builders Guide: Storage I/O Control: 10Gb Intel Ethernet with VMware* vSphere 5.0* SIOCDisclaimers Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See www.intel.com/products/processor number for details.INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER,AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FORA PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITINGBY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHEREPERSONAL INJURY OR DEATH MAY OCCUR.Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked“reserved” or “undefined.” Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents whichhave an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or by visiting Intel’s Web site at www.intel.com.Copyright 2011 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Xeon inside, and Intel Intelligent Power Node Manager are trademarks of IntelCorporation in the U.S. and other countries.*Other names and brands may be claimed as the property of others.

instructions to demonstrate storage I/O control, run end-to-end on VMware* vSphere 5.0 and 10 Gigabit Ethernet (10GbE)1 on Intel server platforms. This addresses one of the key aspects of I/O control usage model requirements defined by Open Data Center Alliance*. Storage I/O Control solutions and 10GbE networks are critical elements of