WIXLIB

Avishay TraegerFAST 2012IBM Research – HaifaFebruary 16, 2012Adding Advanced Storage ControllerFunctionality via Low-Overhead VirtualizationMuli Ben-Yehuda, Eran Borovik, Michael Factor, Eran Rom,Avishay Traeger, Ben-Ami Yassour 2009 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationMotivationNew storagecontrollerrequirements2New functions 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationMotivationNew storagecontrollerrequirementsNew functionsFile Serving3 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationMotivationNew storagecontrollerrequirementsNew functionsFile ServingIn-line compression4 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationMotivationNew storagecontrollerrequirementsNew functionsFile ServingIn-line compressionDatabase5 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationMotivationNew storagecontrollerrequirementsNew functionsFile ServingIn-line compressionIn-line dedupDatabase6 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationMotivationNew storagecontrollerrequirementsNew functionsOften, these functions already exist7 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationMotivationNew storagecontrollerrequirementsNew functionsWhat is the best method for adding the new function?8 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationFirst Method: Deep IntegrationProsBase FunctionOS Good performance Little hardware overheadConsBaseNewFunction FunctionOS9 Time-consuming integration Varying OS versions Difficult resource management– base function assumes adedicated system Core function vulnerable tobugs Dual maintenance 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationSecond Method: External GatewayProsBase FunctionOS Quick integration Protection of core functionConsNew FunctionOS Higher communicationoverheads Expensive hardware costs(CAPEX OPEX)Base FunctionOS10 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationOur Method: VM on the storage controllerProsBase FunctionOS Quick integration Little hardware overhead Protection of core functionCons BaseNewFunction FunctionOS11“VMs have high overhead forI/O – I don't want that on mycritical path!” 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationObservationA storage controlleris a specialpurpose machinewith finely tunedresource controlVMs provide all the features weneed and some that we don'tNeed: fault isolation, resourceisolation, dual environmentsDon't need: resource sharing,over-commit, migrationConclusion:customize the VMbehavior for ourneeds12 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationExternal and Internal CommunicationExternalInternalClients New function (VM)New function (VM) Controller Device assignment SR-IOV virtio block device Fast - bypass the host Fast - shared memory Exits for interrupts Exits for submitting I/Os Exits for IOCs Exits for interrupts Exits for IOCs13 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationI/O Path1GuestHost(KVM Process)(Controller Process)Core 1: File ServerCore 3: Controller3File ServerStorage Controller64Put block requestin shared memoryPut completionin shared memory7Core 2:Guest Polling ThreadCore 4:Host Polling Thread8Poll blockcompletionsPoll file serverblock requests2Poll NIC driver(Poll other sources)5Device assignment14 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationCPU and MemoryStatically allocate CPU cores and memoryBoot guest kernel with idle pollUse HugePages for backing the guest's memoryModify thread priorities and affinities15 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationExperimental Setup Used two servers, each with–CPU: 2 quad-core 2.93GHz Intel Xeon 5500 (w/ EPT)–RAM: 16GB–Ethernet adapter: Emulex OneConnect 10GbE Servers directly connected with 10GbE One server was the load generator, other was our (emulated)storage controller Controller server used 4 cores, unless otherwise specified–VM tests: guest got 2 cores and 2GB RAM–Bare metal tests: host got all cores and RAM Storage back-end: 8GB ramdisk via loopback–Physical disk doesn't become the bottleneck–Assignment of I/O to specific cores (similar to real controller)16 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationNetwork Latency: Ping FloodNo pollingPolling17Bare metalGuest (halt)24 µs21 µs89 µs21 µs 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationNetperf: Request-Response Throughput18 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationNetperf: TCP Send Throughput19 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationNetperf: TCP Receive Throughput20 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationBlock Latency: 4KB Sync WritesTotal LatencyAdded Latency21InitialOptimized50 µs49 µs15.9 µs6.6 µs 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationBlock Throughput: 4KB Sync 5x 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationFile Server: 4KB Read Cache Miss (6 cores)23 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationFile Server: 4KB Read Cache Miss (4 cores)24 2012 IBM Corporation

Conclusions It is feasible to use a virtual infrastructure to integrate newfunctions into a storage controller We demonstrated a set of mechanisms and techniquesthat achieve near zero performance overhead Benefit from performance and hardware cost of deepintegration Benefit from shorter time to market, isolation, and simplerdevelopment model of the gateway approachFuture work:1.Guest polling thread optional while keeping overheadsnear zero (ELI – ASPLOS 2012)2.Benchmark multiple VMs on one host25 2012 IBM Corporation

Backup26 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationBackground: KVM Open source, Linux-based hypervisor Leverages Intel VT-X or AMD-V features to virtualize the CPU Minimalistic hypervisor–VMs look like regular processes–Uses Linux's existing infrastructure (memory manager, scheduler, etc.) Asymmetric modelRegularProcessGuest OSQEMUuser-spacekernelKVM kernel module27 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationBackground: Emulated I/O QEMU may emulate in software–BIOS–PCI bus–USB bus–Standard set of devices (IDE, SCSI, network) Guest OS uses its native drivers to access these devices Guest OS not aware of emulation – no guest changes required! Poor performance – each access to the device must be trappedand emulated “world switches” AKA “exits”28 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationBackground: virtioUses para-virtualized drivers –Guest OS uses drivers that are “aware” that the OS is virtualizedand cooperate with the hypervisor to improve performance Up to 3 exits per I/O–PIO for I/O submission (may be batched)–Interrupts–IOCsGuestvirtio front-endVMMHW29virtio back-enddriverdevice 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationBackground: Device Assignment“Assigns” device to guest –Requires either dedicated device or SR-IOV Guest driver submits I/O directly to device Interrupts delivered by hypervisor – two exitsGuestdriverVMMHW30ISRdevicev. APICAPIC 2012 IBM Corporation

Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationBackground: Guest Execution As long as there are no exits,stay in guest mode and runwith minimal overhead Exit handling can be relativelyfast if not I/O–e.g., interrupts delivery indevice assignment Large overhead if VMM musthandle I/O during exit–virtio–Emulation31 2012 IBM Corporation

storage controller Controller server used 4 cores, unless otherwise specified -VM tests: guest got 2 cores and 2GB RAM -Bare metal tests: host got all cores and RAM Storage back-end: 8GB ramdisk via loopback -Physical disk doesn't become the bottleneck -Assignment of I/O to specific cores (similar to real controller)