The Xen Hypervisor And Its IO Subsystem - Mulix

Transcription

The Xen Hypervisor and its IOSubsystemvirtualizing a machine near youMuli Ben-Yehuda, Jon D. Masonmuli@il.ibm.com, jdmason@us.ibm.comIBM Haifa Research Lab, IBM Linux Technology CenterSystems and Storage Seminar 2005 – p.1/40

Table of ContentsThe Xen HypervisorQuick OverviewFull vs. Para-VirtualizationVirtualizing IOXen IOFrontends and BackendsDriver DomainsDirect Hardware AccessDMAIOMMUsIntroductionSoftware IOMMUsHardware IOMMUsSystems and Storage Seminar 2005 – p.2/40

The Xen HypervisorSystems and Storage Seminar 2005 – p.3/40

Quick OverviewOpen source hypervisor (aka virtual machine monitor),licensed under the GPLProvides secure isolation, resource control and QoSRequires minimal operating systems changes, and nouserspace changesSupports x86, x86-64, ia64 and PPC in varying degreesof maturitySupports Linux, NetBSD, FreeBSD, OpenSolaris, .Close to native performance!Supports live migration of VMsWidespread hardware support, including direct accessXen 3.0.0 just released!Systems and Storage Seminar 2005 – p.4/40

Quick Overview of Xen cont’Developed and maintained at the Cambridge UniversitySystems Research Group, by Ian Pratt, Keir Fraser, lotsof others.Contributions from Intel, AMD, HP, IBM, others.Commercial backing available fromhttp://www.xensource.com/, others.Read about Get the source: http://xenbits.xensource.com/Systems and Storage Seminar 2005 – p.5/40

BibliographyPapers and presentations are available /architecture.htmlXen and the Art of Virtualization, Paul Barham et al,SOSP 2003Xen and the Art of Repeated Research, Brian Clarket al, FREENIX 2004Safe Hardware Access with the Xen Virtual MachineMonitor, Keir Fraser et al, OASIS ASPLOS 2004workshopLive Migration of Virtual Machines, Christopher Clarket al, to be published at NSDI 2005Systems and Storage Seminar 2005 – p.6/40

Full VirtualizationFull virtualization refers to running an unmodified OS ona virtual machine (e.g. VMWare). There are many waysto do this - for example binary rewriting of the runningOS image.Hardware support makes full virtualization much easier.x86 is notoriously difficult to virtualize because someprivileged instructions simply fail silently, rather thanraising a trap.Newer Intel CPUs include virtualization support (VMX,VTx, VTi). AMD’s comparable Pacifica technology willdebut in Q1 2006.Systems and Storage Seminar 2005 – p.7/40

Para-VirtualizationPara-virtualization refers to modifying the OS to makevirtualization faster - modifying the OS to run on thevirtualized environment rather than on bare metal.Easy to do when you have the sourceCan be combined with full virtualization techniques para-virtualize where you can, use full-virtualizationtechniques where you can’t avoid it.XenoLinux - a port of Linux to run under the Xenhypervisor.The bulk of the work is replacing privileged instructions(e.g. cli, hlt, write to cr3) with hypervisor calls.Core concept: modify the OS to the virtualizedenvironment, but expose some details of the hardwarefor optimization.Systems and Storage Seminar 2005 – p.8/40

Virtualizing IOInstead of providing physical devices, provides virtualizedviews of them.The full-virtualization way: emulate real devices (manyways to do this, most common is a softwareimplementation of the hardware state machine)The para-virtualization way: class drivers and devicesWhy not expose physical devices? can’t do it securely, andcan’t do sharing, unless the device knows how to do it(PCI-SIG IOV group is working on it)Systems and Storage Seminar 2005 – p.9/40

Virtualizing IO - The Xen wayWhen Xen boots up, it launches dom0, the firstprivileged domain - currently has to be Linux 2.6, but intheory can be any other OS that has been properlymodifieddom0 is a privileged domain that can touch all hardwarein the system (long term goal is to move all hardwarehandling to dom0, we’re three quarters of the way there)dom0 exports some subset of the the devices in thesystem to the other domains, based on each domain’sconfigurationThe devices are exported as “class devices”, e.g. ablock device or a network device, not as a specific HWmodel.Systems and Storage Seminar 2005 – p.10/40

Xen IOSystems and Storage Seminar 2005 – p.11/40

Frontends and Backendsdom0 runs the backend of the device, which isconnected to each domain’s frontend for that devicenetback, netfront for network devices (NICs)blockback, blockfront for block devicesbackends and frontends communicate at a high leveldevice abstraction - block class, network class, etc. Thedomain doesn’t care what kind of block device it’stalking to, only that it looks like a block device.domains other than dom0 may be granted physicaldevice access, securely [as secure as the architectureallows, anyway]. This used to work in 2.0 but iscurrently broken in 3.0 and the unstable treeSystems and Storage Seminar 2005 – p.12/40

Frontends and Backends cont’Ultimately, all communication between frontends andbackends happens in memory. Xen provides severalmechanisms to make life interesting - err, easy - for driverdevelopers:shared memory, andproducer consumer rings, together withvirtual interrupts,give us event channels. but what about bulk data transfers? grant tables tothe rescue!how do we we tie it all up together?Systems and Storage Seminar 2005 – p.13/40

Driver Domainswe already saw that Xen gives one domain, dom0,access to all HW devices and other domains perform IOthrough itwhat if we could have multiple dom0’s? each with itsown devices and its own consumers (other domainsthat are accessing the hardware through it)with Xen 2.0, we can. In a sense, we run multipledrivers each in its own domain - hence “driver domains”.driver domains require the ability to hide PCI devicesfrom dom0 and expose them to other domains.unfortunately some of the changes to move hardwareinitialization out of the hypervisor to dom0 during the3.0 development cycle broke this functionality, and it’swaiting for a volunteer to fix it.Systems and Storage Seminar 2005 – p.14/40

Direct Hardware AccessOne of the main selling points of virtualization is machineconsolidation. So let’s assume for a second that you putyour database virtual machine and your web server virtualmachine on the same physical machine. Your databaseneeds fast disk access; your web server, fast networkaccess.Xen supports the ability to allocate different physical devicesto different domains (multiple “driver domains”). However,due to architectural limitations of most PC hardware, thiscannot be done securely. In effect, any domain that hasdirect hardware access has to be considered “trusted”.Systems and Storage Seminar 2005 – p.15/40

The Problem with Direct AccessThe reason why is that all IO is done in physical addresses.Consider the following case:domain A is mapped in 0-2GB of physical memorydomain B is mapped in 2-4GB of physical memorydomain A has direct access to a PCI NICdomain A programs the NIC to DMA in the 2-4GBphysical memory range, overwriting domain B’smemory. Ooops!The solution is a hardware unit known as an “IOMMU” (IOMemory Management Unit).Systems and Storage Seminar 2005 – p.16/40

DMASystems and Storage Seminar 2005 – p.17/40

DMA - OverviewDirect memory access (DMA) is the hardware mechanismthat allows peripheral components to transfer their I/O datadirectly to and from main memory without the need for thesystem processor to be involved in the transfer.From "Linux Device Drivers, 2nd Edition" By AlessandroRubini & Jonathan CorbetSystems and Storage Seminar 2005 – p.18/40

DMA - How it worksDevice Driver programs the adapter with physicalmemory address to DMA to/from.Adapter performs DMA on given addressAdapter signals device driver via interrupt that DMA iscompleteSystems and Storage Seminar 2005 – p.19/40

DMA - Problems?DMA limited to address the device can address (ie,32bit devices can only access 32bits of memory)Devices can DMA anywhere they want.Systems and Storage Seminar 2005 – p.20/40

IOMMUsSystems and Storage Seminar 2005 – p.21/40

IntroductionAn I/O Memory Management Unit (IOMMU) are hardwareconstructs that can be emulated in software.IOMMUs provides two main functions: Translation andDevice IsolationThe IOMMU translates memory addresses from “IOspace” to “physical space” to allow a particular device toaccess physical memory potentially out of its range. Itdoes this translation by providing an "in range" addressto the device and either translates the DMA accessfrom the "in range" address to the physical memoryaddress on the fly or copies the data to the physicalmemory address.Also, IOMMUs can limit the ability of devices to accessmemory, used for restricting DMA targets.Systems and Storage Seminar 2005 – p.22/40

Why do we need an IOMMU?Pros32bit DMA capable, non-DAC, devices can accessphysical memory addresses higher than 4GB.IOMMUs can be programmed so that the memoryregion appears to be contiguous to the device on thebus (SG coalescing).Device Isolation and other RAS features.ConsTANSTAAFL, "there ain’t no such thing as a freelunch." Remapping adds a performance hit to thetransfer (can be mitigated by a TLB).Systems and Storage Seminar 2005 – p.23/40

The Main Advantage - IsolationBut, the one we care the most about for virtualization isisolation. For isolation, it is not enough to translate we need a translation to be available to a given device,but not to some other devicewe also need to restrict which domain can programwhich deviceUnfortunately, not many of today’s IOMMUs can actually doisolation.Systems and Storage Seminar 2005 – p.24/40

Types of IOMMUsSoftwareLinux’s swiotlbXen’s grant tablesHardwareAMD GART (translation only)IBM TCEs (translation and isolation)AMD Pacifica (translation and isolation)Systems and Storage Seminar 2005 – p.25/40

swiotlbLinux includes swiotlb which is a softwareimplementation of the translation function of an IOMMU.Or we can just call it “bounce buffers”.Linux always uses swiotlb on IA64 machines, whichhave no hardware IOMMU, and can use it on x86-64when told to do so or when the machine has too muchmemory and not enough IOMMU.As of 3.0.0, Xen always uses swiotlb in dom0, sinceswiotlb provides machine contiguous chunks of memory(required for DMA) unlike the rest of the kernel memoryallocation APIs when running under Xen.Using swiotlb (or any other IOMMU) is completelytransparent to the drivers - everything is implemented inthe architecture’s DMA mapping API implementation.Systems and Storage Seminar 2005 – p.26/40

swiotlb WIPAt the moment, Xen has its own hacked-up copy ofswiotlb. We are working on patches to generalizex86-64’s dma-mapping code and merge Xen’s swiotlbback into the stock swiotlb.The swiotlb code is wasteful in memory, since it requiresa large physically contiguous memory aperture for thebounce buffers. The size of the aperture is configurableand ranges from several to hundreds of megabytes.Implementing support for multiple apertures willalleviate the need for pre-allocating so much memory.Systems and Storage Seminar 2005 – p.27/40

Grant TablesGrant tables are a way to share and transfer pages ofdata between domains. They give (or "grant") otherdomains access to pages in the system memoryallocated to the local domain. These pages can beread, written, or exchanged (with the proper permission)for the purpose of providing a fast and secure methodfor domains to receive indirect access to hardware.They are faster because driver domains are able toDMA directly into pages in the local domain’s memory,instead of having to DMA locally and copying or flippingthe page to the domain. However, it is only possible toDMA into pages specified within the grant table.Of course, this is only significant for non-privilegeddomains (as privileged domains could always accessthe memory of non-privileged domains).Systems and Storage Seminar 2005 – p.28/40

Grant Tables and IOMMUsGrant tables, like swiotlb, are a software implementationof certain IOMMU functionality. Much like how swiotlbprovides the translation functionality of an IOMMU,grant tables provide the isolation and protectionfunctionality. Together they provide (in software) a fullyfunctional IOMMU.Why not allow hardware acceleration if it currentlyexists?It should be possible to replace certain one or both ofthem with their hardware accelerated counterpart. Forexample, replacing swiotlb with AMD GART fortranslation, and still having grant tables to provide theprotection. Unfortunately, the current infrastructure inXen does not apply itself well to this and will require are-write.Systems and Storage Seminar 2005 – p.29/40

How Grant Tables Work - Shared PagesA driver in the local domain’s kernel will advertise apage to be shared (via thegnttab grant foreign access system call). This callnotifies the hypervisor that this page can be accessedby other domains. The local domain then passes agrant table reference ID to the remote domain it is"granting" access to this page. Once the remote domainis done, the local domain removes the grant (via thegnttab end foreign access call).This is used by block devices (and any other device thatreceives data synchronously).Systems and Storage Seminar 2005 – p.30/40

Transferred PagesAlso known as "Page flipping"Used by network devices (and any other device thatreceives data asynchronously)A driver in the local domain’s kernel will advertise apage to be transferred (via thegnttab grant foreign transfer call). This call notifies thehypervisor that this page can be received by otherdomains. The local domain then transfers the page tothe remote domain and takes a free page (viaproducer/consumer ring).Systems and Storage Seminar 2005 – p.31/40

Shared vs. Transfer, Why the difference?Block devices already know which domain requesteddata to be DMA’ed.Incoming network packets need to be inspected beforeit can be transferred.Newer networking technology (such as RDMA NICsand Infiniband) have the ability to DMA directly intodomUs, and will not need to transfer pages.Systems and Storage Seminar 2005 – p.32/40

AMD GARTAMD Graphical Aperture Remapping Table (GART)provides a basic, translation only, IOMMUImplimented in the on-chip memory controllerPhysical memory window and list of pages to betranslatedAddresses outside the window are not translatedFully supported in Linux; Xen support is WIPSystems and Storage Seminar 2005 – p.33/40

Pacifica IOMMUPacifica IOMMU also provides translation and isolation.Translation is done via the AMD GART and isolation isprovided via the Pacifica northbridge. Northbridgeverifies DMAs against the DEV (Device ExclusionVector) defining the access permissions for the device.If it fails, it returns all 1s for a read or suppresses thestore on a write and returns a Master Abort.Each "protection domain" has a device exclusion vector(DEV) that specifies the per-page access rights of thedevices in that domain. Each bit in the DEVcorresponds to one 4kB page in physical memory.DEV checks are applied before the addresses aretranslated via GART.Pacifica IOMMU is very similar to Calgary TCEs.Systems and Storage Seminar 2005 – p.34/40

Calgary TCEsCalgary’s Translation Control Entry (TCE) providesfunctionality to translate and isolateProvides a Unique I/O Address space to all devicesbehind each PCI Host Bridge (PHB)Rather than allowing DMA devices to access memorydirectly, I/O translation uses DMA address as indexesinto a system controlled translation table in memory.Anything not in this table can not be accessed. Thisgives it the ability to protect domains from errant orhostile DMA requests.Currently implemented in IBM’s pSeries servers.Systems and Storage Seminar 2005 – p.35/40

SummaryAt the moment, Xen does not support any HW IOMMU.We are working on it!Due to Xen’s architecture, a large chunk of the work isactually in Linux’s dom0 kernel, e.g. to support multipleIOMMUs on x86-64 cleanly.And we have a few open questions, too:How do we squeeze the most performance out ofsystems with IOMMUs? what are the right interfaces?How can we design better IOMMUs?Systems and Storage Seminar 2005 – p.36/40

Questions?Systems and Storage Seminar 2005 – p.37/40

BackupSystems and Storage Seminar 2005 – p.38/40

Direct Hardware AccessOne of the main selling points of virtualization is machineconsolidation. So let’s assume for a second that you putyour database virtual machine and your web server virtualmachine on the same physical machine. Your databaseneeds fast disk access; your web server, fast networkaccess.Xen supports the ability to allocate different physical devicesto different domains (multiple “driver domains”). However,due to architectural limitations of most PC hardware, thiscannot be done securely. In effect, any domain that hasdirect hardware access has to be considered “trusted”.Systems and Storage Seminar 2005 – p.39/40

The Problem with Direct AccessThe reason why is that all IO is done in physical addresses.Consider the following case:domain A is mapped in 0-2GB of physical memorydomain B is mapped in 2-4GB of physical memorydomain A has direct access to a PCI NICdomain A programs the NIC to DMA in the 2-4GBphysical memory range, overwriting domain B’smemory. Ooops!The solution is a hardware unit known as an “IOMMU” (IOMemory Management Unit).Systems and Storage Seminar 2005 – p.40/40

Xen and the Art of Virtualization, Paul Barham et al, SOSP 2003 Xen and the Art of Repeated Research, Brian Clark et al, FREENIX 2004 Safe Hardware Access with the Xen Virtual Machine Monitor, Keir Fraser et al, OASIS ASPLOS 2004 workshop Live Migration of Virtual Machines, Christopher Clark