Transcription
ISSNSM — International Summer School on Network and Service Management2nd ISSNSM’s Tutorial onUsing Xen Virtualization inResearch Projects(Tutorial T3)Speaker:Kyrre BegnumJune 4, 2008
ISSNM program chaired by Burkhard Stiller, David Hausheer, University of ZürichISSNM laboratory organization chaired by Cristian Morariu, Peter Racz, University of Zürich
Virtualization with XenKyrre Begnum, Oslo University Collegekyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008It’s in the newskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Used in a wide range of areas Server consolidation Personal convenience Predictable computingenvironment Research and DevelopmentEducation and trainingHigh PerformanceComputingProduct demosVirtual hostingLegacy application/environment supportkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Used in a wide range of areas Server consolidation Personal convenience Predictable computingenvironment Research and DevelopmentEducation and trainingHigh PerformanceComputingProduct demosVirtual hostingLegacy application/environment supportYou are here!kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Typical scenariosMore systemson the LANBusiness-likenetworkRouting networkkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008What you should know afterthis course How to install Xen on a Debian-based system Building and running virtual machines Sufficient knowledge to be independent of toolsand build your own solutionThe fundamental components and under-the-hoodfunctioning of XenSetting up networks and connecting virtualmachines togetherkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Basicskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Xen OpenSource virtual machine monitor as part of alarge architecture for dynamic service redeployment Became famous with “Xen and the art ofvirtualization” in 2004 Has become popular for data-centers where largenumbers of virtual machines run on powerfulservers Small entourage of commercial server-sidemanagement tools Bought by Citrix - lets keep our fingers crossed!kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Xen terminology All virtual machines are called domains The other virtual machines are called DomU (Userdomains) The thin layer between the hardware and all thedomains is called the HypervisorThe host operating system is also a virtual machinecalled Domain0, or Dom0ISSNSM 2008Using Xen Virtualization in Research Projectskyrre.begnum@iu.hio.noXen LUMLMain OSAppsAppsAppsVMVMVMwareServer / WorkstationAppsAppsAppsServiceconsoleVMVMMain OSXen HypervisorVMware de LinuxVMware ServerVMware ESX Serverkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
The components of a virtualmachine1. Filesystem - This can be a filesystem image, or diskimage.They are usually pre-built.2. Configuration file - The name, hardware andnetwork connections are defined herekyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008The kernel is outside Paravirtualized guests have their kernel outside ofthe filesystem This goes for the initrd image as well The kernel modules must be on the inside of thefilesystemUsually one uses the same kernel and initrd asdom0, but that can cause problems when usingdifferent distributionskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Installing Xen One has three choices: Install binary package from the distributionInstall binary package from XenCompile from sourceEach approach has benefits and drawbacksXen is still under heavy development, and tends tochange between releasesUsing Xen Virtualization in Research Projectskyrre.begnum@iu.hio.noISSNSM 2008Xen 3.1 on Debian 4.0Commands:apt-get install bridge-utils libc6-xenwget n.tgz/xen-3.1.0-install-x86 32.tgztar xzf xen-3.1.0-install-x86 32.tgzcd dist./install.shdepmod -amkinitramfs -o /boot/initrd.img-2.6.18-xen 2.6.18-xenAdd to /boot/grub/menu.lsttitle Xen 3.1.0 / XenLinux 2.6kernel /boot/xen.gz console vgamodule /boot/vmlinuz-2.6.18-xen root /dev/hda1 ro console tty0module sing Xen Virtualization in Research ProjectsISSNSM 2008
Starting Xen Reboot (cross your fingers! Press your Thumbs! etc. )Choose the Xen hypervisor and kernelWhen booted up, run (as root):/etc/init.d/xend startCheck status with:xm listkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Filesystems Paravirtualization relies on partitions with readymade filesystems (cloning approach) Booting with a CD image and installing from it isnot possible (in PV) It is the filesystem which decides the distributionof the VM Most tools for creating these filesystems arespecific to classes of distributionskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Creating FilesystemsOn a Debian-based system using xen-toolsapt-get install xen-toolsxen-create-image --size 2Gb --hostname first --dir . --dhcp \--kernel /boot/vmlinuz-2.6.18-xen --initrd /boot/initrd.img-2.6.18-xen \--debootsrtap --dist etch --mirror http://ftp.no.debian.org/debianFetch the actual filesystemcp domains/default/disk.img etch.ext3In principle, you should be able to build filesystemsfrom different distribution using xen-toolskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Configuration Files The configuration file defines the “hardware”makeup of the virtual machine Here, you define memory, network cards, harddrives, number of CPUs and more In Xen, the configuration file is actually Pythoncode, which is very picky on syntax! The benefit of using Python, is that one can“program” the configuration and make it moredynamic. (Just like PHP for web-pages)kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Writing a configuration fileA simple VM:# -*- mode: python; -*kernel "/boot/vmlinuz-2.6.18-xen"memory 64disk [ 'file://root/etch.ext3,hda1,w' ]root '/dev/hda1'extra '2'name “etch”vif [ '' ] Note the declaration of the disk with the URI of thefilesystem, device and permission Memory is specified in Megabytes without “M” or “MB”The kernel can point to any Xen-compatible kernel‘vif’ is a list of network interfaces (only one is configuredhere)Using Xen Virtualization in Research Projectskyrre.begnum@iu.hio.noISSNSM 2008Common commands Starting and stoppingxm create [-c] etch.cfg, xm shutdown etch Statusxm [--long] list, xentop Emergenciesxm destroy etch Console accessxm console etchNotice, that you use the configuration file for starting,but use the VM name for the rest.kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
&Using Xen Virtualization in Research Projectskyrre.begnum@iu.hio.noISSNSM 2008Useful GRUB optionstitlerootkernelmodulemodulesavedefaultboot Ubuntu, Xen 3.1, kernel 2.6.18(hd0,0)/boot/xen.gz noreboot console vga dom0 mem 512M/boot/vmlinuz-2.6.18-xen root /dev/sda1 ro quiet splash/boot/initrd.img-2.6.18-xenwill stop the hypervisor fromautomatically rebooting in case of problemsnoreboot dom0 mem 512Mwill only assign 512MB of Ram fordomain0, reserving the rest for virtual machineskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Xend configuration file Located in /etc/xen/xend-config.sxp Limit the minimum memory for dom0: You need to restart the xend daemon after editing:To adjust the networking, edit:(network-script 'network-bridge bridge xenbr0')(vif-script 'vif-bridge bridge xenbr0')(dom0-min-mem 196)/etc/init.d/xend restartkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008SMP virtual machines# -*- mode: python; -*kernel "/boot/vmlinuz-2.6.18-xen"memory 64disk [ 'file://root/etch.ext3,hda1,w' ]root '/dev/hda1'extra '2'name “etch”vif [ '' ]vcpu 2 Virtual machines get virtual CPUs One can pin down a VCPU to a real CPU, but thismight do more damage than goodThe virtual CPUs are moved around between thereal CPUskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Performance CPU usage scales about linear with the number ofvirtual machines Networking scales a bit worse, but not badDisk scales worst, but it is hardest to predict-File images can perform very fast, if the VM isalone and dom0 has a lot of memory-Disk partitions (LVM) perform OK, but scalebetter than fileskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Scheduling of domains Xen offers basic credit scheduling for virtualmachines as default More weight means more CPU-timeA maximum CPU cap can be added as wellSufficient for most scenariosUse xm sched-credit to view credit policyChange the credit like thisxm sched-credit -d domain [-w weight] [-c cpucap]kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Utilizing the configuration file# -*- mode: python; -*name 'myvm'memory 64# Rest of code goes herekernel "/boot/vmlinuz-2.6-xen"disk [ 'file://root/' name '.img,hda1,w' ]root '/dev/hda1'extra '2 'vif [ 'bridge xenbr0' ] Since the configuration file basically is Pythoncode, we can utilize this to create dynamicconfigurations and limit errors The example above re-uses the name of the VM inorder to point to the right filesystemUsing Xen Virtualization in Research Projectskyrre.begnum@iu.hio.noISSNSM 2008Working with differentdistributions There are few tools which can create filesystemsbelonging to other distributions Sites like http://jailtime.org offer downloadablefilesystems for many free distributions (great fortesting) The problem is usually the kernel and initrd (morespecifically, the xen kernel modules)kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Saving and restoring VMs Xen offers two methods for freezing a domain:-Pause.The VM exists in xm list and usesresources, but is not running:xm pause domid xm unpause domid -Save.The VMs memory is saved to a file and canlater be restored.xm save domid state-file xm restore state-file This is not entirely a snapshot, you need tomake a copy of the disk, as wellkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008HVM - on supported platforms HVM means we provide hardware emulation tothe virtual machine This is useful for operatingsystems which do notsupport paravirtualization Recent CPUs and motherboards support HVMkyrre.begnum@iu.hio.noIntel VT / AMD-VYou can test if you have HVM support:xm dmesg grep HVMUsing Xen Virtualization in Research ProjectsISSNSM 2008
Installing a HVM domain HVM domains can be installed from a CD image, inwhich case they need an empty disk-file A HVM filesystem is a disk-image, not a filesystemimage Xen HVM uses QEMU for device emulation, whichperforms slower than paravirtualization HVM domains can be accessed using SDL or VNCkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Installing a HVM domain# -*- mode: python; -*kernel "/usr/lib/xen/boot/hvmloader"builder 'hvm'usb 1usbdevice 'tablet'boot 'd'vnc 1vncviewer 0vncunused 0device model '/usr/lib/xen/bin/qemu-dm'vncpasswd 'akes9womb'memory 256disk [ nXP.iso,hdd:cdrom,r’ ]root '/dev/hda1'extra '2'name 'windows.os10'vif [ 'type ioemu,bridge lan.os10']kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Installing a HVM domain# -*- mode: python; -*kernel "/usr/lib/xen/boot/hvmloader"builder 'hvm'usb 1usbdevice 'tablet'boot 'd'vnc 1vncviewer 0vncunused 0device model '/usr/lib/xen/bin/qemu-dm'vncpasswd 'akes9womb'memory 256disk [ nXP.iso,hdd:cdrom,r’ ]root '/dev/hda1'extra '2'name 'winXP'vif [ 'type ioemu,bridge lan.os10']Change to ‘c’after installationkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Live migration Enables virtual machines to move between serverswith no downtime Live migration is very fast, but depends on theamount of memory assigned to the VM There are two important dependancies:-The two servers have the same CPUarchitecture-Both servers have concurrent access to thefilesystemkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Enabling live migration Accept domains from other serversIn /etc/xen/xend-config.sxp(xend-address '')(xend-relocation-address '')(xend-relocation-hosts-allow '') Restart the xend daemon Live migration of a VM can be done like this/etc/init.d/xend restartxm migrate --live domid new-server kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Performance tips Create a swap file inside the VM filesystemUse separate disk for VM filesystemsMake sure dom0 runs little / few servicesDisable screensavers!Ordinary performance tools will show “ALL OK”even if the dom0 is struggling, look for alternativemeasureskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
PCI-passthrough Xen supports PCI-passthrough, which will give avirtual machine direct, exclusive access to ahardware device on the PCI-bus. This is useful for hardware testing on multipledistributions Support is not superb at the moment, but shouldwork with a little bit of mailing-list readingkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Exerciseskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Basics4. Are the virtual machinesonline?1. Install Xen2. Create three virtualmachines5. Look for more filesystemson http://jailtime.org3. Start all three virtualmachines and connect totheir consolekyrre.begnum@iu.hio.no6. Start all three virtualmachines and connect totheir consoleUsing Xen Virtualization in Research ProjectsISSNSM 2008&1.2.Install hdparm and runhdparm -Tt /dev/hda1What results do you get and what if yourun it on two and three VMs at the sametime?5.Create an SMP virtual machine and re-dosome of the tests.6.For a quick CPU benchmark, run:For a quick CPU benchmark, run:time echo "1234567 123456" bcTry on one VM, dom0 only and on several atthe same time.Write down your results.time echo "1234567 123456" bcTry on one VM, dom0 only and on severalat the same time.Write down your results.3.Use the credit scheduler to give onedomain far less weight. Recreate the CPUtest from above.Were you able to influencethe end-result?4.Edit /boot/grub/menu.lst and limit thememory of dom0kyrre.begnum@iu.hio.no7.Use the credit scheduler to give one domain acpucap. Recreate the CPU test from above.Discuss the benefits / drawbacks of weight vs.cpucap.8.Experiment with saving and restoring domainsUsing Xen Virtualization in Research ProjectsISSNSM 2008
Part II - Networking kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Networking All virtual machines are either connected to the LAN orto local bridge devices Bridge devices function like layer two switches Traffic shaping and QoS is less supportedOne can create arbitrary many bridges and createcomplex network topologiesIdeal point to “tap” for network monitoringkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Bridge devices Bridge devices can be created using the brctl commandbrctl addbr myswitch Virtual machines are connected to the switches in the configuration filevif [ ‘bridge myswitch’ ] The switch has to be created before the virtual machine is startedStatus can using brctl showAccess to the LAN is through the bridge device xenbr0 (renamed afterXen 3.2)kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Several network cards# -*- mode: python; -*kernel "/boot/vmlinuz-2.6.18-xen"memory 64disk [ 'file://root/etch.ext3,hda1,w' ]root '/dev/hda1'extra '2'name “etch”vif [ 'bridge xenbr0',’bridge myswitch’ ] Network cards are added to the vif arrayThe order of the declaration mirrors the order ofthe devices (eth0, eth1 etc.)kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
The network card seen fromdom0 Every NIC has a corresponding interface on dom0 When collecting data from it using ifconfig,remember to switch RX and TX You can manually connect and disconnect theseinterfaces to other bridge-devicesThe syntax for the interface isvif[domid].[nic]kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Setting a MAC address# -*- mode: python; -*kernel "/boot/vmlinuz-2.6.18-xen"memory 64disk [ 'file://root/etch.ext3,hda1,w' ]root '/dev/hda1'extra '2'name “etch”vif [ 'bridge xenbr0,mac 00:16:3E:51:89:6A'] Some software licenses use the MAC address forauthentication Xen can have arbitrary MAC addresses for itsdomains (unlike VMware) Changing the MAC address may confuse somedistributionskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Additional disks# -*- mode: python; -*kernel "/boot/vmlinuz-2.6.18-xen"memory 64disk [ k,hdb1,w' ]root '/dev/hda1'extra '2'name “etch”vif [ 'bridge xenbr0’ ] Disks are added to the disk array Disks can only be shared if defined as read-onlyNotice the different URI of the block-device ratherthan the fileNote, that the device node needs to be present onthe VM in order to use the diskkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Creating empty filesystem files Disks can be created as simple filesdd if /dev/zero of /root/xendisk bs 1M count 1400mkfs.ext3 /root/xendisk# optionalmount -o loop /root/xendisk /mnt They can actually be added in run-time:xm block-attach etch file://root/xendisk /dev/hdc roNext, inside the running VM:mknod /dev/hdc b 22 0mount /dev/hdc /mntkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Adding software to the VMwhile offline Mount the filesystemmount -o loop etch.ext3 /mnt Chroot into the mounted filesystemchroot /mnt bash Install packages and edit filesUnmount the filesystemumount /mntkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Automation In research projects, you need to re-runexperiments a number of times Creation and re-creation of virtual machines mayscale badly with the number of VMs and theirnetwork topology Tools like MLN http://mln.sourceforge.net canfunction as an abstraction layer above Xen Look for these kinds of solutions or write yourown once you’ve decided to stick with Xenkyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Topics we left untouched What if you want to configure parameters of thefilesystem itself (users, software packages,network, etc. )? Xen can change the number of CPUs and amountof memory ONLINE! VMs can also grow network cards (and simulateroaming) MonitoringInstalling Xen from distribution packageskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Exerciseskyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
Networking 1. Create a star topology consistingof three virtual machines2. Make one of the virtual machinesbecome a gateway with twonetwork cards3. Create an empty filesystem andadd it to a running VM4. VM PING-PONG!1. Enable live migration (shown on one of theslides)2. Download ttylinux from:http://legolas.iu.hio.no/ttylinux xen.tar.gz3. Edit the name of the VM so that it has yourname4. Send the VM to other machines when they areready!kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008Thank you :)kyrre.begnum@iu.hio.noUsing Xen Virtualization in Research ProjectsISSNSM 2008
kyrre.begnum@iu.hio.no Using Xen Virtualization in Research Projects ISSNSM 2008 Basics kyrre.begnum@iu.hio.no Using Xen Virtualization in Research Projects ISSNSM 2008 Xen OpenSource virtual machine monitor as part of a large architecture for dynamic service re-deployment Became famous with "Xen and the art of virtualization" in 2004 Has become popular for data-centers where large
