WIXLIB

Educational Virtual Clusters for Ondemand MPI/Hadoop/Condor inFutureGridRenato FigueiredoPanoat Chuchaisri, David WolinskyACIS Lab - University of FloridaAdvanced Computing and Information Systems laboratory

A flexible, extensible platform for handson, education on parallel and distributedsystemsFocus on usability – lower entry barrier Plug and play, open-sourceVirtualization virtual networking tocreate educational sandboxes Virtual appliances: self-contained, pre packaged execution environmentsGroup VPNs: simple management of virtualclusters by students and educatorsAdvanced Computing and Information Systems laboratory2

Guiding principles Full-blown, pre-configured, plug-and-playwell-known middleware stacks Condor – high-throughput, workflows MPI – parallel Hadoop – data parallel, Map/Reduce Quick user start – minutes to first job Isolated sandbox clusters; flexibility Gain access to FutureGrid, or use desktop VMs Shared playground cluster Allow individuals, groups complete control overtheir virtual cluster - root accessAdvanced Computing and Information Systems laboratory3

Outline Overview Deploying an appliance and connectingto playground virtual cluster Virtual appliances, networks FutureGrid resources FutureGrid accounts Condor self-configuration Deploying MPI parallel jobs Deploying Hadoop poolsAdvanced Computing and Information Systems laboratory4

What is a virtual appliance? An appliance that packages softwareand configuration needed for a particularpurpose into a virtual machine “image”The virtual appliance has no hardware –just software and configurationThe image is a (big) fileIt can be instantiated on hardware Desktops: VMware, VirtualBox Clouds: FutureGrid, Amazon EC2Advanced Computing and Information Systems laboratory5

Grid appliances Baseline image: self-configures CondorApplianceimageA Condor nodeAnother Condor nodeinstantiatecopyVirtualizationLayerRepeat Advanced Computing and Information Systems laboratory6

Virtual network, configuration P2P overlay used to self-organize virtualprivate network (VPN) of appliances Akin to Skype Virtual cluster; assign IP addresses in virtualspace through DHCP – support existingmiddleware (Condor, MPI, Hadoop) P2P overlay also used to self-configuremiddleware Akin to Bonjour/UPnP Condor manager advertises itself; Condorworkers discover and register with managerAdvanced Computing and Information Systems laboratory7

FutureGrid gAdvanced Computing and Information Systems laboratoryimage8

Using FutureGrid – accounts 101 Create a portal account Can access and post content, manage profile Identity verification – no resources allocated,but users can interact with portal E.g. cloud computing class community page Create or join a project Request needs to be authorized, and resources grantedPortal users can then be added to the project E.g. a cloud class instructor submits a projectrequest; students request portal accounts;instructor uses portal to add students to classAdvanced Computing and Information Systems laboratory9

Web site – FG accountAdvanced Computing and Information Systems laboratory10

Using FutureGrid – cloud 101 Once a user has a portal account andproject, he/she can use Nimbus orEucalyptus to instantiate appliances onthe different FutureGrid Clouds Tutorials show steps to deploy applianceswith a single-line command Refer to portal.futuregrid.org Under “User information”: Getting started – to get accounts Using Clouds – Nimbus, Eucalyptus Pointers to relevant tutorialsAdvanced Computing and Information Systems laboratory11

User perspective – first steps Deploying the baseline Grid appliance: Nimbus: cloud-client.sh --run --name gridappliance-2.04.29.gz --hours 24 Eucalyptus: euca-run-instance -k mykey -t c1.medium emiE4ED1880 Wait a few minutes ssh root@machine-address You are connected to a pre-deployed‘playground’ Condor cluster condor statusAdvanced Computing and Information Systems laboratory12

Joining Condor poolJoin P2P networkGet DHCP addressDiscover Condor managerSharedPlaygroundcloud client.shAdvanced Computing and Information Systems laboratory13

User perspective – running MPI User can install MPI on their appliance “Vanilla” MPI – just run a script to build Advanced classes - user can also deploy customMPI stacks Condor is used to bootstrap MPI rings ondemand with help of a script Takes executable and number of nodes Dispatches MPI daemons as Condor jobs Waits for all nodes to report Creates configuration based on nodes Submits MPI task Nodes auto-mount the MPI binaries over NFSAdvanced Computing and Information Systems laboratory14

MPI dynamic poolsmpi submit.py –n 4 HelloWorldSharedPlaygroundNFS read-onlyautomount–n 2 HelloWorldNFS read-onlyautomountAdvanced Computing and Information Systems laboratory15

User perspective – running Hadoop User can install Hadoop on their appliance “Vanilla” Hadoop – pre-installed Advanced classes - user can also deploy customHadoop stacks Condor is used to bootstrap Hadoop pools Takes number of nodes as input Dispatches namenodes, task trackers Waits for all nodes to report Creates configuration based on nodes Nodes auto-mount the Hadoop binaries over NFS After pool is configured, submit tasks, useHadoop HDFSAdvanced Computing and Information Systems laboratory16

Hadoop dynamic pools - createhadoop condor.py –n 4 startSharedPlaygroundNFS read-onlyautomount–n 2 startNFS read-onlyautomountAdvanced Computing and Information Systems laboratory17

Hadoop dynamic pools - runhdfs dfsadminhadoop jar app1 argsSharedPlaygroundNFS read-onlyautomounthdfs dfsadminhadoop jar app2 argsNFS read-onlyautomountAdvanced Computing and Information Systems laboratory18

Hadoop dynamic pools - teardownhadoop condor.py –n 4 stopSharedPlaygroundNFS read-onlyautomount–n 2 startNFS read-onlyautomountAdvanced Computing and Information Systems laboratory19

One appliance, multiple ways to run Allow same logical cluster environmentto instantiate on a variety of platforms Local desktop, clusters; FutureGrid; EC2 Avoid dependence on host environment Make minimum assumptions about VM andprovisioning software Desktop: VMware, VirtualBox; KVM Para-virtualized VMs (e.g. Xen) and cloud stacks –need to deal with idiosyncrasies Minimum assumptions about networking Private, NATed Ethernet virtual network interfaceAdvanced Computing and Information Systems laboratory20

Creating private clusters The default ‘playground’ environmentallows new users to quickly get startedUsers and instructors can also deploytheir own private clusters The Condor pool becomes a dedicatedresource Same appliance – what changes is aconfiguration file that specifies whichvirtual cluster to connect toWeb interface to create groupsAdvanced Computing and Information Systems laboratory21

Web site – GroupVPNAdvanced Computing and Information Systems laboratory22

Deploying private virtual poolsDedicatedVirtualpoolStudent 1Student 2cloud client.sh –n 7upload groupVPN configurationAdvanced Computing and Information Systems laboratory23

Summary Hands-on experience with clusters is essentialfor education and trainingVirtualization, clouds simplify softwarepackaging/configurationGrid appliance allows users to easily deployhands-on virtual clustersFutureGrid provides resources and cloudstacks for educators to easily deploy their ownvirtual clustersTowards a community-based marketplace ofeducational appliancesAdvanced Computing and Information Systems laboratory24

Thank you! More information: http://www.futuregrid.org http://grid-appliance.orgThis document was developed with support from theNational Science Foundation (NSF) under Grant No.0910812 to Indiana University for "FutureGrid: AnExperimental, High-Performance Grid Test-bed." Anyopinions, findings, and conclusions or recommendationsexpressed in this material are those of the author(s) and donot necessarily reflect the views of the NSFAdvanced Computing and Information Systems laboratory25

Advanced Computing and Information Systems laboratory26

Advanced Computing and Information Systems laboratory 3 Guiding principles Full-blown, pre-configured, plug-and-play well-known middleware stacks Condor - high-throughput, workflows MPI - parallel Hadoop - data parallel, Map/Reduce Quick user start - minutes to first job Gain access to FutureGrid, or use desktop VMs