Transcription
User-defined Clusters - Introduction to theRocks Cluster Toolkit : Merging Cloudsand ClustersPhilip PapadopoulosUniversity of California, San Diego &San Diego Supercomputer Center
Topicsy QuickIntro to Rocksy Different Types of Cloud Structures –Which ones will make a difference?y Clouds for HPC It’s not about the performancey It’sMy cluster. Campus cloud provisioning. Extending MyCluster using Rocks.
Rocks – www.rocksclusters.orgyyyyLeading Open Source Cluster DistributionActive development for 8 yearsDemonstrated Scalability from 2 – 1000 nodes.Several 1000 Clusters Worldwide 1800 Users on our discussion list About 650 Messages/Month Commercial support availableyyyRedhat Enterprise 4/5 compatible.Solaris Support in Alpha (Target Thumper Storage)Core to many NSF/NIH-funded grants Funding from NSF for core through 20072010UC Regents3
And Rocks Supports More than BeowulfyTiled Display/Cluster of GPUs OpenGL machine Not an MPI machine SAGE DMX ChromiumStandard MPI Clustersy End-user defined specializedy 2007 UC Regents4
Traditional Cluster ArchitecturePublic EthernetPrivateEthernet NetworkApplicationNetwork (Optional)NodeNodeNodeNodeNodeNodeNodeNodeNodeNode 2007 UC RegentsPower Distribution(Net addressable units as option)Frontend Node5
The Modern “Cluster” Architecture is moreInteresting 2007 UC Regents6
The Goal of RocksEnable domain scientists (non-specialists) totrivially build clusters of all functions andsizes(Put in CDs and Start Computing)Key observation – Rocks easily handlesmore than just MPI /HPC Clusters 2007 UC Regents7
Philosophyy Computation/DataAnalysis is now acommon pillar of all fields of Sciencey Reproducibilityof an experiment is thecornerstone of the scientific methody Mustextend this to computationalexperiment – including the infrastructure 2007 UC Regents8
What Rocks SolvesyYou are building a [compute, web, database,visualization, ?] cluster What software should I select ? How should it be configured? How long will it be before I start useful work ?yyI like your cluster configuration, but I have adifferent HW vendor, how do I re-create whatyou have already done?I’d rather drink coffee than configure software 2007 UC Regents9
Software Installation is First ClassDescriptive information toconfigure a node(Rocks Config Graph)Collection of all possiblesoftware packages(Distribution)RPMs(Solaris Pkgs)IO ServerWeb Server 2007 UC RegentsAppliancesCompute NodeKickstart(Jumpstart)file10
It all starts with InstallationyRocks Scales well (Avalanche Installer): 1 node – 10 min, 128 nodes – 20 minutes 700 nodes – 35 minutesySupports updated-at-install time No install-then-patch immediatelyyThe basic core is extensible with add-ons calledRolls Software packages Captures how the expert would configure asubsystem 2007 UC Regents11
Email Yesterday from a Rocks User“I've just installed our new cluster and I must say that I'm really impressedby the scalability of the installer. Look at the rpm to torrent downloadratio on the frontend:[root@stallo log]# grep .rpm.torrent httpd/access log wc -l511526[root@stallo log]# grep ".rpm " httpd/access log wc -l9938I messed up something with the reboot -f scripting so I had to install thething in three batches, if I did them all at once the ratio would havebeen even better.”Æ Avalanche Installer (Rocks) is HTTP based. In this casemore than 98% of packages were NOT downloaded directly fromfrontend 2007 UC Regents12
Kickstart – Allowing the Native OSinstaller to do work for usyRed Hat’s Kickstart Monolithic flat ASCII file No macro language Requires forking based on site informationand node type.yRocks XML Kickstart Decompose a kickstart file into nodes and agraphx Graph specifies OO frameworkx Each node specifies a service and itsconfiguration Macros and SQL for site configuration Driven from web cgi scriptyRocks on Solaris Generates JumpstartFiles 2007 UC Regents13
Configuration/Localization is THE problemto be solvedyySetup & Packages (20%)cdromzerombr yesbootloader --location mbr --useLiloskipxauth --useshadow --enablemd5clearpart --allpart /boot --size 128part swap --size 128part / --size 4096part /export --size 1 --growlang en USlangsupport --default en USkeyboard usmouse genericps/2timezone --utc GMTrootpw --iscrypted nrDq4Vb42jjQ.textinstallrebootPost Configuration (80%)%postcat /etc/nsswitch.conf s dnsbootparams: filesethers:filesEOFcat /etc/ntp.conf 'EOF'server ntp.ucsd.eduserver127.127.1.1fudge127.127.1.1 stratum 10authenticate nodriftfile /etc/ntp/driftEOF/bin/mkdir -p /etc/ntpcat /etc/ntp/step-tickers 'EOF'ntp.ucsd.eduEOF/usr/sbin/ntpdate ntp.ucsd.edu/sbin/hwclock --systohc%packages@Base@Emacs@GNOME 2007 UC Regents14
XML Kickstart 2007 UC Regents15
It looks something like this 2007 UC Regents16
Possible Definitions of CloudsAnything that is a distributed resourceAnother word for Grid or High-ThroughputComputing.y Write your program (workflow) by calling webservicesyy Integrate Google Maps, Weather.com radar, etcySoftware as a Service (SaaS) Mosso Hosting “Cloud” Google Docs (Can I do Computing Here?)yInfrastructure as a Service (IaaS) Sun’s Network.com Amazon EC2 – Run your Software Stack .
Applications/Workflows using web services.yUsing a commercial Map web-serviceyThey have using webservices for 4 yearsWhat about Condor,Globus-based, Avakibased Grids?Workflow using web/gridservicesCalling by differentname does not acloud make
SaaS – Software as a ServiceMosso “cloud” for hosting web sitesyyyIf your web server uses (a combination) of 9software packages, then this worksVerdict: Mosso (and things like it) are notappropriate for general purpose computing.Can be very good for targeted applications
IaaS – Infrastructure as a Service –Cloud with the most promiseSunAmazon EC23TeraGoGridIBMRun (virtual) computers to solveyour problem, using your software
Good/bad of Current IaaS CloudsyPositives User can completely control the software stack – Important forcustom HPC/HTC applications Entry cost is “so low, you trip on it” Cycle cost ( 0.10/CPU-Hour) – Everyone examines cost Dominant cost is now people time for the end-useryBad Performance unpredictable Not Really catering to (small-scale) parallel or high-throughputclusters Long-running machines (weeks/months/years) are actuallyexpensive (Southwest Airlines Model) Not friendly for management (back to the bad old days)
Clouds for HPC?yyYes. Inevitable.Why? (incomplete list) Very Flexible. Low Entry Cost. Done well, the virtual machine (Xen, VMWare,VirtualBox, ) looks/performs like real hardware Hypervisor:Guest performance improves IaaS providers not bearing the cost of supporting yoursoftware.yHPC/Campus centers move to support IaaS paradigm Too Expensive for traditional HPC providers (eg.Teragrid) to support N x K versions of software packages Machines designed for HPC/HTC/High Data Rates can bemuch more effective for scientific problems than EC2implementation
High-level observationsy“Clouds. It’s all new, have to learn a wholenew vocabulary, software, a whole new wayof doing things” If this is true, it is doomed to failureyIaaS Clouds will turn all scientists intosysAdmins If not addressed, Doomed to failureyDial back to 1992 – Introduction of PVM3.The “personal MPP” abstraction was righton target.
The Neonatal stages of cloudsy(Nearly) everybody focusing on starting virtualmachine images E.g., RightScale “dashboard”, 3Tera and othersyHow are the contents of the VM Defined? Start with “base image,” use sysadmin super-skills tomake it what you want.x Run and scream as we re-invent 1995 Beowulf Use a predefined image on EC2, take what you get.What if I want a cluster of VMs with mysoftware?y What if I want a cluster of 1000 VMs?y How are the last two different from a realcluster?y
Rolls Define the How/What aCluster Frontend can Build
Rolls Add Software/Configurationand defined Appliances
Virtual Clusters in PhysicalyTwo Clusters: Physical Hosting Cluster (White Background) Virtual Cluster Å User-defined Stack here(Purple)
The Cloud Provider’s View
Rocks: Create A Virtual Cluster# rocks create cluster vi-1 137.110.119.137 numnodes 3yMust take care of low-level bookkeeping VLAN Isolation, Allocation of Virtual Disks, Allocation ofNetwork Mac Addresses (Everything we take for granted in realHW)
How Rocks Treats Virtual HWy It’sjust another piece of HWy All of the existing techniques carry overy You don’t have to learn “a whole newway”
Campus Cloud: Cluster ExtensionyVMs: Software/OSdefined by thefrontend: Users, file systemmount, queuingsystem, softwareversions, etc
Cluster Extension – CAMERA MetaGenomicsIkelite2.Rocksclusters.orgyConfig is CAMERA-defined BLAST, MrBayes,, MPI-BLAST,GROMACS, (Bio Tools) CAMERA Users, File System Mounts,Queuing System, etc. All “Compute” Nodes are Virtual machinesyRocks 5.1 software structure makesthis very straightforward
Virtual Clusters Performance: HPCC Benchmark SuiteSource: Shava Smallen, SDSC
Many open Operational andFundamental Research IssuesyOperational: How you efficiently handle VM Hosting? Performance optimizations for VMs Performance optimizations for VM ProvisioningyFundamental Two VMs (or real HW) – how closely related aretheir configurations? (Genetics for machines) I need a VM with Characteristics X,Y, & Z – how canyou define and verify (Basic Definition of Instrument) How can I verify if something changed without a fullscan? (Integrity of the computational instrument)
Summaryy Cloudsare making rapid headway They are still in the stone age of mgmt Cost of people finally brought to theforefronty PersonalizedClusters w/o owning any/allof the hardwarey Rocks 5.1 working to make both theprovider side and the user side easier Same set of tools. Different Configurations.
Rocks – www.rocksclusters.org yLeading Open Source Cluster Distribution yActive development for 8 years yDemonstrated Scalability from 2 – 1000 nodes. Several 1000 Clusters Worldwide y 1800 Users on our discussion list About 650 Messages/Month Commercial support available yRedhat Enterprise 4/5 compa
