Transcription
An Introduction to UsingHTCondor2013
The Team - 2012Established in 1985,to do research and development of distributedhigh-throughput computing2
HTCondor does High-ThroughputComputing› Allows for many computational tasks tobe completed over a long period of time› Useful for researchers and other userswho are more concerned with thenumber of computations they can doover long spans of time than they arewith short-burst computations3
HTCondor’s strengths›››››››Cycle scavenging works!High-throughput computingVery configurable, adaptableSupports strong security methodsInteroperates with many types of computing gridsHas features to manage both dedicated CPUs(clusters) and non-dedicated resources (desktops)Fault-tolerant: can survive crashes, network outages,any single point of failure4
HTCondor will .› Keep an eye on your jobs and keep you››››posted on their progressImplement your policy on the execution orderof your jobsLog your job's activitiesAdd fault tolerance to your jobsImplement your policy as to when the jobscan run on your desktop5
Ouresteemedscientist*,has plenty ofsimulationsto do.* and Karen's cousin6
Einstein's SimulationSimulate theevolution ofthe cosmos,assumingvariousproperties.7
Simulation OverviewVarying values for each of: G (the gravitational constant): 100 values Rμν (the cosmological constant): 100 values c (the speed of light): 100 values100 100 100 1,000,000 jobs8
Each job within the simulation: Requires up to 4 GBytes of RAM Requires 20 MBytes of input Requires 2 – 500 hours of computing time Produces up to 10 GBytes of outputEstimated total: 15,000,000 CPU hours or 1,700 compute YEARS 10 Petabytes of output9
Albert will be happy, since HTCondor willmake the completion of the entiresimulation easy.10
DefinitionsJob the HTCondor representation of a piece of work Like a Unix process Can be an element of a workflowClassAd HTCondor’s internal data representationMachine or Resource computers that can do the processing11
More DefinitionsMatch Making Associating a job with a machine resourceCentral Manager Central repository for the whole pool Does match makingSubmit Host The computer from which jobs are submitted toHTCondorExecute Host The computer that runs a job12
Jobs state their needs and preferences: Requirements (needs): I require a Linux x86-64 platform Rank (preferences): I prefer the machine with the most memory I prefer a machine in the botany department13
Machines specify needs and preferences: Requirements (needs): Require that jobs run only when there is nokeyboard activity Never run jobs belonging to Dr. Heisenberg Rank (preferences): I prefer to run Albert’s jobs14
ClassAdsthe language that Condoruses to representinformation – aboutjobs (job ClassAd),machines (machineClassAd), andprograms that implementCondor's functionality(called daemons)15
ClassAd AttributeName ValueorAttributeName Expression16
Part of a Job erCmdRequirements. "cosmos"(Arch "INTEL") BooleanExpression17
The Magic of MatchmakingThe match maker matches job ClassAds withmachine ClassAds, taking into account: Requirements of both the machine and the job Rank of both the job and the machine Priorities, such as those of users and groups18
Getting Started1. Choose a universe for the job2. Make the job batch-ready, whichincludes making the input data availableand accessible3. Create a submit description file4. Run condor submit to put the job(s)in the queue19
1. Choose the Universe› controls how›HTCondor handlesjobsthe many universesinclude: vanillastandardgridjavaparallelvm20
Using the vanilla Universe Allows running almostany “serial” job Provides automatic filetransfer for input andoutput files Like vanilla ice cream,can be used in just aboutany situation21
2. Make the jobbatch-ready› Must be able to run inthe background› No interactive input› No GUI/window clicks22
Batch-Ready:Standard Input & Output› Job can still use stdin, stdout (thekeyboard and the screen) , and stderr ,but files are used instead of the actualdevices› Similar to Unix shell: ./myprogram input.txt output.txt23
Make the Data Available› HTCondor will Transfer data files to the location where the jobruns Transfer result files back from the locationwhere the job runs› Place the job's data files in a place whereHTCondor can access them24
3. Create aSubmit Description File› A plain ASCII text file› File name extensions are irrelevant,although many use .sub or .submit assuffixes› Tells HTCondor about the job› Can describe many jobs at once (acluster), each with different input, output,command line arguments, etc.25
Simple Submit Description File# file name is cosmos.sub# (Lines beginning with # are comments)# NOTE: the commands on the left are not#case sensitive, but file names#(on the right) are!UniverseExecutableInputOutputLogQueue vanillacosmoscosmos.incosmos.outcosmos.log26Put 1 instance ofthe job in thequeue
Input, Output, and Error FilesInput in fileRead job’s standard input from in fileLike shell command: program in fileOutput out fileWrite job’s standard output to out fileLike shell command: program out fileError error fileWrite job’s standard error to error fileLike shell command: program 2 error file27
Logging the Job's Activities› In the submit description file:log cosmos.log› Creates a log of job events, which shows allevents that occur as the job executes› Good advice: always have a log file28
Sample Portion of Job Log000 (0101.000.000) 05/25 19:10:03 Job submitted from host: 128.105.146.14:1816 .001 (0101.000.000) 05/25 19:12:17 Job executing on host: 128.105.146.14:1026 .005 (0101.000.000) 05/25 19:13:06 Job terminated.(1) Normal termination (return value 0).000, 001, and 005 are examples of eventnumbers.29
4. Submit the JobRun condor submit, providing the name ofthe submit description file: condor submit cosmos.subSubmitting job(s).1 job(s) submitted to cluster 100.condor submit then parses the submit description file, checking forerrors creates a ClassAd that describes the job(s) places the job(s) in the queue, which is an atomicoperation, with a two-phase commit30
Observe Jobs in the Queue condor q-- Submitter: MITTED2.0heisenberg1/13 13:593.0hawking1/15 19:184.0hawking1/15 19:335.0hawking1/15 19:336.0hawking1/15 19:34.96.0bohr4/5 13:4697.0bohr4/5 13:4698.0bohr4/5 13:5299.0bohr4/5 13:52100.0einstein4/5 13:55: 128.104.55.9:51883 :RUN TIME ST PRI SIZE CMD0 00:00:00 R 00.0 env0 04:29:33 H 00.0 script.sh0 00:00:00 H 00.0 script.sh0 00:00:00 H 00.0 script.sh0 00:00:00 H 00.0 script.sh0 00:00:000 00:00:000 00:00:000 00:00:000 00:00:00IIIII000000.00.00.00.00.0atoms Hatoms Hatoms Hatoms Hcosmos100 jobs; 1 completed, 0 removed, 20 idle, 1 running, 77 held,0 suspended31
File TransferTransfer Input Files specifies a list offiles to transfer from the submit machine tothe execute machineTransfer Output Files specifies a list offiles to transfer back from the executemachine to the submit machine IfTransfer Output Files is not specified,HTCondor will transfer back all new files inthe execute directory32
More on File TransferFiles need to get from the submit machine to the execute machine. 2possibilities:1.both machines have access to a shared file system2.machines have separate file systemsShould Transfer Files YES: Transfer files to execution machine NO: Rely on shared file system IF NEEDED: Automatically transfer the files, if the submit andexecute machine are not in the same FileSystemDomain(translation: use shared file system if available)When To Transfer Output ON EXIT: Transfer output files only when job completes ON EXIT OR EVICT: Transfer output files when job completesor is evicted33
File Transfer Example# new cosmos.sub fileUniverseExecutableLogTransfer Input FilesTransfer Output FilesShould Transfer FilesWhen To Transfer OutputQueue34 vanilla cosmos cosmos.log cosmos.dat results.dat IF NEEDED ON EXIT
Command Line Arguments# Example with command line argumentsUniverse vanillaExecutable cosmosArguments -c 299792458 –G 6.67300e-112. . . Invokes executable withcosmos –c 299792458 –G 6.673e-112QueueLook at the condor submit man page tosee syntax for Arguments. This examplehas argc 5.35
More Feedback HTCondor sends emailabout job events to thesubmitting user Specify one of these inthe submit descriptionfile:Notification Notification Notification Notification completenevererroralways36Default in 7.8Default in 7.9
ClusterId.ProcID is Job ID› If the submit description file describes multiple jobs, the›››set is called a clusterEach cluster has a cluster number, where the clusternumber is unique to the job queue on a machineEach individual job within a cluster is called a process,and process numbers always start at zeroA Job ID is the cluster number, a period, and the processnumber. Examples: Job ID 20.0Cluster 20, process 0 Job IDs: 21.0, 21.1, 21.2Cluster 21, processes 0, 1, 237
1 ClusterUniverse vanillaExecutable cosmoslog cosmos 0.logInput cosmos 0.inOutput cosmos 0.outQueueJob 102.0 (cluster 102,process 0)38
File OrganizationA logistical nightmare places all input, output,and log files in one directory 3 files 1,000,000 jobs 3,000,000 files The submit description file is 4,000,000 linesThe directory will be difficult (at best) to evenlook at39
Better Organization› Create subdirectories for each job, intentionallynamedrun 0, run 1, run 999999› Implement the creation of directories with a›Python or Perl programCreate or place input files in each of theserun 0/cosmos.inrun 1/cosmos.in run 999999/cosmos.in› The output and log files for each job will becreated by the job, when the job runs40
Einstein’s simulation directorycosmoscosmos.subrun 0cosmos.incosmos.outcosmos.logcosmos.inrun 999999cosmos.outcosmos.log41Submitter or scriptcreates black-fontfilesHTCondorcreatespurple-fontfiles
Better Submit Description File# Cluster of 1,000,000 jobsUniverseExecutableLogOutputInput Dir run 0QueueJob 103.0 (Cluster 103, Process 0)InitialDir run 1QueueJob 103.1 (Cluster 103, Process 1)This file contains 999,998 more instances ofInitialDir and Queue.42
Submit Description File Macros› Queue all 1,000,000 processes with the singlecommand:Queue 1000000› Within the submit description file, HTCondorpermits named macros (Process) will be expanded to the process number foreach job in the clusterValues 0 – 999999 for the 1,000,000 jobs43
Using (Process)› The initial directory for each job can be specifiedInitialDir run (Process) HTCondor expands these to directoriesrun 0, run 1, run 999999› Similarly, command-line arguments could use amacro to pass a unique identifier to each jobinstanceArguments -n (Process) HTCondor expands arguments to:-n 0-n 1 -n 99999944
(Best) Submit Description File# Example: a cluster of 1000000 jobsUniverse vanillaExecutable cosmosLog cosmos.logInput cosmos.inOutput cosmos.outInitialDir run (Process)Queue 100000045
Albert submits the cosmossimulation. Patience required, itwill take a while condor submit cosmos.subSubmittingjob(s) .Logging submitevent(s) .1000000 job(s) submitted to cluster 104.46
the Job Queue condor q-- Submitter: submit.chtc.wisc.edu : 128.104.55.9:51883 : submit.chtc.wisc.eduIDOWNERSUBMITTEDRUN TIME ST PRI104.0einstein 4/20 12:08 0 00:00:05 R 0104.1einstein 4/20 12:08 0 00:00:03 I 0104.2einstein 4/20 12:08 0 00:00:01 I 0104.3einstein 4/20 12:08 0 00:00:00 I 0.104.999998 einstein 4/20 12:08 0 00:00:00 I 0104.999999 einstein 4/20 12:08 0 00:00:00 I 0999999 jobs; 999998 idle, 1 running, 0 held47SIZE CMD9.8 cosmos9.8 cosmos9.8 cosmos9.8 cosmos9.8 cosmos9.8 cosmos
Albert Relaxes› HTCondor watches over›the jobs, and will restartthem if required, etc.Time for a cold one!48
More That You Do With HTCondor49
Remove Jobs with condor rm› You can only remove jobs that you own› Privileged user can remove any jobs root on Linux administrator on Windowscondor rm 4condor rm 4.2condor rm –aRemoves all cluster 4 jobsRemoves only the job withjob ID 4.2Removes all of your jobs.Careful !50
Specify Job Requirements› A boolean expression (syntax similar to C or Java)› Evaluated with respect to attributes from machineClassAd(s)› Must evaluate to True for a match to be madeUniverse vanillaExecutable mathematica.Requirements ( \HasMathematicaInstalled ? True )Queue 2051
Specify Needed ResourcesItems appended to job Requirementsrequest memory – the amount of memory (inMbytes) that the job needs to avoid excessiveswappingrequest disk – the amount of disk space (inKbytes) that the job needs. Will be sum of spacefor executable, input files, output files andtemporary files. Default is size of initial sandbox(executable plus input files).request cpus – the number of CPUs (cores) thatthe job needs. Defaults to 1.52
Specify Job Rank› All matches which meet the requirements can be›sorted by preference with a Rank expression Numerical Higher rank values match firstLike Requirements, is evaluated against attributesfrom machine ClassAdsUniverseExecutable vanilla cosmos. . .Rank (KFLOPS*10000) MemoryQueue 100000053
Job Policy Expressions› Do not remove if exits with a signal:on exit remove ExitBySignal False› Place on hold if exits with nonzero status orran for less than an hour:on exit hold ( (ExitBySignal False) && (ExitSignal ! 0) ) ( (ServerStartTime - JobStartDate) 3600)› Place on hold if job has spent more than50% of its time suspended:periodic hold ( CumulativeSuspensionTime (RemoteWallClockTime / 2.0) )54
Lots ofShort-Running JobsKnow that starting a job is somewhatexpensive, in terms of time.3 items that might help:1. Batch short jobs together Write a wrapper script that will run a set of thejobs in series Submit the wrapper script as the job1. Explore HTCondor’s parallel universe2. There are some configuration variables thatmay be able to help Contact a staff person for more info55
Common Problems withJobs56
Jobs Are IdleOur scientist runs condor q and finds all hisjobs are idle condor q-- Submitter: x.cs.wisc.edu : 128.105.121.53:510 :x.cs.wisc.eduID OWNERSUBMITTEDRUN TIME ST PRI SIZE CMD5.0 einstein 4/20 12:23 0 00:00:00 I 09.8 cosmos5.1 einstein 4/20 12:23 0 00:00:00 I 09.8 cosmos5.2 einstein 4/20 12:23 0 00:00:00 I 09.8 cosmos5.3 einstein 4/20 12:23 0 00:00:00 I 09.8 cosmos5.4 einstein 4/20 12:23 0 00:00:00 I 09.8 cosmos5.5 einstein 4/20 12:23 0 00:00:00 I 09.8 cosmos5.6 einstein 4/20 12:23 0 00:00:00 I 09.8 cosmos5.7 einstein 4/20 12:23 0 00:00:00 I 09.8 cosmos8 jobs; 8 idle, 0 running, 0 held57
Exercise a little patience› On a busy pool, it can take a while›to match jobs to machines, andthen start the jobsWait at least a negotiation cycle ortwo, typically a few minutes58
Look in the Job LogThe log will likely contain clues: cat cosmos.log000 (031.000.000) 04/20 14:47:31 Job submitted fromhost: 128.105.121.53:48740 .007 (031.000.000) 04/20 15:02:00 Shadow exception!Error from starter on gig06.stat.wisc.edu:Failed to test3/run 0/cosmos.in' as standard input: Nosuch file or directory (errno 2)0 - Run Bytes Sent By Job0 - Run Bytes Received By Job.5959
Check Machines' Status condor NUXLINUXLINUXArchX86 64X86 64X86 64X86 64X86 64X86 64X86 1024102410241024ActvtyTime0 00:10:131 19:10:361 22:42:200 03:22:100 03:17:000 03:09:140 19:13:495115110 00:24:170 00:45:01Total Owner Claimed Unclaimed Matched Preempting BackfillINTEL/WINDOWSX86 1060
Try: condor q -analyze condor q -analyze 107.5-- Submitter: crane.cs.wisc.edu : 128.105.136.32:61610 : crane.cs.wisc.eduUser priority for max@crane.cs.wisc.edu is notavailable, attempting to analyze without it.--107.005: Run analysis summary. Of 4 machines,0 are rejected by your job's requirements0 reject your job because of their own requirements4 match and are already running your jobs0 match but are serving other users0 are available to run your job.9evropim617nid
condor q -analyze 102.1-- Submitter: crane.cs.wisc.edu : 128.105.136.32:61610 : crane.cs.wisc.eduUser priority for max@crane.cs.wisc.edu is notavailable, attempting to analyze without it.--107.005: Run analysis summary. Of 3184 machines,3184 are rejected by your job's requirements0 reject your job because of their own requirements0 match and are already running your jobs0 match but are serving other users0 are available to run your jobWARNING: Be advised:No resources matched request's constraints62
(continued)The Requirements expression for your job is:(((((TARGET.Arch "X86 64" ) &&TARGET.OpSys "WINDOWS" ) &&TARGET.Disk RequestDisk ) &&TARGET.Memory RequestMemory ) &&TARGET.HasFileTransfer )Suggestions:Condition---------Machines MatchedSuggestion---------------- ----------1 ( TARGET.OpSys "WINDOWS" ) 0MODIFY TO "LINUX"2 ( TARGET.Arch "X86 64" )31373 ( TARGET.Disk 1 )31844 ( TARGET.Memory ifthenelse(MemoryUsage isntundefined,MemoryUsage,1) )31845 ( TARGET.HasFileTransfer )318463
Learn about available resources condor status –const 'Memory 8192'(no output means no matches) condor status -const 'Memory 4096'NameOpSys ArchStateActivslot1@c001.ch LINUX X86 64 Unclaimed Idleslot2@c001.ch LINUX X86 64 Unclaimed Idleslot3@c001.ch LINUX X86 64 Unclaimed Idleslot1@c002.ch LINUX X86 64 Unclaimed IdleLoadAv0.0000.0000.0000.000MemActvtyTime5980 1 05:35:055980 13 05:37:037988 1 06:00:057988 13 06:03:47Total Owner Claimed Unclaimed Matched PreemptingX86 64/LINUX400400Total40040064
Interact With A Job› Perhaps a job is running for much longerthan expected. Is it stuck accessing a file? Is it in an infinite loop?› Try condor ssh to job Interactive debugging in Unix Use ps, top, gdb, strace, lsof, Forward ports, X, transfer files, etc. Currently not available on Windows65
Interactive Debug Example condor q-- Submitter: cosmos.phy.wisc.edu : 128.105.165.34:1027 ID1.0OWNERSUBMITTEDRUN TIMEST PRI SIZE CMDeinstein 4/15 06:52 1 12:10:05 R 010.0 cosmos1 jobs; 0 idle, 1 running, 0 held condor ssh to job 1.0Welcome to slot4@c025.chtc.wisc.edu!Your condor job is running with pid(s) 15603. gdb –p 15603. . .66
HTCondor is extremelyflexible. Here areoverviews of some ofthe many featuresthat you may want tolearn more about.67
After this tutorial, here are some places youmight find help:1. HTCondor manual2. htcondor-users mailing list. ndorusers3. i4. Developers68
The more time a job takes torun, the higher the risk of being preempted by ahigher priority user or job getting kicked off a machine(vacated), because themachine has something elseit prefers to do HTCondor'sstandard universe mayprovide a solution.69
Standard Universe› Regularly while the job runs, or when thejob is to be kicked off the machine,HTCondor takes a checkpoint -- thecomplete state of the job.› With a checkpoint, the job can bematched to another machine, andcontinue on.70
checkpoint: the entire state of a program saved in afile, such as CPU registers, memory image, I/O,etc.time71
3 Checkpointstime12723
timeKilled!3373
Goodput and Badputgoodputbadputgoodput ?time3374
Standard Universe Features› Remote system calls (remote I/O) The job can read or write files as if theywere local› Programming language independent› No source code changes are typicallyrequired, but relinking the executable withHTCondor's standard universe supportlibrary is required.75
How to RelinkPlace condor compile in front of thecommand used to link the job: condor compile gcc -o myjob myjob.c- OR condor compile f77 -o myjob filea.f fileb.f- OR condor compile make –f MyMakefile76
Limitations› HTCondor’s checkpoint mechanism is not at››the kernel level. Therefore, a standarduniverse job may not : fork() Use kernel threads Use some forms of IPC, such as pipes andshared memoryMust have access to object code in order torelinkOnly available on some Linux platforms77
Parallel Universe› When multiple processes of a single job must be›running at the same time on different machines.Provides a mechanism for controlling parallelalgorithms Fault tolerant Allows for resources to come and go Ideal for Computational Grid environments› Especially for MPI78
MPI Job Submit Description File# MPI job submit description fileuniverse parallelexecutable mp1scriptarguments my mpich linked exe arg1 arg2machine count 4should transfer files YESwhen to transfer output ON EXITtransfer input files my mpich linked exequeue79
MPI jobsNote: HTCondor will probably not schedule all ofthe jobs on the same machine, so considerusing whole machine slotsSee the HTCondor Wiki:Under How To Admin Recipes,"How to allow some jobs to claim the wholemachine instead of one slot"80
VM Universe› A virtual machine instance is the HTCondor job› The vm universe offers Job sandboxingCheckpoint and migrationSafe elevation of privilegesCross-platform submission› HTCondor supports VMware, Xen, and KVM› Input files can be imported as CD-ROM image› When the VM shuts down, the modified diskimage is returned as job output81
Machine Resources areNumerous: The GridGiven access (authorization) to grid resources ,as well as certificates (for authentication) andaccess to Globus or other resources atremote institutions, HTCondor's grid universedoes the trick !82
Grid Universe› All specification is in the submit description file› Supports many “back end” types: Globus: GT2, GT5 NorduGrid UNICORE HTCondor PBS LSF SGE EC2 Deltacloud Cream83
ABCD84Some sets of jobshave dependencies.HTCondor handlesthem with DAGMan.› Interested? Stay forKent's tutorial, laterthis morning.
the Java UniverseMore than java mysimulator› Knows which machines have a JVM installed› Knows the location, version, and performanceof JVM on each machine› Knows about jar files, etc.› Provides more information about Java jobcompletion than just a JVM exit code Program runs in a Java wrapper, allowingHTCondor to report Java exceptions, etc.85
Java Universe Example# sample java universe submit# description fileUniverse javaExecutable Main.classjar files MyLibrary.jarInput infileOutput outfileArguments Main 1 2 3Queue86
In ReviewWith HTCondor’s help, both you andAlbert can: Submit jobs Manage jobs Organize data files Identify aspects of universe choice87
Thank you!Check us out on the ondor-admin@cs.wisc.edu88
Extra Slides with MoreInformation You Might Want toReference89
InitialDir› Identifies a directory for file input and output.› Also provides a directory (on the submit machine) for the›user log, when a full path is not specified.Note: Executable is not relative to InitialDir# Example with InitialDirUniverse vanillaInitialDir /home/einstein/cosmos/runExecutable cosmosNOT Relative to InitialDirLog cosmos.logInput cosmos.inIs Relative to InitialDirOutput cosmos.outError cosmos.errTransfer Input Files cosmos.datArguments -f cosmos.datQueue90
Substitution Macro ( attribute ) will be replaced by the value of thespecified attribute from the Machine ClassAdExample:Machine ClassAd has:CosmosData "/local/cosmos/data"Submit description file hasExecutable cosmosRequirements (CosmosData ! UNDEFINED)Arguments -d (CosmosData)Results in the job invocation:cosmos –d /local/cosmos/data91
Getting HTCondor› Available as a free download d HTCondor for your operatingsystem Available for many modern Unix platforms(including Linux and Apple’s OS/X) Also for Windows, many versions› Repositories YUM: RHEL 4, 5, and 6 yum install condor APT: Debian 5 and 6 apt-get install condor92
HTCondor Releases› Stable and Developer Releases Version numbering scheme similar to that of the›(pre 2.6) Linux kernels Major.minor.release If minor is even (a.b.c): Stable series Very stable, mostly bug fixes Current: 7.8 If minor is odd (a.b.c): Developer series New features, may have some bugs Current: 7.993
General User Commandscondor statuscondor qcondor submitcondor rmcondor priocondor historycondor submit dagcondor checkpointcondor compile94View Pool StatusView Job QueueSubmit new JobsRemove JobsChange a User PriorityCompleted Job InfoSubmit new DAGForce a checkpointLink Condor library with job94
Einstein’s simulation directory cosmos cosmos.sub run_999999 run_0 Submitter or script creates black-font files HTCondor creates purple-font files cosmos.in cosmos.in cosmos.out cosmos.log cosmos.out cosmos.log. 42 Better Submit Descri
