WIXLIB

An Introduction to UsingHTCondor2014Karen Miller

The Team - 2013Established in 1985,to do research and development of distributedhigh-throughput computing2

HT Stands for High ThroughputThroughput: the quantity of work done byan electronic computer in a given period oftime (Dictionary.com)3

HTCondor’s strengths Cycle scavenging works!Very configurable, adaptableSupports strong security methodsInteroperates with many types of computing gridsManages both dedicated CPUs (clusters) and nondedicated resources (desktops) Fault-tolerant: can survive crashes, network outages,any single point of failure4

HTCondor will . Keep an eye on your jobs and keep youposted on their progress Implement your policy on the execution orderof your jobs Log your job's activities Add fault tolerance to your jobs Implement your policy as to when the jobscan run on your desktop5

Our esteemedscientist*, hasplenty ofsimulations todo.* and Karen's cousin?6

Einstein's SimulationSimulate theevolution of thecosmos,assumingvariousproperties.7

Simulation OverviewVarying values for each of:G(the gravitational constant): 100 valuesRμν (the cosmological constant): 100 valuesc(the speed of light): 100 values100 100 100 1,000,000 jobs8

Each job within the simulation: Requires up to 4 GBytes of RAMRequires 20 MBytes of inputRequires 2 – 500 hours of computing timeProduces up to 10 GBytes of outputEstimated total: 15,000,000 CPU hours or 1,700 compute YEARS 10 Petabytes of output9

Albert will be happy, since HTCondor willmake the completion of the entiresimulation easy.10

DefinitionsJob the HTCondor representation of a piece of work like a Unix process can be an element of a workflowClassAd HTCondor’s internal data representationMachine or Resource computers that can do the processing11

More DefinitionsMatchmaking associating a job with a machine resourceCentral Manager central repository for the whole pool does matchmakingSubmit Host the computer from which jobs are submitted toHTCondorExecute Host the computer that runs a job12

Jobs state their needs and preferences: Requirements (needs): I require a Linux x86-64 platform Rank (preferences): I prefer the machine with the most memory I prefer a machine in the botany department13

Machines specify needs and preferences: Requirements (needs): Require that jobs run only when there is nokeyboard activity Never run jobs belonging to Dr. Heisenberg Rank (preferences): I prefer to run Albert’s jobs14

ClassAdsthe language that HTCondoruses to represent informationabout:jobs (job ClassAd),machines (machineClassAd), and programs thatimplement HTCondor'sfunctionality (calleddaemons)15

Part of a Job ClassAdMyTypeString "Job"TargetType "Machine"ClusterId 1ProcID 0IsPhysics TrueOwner "einstein"Cmd "cosmos"IntegerBooleanRequirements (Arch "INTEL")16BooleanExpression

The Magic of MatchmakingThe matchmaker matches job ClassAds withmachine ClassAds, taking into account: Requirements of both the machine and the job Rank of both the job and the machine Priorities, such as those of users and groups17

Getting Started1. Choose a universe for the job2. Make the job batch-ready, whichincludes making the input data availableand accessible3. Create a submit description file4. Run condor submit to put the job(s)in the queue18

1. Choose the Universe controls how HTCondor handlesjobsthe many universesinclude: vanillastandardgridjavaparallelvm19

Using the vanilla Universe Allows running almostany “serial” job Provides automatic filetransfer for input andoutput files Like vanilla ice cream,can be used in just aboutany situation20

2. Make the jobbatch-ready Must be able to run inthe background No interactive input No GUI/window clicks21

Batch-Ready:Standard Input & Output Job can still use stdin (keyboard),stdout (screen) , and stderr , but filesare used instead of the actual devices Similar to Unix shell redirect: ./myprogram input.txt output.txt22

Make the Data Available HTCondor will Transfer data files to the execute host where thejob runs Transfer result files back from the execute hostto the submit host So, place these data files in a place whereHTCondor can access them23

3. Create aSubmit Description File A plain ASCII text file File name extensions are irrelevant,although many use .sub or .submit assuffixes Describes the job Can describe many jobs at once (acluster), each with different input, output,command line arguments, etc.24

Simple Submit Description File# file name is cosmos.sub# Lines beginning with # are comments# Note: the commands on the left are#not case sensitive, but file names#(on the right) are!Universe vanillaExecutable cosmosInput cosmos.inOutput cosmos.outLog cosmos.logQueuePut 1 instance of thejob in the queue25

Input, Output, and Error FilesInput infileRead job’s standard input from infileLike shell command: program infileOutput outfileWrite job’s standard output to outfileLike shell command: program outfileError errorfileWrite job’s standard error to errorfileLike shell command: program 2 errorfile26

Logging the Job's ActivitiesIn the submit description file:log cosmos.log Creates a log of job events, appended withall events as the job executes Good advice: always have a log file27

Sample Portion of Job Log000 (0101.000.000) 05/25 19:10:03 Job submitted from host: 128.105.146.14:1816 .001 (0101.000.000) 05/25 19:12:17 Job executing on host: 128.105.146.14:1026 .005 (0101.000.000) 05/25 19:13:06 Job terminated.(1) Normal termination (return value 0).000, 001, and 005 are examples of event numbers.28

4. Submit the JobRun condor submit, providing the name ofthe submit description file: condor submit cosmos.subSubmitting job(s).1 job(s) submitted to cluster 100.condor submit will parse the submit description file, checking for errors create a ClassAd that describes the job(s) place the job(s) in the queue, which is an atomicoperation, with a two-phase commit

Observe Jobs in the Queue condor q-- Submitter: MITTED2.0heisenberg1/13 13:593.0hawking1/15 19:184.0hawking1/15 19:33.98.0bohr4/5 13:5299.0bohr4/5 13:52100.0einstein4/5 13:55: 128.104.55.9:51883 :RUN TIME ST PRI SIZE CMD0 00:00:00 R 00.0 env0 04:29:33 H 00.0 script.sh0 00:00:00 H 00.0 script.sh0 00:00:00 I0 00:00:00 I0 00:00:00 I0000.00.00.0atoms Hatoms Hcosmos100 jobs; 1 completed, 0 removed, 20 idle, 1 running, 77 held,0 suspended

File TransferTransfer Input Files specifies a list offiles to transfer from the submit machine to theexecute machineTransfer Output Files specifies a list offiles to transfer back from the execute machineto the submit machine. IfTransfer Output Files is not specified,HTCondor will transfer back all new files in theexecute directory. Generally used to limit thenumber files transferred.31

More on File TransferFiles need to get from the submit machine to the execute machine.2 possibilities:1. both machines have access to a shared file system2. machines have separate file systemsShould Transfer Files YES: transfer files to execute host NO: rely on shared file system IF NEEDED: transfer the files, if the submit and executemachine are not in the same file system domain(translation: use shared file system if available)When To Transfer Output ON EXIT: transfer output files only when job completes ON EXIT OR EVICT: transfer output files when job completesor is evicted32

File Transfer Example# changed cosmos.sub fileUniverse Executable Log Transfer Input Files Transfer Output Files Should Transfer Files When To Transfer Output tIF NEEDEDON EXIT

Command Line ArgumentsUniverse vanillaExecutable cosmosArguments -c 299792458 –G 6.673e-112. . .QueueInvokes executable withcosmos –c 299792458 –G 6.673e-112Look at the condor submit man page to seesyntax for Arguments. This example hasargc 5.34

Job Id isClusterId.ProcId()ClassAdattributes A set of related jobs is called a cluster Each cluster has a cluster number, an unsigned integervalue unique to the job queue on a submit host Each individual job within a cluster is given a processnumber, and process numbers always start at zero A Job ID is the cluster number, a period, and theprocess number. Examples: Job ID 20.0cluster 20, process 0 Job IDs: 21.0, 21.1, 21.2cluster 21, processes 0, 1, 235

1 Cluster, 2 JobsUniverse vanillaExecutable cosmosLogInputOutputQueueLogInputOutputQueue cosmos 0.log cosmos 0.in cosmos 0.outjob 102.0 cosmos 1.log cosmos 1.in cosmos 1.outjob 102.136

File OrganizationA logistical nightmare places all input, output,and log files in one directory. 3 files 1,000,000 jobs 3,000,000 files The submit description file is 4,000,000 linesThe directory will be difficult (at best) to evenlook at.37

Better Organization Create a subdirectory for each job, intentionallynamedrun 0, run 1, run 999999 Implement the creation of directories with a program(such as Python or Perl) Create or place input files in each of theserun 0/cosmos.inrun 1/cosmos.in run 999999/cosmos.in The output and log files for each job will becreated by the job, when the job runs.38

Einstein’s simulation directorycosmoscosmos.subrun 0cosmos.inSubmitter or scriptcosmos.outcreates black-fontcosmos.logfilesHTCondorcreatesrun g39

Better Submit Description File# Cluster of 1,000,000 jobsUniverse vanillaExecutable cosmosLog cosmos.logOutput cosmos.outInput cosmos.inInitialDir run 0QueueInitialDir run 1Queuejob 103.0job 103.1This file contains 999,998 more instances ofInitialDir and Queue.40

Queue all 1,000,000 instances of this simulationwith the single command:Queue 100000041

Submit Description File MacrosWithin the submit description file, HTCondor permitsnamed macros: (Process) will be expanded to the process number foreach job in the cluster.For this example, values will be 0 – 999999 for the1,000,000 jobs.42

Using (Process) The initial directory for each job can be specified.InitialDir run (Process)becomesrun 0, run 1, , run 999999 Similarly, command-line arguments may use amacro.Arguments -n (Process)becomes-n 0-n 1 . . .43-n 999999

(Best) Submit Description File# Example: a cluster of 1000000 jobsUniverse vanillaExecutable cosmosLog cosmos.logInput cosmos.inOutput cosmos.outInitialDir run (Process)Queue 100000044

Albert submits the cosmossimulation. Patience required,it will take a while condor submit cosmos.subSubmitting job(s).Logging submit event(s).1000000 job(s) submitted to cluster 104.45

the Job Queue condor q-- Submitter: submit.chtc.wisc.edu : 128.104.55.9:51883 : submit.chtc.wisc.eduIDOWNERSUBMITTEDRUN TIME ST PRI SIZE CMD104.1einstein 4/20 12:08 0 00:00:03 R 0 9.8 cosmos104.2einstein 4/20 12:08 0 00:00:01 I 0 9.8 cosmos104.3einstein 4/20 12:08 0 00:00:00 I 0 9.8 cosmos.104.999998 einstein 4/20 12:08 0 00:00:00 I 0 9.8 cosmos104.999999 einstein 4/20 12:08 0 00:00:00 I 0 9.8 cosmos999999 jobs; 999998 idle, 1 running, 0 held46

HTCondor watches overthe jobs, runs each one tocompletion once, restartingany that do not finish.Time for a cold one!47

More That You Do With HTCondor48

Remove Jobs with condor rm You can only remove jobs that you own Privileged user can remove any jobs root on Linux administrator on Windowscondor rm 4condor rm 4.2condor rm –aRemoves all cluster 4 jobsRemoves only the job withjob ID 4.2Removes all of your jobs.Careful !49

Specify Job Requirements A boolean expression (syntax similar to C or Java) Evaluated with respect to attributes from machineClassAd(s) Must evaluate to True for a match to be madeUniverseExecutable. vanilla mathematicaRequirements ( \HasMathematicaInstalled ? True )Queue 2050

Specify Needed ResourcesItems appended to job Requirementsrequest memory – the amount of memory (inMbytes) that the job needs to avoid excessiveswappingrequest disk – the amount of disk space (inKbytes) that the job needs. Will be sum of spacefor executable, input files, output files andtemporary files. Default is size of initial sandbox(executable plus input files).request cpus – the number of CPUs (cores) thatthe job needs. Defaults to 1.51