Running dependent jobs with DAGMan (condor_submit_dag)

If your work is really several steps that depend on each other — a pre-processing step, then many parallel analysis jobs over its output, then a final merge/cleanup step — submitting and babysitting them by hand quickly becomes tedious and error-prone: you have to wait for each stage to finish, check it actually succeeded, and only then submit the next one. DAGMan does this bookkeeping for you: submit the whole workflow once, and it starts each job automatically as soon as (and only once) its dependencies have completed, retries failures if you ask it to, and lets you resume cleanly from wherever a failed run left off instead of restarting everything from scratch.

DAGMan (Directed Acyclic Graph Manager) lets you describe a workflow of several HTCondor jobs that depend on each other — e.g. “run analyze only after prepare has finished successfully”. HTCondor submits and tracks the individual jobs for you, retries failed ones if you ask it to, and only advances to the next stage once its dependencies are done. Everything below is submitted the same way as any other job on our farm: from a login node (ui, i.e. ui1/ui2/ui3.farm.particle.cz), which forwards your submission to the shared scheduler on condor.farm.particle.cz.

Building blocks

A DAG is described in a plain text .dag file. Each node in the graph is a normal HTCondor job with its own .sub submit file — the .dag file only describes how the nodes relate to each other. As with any job on our farm, keep the .dag file, the .sub files and their output/error/log paths under your /home directory (e.g. ~/jobs/) — /home is shared across ui and all worker nodes and is always available, unlike /mnt/ (see the addendum below).

# ~/jobs/mydag.dag
JOB  prepare   prepare.sub
JOB  analyze   analyze.sub
JOB  cleanup   cleanup.sub

PARENT prepare CHILD analyze
PARENT analyze CHILD cleanup

prepare.sub, analyze.sub and cleanup.sub are ordinary HTCondor submit files for our farm, written exactly as you would for condor_submit — including requesting a container image if you use one, e.g.:

# ~/jobs/analyze.sub
universe      = vanilla
executable    = analyze.sh
+SingularityImage = "/cvmfs/farm.particle.cz/singularity/fzu_wn-centos8"
output        = analyze.out.$(Cluster)
error         = analyze.err.$(Cluster)
log           = analyze.log
requestcpus   = 1
requestmemory = 2G
queue

PARENT ... CHILD ... lines define the dependency order — here analyze only starts once prepare has completed, and cleanup only starts once analyze has completed.

Useful extras

RETRY analyze 3 — retry the analyze node up to 3 times if it fails before giving up. Useful on our farm for nodes that touch a busy shared NFS server and occasionally hit a transient error.
SCRIPT PRE analyze check_input.sh / SCRIPT POST analyze cleanup_tmp.sh — run a local script on ui immediately before/after a node runs, useful for quick checks or bookkeeping that doesn’t need a full HTCondor job.
VARS analyze dataset="run042" — pass a value into a node’s submit file as $(dataset), handy for reusing the same .sub file across many similar nodes (e.g. one per dataset/run).
More advanced control (NOOP, ABORT-DAG-ON, sub-DAGs, throttling with maxjobs, …) is documented in the official HTCondor DAGMan manual and applies unchanged on our farm.

Submitting and monitoring

From ui, in the directory holding your .dag and .sub files:

cd ~/jobs
condor_submit_dag mydag.dag

This creates one extra HTCondor job — the DAGMan controller — which then submits and tracks the individual nodes on your behalf as their dependencies are satisfied. Watch overall progress the same way you would any job on our farm, with condor_q (the controller job and any currently-running nodes will show up there under your username). DAGMan also writes a running log to mydag.dag.dagman.out next to your .dag file, which is the first place to look if something isn’t behaving as expected.

If the whole DAG fails partway through, fix the problem and resubmit the same condor_submit_dag mydag.dag command from ~/jobs — DAGMan keeps track of which nodes already succeeded (in a .dag.rescue file next to it) and only reruns what’s left.

Addendum: DAGs and shared /mnt disks

This section only covers what’s different for DAGs regarding the /mnt/ access restriction described in Requesting shared /mnt disks in HTCondor jobs. Read that page first if you haven’t already.

The DAGMan controller job itself is not affected. It runs in HTCondor’s scheduler universe, directly on condor.farm.particle.cz rather than on a worker node, so the new restriction does not apply to it. Files such as mydag.dag.dagman.out remain accessible exactly as before — nothing to change for the DAGMan job itself, and nothing to add to the condor_submit_dag command line.

The individual node jobs are affected, the same as any other job on our farm. Each node runs on a worker node using its own .sub file, so if a node needs a folder under /mnt/, add the usual line to that node’s own submit file (not to the .dag file, and not to condor_submit_dag):

+<name> = true

Different nodes can request different folders; each .sub file only needs the lines for what that particular node actually uses. For example, if analyze.sub from above reads from /mnt/nfs17 and writes to /mnt/users_private:

# ~/jobs/analyze.sub
universe      = vanilla
executable    = analyze.sh
+SingularityImage = "/cvmfs/farm.particle.cz/singularity/fzu_wn-centos8"
+nfs17 = true
+users_private = true
output        = analyze.out.$(Cluster)
error         = analyze.err.$(Cluster)
log           = analyze.log
requestcpus   = 1
requestmemory = 2G
queue

You can add these lines to your node .sub files now, ahead of the cutover date, with no effect on current behavior — the same as for any other job on our farm.