The best practice to avoid problems with overload and slow response of NFS servers is: read all input files from the NFS server only once when your job starts, use /scratch at a local disk of an assigned worker node as your working directory, write all output files back to the NFS server when your job completes.

A local working directory /scratch is available at a local disk of each worker node, and is intended for temporary files. Upon a job start, the batch system creates a unique subdirectory /scratch/condor/dir_ID/, which is accessible via the $_CONDOR_SCRATCH_DIR variable. When the job finishes, the subdirectory is automatically deleted. This way, the local working directory /scratch is continuously cleaned.

Add a requirement for /scratch to a job_submission_file.

request_disk = 20G
Worker Node NameTotal /scratch/scratch per one HTcore
milan01-293.3 TB25.7 GB
minis01-12722 GB22.5 GB
mahagon01-241.6 TB25 GB
mikan01-51941 GB19.6 GB
Přejít nahoru