- Home
- /
- Documentation
- /
- General Instructions for Job...
- /
- Making Use of /scratch
Making Use of /scratch
Making Use of Local /scratch
If a job produces large files and runs only at one worker node, it is recommended to use a local scratch directory (/scratch) available at a local disk of an assigned worker node.
It is necessary to add a requirement for size and type of a scratch directory to the qsub command. No scratch is set as a default.
#PBS -l select=1:ncpus=8:mem=20gb:scratch_local=50gb
Upon a job start, the batch system creates a unique subdirectory at a local scratch directory,
/scratch*/username/job_JobID.meta-pbs.metacentrum.cz
pointed to by the $SCRATCHDIR variable.
In case you need to check your temporary files at the local scratch subdirectory, log in to the assigned node first. The scratch subdirectory is not accessible directly from the login server where a job has been submitted.
Note: The /scratch*/username directory at worker nodes is NOT WRITABLE – it has permissions 555.
To make use of the scratch directory, modify your job script in following way:
- copy input files to $SCRATCHDIR
- run computations in $SCRATCHDIR
- copy back output files from $SCRATCHDIR to the working directory at /storage/locality/, where the job has been submitted ($PBS_O_WORKDIR)
- clean up $SCRATCHDIR – use a clean_scratch utility to perform safe scratch cleanup. Add the following lines to your job script:
trap ‘clean_scratch’ TERM EXIT ... cp results /storage/... || export CLEAN_SCRATCH=false
Temporary data in a local /scratch are automatically removed in one or two weeks.
Node Type | Total /scratch |
---|---|
luna65-95 | 800 GB |
luna96-99 | 1.7 TB |
luna100-105 | 3.4 TB |
luna106-108 | 3.4 TB |
luna201-206 | 800 GB |
magma1-23 | 3.4 TB |
Making Use of Shared /scratch
The clusters Luna have no shared scratch. It is possible to make use only of a local scratch directory available at each worker node or of a user directory /storage/praha6-fzu/home/username/ NFS mounted from a local disk array.