Submitting Jobs with Late Materialization Feature

When submitting thousands of jobs with a single submission file we recommend to use HTCondor feature „late materialization“ to avoid overloading the batch scheduler. It is essential to limit a number of jobs in the Idle state.

Use two parameters in your job submission file: max_materialize and max_idle.

The max_materialize limit specifies an overall limit on the number of jobs that can be materialized in the scheduler at any one time.
The max_idle limit specifies the maximum number of non-running jobs that should be materialized in the scheduler at any one time.

Example:

max_materialize = 200
max_idle = 10