Automated E-mails from the Golias Computing Farm

The Golias farm automatically sends users two kinds of e-mail, both from an address of the form @condor.farm.particle.cz, with a subject starting with [Golias farm]:. Replies go to fzu@rt.cesnet.cz (RT/CESNET).

Receiving one of these e-mails carries no penalty. They are informational, meant to help you notice and tune workloads before they become a problem. Only users who severely and persistently violate the limits are ever escalated – and even then, the process always starts with the admin team contacting you directly. No account is ever suspended without first being contacted and given a chance to fix the issue.

The one exception: in rare cases where a workload jeopardises the stability of the whole farm (e.g. overloading shared storage or the network for all users), the admin team may put the offending jobs on hold immediately, before contact is made, purely to protect the farm – you would then either be contacted right after, or the hold reason on the job itself will tell you why.

1. Weekly job report

We know that some workloads are inherently hard to tune and behave a bit unpredictably (input data that varies in size, shared/contended storage, etc.). This report isn’t a judgment on that – it exists mainly to let you see how your jobs are doing in the first place, so you have the numbers in hand if something is worth adjusting.

When it arrives: once a week, Monday morning, covering the previous week (Monday–Sunday). Every user who had at least one job finish during that week gets one.
​Subject: [Golias farm]: weekly job report for user ‚<login>‘ (YYYY-MM-DD – YYYY-MM-DD)

The e-mail contains a summary table for the whole week:

• job counts – total, how many completed OK, how many failed “fast” and how many “slow” (see below)
• CPU time and walltime – summed over all of that week’s jobs
• CPU efficiency – the ratio of CPU time actually used to CPU time allocated (cores × walltime)
• core and memory usage – requested vs. actually used
• disk usage – requested vs. actually used on the local scratch space
• the share of “good” vs. “problematic” jobs for the whole week

What the thresholds that flag a job as problematic mean:

Metric	Threshold	Meaning
CPU efficiency	< 30%	the job spends most of its time waiting instead of computing
Core usage	< 75%	more CPU cores were requested than the job actually used (does not apply to 1-core jobs)
Memory usage	< 80%	significantly more memory was requested than the job consumed
Disk usage	< 70%	significantly more scratch disk was requested than the job consumed
Short job	< 20 min runtime	the job finished suspiciously fast – 5 or more such jobs in a week trigger a warning banner
NFS suspect	local disk usage < 50 KiB	the job likely never touched its scratch space and read/wrote directly to NFS instead – 5 or more trigger a warning banner
Fast fail	held/removed within 3 min	the job likely crashed right at startup (bad executable, missing input, config error)
Slow fail	held/removed after > 3 min	the job ran for a while before failing – its usage is still counted in the averages above

Note on NFS suspect: this can severely hurt your own job efficiency. NFS I/O is far slower than local scratch, so a job reading/writing directly to NFS spends much more time waiting on the network instead of computing.

If the share of problematic jobs for the week is 5% or less, the e-mail praises the user anyway – only above that threshold does it explain which specific jobs are flagged and why. In that case a single zip attachment is included, bundling the relevant CSVs (job ID, reason, individual metrics), so the problem can be tracked down without having to query anything yourself. The e-mail also ends with the exact condor_history command needed to pull the same data (for that week) manually.

2. Low running-job efficiency warning

When it arrives: unlike the weekly report, this warning watches jobs that are currently running, checked hourly. If a user’s aggregate CPU efficiency across running jobs stays below 30% for three consecutive hourly checks (i.e. roughly 3 hours in a row), an e-mail is sent. Only users with enough concurrently running jobs are checked, so a random dip in a couple of jobs doesn’t trigger a needless warning.
​Subject: [Golias farm]: low job efficiency for user ‚<login>‘ (warning) or (repeated warning) for a repeat notification.

Rules to keep the e-mails from flooding your inbox:

• at most one notification per user per day, even if the condition persists longer
• repeat notifications within the same uninterrupted low-efficiency streak also state the running average efficiency and the date the problem started
• if the condition lasts more than 3 days, the admin team is informed once (for the whole period) – this typically means the issue is worth a manual look / contacting the user

Low running-job efficiency usually means the same thing as in the weekly report: jobs are waiting (on I/O, network, file locking, etc.) instead of computing, needlessly holding allocated CPU cores that other users could use. The fix is typically the same – check whether jobs are waiting on slow storage/network, whether the requested number of CPU cores (RequestCPUs) matches actual usage, and whether there is unnecessary serial waiting inside the job.

Summary of the difference between the two e-mails

	Weekly report	Efficiency warning
Frequency	once a week (Monday)	live, at most once a day per user
Covers	jobs that finished during the whole week	currently running jobs, last ~3 hours
Purpose	overall overview and resource-request tuning advice	early warning about currently wasteful jobs
Attachments	single zip with CSVs listing the specific problematic jobs (if any)	none

What to do when you receive one of these e-mails

1. Read which metric is over the threshold (see the tables above) – the e-mail and the CSVs in the zip attachment (if any) name the specific jobs.
2. Adjust your submit file (RequestCpus, RequestMemory, RequestDisk) to match actual consumption, or adjust the job itself so it doesn’t needlessly wait on I/O/network.
3. If you’re unsure of the cause, or it keeps recurring, reply to the e-mail (it goes to fzu@rt.cesnet.cz) – the administrators will help.