A cron job cannot tell you it did nothing

31 Aug 2026

A scheduled job has two ways to fail. It can exit non-zero, which your alerting probably catches. Or it can exit zero having accomplished nothing, which it almost certainly does not.

The second one is expensive precisely because every signal you own looks healthy. The job ran. It ran on schedule. It did not complain.

The shape it usually takes

A worker guarded the way every shell style guide tells you to guard one:

set -euo pipefail

grep -q '^ready' /var/lib/app/state   # returns 1 when the marker is absent
process_batch                         # never reached

grep exits non-zero when it matches nothing. That is not an error, it is an answer — but set -e cannot tell the difference, so the script ends there. The scheduler observes a process that started and exited, starts it again on the next tick, and watches it exit again.

The error is not hidden. It is written down — one line, in the log of a job that runs every ten minutes, which is the single least-read location in your infrastructure. What is missing is not the error message. It is anybody asking whether the work happened.

This shape survives for weeks, and the first symptom is usually somewhere else entirely: whatever the job was draining has been filling up the whole time, so you find out from a disk alert on a machine that has nothing to do with the scheduler.

An exit code describes the process, not the work

exit 0 means the shell reached the end of the script. It is a statement about control flow. Whether any rows moved, any files were written, or any queue got shorter is a different question, and nothing in the exit status is even trying to answer it.

Every "job succeeded" check built on exit status inherits that gap.

set -e is not as on as you think

It is worth knowing where it stops applying, because the places it does are the places people put error handling. On bash 5.2:

set -e; if false; then :; fi; echo REACHED    # REACHED — suppressed inside if
set -e; false || true;        echo REACHED    # REACHED — suppressed left of ||
set -e; false | true;         echo REACHED    # REACHED — pipeline status is the last command
set -eo pipefail; false | true; echo REACHED  # not reached

And the one that costs the most:

set -e
f() { false; echo STILL-IN-F; }
f || echo HANDLED

That prints STILL-IN-F and exits 0. Calling a function on the left of || disables set -e for everything inside it, so a function written on the assumption that it halts at the first failure runs to completion and reports success. set -u is the well-behaved one: an unbound variable is fatal, and ${VAR:-default} is how you opt out deliberately.

Guard the failure you expect, and only that one

The reflex fix makes it worse:

grep -q '^ready' /var/lib/app/state || true   # a missing file is now fine too

That converts one known-benign failure and every unknown one into success. Say what you actually mean:

if ! grep -q '^ready' /var/lib/app/state; then
    log "no ready marker; nothing to do"
    exit 0
fi

Same outcome for the expected case, and it is now a decision somebody made rather than a side effect of an option.

Alert on the work, not on the run

The durable fix is a signal that only exists if the job did something:

• A watermark the job advances, and an alarm when it stops moving. Advance it after the batch succeeds, never before — an optimistic watermark turns a crash-loop into silent data loss instead of a silent backlog.
• A count with a time window. "Rows moved in the last 24 hours" is a number that goes to zero the moment the job breaks, whatever the exit code was. It also catches the case where the job runs, succeeds, and processes nothing because a filter is wrong.
• A heartbeat the job emits at the end, with an alert on absence rather than on failure. A job that never runs at all produces no failed run to notice, so monitoring built only on failures cannot see a scheduler that stopped scheduling.

The common property is that all three measure the outcome. An exit code cannot tell you the job did nothing, because from the shell's point of view doing nothing went perfectly.

Alert on the absence of work, not the presence of errors. If the only thing your monitoring would notice is a non-zero exit, then a job that crash-loops before it starts, a job that filters everything out, and a job the scheduler quietly stopped running all look exactly like a job with nothing to do.