Cron Job Alerting That Doesn't Drive You Crazy

Set up monitoring on 15 cron jobs last year. Within a week I had alert fatigue so bad I started ignoring all notifications. The pager was crying wolf constantly and every real issue got lost in the noise.

Here's what went wrong and how I fixed it.

The over-alerting trap

My first setup was straightforward: alert if a job takes longer than expected, alert if it fails, alert if it doesn't run. Sounds reasonable until you realize that your hourly sync job occasionally takes 90 seconds instead of 60, and that 50% threshold triggers an alert every few days.

Multiply that by 15 jobs and you get 5-10 false alerts per day. At that point you stop reading them.

What actually deserves an alert

I ended up with three tiers:

Critical (page me): The job didn't run at all. This is the only thing I wake up for. A missing heartbeat means either the server is down or crontab is broken. Both need immediate attention.

Warning (Slack message): The job ran but something was off. Duration 3x longer than average. Non-zero exit code. Output validation failed (zero rows processed, empty file generated). I check these within a few hours during work time.

Info (daily digest): Duration anomalies under 2x. Successful runs with minor warnings. Job overlap detected but resolved. I review these once a day over coffee.

The baseline problem

Fixed thresholds don't work well for cron jobs. Your sync job takes 30 seconds on Monday and 3 minutes on the first of the month when there's more data. A fixed 60-second threshold fires every month-end.

What works better: track the 95th percentile duration over the last 30 runs. Alert when the current run exceeds that by 2x. This adapts to the job's actual behavior pattern without manual tuning.

I track this in a simple SQLite database. Each run logs its duration, and a quick query gives me the baseline:

`SELECT percentile(duration_ms, 95) FROM job_runs 
WHERE job_name = 'sync' 
AND run_at > datetime('now', '-30 days');
`

Alert routing matters

Not every job is equally important. The billing sync failing at 2am? That needs a phone call. The analytics aggregation failing? That can wait until morning.

I group jobs by business impact:

• Revenue-affecting → PagerDuty, phone call, escalation after 15 min

• User-facing → Slack #alerts, acknowledge within 2 hours

• Internal/analytics → Slack #monitoring, review next business day

Silence windows

Deploy days are noisy. Servers restart, connections drop, jobs fail transiently. Instead of disabling monitoring (dangerous), I set a silence window for warnings during deployment. Critical alerts still fire — if a job completely stops running during deploy, I want to know.

Same for maintenance windows. If the database is down for a planned migration at 3am, I don't need 15 "job failed" alerts.

The current setup

After iterating on this for months, my cron monitoring now generates maybe 2-3 real alerts per week. Each one is actionable. I actually read them and respond.

The key insight: monitoring cron jobs is easy. Making the alerts useful without driving yourself insane — that's the actual work.