Skip to content
Blog

Your cron job stopped and nobody noticed. Heartbeat monitoring, explained.

Uptime checks cannot see a backup script or a queue worker, because there is nothing to probe. Heartbeat monitoring turns the check around: the job reports in, and silence is the alarm. How it works, how to size the grace period, and the mistakes that make it useless.

The failure that hurts most is the quiet one. The nightly backup that has not actually written a file since a credential rotated in March. The queue worker that crashed on Friday and has been holding 40,000 jobs since. The report that stopped emailing and nobody asked why. None of these show up on an uptime monitor, because an uptime monitor asks "does this URL answer?" and a cron job does not have a URL.

Heartbeat monitoring fixes the question. Instead of the monitor reaching out to your job, your job reaches out to the monitor. It pings a unique URL every time it finishes. If the pings stop, that is the outage, and it is detected by absence rather than by error.

This post covers how that works, how to set the two numbers that matter, and the three mistakes that turn a heartbeat into decoration.

Why probing does not work for scheduled work

An HTTP monitor is a pull: a probe sends a request on a schedule and judges the response. That model needs something listening. A cron job runs, exits, and is gone. Between runs there is nothing to talk to, and while it runs there is usually still nothing to talk to. You could expose a "last run" endpoint and probe that, but now you have built a small web service to monitor a shell script, and the endpoint is up even when the job it describes is broken.

The job's own success is the only signal worth having. Heartbeat monitoring makes that signal the whole check.

The mechanism

On RealUptime, a heartbeat monitor is a token. When you create one, you get a URL of the form /api/ping/YOUR_PING_TOKEN. Your job requests it when it completes. Nothing else is sent: no body, no JSON, no customer data. The only thing the monitor learns is that a run finished, and when.

The rules that follow from that are simple.

  • Silence past interval plus grace is down. You tell the monitor how often to expect a ping (the interval) and how much slack to allow (the grace period). A ping that does not arrive within interval plus grace marks the monitor down, and the same alert channels every other monitor uses fire: incidents, status page components, Slack, email, PagerDuty, webhooks.
  • The next ping is recovery. No manual resolution, no second confirmation. The job ran, so the job is back.
  • A never-pinged monitor is not down. A heartbeat you just created but have not wired into anything sits in "awaiting first ping" and pages nobody. Monitoring arms on the first ping the token receives. This matters more than it sounds: a monitor that alarms before it is connected teaches the team to mute alarms on day one.
  • No probes are involved. Nothing reaches into your infrastructure. The ping URL is the only integration surface, so a heartbeat works from a machine with no inbound access at all: a laptop, a box behind NAT, a container with no exposed ports.

The token is the credential. Treat the ping URL like an API key, and regenerate it from the dashboard if it leaks. Pings are rate limited to 120 requests per minute per token, which is far more than any correctly configured job will send and enough to stop a tight-loop mistake from becoming unbounded traffic.

Setting the interval and grace period

The interval is the job's schedule. A job on */5 * * * * gets a five-minute interval. A nightly job gets 24 hours. This is the one monitor type where the interval is not a detection-speed trade-off at all; it is a statement of fact about the job. Heartbeats on RealUptime share the same 60-second floor and 24-hour ceiling as every other monitor type.

The grace period is where the judgment goes. It exists because job runtimes vary, cron fires late under load, and clocks drift. Set it too tight and a job that took four minutes instead of three pages you for nothing. Set it too loose and a dead job hides inside the slack.

A rule that works: grace equals the job's worst normal runtime, plus a little. If a backup usually takes 20 minutes and has taken 35 on a bad night, a 45-minute grace on a 24-hour interval catches "did not run" within 45 minutes of the expected finish, without paging on a slow night. For a five-minute job that finishes in seconds, a one-minute grace is plenty.

Two things people get backwards:

  • Do not fold the runtime into the interval. A job that runs hourly and takes 20 minutes is still an hourly job. Interval 60 minutes, grace 25 minutes. Setting the interval to 80 minutes "to be safe" delays every detection by 20 minutes forever.
  • Grace protects against late, not against missing. If the job genuinely runs at irregular times, a heartbeat is the wrong tool for it; use the widest interval it can honor and accept the detection delay, or fix the schedule.

Ping on success, and only on success

This is the mistake that makes heartbeats worthless, and it is the most common one. The crontab line most people write first is:

*/5 * * * * /usr/local/bin/backup.sh; curl -fsS https://ingest.example/api/ping/TOKEN > /dev/null

The semicolon runs the curl whether the backup succeeded or not. A backup that exits non-zero every night for a month still pings every night, and the heartbeat stays green the entire time. You have monitored that cron is alive, not that the job works.

Use && instead:

*/5 * * * * /usr/local/bin/backup.sh && curl -fsS https://ingest.example/api/ping/TOKEN > /dev/null 2>&1

Now the ping only fires when the script exits zero. A failing job goes silent, silence becomes down, down becomes an alert. For scripts that do not set exit codes carefully, fix the script; a heartbeat cannot be smarter than the signal it is given.

The -f flag on curl matters for the other direction: it makes curl exit non-zero on an HTTP error, so a misconfigured token shows up in the job's own logs instead of being swallowed.

Where to put the ping

At the end of the work, after the thing you actually care about has happened. Some concrete placements:

  • Backups: after the file is written and, if you can, after it is verified. A ping placed before the upload confirms that the script started, which is not the question.
  • Queue workers: workers do not "finish," so ping from the loop. A worker that processes a batch and pings after each batch turns into an hourly heartbeat with an interval a bit above the batch cadence. If the worker crashes, the batches stop, the pings stop.
  • Scheduled reports and emails: ping after the send returns success from the mail provider, not after the template renders.
  • Data pipelines: ping at the end of the last stage. A ten-stage pipeline with a ping after stage one will look healthy for months while stages two through ten rot.

What heartbeats do not catch

Honesty about the limits keeps the tool trusted.

  • A job that runs and does the wrong thing. If the backup script writes an empty file and exits zero, the heartbeat is green. Add a check inside the job (file size, row count) and make the ping conditional on it.
  • A job that runs late but within grace. By design. If "late" is itself a problem for you, tighten the grace and accept more pages.
  • The monitor's own delivery. The ping is an HTTP request from your job; if your job's host loses outbound network, the ping fails and the heartbeat goes down. That is a correct alert (the host cannot reach the internet), but it is not the alert you might assume it is. Read the incident before restarting the job.

Setting one up in a minute

Create a monitor, choose Heartbeat, copy the ping URL, and append the && curl line to the crontab entry. The monitor shows "awaiting first ping" until the job runs once, then arms. The full contract, including the rate limit and the arming rule, is in the heartbeat docs. Heartbeats are included in the free tier alongside HTTP and TCP checks, so the backup you have been meaning to watch can be watched tonight.

The uptime monitor on your website tells you when customers cannot reach you. The heartbeat on your backup tells you, before you need the backup, whether there is one.