Skip to content
Blog

The page you should not have gotten

False uptime alerts train teams to ignore real ones. How we decide a failure is real before waking anyone up, and what we refuse to page on.

Ask an on-call engineer about their monitoring and the complaint is rarely "it misses outages." It is "it pages me for nothing." A timeout that resolved itself. A probe with a bad route. DNS that hiccuped for one resolver in one city. Each false page costs a little sleep and a lot of trust, and a team that has been burned enough starts doing the truly dangerous thing: acknowledging alerts without looking.

The uncomfortable truth is that a single check from a single location cannot distinguish "your site is down" from "the path between one probe and your site had a bad moment." Those two events look identical in the data. The only way to tell them apart is to look again, from somewhere else, before saying anything.

Here is how we make that call.

Confirm before you page

A first failed check on RealUptime does not open an incident and does not page anyone. It schedules a fast confirmation re-check about five seconds later, ahead of the normal probing cycle, so confirmation is quick rather than waiting out a full interval. Only a second consecutive failure opens the incident and starts the alert path (the channels and routing that path can take are documented at /docs/alerting).

This is a tradeoff, stated openly: you learn about a real outage a few seconds later than the theoretical minimum, and in exchange the one-probe-blip page mostly stops existing. For the thing alerts are actually for, getting a human to act, seconds of confirmation delay are noise; a human's response time is measured in minutes. Trust in the pager is the scarce resource, and this is what it costs to keep it.

Four regions, and what disagreement means

Our probes run from Virginia, California, Frankfurt, and Tokyo (the region list is at /docs/checks). When all four agree your site is down, that is about as unambiguous as monitoring gets. The interesting cases are the disagreements.

Frankfurt failing while three regions pass is usually not your bug, but it is not nothing either: your German users are seeing it too. So the regional state is shown as exactly that, a regional problem, rather than being rounded up to "down" or rounded down to "fine." We wrote more about why collapsing the world into one number misleads in One internet, four speeds.

"We could not check" is not "it was down"

There is a third state most monitors quietly fold into one of the other two. Sometimes the failure is on our side: a database blip while recording a result, a slow check abandoned by its deadline, a resolver that accepted a DNS query and never answered. Counting those as your downtime would corrupt your uptime numbers with our operational noise.

So we count them separately, as errors on our side rather than failures on yours, and they do not page you. They page us. Your availability history should contain your outages, not ours.

What to steal from this if you run your own alerting

None of this requires our product; the principles port anywhere. Confirm before paging, always, from a second vantage point if you have one and a second attempt if you do not. Treat single-region failure as its own signal instead of rounding it to up or down. Keep "the check failed" and "the checker failed" in separate columns, forever. And watch your own acknowledge-without-looking rate, because it is the leading indicator that your alerting has started training people to ignore it.

A pager that only goes off for real problems is not a luxury feature. It is the entire product.