The worst failure an uptime monitor can have is not going down. It is staying up and going quiet: probes stop running, dashboards stay green, and the customer finds out their site was down from a customer of their own. The monitor did not lie exactly. It just stopped telling the truth and told nobody.
We have had two brushes with this ourselves. On 2026-08-10, a wedged process held a region's lock for nine hours while every machine reported healthy. On 2026-08-13, a deploy race meant nine minutes where nothing probed at all. Neither incident broke any code path. In both cases the system noticed something and told no one.
So we started keeping a register: every way we know the probe can fail, what the system does about it today, and, the column that actually matters, whether anyone finds out. This post is a tour of the mindset and the mechanisms that came out of it.
The recurring failure is silence, not crashes
Crashes are the easy case. A crashed process stops writing heartbeats, heartbeats go stale within two minutes, and staleness pages us on a single failing poll. The process that would have noticed is dead, so something outside it has to do the noticing. That outside thing exists and is itself monitored.
The hard cases are the ones where everything keeps running:
A region that stops probing while its machines report healthy. Each of our four regions is owned by exactly one process at a time, via a database lock. If a process dies without releasing that lock, the survivors politely wait, and nothing probes the region. Today a passive process watches its region's heartbeat, and if the supposed owner goes quiet it records an operational error naming the runbook, within about a minute. The lock table also names who we believe holds each region and when they last renewed, so a human can check the claim directly.
A check that errors on our side of the wire. A database blip while recording a result is our failure, not the customer's outage. Those used to be swallowed by a pool's catch block and counted nowhere. They are now tallied separately from real check failures, because confusing "we could not look" with "we looked and it was down" is exactly the kind of lie the whole product exists to prevent.
An interval that quietly drifts. We advertise 60 second checks. For a while, production actually delivered them every 75 seconds, an artifact of when a timestamp was stamped relative to a 15 second scheduling grid. No error, no crash, just a product doing 25% less than its own pricing page said. The fix was straightforward; the interesting part is the guard, which now measures the achieved gap directly and fails our own health checks if it drifts past 1.2x, so the same regression cannot happen silently twice.
What we deliberately do not defend
Honesty also means saying what the register does not cover. We do not self-heal a stranded region lock, because forcibly terminating a database session on the strength of a two minute old timestamp is a destructive action, and a human confirms the signature first. And a target that returns a cheerful 200 on a broken page is out of scope for most check types: we can tell you your server answered, not that the answer was any good.
Why write this down publicly
Because "we monitor the monitor" is a claim every vendor makes, and it is unfalsifiable as marketing copy. A register of specific failure modes, with the embarrassing incidents that motivated them, is checkable. It also keeps us honest: the rule in our codebase is that any change to the probe updates the register in the same pull request, so the document cannot rot into aspiration.
If you run infrastructure, we would gently suggest the exercise: list the ways your monitoring can fail, and for each one write down who finds out, and how fast. The rows where the answer is "nobody" are the outages you have not had yet.