The guard had no clock

A scheduled job that watches our products from outside them succeeded for the last time at 14:54:14 UTC on 24 August 2026. Every run after that was created on time, every fifteen minutes, and none of them ran. That went on for eighteen days. Two health probes were pointed at that job for the whole period and both reported green.

The failure under the chart is ordinary and takes a paragraph. The reason nobody heard about it for eighteen days is the subject of the rest of this post, and it is a defect you can go and look for in your own monitoring this afternoon.

A dead credential makes a queue, not an error

The job runs on a self-hosted runner started fresh for each job and thrown away after. A fresh runner has to register itself, and to register it needs a short-lived token, and to get that token it needs a long-lived credential. That credential had stopped working. Asking for a registration token returned 401 Bad credentials.

The last step sets the shape of the whole outage. No registration means no runner. No runner means the job is never picked up. A job nobody picks up does not fail; it waits. Every one of those runs carries an empty runner name and an empty list of steps, which is the fingerprint of a job that never started.

scheduler   creates a run                       -> queued
runner      POST .../actions/runners/registration-token
            <- 401 Bad credentials
            no token -> no registration -> no runner
run record  runner_name: ""   steps: []   conclusion: cancelled

The suspect that fit

An obvious cause sat right there. The account this job runs under had a pending verification and a known CI problem going back weeks. Everyone who looked at the outage reached for that first, including us. It was wrong.

Two things settled it. Other jobs on the same account had successful runs inside the same window, so the account was not blocked. And a job that runs out of minutes leaves a different mark: a failure about two seconds long, no steps, delivered at once. This was a queue, and a queue is a different failure from a refusal.

The setting that took the alarm away

A job with no runner does eventually give up and fail, and a scheduled job that fails sends mail. That mail never went out. The job was set to cancel any run still in progress when the next one starts, and it ticks every fifteen minutes, so each starved run was killed by its successor long before it could fail.

The platform sends mail on a failed scheduled run and sends nothing on a cancelled one, so the concluded state decided whether a human heard anything. That setting did not cause the outage. It converted a loud outage into a silent one, which is a worse thing to own.

The pass condition with no clock

Now the part worth stealing. A separate probe existed for exactly this job, and its one task was to notice if the job stopped. It passed whenever a run newer than the last one it had seen existed. That reads as reasonable until you write down what the broken state looks like.

Under this failure, runs were created every fifteen minutes and killed before they started. A newer run always existed. The outage satisfied the probe's passing condition, permanently, and went on satisfying it. A guard that can be held green by the failure it was built to detect is worse than no guard, because it also occupies the place where a working one would go.

The plate at the top of this page is that condition drawn. The rope still moves on the quarter hour and the clapper is lying on the floorboards. A watcher who checks the rope will report a working bell for as long as anyone keeps pulling.

The fix is one line of thinking and a few lines of code. Recent activity is not evidence of health, because a broken system produces activity as readily as a healthy one. The only event that proves the thing works is the thing working. So measure the time since the last success, and alarm when that number crosses a threshold you chose on purpose.

# before: satisfied by activity, which the outage supplied
latest = runs[0]
if latest.created_at > last_seen_at:   pass

# after: satisfied only by work that finished
ok      = [r for r in runs if r.conclusion == 'success']
drought = (now - max(r.completed_at for r in ok)).minutes
if drought > threshold_minutes:        alarm

# and report the number, always, green or red
"last success 25764 min ago (threshold 90)"

The last line matters as much as the alarm. A guard that prints only a verdict gives you nothing to sanity-check. A guard that prints the measurement it judged lets a person glance at it and say that number looks wrong.

It was not one probe

The same defect was in a second probe, written at a different time, for a different product, by a different hand. It watched an uptime check the same way, and 99 of its last 100 runs had been cancelled while it reported green.

Both were rewritten to measure time since last success. They now report droughts of 25,764 and 25,815 minutes. Those figures are the point. Two authors, working apart, both reached for recent activity when asked to watch a periodic task. It is the easiest thing to fetch, and it tracks health right up to the moment you need it.

A third script had it right all along

There is a third actor. A nightly housekeeping script had this right the whole time. Every day it measured the staleness and wrote a line naming the job and how many hours it had been dead. That line went into a markdown report. Nobody opens that report.

Meanwhile 47 mails a day arrived in the founder's inbox from other automation, none of them about this. The one true finding in the whole system was filed where no human goes, and the channel a human does read was full. A detection written somewhere nobody reads is not a detection; it is a record of the thing you could have known.

That is the cheaper half of the fix, too. Nobody had to write a new check. The check existed, was correct, and ran daily. What it needed was a route to a person, and a budget on the channel it shares with everything else.

The audit

Six checks to run over anything in your system that claims to be watching something else. They take longer to answer honestly than to read. The plausible cause and the real one leave different records, so start with the record.

• Write the passing condition out as one sentence. If it turns on the words recent, latest, new or any, it is reading activity.
• Describe the broken state, then check it against that sentence. If the broken state satisfies the condition, the guard is decoration.
• Look for a clock in the condition. A health claim with no elapsed time in it is not a health claim.
• Make the guard print the number it judged, on green as well as red. A bare PASS gives nobody anything to sanity-check.
• Check whether the job can cancel its own queued runs, and whether the platform notifies on the state that leaves behind.
• Name where the finding lands, and when a person last opened that place. Count what else arrives there in a day.

The generalisation

The three failures here are one failure at different distances. The runner watched for a credential and got a queue. The probe watched for activity and got activity. The report watched correctly and spoke into an empty room. In each case the signal being read was a proxy that holds in the healthy case and breaks in exactly the case the reader cared about.

Bind a guard to the last success. Give it a clock. Make it say the number. Send it somewhere a person goes. Then break the thing on purpose and watch the guard turn red, because a guard you have never seen fail is a guard you have never tested.