Alert fatigue is arithmetic

In the thirty days to 2026-09-11, one monitoring channel sent 1,399 emails to one reader. He opened 103 of them, 7.4 percent. Then a real failure happened. The system detected it correctly and mailed him inside twenty minutes. He found the failure himself two hours later and asked why nothing had told him. Something had, and it sat unread between an all-clear notice and a copy of a customer letter.

The reader is not careless. He was getting about 47 of these a day. At that rate, opening everything costs an hour a day and returns almost nothing, because almost none of them ask for an action. Opening nothing costs nothing and misses one message a month. Alert fatigue is the name we give that sum after a person has already done it.

What the channel was actually sending

The first reading of the census was wrong. 1,399 looked like 1,399 alarms, and they were not. 1,075 of them, 77 percent, were copies of letters the product had already sent to its members. That is a mirror of an outbox, not a warning. True operator alarms came to 324 over thirty days, about 11 a day. Eleven is still several times too many, and it is a different problem from 47.

Open rates rank almost perfectly by how rare a sender is. The alarm that fired five times in thirty days was opened 40 percent of the time, the best rate in the set. The firehose that fired 1,075 times was opened 7.9 percent. A test email sent once was opened every time. Attention is not spread evenly over a channel. It goes to the senders that have earned it by staying quiet.

Four senders, none of them about an incident

• An all-clear notice: 79 sends in thirty days, not one of them opened.
• A pipeline warning: 53 sends about a state that was true on 23 of the 30 days.
• A truth audit: 60 sends.
• A deploy warning: 53 sends, with no deduplication of any kind.

Every one of those fires on a condition being true rather than on a condition changing. A condition that stays true for three weeks produces three weeks of mail and no new information after the first message. That is the defect, and it is unrelated to how important the condition is. Important conditions make the worst repeat senders, because nobody dares turn them off.

One condition, two states

The all-clears are the best small story in the pile. At some point the subject line changed from a dash to a colon. The code that decided whether it had already sent this notice keyed on the subject text, so the two spellings became two independent alert states, each with its own all-clear. One punctuation mark doubled the traffic.

Underneath sat a smaller bug with a very common shape. The code cleared a resolved alert with a DELETE that carried a JSON content type and no body. The API answered 400. A fetch call does not treat a 400 as an error, and the call site swallowed whatever came back, so the clear failed silently on every tick. The stored alert never went away, and every quiet minute resolved it again.

await fetch(url, {
  method: "DELETE",
  headers: { "content-type": "application/json" },   // there is no body
});

// -> 400 Bad Request. fetch resolves. nothing throws. nothing is logged.
// -> the alert row survives, so the next quiet tick resolves it again,
//    and again, and again: 79 all-clears, zero opened.

The monitors saw everything

The counterpoint should worry anyone who runs monitors. Through that same window, the studio's off-box monitoring had been dead for eighteen days. A daily check caught it every single day and wrote the finding into a report file that nobody opens. At the same time, 47 emails a day were arriving about things nobody acts on. Detection worked perfectly, and the routing around it was the entire failure.

Four buckets

So the design question for a new alarm is not whether the condition is worth detecting. It is what should happen when the condition comes true. Four answers cover nearly everything, and only one of them is an email.

• Page: a person has to act within the hour.
• Today: it goes in one daily digest and never travels alone.
• Write it down: a dashboard row or a log line, never an email.
• Delete: it fires on a condition nobody has ever acted on, so remove the sender.

Every alarm now leaves through one function. Its class comes from a registry, and the seam enforces the class rather than asking senders to behave: the two classes that are not urgent cannot send mail at all, whatever the caller passes in. Trusting callers to behave is how the channel filled up the first time.

A page mails once when the condition opens, then on a widening ladder while it stays unresolved, at most five notices ever, then silence. From the second notice the subject says how long the thing has been broken, because the subject is the only part anyone reads at a glance. One resolution mail on the closing edge, sent only if we actually paged.

class     what it means                           may send mail
page      act within the hour                     yes, on the ladder below
digest    act today                               no, one daily digest
record    worth knowing, nobody acts on it        no
delete    fires on a condition nobody acts on     no, remove the sender

page ladder, hours since the condition opened:
  0h   1h   4h   12h   24h    then never again
  from notice 2: "PAGE (2/5, unresolved 1h) <title>"

the daily digest sends nothing on a day with nothing in it.

If one piece of this survives, it should be the guard. It reads every operator email actually sent over a window, groups each one under the registry entry whose pattern claims it, and compares the rate against a ceiling in sends per day. Run against the real thirty days it fails, loudly, because most of the senders have not been moved yet.

channel total: 49.57 sends/day against a ceiling of 2.00  (24.8x over)
21 alarms over budget
UNREGISTERED: <a subject no entry claims>

exit 0   everything inside budget, nothing unclassified
exit 1   a breach, or an alarm nobody has classified
exit 2   could not measure

// exit 2 is deliberately not 0: a check that reports green when it means
// "no credentials" is how a dead monitor stays dead for eighteen days.

Two details there do more work than the ceiling itself. An alarm that no registry entry claims is a failure rather than a warning, so whoever adds a sender has to answer which of the four classes it is before it can ship. And a run that cannot measure exits 2, never 0, so silence from the guard can never be mistaken for good news.

What we got wrong

The first instinct was a cleanup. This channel had been cleaned up before and it refilled within weeks, because nothing stood between a new sender and the inbox. A cleanup fixes the instances in front of you. Only the seam and the guard touch the class, and they cost less than the second cleanup would have.

The ceiling of two a day is a judgment and not a measurement. Two a day is about fourteen a week, which one person can genuinely open and read. The number is picked so that an arriving message means something. Anything that cannot buy a slot at that price was never an email in the first place, and saying so out loud is most of the work.

What has been proven so far is the census and the guard, both run against real history. The channel is still far over its budget and stays there until every sender in the registry has been moved or deleted. Right now the guard measures the problem rather than certifying the fix, which is the honest state for a check written in the same week as the incident.