Three gates, three questions

Eloist makes a short narrated film about a member's recent chess games. Sign up and one gets queued for you automatically. For most of last week we were telling most of our members that a film was coming, while a different part of the same product had already decided it would never be made.

Three gates stood between a member and that film. Each was written by a careful person. Each passed review. Each asked a slightly different question.

eligibility check        does this member have any analysed game
  (in the database)      with a significant blunder, ever

the renderer             is there a qualifying game inside the
  (the worker)           last fourteen days

the "make me one"        does this member have any analysed game
  button (the api)       with a significant blunder, ever

so: the button returned success, then handed the renderer
a job the renderer refused with "no stories in window"

One at a time, all three are defensible. The eligibility check answers whether this is the kind of member the feature exists for. The renderer answers whether there is material for a film this week. The button answers whether the member can use the feature at all. Nothing in any of the three is wrong.

Together they promise on one question and deliver on another. A member whose last real blunder was in March cleared every gate that made a promise and failed the only gate that renders anything.

How many people

Read against the production database on 10 September 2026 at 21:02 UTC: of 282 external members with a linked chess account, 233 were promised-but-undeliverable. That is 82.6 percent, counted one row per member.

Two members ever hit it. A rollout guard left over from launch bounded the automatic queueing to recently created accounts, and that guard is the only reason this was a bug rather than a mailing to everyone. It also hid the bug. Two complaints look like two odd cases.

The first measurement of the blast radius said 18 percent, and it was wrong. The query was fine. It bucketed members by activity over thirty days while the delivery window is fourteen. Same table, same filter, different grain, and a figure four and a half times too small in the number that decides how urgent this is.

One predicate, one place

The fix is dull. One function decides whether a member has a story worth filming. It takes the window as an argument, and every caller reads it. The promise side and the delivery side can no longer disagree, because there is nothing left for them to disagree about.

// before: three call sites, three predicates
database  eligible_for_film(member)          -- ever
worker    games since now() - 14 days        -- window
api       eligible_for_film(member)          -- ever

// after: one predicate, one definition, read by all three
has_story(member, window)

The shared function is not the guard. The guard is a test that asks the promise side and the delivery side the same question about the same member and fails when the two answers differ. It was proved by breaking it twice, once by widening the promise and once by narrowing the window, and watching it go red both times.

An hour later, one layer down

Within the hour, the same engineer wrote a cleanup job. It deletes finished working copies: everything committed, everything merged, nothing left to lose. The protection against deleting live work was a scan of which directories running processes were sitting in.

It deleted the working copy of a running agent. The agent was alive and its work was unfinished. At the instant of the scan, its shell happened to be sitting somewhere else.

The check asked whether a process is in this directory right now. The thing it was protecting was whether anyone is still working here. Those two answers agree most of the time, which is what makes them dangerous.

The repair was to add a second signal and keep the tree if either one says busy: nothing in it has been touched for the last N minutes. A file timestamp sees the gap between two commands, which is exactly where the first signal was blind.

A check that samples one instant cannot answer a question about an interval.

That is the same defect one layer down. Three gates asked three questions about one member. One guard asked a question that was not the question it needed answered. In both cases every component returned a true answer and the system as a whole lied.

What to take from the hour

Two rules come out of this, and they are the same rule at different sizes. A check that decides whether to promise and a check that decides whether to deliver must read one predicate defined in one place. And a check about whether something is in use has to cover the interval, not the instant, because work has gaps in it.

The timing is the part worth keeping. The engineer had spent that morning inside this exact shape, named it, written it down, and reproduced it sixty minutes later on a different layer of the stack. Knowing the lesson did not prevent the lesson. That is the argument for a test rather than a resolution: a resolution has to be remembered at the moment everyone is busy, and a test remembers on its own.