Surfaces that regenerate need a memory

One of the most engaged accounts on Eloist had opened the dashboard on thirteen of the previous sixteen days and left inside thirty seconds every time. Nothing in the aggregates flagged it. Engagement looked fine, nobody had churned, no error had fired. We only saw it by pulling that one player's row-level history: the sequence of items that account had been served, in order, with dates.

They had been shown the same worst move twenty-seven times since 2026-08-07, from a game played on 2026-02-16, while making three comparable blunders in the fortnight they were looking at it. They had been shown the same treatment card thirty times, every one of them marked thin. They had been offered twenty-two drill positions in total and solved twenty-one of them. This was not a disinterested player. They had finished the material and the material would not move.

Later the same night a fourth surface went the same way: the planner for the product's daily newspaper had drafted seven days and two stories were alternating as the lead across six of them. Three of the four fixes landed in one commit on 2026-08-20 and the planner fix in another a little over an hour later.

A stateless ranking over a stable corpus is a constant

Each of the four was a ranking re-rendered every day with no memory of yesterday. The worst-move card was `order(worst_loss desc).limit(1)` over every game the account ever had. An all-time maximum only changes when the player beats their own record, so a daily surface built on one cannot change. The drill set was ordered by worst loss on purpose, because a `?g=` permalink has to mean the same thing twice, and nothing tracked which positions had been solved. The treatment card could only exit by graduating, and graduating needed a readable metric, which a player who has stopped playing that opening never produces. The planner drafted seven days in a loop and every iteration read the table of what had printed; a draft prints nothing, so all seven saw the same spent set and took the head of the same list.

Tie-breaks made it worse in a way we did not expect. Engine losses pile up on the eval ceiling: this player had three separate games at exactly 2400 centipawns, and with no second sort key the database picked whichever it liked. So even the query we thought was deterministic was not, and the one we thought varied did not.

Engagement that decays without churn is the symptom. The user is still coming back; the surface has stopped giving them a reason to.

What we got wrong

The telemetry could not have told us. Every drill view fired an event carrying `{"lossCp": 2400}`. Twenty-seven identical payloads could equally have been twenty-seven different games. A memory needs a stable reference to the item, and the events carried a metric about it instead, so the first thing the fix had to add was the write, not the read.

We also held the wrong mental model of the cost. The drill ordering was stable because stability was a feature: permalinks, determinism, the set you saw yesterday being the set you can link to. We treated stable ordering and stable content as the same property. They are not. You can keep the order byte-identical and change which rows are eligible, and that is what every fix below does.

And the treatment lifecycle had two terminal states where it needed three. A treatment was active, graduated, or dismissed. Dismissed meant the player refused it. Graduated meant the metric improved. A treatment nobody could measure was neither, so it stayed active for ever: seventy-two percent of the treatment cards rendered in the previous thirty days said thin, and the loop that was meant to turn diagnosis into improvement had stopped on the first leak anyone walked away from.

The ledger

The memory is one table, and it already existed. The trainer recorded reviews as (user, card type, card ref, graded at). The drill now writes one row there per position drilled, under its own card type, with a reference built from the game URL and the ply rather than from anything about the move.

// a position's stable identity: the game and the ply inside it.
// NOT the centipawn loss, which is what the events carried.
export const leakRef = (gameUrl: string, ply: number) => `${gameUrl}#${ply}`;

// POST /api/play/leaks  { gameUrl, ply, found }
await supabase.from('srs_reviews').insert({
  user_id:     user.id,          // the account, not the handle: a handle can be relinked
  card_type:   'leak',           // the surface
  card_ref:    leakRef(gameUrl, ply),   // the item
  grade:       found ? 5 : 2,    // the trainer's own scale
  reviewed_at: new Date().toISOString(),
});

A failed write is not a failed drill. The player answered their position; losing the bookkeeping means they may see it again, which is the old behaviour and not worth an error on their screen. The read side loads the refs reviewed inside the cooldown and filters the candidate list against them.

export const LEAK_COOLDOWN_DAYS = 14;

export function recentlySeen(rows, nowMs, cooldownDays = LEAK_COOLDOWN_DAYS) {
  const cutoff = nowMs - cooldownDays * 86_400_000;
  const out = new Set<string>();
  for (const r of rows) {
    if (!r.card_ref) continue;
    const at = r.reviewed_at ? Date.parse(r.reviewed_at) : NaN;
    // a review with no timestamp counts as OLD; treating it as fresh
    // would hide a position for ever
    if (Number.isFinite(at) && at >= cutoff) out.add(r.card_ref);
  }
  return out;
}

export function eligibleLeaks(rows, refOf, seen, { pinned = () => false } = {}) {
  const fresh = []; let skipped = 0;
  for (const row of rows) {
    const ref = refOf(row);
    if (pinned(row) || !ref || !seen.has(ref)) fresh.push(row);
    else skipped += 1;
  }
  // never serve nothing: an exhausted player gets the set back, and is told
  if (fresh.length === 0) return { rows: [...rows], exhausted: true, skipped };
  return { rows: fresh, exhausted: false, skipped };
}

Four decisions are folded into those thirty lines, and each one came from a mistake we either made or nearly made.

• The explicitly requested item bypasses the memory. A `?g=` permalink that quietly serves a different board is a worse bug than a repeat.
• Exhaustion returns the full set and says so. The route reports `drilled: { skipped, exhausted, cooldownDays }` so the page can say you have worked through these, instead of rendering empty. An empty page is the classic over-correction here.
• A memory with no timestamp counts as old. The opposite default hides an item permanently.
• The cooldown is a product decision, so it lives in a named constant with a comment saying why fourteen: long enough that a daily player works through their real list, short enough that a genuinely recurring mistake comes back while it still matters.

Recency first, with the window in the answer

The worst-move card did not need a ledger; it needed a window. It now runs the same query twice, once over the last thirty days and once over everything, takes the recent row when there is one, and the response says which it was: `window: "recent" | "all-time"`. That last field is what keeps the copy honest. Your worst move of the last thirty days and your worst move ever are different sentences, and a returning player after months away still gets a real one. The second sort key is `end_time desc`, so ties on the eval ceiling break toward the game the player can still remember.

A run must read its own claims

The planner is the subtle one, because it did have a ledger. It read what had printed. The defect was that a loop drafting N items, each reading only committed state, makes N identical decisions. The fix is a reservation set the run carries with it and folds into the ledger each draft reads.

interface Reservations {
  claimedOn: Map<string, number>;   // storyId -> day index in this run
  onceEver:  Set<string>;           // stories that may print only once, ever
}

function applyReservations(spentIds, lastUsed, r, dayIndex) {
  const spent = new Set(spentIds), used = new Map(lastUsed);
  for (const [storyId, claimedIndex] of r.claimedOn) {
    if (claimedIndex === dayIndex) continue;       // this day's own claim, if re-drafted
    if (r.onceEver.has(storyId)) { spent.add(storyId); continue; }   // hard ban
    // soft: a future claim is 'more recently used' than anything printed,
    // so LRU sorts it last without removing it
    used.set(storyId, Number.MAX_SAFE_INTEGER - Math.max(0, 1000 - claimedIndex));
  }
  return { spentIds: spent, lastUsed: used };
}

Defer, do not ban, on a thin pool. A hard ban across six evergreen stories leaves holes in the page, and an issue with a hole is worse than an issue with a repeat. The function is pure so the rule can be argued with in a test: claimed earliest sorts first because it has rested longest; a day re-drafted keeps its own claim; the inputs are never mutated.

State that can end

The treatment got a third terminal state. Still thin after `STALL_DAYS = 14` and the episode is retired as `stalled`: not dismissed, because the player refused nothing, and not graduated, because nothing improved. A stalled leak steps aside for thirty days rather than for ever, so if the player picks that opening back up it can be diagnosed again. The migration added the status and deliberately backfilled nothing: retiring long-running actives from a migration would be a silent mass edit of live state, and the code that retires them knows something the migration does not, which is whether the metric is unreadable right now. They stall themselves on the next assessment, the same path every future one takes.

The test renders twice

The unit tests that shipped with the fix pin each rule separately: the cooldown boundary, the unstamped review, the pinned row, the empty set. The one that carries the whole idea is the shape below, which renders the surface, records what it showed, renders again, and asserts the second render moved when the corpus allowed it to.

const rows = [row('g1', 10), row('g2', 20), row('g3', 30), row('g4', 40)];

// day one: nothing drilled, the worst position leads
const first = eligibleLeaks(rows, refOf, new Set());
expect(first.rows[0]).toEqual(row('g1', 10));

// the player drills it; the ledger now holds its ref
const seen = recentlySeen([{ card_ref: leakRef('g1', 10), reviewed_at: today }], NOW);

// day two: the second render differs, the order beneath it does not
const second = eligibleLeaks(rows, refOf, seen);
expect(second.rows.map(refOf)).toEqual([leakRef('g2', 20), leakRef('g3', 30), leakRef('g4', 40)]);
expect(second.skipped).toBe(1);
expect(second.exhausted).toBe(false);

// and when the corpus does NOT allow it, the surface says so instead of going blank
const all = eligibleLeaks(rows, refOf, new Set(rows.map(refOf)));
expect(all.rows).toHaveLength(4);
expect(all.exhausted).toBe(true);

The third block is the one to keep. Without it the suite passes on a surface that correctly varies for a week and then serves an empty page to exactly the users who engaged most.

Where the claim stops: selection versus supply

No picker can vary a pool of one. When we looked at the planner's story bank, the corpus desk held five stories of which one was verified, and the evergreen desk held six, three of them openings. The reservation fix was correct and could not by itself produce a varied week, because the community slot had exactly one eligible story however cleverly it was picked. That repeat is still there tomorrow, and it belongs to the desks that write stories, not to the function that ranks them.

So before shipping a cleverer picker, count the eligible pool for each slot under the real eligibility predicate. If a slot has two candidates or fewer, the work is supply-side and the picker is a distraction, and the honest thing is to say so rather than ship the picker and call it fixed.

The audit

These are the questions we now run against anything that will be re-rendered tomorrow: a daily card, a drill set, a digest, a planner, a recommendation, an ad rotation, a what's-new panel.

• Is the ranking key a constant? Maximum, best-ever, worst-ever, first, oldest. If so, make it recency-first with an all-time fallback, and put the window in the response.
• Where is the ledger, and does its key identify the item? Name the table and column that records this user saw this thing. If the answer is telemetry, check the payload carries a stable ref and not a metric. If the ref does not exist, the write is part of the fix.
• Does the run read its own claims? A loop that drafts N items from committed state makes N identical decisions. Reserve as you go: hard for once-ever items, soft for the rest.
• Can this state ever end? Anything with a lifecycle needs a terminal state for unmeasurable, distinct from refused and from succeeded, and it should expire rather than ban.
• Is it selection or supply? Count the eligible pool per slot before touching the picker.
• Does a test render it twice? And does it assert the exhausted case returns the set with a flag, rather than nothing?

All four were found by reading one real user's history, in order, with dates. The dashboards we had were built to show whether people were coming back, and they were. None of them recorded what a given person was served on a given day, and until the write in the first code block existed, nothing else did either.