Testing a feature that answers differently every time

Here is the failure that organized how we test. It is illustrative, a composite of a failure class with the specifics generalized. A coaching summary comes back clean. It reads well and follows the game in order, ending on the supposed mistake that decided it: a piece thrown away under pressure. That blunder never happened. The piece was traded evenly, the game turned much later, and nothing in the paragraph would tell you which part to doubt.

No test caught it, because we had not written a test that could. The assertion we knew how to write, expected output equals actual output, does not work against generated prose. Run the same game again and the paragraph is rephrased, the emphasis moves, and the claim survives in different words. Our suite had rested on equality the way most product suites do, without anyone having said so.

Old techniques, new surface

The loud version of this argument is that generated output breaks the assumption underneath every test suite ever written. It is false, and anyone who has shipped a parser has the counterexample to hand. Property-based testing has asserted invariants over generated inputs for decades. Metamorphic testing checks relations between outputs where no single one has an oracle. Fuzzing assumes an unpredictable input and asserts the system holds its properties anyway. Statistical acceptance sampling has released physical goods on measured defect rates for a century. None of those needs an equality oracle.

What is new is that the generated prose is now part of the product. Those techniques grew up around internals, a parser or a serializer, where the properties are formal and the reader of a failure is another engineer. Our output is a paragraph a customer reads, so the properties that matter are whether a claim is true of the game it describes and whether the tone is one we will put our name on. Applying invariant and distributional testing to the product surface changes who has to agree on the properties and what a violation costs when it escapes. We can reuse the methods, but we still have to decide which properties matter here, and then write a test that asks whether the output holds them.

• Is every claim grounded, traceable to something that appears in the game record?
• Is it in our voice, at the register a player expects from us?
• Does it decline to assert what we cannot check?
• Does it stay clear of failure classes we have already fixed, like invented moves, or confident narration of thin data?

The other half of the change is the unit under test. One passing run tells you little, because a single run was never the thing being measured: the assertion is over a distribution, so we track failure rates across a representative case set, which tells us which kind of case is failing and whether that rate moved since the last change.

Fluency is not evidence

Obvious wrongness is the cheap case: malformed, empty, truncated, off topic, all caught by a schema check. The expensive failure reads perfectly, with the rhythm of expertise and one specific invented detail. In our own review queue, people catch awkward prose far more readily than a fluent factual error. Someone working down a list is reading for fluency, and this failure passes that read. Checking a claim against the game record is slower than reading the sentence, and nobody with a queue does it on every item.

The check that works refuses to read the prose. Pull the claims out of the paragraph and compare each against the structured record it was written from. Did that move occur, at that point in the game. Was that piece lost. Did the evaluation swing where the sentence says it swung. This tier is deterministic: same input, same verdict, every run. It is cheap enough to run on every case on every commit, and it is decisive, because there is a move list and a claim either agrees with it or contradicts it. A grounding violation blocks the release.

Voice and usefulness do not decompose that way. No table exists to check a sentence against when the question is whether it sounds like us, or whether the summary buried the thing the player needed. We grade those with a model against a written rubric, and the operational properties flip: the grader is itself nondeterministic, it shifts when the grading model is updated, and one score means little while a moving average across the set means a lot. Those scores route work rather than gate it. A rate that slides pulls a batch in front of a person, and the person decides.

// tier 1: deterministic, decisive, blocking
for (const claim of extractClaims(summary)) {
  expect(game.plies).toContain(claim.move);
  expect(game.pieceAt(claim.ply)).toEqual(claim.piece);
}
expect(regressions.check(caseId, summary)).toBe("clean");

// tier 2: model-graded, tracked as a rate, never a gate
const voice = await gradeVoice(summary, rubric);
record(caseId, voice.score);
expect(suiteRate("voice", window)).toBeAtLeast(baseline);

The set is never finished

Tests are normally a phase: written while you build, green when you ship, insurance against failure modes you already enumerated. These failure modes were never enumerable in advance, and the new ones arrive from users. Someone tells us a report about their game said something odd, we find that game, capture the source data alongside the output, and turn the complaint into a permanent case. Over time the set becomes the record of every way this feature has been wrong. We treat it as an operational surface with an owner, the way uptime has one, and every complaint that holds up adds a case to it.

The failure mode that worries us has little to do with any single bad paragraph. Deterministic software breaks loudly: a stack trace, an alert, a number that goes to zero. A generated feature degrades quietly. It hedges a little more, grounds a little less, starts asserting a category of thing it used to be careful about, and throws no errors the whole time.

What compounds

Any team with the same model access can build something that writes about a player's games; the model is a commodity input and the wrapper around the call is small. That wrapper is easy to reproduce. Harder to reproduce is the collection of failures we have already found. Openings whose common name and formal classification disagree. Blitz games where the real story is the clock, and archives where an account changed hands, so the narrative has to notice it is describing two players. Each case records a user-visible failure alongside the source data that should have prevented it, and that pairing is what lets the case survive a model swap.

In conventional software the code is the asset and tests are overhead you accept to change it safely. For a generated feature that ordering inverts. The code around the model call sits close to commodity, while the case set gains value every month it exists. A competitor can read the product and buy the same model, and none of that hands them the failures our users have already found for us. New cases arrive when people use the shipped product and tell us something looked wrong. So we staff the set like product work, across the studio and not only here.

You test a feature that answers differently every time by pinning what is not allowed to move: the claims have to be true of the game, the voice has to be ours, and a failure we have fixed has to stay fixed. Wording can vary, which is the point of generating it. So we write the factual and policy rules down, check them against the game record wherever a machine can, and keep a case for each bug we have already fixed, one that goes red the moment it comes back.