A critique loop that actually converges

Eloist's Creative Studio generates ad images and checks them before export. By August the checks were thorough. Every export is measured with an image library and has to be exact to the pixel for its placement. Content has to sit clear of all four edges. The chess board in the picture is read square by square against its FEN, a check that exists because a defective position went live on 2026-08-04 and paused all spend. The copy is checked against the product's facts. Between them these certify an ad that is correct, honest, exactly 1080 by 1350, and ugly.

Nothing in that list looks at the composition. A ruler can tell you the headline is 40 pixels from the edge; it cannot tell you the headline is fighting the picture for attention, or that the whole thing has drifted back into a look the brand retired. So the pipeline was: one generation, mechanical checks, then a human decides by eye or does not. The quality of a batch was the quality of whatever first draft the model happened to produce. That is the general shape of the problem, and it is not specific to images. Any generator whose output is checked for correctness but never judged for quality has exactly this property.

Correct and good are different questions, answered by different instruments. Passing one says nothing about the other.

One reviewer returns mush

The obvious move is to add a judge: show the image to a vision model and ask whether it is good. The Studio's verify step had already tried a version of that, and it is where we went wrong first: the vision check waved through three cropped ads in one day, and later invented board defects that were not there, which is why the verify module now asks a model nothing and measures everything. A single "is this good?" pass says yes to almost anything, and when it objects, the objection is unactionable ("make it feel more premium"). A loop with a generous reviewer in it looks like it has a judge and behaves as if it has none.

What replaced it is a fixed panel of five lenses, each narrow enough to be answerable, each run as a separate call in parallel on every round, and each carrying its own veto. The lenses are not interchangeable and were not chosen from a textbook. Each maps to a way a real batch failed:

• identity: does this look like our work? A batch on 2026-08-15 drifted back into a retired look (rounded sans, a pill button) and nobody noticed until export. The retired look is named in the brief as blocking.
• legibility: judged as a 300-pixel thumbnail in a feed, because that is where it will be seen. An early batch at low quality rendered the word "Desinboard" and it reached export.
• hierarchy: name what the eye hits first, second, third. Three things competing to be first is no first.
• restraint: the house failure mode is too much text. If a line can be cut without losing the offer, it is a defect.
• truth: does the picture imply a claim the copy never makes and the product cannot keep? A fake interface, an invented number, a promised result.

Each critic returns structured defects, not prose. A defect has a lens, a severity, one sentence about what is wrong with this image, and one imperative fix. A defect with no actionable fix is not a defect. The severity is the critic's own call and the loop trusts it, which is why each lens's brief spells out what counts as blocking for that lens. The lens list and the briefs live in the same file, so a new lens cannot be added without saying what it judges. Adding one is a deliberate act: every lens can veto, and a vague lens makes the loop never finish.

Veto versus score

A critic that only scores produces a number you then have to threshold, and every threshold is a guess. A critic that can veto is simpler: blocking defects stop a ship however pretty the rest is, and a lens with nothing worse than minor outstanding has passed. Both ideas are needed. The veto is how a round gets rejected; the score is how two rounds get compared. The score is where we made the second mistake.

The first instinct is to count defects. Counting is wrong in a way that only shows up in practice: a revision that trades one blocking defect for three minor ones is a clear improvement, and a loop that counts will call it a regression, stop, and keep the worse image. So rounds are scored by severity and never counted. The weights in the module are blocking 9, major 3, minor 1. One blocking defect outweighs eight minors, and a round is better than the last only if its weighted sum is lower.

LENSES   = [identity, legibility, hierarchy, restraint, truth]
WEIGHT   = { blocking: 9, major: 3, minor: 1 }

score(defects)      = sum(WEIGHT[d.severity] for d in defects)
passed(defects, L)  = no defect on lens L worse than minor
blocking(defects)   = defects where severity == blocking

decide(history, maxRounds):
  scores = history.map(r => score(r.defects))
  best   = argmin(scores)                  # never history[last]
  latest = history[-1]
  if no blocking(latest) and every lens clean:  stop 'clean'
  if len(history) >= maxRounds:                 stop 'budget'
  if len(history) >= 2 and scores[-1] >= scores[-2]:
                                                stop 'no-improvement'
  else continue

revisionFrom(defects, cap = 4):
  worst  = sort by severity, blocking first
  chosen = (non-minor if any, else all)[:cap]
  'REVISION. Keep everything already working; change only these:'
  + numbered (lens) fix lines
  + 'Do not restyle the whole image. Do not add elements nobody asked for.'

That whole thing is a pure TypeScript module with sixteen unit tests beside it, and the model calls (generate, five critics in parallel, revise) live in a separate script that does nothing but I/O. The split is the point. What counts as a defect, how a round is scored, when to stop, and what the next generation is told are all arguable in a test rather than only in a production run that cost money. The test that matters most is four lines long: a history of scores 9, 1, 6 must stop on no-improvement and hand back index 1.

Keep the best round, not the last

An image model asked to fix five things fixes two and breaks two, and it does this for ever. So a fixed-round loop that returns its final image is, on most runs, returning something worse than an earlier round. Three stopping rules replace the fixed count. Stop clean when there are no blocking defects and every lens is clean. Stop on no-improvement when the latest round's score is not lower than the previous round's (this needs at least two rounds). Stop on budget when the rounds are exhausted. And whichever rule fires, return the round with the lowest score, never the newest one.

"Three rounds" is a budget, not a standard. Never stop on a count alone and never treat the count as the definition of done. The no-improvement rule also makes the loop cheap: a typical run is two or three rounds rather than the budget.

The same module was ported to a second product, jungl, and its first batch of five concepts is the clearest demonstration we have. Each concept ran up to three rounds (the draft, then two revisions). Four of the five had the first revision as their best and the second revision worse. Scores, draft to final:

concept               draft -> rev 1 -> rev 2   kept
a-year-of-light          12 ->   1   ->   2     rev 1
balcony                   9 ->   1   ->   7     rev 1
empty-corner              3 ->   1   ->   5     rev 1
same-plant-two-fates      3 ->   2   ->   3     rev 1
its-december              3 ->   6               draft   (stopped: no-improvement)

Read the balcony row. The draft scored 9, the first revision 1, the second 7. A loop that returned its last round would have shipped the 7. The its-december row is the other case: the first revision made the draft worse, so the loop stopped after two rounds and kept the draft. Returning the argmin rather than the last element would have been the right call on every row of that table, and it was the rule that proved itself on first contact. The same-plant row also shows the severity rule at work: the first revision traded one major for two minors, score 3 to 2, an improvement even though the defect count rose from one to two. A counting loop would have stopped on the draft.

The revision is ordered and capped

The other half of convergence is what the generator is told between rounds. A model handed eleven simultaneous corrections applies the first two and invents new problems for the rest, so the revision sorts defects by severity, caps at four, and drops minors entirely while anything worse is outstanding. Minors are what you polish once the picture is right. And the instruction is framed as a revision rather than a re-roll: keep everything that is already working, change only these, do not restyle the whole image, do not add elements nobody asked for. Without that framing the second generation is a fresh roll of the dice, and the no-improvement rule fires on noise rather than on a real plateau.

What it costs, and what we still get wrong

A round is one image plus five critic calls. On jungl's published batch of nine ads (18 images, two rounds each, generated 2026-08-22) the ledger came to $2.02 on images and $1.93 on critiques, against a published per-image price of $0.042 for a medium-quality 1024 by 1024 generation. The critics cost about as much as the pictures. We consider that cheap, because the alternative was a human looking at every round, and mostly not doing so. Every round's image, critique and revision is written to disk so the progression can be read afterwards rather than taken on trust, and the ad manifest carries each round's score so the "we kept the best" claim is checkable.

Two things are unsettled. First, what to do when a critic fails to answer. The eloist script records a failed lens as a major defect ("this lens did not answer"), so a round with a silent critic can never be clean. The jungl port scores a failed lens as clean, which makes that round's score optimistic, and writes the fact down rather than smoothing it over. Those are opposite choices and we have not measured which is less wrong.

Second, lenses judge what is present. The first nine jungl ads all passed and not one of them carried the domain, because every lens was reading the words that were there and none was counting the words that should have been. The legibility brief now names absence explicitly, and we assume there are other holes of that shape we have not found. A related limit: the critic is a different model family from the generator, which is better than a model grading its own output and is still not a person looking. Vision reviewers have a measured yes-bias. A score is a filter for your attention, not a verdict, and nothing in this loop has put an ad in front of a stranger.