A critique loop that actually converges

Eloist's Creative Studio generates ad images and checks them before export. By August the checks were thorough. Every export is measured with an image library and has to be exact to the pixel for its placement. Content has to sit clear of all four edges. The copy is checked against the product's facts.

The chess board in the picture is read square by square against its FEN. We added that one because a defective position went live on 2026-08-04 and paused all spend. Between them these rulers certify an ad that is correct, honest and exactly 1080 by 1350. Not one of them can tell us whether it is any good to look at.

Nothing in that list looks at the composition. A ruler can tell you the headline is 40 pixels from the edge. It cannot tell you the headline is fighting the picture for attention, or that the whole thing has drifted back into a look we retired last year.

So the pipeline was one generation, mechanical checks, then a human deciding by eye or not bothering. The quality of a batch was the quality of whatever first draft the model happened to produce. None of this is specific to images. Any generator whose output is checked for correctness but never judged for quality has the same hole in it.

Correct and good are different questions, answered by different instruments. Passing one says nothing about the other.

One reviewer returns mush

The obvious move is to add a judge. Show the image to a vision model and ask whether it is good. We had already tried a version of that in the verify step, and that is where we went wrong first. The vision check waved through three cropped ads in one day, then invented board defects that were not there. The verify module now asks a model nothing and measures everything.

A single "is this good?" pass says yes to almost anything, and when it does object the objection is unactionable. "Make it feel more premium" is not a fix. A loop with a generous reviewer in it looks like it has a judge and behaves as if it has none.

What replaced it is a fixed panel of five lenses, each narrow enough to be answerable, each run as a separate call in parallel on every round, and each carrying its own veto. The lenses are not interchangeable and we did not take them from a textbook. Each one maps to a way a real batch failed:

• identity: whether this looks like our own work. A batch on 2026-08-15 drifted back into a retired look (rounded sans, a pill button) and nobody noticed until export. The retired look is named in the brief as blocking.
• legibility: judged as a 300-pixel thumbnail in a feed, because that is where it will be seen. An early batch at low quality rendered the word "Desinboard" and it reached export.
• hierarchy: name what the eye hits first, second, third. Three things competing to be first is no first.
• restraint: our own failure mode is too much text. If a line can be cut without losing the offer, it is a defect.
• truth: whether the picture implies a claim the copy never makes and the product cannot keep. A fake interface, an invented number, a promised result.

Each critic returns structured defects, not prose. A defect has a lens, a severity, one sentence about what is wrong with this image, and one imperative fix. A defect with no actionable fix is not a defect.

Severity is the critic's own call and the loop trusts it, so each lens's brief spells out what counts as blocking for that lens. The lens list and the briefs sit in the same place, which means a new lens cannot be added without saying what it judges. Adding one is deliberate. Every lens can veto, and a vague lens makes the loop never finish.

Veto versus score

A critic that only scores produces a number you then have to threshold, and every threshold is a guess. A critic that can veto is simpler. Blocking defects stop a ship however pretty the rest is, and a lens with nothing worse than a minor outstanding has passed.

Both ideas are needed. The veto is how a round gets rejected, the score how two rounds get compared. The score is where we made our second mistake.

The first instinct is to count defects, and counting only reveals itself as wrong in practice. A revision that trades one blocking defect for three minor ones is a clear improvement. A loop that counts calls it a regression, stops, and keeps the worse image.

So rounds are scored by severity and never counted. The weights are blocking 9, major 3, minor 1. One blocking defect outweighs eight minors, and a round beats the last only if its weighted sum is lower.

LENSES   = [identity, legibility, hierarchy, restraint, truth]
WEIGHT   = { blocking: 9, major: 3, minor: 1 }

score(defects)      = sum(WEIGHT[d.severity] for d in defects)
passed(defects, L)  = no defect on lens L worse than minor
blocking(defects)   = defects where severity == blocking

decide(history, maxRounds):
  scores = history.map(r => score(r.defects))
  best   = argmin(scores)                  # never history[last]
  latest = history[-1]
  if no blocking(latest) and every lens clean:  stop 'clean'
  if len(history) >= maxRounds:                 stop 'budget'
  if len(history) >= 2 and scores[-1] >= scores[-2]:
                                                stop 'no-improvement'
  else continue

revisionFrom(defects, cap = 4):
  worst  = sort by severity, blocking first
  chosen = (non-minor if any, else all)[:cap]
  'REVISION. Keep everything already working; change only these:'
  + numbered (lens) fix lines
  + 'Do not restyle the whole image. Do not add elements nobody asked for.'

All of that judgement is a pure TypeScript module with sixteen unit tests beside it. The model calls, meaning generate, five critics in parallel, revise, sit in a separate script that does nothing but input and output.

The split is the point. What counts as a defect, how a round is scored, when to stop, and what the next generation is told can all be argued in a test instead of only in a production run that cost money. The test that matters most is four lines long. A history of scores 9, 1, 6 has to stop on no-improvement and hand back the middle round.

Keep the best round, not the last

An image model asked to fix five things fixes two and breaks two, and it does this for ever. A fixed-round loop that returns its final image is, on most runs, handing back something worse than an earlier round.

Three stopping rules replace the fixed count. The loop stops clean when there are no blocking defects and every lens is clean; it stops on no-improvement when the latest round's score is not lower than the previous round's, which needs at least two rounds; it stops on budget when the rounds run out. Whichever rule fires, it returns the round with the lowest score, never the newest one.

"Three rounds" is a budget, not a standard. Never stop on a count alone and never treat the count as the definition of done. The no-improvement rule also makes the loop cheap, because a typical run is two or three rounds rather than the budget.

We ported the same module to jungl, and its first batch of five concepts is the clearest demonstration we have. Each concept ran up to three rounds, the draft and then two revisions. Four of the five had the first revision as their best and the second revision worse. Scores, draft to final:

concept               draft -> rev 1 -> rev 2   kept
a-year-of-light          12 ->   1   ->   2     rev 1
balcony                   9 ->   1   ->   7     rev 1
empty-corner              3 ->   1   ->   5     rev 1
same-plant-two-fates      3 ->   2   ->   3     rev 1
its-december              3 ->   6               draft   (stopped: no-improvement)

On the balcony concept the draft scored 9, the first revision 1, the second 7. A loop that returned its last round would have shipped the 7. The its-december concept failed the other way round. Its first revision made the draft worse, so the loop stopped after two rounds and kept the draft.

Keeping the lowest-scoring round rather than the last would have been the right call on every concept in that batch, and the rule proved itself on first contact. The same-plant concept shows the severity weighting working too. Its first revision traded one major defect for two minors, score 3 down to 2, an improvement even though the defect count rose from one to two. A counting loop would have stopped on the draft.

The revision is ordered and capped

The other half of convergence is what the generator is told between rounds. A model handed eleven simultaneous corrections applies the first two and invents new problems for the rest. So the revision sorts defects by severity, caps at four, and drops minors entirely while anything worse is outstanding. Minors get polished once the picture is right.

The instruction is framed as a revision rather than a re-roll. Keep everything that is already working, change only these, do not restyle the whole image, do not add elements nobody asked for. Without that framing the second generation is a fresh roll of the dice, and the no-improvement rule fires on noise instead of on a real plateau.

What it costs, and what we still get wrong

A round is one image plus five critic calls. On jungl's published batch of nine ads, 18 images at two rounds each, generated 2026-08-22, the ledger came to $2.02 on images and $1.93 on critiques, against a published per-image price of $0.042 for a medium-quality 1024 by 1024 generation. The critics cost about as much as the pictures.

We consider that cheap, because the alternative was a human looking at every round and mostly not doing so. Every round's image, critique and revision is written down so the progression can be read afterwards rather than taken on trust, and the ad manifest carries each round's score so our claim to have kept the best one is checkable.

Two things are unsettled. The first is what to do when a critic fails to answer. In Eloist a failed lens is recorded as a major defect, so a round with a silent critic can never come out clean.

In jungl a failed lens scores as clean, which makes that round's score optimistic, and the port writes the fact down rather than smoothing it over. Those are opposite choices and we have not measured which one is less wrong.

The second is that lenses judge what is present. The first nine jungl ads all passed, and not one of them carried the domain. Every lens was reading the words that were there; none was counting the words that should have been. Nine ads, five critics each, every score green, and no way for a reader to find the site. That one stung.

The legibility brief now names absence explicitly, and we assume there are other holes of that shape we have not found.

The critic is also a different model family from the generator. That beats a model grading its own output and it is still not a person looking. Vision reviewers have a measured yes-bias. A score is a filter for your attention rather than a verdict, and nothing in this loop has put an ad in front of a stranger.