Notebook

We build products with agents.
These are the notes.

A studio running a portfolio of bets, written up from the engineering side by Shaun Robinson: what changed in the economics, what breaks at scale, and what we still get wrong.

0018 min

A flat bill is the failure mode

Launching the video coach put us $266.94 a month underwater, and we did not find out for two weeks. Idle capacity was the cost, not the customers.

Read →
  1. 0026 min

    The guard had no clock

    Scheduled monitoring stopped for eighteen days and two separate probes reported green the whole time. Both asked whether something had happened recently. The broken state produced activity every fifteen minutes, so the answer was always yes.

    monitoringguardsverificationcraft
  2. 0037 min

    Alert fatigue is arithmetic

    A monitoring channel sent its only reader 1,399 emails in thirty days. He opened 7.4 percent of them, and when a real failure came the alarm that fired correctly went unread with the rest.

    monitoringinstrumentationoperationscraft
  3. 0046 min

    Three gates, three questions

    We promised 233 of Eloist's 282 members a film we had already decided we could not make. Every check involved was correct on its own. The defect lived only in the space between them, and it came back an hour after the fix.

    guardscorrectnessagentscraft
  4. FOUR DAYS, BY NIGHTFALLWHAT EXISTED AT THE END OF EACH DAY. THE DIAMONDS ARE CHECKS, SHIPPED WITH OR BEFORE THE FEATURES THEY GUARD.DAY 1ONE REAL GAMEA SILENT CLIPBOARDS DRAWN FROM THE RECORDDAY 2THE SCRIPT, THE VOICETHE ACCURACY GATEDAY 3THE QUEUE, THE WORKERTHE SYNC AUDITDAY 4THIRTY-FIVE LETTERS OUTTHE FIRST REPLY,UNPROMPTEDNO STEP WAITED ON A ROADMAP. EACH DAY SHIPPED SOMETHING A PLAYER COULD HAVE BEEN SENT THAT EVENING.
    0055 min

    The coach shipped in four days

    Eloist had a retention problem: people loved the analysis and did not come back. Four days later a made-up coach was mailing them videos about their own games. The speed came from a shape, and the shape is transferable.

    velocityqueueseconomicsvideo
  5. ONE EPISODE, STATION BY STATIONINK IS RENDERED FROM THE GAME RECORD. BRASS IS GENERATED, AND THERE ARE ONLY TWO.A FAILED CHECK ENDS THE JOB. NOTHING SHIPS.PICKFETCHSCRIPTGATEFRAMESCLOCKSVOICESYNCAUDITRENDERUPLOADCHECKLETTERREFUTE EVERY CLAIM,ONE REPAIR ROUNDCAPTIONS TIMED AGAINSTTHE SPOKEN WORDSEVERY VIDEO CONFIRMED IN STORAGEBEFORE ANY LETTER GOES OUTTHE BOARD PICTURES ARE DRAWN BY ELOIST'S OWN CODE FROM THE GAME RECORD. NO VIDEO MODEL EVER DRAWS A POSITION.
    0065 min

    Rendered, never generated

    Eloist now mails people a short video about their own chess games, narrated by a coach who does not exist. Four days from first commit to thirty-five inboxes. Here is how it works, and what it refuses to do.

    agentsvideoverificationpipelines
  6. 202420252026JUNJULAUGMODELSCOMMITS / WEEKPEOPLECOMMITS / WEEK410484334562OPUS 4.5OPUS 4.6CLAUDE, UNNAMEDSONNET 5OPUS 4.8OPUS 5FABLE 53,245 COMMITS · 16 REPOS · BELOW THE LINE, A PERSON WROTE THE COMMIT ALONE
    0077 min

    One person, eight hands

    Two years of git across 16 repos, 3,200 commits, one engineer. For 22 months the line is flat. Then, in six weeks, models co-author 2,344 commits and the question becomes what that number is actually counting.

    agentsgitvelocitymeasurement
  7. MECHANICAL · A CHECK THAT FIRESSTILL A DOCUMENTSTILL A PERSONJUDGMENTautonomy laddernumber gatedesign barrefusal is workmoney, DNS, voice, tasteFEEDBACKnumber-check + mutation proofgolden journeysdecision ledgerbatch into programmesdream passwhat the feedback meansINTERFACE/ship · /advance · standuptickets → PR under gatesrole botsauto-merge earnedthe merge click, at L0DEVOPSmachine-per-jobwatchtowerheartbeatsagent vaultreboot runbookthe crontab, todayENGINEERworktree guardpre-push gateverify laddercritique looplane territoriesthe last look at the screenINCIDENT → GUARD → CHECKED INBOXES ARE THE STUDIO'S OWN MECHANISMS, AUGUST 2026. DASHED ONES ARE RULES NOTHING ENFORCES YET.
    0088 min

    Intelligence at every layer

    Five layers of a software company, what has become a check that fires, what is still a document, and what stays a person. Measured over six weeks in our own operating repo.

    agentsoperationsorganisationcraft
  8. 20252026AUG 2024JUL 2026FEATURES25 SHIPPEDWORKING MAPADDRESS SEARCHFACILITY POPUPSREP LOOKUPPAGES DEPLOY119TH DISTRICTSCLUSTERING, REVERTEDDUCKDB LAYERVERIFY + SHIPCOMMITSPER MONTH, NO MERGES9560WHO COMMITTEDDEVELOPER B3921DEVELOPER C32CLIENT WEB TEAMTHE ENGINEER24292020OPUS 4.5OPUS 4.6OPUS 4.8SONNET 5
    0097 min

    Two years, one codebase, and the agent that grew inside it

    A client's map app, read back from 384 commits. A team built it, we kept it alive, and for the last eight months the commits carry a model's name. One timeline holds all three.

    agentsgitclient-workcraft
  9. 0109 min

    Two physics bugs the tests caught before a user did

    jungl runs a year of sunlight through a model of your home. The sun part is exact, the room part is a model, and both were confidently wrong on day one. Here is what the tests assert, and why only a deterministic core makes them possible.

    testingsimulationphysicsjungl
  10. 0119 min

    A number is not a number

    Nine wrong figures reached our founder in one day and not one query was broken. The arithmetic was fine. Source, as-of, population and grain were what had gone missing, and now a checker refuses to hand back an integer without them.

    metricsinstrumentationagentscraft
  11. 0129 min

    A critique loop that actually converges

    Rulers certify an ad that is correct, honest, pixel-exact, and ugly. The fix is a panel of narrow critics that can veto, a score that weighs instead of counts, and a loop that returns its best round rather than its last.

    agentsevalsimagescraft
  12. 0139 min

    Sounds like AI, measured

    "This copy sounds like AI" is unfixable as an opinion. Count the tells per hundred sentences against a human control and it becomes a threshold anyone can fail, and a gate that stops the next draft from drifting back.

    copymeasurementagentscraft
  13. 0148 min

    Surfaces that regenerate need a memory

    A daily card ranked over a stable corpus is a deterministic repeat. Four instances of that one defect turned up in a single day, and the fix was the same ledger four times.

    productrankingretentionagents
  14. Four stacked depth layers labelled click, visit, connect and return. A probe descends through all four, its shaft fading from muted to brass as it goes deeper. The shallow layers are crowded with tiny faint marks; the deepest layer holds only six brass marks, each ringed. A depth axis on the left runs from noise at the top to meaning at the bottom.NOISEPROBEREAD AT DEPTHCLICKVISITCONNECTRETURNMEANINGFEW MARKS · SPARSE ON PURPOSE
    01511 min

    The link was already used

    A one-time sign-in link is a bearer token in a URL, travelling through machines whose job is to open URLs. Here is the exact mechanism, the traces, and why the numeric fallback dies with it.

    authemailhttpverification
  15. A tall hatched column marked “capacity, before” stands beside a very short brass column marked “capacity, now”; a brass arrow curves from the top of the tall one down to the short one, and a dimension bracket notes both are the same unit. To the right, five metered bars labelled hosting, database, storage, model and email show each running line billed on its own dial. No figures appear anywhere.COST OF ONE UNIT OF CAPACITYSAME UNITCAPACITY, BEFORECAPACITY, NOWMETEREDHOSTINGDATABASESTORAGEMODELEMAILEACH LINE ON ITS OWN DIAL
    0166 min

    Capacity got cheap, judgment did not

    Production now arrives on a subscription, and every other line of the stack prices itself daily. What is left to be scarce is knowing what to build.

    economicsstrategyagents
  16. Three promise cards on the left, each with a budget shown as a row of filled pips, fire probes into a field of anonymous audience marks. Each probe lands on a node and a return path carries the answer out to three oscilloscope lanes on the right. Two traces are faint and noisy; one, in brass, comes back with a clean strong swing. No percentages or counts are shown.HYPOTHESISAUDIENCESIGNAL RETURNEDH1BUDGETH2BUDGETH3BUDGETH1H2H3EACH WITH A BUDGET ATTACHEDRELATIVE STRENGTH · NO SCALE
    0176 min

    Ads are a research instrument

    A campaign can tell you which description of a problem strangers recognise as their own. That is a narrower answer than it looks, and worth paying for.

    demandadsstrategy
  17. Five stacked bands — land, connect, analyze, report, upgrade — narrowing downward like a machined aperture, with a measurement rail down the right side and a brass bracket marking the single step where the walls close hardest. No values are shown.FIG. 02 · FUNNELELOIST · BACKSTAGEONE STEP LOSES MOST OF THE ROOMNOT TO SCALEENTRY WIDTHEXIT WIDTHLANDCONNECTANALYZEREPORTUPGRADELARGESTDROP
    0186 min

    The funnel is a build artifact now

    A funnel change now ships through the same repository and deploy path as any product change. What did not get faster is finding out whether it worked, and that gap is where the discipline has to go.

    funnelinstrumentationgrowth
  18. A queue of differently sized candidate blocks on the left; four of them selected into a fixed rectangular budget on the right, outlined in brass; the largest candidate left outside, and one block drawn running straight through the floor of the window because it does not fit.FIG. 03 · CONTEXTELOIST · BACKSTAGESELECTION IS A PACKING PROBLEM, NOT A SEARCHNOT TO SCALECANDIDATESSELECTEDCONTEXT WINDOWDOES NOT FITBUDGETFIXED SIZE
    0196 min

    Context is the bottleneck

    Model capability keeps improving and tokens keep getting cheaper. What stays hard is telling an agent which of the things already in the repo is the one the team considers correct.

    contextagentsarchitecture
  19. A wide gap between a self-reported agent success card on the left and the live production surface on the right. Four dashed bridges start at the report and stop short in mid-air; a single brass path crosses the whole gap and lands on the live surface.VERIFICATIONCLAIM VS EVIDENCEREPORTEDOBSERVEDAGENT REPORTSTATUS COMPLETELIVEOBSERVED IN PRODUCTIONSTOPS SHORTDISTANCE TO A REAL USER
    0206 min

    The agent says it shipped. Prove it.

    Every signal we call verification proves something narrower than its name. The discipline is knowing which rung you stopped on, and saying so.

    verificationagentscraft
  20. Five output specimens travel left to right through four vertical hairline gates labelled grounded, in voice, in scope, and not last week's bug. Four specimens pass every gate and reach the rail on the right; the fifth is stopped in brass at the in-scope gate and diverted down into a held box.PROPERTY EVALSSHAPE, NOT EQUALITYGROUNDEDIN VOICEIN SCOPENOT LASTWEEK'S BUGPASSEDOUTPUT SPECIMENSHELD BACK
    0216 min

    Testing a feature that answers differently every time

    Property-based testing, metamorphic testing, fuzzing and statistical acceptance have asserted things other than equality for decades. The new part is applying them to the product surface, where the output under test is prose a customer reads.

    evalstestingquality
  21. Two stacked panels. Above, four agent tracks converge into a single shared lane where a brass burst marks a collision and the lane continues to main as a broken dashed line. Below, the same four tracks each run in their own isolated lane and rejoin at one clean brass merge point before reaching main.PARALLEL AGENTSCONTENTION VS ISOLATIONSHAREDCOLLISIONMAINONE WORKING COPYISOLATEDMAINCLEAN MERGEONE WORKING COPY EACH
    0226 min

    The half of Brooks's law that survives agents

    Running many agents in parallel is the real multiplier for a small team; what limits it is contention over shared state, which draws the same curve as Brooks's law by an entirely different mechanism.

    concurrencycoordinationagents