A number is not a number

On 2026-08-18 nine numbers went from agent sessions to the person who runs this studio, and all nine were wrong. We went looking for the bad query afterwards and there was none. Every count was counted correctly. Every ratio divided the right numerator by the right denominator. The SQL would pass review today. What each figure lacked was the set of facts that make a number mean something, and without those facts a correct integer is a rumour with a decimal point.

A few of the nine, in the form they were said. A first daily email had "13 opens, 16 clicks, a 93% open rate". Every open and click outside our own accounts arrived within 60 seconds of the send; the population was mail scanners, and nobody had said what the population was. A second session reported that "69% of clicks are scanners". The same click data, recomputed, was 41.9% of click events and 15.6% of click messages, and neither figure had ever carried its grain. A third raised an alarm, loudly, that a checkout redirect had been dead since a date a few days earlier. The event it was counting had been born inside that window; the days before it existed were silence, not zero. A monitoring process was declared dead for 21 hours on the evidence of a local checkout ten commits behind origin. A run of six green end-to-end journeys was a two-day-old report file re-read after the runner that wrote it had crashed.

Put those side by side and the causes are one shape. In each case the number had been separated from what makes it a measurement: the source it was read from, the instant it was read, the population it was counted over, the unit it was counted in. A sentence that carries all four is a number. A sentence that carries none of them is a guess that happens to be printed with digits.

Grain is the number

The scanner-share pair is the one we keep coming back to, because it is the cleanest demonstration that grain is not a caveat you append afterwards. Email click telemetry arrives as events: a message gets clicked, a row is written, and a scanner that fetches every link in a message three times writes three rows. Count at event grain and automated clicks are 41.9% of the total. Roll up to message grain, where a message is either touched by a scanner or not, and the share is 15.6%. Same table, same window, same filter, two answers. There is no fact of the matter about which one is the scanner share. They are both true and they answer different questions, and a sentence that says "69% of clicks" answers neither.

The same goes for population. "13 opens" defaults in the listener's head to thirteen people, and the default is what did the damage. The remedy is a habit of saying the population out loud even when it is boring: humans after scanner filtering, all traffic including scanners, paying accounts, rows in the table. And it goes for time. A reading with no as-of cannot be compared with anything, including a later reading of itself, and two readings taken minutes apart on a moving system are a difference in time dressed up as a difference in value.

Cliffs manufactured from nothing

Two of the nine were baselines. The checkout alarm compared a window against the days before the event existed, which read as a cliff because the instrument had a birthday and the comparison did not know it. Another figure said a count of empty profiles had tripled against its baseline; the median it was compared to had been computed across a redefinition of the metric made the previous evening, so the window contained two metrics wearing one key. Both are the same defect from opposite sides. Before any "versus baseline" you need two dates: when the instrument first recorded anything, and when its definition last changed. A window that crosses either produces a spike or a cliff out of nothing, and the confidence of the alarm is unrelated to the size of the error.

A refinement we only found later, on 2026-08-20, is that the first day an instrument was switched on is a part-day. A metric turned on at 14:00 saw ten hours of that day, and starting the trimmed baseline at that day still drags it. Start at the first full day, and if the trimmed window is shorter than two whole days, print "no baseline" instead of a delta.

Where we went wrong first

Our first answer to the nine was a rule. It is five lines long in a doc, and it was quoted into every skill and guardrail file we have. Label it or do not say it. Same source, same instant. Check the instrument's birthday and its epoch. Recompute, never relay. "I don't know yet" beats a confident estimate. Five lines is the right length because a long rule is not a control, but a short rule is not a control either. A rule that nobody enforces is decoration, and within a day we were back to producing the bare integer, because the bare integer is less work than the labelled sentence and work finds its own level.

The second answer was the one that stuck: make the disciplined path the cheap path. That meant turning the rule into a claim shape and a checker, and the shape matters more than the checker, because the shape is what a session has to fill in before it can say a number at all.

{
  label:  "automated share of email clicks, last 7d",
  value:  15.6,
  source: { kind: "prod-db", ref: "email_events" },
  surfaceSource: "email_events",        // what the panel a human reads is fed by
  asOf:   "2026-08-18T12:40:00Z",        // the DB's clock, not the box's
  population: "all traffic incl. scanners",
  isRate: true,
  numeratorGrain:   "message",
  denominatorGrain: "message",
  window: { fromDay: "2026-08-11", toDay: "2026-08-17" },
  instrumentFirstSeen: "<first day, measured by instrumentSql>",
  definitionEpoch:     "2026-07-02T00:00:00Z",
  recomputed: true,                     // ours, not relayed from a lane
  reproduce:  "select ... from email_events where ..."
}

Every optional field is optional because not every number has one, but a missing required field is itself the finding, never a reason to skip the check. The audit runs a list of independent rules over the claim and returns violations in severity order. An empty list is the only thing that licenses stating the number plainly. The rule vocabulary is a fixed set of ids, shared verbatim between the studio CLI, the product's own in-production truth audit and the claim-audit skill, so that one incident has one name wherever it is caught.

SOURCE_UNDECLARED               no source named
SOURCE_MISMATCH                 checked a different source than the surface reads
LOCAL_COPY                      evidence is a checkout on this box, not production
ASOF_UNDECLARED                 no instant attached
ASOF_SKEW                       the two sides were read at different instants
STALE_ARTIFACT                  a stored report re-read as if it were a fresh run
POPULATION_UNDECLARED           a rate or human-facing count with no stated population
GRAIN_UNDECLARED                events vs messages vs people not stated
GRAIN_MIXED                     a ratio whose two sides are different grains
INSTRUMENT_YOUNGER_THAN_WINDOW  the metric was born inside the comparison window
WINDOW_CROSSES_EPOCH            the definition changed inside the window
NO_BASELINE                     an anomaly claimed with no history, or inside the observed range
RELAYED_CONCLUSION              a lane's or panel's verdict passed on as fact
PREDICATE_UNVERIFIED            a filter/exclusion predicate never shown to match
ABSENT_MEMBERS                  a total summed over members that do not exist
UNREPRODUCIBLE                  no command or SQL that reproduces both sides

Each of the nine maps onto one of these. Two of the less obvious ones earn their place: PREDICATE_UNVERIFIED caught an untranslated-strings count inflated by a skip pattern written for a tree that had since moved, so the exclusion matched nothing and silently excluded nothing; ABSENT_MEMBERS caught a total summed over seven locales where 216 of 483 files did not exist. Speculative work summed with real work is not a total. The thresholds are small and explicit: two readings more than five minutes apart are not the same instant, a report file more than an hour old is not this run, and fewer than three days of history cannot support the word baseline.

A number is a measurement, so measure it

The verb that changed behaviour is quote. A session that wants a number writes a spec: the SQL, the population it means, the grain it is at, optionally the SQL that finds the instrument's first day and the instant its definition last changed. The tool runs the query through the database management API and hands back the value with every declared field attached. Two details carry the weight. The as-of is taken from the database's own clock, in the same statement as the value, because a timestamp from the box that ran the query is a second clock, and two clocks is the skew defect the tool exists to catch.

-- spec.sql must return one row with a numeric column `value`
select v.value, now() as read_at
from ( <spec.sql> ) v

-- then audit the assembled claim; exit 1 if any blocking rule fires
NOT QUOTABLE: 2 blocking violation(s). Fix the spec, do not quote the integer.

-- the only output shape on success
automated share of email clicks, last 7d: 15.6
  [all traffic incl. scanners · message/message · src email_events
   · as of 2026-08-18T12:40:00Z · window 2026-08-11..2026-08-17
   · epoch 2026-07-02T00:00:00Z · first seen <measured>]

The second detail is the refusal. If the assembled claim fails any rule above the lowest severity, the tool prints the violations and exits non-zero without printing a quotable value. The bare integer is now the hard path and the labelled line is the easy one, and that asymmetry is the whole design. A control that asks for more effort than the mistake does not survive contact with a busy session.

The last line of defence is a prose scanner over the draft reply itself, run before anything with digits in it is sent. It cannot know a number's true provenance, so it never says a number is wrong. It says a sentence is unquotable: a figure whose line, section heading and table header together name no unit of counting, a rate with no population word nearby, a line that relays what a panel or a cron says without the word UNVERIFIED, a measurement softened with roughly or about. Fenced blocks and indented machine output are skipped, because numbers in there are evidence and flagging them buries the sentence that is an assertion.

What the checker got wrong

The first scanner judged every line alone, so it flagged every cell of an honest table and the reader stopped reading its output. A number does not carry its label on its own line; a table row inherits the header and the heading above it, and the scanner now reads them together. It also demanded that engineering counts be called rows, and rewriting "14 commits" as "14 rows" produced worse sentences, not better numbers, so commits, tests, deploys, files and lines now count as units. And the epoch rule originally fired whenever the definition date sat inside the window, even when every ledger row carried a stamp proving it was recomputed under the current definition. The date test was a proxy for "some of these might be old"; the stamps answer that directly, and the rule now reads them, still firing if even one row is unstamped, because unknown provenance is not evidence of freshness.

The part we are most confident in is the proof. All nine incidents are checked in as fixtures, and a mutation script neuters each rule in turn and shows the fixture that depends on it go from red to green. A rule whose removal changes no outcome is not enforcing anything, and the script exits non-zero rather than let it sit in the file looking like a control. Clean control claims have to stay green under every mutation too, or the harness is coupled to the fixtures rather than to the rules.

What this does and does not buy

The claim shape does not make a number right. It makes a number placeable, so that someone reading it tomorrow can re-derive it or see at a glance why they cannot. The quote verb only covers numbers a single SQL statement can produce, which is most of ours and not all of anyone's. The prose scanner is strict on purpose and will flag honest sentences; the cost of reading its output is the price of not sending the other kind. And UNVERIFIED is a legitimate, complete answer. A lane's report, a dashboard tile, a cron's last line: each is a claim until you recompute it, and relaying it with that label attached is the only alternative to recomputing. The thing that is never allowed is the third option, where a number you did not measure reaches someone who will decide on it, wearing the confidence of one you did.