The funnel is a build artifact now

The drop-off sat on one screen. People arrived at Eloist, read what it does, clicked through, and stopped at the step where you hand over a chess.com or Lichess username. A loss concentrated at one place, where one thing is asked of the visitor, is easier to work with than a loss smeared across the whole path. We had known about it for a while, in the way you know things from a chart somebody looks at monthly.

What is different now is where a change to that screen comes from. It arrives as a pull request in the product repository, written on the day the question came up, reviewed and deployed on the same path as everything else. Funnel work used to live outside the codebase, on a separate tool and a separate schedule, joined to the product by a redirect and an act of faith. Calling the funnel a build artifact is a claim about plumbing, not strategy: the connect screen has a commit history, and reverting it is one command.

What got cheaper

The round trip from noticing that a step leaks to having an alternative version of it live is now short enough that it stopped being the thing we plan around. What that changes most is the threshold of what is worth trying. When implementing a variant was expensive, only large changes cleared the bar, a redesign or a pricing move. Now the narrow fix is worth shipping: the wording on one button, the order of two fields, whether a free no-signup page asks for an email before or after it shows you something. On the Eloist funnel most of what is worth testing is that size, and most of it was already sitting in a doc, unbuilt, because building each one cost more than the answer was worth. Putting a variant live sooner is the obvious change, and the one that reorganizes the work is what it does to measurement: implementation got faster and the evidence did not.

The half that did not get faster

Nothing in the toolchain touches experiment latency. The steps deep in the funnel, past the report, get a thin stream of visitors, and waiting for enough of them to separate a real effect from ordinary variation takes as long as it always did.

That turns the new speed into a hazard with a specific mechanism. A stream of variants shipped against thin traffic does not produce a stream of answers. It produces a stream of differences, most of them noise, and the ones that look largest are the most likely to be noise. Run enough of them and something will always appear to be up, and if the loop is ship, look, keep whatever the query liked, then chance gets promoted into a decision and hardens into a belief about our users that nobody can trace back to anything. The old process forced us to wait longer before judging a change, which sometimes protected us from a noisy early read.

What the change is supposed to move, and how much movement would count, get written down before it goes out. So does the stopping rule: a metric watched continuously will cross any line you care to draw eventually, so the window is fixed in advance rather than when the connect numbers happen to look good.

Novelty is its own trap. A changed screen is unfamiliar, and unfamiliar screens get looked at. We cannot tell from an early read whether a movement on the connect screen is the change or the newness, so the window has to run past the point where anyone would notice that something moved.

The most expensive mistake in this loop is local success. Every step in a funnel is easy to improve on its own terms, and the cheapest way to improve one is to push the difficulty downstream. Ask for less at the connect step and more people get through it, including people we have not yet given a reason to come back. That step reports a win while the report step absorbs the cost, and the readout down there is the slowest one we have. So every hypothesis carries a guardrail: at least one downstream metric with the standing to veto the result, chosen while nobody yet knows how it will read. If connect completions rise and reports opened fall, the change is a loss.

Making the funnel legible

Whatever binds an agentic team in general, on the funnel the limit is instrumentation, and it caps everything above it. The set of questions anyone can ask is bounded by which steps emit events, and missing events do not announce themselves as gaps: a step nobody instrumented looks like a step that is working. So every step on the Eloist funnel emits one, and every event is queryable from the same terminal where the change is written.

The verdict also has to come from somewhere the agent did not author. Ask an agent whether its change helped and you get a fluent account assembled out of the diff it just wrote. So the hypothesis, the metric, the guardrail and the stopping rule go into the repository before the change ships. The record below is what that looked like for the connect screen.

hypothesis: people stall at connect because it is unclear
            which of their two accounts we want
change:     name both sites on the button, drop one field
metric:     connect completions over connect starts
guardrail:  reports opened per connected account (may veto)
window:     fixed up front, sized by the slowest step
            downstream, never by when the number looks good
decide:     by the rule written above, or not at all

Where the advantage sits

Implementation speed will stop separating teams as the tooling spreads. Any team that puts agents on its repository can ship funnel variants at whatever rate it likes, and that capability keeps getting cheaper and more evenly distributed.

What is harder to copy is a funnel where every step emits an event, a history of changes each tied to the metric it was meant to move and the guardrail it had to clear, and the willingness to revert the ones that failed to clear it. That accrues slowly and there is no way to buy it in a hurry. Each experiment leaves the next one cheaper to judge, because the baseline is already measured and the argument about what counts as an effect has been had.

The decision that used to be a hire, somebody to own conversion, is now made at build time. A team that skips it ships variants at the same rate and gets weaker answers out of each one, so we make the call early on every product in the portfolio, before there is a funnel worth optimizing: which steps emit events, and whether anyone can query them without asking permission.