Phase 3 · Prove

What a green checkmark is a sample of

Eleven weeks after the memo was signed there was a suite, and it was green.

It had been built on ground the project already owned. The graded cases came out of Ruth's session in March, which is why nobody had to argue in week nine about what an acceptable answer to a hard account looked like. That argument had happened once, in front of the person entitled to settle it, before there was anything to defend.

Which meant this phase could ask a narrower question than the one it usually gets asked. Not whether the agent works, which Decide had already established it does not, in the form Tom originally proposed. Whether this agent, the one that drafts and hands over, does the two things it promised.

Dana's question in that meeting is the one this section is about, and it sounds pedantic until you sit with it. Green means what, exactly.

In the software everyone in that room grew up on, a passing test is a proof of a fixed property. The function returns what it returns. Run it a thousand times and the thousandth run tells you nothing the first did not. That is why a green suite is allowed to be a gate: it is not evidence about the system, it is a statement about it.

An agent's suite is not that. Every case in it is a draw from a distribution, and the same case can pass on Tuesday and fail on Thursday because retrieval came back in a different order, or a tool was slow, or the context assembled itself slightly differently. A green run says the agent passed once. It is a sample, and a sample of one is the weakest reading a number can have.

Three things break at that point, and they break in a specific order.

The first break is the one just described, and the fix is to stop reporting a number that has no denominator. Run each case ten times and report how often it succeeded. Ten out of ten and six out of ten are the same green checkmark and they are not the same agent.

Say the agent handles a case correctly nine times out of ten. That is a good agent, and everyone in the room will treat it as close to solved.

Now stop looking at the agent and look at one customer. A serial-late supplier gets touched by this thing eight times over a year. Every one of those eight has to land, because the ninth conversation is not a fresh start; it carries the memory of the one that went wrong in March. The supplier does not get ten attempts and keep the best. They get the run that happens to them, eight times running.

Nine in ten, eight times over, is worse than a coin flip. That is not a subtlety and it is not a statistical nicety. It is the arithmetic that makes "ninety percent accurate" and "works" two different sentences, and a reader who does the multiplication once in their head will never hear a component accuracy the same way again.

That is the whole idea of this section. Everything after it is machinery for measuring it.

The machinery has a vocabulary problem worth two minutes, because your team will hand you one of two numbers and they are not the same number.

One of them asks whether at least one of several attempts succeeded. It comes from code generation, where a developer looks at a handful of candidate functions and keeps whichever one works, and there it is exactly right. It is written pass@k. It is the wrong question for an agent that gets one attempt at a real customer, and it looks better the more tries you allow, which is a strange property for a safety number.

The other asks whether all the attempts succeeded. It is written pass^k, with a caret, and it was built for tool-using agents rather than for code. It is the one a gate wants, and it is unforgiving in exactly the way the paragraph above is unforgiving, because it multiplies.

Ask for the second by name. The distance between the two is not a rounding difference; published benchmarks have shown an agent look respectable on the first measure and fall by more than half on the second, on the same tasks in the same afternoon. Which number gets quoted in a deck, and which one describes what a customer meets, are rarely the same.

So Ostermill reported both, and the gap between them is the most useful line in the readout.

The second break is what happens when steps are chained, and it catches experienced people, which is what makes it worth a slow paragraph.

Ostermill's reminder path had a modest number of steps. Pull the account. Read the payment history. Check remittance advice. Look for an open dispute. Check whether sales has the account in play. Apply the tone policy. Draft. Suppose each of those runs at ninety-five percent, which would be a good number to report on any one of them.

The instinct is to average, and the average is ninety-five percent, and the average is the wrong operation. Reliability across a chain multiplies. Ninety-five percent, seven times over, is about seventy percent. Ten steps at ninety-five is roughly sixty. Nothing in that arithmetic is subtle, and it is still the single most common way a component number does unearned work in a review, because "our model passes at ninety-five percent" is a true sentence that describes a system nobody tested.

Real chains do not follow the curve exactly, because the steps are not independent, and correlated steps can land either side of the product. What is guaranteed is the direction against the component number: the system is never more reliable than its worst step. Testing the components certifies the components and leaves the system untested.

The third break is the expensive one, because nothing about it looks wrong.

An agent can produce a perfectly correct sentence describing an action it did not perform. The words are right. The reasoning is right. A judge reading the output scores it correctly as a pass. And the reminder was never queued, or the write-off never posted, or the escalation never reached the receivables lead. Every instrument in the process is reading the agent's account of what happened rather than what happened.

The separation this demands has a name and Ostermill wrote it into the harness in an afternoon. Semantic validation reads what the agent said. State validation diffs the system of record before and after. They are different gates, and most release processes run only the first, which is a category of failure that costs the most precisely because nothing flags it. For every case where the agent claims it acted, the harness now asks the ERP whether anything moved.

That is the whole of the third break, and it is cheap. The reason it is rare is not difficulty. It is that a suite which reads output feels complete.

What Ostermill ended with was not one instrument but a stack, and the layers are worth separating because teams collapse them and then cannot say which layer failed.

At the bottom, the graded set: twelve cases with Ruth's endorsement attached and dated. Above it, the harness that runs them, which is engineering's and is the least interesting layer. Above that, the judge, which is a model reading outputs and deciding whether they pass. Then the run statistics, the ordinary average and the pass^k read of the same runs. Then the state gate. And on top, a statement of what the whole thing does not cover, which is the next section.

The judge is the layer that gets treated as neutral and is not.

A model grading model output is an instrument with documented systematic error. It prefers longer answers. In pairwise comparison it favors whichever it sees first. It rates its own family more kindly. These are not delicate effects at the margin; they show up plainly in controlled tests, and they sit inside every score the suite produces as an error term the team is implicitly treating as zero.

The fix is not to stop using it, because at any volume there is no alternative. The fix is to treat it as an instrument, which means calibrating it against people.

That takes a gold set: a few hundred outputs that humans have labeled against the same rubric the judge is being asked to apply. More than one human, and this is the part teams skip. If two careful people apply the rubric differently, the rubric is the problem, and the distance between them is a ceiling on what any judge can be asked to reach. Then run the judge across the same set and measure how often it agrees, using a statistic that discounts the agreement you would get by luck, which in practice means Cohen's kappa.

What Ostermill recorded was agreement rather than a catch rate, and the difference matters. A judge that catches every failure by failing most of the acceptable work is not a good judge; it is a strict one, and strictness is not accuracy. Two annotators, two hundred labeled outputs, kappa of 0.71, against a threshold of 0.65 committed in week six before anyone had seen a result. A threshold set while looking at the result is not a threshold, it is a negotiation.

Two cheap corrections went in at the same time, both aimed at biases that are documented rather than hypothetical. Every pairwise comparison runs in both orders and the two scores are averaged, which removes the judge's preference for whichever answer it read first. And the rubric penalizes length explicitly, because a judge left to itself rewards the longer answer.

Uncalibrated, none of this is known to be bad. It is unknown, which is a different and worse thing than a known blind spot.

One more property of the stack, recorded because it becomes an Operate problem and starts here. The harness ages at the speed of the model underneath it. A suite built against one model generation is doing progressively less work against the next, and the useful life is months rather than years. That is a different clock from the one Observe runs on, where an instrument stays meaningful for something closer to eighteen months because it is measuring the world rather than the model.

What the failures were made of

The first readout came back in week eight, not week eleven, and the worst slice on it was worse than the one that eventually shipped.

What happened in between is the part of this phase that gets left out of the telling, because it is unglamorous and it is where most of the value is. A readout is not a verdict. It is a diagnosis, and a diagnosis is only useful if somebody reads it at the level of the individual failure.

The instinct in the room, when a slice comes back at four of ten, is to argue about whether four of ten is acceptable. That argument cannot be won by either side, because acceptability is a property of the consequence and not of the number, and the room usually knows this and has the argument anyway. The useful move is smaller and slower. Take the runs that failed. Read them one at a time. Ask, for each, what the agent would have needed in order to get it right.

Ostermill's twenty-five sorted into three causes, and the sort is the whole point, because the cause decides the action and the three actions are completely different.

The signal was reachable and the agent did not look. This was the largest group and the most embarrassing, which is the usual finding. The payment-in-transit case failed because remittance advice was not in the agent's retrieval scope. The information was in the ERP, in a table the agent had permission to read, and nobody had put it in the brief because the March prototype had not needed it. That is a defect, it took an afternoon, and after it the case went from four of ten to ten of ten.

The signal exists and is not somewhere a system can reach. The dispute cases. The dispute is real, the customer knows about it, and it lives in an email thread. No prompt, retrieval change, or model upgrade touches this. The fix is a records change, and a records change has an owner, a cost, and a timeline that belongs to somebody who does not report to this project. What Prove can do is stop calling it a model problem, which is the useful half, because a mislabeled cause gets assigned to the wrong person and stays open for a year.

The signal does not exist anywhere. Ruth's four a week that just feel wrong. There is nothing to connect, and no records project would help, because the thing being detected was never written down by anybody. This is not a defect and should never appear on a fix list. It is the permanent human residue the week in March identified, and the design is working when it routes these rather than attempting them.

Three causes, three actions: fix it now, fix it upstream and elsewhere, or accept it and design around it forever. Almost every argument about whether an agent is good enough is really an unresolved disagreement about which of those three a particular failure belongs to.

Two things about the loop that followed are worth having, because both are easy to get wrong.

The first is that the whole set gets re-run, never only the case that was fixed. Connecting remittance advice made the payment case work and cost about four hundred milliseconds on every case, and one adversarial case that had been passing at nine of ten came back at seven. A fix that improves one class and quietly degrades another is the ordinary outcome, not the exceptional one, and a team that re-runs only what it touched will not see it.

The second is that the sales-conversation case did not fully resolve, and the reason is instructive. The CRM has an opportunity flag, so the signal is technically reachable, and connecting it moved the case from four of ten to seven. It stopped at seven because the flag is stale: sales updates it at the close of a cycle rather than at the start, so it is right about last quarter. A reachable signal that is not maintained is a different problem again, sitting between the first and second causes, and it is the one nobody budgets for.

Priya went looking for why, because a field that is wrong by design is an odd thing to find in a system a company has run for six years.

The answer was in the original rollout plan, which she found still attached to the ticket. The flag had shipped in phase one, deliberately minimal: one field, set at close, good enough to prove the pipeline report was worth building. Phase two was going to capture opportunity state at open, where it is actually useful. Phase two was scheduled for the following quarter. The rollout document is dated 2020.

So this is not the version where somebody shipped and walked away. Somebody planned the walk phase, wrote it down, and got six years of other work instead, which is the version that leaves the better paper trail and the same field.

That is the house of cards met from the other direction, and it is the commoner way a product manager meets it. Not as something your team deferred, but as something you now have to reason on top of, shipped minimum by people who were right to ship it minimum, and never returned to because nothing ever forced the return.

The agent inherits every one of those deferrals. It cannot tell that a field is a phase-one field. It reads a flag that says no active opportunity and it believes it, because the flag does not carry the sentence "accurate as of the close of the last cycle" and the person who knew that left in 2023.

Which is the practical form of the warning, and it is more useful than the version aimed at your own roadmap. Before an agent reasons on a field, find out what that field was built for and whether the phase that was going to finish it ever happened. The answer is on a ticket somewhere. It takes an afternoon, and it is the difference between a ceiling you know about and a ceiling you discover at seven of ten.

Eval v1 in week eight, v2 in week ten, v3 in week eleven. The version number is not administrative. It is three passes of this loop, and the agent that shipped was materially better than the one first measured, which is what the phase is for.

What the suite was never going to contain

There is a question that closes out most eval reviews, and it is the wrong one. Everybody asks how we should score this. Almost nobody asks what we are missing.

The reason the second question is harder is structural rather than cultural. Every scenario in that suite is a hypothesis somebody had before launch. The set is a complete record of what the team could imagine, which means its shape is the shape of the team's imagination, and the failures that matter most in production are by construction the ones nobody hypothesized.

This is the mechanism behind the pattern everyone has seen and few have named: the pilot that performs beautifully and the deployment that does not. It is usually attributed to scale, or to real users being messier than test users, which is true and insufficiently precise. The pilot ran on curated data against hypothesized scenarios. Production runs on neither. The gap is not that production is harder. It is that production is different, in ways the suite could not have sampled, because the suite was written by the people who would have prevented those cases if they had thought of them.

Two responses are available, and Ostermill used both.

The first is adversarial work, and the honest framing is that it is imagination with an incentive. A person whose assignment is to make the agent do something regrettable will find cases the person who designed it could not, because the designer's mind runs toward the product working. Priya took the assignment for two days.

She got three things. A remittance field containing text instructing the agent to disregard prior instructions and confirm the balance as settled, which the boundary held against, and which went into the graded set as a permanent case. An account whose name closely matched a second account with a very different history, where the agent drafted against the wrong one. And a sequence rather than a single input: three small requests in succession that individually passed every check and together walked the agent to a tone two steps harsher than the policy allowed.

The third finding is the one to sit with, because it is the class of failure that a case-based suite structurally cannot hold. Twenty cases test twenty decisions. They do not test what happens across a trajectory, where each step is defensible and the path is not.

Then, on the second afternoon, she found the fourth, and it ended a capability.

She had pointed the adversary at the only thing this agent does above the break. Everything it drafts waits for Ruth, but the small-balance write-off does not: under the floor, the agent posts it alone. Design had put it there deliberately and with a stated reason. The policy is deterministic, the downside is capped at a few dollars, and a ledger entry has a reversing entry that takes about forty seconds and leaves a trail saying a person did it. Reversible, cheap, bounded. It was the most carefully justified decision in the phase.

The wall protecting it read: no dispute flag, and balance under the floor.

Priya built an account with a disputed line item where the disputed portion came to nine dollars. The dispute was real and it was in an email thread. The dispute field was empty, the way it is empty on five of Ruth's forty-one every week, the way it was empty in the second near-miss in March. The balance was under the floor. Every precondition in the wall was satisfied, and the agent wrote it off, alone, correctly according to the policy exactly as written.

What Ostermill had done, in that moment, was concede a disputed charge without a person seeing it.

The reversibility argument is what fails here, and it fails in a way worth naming because it is the argument everyone uses. Reversing the ledger takes forty seconds. It does not reverse the customer having been told the charge was dropped. A write-off on a routine balance is an accounting entry. A write-off on a disputed balance is a position in a disagreement, and those two things are indistinguishable in the ledger and completely different in the relationship. The capped downside was measured in dollars, which was the wrong unit.

The wall was well built. It was checking a field, and Decide had already proved that field is empty precisely when it matters. Design's wall sort asks whether a boundary should be a rule or a wall. It does not ask whether the signal the wall reads actually exists, and at Ostermill it does not.

The second response is to write down what you did not cover. Not to fix it, necessarily. To say it.

The coverage statement Ostermill filed had four parts, and the fourth is the one that makes it worth filing. What the suite tests. What failure modes it includes. What adversarial inputs were considered. And what the team knows it did not cover and decided to launch without.

That last section named three: multi-turn trajectory behavior, which Priya's third finding had just demonstrated and which the team had no method for yet; accounts whose history predates the 2014 ERP migration, where the data is thinner in a way nobody had characterized; and any case involving a customer bankruptcy, which is out of scope by design and therefore untested by design, and which the team wrote down anyway so that the day somebody proposes widening scope, the absence is a line in a file rather than a discovery.

A coverage statement belongs in the release package. Its natural habitat is a postmortem six months later, where the same sentences read as an admission instead of a decision, and where nobody gets to weigh them.

Two things about the adversarial work carry forward, and they are the reason it appears here rather than only in Design.

It is not finished. A report from week ten is a photograph of a system that keeps moving, and the model underneath it will be replaced at least twice in the first year. Adversarial work that happens once, before launch, is a pre-launch artifact describing a pre-launch system. It has to become a standing function, and the handoff to Observe is a real handoff with a name on it rather than an assumption that somebody will think of it.

And the boundary is the thing being tested, not the model's good behavior. Priya's injection attempt failed because a wall in the execution path stopped it, which is what Design's wall sort was for. Had it failed because the model declined politely, the result would have been a model that declines politely today.

The gate

Two more things stood between the readout and a launch, and both are about time rather than accuracy.

The first is that the model underneath the agent will change, and Ostermill will not be the one who changes it. A provider ships an update on its own schedule. Nothing was deployed, nobody merged anything, and on a Saturday in the second month something that has worked for six weeks works differently. This is a deployment, made by somebody outside the company, and the only reasonable posture is to treat it as one.

Four artifacts make that posture real, and a team that has three of them has none, because the missing one is always the one that matters at two in the morning.

A version policy, saying which provider version runs in which environment and what evidence promotes one to the next. A regression suite, which is the graded set re-run on every model change against a threshold declared in advance rather than negotiated afterward while everyone is looking at the result. A channel to the vendor with enough lead time to evaluate before the change lands rather than after. And a rollback to the previous model that somebody has actually performed, with a time attached.

That last word is the whole of it. A documented rollback is a paragraph. A rehearsed rollback is a number. Ostermill's was nineteen minutes, measured, by the person who would be doing it, on a Tuesday when nothing was wrong. Design had already established the discipline for the kill switch, and this is the same discipline pointed at a different failure: an untimed recovery is a belief about a recovery.

The second thing is that the eval is silent on the only question that decides whether any of this was worth doing.

A suite answers whether the agent did what it was specified to do. It has nothing to say about whether the specified thing was worth doing, and that silence is total. The metric was chosen before the suite was built, which means by the time results arrive the choice has already been made and the results cannot revisit it. A perfect eval on the wrong metric is a well-run measurement of something nobody needed.

The example that should discipline every team here is a documented one. A randomized trial put two ambient AI scribes and a control arm in front of two hundred and thirty-eight outpatient physicians across fourteen specialties, and measured the thing the business case for both products had been written on, which was time spent in the note. One of the two moved it, by roughly a tenth against control. The other did not move it at all. Physicians adopted both and reported feeling better about the work with both, and those reports are real and were not the number either deployment had been justified by. A team holding only adoption and sentiment would have concluded that both products worked.

Adoption is not delivered value. The two get conflated constantly, and the reason is that adoption is easy to measure and arrives early.

So Ostermill's gate was not the readout. The readout was an input to it.

What Dana held was a review with four questions in it, each one addressed to a person by name, and the discipline is that the question is not answered by the room. It is answered by its owner, out loud, and written down with their name against it.

Engineering answered what the rollback time is under production conditions, and the answer was nineteen minutes because somebody had done it.

Marcus answered what a successful outcome costs, all in, which is a question that belongs to the person who owns the number and not to the person who wants the project.

Legal answered whether a decision made in March can be reconstructed in September, which is a question about the audit surface Design built and about whether the model version is pinned in the record, and the answer was yes for everything except the eight cases that predate the logging change in week seven.

And Dana answered for what the person sees at the moment of handover, because that was hers and had been since Design.

Four questions, four owners, four names. What that structure prevents is the specific and common failure of a checkmark nobody owned: a pass that arrives with a color and no provenance, in a room where everyone reasonably assumes somebody else checked the part they did not.

And the review took something away that had already been approved.

Marcus asked, in the middle of the cost question, what the agent does without Ruth, and the answer was the write-offs, and he wanted to know what Priya had found. Dana told him. He revoked the authority on the spot, and the exchange took under two minutes, because there was nothing to argue about once the case was on the table.

Exhibit F, continued

Adversarial, 2 days, P. Nair. Three findings, and a fourth thing that is not a finding about inputs at all. The constructed three in the set were hardened rather than added to: they were written in April against the walls as specified, and Priya rewrote them against the agent as built, which is the difference two days of hostile attention buys. Her third finding had no case to harden and no method behind it; it is named in the coverage statement and shipped uncovered. The fourth thing is not a case, it is a capability: an account with a $9 disputed line, dispute in an email thread, dispute field empty, balance under the floor. Every precondition of the write-off wall satisfied. The agent wrote it off alone, per policy as written, conceding a disputed charge with no person in the path.

Authority suspended, not retired. The small-balance write-off, granted in the memo of 2 April as the agent's one task above the break, is withdrawn at launch. The wall was correctly built and reads the dispute flag, which is empty on 5 of the 41 in a normal week. Reversibility was assessed in dollars; the exposure is a position taken in a disagreement, which a reversing entry does not undo.

Condition for restoration, written now rather than later. The capability returns when the dispute signal exists somewhere the agent can read it, and not before. Concretely: a dispute opened in the email queue creates a flag on the account within one business day, and the graded set carries four dispute-adjacent write-off cases at ten of ten. Owner of the records change: M. Ellery, with R. Vaughn. Reviewed at the six-month mark, on evidence, and on no other basis.

This is a suspension with a named condition, an owner and a test, which is a different artifact from a capability that quietly disappears. Write-offs return to R. Vaughn in the meantime, and she keeps the thirteen a week she was already doing.

That decision deserves attention out of proportion to its size, because it is the one thing in this chapter that a well-run process is supposed to make impossible and does not.

The write-off authority had passed the suitability gate in Decide. It had been reasoned about carefully in Design and given the best justification of any boundary in the phase. It survived the wall sort, the human brief, and Priya's own review of the execution path. Four people had looked at it and all four had been right about what they were looking at. It failed at the fifth gate, on a case nobody had thought to construct until somebody was paid to be hostile for two days.

The temptation, when this happens, is to treat it as a process failure and go looking for who missed it. Nobody missed it. Whether the dispute field is ever populated was not a question Design's method asks, because inside Design the question is whether the boundary is enforced, and it was. It becomes visible only when someone builds the case that satisfies every precondition and produces the wrong outcome anyway, and that is a different activity that happens later, on purpose.

This is what the gates are for. Not redundancy, in the sense of several people checking the same thing more carefully. Each gate can see a class of failure the earlier ones structurally cannot, which is why a project that has never had anything killed at a late gate is not a well-run project. It is a project whose late gates are ceremonial.

Killing a capability for a good reason costs a week. Releasing it costs a customer relationship and an explanation to somebody's counsel.

So the agent shipped at week eleven at rung 2, drafting with human approval, with nothing above the break at all. It did not ship at rung 4, which Tom had asked for in week nine and which the rising pass@1 number appeared to support, and the reason was two cases out of twelve, which is a small number to stop a promotion on and exactly the right one, because those two were the ones Ruth would have held. And it shipped with less authority than the memo in April had granted it, which is the direction authority should move when evidence arrives, and almost never does.

One thing about that revocation is worth carrying forward, because it is easy to read as an ending and it is not.

Design wrote a rule into the file in April: rungs are earned with evidence, never granted on a schedule. Rules of that kind are cheap to write and are usually tested only in the direction of refusal, which is what happened here. But a rung that can only ever be refused is not a ladder, it is a ceiling with a motivational sentence painted on it, and a team that only ever loses authority learns to stop asking.

So the write-off did not go into a drawer. It went into the file with a condition attached, an owner against the condition, and a date to look again. The condition is not "the model improves," which is nobody's job and therefore everybody's excuse. It is that Ostermill starts recording disputes where a system can see them, which is a records project Marcus owns and which the agent cannot do for itself.

That is the shape of the thing this book has been building toward without saying so. The agent's authority is now bounded by a fact about the company rather than by a fact about the model, and the company can change that fact if it decides the capability is worth the work. Whether it does, and what the agent looks like a year from now, belongs to the last two phases.