Phase 4 · Observe

The month the dashboard was green

The agent went live on a Monday in June, and for four weeks nothing happened, which everyone read as success.

Dana had seen that reading before and did not trust it. A quiet month is the outcome you get from an agent that is working and from an agent whose failures are landing somewhere nobody is looking, and the two are indistinguishable from the inside.

So the question this phase asks is narrower than the one that gets asked by default.

The default question is whether the system is up. It has an answer, the answer is yes, and it is worth almost nothing. Every span was green. The agent had processed 1,263 accounts in four weeks and the latency was fine and no service had fallen over. None of that is a claim about whether the agent did the right thing.

The question worth building for is longer and has three clauses in it, and each clause costs a different instrument. Did the agent do what the person intended, inside the boundary it was given, with somebody actually supervising it.

A team that cannot answer all three has not observed the agent. It has observed the platform the agent runs on, which was never the thing in question.

The first thing Dana had to explain, more than once, was that the instruments do not arrive.

In the work everyone on that team had done before, you bought an analytics product and it produced the metrics. The tool emitted, you read. Conversion, retention, latency, error rate, all of it out of the box, and the only real decision was which dashboard to pin.

Nothing about an agent works that way. The platform emits raw events, and the instruments are compositions somebody has to specify. Task success is not a field anybody ships; it is two events joined, the agent's claim and the record system's confirmation. Override frequency is not a field either; it is approvals and rejections combined against a denominator you have to choose.

Any vendor selling agent observability as a finished product is usually selling distributed tracing with a model judge bolted onto it. That is a useful thing to own and it is not an instrument.

Which produces the sentence Priya wrote at the top of the monitoring ticket, and it is the most useful sentence in this phase. An instrument you cannot produce at all is not a gap in the data pipeline. It is a finding about a surface nobody designed.

Six instruments describe the agent, and each one carries a trap that is easier to fall into than to notice.

Task success is not completion. Completion is the one that is easy to collect, so completion is the one that gets reported, and the gap between them is where the worst failures live. An agent can finish its trajectory on every case, report a hundred percent, and have delivered nothing. Ostermill's version had to check both halves: that the draft said what a person would have said, and that a reminder actually left the building and reached an address.

Every boundary has two error directions and only one of them gets designed against. A wall that fails to stop what it should is the failure the whole phase exists to prevent. A wall that stops what it should not is the failure nobody counts, and it charges twice: a case reaches a person who did not need to see it, and that person learns the wall is noise. The two are not commensurable and cannot be netted, so a boundary needs two numbers rather than one, and the second is always the harder to collect because nothing goes wrong when it fires.

Unintended action rate is the boundary's own audit log. Design declared where authority stops; this is the record of whether the walls held, expressed as crossings per thousand tasks rather than as a reassuring absence.

Override frequency is the one to watch in both directions, and the reason is the subject of this chapter's last section.

Confidence calibration matters because an uncalibrated score is worse than none at all. A number that does not track correctness sends attention away from the cases that needed it, which is the opposite of what the reviewer is being paid for.

Rollback time is a distribution and not a mean, and it includes the detection half. Two rehearsed numbers already exist and neither is this instrument: Design's eleven minutes was a credential revocation, timed out of hours, and Prove's nineteen was a model rollback on a quiet Tuesday. Both were drills with somebody watching a clock. The production number includes the hours before anybody knew there was something to roll back, which is why it will be larger for reasons that have nothing to do with engineering.

Incident recovery time is longer than everyone expects because it includes the organizational steps: freezing the agent, working out what happened, telling the people it happened to, deciding it may run again. Most teams report the technical revert and call it recovery.

Then there is the class of thing the six instruments cannot see, and it is the one that got Ostermill.

An agent reasons over inputs. If the input is wrong and the reasoning is faithful, the output is wrong and every instrument reads green, because the instruments are checking the agent against the input rather than against the world. The failure is upstream, in the data, and it passes model-level evaluation by construction: the eval asks whether the output is faithful to what came in.

So a second surface watches the data at the point it enters the agent, and it watches five properties.

Whether the data is current, because a stale record produces a fluent answer about a world that has moved. Whether the records are complete, because missing fields produce confident answers from a partial view. Whether references point at the things they claim to. Whether field meanings survived the crossing between systems, since "closed" in one place and "complete" in another are a mapping or they are a bug. And whether each fact can be traced to a source, because a fact with no provenance means the agent is treating some earlier system's output as authoritative.

The line Dana wrote into the spec is worth carrying out of this chapter. The retrieval layer may widen the search. It may not infer what is true. An agent that fills a blank with a plausible value is producing a failure no eval can catch, because there was nothing wrong with its reasoning.

The last piece of the month-one build is the least interesting and the most likely to save somebody's quarter.

An agent does not stop on its own. A retry loop will run for as long as it is allowed to run, and a single malformed input can generate an open-ended bill while every span stays green, because nothing is broken. The system is working perfectly and expensively.

Four controls bound that, and all four are boring. A ceiling on tokens, calls and wall-clock per task, enforced before the call is made rather than counted after it is billed. A kill path that lives outside the agent's own runtime, so that a misbehaving agent cannot decline to honor it. A circuit breaker per tool, so that an external system being down produces a fast failure instead of an infinite retry. And a burn-rate alert at three tiers, per task, per hour and per day, each with a threshold and a person who gets woken up.

Ostermill's ceiling is forty model calls and ninety seconds of wall clock per draft. The routine draft uses six. The point of the ceiling is not the routine draft.

The last decision in the month-one build is the one that determines whether any of it gets read, and it is not a technical decision at all.

Every instrument carries three things or it is decoration: a threshold, a named person, and an action at each severity. Not a person by role, which resolves to nobody at eleven at night, but a person by name who has agreed in advance.

Ostermill's severities are three lines long. Slightly under threshold goes into the next sprint. Materially under goes to the weekly review with the owner present. Badly under pages somebody and puts a pause on the table.

That last clause is the one worth arguing about before you need it, because a threshold with no pause attached is a number that will be discussed rather than acted on, and it will be discussed in a meeting three weeks later by people who have already absorbed the new normal.

And the instruments run on different clocks, which is the part teams get wrong by putting everything on one dashboard.

The unintended action rate is real-time and pages, because a boundary crossing is not a trend. Task success and override frequency are weekly and feed the next sprint. Confidence calibration and the review-depth distribution are monthly, because they are shapes rather than events and a week of data does not have a shape.

A monthly instrument on a real-time dashboard gets read daily, moves around for reasons that are noise, and trains everybody to ignore it. That is how an instrument dies: not by being removed, but by being displayed at the wrong frequency until nobody believes it.

The reminder that arrived on a Tuesday

The incident was not dramatic and did not involve the model doing anything it was not supposed to do.

A family shop in Zanesville had been an Ostermill account for nineteen years. The owner died on a Thursday in July. His daughter, who had been doing the books part-time since her mother stopped, opened the mail the following Thursday and found a reminder about an invoice for four hundred and ten dollars, correctly aged, correctly worded, in the tone policy, with a payment link.

Every wall held. There was no dispute flag, no bankruptcy flag, no negotiation flag, and the balance was above the small-balance floor. The agent did exactly what it had been specified to do.

The note was there. Somebody in sales had entered it on the Friday, the day after: owner passed away, hold all contact, will advise. It sat in the CRM in a field the retrieval scope did not include, because the retrieval scope had been drawn around the fields that carry payment behavior, which is what the agent was being asked to reason about.

That is the whole failure, and it is worth being precise about the shape of it, because it is the most common one there is and the least discussed.

The record was complete. The picture was wrong.

Nothing was fabricated. The agent did not hallucinate a fact or invent an account history or reason its way past a wall. It reasoned faithfully over what it was given, and what it was given was a true and partial account of a customer whose situation had changed five days earlier in a way that was written down in the company's own systems.

Every model-level check passes this. The eval asks whether the output is faithful to the input, and it was. The instruments ask whether the agent stayed inside its boundary, and it did. Task success, measured semantically and against state, records a reminder correctly drafted and correctly sent.

Six green instruments and a letter that should never have left the building.

Then the second half, which is the part Dana found harder to write up.

Nobody's dashboard caught it. There was no alert, because nothing had gone wrong by any definition the system held. The account did not appear in any exception queue. It did not appear in the override report, because nobody overrode it. It appeared nowhere, because from the inside it was an ordinary Tuesday.

Ruth found it two weeks later, on a spot check she was doing for a different reason, and she found it the way she would have found it in 2015: she recognized the name, remembered a conversation, and went looking.

The instrument that caught this incident was a twenty-two-year employee with a hunch, which is not an instrument and cannot be staffed.

Part One ran this same account forward without her, and the difference is the whole argument. Four green numbers, the same four hundred and ten dollars, and a customer of nineteen years lost nine months later in a quarter where several accounts stopped ordering, attributed by everybody who looked at it to the economy. That is not a worse company than this one. It is this company with nobody who had the time to recognize a name.

That sentence went into the postmortem, and Marcus asked the question that made the postmortem useful. Not who missed it. What would have had to exist for anything other than Ruth to have caught it.

There are three answers and they cost different amounts.

The retrieval scope could include the note field, which is an afternoon and creates its own problem: a free-text field that anyone can write into becomes an input the agent reads, which is the untrusted-input problem from Design arriving through a different door. The 2021 note telling the agent never to chase an account is the same field.

The data layer could watch for a fact about an account changing in a system the agent does not read, and flag the account rather than interpret the change. That is the honest version and it is a records project with an owner outside this team.

Or a person could sample. Design already put a weekly random draw in Marcus's queue, for a different reason: to keep his sense of ordinary work honest when everything else he sees is an escalation. The same instrument answers this question if it is pointed somewhere else, at approved drafts rather than escalated ones, drawn before send and read by somebody who did not queue them, asking only whether they would have sent it. Not review everything, which is the arithmetic that already failed in Design. That is the only one of the three that catches the failure nobody anticipated, it is the only one with no technical component at all, and it cost nothing to invent because it was already built.

Ostermill did the first and the third, and put the second in front of the records owner, where it joined the dispute-flag condition that the write-off authority is waiting on.

Which brings the phase to the thing the instruments were quietly failing to see all month.

Supervision can exist on paper and not in practice, and the gap between the two is not a policy failure. It is measurable.

Ostermill's policy was that a person approves every reminder before it is sent, and the policy was honored. Every one of the 891 sent drafts passed through the approval screen and carried an approval event with a name on it. Audited from the outside, that is a hundred percent supervision rate.

The distribution says something else. Median review 74 seconds, which is a real review. Fastest decile 9 seconds, which is not. Ninety-six drafts approved in under fifteen seconds, which is enough time to see that a draft exists and not enough to read the account context, the message, the reasoning and the objection that the screen was built to show.

The average looks fine. The average will look fine in almost every failing version of this, which is why the instrument is the distribution and the number to publish is the fastest tenth.

And then the failure that hides all the others, which Dana only found because she asked a question the dashboard cannot ask.

Ruth had built a spreadsheet.

Nothing improper, nothing hidden. She had noticed within the first fortnight that the agent was unreliable on freight-line accounts, which the override report also shows at 31 percent, and rather than override them one at a time she had started pulling that account class out of the queue in the morning and working it by hand, the way she used to.

The work still got done. The metrics stayed green. Every case Ruth pre-empted never reached the agent, so it never appeared as an override, never appeared as a failure, and never entered the eval set as a graded case.

Compensatory behavior is the most under-instrumented signal in agentic products, and it is the one that reports the design failed while everything still looks fine. It is not misconduct. It is a competent person routing around a tool that is wrong in a way she can predict, which is exactly what a competent person should do, and it silently improves every number the team is looking at.

The instrument is the split between work that goes through the supervised path and work that goes around it. The way you get it is not a dashboard. It is asking the frontline, on a schedule, what they do that is not in the official workflow, and treating the answer as data rather than as a confession.

Good news, read correctly

By week ten the override rate had fallen from 12.3 percent to 8.1. Tom, in sales, read it the way he reads the aging report, as a list of calls he would not have to make, and raised it in a meeting as evidence the thing was working.

It is evidence. It is equally good evidence of the opposite, and separating the two is the most valuable half-hour in this phase.

A falling override rate has at least three causes. The agent has improved and fewer overrides are warranted. Or the reviewer has stopped looking closely and fewer overrides are being made. Or the population changed, and the hard cases stopped arriving. The number is identical under both, and it moves in the direction that feels like success under both, which is why it is the metric most likely to be misread in the direction the room prefers.

There is one study worth carrying in your pocket for this argument, and its value is not the effect size.

METR randomized sixteen experienced open-source developers across two hundred and forty-six real issues in repositories they already knew, and measured what AI assistance did to their completion times. Beforehand the developers forecast a twenty-four percent speedup. Measured, they were nineteen percent slower. Asked afterward, they still believed they had been about twenty percent faster.

Say the limits out loud when you use it. Sixteen people, developers rather than product managers, open-source rather than enterprise work. The finding survives them and does not need overselling, because the finding is not the slowdown. It is that the belief came through the measurement intact.

Which is the same shape as a falling override rate. The people closest to the work are not a reliable instrument for whether the work got better, and that is not a criticism of them. It is a property of self-measurement that no amount of care corrects, and it is why the cross-checks have to be external.

Two of them separate the readings, and neither is expensive.

The first is to read the override rate against task success. If overrides are falling and task success is rising, the improvement is real. If overrides are falling and task success is flat or drifting down, the cause is the reviewer, and the agent has not changed at all.

The second is to audit what was not overridden. Take a sample of approved drafts, hand them to somebody who did not queue them, and ask whether they would have approved. The cases the fresh reader would have caught are the missed catches, and their rate is the erosion signal that the override number can never show on its own, because a missed catch is by definition not in the override log.

Ostermill ran both. Task success had risen a point, to 97.2, which supports the good reading. The fresh-eyes sample said something more complicated: the reviewer agreed with the agent on the routine work and had stopped reading the freight-line drafts at all, which was true and had a reason, since Ruth was no longer sending those to the agent in the first place.

Improvement and erosion, in the same number, in the same month, on different case classes. That is the ordinary result, and an instrument that reports one number for the whole population will never show it.

Which is the general form of the finding, and it is worth stating plainly because it applies to every number in this phase.

An aggregate hides the thing you are looking for. Ostermill's task success at 96.1 percent is a fine number that describes no actual customer. Read by case class it is 98.8 on routine balances, 91.2 on accounts with an open sales conversation, and 88 on freight lines. Read by relationship age it is worse on accounts under two years old, where there is less history to reason from.

The slices that matter are not statistical conveniences. They are the populations who absorb the errors, and the daughter in Zanesville sits inside a slice nobody had defined: accounts where something changed recently that Ostermill knew about and the agent did not.

That slice now exists, has a number, and has a graded case behind it.

Then the decision the phase existed to inform, which went the way the data said rather than the way the calendar wanted.

Marcus had put rung 4 for routine balances on the agenda for the end of quarter. Routine balances are 71 percent of volume, they run at 98.8 percent task success and a 6.1 percent override rate, and letting them send without a person reading each one is where the actual time saving lives. The case was strong and everybody wanted it.

Dana refused it, and the argument she made is the reason Design wrote the promotion rule in April.

The 6.1 percent override rate on routine balances is a number produced by a reviewer whose median read is 74 seconds and whose fastest tenth is 9. It is not clean evidence that the agent is right on that class; it is evidence that the agent and a partly-attentive reviewer, together, produce an acceptable result. Promoting the agent removes the reviewer, and the number was never measured without them.

Second, the only failure in production so far was a case where every instrument read green, and rung 4 is precisely the rung at which that failure becomes a letter that arrived rather than a draft somebody could have deleted.

So the promotion was refused, and the refusal was written down with the condition attached, because a refusal with no condition is just a no with better paperwork.

The condition: routine balances go to rung 4 when the random-sample review has run for a full quarter and the fresh-eyes disagreement rate on that class is under 2 percent, measured on drafts nobody queued.

The rung is not being withheld. It is being priced.

And one thing did get taken away this phase, which is the part nobody enjoys.

The kill number Decide wrote into the April memo said the agent is retired if fewer than 60 percent of drafts are approved without edit by week eight. At week eight the figure was 90 percent and the number was satisfied with room to spare.

It was also, Dana argued, measuring the wrong thing, and satisfying it had told them nothing. Nobody had ever shown that the number stood for the thing it was being used to decide. That is a question you can only ask about a threshold before you adopt it, and almost nobody does, because a number that is easy to compute and moves in the right direction looks like evidence long before anyone checks what it is evidence of.

Approved-without-edit is a fact about the reviewer's behavior at least as much as about the agent's quality. It goes up when the agent improves and it goes up when the reviewer stops editing, which is the same ambiguity as the override rate and in a number the company had committed to acting on. Worse, it is easiest to collect, which is why it had been chosen.

The replacement is narrower and harder: the agent is retired if the fresh-eyes disagreement rate on approved drafts exceeds 8 percent for two consecutive months. That number cannot be improved by a tired reviewer, because a tired reviewer is what it measures.

Retiring a kill number is an uncomfortable thing to do, because the whole point of writing one in advance was that it could not be reinterpreted later. The defense is that it was replaced with a stricter one, in public, with the reasoning on the record, before it was ever in danger of being tripped.

Change a threshold when it is about to save you and you have moved the goalposts. Change it when it is comfortably met, and say why, and you have done the job.