Chapter 6 · The brain, the layers, the loop
What Priya had to hand it
The model that wrote those letters knew a great deal about collections and nothing whatsoever about Ostermill.
It had never seen the Dayton machine shop's six years of paying in the last week of the quarter. It did not know that the two percent tolerance is negotiable for the supplier with eight clean years and not for the one caught padding freight in 2019. It had no view on whether a dispute sitting in an email thread counts, because nobody had told it that disputes live in email threads at this company and in a dispute field at most others.
Priya supplied all of that by hand, for twelve accounts, on a Thursday afternoon, by reading each one and deciding what mattered before she pasted it in.
That is the part nobody scopes, and the demonstration hid it completely. She was the retrieval layer. She read the account, judged which three facts were relevant, and handed the model a tidy paragraph containing exactly what it needed and nothing that would confuse it. Then she showed the room the output and everyone concluded the model was good at collections.
The model was good at writing. Priya was good at collections, for twelve accounts, for one afternoon.
Everything that would make this Ostermill's rather than generic has to be assembled and handed over on every case, by a system, without her sitting there. That assembly is most of the build, and it is invisible in every demonstration ever given.
Six layers, and the one that hurt
Priya drew the stack on a whiteboard the following week, because the team had started arguing and she could tell they were arguing about different layers without knowing it.
It is worth saying what this is not. Chapter 4 listed six parts an agent has to have before the word applies; this lists six layers a team has to build and own. The first is a test of whether you have an agent at all. The second is a map of where the work and the money go. They overlap and they are not the same list, and a reader who tries to line them up one to one will be looking for a correspondence that was never claimed.
At the bottom, the model. Not built here, not inspectable, and due to be replaced under them on a schedule set in another company. Everyone understood this one.
Above it, instructions. The tone policy, the escalation rule, the description of the job. This is the layer everyone works on, because it is the layer you can edit in a text box while somebody watches, and it has less room in it than its popularity suggests.
Above that, retrieval. What the system fetches about this case before it decides anything.
Above that, tools. What it can do rather than say. Each one a permission, and the list of them a more honest description of the agent than any paragraph about it.
Above that, memory. What it carries between cases, which turns out in Design to be a capability question wearing a compliance costume.
At the top, the boundary and the supervisor.
The argument that had been running for three days was about the second layer, and Priya's point in drawing the stack was that the second layer was not where the problem lived.
Retrieval was.
The reminder agent needs the aging report, which is easy. The payment history, which is easy. Remittance advice, which is in the ERP and which the current reminder system has never once read. The CRM opportunity flag, which exists, and which sales updates at the close of a cycle rather than the start, so it is reliably right about last quarter.
And the dispute state, which is not a field.
That is where it stops being a technical problem. Five of Ruth's forty-one holds every week turn on a dispute that lives in somebody's inbox. You cannot retrieve what the company never recorded, and no model improvement changes that, ever. It is a records problem with an owner outside the project, a cost, and a timeline that belongs to somebody who does not report to Dana.
Priya wrote one line under the stack. The model is bought. The context is ours, and some of it does not exist yet.
The hour Dana spent watching it think
Priya offered to show her a run. Dana said yes the way you say yes to a demonstration, which is to say she expected to be shown a result.
What arrived instead was a trace, and it scrolled.
It read the account. It decided the balance would not settle the question on its own and went for the payment history. It came back and reconsidered, found no pattern there that changed the answer, and treated the nothing as information rather than as an absence. It checked whether sales had the account in play. It drafted. Then it read its own draft against the tone policy and revised one sentence in it.
Dana had specified every one of those steps. She had written the sequence into the brief herself, in order, with her own hands. What she had not done was watch it happen, and watching it happen was not the same thing as having written it down.
Somewhere in the third case she stopped reading it as software.
The order it reached for things in was Ruth's order, because the brief had been built out of a week spent at Ruth's desk and Dana had written that order down. Watching it work was the closest thing anyone had to watching the reasoning that had been in Ruth's head for twenty-two years, which nobody at Ostermill had ever seen either, because reasoning does not surface when the person doing it is fast and quiet and gets it right.
Then a case where it did something she had not specified.
It reached a defensible letter through a signal that happened to be present rather than one that meant anything, and it did not know the difference, because nothing in what it read told it there was one. The letter was fine. The route was luck. On a good day those look identical from the outside, and the outside is the only place a demonstration lets you stand.
This is the part of the job that has changed, and it has changed underneath a habit that used to be sound.
For twenty years the product manager defined what and why, and engineering owned how. That was not laziness, it was the deal, and the deal worked because how was decided at build time, tested once, and identical on the millionth run. Ignorance of it was efficient. When the executor is a model, how is decided at runtime, every time, by something that plans, improvises, recovers, cuts corners and occasionally reaches the right answer for no reason. The execution stops being an implementation detail beneath the requirement. The execution is the behavior, and the behavior is the product.
Which means the brief, the boundary, the tone policy and the graded cases are not documentation of the thing. They are its upbringing. And nobody can raise something they have never watched behave.
One more thing about that hour, because it is the one Dana carried into Prove. At the end of its run the agent wrote a short summary of what it had done, and the summary was accurate, fluent and confident. It was also not the same document as the run. Everything hard had happened in the middle, and none of the hard parts were in it.
Why the loop is the cost
The last piece is the one that surprised Marcus, and it surprised him in a meeting where he had expected to be bored.
A model call has one shape. Text in, text out, you pay for both, and the price is on a public page. An agent has a loop, and the loop runs until the agent reaches something it will commit to.
Read the account. Decide it needs the payment history. Fetch it. Reconsider. Check whether sales has the account in play. Fetch that. Reconsider again. Draft something. Check the draft against the tone policy. Revise it.
Every pass is another model call carrying everything gathered so far, which means the cost of a case is set by how many times it goes around and how much it is carrying by the last pass.
Two agents doing the same job at the same quality can differ by a factor of ten on that number, and nothing in the output tells you which one you bought. A case handled in four passes and a case handled in forty produce the same letter, and the letter is the only thing anyone in a demonstration ever sees.
Marcus asked who decides how many times it goes around. Priya said he was looking at her.
The shape of the loop is an architecture decision, made early, usually by somebody who does not know they are making the cost decision at the same time. Procurement cannot negotiate it because it is not in the contract. It is not a price you are quoted. It is a property of a thing your own team built, and it is the single largest determinant of what the machine costs to run.
Which turns out not to be the largest number on the page. Part Two prices the machine and the supervision side by side and the supervision is the multiple, by a distance. Both facts are true, and teams reliably spend their argument on the smaller one, because it is the one with a vendor attached to it.