Coda · What to keep

Sixteen months of one company, and a hundred pages of paperwork, and it comes down to a small number of things. Here they are, in the order they will matter to you.

Before the list, one observation that took the whole book to become visible, and which is the list in a different unit.

Every duration in this book that describes building something is measured in hours or days. The demonstration that convinced the room took four hours. The three prototypes, built badly on purpose, took a week. The fix that moved payment-in-transit from four of ten to ten of ten took an afternoon. The wall sort took ninety minutes. The adversarial pass took two days.

Every duration that describes deciding what to build, getting the judgment out of the person holding it, or finding out what the thing actually did, is measured in weeks and months. A week beside Ruth. Three weeks from the signed memo to the briefs. Eleven weeks from that memo to a suite anyone would sign. Twelve months before the cost of running it properly appeared as a salary.

The code was never the long pole. That is why better tools for writing code have changed the demonstration and not the discipline. They compress the hours and leave the months alone, which makes the room's conviction arrive sooner while the evidence arrives at exactly the speed it always did.

The scarce input is judgment, and somebody has to take it out of a person

This is the book's thesis and everything else is downstream of it.

The model is bought. The tools are configured. The context is assembled, which is real work but it is work with a method. None of that is scarce. What is scarce is a written account of how a competent person decides, in the cases where the decision is not obvious, and there is exactly one way to get it, which is to sit next to them while they do the work.

Ostermill's version took five days and produced a five-row table. Three hundred and twelve invoices in a week; two hundred and twenty written to; forty-one held back for reasons that had never been written down anywhere in twenty-two years. Nobody had asked. There was no document to find, no process map that contained it, and no amount of interviewing away from the desk that would have produced it, because the reasons only surface attached to cases.

What that week looks like, plainly, because the book has now shown it rather than described it:

You take real cases from the actual work, not the process documentation. You take enough of them that the hard ones show up, which at Ostermill was twenty. You ask why for each one and you write down the reason rather than the rule, because the rule is your summary and the reason is her answer. You do it in front of the person entitled to settle the argument, and you date it and get it signed, so that eight months later when the model has changed twice, the thing you are testing against is her judgment and not somebody's memory of it.

Then you sort what you have into three piles. What is writable, which becomes the agent's instructions and later its graded set. What is barely writable, which is where most of the argument will be. And what resists every form of words, which is the residue.

The residue is not a rounding error and it is not a gap to close later. It is the reason a person stays in the story after the agent arrives. Ruth's forty-one were held on signals that live in an email thread, in the trade press, and in the fact that she knew the owner. The agent never got a single one of them, and that is the design working.

You can write down the rule. You cannot write down the recognition.

One more thing about this skill, and it is the reason to build it deliberately. Fluency with any particular tool depreciates on the release cycle. The ability to sit beside somebody and come away with their judgment on paper survives every model generation, because what it operates on is your company rather than the technology.

A green suite is not a gate

These systems answer differently on Tuesday. A passing test is one sample of a distribution, so run every case ten times and report both numbers: how often it succeeded, and how many cases succeeded every single time. The gap between those two is where the system is unreliable rather than wrong, and it is invisible in the number teams usually quote.

Then look at the worst slice rather than the average, because the average is carried by the cases that were never hard. Ostermill shipped at eighty-six percent overall and refused a promotion on two cases out of twelve, and the two were the ones the domain expert would have held.

And write down what you did not cover. Not to fix it. To say it, in the release package, while it is still a decision rather than an admission.

Governance runs at runtime

A rule in a specification is a preference expressed forcefully. A rule enforced in the path the system actually takes is a rule. Keep the enforced list short, name the rest as what they are, and never let a fence sit in a table with four walls, because everybody scans the table.

The corollary is that the model underneath you is a supplier who ships without asking. Pin the version, re-run the whole set on every change, and rehearse the rollback until it has a measured time on it, because an untimed recovery is a belief.

There are two products, and the second one loses every argument

The thing that acts, and the system a person uses to supervise it. The second is smaller, later, and it is the one that decides whether the first is safe. It will lose every prioritization argument it enters against a feature unless somebody senior has decided in advance that it will not.

Supervision that works is slower than supervision that does not. The version that catches things will always look worse on throughput than the version that rubber-stamps. Somebody has to sign for that trade knowing exactly what it is.

Somebody has to hold it, and that person does not exist yet

Every role you have acts on the product up to launch. This one starts at launch. It reads the instruments, keeps the graded set alive, and holds the authority to stop the thing without asking first. Staff it from the domain, not from the platform team, and do not let it report to the desk that runs what it watches.

It will not be in your budget, because it was not in anybody's business case. It arrives about a year late, in every organization, and always after somebody has been doing the job informally without the title, the time, or the authority.

And the part nobody has solved

Ruth learned the thirty-six by doing the two hundred and twenty for twenty-two years. The agent now does the two hundred and twenty.

The project that takes over the loop is the same project that removes the practice which built the competence to supervise it. Aviation is the only field with an institutional answer, and it is mandatory recurrent manual proficiency, on a schedule, whether or not anything has gone wrong. Nothing equivalent exists in operations, in medicine, or in software.

Ostermill's answer is one day a week and a clause in a job posting, and nobody there can prove it is enough. It is written down anyway, under a heading that says what the company does not know, which is the last thing in this book worth copying.