Appendix · Five artifacts, blank
Every exhibit in Part Two is a real document with Ostermill's answers already in it. That is deliberate, because an empty template teaches nothing about what a good answer looks like. It is also insufficient, because the exhibits are easier to admire than to copy.
These five are the ones that are hard to reconstruct from a worked example, and each is an artifact the case shows the value of, by its absence or by the failure it prevented. Fill them badly and you have still done better than not having them, because a bad answer written down is a bad answer somebody can argue with.
One instruction that applies to all five. Date every one, and name an owner. An undated artifact ages into a wrong one silently, and an unowned one is a document rather than a control.
### 1 · The graded case
The unit of the eval suite. Not a test case, which asserts an output; a graded case, which records a judgment and who is entitled to make it.
| Field | What goes in it |
|---|---|
| Case ID | Stable. It will outlive three model versions |
| Class | Routine, signal resolvable, signal not resolvable, adversarial |
| Provenance | Observed, and where from; or constructed, and against which rule |
| The situation | Enough state that the case can be re-run against a system that has changed |
| Correct behavior | What the agent should do, including doing nothing |
| Why | The reason, in the domain expert's words, not the PM's summary of them |
| Endorsed by, date | The person entitled to settle the argument. A name, not a team |
| k, and pass^k to date | Runs attempted, runs clean, at every version |
The distinction that does the work. Observed cases come out of the real week. Constructed cases cannot, because nobody's real week contains a planted instruction, and a set built only from observation has no adversarial half. Mark which is which.
Fails when: the reason is missing. A case with a right answer and no reason cannot be argued with when the model changes, so it gets quietly retired by whoever is inconvenienced by it.
Abstracted from: Exhibit F.
### 2 · The judge calibration record
Filed before the eval runs, not after. Its whole value is that the threshold was committed while nobody knew whether it would be met.
| Field | What goes in it |
|---|---|
| What the judge scores | The specific property. Not "quality" |
| The rubric | The same text the human annotators were given. Verbatim |
| Gold set size, annotators | More than one annotator, or there is no agreement to measure |
| Agreement statistic, threshold | A statistic that discounts chance. Committed and dated before the run |
| Measured agreement | Against that threshold |
| Order effects | Pairwise comparisons run in both orders, scores averaged |
| Length control | The rubric's explicit penalty, or the demonstration that length does not correlate with score |
| Re-calibration due | A date. The judge ages with the model underneath it |
Fails when: the threshold is set after the number is known. Everyone does this in good faith and it converts the gate into a description.
Abstracted from: Exhibit F.
### 3 · The retrieval scope decision record
A configuration file says which fields the agent reads. This says why, and what was deliberately left out, which is the half that causes the incident.
| Field | What goes in it |
|---|---|
| Sources in scope | System, table, field. Specific enough to diff |
| Trust level, per source | Trusted, or untrusted and therefore data rather than instruction |
| Sources out of scope | Named, one line each |
| The class of fact each exclusion loses | The point of the record. "Anything a person typed into a note" |
| Who can change this, and how | A named owner and a review step |
| Reviewed on | A date, and the next one |
Fails when: it does not exist, and the scope is discoverable only by reading a configuration file that has no reasons in it. Every field left out is a class of fact the agent cannot know, and the day one of them matters, nobody can say whether the omission was a decision or an oversight.
Abstracted from: Exhibit H.
### 4 · The constitutional rule
Three lists, and the distinctions between them are the whole artifact. A rule the agent is told is a preference expressed forcefully. A rule that is enforced is a rule.
| Field | What goes in it |
|---|---|
| Rule ID, text | Short enough to quote in an incident review |
| List | Constitutional, enforced in the execution path; mitigated, stopped at the obvious cases; or told, carried in the brief |
| Enforcement point | The specific place it is checked. Blank means the rule is mitigated at best, and told at worst |
| What happens when it fires | Block, route, log. And who sees it |
| Why it is on this list | Especially for the told ones. There is usually a cost that was not paid |
| Version, and the dates it was in force | A number and a date range. A decision record pins the version that was live when the decision was made, or an auditor reads this year's rules against last year's call |
| Last time it fired | Never is a finding, not a reassurance |
Keep the constitutional list short. A long one is not a stronger constitution. It is a told list that has been relabeled.
Fails when: a fence is filed among the walls. Everybody scans the table, four walls and a fence read as five walls, and the organization believes it has a guarantee it does not have.
Abstracted from: Exhibit C and the constitutional runtime in Operate.
### 5 · The supervisory ledger
The cost of running the agent properly, split so that nobody can present a ratio built from one measured number and one asserted one.
| Line | Measured or judged | Owner |
|---|---|---|
| Model and tool cost, per task class | Measured, dated | |
| Review minutes, at a loaded rate, per rung | Measured | |
| Recovery cost when the agent is wrong | Measured, where an incident exists | |
| Eval maintenance, per model change | Measured, and it recurs | |
| Instrument reading, hours per week | Measured, and it recurs | |
| The supervisory seat, when it exists | Measured, and it is a salary | |
| What the agent is worth | Judged. A name signs here |
The rule of the form. Costs are a rate times a quantity. Value is a counterfactual, and the central one is usually a harm that did not occur, which cannot be measured because it did not happen. Keep the two columns apart and do not divide one by the other.
Fails when: the review line is left out. It is almost always the largest line, it is the only one that changes when the rung changes, and a pilot that omitted it becomes a production system that cannot afford itself.
Abstracted from: Exhibit B, Exhibit G, and the ledger in Decide.
### A closing note on all five
None of these is long, and none of them is the point. The point is that each one forces a sentence somebody would otherwise not write: the reason behind a correct answer, the threshold before the result, the field deliberately excluded, the rule that is only a preference, the number that is a judgment. Those five sentences are where the failures in this book actually started.