Appendix B: The Field Manual

Part I: Decide

AI literacy: name the four layers. Take one agent on your roadmap and name all four layers: its model, its orchestration, its tools, its context. The layer you can describe least precisely is the layer your quality problems will come from. Fix that description before you write another line of the spec.

Autonomy: place it on the ladder, honestly. Place one product on the Autonomy Ladder at the rung it has earned, not the one the roadmap wants. Then write the single demonstrated competence that would justify moving it up one rung. If you cannot name it, you do not yet have permission to climb.

Suitability: fill the sheet. For one candidate agent, run the four tests (repeats at volume, bounded tool use, recoverable consequences, measurable and trusted) and the break-even (value × volume against tokens × architecture multiplier + supervision). If all four pass and the break-even holds, build it and you can defend the decision. If any test fails, write the one sentence explaining why you are building it anyway, and notice whether that sentence would survive being read aloud to your CFO.

Part II: Prototype & Collaborate

How the work splits: walk the shape. For your next agent, walk the six seams. Did you run the suitability test and a separate affected-populations scan before the brief? Is the autonomy boundary set before you design the agent, as a constraint on it, not after? Is there one human brief and a derived executable brief, rather than one document trying to be both? Is the agent’s behavior written as outcome-centric specs graded by an eval set, not as pass/fail user stories? Is the release gate two halves, Channel 1 ready and Channel 2 ready, with a named owner for the second? And after launch, who owns the governance stream that never ends? Any step you cannot answer is where your old process is quietly running the new work.

Vibe coding: prototype to decide. Take one roadmap bet. Name the single riskiest assumption, the one-day prototype that would test it, and the specific result that would kill the idea. Build it, put it in front of someone who has the problem, record what it proved or disproved, and hand engineering the learning, not the code.

Collaboration: name the owner of every domain. For one agent, name the owner of each of the five runtime domains (goal definition, control plane / kill switch, rollback, supervisory UX, correctness certification) and the four launch co-owners (engineering = rollback time, CFO = per-task cost, legal = audit surface, product = approval moment). Any domain that resolves to “me” or to no one is an unowned domain about to fail.

Part III: Design

Behavior: the four artifacts as four sentences. For one agent, write the four runtime artifacts as four sentences (boundary, approval moment, audit surface, recovery workflow), then run the five security decisions: (1) input trust tiers, all external input untrusted by default; (2) per-tool blast-radius bound, with per-action approval where the blast radius is unacceptable; (3) memory write / validation / rollback architecture; (4) every action class tagged automated / notify / require-approval, with deletions, payments, outbound communications, and sensitive-record access set to require-approval; (5) tamper-evident, human-readable logging plus a machine-speed incident playbook. Any artifact you cannot state in one sentence is undesigned.

Oversight: which kind does each output need? For each output class the agent produces, assign program-level or transaction-level oversight using the four axes (harm asymmetry, reversibility, regulatory exposure, per-review cost). For every output you mark transaction-level, write the seconds-per-decision the reviewer actually has, and confirm a competent human could exercise judgment in that time. Any transaction-level gate funded at a sampling budget is performative review: either resource it or narrow what flows through it. Check the consequential output classes against current domain rules (EU AI Act Article 14; US healthcare physician-review laws; credit adverse-action reason requirements; sector financial-services sign-off), since the mandates are tightening.

Evals: run the sign-off as a checklist. For one agent: (1) end-to-end pass rate across ten or more runs, with the worst slice, not component averages; (2) state validation present for every action with a state change, not just semantic validation; (3) judge calibrated against a human-labeled failure set, with a pre-committed threshold; (4) coverage statement including the scenarios deliberately deferred; (5) model version policy plus a rehearsed rollback. Any item you cannot produce is a known gap to sign off on explicitly, not a box to leave unchecked.

Part IV: Operate

Guardrails: three ceilings and a name. For one production agent, write the three hard ceilings as numbers: tokens (per run and per rolling window), requests per window, wall-clock per run. Confirm each is enforced before the call (gates the next call), not after (logs the last one). Verify the kill switch lives outside the agent’s runtime and is wired to the burn-rate alert. Put a tool-level circuit breaker (trip on infrastructure failures only, timeouts and 502/503/504, never on 400/401) on every external dependency. Set burn-rate alert tiers (warning at 80% of daily; page at three times baseline hourly rate; kill flag at the daily ceiling) and name the human paged at each tier. Any ceiling you cannot fill in is an uncapped dimension in production.

Observation: a threshold per instrument. For one deployed agent, write the threshold for each of the six instruments (task-success-versus-completion gap; unintended action rate; override frequency, the band, high and low; confidence calibration; rollback time; incident recovery time). Mark which breach pages someone tonight (real-time) and which feeds the next sprint (retrospective). For any instrument you cannot produce at all, name the runtime surface that was never designed. Add a data-layer check: freshness, completeness, referential integrity, retrieval provenance. Model every threshold at production volume, not pilot volume.

Silent degradation: date the last re-calibration. For one deployed agent: (1) write the launch-time scope statement and diff it against what the agent does today; revoke post-launch tools nobody authorized; (2) list every substrate that can drift (model version, training cutoff, context behavior, tokenizer, retrieval corpus, each tool API, system prompt, guardrails) and who watches each; (3) date the last re-calibration of every monitoring instrument, and treat anything older than the last frontier release as suspect; (4) put the currency question in the vendor contract and on the renewal calendar; (5) schedule one external audit on a held-out sample the vendor did not design. Any item you cannot complete is an open drift vector.

Audit: build the sealed decision artifact. For one consequential decision class, specify the sealed decision artifact: (1) decision record (inputs, verbatim reasoning chain, verbatim output, confidence, timestamp); (2) immutable model-version reference (weights hash or pinned provider API version); (3) data and retrieval-corpus version; (4) governance record (who authorized this model for this use, the policy, any waivers); (5) human-intervention record (override or confirm, who, when, why); (6) appeal record (left open, filled on contest). Store write-once and signed at write time, and store the output verbatim, because non-deterministic models do not reproduce from inputs. Set retention to the real limitations period of the relevant claim (roughly two to three years for credit, five to seven for healthcare, seven for SOX-covered finance), not the six-month regulatory floor. The test: could a stranger reconstruct the decision from the bundle alone, with the live agent gone?

Part V: The Human System

Deference: find the disagreement. For one agent where a human reviews the output, find a recent case where the agent and the human disagreed. Which was right, and did the workflow help the reviewer decide or did they defer by habit? Check both collapses: is the agent reliable enough that reviewers have stopped really checking (blind automation), or overridden as a matter of course (reflexive rejection)? Confirm the agent can signal which case this is, that its expressed confidence is calibrated to its actual reliability and not just loud. Anything you cannot answer is a place the human is deferring blind.

Change management: name the supervisor. Name the supervisor for one agent you have shipped. Is it a real role, with time on a calendar and an artifact they own? Or a responsibility bolted onto someone already full? The second means you shipped one product and assumed the other, and the assumed one fails first.

The Loop Test. Pick one agent where your design says a human reviews the output, and run the Loop Test: time, skill, attention. If it passes all three, you have real oversight, and you should be able to say exactly how each holds. If it fails one, you have found the chapter’s point sitting in your own architecture, and you know which fix it needs.

Skill erosion: the three questions. Pick one skill your agent now performs that a human on your team used to perform. Ask: are your experts still practicing it, are your juniors still learning it, and is anyone still checking the agent’s version against their own judgment? Each “no” is one of the three erosions, already underway, traceable to a design decision someone made or defaulted into.

The agent as team member: write the onboarding packet. Write the one-page onboarding packet for one deployed agent, as if it were a new contractor: its scope of work, its access scope, who manages it, how you re-evaluate it on the next upgrade, how you decommission it. Every line you cannot fill is a personnel function you are missing, for an actor already working.

Part VI: Carry the Weight

The people the agent never sees. For one agent: (1) name every party affected beyond the user (the people who absorb the output, not operate it) and the outcome each experiences; (2) run a stratified performance analysis across those populations before launch, treat systematic underperformance for any group as a defect rather than a footnote, and identify who is in the error rate and why their inputs are sparse; (3) enumerate the actions the agent will never take regardless of instruction, enforce the list at runtime, and log refusals as first-class events; (4) apply the heuristic that if the affected person would expect a human and be harmed to learn none was involved, require human accountability at the moment of consequence; (5) confirm the moral architecture is enforced in the product (signature, named owner per action class, refusal mechanism, disclosure posture), not described in a deck.

The before/after week, and the week ahead. Sketch your week three years ago against now, in two columns; mark what an agent now does; in the freed space, mark where the hours actually went versus where you wish they went. Then map your next week against the six parts (Decide, Prototype & Collaborate, Design, Operate, the Human System, Carry the Weight). The part that gets the least of your time is the tell. If it is a choice, write the one-sentence defense. If it is a default, that is the part of the new job you have not yet started.

One diagnostic question per phase

• Decide: would the suitability sentence survive being read aloud to your CFO?
• Prototype: which runtime domain resolves to “me” or to no one?
• Design: which of the four artifacts cannot you state in one sentence?
• Operate: which ceiling is uncapped, and who is paged when it breaks?
• Human system: is the supervisor a role with time on a calendar, or an assumption?
• Carry the weight: who is inside the error rate, and did anyone design for them?

Red flags

A single green checkmark presented as proof on a non-deterministic system. A “freeze” or a boundary that lives in the prompt rather than the architecture. A monthly cost alert described as a safeguard. A transaction-level review funded at a sampling budget. A monitoring instrument no one has re-calibrated since the last frontier model shipped. A consequential decision with a six-month log retention. A supervisor who is “everyone.” An agent doing work no launch review authorized. An aggregate accuracy number with no one asking who is in the error rate.