The People the Agent Is Eroding

Every architecture review for an agentic product reaches the same slide. There is a box for the agent, a box for the action it takes, and between them a smaller box labeled human review. The box lets the team say the system is supervised, lets the regulator say it meets the human-oversight requirement, lets the executive say a person is still accountable. Most of the time it earns none of those claims, and the reason is not that the human is absent. The human is right there and means to review. The reason is that the box assumes a human who can catch the error, and the rest of this part has been about why, eighteen months in, that human is the least reliable component in the system, eroded by the very reliability of the thing they are watching. This chapter is about that human directly, and about the uncomfortable fact that the prior books handed their protection to the one person on the team who cannot provide it.

Lisanne Bainbridge wrote the foundational paper in 1983, about industrial process control, and called it the Ironies of Automation, and it has aged into a description of a future she could not have seen. Her argument was that automating a process does not remove the human; it automates the parts that are easy to specify and leaves the human the residue, the hardest and least structured work, to do under the worst conditions. The human no longer practices the task moment to moment, so their skill at it fades, and they are asked to step in only when the automation fails, which is rarely, unpredictably, and in exactly the situations the automation could not handle, which are the hard ones. The better the automation, the rarer the intervention, and the less ready the human is when it finally comes. Your supervisory design improves the normal case and degrades the abnormal case, and the abnormal case is the only one that ever needed a human. This is the supervision paradox from the opening of this book, and this chapter is where it does its damage, because the failures it produces are not the agent’s and not any single person’s, and the prior books, writing from the product manager’s seat, told the product manager to fix all of them.

The human bends toward the wrong answer, and the expert bends too

The first failure is the one most people would deny applies to them. Automation bias is the tendency to accept an automated system’s output without the independent check you would apply to a colleague, even when your own judgment would have been right, and even when you are an expert. The word expert is load-bearing, because the intuition every team carries is that bias is a novice problem and a senior reviewer is immune. The evidence says otherwise and it is not thin. The canonical review of the field found automation complacency in experts and novices alike, found it could not be instructed away, and found it produced both errors of accepting a wrong recommendation and errors of failing to notice a problem the system did not flag. In a mammography study, expert readers shown an incorrect AI classification moved their own scores toward the wrong answer; the machine pulled trained radiologists off assessments they had gotten right on their own. And a multi-site study found that when the AI added a highlight showing where it was looking, physicians accepted its incorrect reading more often, because the visual cue felt like evidence. The thing teams reach for to make AI more trustworthy, the explanation, made the bias stronger.

One honesty the record requires: the components are documented, the erosion and the bias each on their own, while the composite, that an eroded supervisor catches agent errors at a degraded rate in agentic settings specifically, has not been directly tested anywhere, and the absence of that test is itself a measure of how young this field’s safety evidence is.

This is deference, and it is the root the other human-system failures grow from, and it is worth seeing that it has no single owner, because that absence is the whole point. Whether the human defers well, accepts the agent when it is right and resists it when it is wrong, depends on four things owned by four people. It depends on whether the approval moment shows the human which case this is, which is the designer’s, the decision-package experience from the last chapter. It depends on whether the confidence signal the human reads is calibrated or is the agent’s own untrustworthy certainty, which is the eval owner’s. It depends on whether the human still has the domain judgment to push back at all, which is the eroding skill this chapter is about. And it depends on whether the product manager built any friction into the moment of acceptance or designed a frictionless click that deference flows straight through. Deference looks like a human weakness, a thing the reviewer should just resist, and the entire research record says it is not resistible by instruction, which means it is not the reviewer’s to fix. It is a property of the system around the reviewer, and the system is the team’s.

The Loop Test, and the three different people who fix each leg

The way to tell a real loop from a decorative one is to ask whether the human in it has three things, and the sharpest move this chapter makes is that the three things are owned by three different people, so a product manager who runs the test and finds a failure cannot fix it alone.

The first thing is time. Can the human intervene before the action lands. When the agent acts faster than a human can review, no amount of funding or training rescues the per-decision loop, because the constraint is the clock, and a reviewer who needs ten seconds to judge a case the agent commits in fifty milliseconds is decorative at any staffing level. The fix for a time failure is not a better reviewer; it is a different control, a hard limit the agent cannot cross, a class of action it may not take alone, a pre-commitment rather than a real-time veto. That fix is architectural. It belongs to the architect, who builds the gate that holds the action until the human can reach it, or builds the constraint that means the action never runs autonomously at all. A product manager can specify that the time problem must be solved; they cannot solve it, because the solution lives in the plumbing.

The second thing is skill. Does the human still have the competence the review requires. This is the leg the rest of the chapter is about, and its fix is the one that lands furthest from the product manager. Keeping a human sharp enough to catch the agent’s errors means keeping them practicing the underlying judgment on some cadence even when the agent could do it, the way airlines mandate manual flying hours not because the autopilot is bad but because the pilot’s skill is perishable. Mandating that practice, protecting the time for it against the throughput pressure that argues it away, is a people-management act, and a product manager has no authority to do it. It is the engineering manager’s, the person who owns the humans’ development and can put practice hours on a calendar and defend them. The product manager can name that the skill leg is failing. Only the manager of those people can fix it.

The third thing is attention. Is the human truly reviewing at the volume and the cadence the work arrives, or have they slid into the rubber-stamp. This is the leg the supervision paradox attacks directly, the vigilance that erodes with every success, and its fix is shared: the product manager owns the volume and the interface, not gating so much that the human drowns, and the supervisor owns watching the attention itself, reading override frequency over time as the early signal that the reviewer has stopped reading. A product manager who runs the Loop Test on their own diagram, as the prior book correctly told them to, will find which leg failed, and then discover that the fix for time is the architect’s, the fix for skill is the engineering manager’s, and the fix for attention is split between themselves and a supervisor most teams have not staffed. The test is the product manager’s to run. None of the three repairs is theirs to make alone.

Skill erosion, and the fix a product manager cannot perform

The skill leg deserves its own treatment, because it is where the prior framing put the most on the product manager that the product manager cannot carry, and correcting it is the heart of this chapter. The agent does not only change what the team produces. It changes what the team can still do, and it does so in three erosions on three timelines. The experts who hand a skill to the agent stop practicing it and lose it, slowly, the way any unused skill goes, which is deskilling. The juniors who never had to build the skill because the agent was always there never acquire it, which is never-skilling. And under time pressure, even capable people stop checking the agent’s work and simply accept it, which is the cognitive surrender that is deference hardened into habit. The slowest and most serious version is structural: remove the bottom rung of the ladder, the formative work juniors learned on, and the ladder stands for a while because the people on it climbed before it was gone, and then in a decade the seniors retire and the people meant to replace them spent their formative years managing an agent that did the formative work, and they have the title and the seniority and not the judgment, and they are now the humans in the loop, supervising agents they cannot evaluate because they never built the baseline the evaluation requires.

Here is the sentence the prior book closed this argument with, and it is worth quoting exactly because the re-vantaging turns on it: none of the three erosions is your fault, and all three are your responsibility, because you are designing the conditions that decide which way each goes, and on this team that is you. Written from the product manager’s seat, it is a noble claim and an impossible one. For deskilling, mandate that the experts keep practicing the underlying skill on a cadence, the airline’s manual-hours rule. A product manager cannot mandate a senior engineer’s practice hours; that is the engineering manager’s to require and defend. For never-skilling, protect the juniors’ formative struggle, let them do the hard version before they reach for the agent. A product manager does not design the apprenticeship of engineers or the residency of clinicians; that is the engineering manager’s and the institution’s curriculum. For the pipeline collapse, keep hiring and training the next generation even though the agent makes them look unnecessary this quarter. That is an organizational-leadership decision about workforce, above the product manager entirely, the kind of bet a company makes against its own short-term numbers, and no product manager has the standing to make it.

So the honest version of the close is not on this team that is you. It is on this team that is these people, and the product manager is one of them. The product manager owns the part that is truly theirs: deciding what the agent absorbs and what the humans keep doing, which is a product decision about the division of labor between the agent and the people, and it is real and it is upstream of everything else. But keeping the experts sharp is the engineering manager’s. Protecting the juniors’ formative years is the engineering manager’s and the institution’s. Sustaining the pipeline against the quarterly temptation to stop hiring is leadership’s.

Seven hands, and the person at the end of them

The human system divides across more hands than any other chapter in this part, and the division is the point, because the prior books collapsed it into one.

The product manager owns the division of labor between the agent and the humans, what the agent absorbs and what the people keep doing, and the friction in the moment of acceptance that gives deference something to catch on. The designer owns whether the approval moment shows the human which case they are in. The eval owner owns the calibrated confidence signal that lets the human trust well. The architect owns the time leg, the gate or the hard limit that means the human can reach the action before it commits, or that the irreversible action never runs alone. The engineering manager owns the skill leg, mandating and defending the practice that keeps the experts sharp and protecting the formative work that lets juniors become experts at all. The agent supervisor owns watching the attention leg, reading the erosion in the override numbers before it becomes the incident. And organizational leadership owns the pipeline, the decision to keep building the next generation of judgment even when the agent makes that look like a cost with no return this quarter. Seven hands, and the failure mode is the one the prior books modeled by accident: hand the whole human system to the product manager, who can move exactly one of the seven levers and watches the other six stay still.

There is a binding discipline, and it belongs to whoever leads the team rather than to any single craft: someone has to treat the human’s ability to supervise as a variable that is designed and maintained, not a constant that is assumed. A human reviews it is not a specification. It becomes one only when the team has said which human, whether that human still has the skill, whether they have the time, whether their attention is being watched for erosion, and who is responsible for each of those when the answer starts to slip. The slide says human review. The team that has done this work can say which human, how sharp, how much time, watched by whom, and the team that has not has drawn a box that means nothing and called it oversight.

There is one party even this accounting leaves uncovered, and it is worth naming because it is where the next question lives. The seven hands divide the team’s responsibility for the human system, but they are all answers about how the team manages the supervisor. None of them is the supervisor’s own answer about themselves: the practitioner whose judgment the gate routes through, whose skill is the thing being eroded, whose career runs along the same autonomy ladder the agent climbs, and who has a stake in not being deskilled that no one on the org chart is obviously assigned to protect. The team can decide which human reviews and whether that human is sharp; it is a harder and less settled question what that human should be doing to stay an expert in a craft the agent is quietly hollowing out, and whose job it is to look after the individual rather than the seat. That question is real and this book does not answer it; it belongs to a book about the person, not the team, and naming it here is how this one marks its edge.

It is worth holding, at the end of a chapter that has divided a person into legs and levers and owners, the person who is actually at the end of all of it. Every other failure in this part costs money, or trust, or a customer. This one costs a career that never happened. Picture the engineer who is twenty-six right now, who joined after the agent did, who has shipped real features and been praised for them and has never once written the hard part herself, because the hard part was always already done by the time she opened the file. She is good at her job as her job is currently defined. In fifteen years she will be the senior the team turns to when the agent does something subtle and wrong, and she will look at it with the same uncertainty as everyone else in the room, because the judgment that was supposed to accrue in those fifteen years accrued in the agent instead. No incident will mark the day this happened. There will be no postmortem, because nothing broke; the work shipped the whole time. The cost is invisible precisely because it is a thing that did not occur, a depth that did not form, and it is the one erosion in this chapter that cannot be reversed once it has run, because you cannot give someone back the decade in which they would have become expert. The other failures in this part are about an agent the team did not watch closely enough. This one is about the people who will be doing the watching after the people who know how are gone, and whether anyone with the authority to protect their formative years decided to spend this quarter’s margin doing it. That decision is being made right now, on every team running an agent, mostly by default, mostly by no one, and it is the quietest and most consequential thing this book has to say.