Six published lists of AI product manager interview questions were analyzed for this page, 145 of them transcribed verbatim and tagged one by one. Half of the corpus asks how the model works and how to make the agent behave. Almost nothing asks how the system around it is supervised, and not one question in 145 asks who owns a wrong outcome.
145
questions analyzed, from six published sources
5.3 : 1
questions about the agent's behavior, versus the system supervising it
82%
test the job up to launch. Seven percent test it after
0
ask who owns a wrong outcome
That gap is not an oversight by the people writing these lists. It reflects a real belief about what the job is. The answers below are written from the other belief: that specifying what an agent may decide alone, where it stops, and who carries it when it is wrong is the product work, and the rest is the entry fee.
A note on what this is not. It is not an argument that the technical questions are illegitimate. A product manager who cannot say what a context window is will set a bad boundary, which is why this book has a literacy chapter. The problem is proportion. Literacy is one chapter of fifteen here, and roughly half of what the market asks.
Method. Three sources published every question, so those were counted: a widely shared technical cheat sheet, Product Leadership's list of 25, and IdeaPlan's 30. Three more publish categories and rubrics rather than full question sets and were used as corroboration: Crosschq, KORE1, and LockedIn AI. Every question was tagged by hand to one phase of the lifecycle. Analysis run July 2026.
How to use it. Each answer names what the question is usually testing, then gives the answer from the product seat, then links to the chapter it comes from. Every chapter of Agentic AI for Busy Product Managers is free to read online.
Phase 1
AI Literacy
The market spends about a quarter of its questions here. The book spends one chapter of fifteen. Literacy is real and you cannot set a boundary without it, but it is the entry fee, not the job.
How does an LLM actually work? What is next-token prediction? #
What it is testing Whether you can recite the mechanism.
It predicts the next token from a distribution, one token at a time, conditioned on everything in the window. The product consequence is the part worth saying out loud: there is no retrieval of a stored fact and no lookup of a correct answer. The system produces a plausible continuation. Everything downstream follows from that. It is why the same input can give you two answers, why confidence and correctness are not the same signal, and why a boundary you write in the prompt is a suggestion rather than a rule.
What is a context window, and what happens when you exceed it? #
What it is testing Whether you know the number.
The window is the working memory for one call. Exceed it and something is dropped, and the interesting question is what. Most teams cannot say. If the boundary rules, the policy, and the customer history all live in the window, and the window is trimmed under load, then the thing that got dropped was a governance control. That is not a context problem. It is a design problem: anything the agent must not do belongs somewhere the trimming cannot reach.
What are tokens, and why should a product manager care? #
What it is testing Whether you know the unit.
It is the unit of both cost and memory, which makes it the one piece of model trivia that belongs in a business case. Two consequences. Your unit economics are denominated in it, so a task that re-enters the model six times costs six times what the demo implied, and almost every task does. And the context budget is finite, so the instructions, the retrieved documents, and the policy all compete for the same space. Deciding what wins that competition is a product decision nobody else will make for you.
Why do models hallucinate, and what is your move as a PM? #
What it is testing A definition, then a shrug.
The definition is easy: the system is generating plausible continuations, and a fluent wrong answer costs it nothing. The PM move is not to eliminate hallucination, which you cannot do. It is to decide which outputs need a source attached before anyone acts on them, which need a human in front of them, and which are cheap enough to be wrong. That is a consequence classification, and it is a product decision, not a model decision.
Does a longer, sterner prompt fix hallucination? Why not? #
What it is testing Whether you have tried it.
No, and the reason matters more than the answer. A prompt is context the model weighs, not a rule it obeys. Under an unusual input, a long instruction competes with everything else in the window and can lose. This is the same reason an autonomy boundary written into a system prompt is not a boundary. If the agent must never issue a refund above a threshold, that limit belongs in the execution path, where it is enforced rather than considered.
Why do two identical prompts give different answers? #
What it is testing Sampling and temperature.
Sampling, and that is the honest short answer. The longer one is the reason this question belongs in a PM interview at all: it means single-pass testing proves nothing. One green run tells you the system can pass, not that it does. Everything about how you evaluate, and every promise you make to a stakeholder about quality, has to be stated as a distribution over many runs rather than a result.
What makes something an agent rather than a chatbot? #
What it is testing A taxonomy.
Not autonomy, and not sophistication. The line is whether the system changes state. A chatbot produces text a person reads and acts on; the person is the last step. An agent takes the step itself. That single difference is what moves supervision from a review habit to a product you have to build, because once the system touches state, the question is no longer whether the output was good. It is whether the action was reversible and who is named when it was not.
What it is testing Whether you follow the ecosystem.
It standardizes how a model reaches tools and data so every integration is not written twice. Useful. The PM point is what it does not solve: a standard connection protocol says nothing about which tools this agent may reach, under whose authority, or what gets logged when it does. The tool list is the agent's real authority, and MCP makes it easier to extend that list without anyone deciding they had extended it.
Explain how transformers work. What is attention? #
What it is testing The depth filter. Interviewers use this one to end conversations.
Answer it, briefly and correctly, and then notice what it is doing. Attention lets the model weigh every token against every other, which is why context length is expensive and why what you put in the window matters more than how you phrase it. Beyond that, a PM who can explain attention and cannot say who owns a wrong refund has answered the easy question. Both are fair to ask. Only one of them is the job.
What it is testing Whether you will admit there is such a case.
When the decision is deterministic, when the ground truth is tabular, when the consequence of being wrong is not recoverable, or when the volume is too low to repay the supervision the boundary will require. Four conditions have to hold before an agent is the right shape: the decision repeats at real volume, you can score past decisions as right or wrong without argument, you can enumerate every system it touches, and the worst case at the expected error rate is survivable. If one fails, do more scoping.
LLM or classical ML for churn prediction? Why XGBoost over an LLM for tabular data? #
What it is testing Whether you reach for the fashionable tool.
Gradient boosting, for a tabular prediction with a labelled history, almost every time. It is cheaper per call, deterministic, simple to monitor, and explainable to a regulator. The real answer is that the question is upstream of the model: what decision does this prediction change, who acts on it, and what happens when it is wrong. Answer those and the model choice is usually obvious and usually boring.
Build versus buy versus fine-tune. Walk me through the call. #
What it is testing A framework.
Fine-tuning buys you behavior, not knowledge, and it buys you a maintenance obligation every time the base model moves. Buying buys you speed and a dependency you do not control, which makes the vendor's model release schedule part of your risk register. Building buys you control at a cost most teams understate by more than half. The question that resolves it is not capability. It is which of the three leaves you able to answer, eighteen months from now, why the system did what it did.
Estimate the inference cost for ten million users. #
What it is testing Arithmetic.
Do the arithmetic, then correct it. Token cost per call is the smallest term. Count how many times a single task re-enters the model, because a task that looks like one call is usually six. Then add the architecture multiplier, orchestration, integrations, the audit pipeline, eval maintenance, and the supervision the boundary requires, because a human reviewing the escalated minority is a line item. Take the number in the business case and multiply by 1.5 before you show it to finance.
Your CEO wants AI in every feature. How do you push back? #
What it is testing Whether you can manage up.
Not with a lecture on limitations. With a cost model and a consequence classification. Show two features side by side: one where the decision repeats ten thousand times a month and a wrong answer costs a re-run, and one where it repeats twelve times and a wrong answer costs a customer. The first is a candidate. The second is a slide. The argument that works is arithmetic, and it is much harder to overrule than an opinion about readiness.
How do you decide which features should use AI and which should not? #
What it is testing Prioritization instinct.
Run the four suitability tests on each, and then ask the question that is missing from most frameworks: what does the supervision cost. A feature that passes on volume and measurability but requires a trained human to review every escalated case has a break-even that includes that person's time. Features do not fail this test because the model is not good enough. They fail because the loaded cost, counted in full, never clears the alternative.
What would you do with a model that has ten times the capability at ten times the cost? #
What it is testing Judgment on a trade-off with no right answer.
Route it. Almost no product needs its best model on every call, and paying flagship rates for classification is how unit economics quietly break at volume. Send the hard minority to the expensive model and the rest to a cheap one, and make the router's decision a logged event, because when quality drops you will want to know which model answered. The interesting part of this question is not the model. It is that routing is now a product decision with a cost curve.
Describe a time you decided not to use AI for a feature. #
What it is testing Whether you have ever said no.
A real answer here names the condition that failed, not a general reservation about readiness. The strongest version is a case where the model worked and you still said no, because the consequence class was wrong, or because the supervision the boundary required did not exist and nobody was going to fund it. That is the answer that separates someone who has shipped an agent from someone who has read about one.
How do you think about competitive moats in AI products? #
What it is testing Strategy instinct. Usually answered with proprietary data.
Everyone rents the same model, so the model is not the moat, and proprietary data by itself is not either, because a pile of records nobody has given meaning to is not something an agent can reason over correctly. What is defensible is the semantic layer: the definitions, the business rules, and the context that travel with the data. Then add the supervisory layer. A competitor can copy your feature in a quarter. Reproducing the boundary, the audit surface, and the people trained to operate them takes a good deal longer.
The canon asks 37 questions about designing what the agent does and 7 about designing the system that supervises it. Five to one. This section is the half that goes missing.
Design an AI feature that summarizes unread messages. #
What it is testing Whether you can spec a pipeline.
Spec the pipeline, then keep going, because the pipeline is the easy half. What may this feature do without asking: read only, or also mark as read, also archive, also reply. What does the person see at the moment they accept a summary, and can they get to the original. What is the record if the summary dropped the one message that mattered. A summarizer that only reads is a low-consequence product. The version that acts is a different product with the same demo.
RAG versus fine-tuning versus prompting. When do you pick each? #
What it is testing A decision tree.
Prompting for behavior you will change weekly, RAG for knowledge that changes faster than you can retrain, fine-tuning for a form or a style you need consistently and cheaply. The part the decision tree leaves out: RAG makes your retrieval layer part of your attack surface and part of your audit surface at the same time. If you cannot say which documents the agent read to produce a given answer, you have not finished the design, whatever the retrieval metrics say.
What it is testing Vector stores and summarization strategies.
Answer the mechanics, then separate two things the question conflates. Memory is what the agent recalls and may weigh. Policy is what the agent may do. Putting a limit into a memory file makes it a preference the model balances against everything else in the window. Memory that drifts is inconvenient. Policy that drifts is an incident. Decide which of your files are which, and enforce the second category outside the model.
Where do you draw the safety redlines? How would you define a redline? #
What it is testing Values. This is one of the two good supervisory questions in circulation.
A redline is not a strong preference. It is a limit the execution path enforces, which the agent cannot be argued out of by an unusual input or a persuasive user, and which fails closed. The test is mechanical: if the only thing standing between the agent and the action is a sentence in a prompt, you have a guideline. Draw the line at the actions whose consequences are not recoverable, and write it where the code, not the model, decides.
How do you design for failure, with bad tool calls and timeouts? #
What it is testing Engineering resilience.
Retries and circuit breakers are the engineering answer and they are correct. The product answer is the recovery workflow: what compensating action exists when the agent was wrong and the call succeeded. Is rollback reachable mid-execution, not only after completion. What cannot be undone, and is the person told that before the action rather than after. For actions whose consequences move faster than any alert cycle, the only real control is a kill switch upstream of execution.
What is human-in-the-loop, and when should a product use it? #
What it is testing A definition. It usually gets one.
The definition is the least useful part. HITL is a design commitment that someone competent is positioned to catch the error, and it decays quietly in production: a person approving forty of these an hour is not reviewing, they are clicking. Two separate things get called HITL. Transaction-level review, a human on each action, is expensive and does not scale. Program-level review, a sampled audit with a named owner, does. Say which one you mean and price it.
How would you design a human-in-the-loop feedback system? #
What it is testing An interface. The best question in the whole circulating set.
Design the approval moment as a product surface. What the person sees is a decision package, not a confirmation dialog: what the agent proposes, what it relied on, what the alternatives were, and what happens if nobody decides inside the window. Frame it as authorization within a designed boundary rather than validation of the agent's reasoning, because the second framing does not survive the predictable erosion of the reviewer's competence. Then instrument override frequency, because that number is your early warning.
Not by surfacing a confidence number, which most users read as an accuracy guarantee and which is usually poorly calibrated anyway. By deciding, before launch, what the system does when it is unsure: escalate, ask, degrade to a narrower answer, or refuse. Ambiguity is not an edge case to handle in the UI. It is a behavior to specify, with a named destination for each case, in the brief the engineers build from.
What is your framework for deciding whether a feature is too risky to ship? #
What it is testing Whether you have a framework.
Classify by consequence, not by proximity to the customer. For each action type: is it reversible, at what cost, on what timescale, and who is named to carry it when it goes wrong. Then a gate with three non-negotiables before build, not before launch. The boundary is enforced in the execution path rather than the prompt. The decision record can be reconstructed later. The kill switch is reachable and has been rehearsed. A spec that violates one does not proceed, however good the demo looks.
Twelve percent of the questions, and the one area where the market's instincts are close to right. The gap here is not coverage. It is that almost every question assumes a single pass.
How would you evaluate this feature before launch? #
What it is testing Whether you know what an eval set is.
Curate the cases, including the ones the old workflow handled badly and the failures that must never ship. Then run them many times, not once, and report a pass rate with its variance rather than a result. Report the worst slice alongside the median, because the average hides the population you are about to hurt. And validate state, not just text: an agent that says it issued the refund and did not has passed a semantic check and failed the only one that matters.
Whiteboard an evaluation framework from scratch. #
What it is testing Structure under pressure. Asked at Meta, among others.
Five parts. The curated set with the endorsed outcome for each case. The distribution: how many runs, what pass rate, what variance, what floor. State validation separate from output validation. A coverage statement that names which intents, failure modes, and adversarial inputs were tested and which gaps you deliberately deferred. And the sign-off: who accepts the residual risk, by name. The last one is the part most whiteboard answers leave out and the part that makes the rest binding.
What it is testing Whether you understand the core problem.
You stop asking whether it passed and start asking how often. Pass@K over many runs, with the variance reported, and a stated floor rather than an average. Then project the compound: a five-step workflow at ninety-five percent per step is not a ninety-five percent workflow. Most agentic quality surprises are this arithmetic arriving late.
A model scoring another model's output, which is cheap and scales and is the only practical option at volume. Trust it after you have characterized its false-pass rate against human labels, and account for the documented biases: it prefers longer answers, it is sensitive to position, and it is easier on models from its own family. An uncalibrated judge is not a measurement. It is a second opinion from something with the same blind spots.
How do you set accuracy targets when you have no baseline? #
What it is testing Whether you invent a number.
You do have a baseline. It is the current process, and it is worse than everyone remembers. Sample what humans actually do today, score it with the same rubric you will use on the agent, and you have the comparison the business case needs. Setting a target without that number is how teams end up defending ninety-two percent against an imagined ninety-nine, when the humans were at eighty-four.
Offline evals passed but users are unhappy. What now? #
What it is testing Debugging instinct.
Assume the coverage statement is where the answer is, and go read it. The eval suite tested the distribution you imagined; production sent a different one. Look for the inputs the eval assumed were clean, the intents nobody wrote a case for, and the slice you averaged away. This is also the moment to check whether users are unhappy about the output or about the experience around it, because a correct answer delivered at the wrong moment is a design failure that no eval was ever going to catch.
Explain precision, recall, and F1 to a stakeholder. #
What it is testing Whether you can translate.
Do it in their currency, not in statistics. Precision is how often we are right when we act. Recall is how much of the real thing we caught. F1 balances them and hides which one you chose to sacrifice. Then ask the question the metrics exist to serve: in this workflow, which error costs more, acting wrongly or missing something. That answer sets the threshold, and it belongs to the domain expert, not to the model.
How do you catch regressions after a model swap? #
What it is testing Regression testing.
Re-run the golden set and compare distributions rather than headline scores, because a swap that improves the median and degrades the tail reads as an upgrade and behaves as an incident. Then note the harder problem: your vendor can change the model underneath you without a version number. Deployment events have to be logged as first-class product events, or your quality signal has no independent variable and you are debugging in the dark.
How do you evaluate faithfulness versus retrieval quality? #
What it is testing Whether you know these are two different failures.
They fail independently and they need separate numbers. Retrieval quality asks whether the right documents came back. Faithfulness asks whether the answer stayed inside what came back. A system can retrieve perfectly and then assert something the documents never said, and a single blended quality score hides which half broke. Measure them apart, then add the third that most teams skip: whether the retrieved source was still current. An answer faithful to a superseded policy is wrong in the way that reaches a regulator.
Seven percent of the circulating questions cover everything that happens after launch. Eighty-two percent cover everything before it. An interview shaped this way selects for someone who can get an agent shipped, not for anyone who can still tell, two years later, whether it is doing the right thing.
What it is testing Whether you will say logs, traces, and metrics.
Those are the substrate, not the answer. Six things need an instrument: task success on the real distribution, unintended actions, override frequency, confidence calibration, time to roll back, and time to recover from an incident. Notice that two of them measure the human rather than the model. Override frequency drifting toward zero is the single most useful number you will have, because it is how you find out that supervision became clicking.
Whether anything changed underneath you, before you touch your own code. The model version, the retrieval corpus, the upstream data, the prompt someone edited on Friday. Most agentic quality incidents are not regressions in your system; they are changes in something your system depends on and does not control. If you cannot answer what version answered this request, that is the finding, and it outranks the quality drop.
What it is testing Whether you have a stable answer in a moving field.
The boundary, the record, the named owner, and the recovery path. Capability moves every quarter and the questions do not: what may this system do alone, how do we know it did, who carries it when it is wrong, and how do we undo it. Everything you build against a specific model's behavior is a depreciating asset. Everything you build against those four questions survives the swap.
How do you measure the success of an AI feature? #
What it is testing Metrics.
Start by naming the seductive wrong metric and excluding it, because for most agents it is share resolved without a human, and optimizing it means training the system to avoid escalating the cases that needed escalating. Success is the outcome the workflow existed to produce, measured on the full distribution, with the escalation rate tracked as a health signal rather than a cost. If escalations fall and outcomes do not improve, the agent got more confident, not better.
Satisfaction is 95 percent but NPS dropped after launch. What happened? #
What it is testing Analytical instinct.
Satisfaction is measured on the people who completed the flow. NPS includes the people who gave up, escalated, or quietly stopped trusting it. Look at the five percent, because in an agentic product the tail is not noise; it is the population whose case was exactly the kind the old workflow handled badly, which is the population the agent was built for. A green dashboard measuring the easy majority is the most common way this goes wrong.
How do you build a roadmap when capabilities evolve this fast? #
What it is testing Planning under uncertainty.
Separate the roadmap into what depends on model capability and what does not. The supervisory layer, the audit surface, the recovery path, and the accountability model do not get cheaper when the model gets better; they get more necessary, because better models are trusted with more consequential actions. Plan those on a normal horizon. Plan the capability-dependent work in short cycles and expect to throw some of it away.
You need the decision trajectory: what the agent was asked, what it retrieved, which tools it called with what arguments, and what came back. Not chain-of-thought, which is a narrative rather than a record. If that trajectory is not reconstructable for an arbitrary request from six months ago, you cannot debug the product and you also cannot answer a regulator or an affected customer. Debuggability and auditability are the same build, and teams that treat them separately do the work twice.
How would you design an A/B test for an AI feature? #
What it is testing Experiment design.
The mechanics are the same and one assumption is not. In an ordinary test the treatment holds still. Here the model underneath can change mid-experiment, without a version number and without telling you, which means the treatment arm is not one thing. Log the model version on every call and treat a vendor deployment as an event that ends the experiment. Then choose the metric with care, because the tempting one, share resolved without a human, improves when the agent stops escalating the cases it should have escalated.
Across 145 questions from six published sources, these got zero. Not one asked who owns a wrong outcome. Not one asked where the autonomy boundary sits, what the human sees at the approval moment, whether the record can be reconstructed, or what happens to the agent's knowledge when you turn it off. Ask these in your next interview, from either side of the table.
Who owns a wrong outcome, by name, and where does that name appear? #
Not the team, not the function. One person per action type. And the name belongs in the product interface, not only in the governance document, because a name in a policy is a name nobody sees at the moment it matters.
The autonomy boundary is the single most consequential product decision in an agentic system and it is usually made implicitly, by whoever wired the tool list. Ask a candidate to state it for a system they shipped. Most cannot, which is the finding.
A decision package, or a confirmation dialog. One of those is a supervisory product and the other is a speed bump that trains people to click. Ask what information is on the screen, what the alternatives were, and what happens when nobody responds inside the window.
Can you reconstruct why the agent did what it did, six months later? #
Observable actions and retrieved context, for an arbitrary past request. If the answer is that the logs roll off after thirty days, the system has no audit surface, and the incident that needs one has not happened yet.
What is the compensating action when the agent was wrong? #
Rollback reachable mid-execution, not only after completion. What cannot be undone at all, and whether the person is told that before the action rather than after.
Is the boundary enforced in the execution path, or written in the prompt? #
This is the one question that separates a governed agent from a well-intentioned one, and it takes ten seconds to ask. A limit the model weighs is a preference. A limit the code enforces is a boundary.
What causes a demotion, and who is named to act on it? #
Autonomy is earned, not scheduled. If a system moves up the ladder on a clean calendar rather than on demonstrated competence in the failure modes that matter, the autonomy was scheduled. And if the demotion thresholds are not defined before launch, they get defined under pressure after the first incident.
Is your kill switch reachable, and when did you last rehearse it? #
Every team says they have one. Far fewer have used it outside an incident. For actions whose consequences move faster than an alert-and-respond cycle, the kill switch is not a backstop; it is the only control that operates on the right timescale.
How will you know when your supervisor has stopped really supervising? #
Sustained use of a competent system erodes the competence of the person supervising it. That is the supervision paradox, and it means the human in the loop on day 400 is not the human you designed for on day one. Override frequency trending toward zero is the signal. Somebody has to be watching it.
What happens to the agent's knowledge when you decommission it? #
Retiring an agent is closer to an employee leaving than to deleting a service. The audit trail has to survive it, the tasks have to go somewhere, and the judgment encoded in its configuration is institutional knowledge that walks out of the building unless someone writes it down.
Who is trained to do this work without the agent, and how often do they practice? #
Aviation solved this with recurrent proficiency requirements decades ago. Software has not adopted it. If the answer is that nobody has done this manually in eighteen months, the recovery plan for an outage is a hope.
The eleven questions above are the cheapest diagnostic in this document. They take a minute each, they cannot be prepared for from a listicle, and the answers separate a candidate who has operated an agent in production from one who has read about it. A candidate who can explain attention and cannot tell you who owns a wrong refund has answered the easy question.