AI incident response plan for product teams

AI incidents rarely announce themselves with an error spike. Anthropic's postmortem of three recent issues describes a bug that started out "affecting approximately 0.8% of requests made to Sonnet 4" and later reached 16% of Sonnet 4 requests at the worst impacted hour, while the company's first signal was user complaints that "were difficult to distinguish from normal variation in user feedback." A team with far fewer users and far less tooling than a model provider should expect detection to be harder, not easier.

That is the case for writing the plan before launch. The AI feature launch checklist gets a feature out the door with a kill switch, traces and a fallback. This page covers what happens after one of those gates is tested for real: the definition, the severity call, containment, communication, any regulatory clock, and the review.

What counts as an AI incident for a product team?

An AI incident is harm that an AI system caused or contributed to, and the definition worth adopting is the OECD's, because regulators and incident trackers already use it. The OECD.AI incidents monitor methodology states that "An AI incident is an event, circumstance or series of events where the development, use or malfunction of one or more AI systems directly or indirectly leads to any of the following harms" and lists harm to health, disruption of critical infrastructure, rights violations, and harm to property, communities or the environment. The same page calls an event that "could plausibly lead to an AI incident" an AI hazard.

Most product teams need one more tier below those two: a quality incident, where the feature is wrong often enough or visibly enough that users lose trust before anyone is harmed. A support assistant inventing policy, a summariser dropping the key number, or a guardrail that a user bypassed and shared publicly all belong here. Writing the three tiers down ahead of time stops the first hour of every incident from turning into an argument about whether it is one.

How should you set severity for an AI incident?

Severity for an AI incident should follow harm and reach, not how embarrassing the output looks in a screenshot. The table below is a starting scale; adjust the examples to your feature, but keep the decision rule in the last column.

Severity	What it looks like	Default containment	Who decides
SEV 1: harm	Output that hurt someone, exposed personal data, or took an action the user did not approve	Kill switch on the AI path immediately, human fallback for all users	Incident commander, with legal looped in the same day
SEV 2: hazard	A reproducible failure that could cause harm, such as a working prompt injection or a wrong answer about money, health or eligibility	Route the affected intent to human review or block the input pattern	Incident commander and PM
SEV 3: quality	A pattern of wrong, off-tone or degraded answers with no harm found	Roll back the last prompt or model change, keep the feature on	PM, recorded in the incident log

Two rules keep the scale honest. Raise severity when you cannot yet tell how many users saw the failure, since AI faults are often partial and sticky to some users. Lower it only with evidence from traces, never on the strength of one engineer failing to reproduce the problem.

Who owns what during an AI incident?

An AI incident needs the same three separated roles as any production incident: someone in command, someone changing the system, and someone talking to people. The Google SRE book chapter on managing incidents says "The incident commander holds the high-level state about the incident" and that "The operations team should be the only group modifying the system during an incident." It describes the communications owner as "the public face of the incident response task force."

The AI twist is who fills those seats. A model failure often has no single owning engineer: the prompt belongs to product, the retrieval index to data, the model to a vendor. Name the incident commander per severity level in the plan, and give the PM a defined seat, usually communications for SEV 3 and SEV 2, because explaining a wrong answer to users is a product judgment more than an operational one.

How do you contain an AI incident without a deploy?

You contain an AI incident by flipping a runtime control that was built and tested before launch: a kill switch on the AI path, a route to human review, an input block, or a rollback to the last known good prompt and model version. The NIST AI RMF Core, in its Manage function, asks that "Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use." The words that matter for a runbook are "responsibilities are assigned": a switch nobody is authorised to flip outside office hours is not a mechanism.

Snapshot before you change anything. Save the active prompt, model and version, retrieval configuration, guardrail settings and a sample of failing traces, because a rollback erases the state you need for the review. Providers of high-risk systems under the EU AI Act have a further reason: Article 73 says that, after reporting a serious incident, the provider "shall not perform any investigation which involves altering the AI system" concerned "in a way which may affect any subsequent evaluation of the causes of the incident, prior to informing the competent authorities of such action." The rule covers investigations that change the system in ways that could affect the later evaluation of causes, not every change; how it applies to a specific emergency rollback is a question for counsel, not for this page. Setting up the switch itself is covered in the launch checklist, and the control types are in the AI guardrails glossary entry.

Why are AI incidents harder to detect and reproduce than ordinary bugs?

AI incidents are harder to detect because the failure is probabilistic and partial, so dashboards built for crashes and error rates stay green. In the Anthropic postmortem, the team wrote that "The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes." Of the compiler bug, the same postmortem says "The same prompt might work perfectly on one request and fail on the next."

Three detection channels follow from that. First, user reports with a one-click path, routed to whoever is on call rather than to a weekly feedback review. Second, evals run continuously against production traffic, which the postmortem lists among its changes: "we will run them continuously on true production systems" to catch problems like the routing error. Third, a written rule, borrowed from the same postmortem, for linking a spike in reports to the most recent prompt, model or infrastructure change, since it admits "we didn't immediately make the connection to an otherwise standard load balancing change."

Who talks to customers, and what should they say?

Customers hear from one named communications owner, with a first message that states what the feature did wrong, who was affected, what has been switched off, and when the next update comes. Avoid model-speak. A line saying the assistant gave incorrect refund terms to some customers between Tuesday and Thursday, and now hands refund questions to a person, is a message a user can act on. A line saying the team is investigating anomalous outputs is not.

The Anthropic postmortem is a useful model for the final write-up because it answers the suspicion users actually had, in one line: "To state it plainly: We never reduce model quality due to demand, time of day, or server load." It also owns the miss without hedging: "In these recent incidents, we didn't meet that bar." Find the question your users are really asking during your incident, and answer it in the first paragraph.

When does the EU AI Act require you to report an AI incident?

The EU AI Act requires reporting when a provider's high-risk AI system is involved in a serious incident, under Article 73, which says providers "shall report any serious incident" to the market surveillance authorities of the Member States where it occurred.

Article 3 defines a serious incident as an incident or malfunctioning of an AI system that directly or indirectly leads to death or serious harm to health, serious and irreversible disruption of critical infrastructure, infringement of obligations protecting fundamental rights, or serious harm to property or the environment.

Situation under Article 73	Reporting deadline in the text	Source
Any serious incident (general rule)	Immediately after a causal link or its reasonable likelihood is established, and "not later than 15 days after the provider" or deployer becomes aware	Article 73(2)
Widespread infringement, or serious and irreversible disruption of critical infrastructure	Immediately, and "not later than two days after the provider" or deployer becomes aware	Article 73(3)
Death of a person	Immediately, but "not later than 10 days after the date on which" the provider or deployer becomes aware	Article 73(4)

The deadlines run from awareness, which is why the plan should record the time the first credible report arrived. The Article 73 text also allows a provider to "submit an initial report that is incomplete, followed by a complete report", so waiting for the full root cause is not a reason to miss the clock. Article 73 applies only to high-risk AI systems, so check whether your feature is classified as high-risk; which obligations apply when is covered in the EU AI Act for product managers. Nothing here is legal advice; put a check of the reporting duty with legal on the SEV 1 checklist and let counsel decide.

What does a worked hallucination incident look like?

A hallucination incident follows the same five steps as any other, and an illustrative example makes the order clear. Picture a support assistant that answers billing questions from your help centre. On a Monday, two support agents flag tickets where customers quote a refund window that does not exist in your policy.

1. Declare and classify. The on-call PM declares an incident. Customers are acting on a wrong answer about money, so it starts at SEV 2, with a note to raise it to SEV 1 if anyone was charged or denied a refund because of the answer.
2. Snapshot and contain. Engineering saves the active prompt, model version and a sample of traces mentioning refunds, then routes every refund intent to a human agent behind the existing flag. The rest of the assistant stays on.
3. Size it. Traces show which answers mentioned the invented window, and support pulls the affected tickets. The team finds the help centre article was rewritten the previous week, and the retrieval index still holds the old and new versions side by side.
4. Communicate. Support contacts each affected customer with the correct policy. The PM posts an internal update and a short public note on the help centre.
5. Review and feed the evals. The review adds every failing refund question to the eval set, adds a check that retrieval never returns two versions of the same policy, and only then turns the refund intent back on.

The step teams skip is the fifth. Without the new eval cases, the next index refresh can bring the same failure back and nobody will know until support notices again. The AI hallucination glossary entry covers why models fill gaps like this.

How should the post-incident review feed the eval set?

The post-incident review should end with new eval cases and a named owner for each action, not only a document. The Google SRE book chapter on postmortem culture defines the artifact as "a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring" and adds that a blameless review "must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior."

For an AI feature, the follow-up actions have a specific home. Each failing input goes into the regression set described in how to write evals for AI products, and the NIST AI RMF Core asks that "Processes for tracking, responding to, and recovering from incidents and errors are followed and documented." If you use AI to draft the review itself, the guide to AI for incident reviews covers where a model helps and where it quietly adds blame.

Is a separate AI incident plan overkill if you already have an incident process?

A separate AI incident plan is overkill if it duplicates roles, paging and postmortem templates you already run, and that objection deserves a straight answer: keep your existing process. Engineering on-call, severity paging and blameless reviews work the same way for a model as for a database.

What the existing process usually lacks is four things: a definition of a quality incident that nobody would page for today, a runtime switch for the AI path specifically, a rule that every incident adds eval cases, and a reporting clock if you provide a high-risk system. Add those as an appendix to the runbook you have. If the appendix grows past a page, it is duplicating something.

Where do you practise handling AI risk before it happens in production?

Builders Camp's AI Product Management bootcamp, led by its own programme director, a working practitioner, runs for 2 weeks with 4 live sessions and 7 self-paced microlessons. Its stated focus is to design AI-native experiences, evaluate model behavior, manage risk, and deliver AI features users trust. The bootcamp is open to product managers at all levels, its practical challenge is rated advanced, and it is part of the AI Product Expert Track. For the agent side of the same problem, where a model can take actions rather than only answer, the AI Agents bootcamp covers defining tools and guardrails.

See the AI Product Management bootcamp