How to measure success of an AI feature: quality, outcome and cost metrics that have to agree

An AI feature has no single success number, and the teams that report one usually hide a failure behind it. Anthropic's own documentation on defining success criteria says it directly: "Most use cases need multidimensional evaluation along several success criteria." The same page lists latency and price next to task fidelity and tone as criteria to set before you build. GitHub's best-known Copilot result shows why the outcome layer matters as much as the quality layer: in a controlled experiment with 95 professional developers, Eirini Kalliamvakou reported on the GitHub Blog that developers using Copilot took on average 1 hour and 11 minutes to finish the task against 2 hours and 41 minutes for those without it, or 55% faster. One task in one language is not your product, but the experiment measured the outcome users cared about, not how often the model was called.

You need three layers of metrics, all defined before launch, and a rule for what you do when they disagree.

What are the three layers of AI feature metrics?

Every AI feature needs a quality layer, an outcome layer and a system layer, because each one answers a question the other two cannot.

Layer	Question it answers	Example metrics	How fast it moves
Quality proxy	Is the output good enough that users act on it?	Acceptance rate, edits before sending, regenerations, thumbs down, escalation to a human	Hours to days
Product outcome	Did the job the feature exists for get done better?	Tickets resolved, task completion time, retention of users who adopt the feature	Weeks
System health	What does each success cost to deliver?	p95 latency, error and timeout rate, cost per request, cost per successful outcome	Minutes

The quality proxy is your early warning, the outcome is the verdict, and the system layer decides whether the verdict is affordable. Google's People + AI Guidebook frames the team-level version of this: "Everyone on your team should feel aligned on what both success and failure look like for your feature, and how to alert the team if something goes wrong."

How is measuring an AI feature different from measuring any other feature?

The chain in how to measure feature success still holds for an AI feature: name the business outcome, the behaviour that has to change, the metric for that behaviour, the events that compute it, and a guardrail. Start there.

An AI feature breaks two assumptions that chain quietly relies on. A normal feature behaves the same way every time, so once it works in QA you only measure whether people use it. A model returns a different output for each input, so quality is a distribution you have to sample in production, not a property you verify once. And a normal feature costs roughly nothing per click, while every model call has a price, so higher usage can mean higher losses. The quality proxy and the system layer exist to cover those two gaps.

Where do offline evals stop and production metrics start?

Offline evals answer "is the output good on cases we chose?" and production metrics answer "did users and the business get what they needed?" The guide to writing evals for AI products covers the first question: a labeled set of real traces, a pass or fail rubric, and a re-run before every significant change. Treat that eval set as the launch gate.

Passing it does not make the feature a success. Your eval set is a sample you picked, and real users bring inputs you never thought of. The production metrics below measure what happens after the gate: whether people accept the output, whether the outcome moves, and what it costs. When a production metric surfaces a new failure, the failing traces go back into the eval set, which is how the two halves feed each other.

Which quality proxy metrics work in production?

The best quality proxy is a behaviour the user performs anyway that shows whether the output was good enough to use. Google's Feedback and Control chapter separates the two sources: "Implicit feedback is data about user behavior and interactions from your product logs." Explicit feedback, in the same chapter, is commentary users deliberately give on an output. Implicit signals arrive on every interaction; explicit signals such as a thumbs down are rarer, and the same chapter warns that they can lean toward extreme opinions.

Which implicit signal tracks what users actually feel was tested in a case study of GitHub Copilot users. The paper Productivity Assessment of Neural Code Completion, by Albert Ziegler and colleagues, compared Copilot users' self-reported productivity with usage data and found that "the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers' perception of productivity." Acceptance rate beat the more sophisticated metric. For your feature, pick the equivalent action:

• Suggestion features: acceptance rate, and how much the user edits the suggestion before keeping it.
• Generation features: regenerate clicks, copy or send events, and the share of drafts sent with no edit.
• Agent features: task completion without a takeover, and how often a person has to step in, which is where a human-in-the-loop review step doubles as a measurement point.

Read every proxy with one caveat in mind: the user accepting an output only shows it looked good enough at that moment. A wrong answer that sounds confident gets accepted too, which is why the outcome layer is not optional.

How do you tie an AI feature to a product outcome?

Pick the outcome the feature was funded to move, measure it against a group that did not get the feature, and ignore model metrics when you report it. The Copilot experiment is the template: one group with the tool, one without, the same task, and time to completion as the outcome. A holdout group or a staged rollout gives you the same comparison inside a live product.

Support automation shows the outcome layer and the quality layer merging. Intercom's help center counts a Fin resolution when, after Fin's last answer, the customer confirms the answer was satisfactory or exits without requesting further help, and the same help article puts the price at $0.99 per resolution. On its Fin product page, Intercom states that Fin is "averaging 76% across 12,000+ customers" on resolution rate. That figure is a vendor claim about its own product, measured with its own definition, so do not borrow it as a benchmark. What it does show is a company that chose a business outcome, a resolved conversation, as both its success metric and its billing unit.

If your product has a North Star, the AI feature's outcome metric should be one of its inputs. The North Star metric examples guide shows how to find the input a feature can actually move.

What system, cost and latency metrics belong on the dashboard?

Put latency, error rate and cost per successful outcome on the same dashboard as quality and outcome, with a ceiling for each set before launch. Anthropic's success-criteria page gives a worked example of a multidimensional target that includes "95% response time" under 200 milliseconds next to accuracy and toxicity thresholds, and it tells teams to consider "the cost for each API call, the size of the model, and the frequency of usage" when setting a price criterion.

Cost per request is the wrong unit on its own. A cheap call that produces an answer nobody uses still costs money and delivers nothing. Divide total model and infrastructure spend by the count of accepted suggestions, resolved conversations or completed tasks, and track that number weekly. A prompt change that raises acceptance but doubles tokens per request can make each success more expensive, and only this ratio shows it. If you price the feature by outcome, the AI agent pricing guide explains why this ratio decides your margin.

What happens when the proxy metric and the outcome disagree?

When the layers disagree, trust the outcome and investigate the proxy, because the proxy is the layer most likely to be measuring the wrong thing. The strongest objection to everything above is that proxies are gameable: a resolution that counts a customer who left without asking for more help also counts a customer who gave up. Intercom's definition handles part of this by deducting a resolution if the customer returns, which is a good pattern to copy. Define a reopen or retry window for your own proxy, and subtract the cases that came back.

The other common disagreement runs the other way: quality looks fine, the outcome is flat. That usually means the feature works but sits in the wrong place in the workflow, so users accept its output and then do the task the old way anyway. Check the event sequence after acceptance before you retrain anything. And when a quality metric drops for one user segment while the average holds, treat it as a safety issue, not a metrics curiosity: segment-level failures are what AI guardrails and bias reviews exist to catch.

How do you write the metrics plan for one AI feature?

Write it as one page attached to the feature ticket, filled in before launch, with a baseline, a target and a read date for each line. For an AI reply drafter in a helpdesk, the plan looks like this:

Line	What you write down
Outcome	Median time from ticket open to first useful reply, compared with a holdout group
Quality proxy	Share of drafts sent with light or no edits; regenerate rate; agent thumbs down rate
Guardrail	Reopen rate within 7 days and CSAT on AI-assisted tickets must not fall below the holdout
System	p95 draft latency ceiling, timeout rate, cost per sent draft
Segment check	Quality proxy split by language and ticket category
Read dates	Proxy and system daily from launch; outcome after several weekly cycles

Fill the baseline column from your current data before writing any target. If you cannot measure a line today, add the event before launch, not after the first review meeting.

Where do you learn to build this metric stack with practice?

Builders Camp's AI Product Management bootcamp runs for 2 weeks with 4 live sessions and 7 self-paced microlessons, one of which is titled "What metrics do you use to measure the success of AI products?". Its sessions cover evals as the launch gate, agent design, and how AI products capture value, and the practical challenge works through a live AI feature that fails a bias audit, where you have to set the acceptance criteria a retrained model must pass. The bootcamp is part of the AI Product Expert Track and, like each bootcamp, is led by its own programme director.