How to Write Evals for AI Products

What is an eval set, exactly?

An eval set is a fixed list of real inputs, paired with what a correct or acceptable output looks like, that you score every time a prompt, model, or retrieval step changes. Think of it as the thing that stands in for "did this actually get better" once you no longer have a single obvious right answer to check against. A traditional feature ships when the code passes its tests. An AI feature ships when its outputs pass a rubric a person defined in advance, because the model's behavior is probabilistic, not fixed.

The set itself is not complicated. Eugene Yan's write-up on product evals describes the simplest version as a spreadsheet: one column for the input, one for the model's output, one for any context that helps a grader judge it, and one for the label, pass or fail. You do not need a framework to start. You need a frozen list of cases and a written definition of what passing means for each one.

Why are evals called the new PRD?

For an AI feature, the most precise specification you can write is an eval, because a paragraph describing good behaviour cannot be run and an eval can. Anthropic's documentation on defining success criteria puts the order plainly: "Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them." The same page asks for criteria that are specific and measurable, so "good performance" becomes something a test can check.

Used as a spec practice, the loop has four steps:

• Pick the hero use case. Name the one job the feature must do well for the launch to count, for example "answer a billing question from the customer's own invoice history."
• Write the ideal answer. For 10 to 20 real inputs to that job, write what a great response looks like, including what it must not do.
• Turn the answers into an eval. Each ideal answer becomes one or more pass or fail checks that a person, a code rule or a validated LLM judge can apply.
• Hill-climb. Change the prompt, retrieval or model, re-run the set, keep the change only if the pass rate goes up without new failures.

The document still has a job: it explains why this use case, for whom, and what the team decided not to build. What moves out of the document is the acceptance criteria, which is also why reviewing an AI written PRD ends with the eval set rather than the prose.

How many examples do you actually need to start?

Twenty to fifty real outputs is the floor. Hamel Husain and Shreya Shankar's AI evals FAQ describes a minimum viable setup as spending 30 minutes manually reviewing 20 to 50 outputs whenever you make a significant change, and recommends a larger pool of about 100 diverse traces when you are hunting for failure modes. It is worth taking seriously because it is written for teams that already shipped and got burned skipping this step. The number that matters more than the count is where the examples come from: real usage logs, real support tickets, or your own attempts to break the feature, not a list you imagined in a planning meeting. A handful of real, hard cases teaches you more than a hundred synthetic ones, because synthetic inputs rarely reproduce the exact way your product actually fails in front of a user.

Weight the set toward failures on purpose. If your AI feature is 90 percent accurate on easy inputs and falls apart on the remaining 10 percent, an eval set built from a random sample of traffic will mostly test the easy 90 percent, and you will feel confident right up until a user hits the gap. Go looking for the hard cases instead: the ambiguous request, the edge-case input, the thing a less capable model gets wrong that a more capable one might paper over.

What makes a good pass or fail criterion?

A pass or fail criterion is a written yes or no question, specific enough that two different people would grade the same output the same way. "Is this a good answer" is not a criterion. "Does this answer address what the user actually asked, without inventing a fact that is not in the source material" is a criterion, because it names the failure mode you are actually worried about.

Binary labels beat a 1-to-5 scale for the same reason a clear rubric beats a vague one: a scale lets a grader avoid the hard call in the middle, and the hard call in the middle is exactly the judgment you need someone to make out loud. Write the rubric before you look at any outputs, or you will unconsciously write it to match whatever the model already produced. Treat it the same way you would treat acceptance criteria in a PRD: specific enough that someone else could apply it without asking you what you meant.

Grading method	Best for	What it costs	Where it breaks
Human review	Defining the first rubric, ambiguous or high-stakes failure modes	Slow, does not scale past a few hundred cases a week	Reviewer fatigue produces inconsistent grading over time
Code-based check	Objective, deterministic rules: does the JSON parse, is the citation present, is the tool call formatted correctly	Fast, cheap, runs on every commit	Cannot judge tone, factual accuracy, or anything requiring interpretation
LLM-as-judge	Scaling a rubric a human already validated, subjective quality checks at volume	Cheap and fast once set up, needs ongoing spot checks	Drifts from human judgment if the rubric or the underlying model changes and nobody re-checks agreement

How do you read traces to find failure modes?

A trace is the full record of one user session: the input, any retrieved context and tool calls, and the output. Reading them is the habit that keeps an eval set honest, because the checks you write should come from failures you have seen, not failures you imagined in a planning meeting. Husain and Shankar's FAQ is direct about this: "Write evaluators for errors you discover, not errors you imagine."

The routine is simple enough to run in an afternoon:

• Read about 100 real traces. The FAQ recommends reviewing at least 100 and continuing past that while you are still learning something new. Write a short note on every trace that is wrong.
• Name the failure modes. Group the notes into named patterns, such as "answers from the wrong customer's invoice" or "promises a refund policy we do not have," and count how often each appears.
• Write one binary check per failure mode. One pass or fail question per pattern, never a single check that tries to judge everything. Eugene Yan makes the same point about LLM judges: one evaluator per dimension is easier to align than a single evaluator that grades many dimensions at once.

The strongest objection to "evals are the new PRD" comes from the same FAQ, which answers "Should I practice eval-driven development?" with "Generally no," because an LLM can fail in ways nobody predicts before building. Both positions hold once you separate the two kinds of check. The hero use case and its ideal answers are the target you write up front. The failure-mode checks come afterwards, from reading traces, and they are where most of the eval set's value ends up.

How do you run an eval set without slowing down every ship?

Automate it into the same place a prompt or model change already gets reviewed, the same way a test suite runs before a pull request merges. The eval run itself should take minutes: feed the frozen input list through the new prompt or model, score each output against the rubric, and flag anything that flips from pass to fail compared to the last run. That comparison, not the raw pass rate, is what tells you whether a change made things better or worse.

The part that actually takes time is building the first set and getting the rubric right, not running it afterward. Budget a real afternoon for that step rather than trying to improvise it the night before a launch, because a rushed rubric produces a set that passes everything and catches nothing.

Should a person or a model grade the output?

Start with a person, always, for any new failure mode. You cannot hand a rubric to an LLM judge until a person has proven the rubric actually separates good outputs from bad ones on real examples. Once the rubric holds up on real examples, an LLM judge can take over the repetitive grading at volume, scored with a clear yes or no question. Budget the labels honestly: Husain and Shankar's FAQ plans for 100 to 200 examples per failure mode, labeled by a trusted domain expert, before trusting a judge on a subjective failure, and it measures the judge by how many of the human-labeled failures it catches.

Keep checking the judge's agreement against a fresh batch of human-labeled cases every so often. An LLM judge that agreed with your team last quarter can quietly drift once the underlying model updates or your product's inputs shift, and nobody notices until the pass rate looks great and users are still unhappy.

When should you re-run evals and add new cases?

Re-run the set on every prompt change, retrieval change and model change, including a provider's model update you did not ask for. A new model can fix three failure modes and open two you had closed, and only a fixed set run before and after will show you which. Husain and Shankar's FAQ lists the triggers for a fresh round of trace reading: "Re-run error analysis when making significant changes: new features, prompt updates, model switches, or major bug fixes."

Every production failure is also a new eval case. When a user reports a bad answer, or a trace review turns one up, add that input and the correct behaviour to the set before you fix it, then confirm the fix turns it from fail to pass. Over a few months, the set becomes a record of every way the feature has broken, which is exactly the regression suite a probabilistic product needs. Hamel Husain's earlier post, Your AI Product Needs Evals, frames the same idea as three levels, unit tests, human and model review, and A/B testing, run at different cadences because each costs more than the one before.

What happens when you skip evals entirely?

You ship on vibes, which works exactly until the first prompt tweak that looks like an obvious improvement in the three examples someone tried by hand and turns out to break a case none of them thought to check. The AI Product Management bootcamp's own practical challenge is built around a version of this failure: an AI feature passes its stated accuracy threshold and still causes real harm, because the threshold measured the wrong thing and nobody had a rubric that would have caught it earlier. Evals do not prevent every failure. They prevent the specific failure of finding out from a user instead of from your own test run.

Who this guide is for, and who it is not for

Writing evals as described here suits a product manager, founder, or builder who has an AI feature already shipped or close to shipping and needs a repeatable way to know whether the next change made it better or worse, not a data scientist looking for a paper on evaluation metrics. It assumes you can write a specific yes or no question about what a good answer looks like for your product, because that judgment call cannot be automated away.

It is not for someone looking for a guaranteed accuracy number to hit before launch. No such number exists in the general case, and Builders Camp does not publish one because the right bar depends entirely on what a wrong answer costs your specific users. It is also not a substitute for talking to users directly. Evals check a rubric you already trust; user research is how you find out when the rubric itself needs to change.

Where evals fit next to the rest of the AI product job

Evals are one piece of a larger discipline that also includes designing what an AI agent is allowed to do on its own, deciding when retrieval-augmented generation actually improves an answer instead of just adding latency, and knowing how to build an AI product end to end rather than as a series of disconnected demos. If evals are the piece you are missing, Builders Camp's AI Product Management bootcamp walks through building an evaluation toolkit as one of its four live sessions, alongside the same responsible-AI patterns this guide only has room to summarize.

Builders Camp runs the AI Product Management bootcamp live over 2 weeks, 4 sessions, with a certification quiz and a practical challenge built around a realistic AI incident scenario, not a toy example.