AI classifier prompt practice exercise for product managers

The scenario

A field service app used by technicians gets 60 to 80 app store reviews a week across iOS and Android. Three weeks ago, a PM automated the weekly triage process: every review gets sent to GPT-4 with a single instruction to classify it as Bug Report, Feature Request, Praise, or Churn Signal, and the category gets written back into a shared sheet.

The prompt itself reads reasonably. It names the four categories and asks for accuracy and concision. It has been running unreviewed for three weeks. This morning, ahead of sprint planning, someone opens the sheet and notices half the classifications look wrong, and the PM who built it is on holiday.

Several specific errors stand out: a crash report on photo upload classified as praise, a request for parts inventory classified as a churn signal, a review about switching to a competitor classified as a bug report, a cancelled subscription classified as a feature request, and a complaint about missed job alerts classified as praise. These are not random noise. They share one or two underlying causes in how the prompt was written, and finding those causes, not just relabeling the six examples by hand, is the actual job.

What you are asked to do

The exercise has five connected steps:

• For each of the five worst misclassifications, state what the correct category should have been and why the prompt's wording produced the wrong one instead.
• Name the one or two root causes shared across the errors, specifically what was missing or ambiguous in the prompt, not a general statement that it was "unclear," and name the one thing the original author likely assumed the model would figure out on its own.
• Rewrite the system prompt to address both root causes, including context about the product and its users, a precise definition of each category with the ambiguous boundaries resolved, and instructions for reviews that could plausibly fit more than one category.
• Predict how your new prompt would reclassify each of the five reviews, explaining what specifically in your new category definitions makes the new answer correct.
• Define a monitoring plan naming a specific, measurable signal to track, how often to check it, and the threshold that should trigger a manual review of the prompt.

What a strong answer covers

The exercise's own objectives are what a strong submission has to satisfy:

• Does the diagnosis for each misclassified review name the specific gap in the prompt's category definitions, rather than a general note that the model got it wrong?
• Are the named root causes shared across multiple errors, showing that the diagnosis found the pattern rather than treating six mistakes as six unrelated problems?
• Does the rewritten prompt resolve the specific ambiguous boundary that caused the original errors, such as the line between a bug report and a churn signal?
• Do the predicted reclassifications follow logically from the new prompt's own definitions, rather than simply asserting the correct answer?
• Is the monitoring signal specific and measurable, with a stated check frequency and a concrete threshold, rather than a general instruction to "keep an eye on it"?

Skills this exercise practises

Tracing a run of bad AI outputs back to the specific prompt language causing them, instead of relabeling each mistake by hand. Writing category definitions precise enough to resolve a genuinely ambiguous case. Predicting how a prompt change will behave before deploying it, which is what makes a fix testable rather than hopeful. Defining a monitoring threshold specific enough to actually catch the next drift. These map onto the bootcamp's own curriculum on prompting and ideation workflows and scaling LLMs and workflows reliably. If you want to work a comparable diagnosis on a much higher-stakes classifier, the AI bias audit exercise from AI Product Management runs the same kind of root cause analysis in a healthcare setting. For the broader groundwork, see how to build an AI product and how to become an AI product manager.

Which bootcamp this comes from

This exercise is the practical challenge from Foundations of AI, a two week bootcamp on Builders Camp with four live sessions covering generative AI and agents explained for builders, AI across the product lifecycle, and prompting and ideation workflows. Completing the practical challenge counts toward the bootcamp's completion requirement and its certificate, alongside the certification quiz.

Builders Camp runs this bootcamp both live and self-paced, included with the Builders Camp Membership alongside every other bootcamp, track, and masterclass.