Prompting vs RAG vs fine-tuning: how a PM decides

Start with the cheapest option. OpenAI's own model optimization guide says "The prompt engineering process may be all you need in order to get great results for your use case." The same page now opens its fine-tuning section with "OpenAI is winding down the fine-tuning platform." OpenAI's deprecations page says that from May 7, 2026, creating fine-tuning jobs "is not available to organizations that have not previously run fine-tuning"; that from July 2, 2026, it is also closed to organisations that have not run inference on a fine-tuned model in the past 60 days; and that "Active existing customers will no longer be able to create new fine-tuning jobs" from Jan 6, 2027. Inference on models already fine-tuned continues until their base models are deprecated.

Those dates were checked on 9 November 2026, so check the page again before you plan around them.

One vendor's platform decision does not settle the technique. It does change the risk of building a roadmap around a custom model you may not be able to retrain. The useful question for a PM is narrower: which of the three fixes the specific failure your evals show. If you need the basics of retrieval first, RAG for product managers explains how it works; this page is about choosing between all three.

What does each option actually change?

Each option changes a different input to the answer: prompting changes the instructions, RAG changes the knowledge available at answer time, and fine-tuning changes the model's learned behaviour. Mixing them up is how teams end up training a model to memorise a refund policy that changes every quarter.

	Prompting	RAG	Fine-tuning
What it changes	Instructions, examples and context sent with each request	The documents retrieved and added to the prompt at answer time	The model's weights, through training on example inputs and outputs
Fixes	Unclear task, wrong format, missing examples, tone	Missing, private, large or fast-changing knowledge	A consistent behaviour the prompt cannot hold, such as a strict format or a narrow classification
Does not fix	Knowledge the model never saw and that will not fit in the prompt	Behaviour problems, such as ignoring the format	Facts that change after training
Time to change	Minutes: edit and rerun the evals	Hours to days: build the index, then update documents as they change	Days to weeks: collect data, train, evaluate, repeat
What you maintain	A prompt and its eval set	An index, a retrieval pipeline and the source documents	A training dataset, training runs and a retrain plan when the base model changes

The RAG paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, framed the knowledge problem in 2020: it notes that for language models, "providing provenance for their decisions and updating their world knowledge remain open research problems." Retrieval answers both, because the source of an answer is a document you can show and edit. Fine-tuning answers neither.

Why should you start with prompting?

Start with prompting because it is the cheapest change to make, the fastest to test, and the only one you can reverse in minutes. The OpenAI guide lists the prompting basics as including relevant context, providing clear instructions, and providing example outputs, the last of which it calls few-shot learning. A wrong format, a missed edge case or the wrong tone are instruction failures, and the prompt engineering glossary entry covers the techniques.

Prompting has limits, and the vendor docs say so. Anthropic's prompt engineering overview states "Not every success criterion or failing eval is best solved by prompt engineering. For example, you can sometimes improve latency and cost more easily by choosing a different model." The same page assumes you start with "A clear definition of the success criteria for your use case" and "Some ways to empirically test against those criteria". Without those, you cannot tell a prompt that improved from one that got lucky on three examples. Build the test set first, as how to write evals for AI products describes.

When does a problem need RAG instead of a better prompt?

A problem needs RAG when the knowledge the model needs is too large to paste into every prompt, changes often, or has to be scoped to the user asking. If the knowledge is small and stable, skip RAG and put it in the prompt. Anthropic made that point in its September 2024 post Introducing Contextual Retrieval: "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods."

The same post said prompt caching made that approach cheaper, describing it at the time as "reducing latency by > 2x and costs by up to 90%". Those figures are Anthropic's own, for its own models, from 2024; treat them as evidence that caching changes the sum, not as a number for your feature. Run your own estimate with how to estimate LLM API costs for a product feature, and check the context window of the model you use.

Once the knowledge outgrows the prompt, retrieval quality becomes the product. In the Contextual Retrieval post, Anthropic reported that its method "can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%." Those results come from Anthropic's own benchmarks, so the takeaway for a PM is the size of the gap between basic and improved retrieval, not a promise about your data. After launch, expect most RAG work to go into the search step and the vector store behind it, not the model.

When is fine-tuning worth the cost?

Fine-tuning is worth it when evals show a behaviour gap that prompting and RAG have not closed, and the task is narrow and high-volume enough to repay the training work. The OpenAI model optimization guide describes fine-tuning as "a time-consuming process" that "can also enable a model to consistently format responses in a certain way or handle novel inputs."

The same guide lists four benefits over prompting alone: you can provide "more example inputs and outputs than could fit within the context window of a single request"; you can use shorter prompts, which "saves on token costs at scale and can be lower latency"; you can train on proprietary data without sending it as examples in every request; and you can train a smaller, cheaper, faster model for one task. Its table of methods names supervised fine-tuning as best for classification, nuanced translation, generating content in a specific format and correcting instruction-following failures.

Notice what is missing from that list: teaching the model facts that change. A fine-tuned model knows what was true in its training data. When the facts move, you retrain, and when the vendor retires the base model, you retrain on the new one, if the vendor still lets you.

How do cost, latency and maintenance compare?

Cost, latency and maintenance move in different directions for each option, which is why no single option wins on all three. The table compares where the cost comes from rather than quoting price ranges, because published ranges depend on the model, the volume and the date, and most of the ones in circulation have no primary source.

	Prompting	RAG	Fine-tuning
Upfront work	Low: write the prompt and the evals	Medium: chunk, embed and index documents, tune retrieval	High: collect and label a training set, run training, evaluate
Per-request cost driver	Prompt length, cut by caching	Prompt length plus retrieved passages, plus the search call	Often lower per request if a shorter prompt or a smaller model works
Latency driver	Prompt length and model choice	The retrieval step added before generation	Shorter prompts can lower latency, per OpenAI's guide
What goes stale	The prompt, when the model changes	The index, when documents change	The whole model, when facts or the base model change
Who usually owns it	PM and engineer together	Engineering and whoever owns the source content	ML or platform engineering

The row that decides most cases is the last but one. A prompt goes stale slowly and is cheap to fix, an index goes stale every time someone edits a document, and a fine-tuned model goes stale all at once.

Which option fits which problem?

Match the option to the failure your eval set shows, not to the feature description. The table maps common symptoms to the first fix to try.

What the evals show	Likely cause	Try first	Escalate to
Right facts, wrong format or tone	Instructions	Clearer instructions and two or three examples	Fine-tuning, if the format still drifts at volume
Invented answers about your product or policies	Missing knowledge	Paste the relevant documents into the prompt	RAG, once the documents do not fit or change often
Correct last month, wrong this month	Stale knowledge	RAG over the current source	Better retrieval, not a bigger model
A user sees another customer's data	Access scoping	Retrieval filtered by user permissions	Never fine-tuning, which cannot scope per user
Good output, but too slow or too costly at volume	Model size and prompt length	A smaller model, caching, a shorter prompt	Fine-tuning a smaller model, if your vendor offers it
Inconsistent labels on a narrow classification task	Behaviour	Better label definitions and examples	Fine-tuning on labelled data

The permissions row is the one teams miss. Anything trained into a model can surface for any user of that model, so per-user or per-customer knowledge belongs in retrieval, where access rules can be applied before the passage reaches the prompt.

In what order should a PM try them?

Try them in a fixed order and stop as soon as the evals pass, because each step adds cost and maintenance you have to carry.

1. Write the evals. Collect real inputs, define what good looks like, and record a baseline score.
2. Prompt. Fix instructions, add examples, and try a different model if latency or cost is the problem.
3. Add context directly. If the model is missing knowledge that fits in the prompt, put it there, and use caching if the same context repeats.
4. Add RAG. When the knowledge is too large, changes often or needs per-user scoping, retrieve it instead.
5. Fine-tune. Only for a behaviour gap that survived steps 2 to 4, on a task with enough volume and labelled data to repay it, and only after checking your vendor still supports it.

Combining options is normal at the end of that sequence. The original RAG paper itself explores "a general-purpose fine-tuning recipe for retrieval-augmented generation", and a product might use RAG for current knowledge and a fine-tuned model for format. Add each layer only when it moves the eval score on its own.

Is "prompt first" bad advice for some products?

"Prompt first" is weaker advice for products that run one narrow task at very high volume, and that objection deserves its own answer. If a classifier runs millions of times a day, a long prompt with many examples is a cost you pay on every call, and the OpenAI guide is explicit that fine-tuning can cut prompt length and let a smaller model do the job. Teams in that position sometimes plan for fine-tuning from day one.

Even then, the prompt comes first in time, because the prompt-based version is how you build the labelled examples, the eval baseline and the proof that the task is worth automating. What has changed in 2026 is the platform risk. A plan that depends on retraining a custom model needs a fallback for the day your vendor stops offering training, and OpenAI's deprecations page shows that day is not hypothetical. Ask your engineering lead which vendors you could retrain on, and whether a smaller general model with a good prompt would pass your evals today.

Where do you build the mental model for these choices?

Before your next AI feature review, take the five worst cases from your eval set and label each one instruction, knowledge or behaviour. If most are instruction failures, the team does not need a fine-tuning project; if most are knowledge failures, the next sprint is retrieval. Builders Camp's Foundations of AI bootcamp, led by its own programme director, a working practitioner, runs for 2 weeks with 8 self-paced microlessons and is built to make you fluent in Generative AI and AI Agents from a product perspective. Its stated curriculum covers prompting and ideation workflows and scaling LLMs and workflows with APIs, templates and automation, and it sits in the AI Product Expert Track. For the prompting step on its own, the AI Prompting for Product bootcamp covers the patterns product people use day to day.

See the Foundations of AI bootcamp