"Should we fine-tune or use RAG?" is one of the first questions teams ask when an LLM prototype is not good enough. It is usually the wrong first question. Ask what exactly is failing. A model that lacks some knowledge and a model that behaves the wrong way are different problems with different fixes.
What changed in 2026
Most articles on this topic predate two shifts that change the answer.
The first is that fine-tuning frontier API models is fading out. OpenAI is winding down its fine-tuning platform. New organizations lost access in May 2026, inactive ones in July, and on January 6, 2027 no customer can start new fine-tuning jobs. Existing fine-tuned models keep serving until their base models are retired. For most teams, "fine-tuning" now means an open-weights model.
The second is that long context became cheap enough to be a real option. Frontier models accept hundreds of thousands to a million tokens, and prompt caching makes repeated context far cheaper to resend. For a small knowledge base, "just put it in the prompt" competes with RAG.
So the real choice today is between prompting (including long context with caching), RAG, and fine-tuning an open model.
The one-line version
Prompt engineering changes the instructions. RAG (retrieval-augmented generation) changes what the model knows at the moment it answers, and fine-tuning changes how the model behaves by default.
Side by side
| Prompt engineering | RAG | Fine-tuning | |
|---|---|---|---|
| Fixes | Unclear instructions, format, reasoning steps | Missing or outdated knowledge, your private data | Consistent style, format, narrow task behavior |
| Doesn't fix | Knowledge the model never had | Behavior or tone problems | Reliable recall of specific or changing facts |
| Time to first result | Hours | Days to weeks | Weeks (data collection is the slow part) |
| Update cost | Edit a prompt | Re-index documents | Re-train on a new dataset |
| Citations and permissions | No | Yes: cite sources, filter by user access | No |
| Main ongoing cost | Tokens | Retrieval infrastructure plus longer prompts | Training runs plus hosting or premium inference |
Option zero: put it in the prompt
Before building a retrieval pipeline, check how big your knowledge base actually is. If it fits comfortably in the context window and changes rarely, load all of it into the prompt and cache it. Anthropic's guidance is that knowledge bases under roughly 200,000 tokens (about 500 pages) can often skip RAG entirely. You avoid chunking and retrieval errors, and caching keeps the repeated context cheap. See our guide to reducing LLM API costs for how caching is priced.
It stops working when the corpus is large or changes constantly, or when different users may see different documents.
When to use RAG
Use RAG when answers depend on information the base model does not have, or on information that changes. Internal docs and wikis are the classic case. So are contracts, support tickets and product catalogues. RAG also fits anywhere users need to see where the answer came from, and any data with access rules where different users may see different documents.
If the knowledge lives in live systems such as a CRM, ticketing tool or database rather than in documents, consider letting the model fetch it at answer time through tools instead of indexing copies. That is what an MCP server is for.
Most of a RAG system's quality comes from the retrieval step. Good chunking, hybrid keyword and vector search, a reranker and filtering by metadata matter more than which LLM you pick. Anthropic's published contextual retrieval experiments are a useful reference. Adding context to each chunk before embedding cut retrieval failures by about a third. Combining it with keyword (BM25) search roughly halved them, and adding a reranker brought the reduction to about two thirds. So if answers are wrong, check first whether the right passage was retrieved at all.
When to fine-tune
Fine-tuning earns its cost when the task is narrow and repetitive, like classifying tickets, extracting fields in a fixed schema or rewriting text in a house style. You also need hundreds to thousands of high-quality examples of the input and ideal output. And your evals should show a consistent behavior gap that better prompts and few-shot examples did not close.
Distillation is often the best reason of all. You train a smaller model that is cheaper and faster to run so it copies a large model's behavior for one step of a pipeline, which can cut LLM costs and latency at the same time.
In practice this now means an open-weights model that you host yourself or run on a managed training and inference platform. Budget for the operational side too. Someone has to host and monitor it, and re-train it when the base model is superseded.
Avoid fine-tuning to teach facts. The model may absorb them partially and blend them with what it already believes, and you cannot easily update or cite them.
When prompt engineering is enough
More often than teams expect.
A large share of "the model is not good enough" complaints go away with clear instructions and a defined output schema, plus a few worked examples and room for the model to reason before answering. Prompting is also the cheapest thing to change. That is why it should always be the baseline you measure the other options against.
The order we recommend
- Build an eval set first, with 50 to 200 real questions that have known good answers. Without it you cannot tell if any change helped.
- Work on the prompt. Get the most out of instructions, schema and examples, and measure.
- If the knowledge base is small, try long context. Put it in a cached prompt and measure.
- Add RAG if the corpus is too big, changes often or needs per-user access control. Measure retrieval quality separately from answer quality.
- Fine-tune an open model only if the evals show a behavior gap that remains and you have the data to fix it.
- Combine approaches where it helps, for example a retrieval-backed main model with a small fine-tuned model for classification or extraction.
Common mistakes
The first is fine-tuning before you have evals. You will not know whether it helped, or what it broke.
Teams also blame the model for bad retrieval. Log the retrieved chunks next to every answer.
Stuffing the whole knowledge base into the prompt is another. Long contexts are expensive and slower, and models still miss details buried in the middle.
And don't treat this as a one-time decision. Models change every few months. A system with good evals lets you swap models or techniques safely.