Skip to content
Cortexa ConsultingCortexa Consulting
Cortexa Ground Truth
Explainer5 min readBy Joe Coffman
0:00 / 3:29

Tip: once it’s playing, tap any line to jump the audio there.

Fine-tune, retrieve, or just ask better.

Most “train it on our data” asks don't need training. A simple ladder — prompt, then ground (RAG), then fine-tune — tells you which one you actually need.

“Should we fine-tune a model?” — or “shouldn't we train the model on our data?” — might be the most expensive question in AI, because the answer is usually no. Three different tools keep getting mistaken for one another: prompting, retrieval-augmented generation (RAG), and fine-tuning. Picking the right one saves months and budget.

Three tools people mix up

Prompting is telling the model clearly what you want — the instructions, examples, and format, written in plain language. RAG hands the model the right documents so it answers from your material instead of its memory. Fine-tuning actually retrains the model on your examples, adjusting its internal wiring so it defaults to a particular style or task.

Three ascending platforms — a prompt card, an open book (RAG), and a gear meshed with a brain in a jar — cheapest to most expensive. A figure steps onto the middle “ground it with RAG” platform, highlighted in chartreuse as the usual stopping point, while an arrow climbs past it into a fading dashed line.Three ascending platforms — a prompt card, an open book (RAG), and a gear meshed with a brain in a jar — cheapest to most expensive. A figure steps onto the middle “ground it with RAG” platform, highlighted in chartreuse as the usual stopping point, while an arrow climbs past it into a fading dashed line.

Climb the ladder — don't start at the top

The mistake is reaching for fine-tuning or “training the model” first because it sounds the most serious. It's the opposite: start on the cheapest rung and climb only when the one below genuinely isn't enough.

The decision ladder

What's actually real?

  • Most “train on our data” asks are really RAG. If the goal is accurate answers grounded in your content, retrieval gets you there — the foundational retrieval research2 and model-builder guidance both favor grounding over retraining3 for factual, up-to-date answers.
  • Prompting solves more than people expect. The prevailing engineering guidance is to use the simplest approach that works1 before adding machinery — a clearer prompt often closes the gap you were about to spend a quarter fine-tuning to fix.
  • Fine-tuning changes style and format, not facts. It's the right tool when you need a consistent voice, a strict output shape, or a specialized task at scale — not when you need the model to “know” your latest data. For knowledge, retrain nothing: ground it.

The “so what” — for anyone serving tech clients

  • When a client says “fine-tune it,” ask what they actually want: current, accurate answers (RAG), a specific voice or format (maybe fine-tuning), or just better results (often a better prompt). Scoping the real need protects their budget and your timeline.
  • You can deliver most “custom knowledge” projects with prompting plus retrieval — faster, cheaper, and updatable without a retraining cycle. Lead with that, and reserve fine-tuning for when it's genuinely earned.

Sources

  1. Anthropic — Building effective agents
  2. Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  3. Anthropic — Reduce hallucinations

Next Reads

Start a conversation