Fine-tune, retrieve, or prompt: which one?

You may have seen a company announce that it trained its own artificial intelligence (AI) on all of its documents. It sounds impressive, and sometimes it's the right call. But often a clearer request, or letting the tool look things up, would have done the same job with far less work. So how do you tell which fix a problem needs?

There are three ways to make an AI tool better at a particular job, and they run from light to heavy.

• Prompting: changing what you ask, and how you ask it.
• Retrieval: letting the tool look up the right documents before it answers.
• Fine-tuning: training the model a little further on your own examples, so it changes how it behaves.

Each one fixes a different kind of problem. The trick is to start with the lightest one that works.

Prompting is the cheapest fix and the quickest to try. You give clearer instructions, say who the answer is for, and show an example or two of what good looks like. It costs a few minutes, and you can change it again tomorrow. OpenAI's guide to improving a model's results starts there: give it clear instructions and the context it needs, measure how it does, and only then think about training. The guide also says that for some jobs, a better prompt may be all you need.³

Sometimes the model just doesn't know something. Your return policy. Last week's price list. A contract it has never seen. A better prompt can't fix that, because the facts aren't in the model. That's the job for retrieval. The tool searches your documents first, then writes its answer from what it found. It suits facts that are private or that change often, because when you update the documents, the answers follow. And you can see where each answer came from.²

Fine-tuning changes the model itself. You train it further on examples of the behavior you want, and it starts doing that by default. That suits a lasting style or format: a support assistant that always answers in your brand's tone, or a tool that has to fill in the same form the same way every time. It's heavier work, because someone has to gather good examples, run the training, and test the result. It's also a weak way to teach facts. In a study published in 2024, researchers at Microsoft found that retrieval did better than fine-tuning at giving a model new knowledge.¹⁴⁵

So when a tool falls short, ask one question first. Is the problem what it knows, or how it behaves? If the answers are vague or messy, work on the prompt. If it's missing facts, add retrieval. And if it knows enough but won't keep to the style you need, even with good instructions, that's when fine-tuning is worth its cost. Plenty of teams end up using all three together.

Next time a company says it trained its own AI, you'll have a good question ready. What problem was it fixing, and did anyone try a better prompt first?