LLM fine tuning vs RAG: how to tell which one your problem needs

A support assistant gets two kinds of questions wrong. Ask it what the refund window is and it invents a number, because the current policy isn't anywhere in front of it. Ask it to answer in the format your helpdesk requires and it sends a friendly paragraph, because nothing has taught it the house style. The first is a knowledge problem and the second is a behavior problem, and they have different fixes.

Teams often reach for fine tuning first because it sounds like the thorough option, even when the problem is knowledge. Picking the wrong one can cost weeks of rework.

LLM fine tuning vs RAG: what each one changes

Fine tuning continues the training of an existing model on examples of your own, so its behavior shifts. It learns patterns: your terminology, your tone, a fixed output format, the way your team makes a particular call. What it learns is baked in. It can't look anything up, and it can't tell you where a fact came from.

RAG, short for retrieval augmented generation, leaves the model alone. At question time a retrieval step finds relevant passages in your content and puts them in the prompt, and the model answers from them. Knowledge lives in your documents, so changing an answer means changing a document.

Three scenarios and their answers

Fine tuning and RAG solve different problems, so here are three invented cases and the answer each one tends to get.

An HR assistant answers staff questions about leave, benefits and expenses. The questions are about facts that live in policy documents and change a few times a year, which makes this a retrieval problem. Index the policies, show the source, refresh when a policy changes, and tune nothing. A tuned model would learn last year's leave policy and keep quoting it.

A system tags incoming contracts with clauses from the company's own taxonomy: forty categories, some of them subtle. The taxonomy is the company's invention, so a general model keeps mislabeling the borderline categories even with careful prompts, and there's an archive of several thousand contracts that lawyers have already tagged. That is a fine tuning problem, and a reasonable one: the behavior is narrow, the examples exist, and the output is a label with no fact to look up.

A support assistant writes replies in the company's tone, in the format the helpdesk requires, using current policies. It may need both. Retrieval brings the current refund policy and the customer's order status into the prompt. Tuning, if prompting can't hold the house style and format consistent, teaches the style. In practice the first step is retrieval plus a good prompt, and tuning gets added only if the format still wobbles.

Check prompting first

Before either, see how far a better prompt gets you. Write clear instructions, add a few worked examples, and ask for a structured output format. Many problems that look like they need tuning turn out to need a better description of the task, and if careful prompting fixes the problem, tuning only adds cost.

Then work through these questions in order.

1. Does careful prompting with a few examples produce the right behavior? If yes, stop there.
2. Does the answer sit in a document the model could read? If yes, start with retrieval: it usually costs less and can show its source.
3. Does the output need a fixed format, tone or house terminology that prompting can't keep consistent from one answer to the next? That points toward fine tuning.
4. Do you have consistent examples of the task done well, enough to train on and to hold some back for testing? Without them, tuning has little to learn from.
5. Do you need both facts and behavior? Then plan to use both.

Prompting techniques worth trying before you tune

Check prompting properly before declaring it insufficient. A few techniques cover most of the ground.

Show examples. Two to five worked examples in the prompt, chosen to cover the awkward cases, often fix format and tone problems that instructions alone don't. Ask for structure. If the output feeds another system, request JSON that matches a schema and validate it in code, retrying when it doesn't parse. Break the task up. A prompt that asks a model to read, decide, format and explain in one go does worse than a short chain where each step has one job, and the intermediate outputs are easier to inspect. Add a check step, where a second pass compares the draft against the instructions or the source passages and flags problems. And try a stronger model before you try a tuned one: sometimes the problem is capability, and a larger model with the same prompt solves it with less effort than a tuning project.

Keep notes as you go. Record what you changed, what the score did, and what the failures look like afterwards. If you've exhausted these and the same failure remains, you have the evidence tuning needs, and the examples you collected along the way are the start of a dataset.

How LLM fine tuning works

You assemble examples, each one an input and the output you wanted, and train the model on them for a short time. Then you test it on examples it has never seen and compare the result with the base model.

Among LLM fine tuning techniques, parameter efficient ones are now the usual choice for open models. Full fine tuning updates every weight and needs a great deal of GPU memory. LoRA, short for low rank adaptation, freezes the original weights and trains small extra matrices that sit on top, which cuts the memory needed for training and leaves a small adapter file beside the original model. QLoRA goes further by loading a compressed copy of the base model, and its authors tuned a 65 billion parameter model on a single 48 GB GPU. How much memory you need depends on the model size and the method, but the base model still has to fit.

Fine tuning LLM models can also make them worse at things they used to do well. A model that specializes may lose general abilities, a problem known as catastrophic forgetting. Adapter methods tend to forget less than full tuning, though not nothing. After tuning, run regression checks: a set of tasks the base model handled correctly, to confirm that a gain in one area hasn't hidden a loss in another.

Fine tuning LLM datasets: build them from work already done well

The dataset is the long part. Training can take hours; assembling and checking the examples often takes weeks. Build it from work your team already did well: replies a good agent wrote, summaries an expert approved, classifications that were later confirmed. Clean out duplicates, near duplicates and the noise that collects in real records, and have a person review the set before training, because a labeling error turns into a model behavior.

Consistency comes before volume. If two of your senior people would label the same case differently, the model will learn the disagreement, and more data won't resolve it. For a narrow task, a few hundred examples that agree with each other can be enough to show whether tuning helps. Hold back a portion, perhaps ten to twenty percent and at least fifty cases, as an evaluation set that the model never trains on. Scoring a tuned model on its own training examples proves nothing, because it has already seen the answers.

Using both, and how to evaluate

A support assistant can use RAG to look up the current refund policy and a tuned model to write in your house style and your helpdesk's format. If you can only do one first, match it to the failure you see: wrong facts point to retrieval, and wrong behavior, once prompting has been tried, points to tuning.

In the support case, a question arrives first, retrieval finds the current policy and the order record, a prompt assembles them with the house style instructions, and the tuned model writes the reply in the right format. A separate check compares the reply with the cited passages before it goes out. Each piece can be tested on its own: retrieval on whether the right passage came back, the tuned model on format and tone, the check on whether it catches a reply that contradicts the policy. When something goes wrong, you can say which piece did it.

Either way you need a test you wrote before you started. Collect fifty to a hundred real tasks with known good outputs and score the base model on them, so you have a baseline. Then score the system after your change. For RAG, also check that the cited passage supports the answer in each case, since a fluent answer from the wrong passage is the typical failure. For a tuned model, include regression tasks and cases it should refuse. A public benchmark tells you very little about your documents or your tasks.

Diagnosing which part failed

In a system that uses both, a wrong answer has a traceable cause, and finding it saves time. Look at what the model was given before you look at what it said. If the right passage wasn't in the prompt, the failure is in retrieval: the index, the splitting or the search settings. If the right passage was there and the model ignored or garbled it, look at the prompt and the model. If the content was right and the tone or format was wrong, that's the behavior problem tuning addresses. If the content was right, the format was right, and the answer was still wrong, check whether the source itself was out of date.

Log the retrieved passages with every answer so this diagnosis takes minutes. Teams that don't keep that log end up arguing about whether the model is the problem without any evidence either way.

Building an evaluation set you can trust

Both options depend on a test written up front, and its quality limits how far you can trust everything after it. Sample real tasks, not invented ones, spread across the categories you care about, including the rare and the awkward. For each, record what a good output looks like: an exact answer where there is one, otherwise a short rubric with the points that must appear and the mistakes that must not.

Decide how to score. Exact match works for labels and extraction. For free text, human reviewers are the reference, and a second model can grade at scale if you've checked its grades against humans on a sample; if the two disagree often, don't trust the model grader yet. Have two people grade a subset and measure how often they agree, because if people can't agree on what good looks like, any score against it will be noisy.

Freeze each version of the set. Keep it out of training, keep a version number so that a score from March means the same thing in June, and report results by category as well as in total, since an average can hide a category that got worse. Add to it over time: every real failure you find in production is a new test case.

What each one takes to keep running

RAG needs the index kept current: refreshing as documents change, removing deleted ones, and adjusting retrieval settings as the questions change. A tuned model needs watching for drift as your domain and your data move, and retraining before quality slips.

Open models give you control, because the weights can stay inside your environment, which counts for a lot where data can't leave it. Check the license too, since some open weight licenses restrict commercial use. Some providers let you tune their hosted models, under their own terms for data and ownership, so read those first. Before launch, decide who owns the evaluation set and the refresh or retraining schedule.

Costs, hosting and serving a tuned model

The headline cost of fine tuning is often small. Parameter efficient training of an open model can take hours on a single GPU, and the money goes into the data work before it and the engineering after. Preparing and reviewing examples takes weeks of people's time. Serving the result takes infrastructure: a tuned model is a model you host, or a tuned variant at a provider, with the running costs that implies.

Adapters make some of that cheaper. One base model can serve several adapters, each for a different task or customer, which shares the expensive GPU memory. Merging an adapter into the base weights simplifies serving at the price of flexibility. Either way, plan for retraining: your domain will drift, a new base model will arrive, and an adapter tuned for the old base can't simply be moved to a new one. Budget a retraining cycle and the evaluation run that goes with it, and name the owner. RAG has running costs of its own in the index, the refresh jobs and the retrieval calls on every question, plus the longer prompts that retrieved passages add, and none of it stops.

Mistakes teams make

The same few mistakes turn up in many disappointing projects:

• Tuning to teach a fact, which the model forgets or garbles, when retrieval would have supplied it with a source.
• Training on contradictory examples and expecting the model to resolve the contradiction.
• Evaluating on the training examples.
• Skipping the regression run and discovering a month later that the tuned model got worse at something nobody tested.
• Choosing the largest model by default, when a smaller one tuned well is cheaper to run and fast enough.
• Forgetting the license terms on the base model.
• Leaving nobody in charge of the dataset, so the next round of tuning starts from a pile of files and a vague memory of what was in them.

Questions for anyone pitching fine tuning

Whoever is selling it, ask a few things. What is the baseline, meaning how does the untuned model with a good prompt score on my tasks? What data will you train on, where does it come from, and who reviews it? How will you evaluate, on what held back examples, and will I get the scores? What happens when the base model is updated? Who owns the tuned weights and the dataset at the end? And can the whole thing run in my environment if my data can't leave it? A good answer to each is specific, and a vague answer to the first one usually means there's no baseline at all.

How we approach the decision

Our LLM fine tuning work begins by checking whether prompting or retrieval already solves the problem, and when they do, we say so. When tuning is the right call, the order is assessment, dataset, baseline, tuning with regression checks, then deployment with drift monitoring. When retrieval fits better, that's a RAG system, and our posts on data sources for RAG systems and the privacy implications of RAG systems go deeper on that side. If you're not sure which side your problem sits on, talk to us and bring three examples of the model getting it wrong.