Fine-tuning or RAG: when to use each approach

When a company wants AI to “know” its products, policies or customers, two proposals soon appear. One is to train the model on the company’s data, so-called fine-tuning. The other is to leave the model as it is and connect it to the documents, the technique known as RAG.

The two touch different things. This article explains what each one does, what published research says about them and how to decide, in practice, where to start.

What each approach changes

A language model is, in essence, a huge set of numbers adjusted during training. These numbers are the parameters. Everything the model “knows” permanently is stored in them. If you want to review the basics, see what LLMs are.

Fine-tuning is continuing the training of an existing model with a smaller set of examples you choose. At the end, some of the parameters change, and the model starts answering differently even without extra instructions in the prompt.

RAG (retrieval-augmented generation) does not touch the parameters. With each question, the system searches for relevant passages in your documents and places them in the prompt, along with the question. The model answers by reading this material. The mechanism is detailed in RAG: how to connect AI to documents.

A simple way to remember the difference:

• Fine-tuning changes how the model answers.
• RAG changes what the model has at hand to answer.

What research says about teaching facts

The most common question is: “if I train the model on my documents, will it know their content?” Two studies help answer it.

Direct comparison between fine-tuning and RAG

In the paper Fine-Tuning or Retrieval?, by Oded Ovadia and co-authors (December 2023), the researchers compared the two techniques for “injecting knowledge” into three 7-billion-parameter models (Llama 2, Mistral and Orca 2). The fine-tuning used was unsupervised: the model simply continued training on the texts.

The tests used multiple-choice questions on topics such as anatomy and astronomy and, in a second set, on events from August to November 2023, after the models’ training. According to the authors, RAG consistently outperformed fine-tuning, both for knowledge the model had already seen and for new knowledge.

On recent events, the difference was large. Mistral 7B, for example, scored 48.1% unaided, 50.4% after fine-tuning and 87.5% with RAG. Combining the two (fine-tuned model with RAG) came in below RAG alone on this test set.

There was a useful nuance. When the researchers trained the model with several paraphrases of the same content (ten versions of each passage), fine-tuning improved. The authors treat this result as preliminary, but it suggests the model needs to see the same fact written in many ways to absorb it.

New facts and hallucination

The second study, Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, by Zorik Gekhman and co-authors (May 2024, accepted at the EMNLP 2024 conference), looked at the problem from another angle. The authors varied the proportion of training examples with facts the model did not yet know.

The reported results:

• The model learns examples with new facts much more slowly than examples consistent with what it already knows.
• When it finally learns these new facts, its tendency to hallucinate, that is, to make up answers, increases linearly.

The authors’ conclusion is that models acquire factual knowledge mainly in pre-training (the initial phase, with enormous volumes of text), and that fine-tuning mostly teaches them to make better use of what is already there.

These are two studies, with specific models and tests. They do not settle the matter. But they point in the same direction: if the goal is for the model to answer based on your company’s facts, training on those facts is a weak and risky path.

What fine-tuning is good for

This does not make fine-tuning useless. It shines on another type of problem: behavior.

OpenAI’s Model optimization page, consulted on October 8, 2026, lists as good uses of supervised fine-tuning:

• text classification;
• nuanced translation;
• content in a specific format;
• correcting failures to follow instructions;
• tone and style.

The same page cites other gains: responses with more consistent formatting, shorter prompts (because the model already “knows” what to do without so many examples) and the possibility of getting a smaller, cheaper model for a specific task.

A concrete example: a team that classifies thousands of support tickets a day into twenty internal categories. The categories do not change every week, there is a large history of tickets already classified by people, and what matters is always answering in the same format. This is the typical profile where fine-tuning can pay off.

OpenAI itself also says, on the same page, that prompt engineering may be all you need to get good results. Before training anything, it is worth exhausting what can be done with well-written instructions; see prompt engineering.

A market signal

In the same consultation, OpenAI’s page states that the company is shutting down its fine-tuning platform: the service does not accept new users, current users can create training jobs “in the coming months” and already fine-tuned models remain available until the base models are deprecated. This is one vendor’s situation in October 2026, not an industry rule. Other providers and open models continue to offer fine-tuning.

In the newsroom’s assessment, the episode illustrates a practical risk: a fine-tuned model is tied to the base model and the vendor. When the base is retired, the training work has to be redone. A RAG system is usually easier to move to another model, because the knowledge stays in the documents.

Cost, effort and maintenance

The two approaches have costs of a different nature.

Fine-tuning

The cost is concentrated before use: building a quality set of examples, running the training, evaluating and repeating. And every time the information changes, you need to train again.

Techniques such as LoRA have greatly reduced the computational cost. In the LoRA paper, by Edward Hu and co-authors (2021), instead of changing all parameters, the method freezes the model and trains small additional matrices. The authors report, compared with full fine-tuning of the 175-billion-parameter GPT-3, a 10,000-fold reduction in the number of trainable parameters and about a threefold reduction in the GPU memory required. QLoRA, by Tim Dettmers and co-authors (2023), went further and made it possible to fine-tune a 65-billion-parameter model on a single 48 GB GPU.

These advances lower the hardware bill. They do not lower the work of preparing good data, which is usually the most time-consuming part.

RAG

The cost is spread across usage. You need to build and maintain the document index, and each question carries extra passages in the prompt, which means more tokens billed per answer.

On the other hand, updating knowledge means replacing or adding documents. The original RAG paper, by Patrick Lewis and co-authors (2020), already pointed to this advantage and to that of indicating the source of each answer, two hard problems for models that rely only on internal memory.

When to combine the two

The approaches are not mutually exclusive. The RAFT method, described by Tianjun Zhang and co-authors (2024), uses fine-tuning precisely to improve the use of RAG: the model is trained to ignore retrieved documents that do not help and to quote verbatim the passage that supports the answer. The authors report gains over the baseline approaches on datasets of medicine (PubMed), multi-hop questions (HotpotQA) and API usage (Gorilla).

Note the division of roles: knowledge stays in the documents, and fine-tuning teaches a skill, that of reading those documents well. This is consistent with what the other studies indicate.

Remember also that, in Ovadia’s study, combining the two techniques did not improve the result compared with RAG alone. Combining is only worthwhile when each part solves a problem you have measured.

A roadmap to decide

In the newsroom’s assessment, the order below avoids most waste. It is not a universal rule, and each case calls for testing.

1. Start with a good prompt. Clear instructions, examples and a defined output format solve many cases.
2. If the problem is missing information, use RAG. Internal documents, data that changes, the need to cite the source or to control who sees what point to retrieval.
3. If the knowledge base is small, test putting all of it in the context. The topic is covered in context and memory in AI.
4. If the problem is behavior, consider fine-tuning. A format the model keeps getting wrong, a very specific tone, high-volume classification, or the need for a smaller, cheaper model.
5. Measure before and after. Build a set of test questions with known answers and compare the alternatives. Criteria in how to evaluate AI responses.

A few questions help classify your case:

• Does the information change frequently? That favors RAG.
• Do you need to show where the answer came from? That favors RAG.
• Do you have hundreds or thousands of examples of correct inputs and outputs? Without that, fine-tuning is unlikely to pay off.
• Is the model’s error about content or form? Content calls for RAG; form may call for fine-tuning.

Risks that apply to both

• Sensitive data. In fine-tuning, the training data comes to influence the model and, if training is done by a vendor, leaves your environment. In RAG, it sits in an index that must respect access permissions.
• Quality of the base. Wrong documents produce wrong answers in RAG; bad examples produce a bad fine-tuned model.
• Insufficient evaluation. Without a test set, there is no way to know whether the change improved or worsened the system.

Key takeaways

Fine-tuning and RAG solve different problems. RAG gives the model access to information it does not have, with simple updates and a traceable source. Fine-tuning adjusts behavior: format, tone, repetitive tasks and cost per answer.

The studies cited indicate that training the model on new facts is inefficient and can increase hallucinations. That is why, for most projects that want “the AI answering about our documents,” the safest path is to start with a good prompt, move to RAG and leave fine-tuning for when there is a clear, measured behavior problem.