RAG: how to connect artificial intelligence to documents

A language model learns from a large volume of text up to a certain date. After that, its knowledge is frozen. And it has never seen your company’s internal rules, the contract you signed yesterday or this week’s price sheet.

The most widely used technique to fill this gap is called RAG. Instead of hoping the model knows the answer, the system first looks for the relevant passages in your documents and only then asks the model to answer based on them. This article explains how it works, where the idea came from and where it usually fails.

What RAG is

RAG stands for retrieval-augmented generation. “Retrieval” here means search: finding, in a collection of documents, the passages that help answer a question. “Generation” is the text the model produces.

The term was introduced in the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and co-authors, posted on arXiv in May 2020 and accepted at that year’s NeurIPS conference. The authors start from a diagnosis: pretrained models store factual knowledge in their parameters, but access and manipulate that knowledge imprecisely. Moreover, it is hard to know where an answer came from and to update what the model knows.

The paper’s proposal is to combine two kinds of memory:

• Parametric memory: what the model learned and stored in its parameters (the internal numbers adjusted during training).
• Non-parametric memory: an external document index that can be queried and swapped without touching the model.

If you want to review how a model generates text, see what LLMs are and how generative AI works.

How it works, step by step

Details vary between systems, but the basic structure is similar. The description below follows the original paper and Anthropic’s post on contextual retrieval, from September 2024, which describes a typical RAG pipeline.

1. Split the documents into chunks

Documents are broken into smaller pieces, called chunks. Anthropic mentions chunks of a few hundred tokens (the text units the model processes; see tokens in AI models). In the 2020 paper, the authors used the December 2018 Wikipedia, split into 100-word blocks, for a total of 21 million passages.

2. Turn each chunk into a vector

Each chunk goes through an embeddings model. An embedding is, according to the OpenAI documentation, a list of floating-point numbers that represents the text. The distance between two of these vectors indicates how related the texts are: a small distance suggests high relatedness; a large distance, low relatedness. The same documentation recommends cosine similarity as the measure.

The vectors are stored in an index, often called a vector database. In the original paper, the index was built with the FAISS library, designed for fast searches of this kind.

3. Retrieve the chunks most similar to the question

When you ask a question, it is also turned into a vector. The system searches the index for the closest chunks and keeps the top results. In the 2020 paper, that was 5 or 10 documents per question.

Vector search captures meaning but can miss exact terms. Anthropic gives the example of an error code such as “TS-999”. That is why many systems combine vector search with BM25, an older method that matches words literally, and merge the two sets of results.

4. Rerank (optional)

Some systems add a reranking model, the reranker, which reassesses the candidates and keeps only the most relevant ones. Anthropic describes a pipeline that retrieves about 150 chunks and sends the top 20 to the model. The company notes that this improves accuracy but adds time and cost.

5. Generate the answer

The selected chunks go into the prompt, together with your question and an instruction along the lines of “answer based on the documents below and indicate where you took each piece of information from”. The model then writes the answer.

What the original paper showed

In the 2020 paper, RAG was tested on tasks that require a lot of factual knowledge. The authors report state-of-the-art performance on the open-domain question answering tasks evaluated, outperforming models that relied only on parametric memory.

In a human evaluation of Jeopardy-style question generation, raters judged RAG’s text more factual in 42.7% of cases, against 7.1% in which the model without retrieval (BART) was judged more factual.

The most instructive experiment was the index swap. The authors built an index from the 2016 Wikipedia and compared it with the 2018 one, asking who held world leader positions that changed in that interval. With the 2016 index, the system got 70% of the 2016 leaders right; with the 2018 index, 68% of the 2018 ones. With the “wrong” index for the period, accuracy dropped to 12% and 4%. The authors’ conclusion: you can update the system’s knowledge by swapping the documents, without retraining the model.

A word of caution. In the paper, the retrieval component and the generator model were trained together. In many current applications, such as the one Anthropic describes, the model is not retrained: retrieval is built separately, and the chunks found are simply placed in the prompt.

RAG or put everything in the context?

Recent models accept very long prompts. So the question arises: why not paste all the documents at once?

For small bases, that can make sense. Anthropic suggests that if the knowledge base is smaller than about 200,000 tokens (something like 500 pages), it can be included whole in the prompt, using caching to reduce cost and latency. That number reflects the company’s guidance in 2024 and depends on the model used.

A long context, however, does not guarantee careful reading. The study Lost in the Middle, by Nelson Liu and co-authors (2023), showed that the performance of the models tested is usually higher when the relevant information is at the beginning or end of the input, and drops when it is in the middle. The authors conclude that the models evaluated do not reliably use all the information in long contexts. The topic is covered in context and memory in AI.

In the editorial team’s assessment, the choice is practical: small, stable bases fit in the context; large bases that change often or require access control call for RAG.

Where RAG fails

RAG reduces the problem of unsupported answers but does not eliminate it. These are the most common points of failure.

Retrieval brings back the wrong chunk

If the right chunk is not retrieved, the model has no way to use it. Anthropic shows a typical case: a chunk saying “the company’s revenue grew by 3% over the previous quarter” loses its meaning when separated from the document, because it does not say which company or which quarter.

In Anthropic’s own tests, the retrieval failure rate (when the needed chunk does not appear among the top 20) was 5.7% with the traditional method. Adding a context summary to each chunk, combining it with BM25 and using reranking reduced that rate by 67%, to 1.9%. These are results published by the company using its own data, not a general measure of RAG.

The model ignores or distorts the right chunk

Even with the correct document in the prompt, the model can get things wrong, mix up information or fill in with something made up. This is known as hallucination. The “lost in the middle” effect described above also applies when many chunks are sent at once.

The documents are wrong or outdated

RAG answers based on what it finds. If the base holds old versions of a policy, or two documents that contradict each other, the answer inherits the problem. In the editorial team’s assessment, keeping the base organized is as important as choosing the model.

Documents can contain attacks

A retrieved document is text that goes into the prompt. The paper by Greshake and co-authors (2023) describes indirect prompt injection: an attacker plants instructions in data likely to be retrieved by the application, without needing direct access to the system. If your base accepts third-party files, such as résumés, emails or web pages, this risk is real.

Permissions and privacy

In the editorial team’s assessment, retrieval must respect who can see what. If the index mixes documents from every department, an innocent question can surface a confidential passage.

Cost

Every question now includes the retrieved chunks, and more text in the prompt means more billed tokens. Extra steps, such as reranking, add time and cost, as Anthropic notes.

How to evaluate a RAG system

A good practice is to measure the two parts separately:

• Retrieval: for a set of test questions with known answers, does the right chunk appear among the top results? This is the kind of measure Anthropic uses in the tests cited.
• The answer: is the final text correct, faithful to the chunks and citing the right source?

Always ask the system to indicate which document each piece of information came from, and spot-check it. Criteria for this check are in how to evaluate AI responses. How you instruct the model also matters; see prompt engineering.

Key takeaways

RAG is a way of giving the model access to documents it does not know: retrieving the relevant passages for each question and handing them over along with the prompt. The idea was formalized in 2020 and showed a clear advantage: updating knowledge by swapping documents instead of retraining the model.

The technique depends on a chain of steps, and each one can fail: how texts are split, retrieval, the model’s reading and the quality of the base itself. Treat the answers as a good starting point, check the cited sources and protect the base against malicious content.