Context, memory and information retrieval in AI

When you talk to an AI assistant, it seems to “remember” what was said minutes earlier, sometimes even what you told it weeks ago. Behind that impression are three different mechanisms that are often confused: context, memory and information retrieval.

Understanding the difference helps you see why an assistant forgets details in long conversations, why it sometimes ignores part of a document and what happens to what you tell it.

Three ideas that are often confused

A language model learns patterns during training and stores that learning in its parameters, the internal numbers adjusted during that phase. A 2020 paper calls this parametric memory (Lewis et al., 2020). It is fixed after training: the model doesn’t learn anything new just because you talked to it. More about this process in what LLMs are.

Everything else comes from outside the parameters:

• Context: the text that goes into the current request, including the conversation so far.
• Memory: information saved by a system to be reused in future conversations.
• Information retrieval: an on-the-spot search for relevant passages in a document base, to include them in the context.

Context window: working memory

Anthropic’s documentation defines the context window as all the text a model can reference when generating a response, including the response itself. The company describes it as “working memory,” distinct from training data (Anthropic, context windows).

This space is measured in tokens, pieces of text the model processes. According to the same documentation, everything counts toward the limit: system instructions, every message exchanged, documents, images, tool results and the generated text.

In a conversation, each new question is sent along with the history. The conversation grows until it approaches the limit. What happens then depends on the product. Anthropic states that chat interfaces may drop the oldest messages first. Another option the company describes is compaction: automatically summarizing the older part of the conversation in order to continue.

Window sizes have changed quickly in recent years and vary by model. On the date consulted (October 8, 2026), Anthropic’s documentation mentions windows of up to 1 million tokens, depending on the model. Since this number changes often, check the documentation for the model you use.

More context isn’t always better

A large window doesn’t guarantee the model makes good use of everything in it.

In 2023, Nelson F. Liu and co-authors tested models on two tasks: answering questions from multiple documents and finding a value in a list of key-value pairs. Performance was usually higher when the relevant information was at the beginning or end of the text and dropped significantly when it was in the middle. The effect appeared even in models built for long contexts. The study became known by its title, “Lost in the Middle” (Liu et al., 2023).

Anthropic describes a related phenomenon it calls context rot: as the number of tokens grows, accuracy and the ability to recall details decrease (Anthropic). In an engineering post published in September 2025, the company proposes treating context as “a finite resource” and seeking the smallest set of relevant information that leads to the desired outcome (Anthropic, context engineering).

What this means for you:

• In very long conversations, a detail mentioned early on may be ignored. Repeating the important point close to the question helps.
• When pasting long documents, Anthropic recommends placing the material at the beginning and the question at the end, and asking the model to quote the relevant passages before answering (Anthropic, best practices).
• Pasting “everything, just in case” can make the answer worse. Select what matters.

These practices are covered in more detail in prompt engineering.

Memory across conversations

By default, each conversation starts from scratch. For an assistant to “remember” your preferences the following week, the system needs to save that information somewhere and reinsert it into the context later. That is what products call memory.

Anthropic’s memory tool for developers shows the mechanism transparently. The model can create, read, edit and delete files in a memory directory, and it checks that directory before starting a task. The files live in the storage of the application using the model, not inside the model (Anthropic, memory tool).

Note the logic: memory doesn’t change the parameters. It is stored text that returns to the context window when useful. That is why it also takes up space and is subject to the same attention limits.

The same documentation lists precautions that fall to whoever builds the system: avoid storing sensitive information, limit file sizes, delete old memories and block paths that would allow access to files outside the memory directory.

For you, as a user, the practical consequence is different: what you tell an assistant with memory turned on may resurface in future conversations. Each product has its own controls to view, delete or turn off memory, and they change frequently. It is worth looking for these options in the settings of the service you use.

Information retrieval: searching before answering

And when the information is spread across thousands of documents that don’t fit, or shouldn’t fit, in full in the context? That’s where information retrieval comes in.

The best-known approach is retrieval-augmented generation (RAG). The paper that coined the term combined a generator model with a document index queried by a retriever, which works as a “non-parametric” memory. According to the authors, the answers were more specific and factual than those of the model without search (Lewis et al., 2020).

Broadly speaking, the process has three steps:

1. Documents are split into pieces (chunks) and indexed.
2. When a question arrives, the system searches for the most relevant pieces.
3. Those pieces enter the context along with the question, and the model answers based on them.

The search usually uses embeddings: lists of numbers that represent the meaning of a text. According to OpenAI’s documentation, the distance between two of these vectors indicates how related the texts are; small distances suggest high relatedness (OpenAI, embeddings). Many systems combine this with keyword search, which is useful for codes and exact names.

There is a known problem: an isolated chunk can lose its meaning. Anthropic gives the example of a passage saying that revenue “grew by 3% over the previous quarter” without saying which company or which period. The technique the company proposes, called contextual retrieval, adds to each chunk a brief explanation of where it came from. In Anthropic’s own tests, applying this technique to both the embeddings and keyword search reduced retrieval failures by 49%; with an extra step of reranking the results, the reduction reached 67% (Anthropic, contextual retrieval). These are the company’s figures, measured on its own test sets.

The same post, from September 2024, notes that if the knowledge base is small (under 200,000 tokens, about 500 pages, by the company’s estimate), it may be simpler to put it whole into the prompt. How RAG works in full is covered in RAG: AI with documents.

When to use each strategy

Situation	Most suitable strategy
A document or conversation that fits comfortably in the window	Paste the content into the context, with the question at the end
Preferences and facts that should apply across several conversations	Memory, with control over what is saved
A large base, or one that changes frequently	Information retrieval (RAG)
Long tasks for AI agents	A combination: compaction, saved notes and on-demand search

The last row follows what Anthropic describes for agents: loading information “just in time” through tools, summarizing the history as it grows and keeping notes outside the context window (Anthropic, context engineering).

This table is guidance from our newsroom, not a technical rule. The right choice depends on data volume, cost and how much error is acceptable.

Risks you need to know about

Wrong retrieval becomes a wrong answer. If the search returns the wrong or an outdated passage, the model may answer confidently based on it. It may also mix the passage with what it learned in training. See how to spot this in AI hallucinations and in how to evaluate AI responses.

Retrieved content may contain malicious instructions. A 2023 study showed that attackers can plant instructions in pages or documents a system is likely to retrieve. When processing that content, the model may follow the hidden orders. The authors called this indirect prompt injection and stated that effective defenses were lacking at the time (Greshake et al., 2023). The more a system searches the web or emails, the larger the attack surface.

Memory is stored personal data. Information you share may be saved and resurface. Avoid telling assistants with memory turned on your passwords, banking details or third parties’ health data.

Key takeaways

• The model doesn’t learn from your conversation. It answers with what it learned in training plus what is in the context at that moment.
• A large context is not a well-used context: information in the middle of long texts tends to be used less. Select, organize and put the question at the end.
• Memory and document retrieval are external systems that feed the context. They expand what AI can do, but bring risks of privacy, retrieval errors and instruction injection.