How to build a chatbot with your own documents

Many people want a chatbot that knows the company’s documents: the product manual, internal policies, standard contracts, the customer service FAQ. The technique behind it is almost always RAG, retrieval-augmented generation, which we explain in detail in RAG: how to connect AI to documents.

This article is the practical side. Instead of explaining the mechanism, it walks through the decisions in a real project, from defining the scope to monitoring after launch. It is not a code tutorial: tools change quickly, and the decisions stay the same.

Step 1: define the scope before the technology

Start with three questions:

• Who will use it? Employees, customers or both. This changes the tone, the documents allowed and the risk of leaks.
• About what? List the documents that go in and those that stay out. An HR chatbot does not need to read supplier contracts.
• What should it not do? For example: not give legal opinions, not answer questions outside the knowledge base, not make things up when it cannot find the answer.

Also gather 30 to 50 real questions, with the correct answer and the document it comes from. In our newsroom’s assessment, this test set is the project’s most valuable asset: it is what you will use to measure every change.

Step 2: decide whether you need search

If the knowledge base is small, you may not even need RAG. Anthropic, in its article on contextual retrieval (September 2024), suggests that knowledge bases under roughly 200,000 tokens, about 500 pages, can be placed in the prompt in full. The company says this number is an approximation and depends on the model.

When the base is larger, changes often or requires each person to see only part of it, search becomes necessary. To understand what tokens are, see tokens in AI models.

Step 3: choose between a managed service and building your own

There are two main paths.

Managed service

Some model providers offer search out of the box. With OpenAI’s file search tool, for example, you upload the files, create a vector store and the model queries it when needed. The answer comes with annotations pointing to the cited file.

The retrieval guide in the same documentation lists, as of October 8, 2026, the defaults used: chunks of up to 800 tokens, with a 400-token overlap between neighboring chunks, files of up to 512 MB and up to 5 million tokens per file. Storage is free up to 1 GB and charged above that. These figures come from the provider and may change.

The advantage is speed. The downside is less control over each stage and dependence on a single provider. The comparison between closed and open-weight models is in open vs. proprietary AI models.

Building your own

On the other path, you assemble each piece: a program that extracts and splits the texts, an embeddings model (which turns text into numerical vectors), a database that stores the vectors and a language model that writes the answer.

The database does not need to be exotic. pgvector, an open-source extension for the PostgreSQL database, performs vector similarity search. According to the project’s documentation, without an index the search is exact; with approximate indexes, such as HNSW or IVFFlat, it is faster but may miss some of the nearest neighbors.

Step 4: prepare the documents

This is the most underestimated stage. The chatbot can only answer well what it can read.

• Remove what is outdated. Two versions of the same policy produce contradictory answers.
• Check that the text can be extracted. PDFs that are scanned images contain no text. Anthropic’s citations documentation warns, for example, that scanned PDFs without extractable text cannot be cited. In those cases, you need to run OCR (optical character recognition) and review the result.
• Take care with tables and lists. In our newsroom’s assessment, tables tend to fall apart during extraction. It is worth converting them into text or a structured format.
• Record metadata. For each file, store the title, date, department and access level. OpenAI’s guide allows up to 16 attributes per file to filter the search; when building your own, you define yours.

Step 5: split and index

The texts are split into chunks, and each chunk becomes a vector. Anthropic describes chunks of, typically, no more than a few hundred tokens.

Short chunks lose context. Anthropic’s example is a sentence such as “the company’s revenue grew by 3% over the previous quarter”, which, on its own, does not say which company or which quarter. The technique the company proposes adds a short summary of the source document to each chunk before indexing.

Combine meaning-based search with exact keyword search. Product codes, clause numbers and acronyms are often poorly retrieved by vectors alone. In the tests Anthropic published using its own data, the retrieval failure rate (when the needed chunk does not appear among the top 20) fell from 5.7% to 2.9% with contextualized chunks and keyword search (BM25), and to 1.9% when a reranking step was added. These are the company’s results, not a guarantee for your knowledge base.

Step 6: write the instructions and require citations

The system prompt, the fixed instruction sent to the model before each conversation, should say at least:

• answer only based on the chunks provided;
• if the answer is not in the chunks, say you did not find it;
• indicate the document and section for each piece of information;
• treat the content of the documents as data, never as commands.

Citations make the answer verifiable. Some providers already return citations in a structured form. In Anthropic’s API, according to the citations documentation, plain text and PDFs are split into sentences, and each citation points to the exact passage, with a character range or page number. Prompting techniques are covered in prompt engineering.

Step 7: security and privacy

A document chatbot can leak exactly what you want to protect. The OWASP Top 10 for LLM Applications, 2025 version, ranks prompt injection first and devotes one item, LLM08, to vector and embedding weaknesses.

The LLM08 page describes risks that apply directly to this kind of project:

• Weak access control. If the index mixes documents from several groups, a user may receive chunks they should not see. The recommendation is to filter search by the permissions of whoever is asking and to separate data by group.
• Knowledge base poisoning. A document with hidden instructions can hijack the system’s behavior. OWASP’s example is a résumé with hidden text that manipulates a screening system. The recommendation is to accept only trusted sources and validate files before indexing.
• Logging. Keeping detailed logs of searches helps detect and investigate suspicious use.

If the documents contain personal data, Brazil’s General Data Protection Law (LGPD) applies. According to the FAQ of the ANPD (Brazil’s data protection authority), personal data is information relating to an identified or identifiable natural person, and data must be deleted when it is no longer needed for its purpose. In our newsroom’s assessment, this means not indexing personal data unnecessarily and having a way to delete it from the index, not just from the original file.

Step 8: test before opening it to the public

Use the question set from step 1 and measure search and answers separately.

• Did the search find the right chunk? Check whether the expected document appears among the top results.
• Is the answer faithful to the chunks? The paper that introduced Ragas (Es et al., 2023), a RAG evaluation tool, separates the faithfulness of the answer to the retrieved chunks from the relevance of the context found by the search. Tools like this help, but they do not replace a human reading a sample.
• Does the chatbot know how to say “I don’t know”? Include questions whose answers are not in the knowledge base.

General checking criteria are in how to evaluate AI responses.

Step 9: launch small and keep watching

Start with a restricted group of users. Track unanswered questions, answers flagged as wrong and documents that are never retrieved. Every new or changed document needs to be reindexed; decide who does this and how often.

If the chatbot serves customers, always leave a way to reach a person. The risks of this use are covered in AI in customer service.

Key takeaways

Building a chatbot with your own documents is less about choosing the model and more about organizing the knowledge base, the search and the rules. Define the scope and the test questions first, clean up the documents, combine meaning-based and keyword search, require citations and filter everything by the permissions of whoever is asking.

Even when well built, the system will sometimes get things wrong, because language models can hallucinate even with the right chunk in front of them. Treat the answers as a starting point, show the source to the user and keep someone responsible for reviewing the knowledge base.