Artificial intelligence in contract analysis

A 40-page service agreement arrives on a Friday afternoon. Someone suggests: “run it through AI and ask for the points to watch”. In seconds, out comes a neatly organized list. The question that matters is how much of that list is right and what was left out.

This article explains what artificial intelligence does in contract analysis, what the available research shows about its hits and misses, and which confidentiality and data protection concerns come into play. It is for information only and is not legal advice: decisions about a specific contract depend on a qualified professional.

What “analyzing a contract” with AI means

In practice, the expression covers very different tasks, with different levels of risk.

• Locate and extract. Find the termination clause, jurisdiction, term, penalty and confidentiality provisions in the text.
• Summarize. Produce an overview of the document in plain language.
• Compare. Point out differences between two versions, or between the contract received and the company’s standard template.
• Check against a list. Verify whether the document contains the clauses your policy requires.
• Assess and recommend. Say whether a clause is abusive, valid or risky, and what to negotiate.

The first four are, in essence, reading and organizing tasks. The last one is legal interpretation. This distinction runs through the whole article, because that is where the risk moves to another level.

How the tools work

Most current tools use a large language model (LLM), the same kind of technology behind chat assistants (see what LLMs are). The contract is placed in the model’s context along with an instruction, and the model generates the answer.

When there are many documents, retrieval-augmented generation (RAG) comes into play: the system first searches for the relevant passages and only then asks the model to answer based on them. We explain the method in RAG: how to connect AI to documents. Scanned contracts also require reading images, the subject of multimodal AI.

Two technical limits matter here. Long contracts can take up a large part of the context window, and models tend to make poorer use of information located in the middle of long texts (details in context and memory in AI). And the model can confidently generate something that is not in the document, which is called hallucination.

What the studies show

There is no general accuracy rate for “legal AI”. Results depend on the task, the model, the language and the legal system. The studies below are dated and were carried out in specific contexts, generally with documents in English and US law.

Locating clauses is a measurable task

CUAD, a dataset published in 2021, was created precisely to measure machine contract review. It brings together 510 commercial contracts with more than 13,000 annotations made by legal experts from the Atticus Project, covering 41 clause types (Atticus Project; Hendrycks et al., 2021). The task defined by the authors is to highlight the parts of the contract a person should review. At the time, they rated model performance as nascent, with much room for improvement.

The design of the task already says something: the goal was to point out what to review, not to replace the review.

Legal reasoning varies by type of task

LegalBench, from 2023, brought together 162 legal reasoning tasks designed by legal professionals, divided into six types, and evaluated 20 models (Guha et al., 2023). The existence of distinct categories reinforces that “being good at law” is not a single capability: a model can do well at identifying a rule and poorly at applying it to facts.

Comparisons with lawyers

A January 2024 paper compared LLMs with junior lawyers and outsourced review companies in identifying legal issues in contracts, using senior lawyers’ assessment as the benchmark (Martin et al., 2024). The authors state that the most advanced models matched or exceeded human accuracy, took seconds instead of hours and cut costs by 99.97%.

It should be read with caution. The work is a preprint, the authors are affiliated with the Onit AI Centre of Excellence, and the abstract does not detail the sample, the models or how costs were calculated. It is an indication that well-defined identification tasks can be automated with good results, not proof that human review has become unnecessary.

Open legal questions produce more errors

When the task stops being “find it in the document” and becomes “say what the law or case law establishes”, errors increase. A study published in the Journal of Legal Analysis found hallucinations in about 58% of ChatGPT 4’s answers and 88% of Llama 2’s on verifiable questions about US federal court decisions. The models also frequently failed to correct incorrect legal premises embedded in the question (Dahl et al., 2024).

Specialized tools reduce the problem but do not eliminate it. A May 2024 study evaluated AI legal research tools, such as Lexis+ AI and Westlaw’s, and found hallucinations between 17% and 33% of the time, less than a general-purpose chatbot. The authors concluded that vendors’ claims of “hallucination-free” results were overstated (Magesh et al., 2024).

The finding about false premises is especially relevant for contracts. If you ask “is this penalty within the legal limit?”, the answer may assume that an applicable limit exists, without questioning it.

Where AI helps most

In the editorial team’s assessment, based on the studies above, AI delivers the most when the task is well defined and the answer can be checked in the document itself:

• Volume screening. Sorting, among hundreds of contracts, those that contain a given clause, such as automatic renewal or change of control.
• Extraction into a spreadsheet. Building a table with parties, amounts, terms, adjustment indexes and expiration dates.
• Version comparison. Pointing out what changed between the draft sent and the one returned by the other party.
• Internal checklist. Indicating whether the clauses required by company policy are present.
• Plain-language explanation. Helping non-specialists understand the document’s structure before talking to a professional.

In all these cases, the result is a starting point. Checking is quick because you only need to go back to the cited passage.

Where AI should not decide

Assessing whether a clause is valid, whether it is abusive or what to negotiate is legal interpretation. In Brazil, Law No. 8,906/1994, the Statute of the Legal Profession, includes legal consulting, advisory and management among the activities reserved for lawyers (Law 8,906/1994, art. 1).

The Federal Council of the OAB (the Brazilian Bar Association) approved recommendations in 2024 for the use of generative AI in legal practice, released in November of that year (OAB SP, November 27, 2024). The text organizes the guidelines into four pillars and states, among other points (text of the recommendation, OAB):

• that professional judgment should not be exercised by generative AI systems without human supervision, and that no activity reserved for lawyers should be delegated to them;
• that excessive reliance on these tools cannot replace the analysis carried out by the lawyer;
• that transparency with the client about the use of generative AI is recommended.

The recommendations are addressed to lawyers, but the reasoning applies to any company: the tool can prepare the ground, and the legal conclusion rests with whoever is qualified and accountable for it.

Confidentiality and data protection

Contracts usually contain personal data (names, CPF taxpayer numbers, addresses, salaries) and sensitive commercial information. Sending them to an AI tool is a form of data processing, and some questions need to be answered first.

What the LGPD requires. Brazil’s General Data Protection Law (Law No. 13,709/2018) establishes, among others, the principles of purpose, necessity (limiting processing to the minimum necessary) and security. It also requires processing agents to adopt technical and administrative measures to protect data from unauthorized access and from accidental or unlawful situations (LGPD, arts. 6 and 46). If the tool processes data outside the country, the international transfer rules of art. 33 also come into play. Since each case depends on who the parties are, what data exists and what the contract with the vendor says, the concrete application should be assessed by the organization’s legal or data protection officer.

What the OAB recommends. For lawyers, the guidelines call for attention to data that could make the client identifiable, diligence in choosing the vendor (protection of information, security measures and the option of not using submitted data to train the system) and checking the data-sharing purposes set out in the privacy policy. The text itself notes that generative AI tools may use prompts and uploaded documents for training and even share them with third parties.

In practice, before sending a contract, check:

1. Whether the tool is approved by your organization and under what contract it operates.
2. Whether submitted data can be used to train models, and whether that can be turned off.
3. Where the data is processed and how long it is kept.
4. Whether personal data and amounts can be removed or masked before sending, without compromising the analysis.
5. Whether the contract has a confidentiality clause restricting sharing with third parties, including technology vendors.

A safer workflow

1. Define the task precisely. “List the termination clauses, with number and verbatim text” is better than “analyze the contract”.
2. Ask for verbatim passages before conclusions. With the quote in hand, you can check in seconds whether the clause exists and says what the AI claims.
3. Do not ask what the law says without a source. If the answer depends on legislation or case law, it must come from the official source or the professional, not from the model’s memory.
4. Test before adopting. Use contracts you already know well to see what the tool gets right and what it misses. We show how to set up this check in how to evaluate AI responses.
5. Record the use. Note which tool was used, for what and who reviewed it. This makes accountability and error correction easier.
6. Leave the decision with whoever is accountable for it. AI prepares; the qualified professional concludes.

A note on vocabulary: a “smart contract” is something else. The term refers to programs that execute rules on a blockchain, not to contract-reading tools. We explain the difference in smart contracts.

Key takeaways

• AI is useful for locating, extracting, summarizing and comparing clauses, tasks in which the answer can be checked in the document itself.
• Studies show good results on well-defined tasks and significant error rates on open legal questions, including in specialized tools.
• Interpreting the validity of a clause and guiding a negotiation is legal analysis. In Brazil, legal consulting is an activity reserved for lawyers, and the OAB recommends human supervision when using AI.
• Sending a contract to an AI tool involves confidentiality and the LGPD. Check the vendor, the use of data for training and what can be masked beforehand.
• Always ask for the verbatim passage and check it. A well-formatted list is no guarantee that it is complete or correct.