How to evaluate the quality of AI responses

You ask an AI assistant a question and get back fluent text, organized into bullet points, with a confident tone. It looks great. But looking great says nothing about being right.

This article shows how to evaluate AI responses deliberately. First, for people who use a chatbot day to day. Then, for those who put AI into a work process and need to measure quality more rigorously.

Why appearances deceive

A large language model (LLM) generates text by predicting the most likely continuation based on what it learned from large volumes of data. If you want to understand this mechanism, see what LLMs are and how generative AI works.

The central point for evaluation is this: the same process that produces correct sentences also produces wrong ones, with the same fluency. The generative AI profile from NIST, the US standards institute, describes this risk as confabulation: the production of confidently stated but erroneous or false content, popularly known as “hallucination” (NIST AI 600-1, July 2024). The document notes that the risk arises when users believe the content precisely because of its confident tone.

The same NIST text offers a useful warning for evaluators: the model may present confabulated justifications and even citations to support a wrong answer. In other words, a response “with sources” still needs its sources checked. The topic is explored further in our article on AI hallucinations.

There is also an explanation for why models tend to “guess.” A paper by Adam Tauman Kalai and co-authors, published in September 2025, argues that many performance tests award points only for the right answer and zero for “I don’t know.” Under that system, guessing pays off more than admitting doubt (Kalai et al., 2025). In practice, this means you should not treat the absence of caveats as a sign of reliability.

The criteria that matter

Before judging a response, define what “good” means for that task. Anthropic’s documentation on evaluation recommends criteria that are specific, measurable, achievable and relevant, and notes that most use cases require evaluating several dimensions at once (Anthropic, “Define success criteria and build evaluations”). Among the criteria listed there are task fidelity, consistency, relevance and coherence, tone, privacy, use of the provided context, response time and cost.

Academic research follows the same line. Stanford’s HELM project proposed evaluating language models on seven metrics, not just accuracy rate: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency (Liang et al., 2022). Calibration is the extent to which the confidence a system expresses matches its actual chance of being right.

For everyday use, our newsroom proposes condensing these references into six questions:

1. Accuracy. Are the facts, numbers, dates and names correct?
2. Grounding. Do the important claims have a verifiable source, and does the source actually say that?
3. Adherence to the request. Does the response do what you asked, in the format and scope you asked for?
4. Coherence. Are the parts of the response consistent with one another and with the material you provided?
5. Completeness. Is something essential missing, such as an exception, a risk or a condition?
6. Honesty about uncertainty. Does the response indicate what it doesn’t know, or does it state everything in the same tone?

Quick checklist for daily use

Use this list whenever the response will inform a decision, even a small one.

• Separate what is verifiable. Mentally underline names, numbers, dates, quotes and references. These are the points most likely to be wrong and the easiest to check.
• Open the sources. If the response cites an article, law or page, open the link and check that it exists and supports the claim. Be suspicious of references you cannot locate.
• Check dates. Ask yourself whether the information might have changed. Prices, product versions, rules and rankings change quickly.
• Ask the same thing again. Ask the question again, or rephrase it. If the details change with each attempt, treat them as suspect. This is the principle behind the SelfCheckGPT method: when the model makes things up, sampled responses tend to “diverge and contradict one another,” according to the authors (Manakul et al., 2023).
• Ask about uncertainty. Ask the model to point out what it doesn’t know or what needs verification. This doesn’t guarantee accuracy, but it helps locate the weak points.
• Compare with the source material. If you provided a document and asked for a summary, check that every claim in the summary is in the document. Additions that aren’t there are a common type of error.
• Weigh the cost of being wrong. How much does an error cost? In health, law, finance or security, the AI’s response is a starting point, not the final word. Seek an official source or a professional.

A note about questions: how you ask influences the quality. Vague requests produce vague answers. To improve what goes in, see our guide to prompt engineering.

When AI becomes part of a work process

If AI is going to answer customers, summarize contracts or classify tickets, case-by-case checking is not enough. You need an evaluation (or eval): a set of tests that measures quality in a repeatable way.

OpenAI’s evaluation best practices guide describes a five-step cycle: define the objective, collect a dataset, define metrics, run and compare, and evaluate continuously (OpenAI, “Evaluation best practices”). The same guide warns against “vibe-based” evaluations, of the “it seems to be working” kind.

In practice, this translates into a playbook:

1. Write down the success criterion. Instead of “good responses,” something like “the summary cites the contract’s term, penalty and jurisdiction, without adding clauses.”
2. Build a set of real examples. Use cases from your daily work, including difficult and rare ones. For each, record what an acceptable response would be.
3. Choose how to score. Some things an automated rule can check (format, presence of a field, exact match). Others require judgment.
4. Run before and after every change. Did you switch the model, the prompt or the supporting documents? Run the same set and compare.
5. Add the errors that show up. Every failure found in real use becomes a new test case.

The NIST profile points in the same direction, noting that confabulation risk warrants special monitoring when AI is integrated into applications involving consequential decisions (NIST AI 600-1).

Who evaluates: people, rules and AI judges

There are three main ways to score responses, according to OpenAI’s guide.

• Automated metrics. Exact text comparison, presence of keywords, format validation. They are cheap and good at detecting regressions, but they miss nuance.
• Human evaluation. The highest quality, but slow and expensive. Evaluators also disagree with one another, so the guide suggests pass/fail criteria and consensus voting.
• AI as a judge. A second model evaluates the first model’s response. It is cheaper and scales better.

The AI-as-judge technique has an empirical basis, but also known limits. A 2023 study found agreement above 80% between GPT-4 as a judge and human evaluators, a level comparable to agreement among humans (Zheng et al., 2023). The same study documented biases: preference based on the position in which a response appears, for longer responses and for the model’s own responses.

That is why OpenAI’s guide recommends validating the AI judge against human labels before relying on it. In our newsroom’s assessment, a good starting point is to ask the judge for a simple verdict (pass or fail, with a reason) and to check a sample manually.

What public rankings say and what they don’t

Rankings and benchmarks help compare models under standardized conditions. They don’t tell you how a model will perform on your task.

A classic example is TruthfulQA, published in 2021: 817 questions across 38 categories such as health, law and finance, designed to elicit answers that repeat popular false beliefs. At the time, the best model tested was truthful on 58% of questions, compared with 94% for humans, and the largest models were generally the least truthful (Lin et al., 2021). These figures describe 2021 models, not current ones. The study’s value today lies in its methodological lesson: a model can be highly capable and still reproduce common errors present in its data.

Furthermore, as Kalai and co-authors argue, many tests reward guessing. A model with a high score may have got there by venturing answers that a more cautious system would leave open (Kalai et al., 2025). A ranking measures what the test measures, and nothing more.

Red flags

Some patterns call for extra checking:

• references with a title, author or link you cannot find;
• very precise numbers with no stated origin;
• responses whose details change with each new attempt;
• immediate agreement with a premise of yours that may be wrong;
• claims about recent events without a date;
• a summary that includes something not in the original document.

If the tool works with your documents, it is also worth understanding how RAG and AI with documents and context and memory in AI work, because many errors stem from what the model received, or failed to receive.

Key takeaways

• Fluency and confidence are not evidence of accuracy. Evaluate facts, sources and coherence, not tone.
• Define what a good response is before looking at it, with criteria specific to your task.
• In daily use, apply the checklist: separate what is verifiable, open the sources, repeat the question and weigh the cost of error.
• In work processes, replace impressions with repeatable tests, run after every change, and validate any AI judge against human evaluation.
• Public rankings are a starting point, not a verdict on your use case.