AI evaluations · Lesson 1

What an AI eval can tell you

Estimated time: about 10 minutes

In plain English. AI answers can sound convincing while being wrong. An evaluation, or eval, is a repeatable check using examples that represent a real task. It helps you see what the system handles well, where it fails, and whether a change made it better.

An eval does not reveal whether an AI is “correct” in every possible situation. It checks whether the system meets clear expectations on a chosen set of cases. The choice of cases and expectations is part of the work.

Start with a real task

Suppose you build an assistant that answers questions from a rulebook. “Is the AI good?” is too broad to check. Instead, ask: “When a user asks about a rule, does the assistant give the right ruling and point to the right source?” That question can be turned into examples and observable checks.

Question in the evalWhat a case might containWhat a pass could mean
Can it answer a supported question?Question plus known source and key factsAnswer includes the required rule and cites its source
Does it avoid guessing?A question the reference material cannot answerIt says it cannot determine the answer from the material
Does retrieval find the evidence?Question plus the documents neededAt least one required document appears in the retrieved results

One eval can have many cases. A grader is the rule or reviewer that decides whether a case passed. For an exact requirement, such as “the response must include a citation,” code can check it. For meaning, such as “the answer follows the cited rule,” a written rubric and human review may be needed. A model can also apply a rubric, but its judgments should be checked against human ratings.

One score can hide a failure

Imagine 20 questions: 18 supported questions and 2 that should receive “I don’t know.” The assistant answers all 18 supported questions correctly, but also invents answers to both unsupported questions. A single score of 90% may look good, yet it hides a serious guessing problem.

So report the relevant kinds of success separately. For this example, show supported-answer accuracy and unsupported-question abstention accuracy. Then read the two failing answers. The overall score tells you how often cases passed; the examples help you understand why.

Practice: pick a check that matches the risk

1. Your assistant must return valid JSON with a field named total. What is a strong first check?

Parse the output as JSON and check that total exists. This checks the exact format requirement without asking a language model to judge it.

2. The score is 95%, but users report made-up answers to questions outside the source material. What should you inspect next?

Check results specifically for unsupported questions and inspect the failing examples. The overall score can hide a weakness concentrated in that case type.

A useful first workflow

  1. Write down the user task in one sentence.
  2. Collect representative examples, including difficult cases and cases where the assistant should decline to guess.
  3. For each example, state what a good result must do.
  4. Choose a check that measures that behavior.
  5. Run the system, inspect failures, and add useful failure cases to the dataset.
  6. After changing the system, rerun the same cases to see what changed.

The example set is called an evaluation dataset. It is a measuring instrument, so it can be incomplete or biased. A high score means “this system did well on these cases under these checks.” It does not guarantee correctness on future requests.

Further reading

Start with OpenAI’s Evaluation best practices, which explains how to define success, select cases, choose graders, and improve evaluations over time. For the platform-specific workflow, see Working with evals.

Keep the steps handy in the AI eval design reference. Ask me any follow-up question; we can work through an example from a system you care about.

Practice

Answer before you reveal the explanation.

Quick recall

Before choosing an eval metric, what should you define first?

Say your answer first, then reveal the explanation.

Quick recall

An assistant scores 95% overall but often guesses on questions with no source answer. What should you measure separately?

Say your answer first, then reveal the explanation.

Quick recall

What does a 100% score on an eval dataset establish?

Say your answer first, then reveal the explanation.