AI evaluations · Lesson 3

Choose a grader that fits

Estimated time: about 10 minutes

In plain English. A grader is the check that decides whether an AI result meets an expectation. Use the simplest check that can judge the thing you care about, and verify the grader when its judgment is uncertain.

Different requirements need different kinds of checks. A strict format can be checked by code. Meaning often needs a rubric and human judgment. A model can apply that rubric at scale, but “a model said it passed” is not proof by itself.

What must be right?Good first graderWhat to watch for
Valid JSON with required fieldsParse the JSON and check the fieldsA structural check does not prove the values are correct
Exact label from a fixed listCompare with the expected labelNormalize only differences that do not matter
Answer follows a source documentUse a written rubric, with human review on a sampleText similarity can reward matching words while missing a false claim
Answer is helpful and completeUse a task-specific rubric; compare with human judgments“Helpful” is too vague until you say what a good answer contains

Write a rubric that can be applied

A rubric is a short set of criteria for judging a result. For a policy answer, one criterion might be: “Does the answer state the exception that applies to this user’s situation?” Give clear pass and fail examples. If two reviewers disagree often, clarify the rule before trusting an automatic grader.

A model grader can help with judgments that are hard to express as simple code. Keep its task narrow, for example “Does the answer mention the policy’s holiday exception? Return yes or no and quote the evidence.” OpenAI’s guidance recommends specific criteria and calibrating automatic scores against human feedback.

Check the grader before trusting its score

Take a small set of outputs that people have already judged. Run the grader on those same outputs, then compare its decisions to the human decisions. If it misses a critical error, change the rubric or use human review for that criterion.

Also look at errors in both directions. A grader can approve a bad answer, or reject a good answer because it uses different wording. A single agreement percentage can hide which of those mistakes is happening, so inspect disagreements and separate them by severity.

Practice: match the check to the requirement

1. A response must contain the exact label refund_eligible. Which grader is the best first choice?

A simple exact comparison against the expected label. A model judge would add needless uncertainty to an exact requirement.

2. A model grader says an answer is grounded. How can you check that the grader itself is reliable enough?

Compare its judgments with human judgments on a set of representative answers, inspect disagreements, and refine the rubric or keep human review for risky cases.

Further reading

See OpenAI’s evaluation guidance for advice on metrics and human calibration. The working with evals guide illustrates grader types in OpenAI’s platform, which is scheduled for retirement later in 2026.

For the compact decision guide, see Eval design and the eval glossary.

Practice

Answer before you reveal the explanation.

Quick recall

When is a programmatic exact check a better choice than a model grader?

Say your answer first, then reveal the explanation.

Quick recall

What evidence should you collect before relying on a model grader for a meaningful judgment?

Say your answer first, then reveal the explanation.

Quick recall

What should you do if two reviewers often disagree about whether answers pass a rubric?

Say your answer first, then reveal the explanation.