Quick recall
When is a programmatic exact check a better choice than a model grader?
Say your answer first, then reveal the explanation.
When the requirement is exact and observable, such as valid JSON, a required field, or a fixed label.
AI evaluations · Lesson 3
Estimated time: about 10 minutes
Different requirements need different kinds of checks. A strict format can be checked by code. Meaning often needs a rubric and human judgment. A model can apply that rubric at scale, but “a model said it passed” is not proof by itself.
| What must be right? | Good first grader | What to watch for |
|---|---|---|
| Valid JSON with required fields | Parse the JSON and check the fields | A structural check does not prove the values are correct |
| Exact label from a fixed list | Compare with the expected label | Normalize only differences that do not matter |
| Answer follows a source document | Use a written rubric, with human review on a sample | Text similarity can reward matching words while missing a false claim |
| Answer is helpful and complete | Use a task-specific rubric; compare with human judgments | “Helpful” is too vague until you say what a good answer contains |
A rubric is a short set of criteria for judging a result. For a policy answer, one criterion might be: “Does the answer state the exception that applies to this user’s situation?” Give clear pass and fail examples. If two reviewers disagree often, clarify the rule before trusting an automatic grader.
A model grader can help with judgments that are hard to express as simple code. Keep its task narrow, for example “Does the answer mention the policy’s holiday exception? Return yes or no and quote the evidence.” OpenAI’s guidance recommends specific criteria and calibrating automatic scores against human feedback.
Take a small set of outputs that people have already judged. Run the grader on those same outputs, then compare its decisions to the human decisions. If it misses a critical error, change the rubric or use human review for that criterion.
Also look at errors in both directions. A grader can approve a bad answer, or reject a good answer because it uses different wording. A single agreement percentage can hide which of those mistakes is happening, so inspect disagreements and separate them by severity.
1. A response must contain the exact label refund_eligible. Which grader is the best first choice?
2. A model grader says an answer is grounded. How can you check that the grader itself is reliable enough?
See OpenAI’s evaluation guidance for advice on metrics and human calibration. The working with evals guide illustrates grader types in OpenAI’s platform, which is scheduled for retirement later in 2026.
For the compact decision guide, see Eval design and the eval glossary.
Answer before you reveal the explanation.
Quick recall
Say your answer first, then reveal the explanation.
When the requirement is exact and observable, such as valid JSON, a required field, or a fixed label.
Quick recall
Say your answer first, then reveal the explanation.
Compare its decisions with human judgments on representative examples and inspect where they disagree, especially on serious errors.
Quick recall
Say your answer first, then reveal the explanation.
Clarify the criterion and add examples until careful reviewers can apply it consistently; do not rely on an unclear rule as an automatic score.