AI evaluations · Lesson 5

Turn failures into progress

Estimated time: about 10 minutes

In plain English. An eval is most useful when it changes what you do next. Treat failures as clues: inspect the examples, find the likely cause, fix that cause, and rerun the same checks.

A failed case is not just a lower score. It is a concrete example of where the system missed an expectation. Review the input, the system’s intermediate steps when available, the output, and the grader’s reason for marking it wrong.

Find the part that failed

Suppose a document assistant gives a wrong answer and cites a source. The failure could come from a missing document, a poor retrieval result, an incorrect reading of the source, or a grader that misunderstood the answer. Each cause calls for a different change.

What you observeWhat it may meanWhat to check next
Expected source is absentRetrieval or indexing problemQuery, indexed text, and retrieved passages
Source is present but the answer contradicts itGeneration or instruction-following problemPrompt, context, and the exact unsupported claim
Answer seems acceptable but grader says failRubric or grader problemCompare with human judgment and clarify the criterion
Provider request failed or timed outService reliability issueReport it separately from answer quality

Keep a simple improvement loop

  1. Run the eval set and save the version of the system and settings.
  2. Inspect failures and group them by likely cause.
  3. Make one focused change based on the evidence.
  4. Rerun the full set, not only the case that failed.
  5. Check that the target behavior improved and important other behaviors did not regress.
  6. Add a useful, reviewed failure case to the regression set.

A regression set is the collection of past cases you rerun after changes to catch previously fixed problems returning. Keep some new or held-back examples for a less biased comparison when making bigger decisions. If you tune repeatedly against every case, the score can become a measure of how well you fit those cases rather than how well the system handles new requests.

Keep checking as the system changes

AI results can vary, and real user requests change over time. Run the checks after meaningful changes to prompts, models, tools, or data. When new failure patterns appear, add good examples. Watch live quality signals and human feedback too; a fixed eval set cannot predict every new situation.

OpenAI’s best-practices guide describes evaluation as an ongoing process and recommends using human feedback to calibrate automatic scoring. The same idea applies whether your checks run by hand, in a command line, or in continuous integration.

Practice: choose a next step

Scenario. After a prompt change, the assistant stops making unsupported claims but now refuses many answerable questions. What is the next useful action?

Inspect the new refusal failures, compare them with the system’s prior behavior, and adjust the prompt based on the cases. Then rerun the full eval set to check both unsupported-question behavior and answerable-question accuracy.

Further reading

See the sections on iteration and ongoing evaluation in OpenAI’s evaluation best practices. The platform-specific Evals API is being deprecated, so use the guidance here as a process, not a requirement to use that API.

Review the eval design checklist and case template.

Practice

Answer before you reveal the explanation.

Quick recall

Why should a timeout be reported separately from an incorrect AI answer?

Say your answer first, then reveal the explanation.

Quick recall

After fixing one failed eval case, why rerun the full regression set?

Say your answer first, then reveal the explanation.

Quick recall

What should happen to a useful, confirmed failure example after you diagnose it?

Say your answer first, then reveal the explanation.