Quick reference · AI evaluations

Design an eval that answers a real question

An evaluation (eval) checks how an AI system behaves on examples chosen to represent a real task.

  1. Name the task. What should the system do for a user?
  2. Choose cases. Include ordinary requests, edge cases, and cases where the right behavior is to refuse or ask a question.
  3. Write pass criteria. Describe observable properties of a good result, not a vague goal like “be smart.”
  4. Choose a grader. Use exact checks for exact requirements, a rubric for judgment, or human review when errors matter or graders are uncertain.
  5. Inspect failures. Read examples behind the scores. Group errors by cause before changing the system.
  6. Repeat. Rerun the same cases after changes, then add new cases for failures you discover.

Keep different questions separate

QuestionExample measurementWhat it cannot prove alone
Did retrieval find the needed source?Recall at kThat the final answer is correct
Did the answer follow the evidence?Grounding or citation checksThat the evidence itself is true or complete
Did it answer the user’s actual question?Task-specific rubric or human reviewThat it will work on every future request
Did the service work reliably?Failure rate and latencyAnswer quality

Return to Lesson 1.