Glossary ยท AI evaluations
Words used in AI evals
- Eval (evaluation)
- A repeatable check of how an AI system behaves on examples that represent a task.
- Eval set
- A collection of inputs and expected behaviors used to check a system.
- Expected behavior
- A clear description of what a good result should do for one case.
- Grader
- A rule, program, person, or model that decides whether a result meets the expected behavior.
- Rubric
- A short set of criteria that a reviewer uses to judge a result.
- Metric
- A number that summarizes a measured behavior, such as the share of cases answered correctly.
- Regression set
- Past examples rerun after changes to catch known failures returning.
- RAG (retrieval-augmented generation)
- A system that searches for relevant source text and gives that text to a language model to help it answer.
- Recall at k
- The share of expected relevant items found within the first k search results.
- Calibration
- Checking and adjusting an automatic grader by comparing its judgments with trusted human judgments.
See Eval design for the workflow and Eval case template for writing examples.