Glossary ยท AI evaluations

Words used in AI evals

Eval (evaluation)
A repeatable check of how an AI system behaves on examples that represent a task.
Eval set
A collection of inputs and expected behaviors used to check a system.
Expected behavior
A clear description of what a good result should do for one case.
Grader
A rule, program, person, or model that decides whether a result meets the expected behavior.
Rubric
A short set of criteria that a reviewer uses to judge a result.
Metric
A number that summarizes a measured behavior, such as the share of cases answered correctly.
Regression set
Past examples rerun after changes to catch known failures returning.
RAG (retrieval-augmented generation)
A system that searches for relevant source text and gives that text to a language model to help it answer.
Recall at k
The share of expected relevant items found within the first k search results.
Calibration
Checking and adjusting an automatic grader by comparing its judgments with trusted human judgments.

See Eval design for the workflow and Eval case template for writing examples.