Quick reference · AI evaluations
A useful eval case, one field at a time
Copy this structure when adding an example to an evaluation set. Keep the expected result specific enough that another person can judge it consistently.
{
"id": "stable-name-for-this-case",
"input": "The request and any needed context",
"expected_behavior": "What a good system must do",
"must_include": ["required fact or action"],
"must_not": ["harmful or unsupported behavior"],
"case_type": "ordinary | edge | unsupported | adversarial",
"source": "Where the expected answer came from"
}
Before you add a case
- Can you explain why this case matters to a real user?
- Is the expected behavior supported by a trusted source or domain reviewer?
- Could two careful reviewers agree whether the output passes?
- Does the set already contain a near-duplicate?
- Is this case for development and regression checks, or reserved for a final comparison?
A case can test one or several requirements, but make the requirements explicit. An ambiguous case with no defensible expected behavior can make your score less trustworthy.