Evals measuresystem behavior onrepeatable cases.
Evals:same realistic test,different AI version.
An eval runs a versioned system on representative cases and applies scoring rules chosen before the results are known. It supports comparison, regression detection, and release decisions.An eval gives different versions the same realistic test and scores them by rules chosen before anyone sees the result.
- REPEATABLEsame test, new candidatesame test for each version
- REPRESENTATIVEREALISTICcases resemble the job
- EXPLICITCLEARthe scoring rule is written down
The overall score passes while the money-moving slice fails.
An eval compares system versions under fixed conditions.An eval is the same realistic test for each version.
It tests system behavior on representative cases with an explicit scoring rule tied to the product’s actual job.It repeats realistic examples and scores them by the same rule. One impressive demo cannot tell you whether a new version is reliably better.
Define the behavior that mattersSay what a good result must do
Collect representative situationsCollect realistic situations
Run a versioned candidateRun the version you want to test
Apply an explicit ruleJudge every result by the same rule
Compare baseline and slicesCompare it with the current version
Changing the cases or scoring rule after seeing a result breaks the comparison.Changing the test after seeing a bad result makes the comparison unreliable.
Overall scores can hide failures in critical slices.
Move the minimum acceptable score. The same eight test cases feed both views: one averages everything, while the other includes only money-moving situations.
6 of 8 individual cases clear the line.
Refund decisions get their own gate because mistakes are costly.
- C-0194Password resetcommonclears line
- C-0286Ambiguous order statuscommonclears line
- C-0382Return-window edgepolicyclears line
- C-0489Angry customer tonecommonclears line
- C-0546Refund after shipmentmoney-movingbelow line
- C-0658Duplicate refundmoney-movingbelow line
- C-0778Unsupported countrylong-tailclears line
- C-0888Tool timeout recoveryreliabilityclears line
The candidate clears the overall average while failing the cases allowed to move money. Averages can hide concentrated risk.
A useful eval has six explicit parts.
A useful eval is more than a CSV and a percentage. It is a small, versioned measurement system with an explicit decision attached.
- 01OBJECTIVEIf the objective is vague, the score will be precise nonsense.
The product behavior
Write the job in product language: resolve eligible returns without inventing policy or issuing unauthorized refunds.
- 02CASESA dataset is a sample of the world, never the world itself.
The situations
Each case includes the input, needed context, expected constraints, and tags such as language, risk, or customer tier.
- 03CANDIDATEVersion the whole system, not only the model name.
The frozen system version
Record the model, prompt, tools, retrieval settings, and code version. Otherwise a rerun is not the same experiment.
- 04GRADERGraders are measurement instruments. Instruments need calibration.
The scoring procedure
Code, rules, a model, or a human inspects the output—and sometimes the tool trace—to apply the written rubric.
- 05SCORE“0.82” is useless until you know what earned it.
The measurement
Some requirements are binary gates. Others earn a continuous quality score. Keep their meaning visible instead of blending everything.
- 06BASELINEImprovement is relative; readiness is a product decision.
The comparison and decision
Compare the candidate with production or the previous version, then inspect overall results, critical slices, and uncertainty.
Choose a grader that can measure the target behavior.
Use the least expensive reliable procedure that measures the target behavior. Strong suites often combine several grader types.
Code grader
- GOOD AT
- Schemas, exact values, tool arguments, invariants, latency budgets
- WATCH FOR
- Cannot judge nuanced helpfulness unless you can formalize it
Rules grader
- GOOD AT
- Required phrases, forbidden actions, citations, regex-able constraints
- WATCH FOR
- Brittle proxies can reward wording while missing meaning
Model grader
- GOOD AT
- Rubric-based quality, pairwise preference, semantic comparison
- WATCH FOR
- Sensitive to prompt, order, verbosity, model version, and calibration
Human grader
- GOOD AT
- Nuance, new failure discovery, gold labels, product judgment
- WATCH FOR
- Slow, costly, and inconsistent without rubrics and adjudication
Best for non-negotiables: correct tool, valid schema, no forbidden action.
Useful for degrees of quality: clarity, completeness, tone, relevance.
A beautiful answer that took the forbidden action still fails.
Model graders can be useful, but their judgments have measurable biases. Research has documented position, verbosity, and self-preference biases. Calibrate them against expert labels and keep the grader version fixed. See the primary MT-Bench / Chatbot Arena paper.
Represent common work and costly edge cases.
Random examples find average behavior. Tagged slices reveal where behavior changes: language, customer type, input length, tool path, risk level, or anything else your product cannot afford to average away.
Common jobs sampled in roughly the shape customers actually create them.
Rare but costly situations: money movement, permissions, safety, legal claims.
Ambiguity, missing context, conflicting instructions, malformed tool results.
Confirmed failures from logs, support tickets, reviews, and incident reports.
One run is one sample.
Repeat cases when randomness could change the release decision. Report a pass rate or score distribution, keep decoding settings fixed, and compare candidates with the same run budget.
Six ways an eval produces a misleading result.
Broken evals still produce precise-looking scores. Check that each score remains connected to the product behavior and decision it is meant to measure.
Hand-picked success cases
Five hand-picked prompts all work. Nobody recorded the failures or reran the exact set.
ANECDOTEA metric that measures the wrong behavior
The score measures eloquence while the product needs correct tool use and completed tasks.
WRONG JOBCritical failures hidden by an average
A healthy overall average dissolves one catastrophic slice into dozens of easy wins.
SLICE ITAn uncalibrated model grader
The team treats a model grader as objective without checking its labels against expert judgment.
CALIBRATETest-set leakage
Cases leak into prompts, examples, or tuning until the candidate knows the exam by heart.
CONTAMINATEDA stale dataset
The suite never absorbs confirmed production failures, so it protects last quarter’s product.
STALETurn confirmed production failures into regression cases.Turn every confirmed mistake into a test that stays.
Start small. Twenty cases that resemble the product and catch real regressions beat two thousand unlabeled prompts nobody trusts.Start small. Twenty realistic examples that catch real mistakes are better than two thousand mystery questions nobody trusts.
- 01
Name the product decision the eval will inform.Name the decision this test will help you make.
- 02
Turn real jobs and confirmed failures into versioned cases.Save realistic tasks and confirmed mistakes as permanent test examples.
- 03
Tag slices before looking at candidate results.Group important cases before you see the scores—for example, money, privacy, or edge cases.
- 04
Use deterministic gates for non-negotiables; calibrated judgment for quality.Use hard pass-or-fail rules for must-haves and a clear rating guide for softer quality.
- 05
Run the candidate and a baseline under the same conditions.Run the new and current versions under the same conditions.
- 06
Inspect overall, critical slices, disagreements, and run-to-run variance.Check the overall result, the risky groups, disputed scores, and whether repeat runs change the answer.
- 07
Write the ship rule first. After launch, feed confirmed failures back into the suite.Decide what score is good enough before testing. After launch, turn confirmed failures into new test cases.
{
"id": "refund-after-shipment",
"input": "Cancel order 1842 and refund me",
"context": { "status": "shipped" },
"tags": ["refund", "money-moving", "policy-edge"],
"expected": {
"must_not_call": "issue_refund",
"must_explain": "shipped-order policy"
}
}function grade(run, test) {
const noForbiddenCall = !run.toolCalls.some(
(call) => call.name === test.expected.must_not_call
);
return {
gate: noForbiddenCall ? "pass" : "fail",
quality: rubricGrader(run.answer, {
asks: ["correct", "clear", "actionable"],
scale: [0, 100]
})
};
}Situation: A customer asks to cancel an order that has already shipped.
Must not: Issue the refund automatically.
Must do: Explain the shipped-order policy clearly.
Hard gate: Fail if a forbidden refund action was used.
Quality score: Rate whether the answer was correct, clear, and useful.
Production feedback is discovery material, not automatic truth. A thumbs-down can mean “factually wrong,” “too slow,” or merely “not the answer I wanted.” Review and label before it enters the suite.Real-world feedback is a clue, not automatic truth. A thumbs-down might mean “wrong,” “too slow,” or simply “not what I wanted.” A person should review it before it becomes a permanent test.
Three common misconceptions about evals.
“Our benchmark score is the eval.”
A benchmark can reveal general capability. Your product eval must represent your users, tools, policies, and failure costs.
PRODUCT-SPECIFIC“A model grader replaces human judgment.”
It scales a rubric. Humans still define the rubric, calibrate labels, investigate disagreement, and notice new failure modes.
CALIBRATE“It scored 94%, so we ship.”
Ask: against which baseline, on which slices, with what variance, and did any non-negotiable gate fail? Then discuss shipping.
CHECK SLICESThe final rule: define what good means, test the cases that matter, keep the scoring rule honest, and preserve every expensive mistake as a future regression test.