Evals:applause isnot evidence.
A dazzling demo proves that your system worked once. An eval asks it to work again, on purpose, under rules you wrote before seeing the score.
- REPEATABLEsame test, new candidate
- REPRESENTATIVEcases resemble the job
- EXPLICITthe scoring rule is written down
The average brought confetti. The slice brought a fire extinguisher.
An eval is a contract with reality.
It is a repeatable test of system behavior on representative cases with an explicit scoring rule. Not one impressive example. Not one universal number. A test tied to the actual job.
Define the behavior that matters
Collect representative situations
Run a versioned candidate
Apply an explicit rule
Compare baseline and slices
If the course changes whenever the candidate looks bad, you are not measuring improvement. You are moving the cones.
One system. Two verdicts. Zero paradox.
Move the minimum acceptable score. The same eight test cases feed both boards: one averages everything, the other watches only money-moving situations.
6 of 8 individual cases clear the line.
Refund decisions get their own gate because mistakes are costly.
- C-0194Password resetcommonclears line
- C-0286Ambiguous order statuscommonclears line
- C-0382Return-window edgepolicyclears line
- C-0489Angry customer tonecommonclears line
- C-0546Refund after shipmentmoney-movingbelow line
- C-0658Duplicate refundmoney-movingbelow line
- C-0778Unsupported countrylong-tailclears line
- C-0888Tool timeout recoveryreliabilityclears line
The candidate clears the overall average while failing the slice allowed to move money. Averages can hide concentrated harm.
Six parts. Remove one, invite a ghost.
A useful eval is more than a CSV and a percentage. It is a small, versioned measurement system with an explicit decision attached.
- 01OBJECTIVEIf the objective is vague, the score will be precise nonsense.
The product behavior
Write the job in product language: resolve eligible returns without inventing policy or issuing unauthorized refunds.
- 02CASESA dataset is a sample of the world, never the world itself.
The situations
Each case includes the input, needed context, expected constraints, and tags such as language, risk, or customer tier.
- 03CANDIDATEVersion the whole system, not only the model name.
The frozen system version
Record the model, prompt, tools, retrieval settings, and code version. Otherwise a rerun is not the same experiment.
- 04GRADERGraders are measurement instruments. Instruments need calibration.
The scoring procedure
Code, rules, a model, or a human inspects the output—and sometimes the tool trace—to apply the written rubric.
- 05SCORE“0.82” is useless until you know what earned it.
The measurement
Some requirements are binary gates. Others earn a continuous quality score. Keep their meaning visible instead of blending everything.
- 06BASELINEImprovement is relative; readiness is a product decision.
The comparison and decision
Compare the candidate with production or the previous version, then inspect overall results, critical slices, and uncertainty.
Choose the instrument that can see the failure.
No grader is “the smart one.” The useful grader is the cheapest reliable procedure that measures the behavior you care about. Strong suites usually combine several.
Code grader
- GOOD AT
- Schemas, exact values, tool arguments, invariants, latency budgets
- WATCH FOR
- Cannot judge nuanced helpfulness unless you can formalize it
Rules grader
- GOOD AT
- Required phrases, forbidden actions, citations, regex-able constraints
- WATCH FOR
- Brittle proxies can reward wording while missing meaning
Model grader
- GOOD AT
- Rubric-based quality, pairwise preference, semantic comparison
- WATCH FOR
- Sensitive to prompt, order, verbosity, model version, and calibration
Human grader
- GOOD AT
- Nuance, new failure discovery, gold labels, product judgment
- WATCH FOR
- Slow, costly, and inconsistent without rubrics and adjudication
Best for non-negotiables: correct tool, valid schema, no forbidden action.
Useful for degrees of quality: clarity, completeness, tone, relevance.
A beautiful answer that took the forbidden action still fails.
Model graders can be very useful, but they are not neutral measuring sticks. Research has documented position, verbosity, and self-preference biases. Calibrate them against expert labels and keep the grader version fixed. See the primary MT-Bench / Chatbot Arena paper.
Your dataset is a map. Draw the dangerous roads.
Random examples find average behavior. Tagged slices reveal where behavior changes: language, customer type, input length, tool path, risk level, or anything else your product cannot afford to average away.
Common jobs sampled in roughly the shape customers actually create them.
Rare but costly situations: money movement, permissions, safety, legal claims.
Ambiguity, missing context, conflicting instructions, malformed tool results.
Confirmed failures from logs, support tickets, reviews, and incident reports.
One run is a sample, not a personality trait.
Repeat cases when randomness could flip the ship decision. Report a pass rate or score distribution, keep decoding settings fixed, and compare candidates with the same run budget.
Six ways an eval can lie politely.
Most broken evals still produce decimals. That is what makes them dangerous: the machinery looks scientific after the connection to product reality has snapped.
The demo reel
Five hand-picked prompts all work. Nobody recorded the failures or reran the exact set.
ANECDOTEMetric costume
The score measures eloquence while the product needs correct tool use and completed tasks.
WRONG JOBAggregate soup
A healthy overall average dissolves one catastrophic slice into dozens of easy wins.
SLICE ITGrader worship
The team treats a model grader as objective without checking its labels against expert judgment.
CALIBRATETest-set telepathy
Cases leak into prompts, examples, or tuning until the candidate knows the exam by heart.
CONTAMINATEDThe frozen museum
The suite never absorbs confirmed production failures, so it protects last quarter’s product.
STALETurn one production bruise into a permanent crash test.
Start small. Twenty cases that resemble the product and catch real regressions beat two thousand unlabeled prompts nobody trusts.
- 01
Name the product decision the eval will inform.
- 02
Turn real jobs and confirmed failures into versioned cases.
- 03
Tag slices before looking at candidate results.
- 04
Use deterministic gates for non-negotiables; calibrated judgment for quality.
- 05
Run the candidate and a baseline under the same conditions.
- 06
Inspect overall, critical slices, disagreements, and run-to-run variance.
- 07
Write the ship rule first. After launch, feed confirmed failures back into the suite.
{
"id": "refund-after-shipment",
"input": "Cancel order 1842 and refund me",
"context": { "status": "shipped" },
"tags": ["refund", "money-moving", "policy-edge"],
"expected": {
"must_not_call": "issue_refund",
"must_explain": "shipped-order policy"
}
}function grade(run, test) {
const noForbiddenCall = !run.toolCalls.some(
(call) => call.name === test.expected.must_not_call
);
return {
gate: noForbiddenCall ? "pass" : "fail",
quality: rubricGrader(run.answer, {
asks: ["correct", "clear", "actionable"],
scale: [0, 100]
})
};
}Production feedback is discovery material, not automatic truth. A thumbs-down can mean “factually wrong,” “too slow,” or merely “not the answer I wanted.” Review and label before it enters the suite.
Three sentences to retire.
“Our benchmark score is the eval.”
A benchmark can reveal general capability. Your product eval must represent your users, tools, policies, and failure costs.
NOT THE JOB“A model grader replaces human judgment.”
It scales a rubric. Humans still define the rubric, calibrate labels, investigate disagreement, and notice new failure modes.
NICE TRY“It scored 94%, so we ship.”
Ask: against which baseline, on which slices, with what variance, and did any non-negotiable gate fail? Then discuss shipping.
SHOW RECEIPTSThe final rule: define what good means, test the cases that matter, keep the scoring rule honest, and preserve every expensive mistake as a future regression test.