Here is a test. Ask the team that owns your AI feature this question: the prompt changed last Tuesday — did quality go up or down, and by how much? If the answer involves the words "it feels", you do not have a product. You have a demo that happens to be in production.
An evaluation set is the smallest thing that converts an AI feature into software you can maintain. It takes about a week to build the first useful version, and teams keep not doing it because it feels like overhead compared with shipping another prompt tweak. It is the opposite: it is what makes the next fifty prompt tweaks cost minutes instead of meetings.
What a graded set actually is
A list of realistic inputs, each with an expected output or a set of criteria the output must satisfy, written by the people who will judge the feature in production. Not by the engineers. Not synthetic. Between 100 and 200 examples is enough to start; we have never needed more than 400 for a single feature.
Composition matters more than volume. A useful split:
- Around half: the common cases, sampled from real traffic or real documents.
- A quarter: known-hard cases — ambiguity, multiple valid answers, information spread across sources, the things your experts argue about.
- A fifth: cases where the correct behaviour is to refuse or say the answer is not available. These are the examples teams skip, and they are the ones that catch confident nonsense.
- The remainder: adversarial and malformed input — prompt injection attempts, truncated documents, the wrong language, an empty file.
Building it in a week
The process that works: two or three subject-matter experts, three sessions of two hours. Session one collects inputs and everyone writes expected outputs independently. Session two compares them — and the disagreements are the valuable output, because a case where two experts disagree is a case where your specification is wrong, not where the model will be. Session three resolves the disagreements and turns the resolutions into written criteria.
Do not let engineers author the expected outputs. They will unconsciously write what the current system produces, and the set will encode today's behaviour as correct forever.
Metrics that survive contact with reality
For structured extraction, this is easy and you should take the easy win: per-field precision, recall and F1 against the expected values, reported per field rather than averaged. An average hides the one field that dropped from 0.9 to 0.4.
For generative outputs, use a layered approach:
- Deterministic checks first. Schema validity, required fields present, citations resolve to real spans, no personal data in the output, length within bounds. These are cheap, fast and catch a surprising share of regressions.
- Reference-based scoring where a canonical answer exists.
- Model-as-judge last, and with discipline (below).
Using a model as a judge, honestly
Model-as-judge is useful and routinely misused. Three rules make it defensible.
Calibrate the judge against humans. Have your experts grade 50 outputs, have the judge grade the same 50, and report agreement. If agreement is poor, fix the judge rubric before trusting a single number it produces. Re-check when you change the judge model.
Use a different model, and pin its version. A model judging its own family's output has a measurable bias toward it. Pin the version, because a silent judge upgrade will move all your metrics and you will spend a day investigating the wrong system.
Score narrow attributes, not vibes. "Is every factual claim supported by the provided context: yes/no" is answerable. "Rate the quality from 1 to 10" is noise with a number attached.
Wiring it into CI
The harness runs on every change to a prompt, model version, retrieval configuration, index build or chunking rule. Practical arrangement:
- A fast subset (about 30 examples, deterministic checks only) on every commit, finishing in under two minutes.
- The full set nightly and on any pull request touching AI configuration.
- Results written to a store, not just printed — you want the trend line across releases.
- A merge gate on regressions beyond a defined tolerance per field, with an explicit override that requires a written reason.
Treat prompts as versioned artefacts with the same review discipline as code. A prompt edited directly in a vendor console at 6 p.m. on a Friday is an unreviewed production deployment, and everyone involved knows it except the process.
Keeping the set alive
Route production disagreements back in. When a reviewer corrects an output, that becomes a candidate example. Review the candidates monthly, add the ones that represent a genuine class of failure, and note which release first passed each one.
Guard against overfitting the same way you would anywhere else: hold out 20% of examples that the team building prompts does not look at. When headline scores rise and the holdout does not move, you have been tuning to the test.
What this buys you
Three things, in ascending order of value. You can ship prompt and model changes without a review meeting. You can evaluate a new model release against your actual use case in an afternoon instead of taking a vendor's benchmark on faith. And when someone senior asks whether the feature is good enough yet, you answer with a number and the date it was measured.
A week of expert time. That is the whole cost.
Written by the Bytescope Tech engineering team from live engagement work. If you want to argue with any of it, or you have a case where the opposite held, tell us — those are our favourite emails.