MODULE 08 OF 12 · RELEASE 2

Evaluation Engineering

Proving a change made things better, instead of hoping it did.

So what is this really about?

AI systems are unusually easy to fool yourself about. You tweak the wording, try two examples, and they look better — so you ship. Then a whole category of input quietly breaks. The fix is a fixed set of examples, run every time, with the results compared against last time.

The words you will hear

Evaluation set

Your fixed tasting menu. The same examples every time, including the ones that used to fail.

Baseline

Last known score. New changes have to beat it, or at least not lose to it.

Regression

Something that used to work and now does not.

Model judge

Using an AI to grade answers where there is no exact right answer — after checking it agrees with humans.

By the end you will be able to

• Build a fixed set of examples drawn from real failures.
• Run it automatically and block changes that break things.
• Use an AI grader only where you have checked it agrees with people.

Going deeper

Why this is unusually easy to get wrong

Ordinary software gives you honest feedback. Break something and a test goes red, or a user reports a clear bug. The signal arrives quickly and points somewhere specific.

AI systems do not behave like that. Break something and the output is still fluent, still plausible, still shaped like an answer. Nothing crashes. The feedback loop that normally protects you is quietly absent, and the gap fills with impressions: it feels better, it seems smarter.

Evaluation is how you get the feedback loop back.

Build the set from failures

The temptation is to collect examples that show the system working. Those are pleasant and nearly useless — they will keep passing through changes that break everything else.

The valuable cases are the ones that failed. The question your system answered wrongly last month. The input that made it hallucinate a policy. The phrasing that produced a confidently incorrect number. Each of those is a case worth keeping forever, because each represents a mistake you have already made once.

Fifty cases is plenty to start. It is more than most systems have, and it is enough to catch real regressions.

Assert what you can, judge only what you must

Split every case into properties you can check exactly and properties you cannot.

Far more is exactly checkable than people assume. Did it return valid structured output? Is the required field present? Is the total the sum of the line items? Did it cite a real document? Did it refuse the thing it should refuse? None of that needs a judge — it needs an assertion, and assertions are fast, free and deterministic.

What is left is genuinely subjective: is this explanation clear, is the tone right, is this a good summary. For those, a model judge with an explicit rubric is reasonable.

But a judge is itself a system you are trusting, so measure it before you rely on it. Have a person grade thirty outputs, have the judge grade the same thirty, and compare. If they agree most of the time, the judge is useful. If they do not, you have replaced an unmeasured system with an unmeasured measurement — which is worse, because now it produces numbers.

Averages hide exactly the thing you need to see

This is the failure mode that makes evaluation worth engineering properly.

You change a prompt. The average score goes from 82% to 86%. Ship it?

Not yet. That improvement might be forty cases getting slightly better while three cases that used to pass now fail completely. Whoever depended on those three has just watched the system stop working, and your dashboard shows an improvement.

So report per case, and fail the build on any regression — a case that passed before and does not now — regardless of what the aggregate did. Aggregates are for trends. Individual cases are for decisions.

Run it where it can block

An evaluation you run when you remember is a evaluation you will stop running. Put it in CI alongside the tests, with a stored baseline, and let it fail the build.

Two practical notes. Store results per commit so a regression is attributable to a change rather than to a vague window of time. And keep the suite fast enough that people do not resent it — if a full run takes twenty minutes, run a subset on every commit and the whole thing nightly.

What good looks like

Changing a prompt produces a number rather than an argument. A regression in one category cannot hide behind an improved average. And when someone asks whether the system got better last quarter, the answer is a chart rather than a recollection.

How it is put together

An evaluation harness runs a fixed set of cases against the current system and reports per-case results and an aggregate. Deterministic properties are asserted exactly; subjective properties use a model judge with its own rubric and its own measured agreement with human labels. Results are stored per commit so a regression is attributable.

Where you start

A handful of manual test prompts run by hand before each release.

The lab

1. Convert manual prompts into a structured case file with expected properties.
2. Add cases drawn from real failures, not only from cases that already pass.
3. Assert deterministic properties exactly; reserve the model judge for the rest.
4. Measure the judge against human labels and record the agreement rate.
5. Run the harness in CI with a baseline that fails the build on regression.

What goes wrong, on purpose

The scenario. A prompt change improves the aggregate score while breaking a specific class of input.

What should happen. Per-case reporting surfaces the regression even though the aggregate improved, and the build fails on the affected cases.

How you prove it works

• A test asserting the harness fails when a known-good case regresses.
• A judge-agreement measurement recorded alongside the results.
• A CI run producing a per-case report, not only a single number.

What you walk away with

An evaluation harness in CI, a baseline that blocks regressions, and a measured agreement rate for any model judge you rely on.