Your LLM eval set is too small: build it from production failures

Most LLM eval sets are written once, at launch, from the queries the team imagined users would send. Then they never change. The problem is that production traffic has a long tail no hand-authored set ever captures, so the eval stays green while real users hit failures nobody wrote a test for. The fix is a loop: treat production traces as the source of truth, and turn every real failure into a permanent regression case. Your eval set should grow with each incident, not stay frozen at the size it was on launch day.

Why the launch-day eval set goes stale

When a team ships an AI feature, someone usually writes 30 to 50 eval cases the week before launch. They are written from memory: the demo prompts, the obvious questions, a few edge cases someone thought of in a meeting. The set passes, the feature ships, and the set is filed away. Six weeks later it still passes, and everyone reads that as the feature being healthy.

It is not evidence of health. It is evidence that the feature still handles the 40 things you thought of in one afternoon. A hand-authored set has three built-in blind spots.

It encodes what you imagined, not what users do. Real users phrase questions in ways you would never write down, combine two requests in one message, paste in malformed data, and ask about features you shipped after the eval set was frozen. None of that is in a set written from the demo script.

It does not move when the traffic moves. Your product ships new features, onboards a different customer segment, and picks up seasonal patterns. The distribution of real inputs drifts every month. A frozen eval set measures last quarter's product against last quarter's users.

It gives false confidence precisely when you need real signal. A green eval run feels like permission to ship a prompt change or a model swap. But if the set never covered the failure a change reintroduces, green means nothing. This is the gap we described in observability versus evals: observability tells you the feature ran, evals are supposed to tell you it was right, and a stale set quietly stops doing its job.

Production traces are the real test set

The test cases you actually need are already being generated, thousands a day, by your users. Every production request is a labeled example waiting to happen. The queries you would never invent, the phrasings that break your retrieval, the multi-part questions that confuse your agent are all sitting in your logs. Production is the only place the long tail exists.

To use it, you have to log enough to replay a request, not just enough to debug latency. A trace that is useful for evals captures the full input, any retrieved context or documents, every tool call and its response, the final output, and the metadata to reproduce the run: the prompt version, the model, and the relevant configuration. If you cannot replay a trace deterministically, you cannot turn it into a stable regression case.

This is also why the dangerous failures are the ones worth mining hardest. As we covered in silent failures in production AI agents, the incidents that hurt are the ones that return a confident, well-formed, wrong answer and never throw. Those never show up in error dashboards. They show up in traces, if you captured enough of them.

The loop: turn a failure into a permanent regression case

The discipline is a closed loop that runs every time a real failure surfaces. Four steps, and the output of the fourth feeds the first the next time.

1. Capture the failing trace

When a failure is found, whether from a user report, a support escalation, or a monitoring alert, pull the full trace, not a screenshot of the answer. You want the exact input, the retrieved context, and the config that produced the bad output, so the case you build reproduces the real conditions rather than an approximation of them.

2. Label the failure mode

Assign the failure to a small, fixed taxonomy rather than a free-text note. Categories like stale or wrong retrieved context, ignored instruction, hallucinated fact, malformed output, wrong tool call, and unsafe response cover most of what you will see. A small taxonomy is what lets you count failure modes, spot the one that recurs, and decide where to spend engineering time. Free-text labels give you a pile of anecdotes; categories give you a distribution.

3. Promote the example into the eval set with an expected behavior and a scorer

Add the trace to your eval set as a new case: the input, the expected behavior, and a scorer that checks it. The expected behavior is not always an exact string; often it is a property, such as must cite the returned policy, must not invent an amount, or must return valid JSON with these fields. The scorer is the code that decides pass or fail. This is the same acceptance-criteria thinking we described in defining acceptance criteria for AI features, applied one real failure at a time.

4. Gate CI and monitor live traffic with the same scorer

Run the new case in CI so any future prompt edit, model swap, or retrieval change that reintroduces the bug fails the build before it merges. Run the same scorer against a sample of live traffic so you learn whether the failure mode is recurring in production even when CI is green. One scorer, two places: the pull request and the live stream.

Finding failures when nothing throws

Mining production only works if you can find the failures, and most of them are silent. You need signals that flag a candidate trace for a human to look at.

Explicit signals are the cheapest to act on: a thumbs-down, a support ticket that quotes the AI answer, a user who rephrases the same question three times, an escalation to a human agent, an abandoned session right after a response. Each is a strong hint that the last trace is worth labeling.

Implicit signals catch the failures users do not report: a low score from an automated judge, a schema violation on a structured output, a tool call that returned an error the model talked over, an input that looks nothing like anything in your eval set. These need instrumentation, but they surface the silent-but-wrong answers that no user bothers to flag.

You cannot label everything, so sample deliberately. Stratify by feature and intent so a high-volume, low-stakes path does not drown out a low-volume, high-stakes one, and always oversample the traces your signals flagged. The goal is not to review all traffic; it is to make sure the failures that matter reliably reach a human who can label them.

Scorers: keep them narrow, prefer deterministic

A regression case is only as good as the scorer attached to it, and the most common mistake is reaching for a language model to grade everything. Use plain code wherever the check can be plain code: an exact tool name, a required set of fields, a value that must or must not appear, valid JSON, a number within a range. Deterministic scorers are fast, free, and never flaky, so they can run on every commit.

Reserve a model-based judge for the cases that genuinely need judgment, such as whether an answer is faithful to the retrieved context or appropriately refuses. When you do use a judge, calibrate it before you trust it, because an uncalibrated judge ships regressions while the dashboard stays green. We walked through that calibration in using an LLM as a release gate. Whatever the scorer, keep it narrow and binary. One case should test one behavior, so a failure points at one thing.

A worked example: a B2B support copilot

Consider a support copilot for a billing product. At launch the team wrote 40 eval cases, all green, covering the obvious questions: how do I update my card, why was I charged, how do I cancel. The feature shipped and the set kept passing.

Five days later a customer on the Growth plan asked a two-part question: why their invoice jumped and whether the overage would repeat next month. The copilot answered the first part from a stale plan record and confidently quoted the wrong overage rate. Nothing errored. The trace looked healthy. The failure surfaced only because the customer left a thumbs-down, which a weekly review caught.

The team ran the loop. They pulled the trace, labeled it stale retrieved context plus multi-part question, and added it to the eval set as a case with a scorer that checks the answer cites the current plan's overage rate and addresses both parts of the question. That case went into CI. Over the next eight weeks the same loop ran on every flagged failure, and the set grew from 40 cases to roughly 180, each one a real thing that had actually gone wrong. When a later prompt edit meant to make answers shorter would have dropped the second half of multi-part answers again, CI caught it before merge. The numbers here are illustrative, but the shape is what matters: the set stopped being a launch artifact and became a record of every failure the feature had survived.

When a small eval set is fine

This loop is not free, and it is overkill for some situations. Before launch, when you have no real traffic, a hand-authored set is exactly right and the only thing you can build. For a low-volume internal tool where a wrong answer costs a shrug, a dozen cases you maintain by hand is a reasonable place to stop. For a throwaway prototype, do not build any of this.

The loop earns its keep once two things are true: you have real traffic with a long tail, and a wrong answer has a real cost, whether that is a refund, a churned customer, or a compliance problem. That is the point where a frozen set becomes a liability and a living one becomes the cheapest insurance you have. It pairs naturally with a careful rollout process, since the same regression cases that gate CI also give a shadow or canary release something concrete to measure the candidate against.

An eval set is not a document you write once and archive. It is the accumulated memory of everything your AI feature has gotten wrong, and it should get larger and sharper every week the feature is live. If you want help building that loop into a feature you are shipping, that is the kind of work our team does; the AI engineering playbook covers where it fits alongside evals, observability, and rollout.