Building a Golden Eval Dataset from Production Traffic

extract

Building a Golden Eval Dataset from Production Traffic OpenRouter · You update a prompt, or a provider rolls out a new checkpoint under the same model ID, and something in production regresses. The last week of user complaints looks slightly different from the week before. A public benchmark like MMLU won’t catch that. It measures general capability across academic subjects, not how the model handles your product’s traffic. A golden eval dataset closes that gap. It’s a curated collection of production inputs paired with reviewed expected outputs, versioned in Git, and run before every deploy. It answers the question a benchmark can’t. Does this change help or hurt on the traffic you serve? This guide covers what a golden set is, why production data is a better foundation than synthetic data, and a five-step process for building one from live traffic. It also covers how to run the same set against many candidate models through one API, so you can pick your next model on evidence from your own traffic rather than leaderboard rank. Tl;dr - A golden eval dataset is a curated set of production inputs paired with reviewed expected outputs, used as a regression test before every meaningful change. - Once the set exists, you can run it across many models through one API and pick your next model on evidence from your own traffic. - Match the size to the job. Roughly 10 items to explore a single issue, 100 to 1,000 for a full regression set. The right size depends on the metric, the variance, and the smallest difference you need to detect. - Coverage of failure modes matters more than volume. Production examples carry the failures your users hit. Synthetic examples carry the failures someone imagined. - Version the dataset, the rubric, and the baseline together. Otherwise you can’t tell whether a regression came from a model change, a prompt change, or a dataset change. What a golden eval dataset is A golden eval dataset is a collection of production examples with reviewed expected outputs, held out from training and evaluated on every release candidate. Someone with domain knowledge confirmed each expected output, or decided the item doesn’t need one because a reference-free check like format, safety, or tone is the pass criterion. That review is what makes the set useful. Without it, when a score drop appears, you can’t tell whether the change regressed quality or item 23 has a wrong label. Treat it as a regression test suite for behavior rather than code. It runs in CI, it has known-good expected outputs, and it catches drift between deploys. The difference from a unit test is that the expected output is a judgment, so the harness has to do more than compare strings. A golden set is not a benchmark, a training set, or an A/B test. A benchmark measures general capability. A golden set measures capability on your traffic. A training set teaches the model. A golden set catches the model when it regresses. An A/B test measures live user outcomes. A golden set measures whether a change is safe enough to run an A/B test on at all. Why production data is the better foundation Synthetic evals test a model’s ability to answer questions someone imagined a user might ask. What you need to test is the questions your users asked, in the phrasing they used. Production traffic is a better foundation for three reasons. The distribution is right. If 70 percent of your traffic is about pricing, 70 percent of your eval set should be about pricing. A synthetic set drawn from imagined edge cases doesn’t preserve that distribution. Preserving the observed distribution is what makes a regression score reflect user impact. The failure modes are the ones you care about. Users find failure modes you wouldn’t think to write. A message that mixes three languages, a support ticket that pastes an entire error log, a query that references a product feature you renamed last quarter. A synthetic set won’t contain these unless someone thought of them. The examples track your product. A support bot that launched handling password resets six months ago now gets multi-account questions, refund escalations, and screenshots of a competitor’s UI. A golden set drawn from production tracks that drift. A synthetic set frozen at launch doesn’t. Synthetic examples still have a place as an extension rather than a foundation. Use them to fill coverage gaps for known failure modes you have too few real examples of. You can generate them with a model call that uses structured outputs so each example lands in your dataset schema, or by paraphrasing existing items. Mark them as synthetic in metadata, and keep them a minority of the set. The five-step process Step 1: Pull a sample of production traffic Start with a week or two of logged inputs and outputs. Use a longer window if the feature is seasonal or thinly used. Sample randomly first, then look at what came out. If your traffic has a long tail, a random sample over-represents the head. That’s often what you want, since regressions in the head affect the most users, but check whether rare and critical intents show up at all. Log every field you might want to slice on later. That includes the intent, the feature area, the user segment, the timestamp, and the model and prompt version that produced the response you sampled. You can’t reconstruct this metadata later. Sampling production traffic means sampling user data. Check your terms of service, run your PII scrubbing pipeline before the data reaches an eval harness, and log which examples you transformed so you can reproduce the scrubbing later. Aim for a raw pool of a few hundred examples going into step 2. You’ll trim heavily. Step 2: Deduplicate and cluster Production traffic is repetitive. A support chatbot might see the same “how do I reset my password” question a hundred times a day in slightly different phrasings. You want one of those in the set, not fifty. Exact-match deduplication misses paraphrases. To catch them, check whether the dataset already covers the same intent, either by exact-matching normalized inputs or by comparing embedding similarity between inputs. Once you have a deduplicated pool, sample within it for coverage. If half your traffic is one intent, that intent should get roughly half the golden set, but not all of it. An under-represented intent that fails badly costs more than a well-represented intent that fails gracefully, so weight toward the failure modes you can’t afford. For target size, Langfuse’s guidance is a useful starting point. Treat the numbers as reference points and adjust to your traffic and evaluation goal. Keep a subset of tens to low hundreds for the pull request gate so it stays fast, and reserve the full set for release branches or nightly runs. Step 3: Add expected outputs For every input, someone qualified has to write down what the right output looks like. Sometimes that’s a single string. More often it’s a rubric. Which facts must be present, which claims must not be made, and what tone or format is required. Two rubric choices make grading more consistent. Use binary criteria, where each criterion is either MET or UNMET. Use analytic rubrics, which score each criterion separately, instead of holistic rubrics that assign one overall score. Analytic rubrics tell you what got worse, not only whether something did. Have two people annotate a subset independently. Where they disagree, treat the rubric as the problem, not the annotators. Fix the rubric until two reviewers grading the same model output would grade it the same way. Not every item needs a hard-coded expected output. Items checked only by reference-free evaluators, such as format validity, safety, or tone, need no expected output at all. A golden set can mix both kinds. Step 4: Run a first evaluation and fix the rubric Before you trust the set, run your current production model against it and look at every failure. Sort the failures into three categories. Real failures, where the model got it wrong and the expected output is right. Keep those. Rubric problems, where the model gave a reasonable answer that your expected output didn’t anticipate. Fix the expected output. Ambiguous examples that no reasonable rubric can grade. Cut them. Expect to cut part of the set on this pass. How much depends on how carefully you wrote the expected outputs in step 3. The cut is the point of the pass. A golden set that produces the same pass rate regardless of the model change isn’t measuring anything. Don’t skip this step. A rubric problem you don’t catch here shows up later as a false regression during a real deploy. Step 5: Commit to Git, wire into CI, iterate The golden set belongs in your codebase, versioned alongside the prompts and the code that runs it. That’s what turns it from an ad hoc quality check into a regression test. A minimal directory layout looks like this. /evals /golden dataset.jsonl # one example per line rubric.md # how to grade run.ts # loader and harness baseline.json # pass rates on the current production model /synthetic dataset.jsonl # rare failure modes, marked with synthetic: true Store each example as a JSON line with fields for the input, the expected output, tags, and metadata. Store the rubric as a human-readable document that the grader reads from, so anyone reading a pull request diff can see whether a rubric change is what moved the pass rate. A single line in dataset.jsonl looks like this. {"id":"pw-reset-locked-account","input":"i cant log in, tried resetting three times and its saying account locked. help","expected":"The bot should acknowledge the lockout, ask for the account email, and route to the account-recovery flow. It must not offer to reset the password directly.","rubric":"must_ask_email;must_route_recovery;must_not_reset_directly","tags":["password_reset","edge_case"],"synthetic":false} The fields serve different readers. id gives each item a stable reference that survives Git diffs. input is the production input, verbatim after scrubbing. expected is human-readable prose that an LLM judge can read. rubric holds the machine-checkable criteria the judge grades against. tags let you slice pass rates by intent. synthetic keeps real and synthetic items separate in aggregate metrics. Wire the harness into CI to run on every prompt change or model swap. Fail the build if the pass rate drops more than a defined threshold, and require a written acknowledgment on any smaller drop. A GitHub Actions workflow that calls your harness on every pull request is enough to start. Refresh the set on a cadence. A quarterly review of items older than six months, checking each against current product behavior, is a reasonable default. Review monthly if the product is changing fast. A stale golden set eventually stops predicting production behavior. Version the rubric alongside the data. A failing run is only useful if you can diagnose it, and rubric changes are the hardest kind of change to catch after the fact. If someone edits a criterion to fix what looked like a false regression, that edit should appear as a visible change in a pull request diff, not as an unexplained pass rate move six weeks later. Don’t retire items because they keep passing. A passing item is proving that the behavior still holds. Retire an item only when the behavior it tests no longer exists in the product, or when the expected output is now wrong. Cross-model benchmarking through one API Once you have 100 examples with expected outputs, running the same set against a different model tells you how that model performs on your workload. Not on MMLU, and not on someone else’s coding harness. On the questions your users send. Comparing models across vendors usually means a different SDK, different authentication, a different response shape, and different rate limits for each candidate. Through OpenRouter, the same OpenAI-compatible request body works across the model catalog. For candidates that share an interface and support the parameters your request uses, you can compare them by changing model: "openai/gpt-5.1" to model: "anthropic/claude-fable-5.1" and rerunning the harness. Where models differ on context length, tool support, or supported parameters, expect some integration work per candidate. Each entry in the models endpoint lists the model’s context length and a supported_parameters array, so you can check before switching. A minimal harness looks like this. import OpenAI from "openai"; import { readFileSync } from "fs"; const client = new OpenAI({ apiKey: process.env.OPENROUTER_API_KEY ?? "", baseURL: "https://openrouter.ai/api/v1", }); const dataset = readFileSync("./evals/golden/dataset.jsonl", "utf-8") .trim() .split("\n") .map((line) => JSON.parse(line)); const modelsToTest = [ "openai/gpt-5.1", "anthropic/claude-fable-5.1", "google/gemini-3.8-flash", ]; const results: Record<string, { pass: number; total: number }> = {}; for (const model of modelsToTest) { results[model] = { pass: 0, total: 0 }; for (const example of dataset) { const response = await client.chat.completions.create({ model, messages: [{ role: "user", content: example.input }], }); const output = response.choices[0].message.content ?? ""; const passed = grade(output, example.expected, example.rubric); results[model].total += 1; if (passed) results[model].pass += 1; } } console.table(results); The grade function is where your rubric lives. For string-match cases, it can be output.includes(expected). For rubric-based grading, it’s usually another LLM call to a strong judge model with the rubric and the response, asking whether the response satisfies each criterion. A rubric-check prompt to the judge looks like this. Rubric criteria (each MET or UNMET): {criteria} Response to grade: {response} For each criterion, output MET or UNMET on its own line. No prose. The judge returns one MET or UNMET per criterion, which grade() parses into a pass if every criterion is MET and a fail otherwise. Binary criteria and the “no prose” instruction keep the judge output consistent enough to parse across runs. An LLM judge is reliable enough for regression detection on a well-designed rubric with binary criteria and anchor examples. Agreement with human graders varies by task, rubric, and judge model, so calibrate the judge against a set of human-labeled examples before you rely on its scores. Judge reliability also varies with the random seed alone. One study had three LLM judges grade responses to BIG-Bench Hard questions 100 times each, changing nothing but the seed, and measured inter-rater reliability between the judges ranging from 0.167 to 1.00 across replications. On that scale, 1.0 is perfect agreement. Use a different model as the judge than the model under test to avoid self-preference bias, and don’t treat a single judge run as ground truth on high-stakes decisions. Score the same set on every model. If you change the dataset between runs, you’re measuring two things at once. Freeze the set, record the commit hash, then swap models. Watch cost as well as quality. The most accurate model on your golden set might cost 30 times as much per call as the second-most-accurate model, for a quality difference of two percentage points. Surface that tradeoff to whoever owns the budget. Our model comparison page shows the price gap between two models before you run the full benchmark. Use provider routing controls when you need reproducibility. The same model slug can route to different providers, and providers can differ slightly in behavior. To pin a benchmark run to one provider, set the order field in the provider object to that provider’s slug and set allow_fallbacks to false. With order alone, we still fall back to other providers if the listed ones are unavailable. Conclusion A golden eval dataset tells you whether a change helps or hurts on the traffic you serve. Build it from production data, review every expected output, version the set with its rubric and baseline, and run it on every deploy. Once you have it, running the same set across many models through one API is the payoff. For compatible candidates, comparison becomes a model identifier change and a rerun, with per-candidate checks for context length, tool support, and supported parameters. Frequently asked questions How large should a golden eval dataset be? Start with 20 to 50 reviewed items that cover your most important failure modes. Grow toward 100 to 1,000 items for a full regression set, and keep a smaller subset for per-PR checks. The right size depends on the metric, the traffic distribution, and the smallest quality difference you need to detect. Coverage of failure modes matters more than raw count. Should I use production data or synthetic data? Use production data as the foundation and synthetic examples to fill known coverage gaps. Real traffic preserves the distribution and failure modes your users generate. Add synthetic cases for rare failure modes you have too few real examples of, mark them as synthetic in metadata, and keep them a minority of the set. How often should I refresh the golden set? Review the set quarterly, and more often when prompts, models, or product behavior change quickly. Check older items against current behavior, add newly observed failure modes, and retire only items whose behavior no longer exists in the product. Keep prior versions so you can reproduce historical regressions. Can I use an LLM as the judge? Yes, with two conditions. The rubric must be specific enough that two humans grading the same output would grade it the same way, and you must calibrate the judge against human-labeled examples before relying on its scores. Avoid single-run judgments for high-stakes decisions, and don’t use the model under test as its own judge. What is the difference between a golden set and a benchmark? A benchmark measures a model’s general capability against a public test. A golden set measures capability on your own traffic. Benchmarks help you narrow the model shortlist. A golden set tells you which of the shortlisted models works for your product. Neither replaces the other. References - Langfuse: Golden dataset evaluation. Practitioner guidance on building and maintaining golden sets, including the size table above. - Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge. Empirical work on LLM-judge reliability under random-seed variation. - OpenRouter models. The current catalog of models available through one API. - Provider selection. How to pin a specific provider for reproducibility. - Model rankings. Usage-based rankings across the catalog, useful for shortlisting judge models and benchmark candidates.