The LLM evaluation framework that actually works in production

Most teams pick a model the same way they pick a coffee machine — a quick online comparison and a gut feeling. Here is what an evaluation framework looks like when your business depends on it.

The benchmark trap.

In a lot of AI project kick-offs, the model selection conversation goes roughly like this: someone opens a leaderboard, the team debates which frontier model is "smarter," someone else suggests running both through a few demo prompts, and within forty minutes a decision is made that will govern every output the product ships for the next twelve months.

Then, two months into production, the support queue fills up with the exact failure mode nobody thought to test for. The model is technically "better" on MMLU. It is still confidently wrong about the specific thing your users actually ask.

Public benchmarks are not lies. They measure what they measure, accurately. The problem is that they measure capability across a wide distribution of tasks, and your production use case occupies a narrow slice of that distribution with its own statistical properties, its own failure modes, and its own quality bar. A model that ranks first on HumanEval may still mangle your domain-specific code generation because the benchmark never included your API patterns, your naming conventions, or your library versions.

The gap between benchmark rank and production quality is not random. It is systematic, and it is reproducible. Which means you can close it with the right evaluation approach before you ship.

What "good enough" actually means.

Before you can evaluate a model, you need a definition of success that is specific enough to test. This sounds obvious. It is almost universally skipped.

Vague success criteria — "the outputs should be high quality," "the responses should be helpful," "the model should be accurate" — produce vague evaluation results. When the team sits down to review model outputs and disagrees about which is better, the problem is not the model. The problem is that nobody specified what better meant before looking at the outputs.

Take a B2B document summarisation feature whose initial brief says "the summaries should be accurate and concise." A sprint of evaluation work should turn that into a checklist of specific criteria: maximum sentence count by document type, required inclusion of monetary figures and dates, prohibited language patterns, handling of ambiguous attribution, formatting requirements for bullet points, and the edge cases involving redacted content. Reviewers working from a checklist like that agree with each other far more often than reviewers working from the one-line brief — and agreement between reviewers is what makes an evaluation result mean anything.

The checklist becomes the evaluation rubric. The rubric becomes the prompt specification. The prompt specification becomes the acceptance criteria for every model upgrade. One document, used three ways.

The architecture of a working eval set.

A production evaluation set has three layers, and most teams only build the first one.

The first layer is the golden set — representative examples of normal inputs with known-correct outputs. These are your baseline. They tell you whether the model can do the core job at all. Aim for 50 to 100 examples that span the realistic input distribution: short inputs and long ones, clean formatting and messy formatting, clear intent and ambiguous intent. Human experts label the correct output for each. This is the only layer most teams build, and it is necessary but nowhere near sufficient.

The second layer is the adversarial set — inputs specifically designed to trigger known failure modes. If your feature generates customer-facing email copy, your adversarial set includes: requests that could produce legally problematic claims, inputs where the user's intent conflicts with your brand guidelines, edge cases where the model must decline to do something, very short inputs with insufficient context, and inputs in languages or registers outside your primary use case. The adversarial set is not about being hostile. It is about testing the surface area your users will eventually find, before they find it in production.

The third layer is the regression set — real production failures that have been reviewed and labelled. This layer does not exist at launch. It grows over time as real users interact with your system in ways you did not anticipate. Every failure that reaches a human reviewer is a candidate for the regression set. A mature deployment will have hundreds of regression examples. They are your most valuable evaluation data because they represent real failures, not imagined ones.

Scoring: the three things you are actually measuring.

When you run an evaluation, you are measuring three distinct things, and conflating them is a common source of confusion.

The first is correctness — did the model produce the right answer? For factual tasks, this is binary. For open-ended tasks, it requires a rubric. Correctness is the dimension that LLM-as-judge evaluation handles worst, because the judge model can be wrong about the same things as the model under evaluation. For correctness on domain-specific content, human expert review against ground truth is the only reliable method.

The second is format compliance — did the output match the required structure? JSON validity, required field presence, length constraints, citation formatting, prohibited phrases. This is highly automatable and should be fully automated. Any format compliance failure that reaches a human reviewer is a process failure upstream.

The third is quality — is the output good, not just correct and well-formatted? Quality evaluation is where LLM-as-judge actually earns its place. Trained on calibrated examples from your domain, an LLM judge can evaluate fluency, tone appropriateness, completeness, and conciseness at scale and speed that human review cannot match. The key word is calibrated: your judge model must be explicitly aligned to your quality rubric through examples, not just given a vague instruction to "rate quality from 1 to 10."

The adversarial mindset.

The evaluation work that separates teams with reliable AI features from teams with unreliable ones is adversarial thinking — actively trying to find the ways the system fails before users do.

This means thinking about your input distribution the way a security researcher thinks about an attack surface. What inputs, if they arrived, would cause the most damage? What assumptions does your prompt make that a real user will violate? Where does the model's training data distribution diverge from your user's language and context?

Consider a WhatsApp-based customer intake system whose primary users are non-native English speakers in Southeast Asia. Its adversarial set should include: messages in Singlish and Manglish, misspellings common to mobile autocorrect, requests that mix English and Bahasa Indonesia, messages that are screenshots of text, and voice-memo transcripts with characteristic disfluencies. None of these tend to appear in an initial golden set written by the team. All of them are the kind of input real users send from the first week.

The adversarial set does not need to be exhaustive. It needs to be systematic. If you can articulate the axes of variation in your input distribution, you can generate adversarial examples that stress-test each axis. Inputs that are too long, too short, too formal, too casual, ambiguous about intent, clear about the wrong intent, in the wrong language, in no language at all.

Running the evaluation: practical mechanics.

An evaluation pipeline for a production AI feature should be runnable in under thirty minutes and produce a report your entire team can interpret without a data science background. If it takes two days and requires a specialist to read, it will not get used.

The structure that works: every evaluation run produces three numbers — a correctness rate on the golden set, a format compliance rate on the full set, and a quality score on a calibrated 1-5 rubric. Any individual example that fails correctness or format compliance is logged with the input, the expected output, the actual output, and the failure category. This log is the artifact that drives the next sprint of prompt engineering or fine-tuning work.

Model upgrades trigger a full regression run against the current production prompt. Prompt changes trigger a regression run before deployment. This is not onerous if the evaluation pipeline is automated. It takes thirty minutes to run and fifteen minutes to interpret. The alternative — deploying prompt changes without regression testing — costs you hours of incident management when the change that improved one behaviour degraded another that you forgot about.

• 01
  Define success before you look at outputs. A rubric written after seeing model outputs is rationalisation, not evaluation. Write the acceptance criteria first, even if it takes a full day. That day will save weeks of disagreement downstream.
• 02
  Build the adversarial set before launch, not after. Wait for production failures to teach you your failure modes and you will be teaching users your failure modes. Spend two days thinking adversarially before launch and you will intercept most of the worst cases yourself.
• 03
  Automate format compliance checks completely. Any format failure that reaches a human is a pipeline failure. Structural validation should be automated, fast, and run on every output in production, not just in evaluation.
• 04
  Calibrate your LLM judge before trusting it. LLM-as-judge without calibration produces consistent ratings that correlate poorly with human expert ratings. Run 50 examples through both your judge and human experts, measure agreement, and tune your judge prompt until agreement exceeds 80% before relying on it at scale.
• 05
  Add every production failure to the regression set. The regression set is your institutional memory. A team that does not maintain it will fix the same class of bug three times across three model upgrades. A team that does will catch regressions before users do.
• 06
  Make the eval report readable by non-specialists. If the output of an evaluation run is a Jupyter notebook, it will not influence the people who make shipping decisions. Three numbers and a log of failures, formatted as a one-page summary, will.

Frequently asked questions.