How to Build LLM Evals That Catch Real Failures

TL;DR: The Verdict on LLM Evals

A developer pastes a prompt into a playground, reads ten outputs, nods, and ships. Two days later a customer pastes a malformed product code, the model answers with a confident fabrication, and support tickets pile up. Manual spot-checking does not scale. “It looked fine in the playground” is not a test strategy.

The fix is boring and it works: a golden eval set of ten to fifty real cases, one deterministic checker, and one LLM-as-judge calibrated against human ratings. Run all three each time you touch a prompt, a model version, or a retrieval config. Do this before you pay for an eval platform. A platform scales a process you already trust. It never creates one.

I have watched teams skip this and pay for it. A model update improves most outputs with no warning. Without evals you learn about it from churn instead of from CI. Build the eval set first. Everything else in this article shows you how.

Also Read:

Prompt Caching Explained: Slash Your LLM API Bills

Why LLM Outputs Break Traditional Unit Tests

Unit tests assert values. Evals assert behavior. That gap is the whole problem.

Traditional tests check exact outputs: given this input, return that string. LLM outputs refuse to cooperate. The same prompt produces different phrasing on each run. Two answers can differ word by word and both be correct. A third can reads well, matches the tone, and contains one fabricated fact that costs you a customer. Exact-match assertions catch none of this.

Quality in LLM output is rubric-shaped, not boolean-shaped. You care about factuality, completeness, tone, format compliance, and refusal behavior on out-of-scope questions. None of these reduce to assert output == expected. Anthropic’s eval guidance frames it this way: “Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them.” The criteria come first. The checks follow.

Also Read:

Vector Databases: pgvector vs Pinecone vs Qdrant

There is a second trap. Teams version their prompts like code but test them like demos. A prompt change is a behavior change, and behavior needs a regression suite. The workflow that works for code reviews, write the failing test, make the change, watch the suite, ports to prompts with one substitution: the “test” is a small dataset plus a judge that scores each output. The same discipline that makes testing AI-generated code tractable applies to prompts. Concrete cases in, scores out, diff the scores.

The Four Evaluator Types Compared

Every eval you will ever build uses one of four evaluator types. LangSmith’s evaluator taxonomy names them: code rules, human review, LLM-as-judge, and pairwise comparison. Each catches different failures and each lies to you in a different way.

Notice the pattern. Deterministic checks are cheap and blind to meaning. Human checks see meaning and cost a fortune. LLM judges sit in the middle, which is why they dominate production eval stacks, and why section five of this article exists: an uncalibrated judge is a random number generator with confidence.

Also Read:

LLM Structured Outputs: OpenAI vs Anthropic vs Google

Start with code rules. They cost nothing, they run in CI, and they catch the embarrassing failures: unparseable JSON, truncated outputs, wrong languages. Then add one LLM-as-judge for the semantic layer. Keep human review as the calibration source and the appeals court, not the front line.

Building a Golden Eval Set That Catches Regressions

A golden eval set is a folder of real cases with known-good expectations. Ten cases beat zero. Fifty well-chosen cases beat two hundred synthetic ones. The recipe below is the one I recommend to teams shipping their first LLM feature.

1. Harvest from production traces. Pull recent real inputs from your logs: the successes, the failures, and the weird ones. Production data beats imagined data because customers are more creative than your test plan. Strip personal data first. Ten to twenty cases from real traffic give the set its backbone.

Also Read:

Foreman Review: Reliability Gates for Scheduled AI Agents

2. Add the edge cases on purpose. Write the cases you hope never arrive: empty input, contradictory instructions, prompt-injection attempts, non-English input, input ten times longer than any real one. These are the cases that break models in production, and they are the ones spot-checking never covers.

3. Define pass criteria you can measure. Anthropic’s docs give the template: criteria must be specific and measurable. Their illustrative example sets a bar like “less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter.” That number is an example, not a vendor SLA, but the shape is right. “Answers must be accurate” is a wish. “No fabricated order numbers in 50 sampled outputs, checked by a judge calibrated to 95% human agreement” is a criterion.

4. Run offline evals on every change. LangSmith splits evaluation into two loops and the names say it all: offline evaluation is “test before you ship,” online evaluation is “monitor in production.” Wire the golden set into CI. Change a prompt, swap a model, tune retrieval, and the suite scores the new outputs against the old baseline. A red diff blocks the merge. This is reliable output via prompt chaining taken as a rule: evaluate each chain stage, the final answer included, so a regression points at the stage that caused it.

One warning about benchmarks. Public leaderboards measure general capability, not your product. HELM, the Stanford benchmark suite, showed the field had a measurement problem: its paper reports 16 core scenarios times 7 metrics, with 30 models run across 42 scenarios, and notes that before HELM, models saw evaluation on 17.9% of the core scenarios on average, rising to 96.0% under HELM. Those figures are from the original 2022 paper, not current totals. The lesson stands regardless of version: standardized suites fixed the field’s inconsistency, but your golden set still measures your product. Use benchmarks to pick a model. Use your eval set to ship it.

LLM-as-Judge Done Right

An LLM judge reads a model output and scores it against a rubric. It sounds circular, grading AI with AI, and the research says it works better than the old metrics and worse than careful humans. Both facts matter.

The evidence for judges: G-Eval, a GPT-4-based judging method, reached a Spearman correlation of 0.514 with human ratings on summarization, beating BLEU and ROUGE, which the paper notes “have been shown to have relatively low correlation with human judgments.” That 0.514 is for summarization with a GPT-4 backbone. Do not generalize it to every task. It proves judges can work, not that yours does.

The evidence against blind trust: position bias is documented and gameable. One study showed Vicuna-13B beating ChatGPT on 66 of 80 tested queries with ChatGPT itself as the evaluator, and the change that flipped the result was which answer appeared first in the context. Swap the order and the verdict flips. Any judge you deploy without controlling for this is a coin flip wearing a lab coat.

Three practices make a judge trustworthy:

Demand reasoning before the score. Ask the judge to write its analysis of the output first, then assign the number. Chain-of-thought before scoring raises correlation with human ratings in the G-Eval work and makes bad scores auditable. A bare number with no reasoning is unverifiable output from the component you built to verify outputs.

Swap answer order and average. Run each judgment twice with the candidates in opposite positions. If the verdict changes with the order, the judge is unreliable on that case and the case needs human eyes. This doubles judge cost and halves a specific, known bias. Worth it.

Calibrate against humans on a sample. Score 50 to 100 outputs with both your judge and human reviewers, then compute agreement. If the judge matches humans on 90% or more of cases, promote it to the overnight suite. Below that, fix the rubric or the judge model. Human review is the ground truth you spend to buy a cheap, scalable proxy. Recalibrate each time you change the judge model or the rubric.

Eval Tools Head-to-Head: promptfoo vs LangSmith vs OpenAI Evals

Three tools dominate the conversation, and they solve different problems. Pick by team shape, not by hype. The short version: promptfoo for the terminal, LangSmith for tracing plus evals, OpenAI Evals for shared eval definitions.

Star counts move each day, so treat those numbers as snapshots, not rankings. The decision that matters: a solo builder gets 80% of the value from promptfoo in an afternoon. A team with production traffic and on-call rotations gets value from LangSmith’s tracing plus its online eval loop. A lab or platform team defining evals as shared artifacts gets value from the OpenAI Evals registry format. All three assume you did section four first. No tool invents your golden set for you.

Cost deserves one honest paragraph. Eval runs burn tokens. A 50-case suite with a GPT-4-class judge costs real money per run, which is why teams run the full suite each night and a 10-case smoke subset on every commit. Small models in production change this math in your favor: evals are how you prove a cheaper model is good enough before you downgrade. A $200 eval run that justifies switching to a model at one-tenth the inference cost pays for itself the same week. Keep the judge strong and the candidate cheap, then let the scores decide.

Conclusion: From Benchmarks to Production

Offline evals catch regressions before deploy. They say nothing about the traffic you have never seen. Close the loop with online evals: sample a slice of production traffic, run your judge over it on a schedule, and alert when scores drift. LangSmith’s “monitor in production” loop is this idea productized, but a cron job and a spreadsheet implement the core of it.

The feedback loop is the whole game. A failing production trace becomes a new golden case. The new case becomes a regression test. The regression test blocks the next bad deploy. Each incident makes the suite stronger instead of making the postmortem longer. Teams that run this loop stop fearing model updates. Teams that skip it discover every update in production.

Your starter plan, in order:

This week: collect ten real inputs from your logs and write expected behaviors for each. Add five edge cases. That is your v1 golden set.

Next: add one deterministic checker for format and one LLM-as-judge with a short rubric. Calibrate the judge on fifty human-scored samples.

Then: wire the suite into CI so prompt and model changes trigger it with no manual step. Add production sampling when traffic justifies it.

Anthropic’s engineering team, writing about agents, observed that “the most successful implementations use simple, composable patterns rather than complex frameworks.” Evals are the same story. A small dataset, a cheap assertion, and a calibrated judge beat an elaborate platform with no ground truth underneath. Build the ten-case set this week. Your future self, the one not debugging a 2 a.m. hallucination incident, will be grateful. And when your retrieval layer needs its own scorecard, the same playbook decides your RAG vs fine-tuning decisions: measure both, keep the winner, and let the eval set remember why.