LLM-as-judge scores feel safer than vibes because they are numbers. That is exactly why they are dangerous.
A judge prompt is still a model call. It can over-reward fluent nonsense, miss a tool-use error, drift when the judge model changes, and disagree with the human reviewers who actually own the product risk. The engineering mistake is not using judges. The mistake is letting an uncalibrated judge become a release gate.
The harness rule I use is simple:
Decision rule: A judge is not allowed to block or approve a release until it has a calibration set, a disagreement budget, and trace-shaped inputs.
That rule turns evaluation from a dashboard exercise into harness engineering. The goal is not to produce one impressive score. The goal is to know when the score is trustworthy enough to act on.
The judge is part of the harness, not the oracle
OpenAI's eval guidance defines evals as tests that check model outputs against specified style and content criteria, then uses a loop of describing the task, running test inputs, analyzing results, and iterating. That sounds close to behavior-driven development for LLM systems, and it is the right mental model: specify the behavior before you trust the implementation.
But a production harness has more than one kind of assertion. OpenAI's grader docs describe several grader types, including string checks, text similarity, score-model graders, and Python code execution. That taxonomy matters because it prevents a common anti-pattern: asking one LLM judge to grade everything.
A good harness splits the work:
| Harness layer | Best for | Failure mode if skipped |
|---|---|---|
| Deterministic checks | JSON validity, schema, tool name, required citations, forbidden actions | The judge wastes tokens grading outputs that should fail immediately |
| Reference-based checks | Known answers, required fields, expected tool arguments | Rubric scores hide objective regressions |
| LLM judge rubric | Helpfulness, faithfulness, policy nuance, synthesis quality | Teams reduce subjective product quality to brittle string matching |
| Human calibration | Ambiguous cases, severity labels, false-pass review | The judge optimizes for its own taste instead of product risk |
| Online sampling | Production drift and rare edge cases | The offline set ages while real traffic changes |
If a failure can be detected by code, do not delegate it to a judge. Save the judge for cases where human-like judgment is the point.
Build calibration sets before scorecards
LangSmith's evaluation concepts recommend identifying critical components before building evals: LLM calls, retrieval steps, tool invocations, output formatting, and quality criteria for each. The same guide recommends starting with manually curated examples, often 5 to 10 examples of what good looks like for each critical component.
That small curated set is not a benchmark. It is a calibration instrument.
For a RAG answer harness, the first calibration set might include:
- one answer with correct citations but missing an important caveat
- one answer with a plausible unsupported claim
- one answer that refuses correctly because evidence is insufficient
- one answer that cites the wrong retrieved chunk
- one answer that is factually correct but violates the requested format
For an agent harness, LangSmith separates evaluation targets into final response, single step, and trajectory. That gives you a better calibration shape:
id: refund-policy-tool-wrong-args
input:
user_message: "Can I return an opened device after 40 days?"
trace_expectations:
required_tool: lookup_policy
forbidden_tool: issue_refund
required_arguments:
policy_area: returns
judge_expectations:
correctness: fail
reason: "The answer is confident but the tool call queried warranty policy, not returns policy."
human_severity: release_blocker
The important field is not only the expected score. It is the reason. A calibration set teaches the evaluator what kind of mistake matters. It also gives reviewers a way to inspect judge disagreements without reverse-engineering the prompt.
Trace-shaped inputs beat answer-only judging
Final-answer judges are attractive because they are easy to bolt onto a demo. They are also hard to debug. LangSmith's application-specific evaluation guide notes that final-response evaluation treats the agent as a black box, can take a while to run, and does not evaluate what happens inside the agent.
That is a poor fit for release gates. Senior engineers rarely need a single scalar that says "bad." They need to know which contract failed.
For agents, include the trace features the judge needs:
{
"case_id": "ticket-triage-017",
"user_goal": "Classify an IT support ticket",
"final_answer": "Hardware",
"tool_calls": [
{
"name": "classify_ticket",
"arguments": { "category": "Hardware" },
"latency_ms": 184
}
],
"retrieved_documents": [],
"deterministic_checks": {
"schema_valid": true,
"allowed_tool": true,
"required_category_present": true
},
"rubric_version": "judge.ticket_triage.v3"
}
The judge should see enough context to evaluate the actual system path, not just the final prose. This also aligns with the direction of production observability. OpenTelemetry's GenAI semantic convention work exposes standard surfaces around GenAI spans, events, and metrics. Whether you use that exact schema or an internal one, the harness should consume trace-shaped records rather than copy-pasted transcripts.
Use disagreement budgets, not average scores
Averages are comfortable and misleading. A model can improve its mean score while introducing one release-blocking false pass. For harness judges, the more useful question is:
How many known bad cases did this judge allow through, and are those cases in risk classes we care about?
A practical calibration report should include:
- false-pass count by severity
- false-fail count by severity
- agreement with human labels
- agreement by component, such as retrieval, tool call, format, safety, and final answer
- drift from the previous rubric or judge model
- examples whose labels are unstable across repeated runs
That report changes the release policy. Instead of "ship if average judge score is above 0.85," use a gate like:
ship if:
deterministic_failures == 0
critical_false_passes == 0
high_severity_false_pass_rate <= 1%
judge_human_agreement >= 0.9 on calibration set
no regression on release-blocker fixtures
The exact numbers should match the product risk. The key is that the policy talks about false passes and severity, not just score.
Online judges need filters and sampling
Offline calibration is necessary, but it is not enough. Real users create new ambiguity. Tools fail in new ways. Retrieval indexes age. Model upgrades change style and uncertainty behavior.
LangSmith's online LLM-as-judge docs describe evaluators that run on production traces, with filters for selecting runs and sampling rates for cost control. That is the production pattern: do not judge every trace forever. Select the traffic that teaches the harness something.
Good online filters include:
- runs where users left negative feedback
- runs that invoked high-risk tools
- runs with low retrieval confidence
- runs from newly deployed prompts or model versions
- runs with long trajectories or retries
- enterprise or regulated workflows that have different risk budgets
Then feed the disagreements back into the offline set. OpenAI's dataset guide recommends expanding evaluation data over time as edge cases or blind spots appear. That is how an eval harness stays alive: production disagreements become new calibration fixtures, not just dashboard rows.
Calibrate the rubric like code
LangSmith's few-shot evaluator guide is useful because it makes one operational point explicit: LLM-as-judge effectiveness depends on quality and alignment with human reviewer feedback, and human corrections can be inserted as few-shot examples to guide future scores.
Treat those corrections as rubric migrations.
Every judge should have:
- a rubric version
- a calibration dataset version
- example-level explanations
- a changelog for scoring semantics
- a replay report against prior release-blocker cases
- an owner who can decide whether a disagreement is a judge bug or a product-policy change
When the rubric changes, replay it. When the judge model changes, replay it. When the application prompt changes, replay it. If the harness cannot replay calibration cases, the judge is not a release gate. It is a comment generator.
The production checklist
Before an LLM judge graduates from advisory signal to release gate, require this checklist:
- Deterministic assertions run first and fail closed.
- The judge receives structured, trace-shaped inputs.
- The calibration set includes good, bad, borderline, and high-severity cases.
- Human labels include explanations, not only scores.
- The release policy uses false-pass budgets by severity.
- Online sampling feeds production disagreements back into the offline set.
- Rubric, model, and dataset versions are recorded with every score.
- Replays run before model, prompt, tool, or rubric upgrades.
That is harness engineering. The artifact is not a judge prompt. The artifact is a calibrated decision system that knows which failures are objective, which failures need judgment, and which disagreements deserve human review.
The winning team is not the one with the most eval scores. It is the one that can explain why a score is safe enough to block a deploy.
Sources
- OpenAI, Working with evals
- OpenAI, Getting started with datasets
- OpenAI, Graders
- LangSmith, Evaluation concepts
- LangSmith, Application-specific evaluation approaches
- LangSmith, How to improve your evaluator with few-shot examples
- LangSmith, Set up LLM-as-a-judge online evaluators
- OpenTelemetry, Generative AI semantic conventions
References
- OpenAI, Working with evals
- OpenAI, Getting started with datasets
- OpenAI, Graders
- LangSmith, Evaluation concepts
- LangSmith, Application-specific evaluation approaches
- LangSmith, How to improve your evaluator with few-shot examples
- LangSmith, Set up LLM-as-a-judge online evaluators
- OpenTelemetry, Generative AI semantic conventions