Eval Fixtures Should Start as Incidents

The strongest eval datasets begin as production incidents, not brainstorming sessions. When a real agent failure becomes a small, replayable fixture with its trace, inputs, expected behavior, and release gate, evaluation stops being a dashboard ritual and starts protecting the next deploy.

Credit: Original Sieon Labs diagram, rendered with Kroki from a Mermaid source.

Why prompt spreadsheets miss production risk

A spreadsheet of clever prompts can be useful during prototyping, but it is a weak memory of production risk. Real failures are rarely just bad answers. They are usually combinations of retrieval drift, tool latency, ambiguous state, missing authorization, planner loops, stale cache entries, or a model upgrade that changed how the agent interprets a boundary condition.

That is why an eval set should be treated as a reliability artifact. OpenAI describes evals as tests of model outputs against criteria you specify and frames the basic workflow as describing the task, running test inputs, analyzing results, and iterating on the application OpenAI evals guide. LangSmith makes the same operational point from the application side: evaluations are quantitative measurements because small prompt, model, or input changes can significantly affect LLM application behavior LangSmith evaluation quick start.

The trap is to stop at the input and expected answer. For a production agent, the expected answer is only the visible symptom. The more important asset is the path that produced it. Did the agent retrieve the right document but ignore it? Did it call the right tool with the wrong identifier? Did it retry a non-idempotent operation? Did it lose the deadline while waiting for a model fallback? A prompt spreadsheet cannot answer those questions. A fixture built from an incident can.

The incident-to-fixture pipeline

The pipeline is simple enough to run after every serious failure review:

1. Capture the production trace and the user-visible outcome.
2. Reduce the trace into a minimal reproduction.
3. Name the failure mode in engineering language.
4. Add an evaluator that scores that failure mode directly.
5. Put the fixture into a pre-release gate.
6. Feed the same signal back into runtime monitoring.

This turns incident review from narrative memory into executable memory. It also stops the team from debating whether a new model is "better" in the abstract. The question becomes narrower and more useful: does this release still handle the production failures we already paid to discover?

Tracing is the hinge. LangSmith's observability quick start focuses on tracing LLM applications and points teams toward tracing more than just single LLM calls, including LangChain and LangGraph applications LangSmith observability quick start. OpenTelemetry has also moved its GenAI semantic conventions into a dedicated repository OpenTelemetry GenAI conventions, with separate documentation for agent spans and GenAI events agent spans GenAI events. The industry direction is clear: model calls are no longer enough. The runtime path matters.

What a good fixture contains

A useful incident fixture is small, boring, and specific. It should not attempt to simulate the whole production system. It should preserve exactly enough context to reproduce the failure mode and catch the regression again.

A practical fixture usually contains:

Field	Why it matters
Incident id	Ties the fixture to a real failure review, support ticket, or postmortem.
User task	Captures what the user or upstream service was trying to accomplish.
Inputs	Preserves prompt, retrieved snippets, tool responses, policy state, and relevant metadata.
Trace links	Keeps the fixture connected to spans, events, latency, retries, and tool calls.
Failure mode	Names the engineering bug, such as wrong tool argument, stale retrieval, or unsafe retry.
Expected behavior	Defines the smallest acceptable behavior, not a perfect answer.
Evaluator	Scores the failure mode with deterministic checks or a constrained judge.
Owner	Assigns maintenance to the team that owns the runtime boundary.

The most important field is the failure mode. Without it, evals drift toward taste tests. With it, the team can pick the right scoring strategy. A retrieval freshness bug needs source and timestamp checks. A tool argument bug needs schema and side-effect checks. A refusal policy bug needs policy-state fixtures. A planner loop needs step count, deadline, and termination checks.

Evaluators should score the failure mode, not the model personality

Many teams start with LLM-as-judge scoring because it is flexible. That is fine as a starting point, but the judge should not be asked whether the answer is "good." It should be asked whether the known failure mode is absent.

LangSmith describes evaluations as three pieces: a dataset, a target function, and evaluators that score the target function's outputs LangSmith evaluation quick start. That decomposition is useful because it forces the team to separate what is being tested from how it is scored. The dataset is the incident fixture. The target function is the current agent or a narrowed subsystem. The evaluator is the release contract.

For production agents, start with deterministic checks whenever possible:

• Did the answer cite one of the retrieved documents?
• Did the tool call use the expected account, workspace, or resource id?
• Did the agent avoid a write operation when approval state was absent?
• Did the run finish inside the deadline budget?
• Did the planner stop after the required handoff receipt was created?

Use a judge only for the part that is genuinely semantic. Even then, constrain it with a rubric tied to the incident. A judge can decide whether the answer explains a billing policy accurately. It should not decide whether the agent was allowed to call the refund tool. That is policy state and tool authorization, so it should be checked directly.

Wire fixtures into release gates and runtime guardrails

An incident fixture earns its keep only when it blocks the next unsafe change. Put the fixture into the same release path that model upgrades, prompt edits, retrieval changes, and tool schema updates must pass. If a fixture fails, the release should either stop or require an explicit risk acceptance from the owning team.

This does not mean every fixture must run on every commit. Mature systems use tiers:

• Smoke fixtures run on every prompt or policy edit.
• Component fixtures run on retrieval, tool, memory, or planner changes.
• Full incident suites run before model upgrades and production deploys.
• Canary fixtures run against live traffic shadows when a runtime change cannot be simulated fully.

The same fixture can also feed runtime monitoring. If a postmortem found that an agent silently used stale retrieved content, add the source timestamp check to the eval suite and emit the same condition as an alert in production. OpenTelemetry's GenAI work is valuable here because it gives teams a common vocabulary for spans and events around generative AI interactions OpenTelemetry GenAI events. The naming does not remove the need for domain-specific checks, but it makes traces less bespoke.

Minimal runnable fixture example

A fixture can start as a plain JSON object and a small Python evaluator. The point is not to build a framework on day one. The point is to preserve the failure mode in executable form.

{
  "incident_id": "INC-2026-08-07-014",
  "failure_mode": "retrieval_answer_ignored_source_timestamp",
  "user_task": "Summarize the current refund policy for annual plans.",
  "retrieved_documents": [
    {
      "id": "policy_2025_old",
      "title": "Refund policy, 2025",
      "updated_at": "2025-11-02T10:00:00Z",
      "text": "Annual plans are refundable for 30 days."
    },
    {
      "id": "policy_2026_current",
      "title": "Refund policy, 2026",
      "updated_at": "2026-06-15T09:00:00Z",
      "text": "Annual plans are refundable for 14 days."
    }
  ],
  "expected_current_document_id": "policy_2026_current",
  "candidate_answer": "Annual plans are refundable for 30 days."
}

from datetime import datetime, timezone

fixture = {
    "expected_current_document_id": "policy_2026_current",
    "retrieved_documents": [
        {"id": "policy_2025_old", "updated_at": "2025-11-02T10:00:00Z"},
        {"id": "policy_2026_current", "updated_at": "2026-06-15T09:00:00Z"},
    ],
    "candidate_answer": "Annual plans are refundable for 30 days.",
}


def parse_z(ts: str) -> datetime:
    return datetime.fromisoformat(ts.replace("Z", "+00:00")).astimezone(timezone.utc)


def score_refund_policy_fixture(fx: dict) -> dict:
    newest = max(fx["retrieved_documents"], key=lambda d: parse_z(d["updated_at"]))
    used_current_document = newest["id"] == fx["expected_current_document_id"]
    answer_mentions_current_window = "14 days" in fx["candidate_answer"]
    passed = used_current_document and answer_mentions_current_window
    return {
        "passed": passed,
        "newest_document_id": newest["id"],
        "reason": "answer must follow the newest policy document",
    }


if __name__ == "__main__":
    result = score_refund_policy_fixture(fixture)
    if not result["passed"]:
        raise SystemExit(result)

This is intentionally small. In a real suite, the target function would run the current agent against the fixture inputs, then the evaluator would inspect both the answer and trace. If the failure was a wrong tool call, the evaluator would inspect the captured tool call. If the failure was a planner loop, it would inspect step counts and terminal state. If the failure was stale retrieval, it would inspect source metadata.

Operating model and failure review cadence

The hard part is not writing the first evaluator. The hard part is keeping fixtures useful after the system changes. Assign ownership the same way you assign service ownership. Retrieval fixtures belong to the retrieval owner. Tool authorization fixtures belong to the integration owner. Planner and handoff fixtures belong to the runtime owner.

During incident review, add one question after root cause: what is the smallest fixture that would have failed before this incident reached production? If nobody can answer, the team probably does not understand the failure yet. If the answer requires a huge end-to-end simulation, reduce it until the failure mode is isolated.

Review the suite monthly. Delete or archive fixtures that no longer represent a real risk. Split fixtures that test too many things. Promote recurring failures into release gates. Most importantly, keep the fixture connected to the original trace. When a future engineer asks why a strange evaluator exists, the trace should answer faster than a wiki page.

This is how evals become engineering infrastructure. They are not a scorecard for model vibes. They are executable postmortems that make the next release remember what production already taught the team.

FAQ

Should every production incident become an eval?

No. Convert incidents that expose a repeatable failure mode, a boundary the system must preserve, or a regression risk that future changes could reintroduce. One-off provider outages usually belong in resilience testing, not necessarily in an eval dataset.

How many examples does a fixture need?

Start with one minimal reproduction from the incident. Add variants only when they protect a real dimension of risk, such as another tool, policy state, tenant shape, or retrieval source. A small precise fixture is better than twenty vague examples.

Should teams use LLM-as-judge evaluators?

Yes, but narrowly. Use deterministic checks for ids, timestamps, tool calls, approvals, deadlines, and schema behavior. Use a judge for semantic quality only when the rubric is tied to the incident and the judge output is not the only gate.

Where should traces live after an eval is created?

Keep a durable link from the fixture to the original trace or a sanitized trace export. The fixture should be runnable without production data, but the trace should remain available for debugging, ownership, and future suite maintenance.

References

1. OpenAI, Working with evals
2. LangSmith, Evaluation Quick Start
3. LangSmith, Observability Quick Start
4. OpenTelemetry GenAI semantic conventions
5. OpenTelemetry GenAI agent spans
6. OpenTelemetry GenAI events