Replay Adapters Make Harnesses Trustworthy

A harness is trustworthy only when a failure can be replayed from the same normalized evidence: case data, prompt boundary, retrieval snapshot, tool IO, model settings, trace identifiers, and assertions. OpenAI describes evals as a dataset plus an eval class, so replay adapters should make production failures become stable dataset cases, not screenshots.

The harness bug nobody sees in the green build

Most AI evaluation harnesses start with a responsible instinct: capture examples, run the model, score the output, and block a release when quality drops. That shape matches the public OpenAI Evals framing that an eval combines a dataset with an eval class and then runs that registered evaluation OpenAI Evals. The trouble is not the shape. The trouble is that the dataset is often weaker than the incident it came from.

A production agent failure usually contains more than input and output. It has the system prompt version, tool schema, retrieved documents, model parameters, streamed tool calls, user-visible partial state, latency, token usage, and the trace that tells you where the bad decision entered the run. OpenTelemetry now maintains Generative AI semantic conventions across spans, events, exceptions, metrics, MCP, and LLM-call examples, which is a strong signal that AI runtime evidence is becoming structured operational data OpenTelemetry.

The decision rule is simple: if a failed run cannot be replayed without asking an engineer to remember what happened, the harness is not protecting the system. It is protecting a story about the system. Senior teams should treat replay as a first-class adapter boundary, just as pytest treats setup resources as explicit fixtures requested by tests pytest fixtures.

What a replay adapter is

A replay adapter converts volatile run evidence into a stable evaluation case. It is not the judge, not the fixture, not the trace exporter, and not the dataset itself. Its job is to take one production or staging run and produce a case that can be stored, reviewed, scrubbed, versioned, and executed by the harness.

That boundary matters because each neighboring component has a different failure mode. A fixture provides setup state for a test, and pytest documents fixtures as reusable resources that tests request explicitly pytest fixtures. A judge scores behavior. A trace exporter records operational spans and events. A dataset stores the cases that an eval runner consumes. The replay adapter is the translator that decides whether runtime evidence is complete enough to become a durable case.

A good adapter emits fewer cases than a naive logging pipeline. That is a feature. The adapter should reject a trace when required evidence is missing, when a retrieval snapshot cannot be identified, when tool outputs include unsafely retained secrets, or when the expected assertion is too vague to survive a model upgrade. LangSmith’s evaluation documentation frames evaluation around datasets, evaluators, and experiments for LLM applications, and that separation is useful because replay should feed the dataset layer before scoring begins LangSmith evaluation.

The contract I want in every replay case

The minimum contract is small enough to implement but strict enough to catch harness drift:

Field	Why it belongs in the replay case	Failure if omitted
schema_version	Lets the harness migrate old cases intentionally	Old incidents silently run with new assumptions
source_trace_id	Connects eval evidence to the observed run	Engineers cannot audit where a case came from
input_messages	Preserves the user and system boundary	Prompt regressions look like model regressions
model_config	Captures model, temperature, tool choice, and limits	Replay changes when runtime defaults change
retrieval_refs	Pins document ids, corpus version, or snapshot time	RAG failures disappear after reindexing
tool_calls	Captures tool names, arguments, responses, and errors	Agent planning is scored without action evidence
assertions	Names the invariant the case must protect	Judges grade vibes instead of release rules
scrubbed_metadata	Keeps routing, tenant class, and risk labels without secrets	Privacy cleanup removes operational context

OpenAI’s documented eval-building flow begins by building the dataset, then registering an eval with that dataset, and then running it OpenAI Evals. A replay adapter should make that first step boring. The output is no longer a handmade YAML example created during a postmortem. It is a normalized case whose provenance, completeness, and safety checks are enforced before the case reaches the harness.

A minimal replay adapter

This example is intentionally small. It shows the adapter boundary, not a full tracing stack. The adapter refuses to create a case unless the run contains the fields that make replay meaningful.

from dataclasses import dataclass
from typing import Any

REQUIRED = [
    "trace_id",
    "input_messages",
    "model_config",
    "retrieval_refs",
    "tool_calls",
    "expected",
]

@dataclass(frozen=True)
class ReplayCase:
    schema_version: str
    source_trace_id: str
    input_messages: list[dict[str, str]]
    model_config: dict[str, Any]
    retrieval_refs: list[dict[str, str]]
    tool_calls: list[dict[str, Any]]
    assertions: list[dict[str, str]]
    scrubbed_metadata: dict[str, str]


def to_replay_case(run: dict[str, Any]) -> ReplayCase:
    missing = [name for name in REQUIRED if not run.get(name)]
    if missing:
        raise ValueError(f"cannot replay run; missing {missing}")

    return ReplayCase(
        schema_version="replay-case.v1",
        source_trace_id=run["trace_id"],
        input_messages=run["input_messages"],
        model_config={
            "model": run["model_config"]["model"],
            "temperature": run["model_config"].get("temperature", 0),
            "tool_choice": run["model_config"].get("tool_choice", "auto"),
            "max_output_tokens": run["model_config"].get("max_output_tokens"),
        },
        retrieval_refs=run["retrieval_refs"],
        tool_calls=run["tool_calls"],
        assertions=[{"kind": "must_satisfy", "value": run["expected"]}],
        scrubbed_metadata={
            "route": run.get("route", "unknown"),
            "risk": run.get("risk", "unknown"),
        },
    )

The most important line is the rejection path. A partial trace is useful for debugging, but it should not automatically become a release gate. Pytest’s tmp_path fixture gives each test its own temporary directory for isolated filesystem state pytest tmp_path. Replay adapters need the same discipline for AI evidence: isolate the case, keep the boundary explicit, and fail closed when the setup cannot be reconstructed.

Where replay belongs in the harness pipeline

Put replay before scoring. The pipeline should look like this:

flowchart LR
    A[Production or staging trace] --> B[Replay adapter]
    B -->|complete and scrubbed| C[Versioned eval case]
    B -->|missing evidence| D[Quarantine queue]
    C --> E[Harness fixtures]
    E --> F[Model or agent run]
    F --> G[Evaluator or judge]
    G --> H[Release decision]

This ordering keeps the judge honest. A model grader can be useful, and LangSmith documents evaluator-driven application evaluation workflows LangSmith evaluate. But an evaluator should not repair a missing trace, invent the retrieval snapshot, or infer which tool response was available. Those are adapter responsibilities.

The quarantine queue is also important. It should not be a graveyard. It is the backlog of instrumentation debt. If traces are frequently rejected because retrieval snapshots are missing, the retrieval service has a harness integration bug. If traces are rejected because tool output contains unsafe material, the tool boundary needs a scrubber. If traces are rejected because assertions are unclear, the product invariant is not ready for automated gating.

Tradeoffs that make replay adapters worth the cost

Replay adapters add engineering work, and the cost is real. You need schema migrations, artifact storage, privacy review, source trace links, and a policy for expiring cases that no longer represent supported product behavior. That is still cheaper than a harness that passes while production regresses.

The biggest tradeoff is freshness versus reproducibility. A RAG application may rely on a live index, but a replay case needs to explain which evidence the original run saw. Store document ids, corpus version, and retrieval parameters when possible. When the product requires live freshness, split the eval into two cases: one deterministic replay case for regression safety and one freshness probe that intentionally checks the current index. OpenTelemetry’s Generative AI conventions give teams a public vocabulary for recording AI spans and events, which helps make that split observable OpenTelemetry.

The second tradeoff is coverage versus signal. Not every trace deserves to become an eval. Prioritize incidents, high-value routes, policy-sensitive flows, and examples that reveal a specific invariant. OpenAI’s eval-building guidance asks contributors to think about what makes an eval interesting and identifies categories such as safety, steerability, hallucinations, reasoning, and real-world use cases OpenAI Evals. Internal harnesses should use the same taste. A replay case should teach the system a rule it must keep.

The third tradeoff is determinism versus stochastic behavior. Replay does not make a model deterministic by magic. It makes the evidence deterministic enough that remaining variance is visible. If the same case fails only at temperature 0.7, the harness learned something. If it fails only when the retriever silently changed, the harness also learned something. Without replay, those two failures look the same.

Implementation pitfalls

Do not store raw production payloads as eval cases. The adapter must scrub secrets and private data before saving the case, and it should keep only metadata that matters for routing, risk, and debugging. A public harness article should never need private notes to explain why the case exists.

Do not let the adapter mutate the case during scoring. Once a replay case is accepted, the evaluator should read it as an immutable artifact. If a schema change is needed, write a migration and bump schema_version. Silent case mutation is how evaluation history becomes unreproducible.

Do not collapse tool errors into text summaries. Agent behavior often depends on whether a tool timed out, returned a validation error, or produced a partial result. Preserve tool name, arguments, response shape, error class, and whether the model saw the error. OpenTelemetry’s conventions include GenAI exceptions and events as part of the semantic convention surface, which supports treating those details as runtime evidence rather than prose OpenTelemetry.

Do not treat replay as a replacement for unit tests. Use regular tests for deterministic helper logic, fixtures for controlled setup, and replay cases for AI behavior that depends on prompts, retrieval, tools, and model outputs. Pytest’s fixture model is a useful comparison because it keeps setup explicit and composable pytest fixtures.

The release gate

The release gate should ask three questions before it trusts a replay-backed harness. First, can every failing case link back to a source trace or incident? Second, can every passing case state the assertion it protects? Third, can the harness explain which adapter version produced the case?

If the answer to any question is no, the gate should not block a release by itself. It can warn, quarantine, or request review, but it should not pretend to be a production control. Once those answers are yes, the harness becomes much more valuable. A failed case points to an auditable run. A passed case documents a protected invariant. A schema migration becomes an explicit engineering change.

That is the memorable idea: replay adapters turn evals from opinions into receipts. They do not make AI systems simple. They make the evidence stable enough that senior engineers can debug, compare, and govern them.

FAQ

Is a replay adapter just another fixture?

No. A fixture sets up resources for a test, and pytest documents fixtures as requested setup resources pytest fixtures. A replay adapter converts runtime evidence into a versioned evaluation case before fixtures and evaluators run.

Should every production trace become an eval case?

No. The adapter should select high-signal traces: incidents, risky routes, recurring regressions, and examples tied to clear product invariants. OpenAI’s eval guidance emphasizes datasets that test meaningful behavior, not arbitrary logs OpenAI Evals.

Where should replay artifacts live?

Store them with the evaluation dataset or in an artifact store referenced by the dataset. The key is that the eval runner can resolve the exact case, retrieval references, tool evidence, and adapter schema without depending on a live incident dashboard.

Does replay make stochastic models deterministic?

No. Replay fixes the evidence boundary. Model variance can still exist, but it becomes easier to separate model behavior from changing prompts, tools, retrieval indexes, and missing trace data.

References

1. OpenAI Evals, Building an eval
2. pytest, How to use fixtures
3. pytest, How to use temporary directories and files in tests
4. OpenTelemetry, Generative AI semantic conventions
5. LangSmith, Evaluation concepts
6. LangSmith, Evaluate an LLM application