Agent Evals Need Traces, Not Just Test Prompts

Prompt tests are still useful, but they are the wrong center of gravity for agent quality. A production agent should be evaluated from its trace: the model turns, tool calls, arguments, handoffs, guardrails, state changes, and final answer that made one real workflow succeed or fail.

Why prompt tests miss agent failures

Most LLM evaluation programs started with a simple shape: input prompt, expected output, grader. That shape works for summarizers, classifiers, extractors, and many single-turn assistants. It breaks down once the system starts acting through tools.

The failure is not subtle. A customer-support agent can give a reasonable final response after calling the wrong internal lookup. A coding agent can produce a correct patch after reading a file it was not allowed to read. A research agent can cite the right source while skipping the retrieval step that would have prevented a stale answer. A workflow router can hand off to the right specialist only after leaking private context to the wrong one.

OpenAI's agent evaluation guidance makes the same operational distinction: trace grading is meant for workflow-level issues, and a trace captures model calls, tool calls, guardrails, and handoffs for one run (OpenAI agent evals). LangSmith's complex-agent evaluation docs separate final-response evaluation from trajectory evaluation and single-step evaluation, because the path matters as much as the final text for agent systems (LangSmith complex-agent evals).

That is the core shift. Agent evals should not ask only, "Did the answer look right?" They should ask, "Did the system make the right decisions under the same constraints a production run had?"

The evaluation unit should be a decision trace

A useful agent eval case is closer to an incident replay than a prompt fixture. The input prompt is only the first line of the record. The case should preserve the decisions that define the behavior:

Layer	What to evaluate	Example failure
Routing	Which agent, graph node, or policy branch handled the request	Refund request sent to general Q&A
Retrieval	Which sources or records were selected	Old policy document outranked current policy
Tool choice	Which tool was selected and why	Write tool used when read-only lookup was required
Tool arguments	Whether arguments were valid, minimal, and authorized	user_id inferred from text instead of session identity
Handoffs	Whether control moved to the right specialist	Security review skipped for a destructive action
Guardrails	Whether policy checks fired at the correct point	Approval requested after the side effect
Final output	Whether the user-facing answer is accurate and useful	Correct action, vague or misleading explanation

OpenAI's Agents SDK tracing records spans for runner invocations, model turns, agents, generations, function tool calls, guardrails, handoffs, and custom events (Agents SDK tracing). That span model is a good mental model even if your stack is not OpenAI. You need a durable representation of what the agent did, not just what it said.

Here is the architecture I usually want before I trust an agent release:

flowchart LR
  A["Production run"] --> B["Trace capture"]
  B --> C["Incident triage"]
  C --> D["Eval case"]
  D --> E["Replay harness"]
  E --> F["Trajectory graders"]
  E --> G["Step graders"]
  E --> H["Final answer graders"]
  F --> I["Release gate"]
  G --> I
  H --> I
  I --> J["Deploy or rollback"]

This is an original Sieon Labs diagram. It has no external image dependency.

What to store in a replayable eval case

A trace is not automatically an eval. Raw traces are often too noisy, too sensitive, and too dependent on production infrastructure. The job is to compress the trace into a replayable case that preserves the decisions you care about.

A practical case file can look like this:

{
  "case_id": "refund_policy_2026_07_22_001",
  "user_goal": "Customer asks whether a refund is allowed after 34 days.",
  "input": {
    "messages": [
      {"role": "user", "content": "Can I still get a refund for an order from last month?"}
    ],
    "tenant_policy_version": "refunds-2026-07"
  },
  "expected": {
    "route": "policy_lookup",
    "required_tools": ["get_current_policy"],
    "forbidden_tools": ["issue_refund"],
    "final_answer_contains": ["30 days", "support exception path"]
  },
  "fixtures": {
    "get_current_policy": {
      "refund_window_days": 30,
      "exception_channel": "support_review"
    }
  }
}

This is intentionally boring. It stores the user goal, the important input context, expected decisions, and deterministic tool fixtures. It does not store a full production database snapshot. It does not require the same model to produce byte-for-byte identical text. It gives graders enough structure to catch the regression that matters.

OpenAI's Evals API represents evals as configured runs with testing criteria, result counts, and reports for inspection (OpenAI evals). That is the right abstraction level for release gates. The case defines the behavior contract. The run compares a candidate system against that contract.

A production evaluation stack

For a real agent, I would split evaluation into four layers.

1. Contract tests for tools

Tool schemas are production contracts. Before an agent ever runs, validate that tool descriptions, argument schemas, permissions, and return shapes match the actual implementation. These tests are fast and deterministic.

The point is not to prove the model will pick the tool. The point is to make sure a correct tool call can succeed. If lookup_customer(order_id) silently accepts an email address because the schema is too loose, the agent eval suite will waste time diagnosing model behavior when the contract is the bug.

2. Step evals for local decisions

Step evals isolate a single decision: route this request, choose the first tool, approve or block this action, select the retrieval query, or decide whether a handoff is required. LangSmith's docs call out single-step evaluation as a separate mode, including whether an agent selects the appropriate first tool for a step (LangSmith complex-agent evals).

Step evals are where you want many examples. They are cheaper than full workflow replays and easier to debug.

3. Trajectory evals for workflow behavior

Trajectory evals check the path. They answer questions like:

• Did the agent call the read tool before the write tool?
• Did it use the current policy source instead of a cached summary?
• Did it request approval before a side effect?
• Did it stop after a guardrail rejection?
• Did it avoid tools that are irrelevant or overprivileged?

OpenAI's agent-eval guide lists similar trace-grading questions, including whether the agent picked the right tool, whether a handoff happened when it should have, and whether the workflow violated an instruction or safety policy (OpenAI agent evals).

4. End-to-end evals for user impact

Final answer grading still matters. The user sees the answer, not the trace. But by the time an end-to-end case fails, you should already know whether the failure came from routing, retrieval, tool arguments, policy, or wording.

That separation is what makes agent quality work scalable. Without it, every failed example becomes a prompt debate.

Example: turning an incident into a regression test

Consider a support agent that accidentally issued a refund before checking the current refund policy. The final answer was polite. The customer was happy. The business process was wrong.

A prompt-only eval might pass if the model says, "I processed your refund." A trace-first eval should fail because the sequence is unsafe.

EXPECTED = {
    "must_happen_before": [
        ("get_current_policy", "issue_refund"),
        ("approval_check", "issue_refund"),
    ],
    "forbidden_without_approval": ["issue_refund"],
}


def grade_trace(tool_calls: list[dict]) -> dict:
    names = [call["name"] for call in tool_calls]
    failures = []

    for before, after in EXPECTED["must_happen_before"]:
        if after in names and (before not in names or names.index(before) > names.index(after)):
            failures.append(f"{before} must occur before {after}")

    if "issue_refund" in names and "approval_check" not in names:
        failures.append("issue_refund requires approval_check")

    return {
        "pass": not failures,
        "failures": failures,
    }

This grader is deliberately simple. It does not ask a judge model to infer whether the run was safe. It checks the invariant directly. Use model graders when the judgment is semantic. Use deterministic graders when the expected behavior is structural.

Observability and evals should share identifiers

The best eval cases usually come from production traces. That means observability and evaluation need shared identifiers.

OpenTelemetry's GenAI registry includes attributes for conversation identifiers, input messages, output messages, evaluation names, evaluation score labels, and evaluation score values, while noting that the GenAI attributes have moved to a dedicated GenAI semantic conventions repository (OpenTelemetry GenAI attributes). You do not have to adopt every attribute on day one. You do need stable names for the things you will join later:

• workflow name
• trace id
• conversation or task id
• agent version
• prompt version
• tool schema version
• retrieval index version
• policy version
• grader version
• release candidate id

If those identifiers are missing, your post-incident workflow becomes archaeology. Someone exports logs, someone screenshots traces, someone guesses which prompt was deployed, and the regression test never lands.

Privacy is part of the eval design

Trace-first evaluation can become a data leak if teams are careless. The OpenAI Agents SDK tracing docs warn that generation spans and function spans can store model inputs, outputs, and function inputs or outputs, and provide configuration to disable sensitive data capture (Agents SDK tracing). That warning applies broadly, not only to one SDK.

For production systems, decide this before you collect cases:

• Which fields are safe to store as eval fixtures?
• Which fields must be redacted before a trace leaves production logs?
• Which tool outputs should be summarized instead of replayed exactly?
• Who can inspect failed traces?
• How long should eval cases derived from user data live?

A strong pattern is to keep raw traces in the observability system with normal retention and access controls, then promote only minimized cases into the eval repository. The eval case should contain the behavior contract, sanitized inputs, and deterministic fixtures. It should not become a shadow copy of production data.

Rollout checklist

If your agent eval suite is still prompt-first, I would not rewrite everything at once. Add trace-first coverage around the riskiest paths.

1. Pick one workflow with real side effects, such as refunds, deployments, account changes, or database writes.
2. Capture traces for successful runs and incidents.
3. Define the few structural invariants that must never break.
4. Build deterministic graders for those invariants.
5. Add semantic graders only where deterministic checks cannot express the expected behavior.
6. Promote incidents into sanitized replay cases within one business day.
7. Gate releases on the replay suite before changing prompts, tools, routing, or policies.
8. Review failed evals by layer: contract, step, trajectory, final answer.

The important cultural change is that an agent incident should not end with a better prompt. It should end with a replayable case.

FAQ

Why are prompt-only evals not enough for agents?

Because agents fail through actions, not only through text. A final response can look correct while the agent used the wrong tool, skipped a guardrail, chose a stale source, or performed a side effect in the wrong order. Trace grading exists to evaluate those workflow-level decisions (OpenAI agent evals).

What is the minimum trace data an eval should keep?

Keep the user goal, relevant input context, selected route, tool calls, tool arguments, guardrail outcomes, handoffs, final response, and the version identifiers needed for replay. Avoid copying sensitive production data unless the case genuinely requires it.

Should every production trace become an eval case?

No. Most traces are observability data. Promote traces when they represent an incident, a high-value workflow, a known edge case, or a release-blocking behavior. Once the expected behavior is clear, convert the trace into a minimized replay case.

How do teams avoid storing sensitive data in eval traces?

Separate raw observability from eval fixtures. Keep raw traces under normal access controls and retention. Promote only sanitized inputs, deterministic tool fixtures, and behavior expectations into the eval repository. Also configure tracing systems to avoid capturing sensitive span data when your policy requires it (Agents SDK tracing).

References

1. OpenAI, Evaluate agent workflows
2. OpenAI Agents SDK, Tracing
3. LangChain Docs, Evaluate a complex agent
4. OpenTelemetry, Gen AI attributes
5. OpenAI, Working with evals