On this page
- The decision rule: budget the failure before judging the answer
- Split the oracle into evidence, invariants, and tolerances
- Inject known failures, not just golden paths
- Score reliability at the trace boundary
- What this costs
- FAQ
- Where should oracle contracts live?
- Do model graders replace deterministic checks?
- How many failure cases are enough?
- What should happen when a dependency is flaky?
- References
Harness oracles should answer one question before they judge a model: which failures can this product safely tolerate? A senior AI team should encode acceptable timeout rates, schema drift, degraded answers, and missing evidence as versioned test data, because binary expected answers miss the reliability shape of agent systems.
The decision rule: budget the failure before judging the answer
I use a simple rule when an agent evaluation starts getting noisy: do not add another judge until the oracle says what failure it is allowed to forgive. LangSmith frames evaluation as defining what good looks like and measuring non-deterministic LLM behavior, but “good” is not only the final sentence. In an agent, good can mean the model chose the right tool, stayed under a retry budget, preserved a required citation, and returned a degraded answer when the dependency was down.
That is why I treat the oracle as a reliability contract. OpenAI's agent evaluation guide recommends trace grading for workflow-level issues, which is the right level for systems that call tools, run guardrails, and branch. A trace-level oracle can say, “the final answer may be shorter during search outage, but it must disclose degraded retrieval, must not fabricate document ids, and must finish within two retries.” A text-only oracle usually cannot express that.
The memorable decision rule is this: budget the failure before judging the answer. If a harness has no budget for timeout, partial evidence, schema rejection, or tool retry, then every unexpected dependency behavior becomes either a flaky test or a hidden production incident. OpenAI Agents SDK tracing records LLM generations, tool calls, handoffs, guardrails, and custom events, so the evidence exists in many agent runtimes. The harness problem is deciding which evidence becomes an oracle.
A failure budget is not permission to lower quality. It is a way to turn product promises into executable tolerances. A support agent may tolerate one slow retrieval call if it returns a clear fallback. A regulated workflow may tolerate no unsupported answer at all. A code-generation agent may tolerate multiple tool calls if the final patch compiles. The oracle makes those differences explicit, reviewable, and portable across models.
Split the oracle into evidence, invariants, and tolerances
I prefer an oracle file with three layers: evidence, invariants, and tolerances. Evidence is the captured input, tool transcript, retrieval snapshot id, model settings, and expected product context. Invariants are the facts that must always hold, such as valid JSON, no unsupported citation, no unsafe action, or required handoff. Tolerances are bounded exceptions, such as “one retry allowed,” “latency warning allowed,” or “answer may omit optional examples during degraded retrieval.”
This split keeps the harness portable. Python's json module supports encoding and decoding structured JSON data, which is enough for a minimal oracle artifact. Pydantic models validate data against declared fields and types, which is useful when a team wants stronger schema validation. pytest fixtures provide explicit setup resources requested by tests, which makes the same oracle easy to load in CI.
Here is a small, runnable Python 3.13 example. It does not need an LLM call. The point is the contract shape: the oracle checks reliability properties around a run result, not only answer equality.
from dataclasses import dataclass
from typing import Any
@dataclass(frozen=True)
class FailureBudget:
max_tool_retries: int
allow_degraded_answer: bool
require_citation: bool
@dataclass(frozen=True)
class AgentRun:
answer: str
citations: list[str]
tool_retries: int
degraded: bool
schema_valid: bool
def judge_run(run: AgentRun, budget: FailureBudget) -> tuple[bool, list[str]]:
failures: list[str] = []
if not run.schema_valid:
failures.append("response schema was invalid")
if run.tool_retries > budget.max_tool_retries:
failures.append("tool retry budget was exceeded")
if run.degraded and not budget.allow_degraded_answer:
failures.append("degraded answer was not allowed")
if budget.require_citation and not run.citations:
failures.append("required citation was missing")
if "I do not know" in run.answer and not run.degraded:
failures.append("uncertainty was not tied to degraded evidence")
return not failures, failures
if __name__ == "__main__":
budget = FailureBudget(
max_tool_retries=1,
allow_degraded_answer=True,
require_citation=True,
)
run = AgentRun(
answer="Retrieval was degraded, so I can only confirm the summary.",
citations=["trace://case-184/search-result-2"],
tool_retries=1,
degraded=True,
schema_valid=True,
)
passed, reasons = judge_run(run, budget)
print({"passed": passed, "reasons": reasons})
Notice what is absent. There is no model name in the oracle. A model can change without changing the reliability contract. There is also no private prompt dump in the assertion. OpenAI's tracing docs warn that traces can capture sensitive input and output data, so a public article, a shared eval repository, or a vendor dashboard should not become a secret store by accident.
Inject known failures, not just golden paths
Golden-path fixtures prove the system can work. Failure-budget oracles prove the system fails in the way the product promised. I want at least one case for each dependency that regularly hurts users: search timeout, empty retrieval, malformed tool output, rate-limit retry, schema mismatch, and unsafe side-effect request.
pytest monkeypatch can safely set or delete attributes, dictionary items, environment variables, sys.path entries, and the current directory, then undo those modifications after the test. That makes it a practical way to inject missing keys, fake responses, and broken configuration. Python unittest.mock provides patching and mock objects for replacing dependencies and asserting calls, which is useful when the harness must prove that a dangerous tool was not invoked. Tenacity documents retry decorators with explicit stop and wait policies, which gives teams a vocabulary for retry budgets instead of vague “try again” behavior.
The tradeoff is that injected failures must be realistic. A fake timeout that never resembles the real HTTP client exception trains the agent runtime against theater. A fake schema error that omits the production field path makes the repair path untestable. A fake rate limit without a retry-after value hides scheduling behavior. The harness owner should keep failure cases close to production evidence, but scrub the payload before it becomes a reusable artifact.
I usually divide injected failures into three classes. First, deterministic contract failures: invalid JSON, missing citation, forbidden tool, wrong handoff. These should fail every time. Second, bounded dependency failures: one timeout, one retry, empty search result, stale cache. These may pass if the system follows the fallback contract. Third, budget exhaustion failures: repeated retries, unresolved schema mismatch, missing required evidence. These should stop the run and produce a user-visible limitation instead of a confident answer.
This is where senior engineering judgment matters. Too many failure cases make the harness slow and brittle. Too few make it ceremonial. The right set is not “all possible failures.” It is the smallest set that protects the product promise, the incident history, and the interfaces most likely to change.
Score reliability at the trace boundary
A failure-budget oracle becomes much more useful when it scores traces, not screenshots of final text. OpenTelemetry Python instrumentation creates tracers and records spans with attributes and events. OpenTelemetry also publishes GenAI semantic conventions for spans, events, metrics, and model-provider attributes. That public vocabulary lets a team attach eval results to the same run structure used for operations.
I like to emit four numbers from the harness: invariant failures, tolerated failures, exhausted budgets, and missing evidence. Invariant failures catch correctness problems. Tolerated failures show that fallbacks were used. Exhausted budgets catch reliability regressions. Missing evidence catches instrumentation gaps. If missing evidence rises after a deploy, the model may be fine while the harness has gone blind.
This also changes how I review eval dashboards. A pass rate of 94 percent is not actionable unless I know which failures were tolerated. Ten tolerated search degradations can be acceptable during a provider incident. Ten tolerated citation omissions are not tolerable if citations are the product promise. The oracle should make those categories visible before anyone celebrates a green run.
The privacy boundary belongs here too. OpenAI Agents SDK tracing supports custom traces and spans, but the harness should decide what to persist. Store stable ids, scrubbed tool classes, latency buckets, schema versions, and invariant outcomes. Avoid storing raw customer text unless the data policy and retention model allow it. A trace-aware harness without a data boundary is just another place to leak prompts.
What this costs
Failure-budget oracles cost more than plain expected answers. They require schema discipline, case curation, and agreement between product, platform, and model teams. LangSmith distinguishes offline evaluations for pre-deployment testing from online evaluations for production monitoring, and that distinction matters because failure budgets live in both places. Offline harnesses catch regressions before release. Online scoring shows whether tolerated failures are becoming normal behavior.
They also create migration pressure. When a tool schema changes, the oracle has to declare whether old evidence is still valid. When a model starts using a new tool path, the invariant must decide whether that path is equivalent or risky. When a product changes its promise, the tolerance changes with it. That is maintenance, but it is better than pretending the expected answer string was the contract.
False positives are another cost. A strict citation invariant can fail a good answer that cites a new but equivalent source. A retry budget can fail during a temporary provider outage. A schema check can block a harmless field addition. The answer is not to remove the oracle. The answer is to version it, review tolerated failures, and keep a small set of representative cases rather than a giant graveyard of stale incidents.
My practical rollout path is boring. Start with ten cases around the highest-risk interface. Add one failure budget per case. Record why each tolerance exists. Put the oracle under code review. Run it in CI before model or prompt releases. Promote production traces into the harness only after scrubbing. Once the team trusts the categories, wire the same outcomes into trace metrics.
A harness oracle is the place where an AI system admits what can go wrong. If that admission is missing, the harness can still produce scores, but the scores will hide the difference between resilience and luck.
FAQ
Where should oracle contracts live?
Keep the canonical oracle near the code or eval repository so changes go through review. The same oracle can be mirrored into an eval platform, but the reviewable contract should not disappear into a dashboard configuration.
Do model graders replace deterministic checks?
No. Use deterministic checks for schema, citations, tool calls, safety gates, and budget exhaustion. Use model graders for semantic quality only after the evidence and invariants are stable.
How many failure cases are enough?
Start with the smallest set that covers the product promise and recent incidents. I usually begin with about ten high-risk cases, then add a case only when a new failure mode would have escaped the existing oracle.
What should happen when a dependency is flaky?
The oracle should distinguish tolerated degradation from budget exhaustion. One documented retry and a clear degraded answer may pass. Repeated retries, missing required evidence, or hidden uncertainty should fail.