Harness Promotions Need Retirement Rules

A harness case should become release-blocking only after it has a source, oracle, owner, and retirement rule. Evaluation platforms make it easy to collect datasets and run experiments, but production trust comes from deciding which cases graduate, which stay advisory, and when each one leaves the suite.

The failure mode: every interesting row becomes permanent

AI teams are getting better at capturing failures. OpenAI describes evals as tests that check model outputs against specified style and content criteria, and LangSmith frames evaluation around datasets, criteria, benchmarking, regression detection, and quality tracking. Those are the right primitives, but they do not answer the lifecycle question by themselves: should this new case block the next release forever?

That question matters because agent and LLM harnesses tend to rot in a different way than ordinary unit test suites. A unit test often maps to a stable function contract. An eval case may come from a product incident, a synthetic adversarial prompt, a production trace, a human review, or a temporary model weakness. LangSmith documents datasets created from manual examples, historical traces, and synthetic data, which is useful precisely because cases enter from different sources. The danger is treating all sources as equally permanent.

The memorable rule is simple: if a case cannot explain why it was promoted and what would make it obsolete, it is not a regression test yet. It is a candidate signal.

Promotion is a contract, not a folder move

Many teams call a dataset split regression and assume governance is done. That is too weak. LangSmith dataset management includes versions, past read-only versions, metadata filtering, splits, and tagged or timestamped versions, so the platform can preserve useful structure. The harness still needs a promotion contract that says why a row has authority over releases.

For senior AI engineering teams, the contract needs four fields:

Field	Decision it answers	Failure if missing
source_ref	Where did this case come from?	Nobody can tell whether it reflects a real risk or a synthetic curiosity.
oracle_type	How do we know pass or fail?	A model judge, regex, human label, and trace assertion get mixed as if they have equal strength.
owner	Who maintains the case?	Broken or stale rows linger because everyone assumes someone else owns them.
retire_when	What makes this case stop blocking?	The suite grows forever and starts punishing healthy architecture changes.

This is not bureaucracy. It is the minimum information needed to know whether a failed row represents a product regression, a known flaky signal, a policy choice, or a historical artifact.

Candidate to regression flow

A good harness separates discovery from release policy. OpenAI's datasets guide positions datasets as a way to get started with evals and prompt testing. That starting point should feed a promotion path, not immediately become the production gate.

flowchart LR
  A["Candidate case"] --> B["Source review"]
  B --> C["Oracle review"]
  C --> D["Owner assigned"]
  D --> E["Retirement rule written"]
  E --> F{"Blocking policy?"}
  F -->|"yes"| G["Regression split"]
  F -->|"not yet"| H["Advisory split"]
  G --> I["Release gate"]
  H --> J["Trend dashboard"]

This flow keeps weak but useful examples visible. A synthetic jailbreak idea can stay advisory until it maps to a real product boundary. A production trace can become blocking after the team confirms the oracle. A temporary vendor behavior can have a retirement date tied to a model or tool upgrade.

The point is not to slow down eval creation. The point is to prevent the release gate from becoming a museum of old surprises.

A minimal promotion manifest

The manifest below is intentionally small. It uses ordinary JSONL because harness metadata often needs to move between evaluation platforms, CI jobs, and review tools. JSON Schema validates JSON documents and can require fields and nested structure, so teams can enforce this shape without building a custom control plane.

{
  "case_id": "approval-boundary-2026-09-21-001",
  "source_type": "production_trace",
  "source_ref": "trace:4bf92f3577b34da6a3ce929d0e0e4736",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "oracle_type": "deterministic_and_rubric",
  "owner": "ai-platform",
  "reviewer": "evals-oncall",
  "split": "regression",
  "promotion_reason": "agent attempted a side effect before approval",
  "promoted_at": "2026-09-21T13:00:00Z",
  "severity": "release_blocking",
  "release_policy": "must_pass_before_agent_runtime_release",
  "retire_when": "approval token is cryptographically bound to the tool call"
}

Notice the trace fields. W3C Trace Context standardizes propagation of trace context across service boundaries, and OpenTelemetry defines trace IDs, span IDs, span context, spans, links, events, attributes, and status. A harness does not need to expose private traces publicly, but internally it should preserve enough linkage to reproduce the behavior that justified promotion.

Here is a small gate that catches the most expensive mistakes before a case enters the blocking split.

import json
from pathlib import Path

REQUIRED = {
    "case_id",
    "source_type",
    "source_ref",
    "oracle_type",
    "owner",
    "reviewer",
    "split",
    "promotion_reason",
    "promoted_at",
    "severity",
    "release_policy",
    "retire_when",
}

VALID_BLOCKING_SPLITS = {"regression", "release"}


def validate_manifest(path: str) -> list[str]:
    errors: list[str] = []
    for line_number, line in enumerate(Path(path).read_text().splitlines(), start=1):
        if not line.strip():
            continue
        row = json.loads(line)
        missing = sorted(REQUIRED - set(row))
        if missing:
            errors.append(f"line {line_number}: missing {missing}")
        if row.get("split") in VALID_BLOCKING_SPLITS and not row.get("retire_when"):
            errors.append(f"line {line_number}: blocking case has no retire_when")
        if row.get("oracle_type") == "llm_judge" and "calibration_set" not in row:
            errors.append(f"line {line_number}: judge case lacks calibration_set")
    return errors

The exact field names can change. The invariant should not. Blocking cases need provenance, oracle strength, ownership, and an exit condition.

The promotion matrix

A release gate should treat signals differently. OpenAI evals separate data source configuration from testing criteria, which is a useful mental model: the row and the grader are separate design choices. A production harness should make that separation visible in policy.

Case type	Good first split	Promotion requirement	Retirement trigger
Production trace	Advisory	Reproduced, owned, and tied to a product invariant	Invariant is enforced by runtime policy
Human review case	Advisory or regression	Labeler agreement and stable rubric	Rubric changes or feature scope changes
Deterministic schema failure	Regression	Clear contract and low flake risk	Contract is removed or replaced
LLM judge failure	Advisory first	Calibrated against known examples	Judge disagreement rises or calibration set is superseded
Cost or latency case	Release budget	Budget owner and measurement window	Budget changes through capacity planning

The mistake is promoting by emotional weight alone. A painful incident deserves attention, but a release-blocking gate needs a reliable oracle. If the oracle is not reliable yet, keep the case visible as advisory and use it to improve the grader.

Retirement is a safety feature

Retirement sounds risky because teams hear it as deletion. In a harness, retirement should mean removing release-blocking authority while preserving history. LangSmith's dataset versioning and read-only past versions are useful because retired cases can remain inspectable even when they stop blocking the next release.

Retirement rules also force better architecture. Suppose an agent bypassed an approval boundary. The case should block releases while the boundary is implemented in prompts or application code. Once the runtime requires a signed approval token at the tool-call layer, the old prompt-level failure may no longer be the best release signal. The retirement rule can say that this row moves to historical coverage after the runtime policy ships and a lower-level deterministic test covers the boundary.

That is safer than leaving both tests as permanent blockers. Duplicate gates create noise. Noise trains engineers to distrust the harness.

Retirement also gives product teams a cleaner way to absorb platform upgrades. If a new model, tool sandbox, or policy engine changes the failure surface, the harness should not pretend the old row still measures the same risk. The row can move to historical coverage, the manifest can point to the replacement control, and the active gate can focus on the current invariant. This keeps the suite small enough for engineers to investigate every failure instead of negotiating around a backlog of stale blockers.

A practical rollout pattern

Start with three queues. The candidate queue accepts everything useful: incidents, traces, red-team examples, synthetic edge cases, and human review misses. The advisory queue runs in CI or nightly jobs but does not block releases. The regression queue blocks releases only when promotion fields are complete.

Then add a weekly review that asks five questions:

1. Which advisory cases failed more than once?
2. Which failures map to a product invariant rather than a model preference?
3. Which cases have oracles strong enough to block a release?
4. Which blocking cases lack active owners?
5. Which blocking cases met their retire_when condition?

This review is where harness engineering becomes engineering rather than dashboard watching. LangSmith's evaluation workflow covers running experiments and comparing results, but the team still owns the release policy that converts experiment evidence into action.

The decision rule

Promote a case only when the team can say this sentence without hedging:

This case blocks releases because it came from this source, checks this invariant with this oracle, is owned by this team, and retires when this stronger control exists.

If the sentence cannot be completed, the case may still be valuable. It just belongs in discovery, advisory evaluation, or grader development. Harness quality is not measured by how many cases block the release. It is measured by how confidently the remaining blockers represent real product risk.

FAQ

When should an eval case become release-blocking?

A case should become release-blocking when it protects a stable product or safety invariant, has a reliable oracle, has an owner, and has a written retirement rule. Evaluation systems can run cases against datasets and criteria, but the release-blocking decision is a policy decision owned by the engineering team.

Should production traces automatically become evals?

No. Production traces are strong candidates because they reflect real behavior, and Trace Context plus OpenTelemetry concepts make trace linkage practical. They still need review, redaction, an oracle, and a promotion reason before they block releases.

How do we retire old cases without losing safety?

Keep the historical record, remove release-blocking authority, and replace the old case with a stronger lower-level control when possible. Dataset versioning and tagged versions help preserve history while allowing the active gate to change over time.

How is this different from dataset lineage?

Lineage explains where a case came from. Promotion and retirement rules explain why it has authority today. A strong harness needs both: provenance for trust, and lifecycle policy for control.

References

1. OpenAI, Working with evals
2. OpenAI, Getting started with datasets
3. LangSmith, Evaluation
4. LangSmith, Manage datasets
5. W3C, Trace Context
6. OpenTelemetry, Tracing API
7. JSON Schema, Creating your first schema