Harness Datasets Need Lineage

A harness dataset is not a spreadsheet of prompts. It is evidence with custody. If a failing eval row cannot tell me where it came from, which version promoted it, which oracle owns it, and when to retire it, I treat the result as a hint, not a regression.

The dataset is part of the harness API

Most eval failures are reported as if the model suddenly became worse. In practice, the harness often changed under the model. A row was edited, a judge rubric drifted, a prompt variable was renamed, a production trace was copied without context, or a fixture started constructing a different world than the one the oracle expects.

That is why I treat the dataset as part of the harness API. OpenAI's eval guide describes two core ingredients for an eval: data_source_config, which defines the schema for test data, and testing_criteria, which defines the graders that judge model output (OpenAI evals guide). Those are not clerical details. They are the contract between the case, the application, and the scoring layer.

The same idea shows up in LangSmith's evaluation workflow. LangSmith scores agent outputs against datasets and criteria so teams can benchmark versions, catch regressions, and track quality over time (LangSmith Evaluation). A benchmark cannot do that job if its examples are anonymous. A dataset row needs enough metadata for a reviewer to answer four questions quickly:

1. Why does this case exist?
2. What behavior is expected?
3. Which code path or product surface does it cover?
4. What event would make it obsolete?

Without those answers, the row is not really a test case. It is an opinion in JSONL.

The practical design rule is to make lineage required at ingest time. The row should carry a stable case ID, source type, source reference, owner, reviewer, created timestamp, last promotion reason, dataset split, oracle type, and retirement condition. If those fields are missing, the harness should reject the row before it reaches CI.

Lineage turns incidents into reusable tests

The strongest eval cases usually start as incidents. A customer asked a weird question, a retrieval chain cited stale context, a tool call used the right tool with the wrong arguments, or an agent completed the task but skipped the approval step. LangSmith states the pattern directly: add real traces to a dataset so a failure you saw once becomes a test you run every time (LangSmith Evaluation).

That sentence contains the important word: becomes. A trace is not automatically a test. It becomes a test after someone decides what the failure means, removes private data, chooses an oracle, and writes down why the case belongs in the suite.

I like this promotion path:

production trace -> incident note -> reviewed case -> dataset row -> tagged dataset version -> experiment result

Each arrow is a review boundary. The production trace preserves what happened. The incident note explains why it mattered. The reviewed case removes secrets and narrows the expected behavior. The dataset row records the oracle. The tagged version tells CI what was actually tested. The experiment result connects a model or prompt change to a known evidence set.

OpenAI's datasets guide recommends treating datasets as dynamic spaces that expand as edge cases or blind spots are identified (OpenAI datasets guide). Dynamic does not mean casual. The more often a suite changes, the more important row provenance becomes. Otherwise a team cannot tell whether a score moved because the application improved, the model changed, or the benchmark quietly became harder.

The minimal metadata I require for promoted incidents is boring by design:

{
  "case_id": "support-refusal-2026-09-14-001",
  "source_type": "production_trace",
  "source_ref": "trace:4bf92f3577b34da6a3ce929d0e0e4736",
  "owner": "ai-platform",
  "reviewer": "evals-oncall",
  "promotion_reason": "agent skipped refund approval boundary",
  "oracle_type": "deterministic_and_rubric",
  "split": "regression",
  "retire_when": "refund tool requires signed approval token"
}

The values are examples, not a vendor format. The point is that the harness can explain the case without a Slack archaeology project.

Version the example, not only the prompt

Prompt versioning helps, but it is not enough. OpenAI's dataset interface says saved prompts use versioning so updates can be referenced safely (OpenAI datasets guide). LangSmith applies the same discipline to datasets: every add, update, or delete creates a new dataset version, and past versions are read-only (LangSmith manage datasets).

That matters because evals compare systems across time. If I run model A on Monday and model B on Friday, I need to know whether both runs saw the same examples. If the dataset changed between runs, the comparison is not wrong, but it is a different claim. It should be labeled as a new benchmark version.

I use three versioning layers:

Layer	What changes	Why it matters
Case version	The row's input, metadata, or expected behavior	Preserves review history for a specific example
Dataset version	Membership of cases in a named suite	Makes model and prompt comparisons reproducible
Split tag	smoke, regression, canary, adversarial, or retired	Controls cost, runtime, and release gating

LangSmith supports tagged versions, filtered views, and split-based evaluation (LangSmith manage datasets). Promptfoo supports weighted assertions and test thresholds, so one suite can collect several signals without making every weak signal block a release (Promptfoo assertions). Those mechanisms are useful only if the row itself says which split and gate it belongs to.

For senior teams, the key decision is not whether to version. It is what gets to change without review. My default is strict: prompt experiments can be cheap, dataset membership cannot. A case entering regression should require a reviewer and a retirement condition. A case entering canary can be noisier, but its failures should not block deploys until the oracle stabilizes.

Separate provenance from the oracle

Lineage and correctness are related, but they are not the same field.

Provenance answers where the case came from. The oracle answers what should happen. A case can come from a production trace and still use a deterministic JSON assertion. A synthetic red-team case can use an LLM rubric. A manually written support ticket can use a reference label. Mixing those ideas makes the harness hard to debug.

OpenAI's eval examples separate test data from testing_criteria such as string checks against a reference label (OpenAI evals guide). Promptfoo's assertion model makes the same split explicit: assertions can check equality, containment, regex, JSON validity, latency, cost, trace spans, tool trajectories, or model-graded rubrics (Promptfoo assertions).

That gives a useful matrix:

Provenance	Better first oracle	Reason
Incident trace	Deterministic regression check	The original failure should be crisp enough to replay
Ambiguous user preference	Human or pairwise evaluator	The desired behavior may be subjective
Retrieval failure	Context faithfulness plus citation checks	The output must stay grounded in supplied context
Tool-use bug	Trajectory and argument assertions	The path matters, not only the final text
Policy boundary	Deterministic refusal or approval-token check	Release gates need low variance

The mistake is asking an LLM judge to compensate for missing lineage. If the row does not say why it exists, a rubric cannot recover that intent. I use LLM judges where semantic judgment is necessary, then I pin deterministic checks around the parts that must never drift.

Make trace IDs the join key

A harness result should lead to the exact run that produced it. W3C Trace Context defines standard HTTP headers and value formats for propagating context that identifies individual requests in distributed systems (W3C Trace Context). OpenTelemetry's tracing API defines core concepts such as SpanContext, trace IDs, span IDs, span attributes, events, links, and status (OpenTelemetry Tracing API).

That is enough to make trace identity a harness primitive. Do not store a screenshot of a trace in the eval row. Store the trace ID, span ID, environment, service version, and selected attributes that explain the case. Then the harness can join failures to telemetry, replay logs, tool calls, and deployment metadata.

For agent systems, trace-linked lineage also prevents a common false fix. A developer sees a bad final answer and edits the final-answer prompt. The trace shows that retrieval returned stale documents or a tool returned a timeout. The right fix is upstream. Promptfoo even includes trace-oriented checks such as span counts, span duration, error spans, and trajectory assertions (Promptfoo assertions). Those checks are more useful when the dataset row already names the trace boundary it represents.

The same dependency clarity exists in ordinary test harnesses. pytest fixtures are requested by test function arguments, and fixtures can request other fixtures, making test dependencies explicit (pytest fixtures). Eval rows need the same explicitness. If a row depends on a specific retriever corpus, tool schema, policy version, or trace shape, put that dependency in metadata instead of hoping the runner knows.

A minimal lineage manifest

Here is a small Python 3.11+ example I use as a starting point. It validates a JSONL case manifest, rejects duplicate IDs, requires lineage fields, checks stable splits, and emits a run manifest that CI can archive beside eval results. It uses only the standard library.

import datetime as dt
import hashlib
import json
from collections import Counter
from pathlib import Path

REQUIRED = {
    "case_id",
    "source_type",
    "source_ref",
    "owner",
    "reviewer",
    "promotion_reason",
    "oracle_type",
    "split",
    "retire_when",
}

VALID_SPLITS = {"smoke", "regression", "canary", "adversarial", "retired"}


def load_cases(path: Path) -> list[dict]:
    rows = []
    for line_number, line in enumerate(path.read_text().splitlines(), start=1):
        if not line.strip():
            continue
        case = json.loads(line)
        missing = sorted(REQUIRED - case.keys())
        if missing:
            raise ValueError(f"line {line_number}: missing {missing}")
        if case["split"] not in VALID_SPLITS:
            raise ValueError(f"line {line_number}: invalid split {case['split']!r}")
        rows.append(case)
    counts = Counter(row["case_id"] for row in rows)
    duplicates = sorted(case_id for case_id, count in counts.items() if count > 1)
    if duplicates:
        raise ValueError(f"duplicate case_id values: {duplicates}")
    return rows


def digest_cases(rows: list[dict]) -> str:
    canonical = "\n".join(json.dumps(row, sort_keys=True) for row in rows)
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


def build_run_manifest(rows: list[dict], dataset_name: str) -> dict:
    split_counts = Counter(row["split"] for row in rows)
    return {
        "dataset_name": dataset_name,
        "case_count": len(rows),
        "split_counts": dict(sorted(split_counts.items())),
        "dataset_sha256": digest_cases(rows),
        "created_at": dt.datetime.now(dt.UTC).isoformat(),
    }


if __name__ == "__main__":
    cases = load_cases(Path("cases.jsonl"))
    manifest = build_run_manifest(cases, "refund-agent-regression")
    Path("run-manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
    print(json.dumps(manifest, indent=2))

The manifest does not replace vendor dataset versions. It gives the repository a portable proof of what the harness believed it was testing. JSON Schema can validate richer versions of this contract, including required object properties and nested structures (JSON Schema tutorial). Vendor platforms can still own execution, dashboards, and experiment comparison.

In a real harness, I add three more gates:

1. A privacy check that blocks raw customer text unless it has a redaction marker.
2. A source check that rejects production-derived rows without trace IDs or incident links.
3. A promotion check that prevents canary cases from moving to regression without a reviewer.

Those gates are not bureaucracy. They keep the benchmark from becoming a landfill.

Decision rule

My decision rule is simple: promote a case only when source, oracle, owner, and retirement condition are explicit.

If the source is missing, the team cannot investigate the failure. If the oracle is vague, the team cannot tell whether the model or judge drifted. If the owner is missing, the row will outlive the product behavior it was meant to protect. If the retirement condition is missing, the benchmark accumulates stale fears.

Harness Engineering is mostly about turning model behavior into repeatable signals. Fixture contracts, calibrated judges, replay adapters, oracle budgets, and mutations all help. Dataset lineage is the layer that tells those signals what they mean. A lineage-aware dataset lets a senior engineer say, with a straight face, that a red build represents a product regression rather than a mysterious spreadsheet change.

FAQ

How is lineage different from a golden dataset?

A golden dataset says which examples are important. Lineage says why each example is important, where it came from, who reviewed it, which oracle owns it, and when it should be retired. You need both.

Should production traces be copied directly into evals?

No. Use production traces as evidence, then promote reviewed, redacted, narrowed cases. LangSmith's workflow supports using historical production traces as dataset material, but the case should still be reviewed before it becomes a release gate (LangSmith Evaluation).

How often should an evaluation dataset be versioned?

Every time membership or example content changes. LangSmith creates a new dataset version whenever examples are added, updated, or deleted (LangSmith manage datasets). Even if your harness is homegrown, use the same rule.

Where do LLM judges fit when deterministic assertions exist?

Use deterministic checks for invariants such as JSON shape, required refusal, tool arguments, citation presence, and latency budgets. Use LLM judges for semantic quality that cannot be reduced to a stable programmatic assertion. Keep the provenance fields separate from both.

References

1. OpenAI: Working with evals
2. OpenAI: Getting started with datasets
3. LangSmith: Evaluation
4. LangSmith: Manage datasets
5. Promptfoo: Assertions and Metrics
6. W3C: Trace Context
7. OpenTelemetry: Tracing API
8. pytest: How to use fixtures
9. JSON Schema: Creating your first schema