On this page
- The decision rule: promote a fixture only with a contract
- Split fixtures by interface, not by model
- Make the harness executable in CI
- Close the loop from production traces
- Trade-offs senior teams should price explicitly
- FAQ
- How many fixtures should a team start with?
- Are online evals enough if we already monitor production?
- Can one metric cover RAG and agents?
- Should fixture contracts live in an eval SaaS product?
- References
Harness fixtures become useful when I treat them as APIs: versioned inputs, explicit oracles, scorer budgets, and provenance. Without that contract, an eval suite can still run green while the product changes underneath it, especially across model upgrades, retrieval changes, and tool-calling rewrites.
The decision rule: promote a fixture only with a contract
My rule for harness engineering is simple: a production example is not a fixture until another engineer can tell what interface it is testing, what evidence makes the answer correct, and what regression should block a release. That rule matters because LangSmith frames evals as the process of breaking down what good looks like and measuring it, not as a folder of prompts that happened to fail once.
The contract starts before a test runner. LangSmith recommends identifying critical components such as LLM calls, retrieval steps, tool invocations, and output formatting before building evaluations. I translate that into a fixture interface field. If the example checks citation grounding, it belongs to the retrieval interface. If it checks whether an agent selected the right payment-refund tool with the right arguments, it belongs to the tool interface. If it checks JSON schema stability, it belongs to the output-format interface.
This is the difference between a harness and a scrapbook. A scrapbook says, “the model once answered this wrong.” A harness says, “for task refund_status_v2, with this input and this retrieved evidence, any candidate release must satisfy this oracle under this scorer and budget.” OpenAI’s eval guide uses the same shape at the API level: data source configuration describes the test data, while testing criteria define graders that decide whether output is correct. The names vary by platform, but the engineering boundary is stable.
I also keep the starting set small. LangSmith recommends 5 to 10 manually curated examples of good behavior for each critical component. That is enough to force precise contracts without pretending that the first fixture set is statistically complete. The first pass should make ambiguity visible. Scale comes later, after the contract survives review.
Split fixtures by interface, not by model
The most expensive harness mistake I see is organizing fixtures by model name. gpt-5.6_cases.yaml tells me almost nothing after the next routing change. retrieval_grounding/citation_span_v1.yaml tells me exactly which product boundary is under test. LangSmith defines offline examples as inputs, optional reference outputs, and optional metadata, and I use metadata aggressively because it is the harness control plane.
A fixture contract should include these fields:
{
"id": "citation_span__contract_refund_001",
"interface": "retrieval_grounding",
"task": "answer_with_cited_policy",
"input": "Can I get a refund after 35 days if the shipment was late?",
"evidence_ids": ["policy_refunds_2026_04", "shipping_exceptions_2026_05"],
"oracle": {
"must_include": ["late shipment exception", "35 days"],
"must_not_include": ["unconditional refund"]
},
"scorers": [
{"name": "citation_present", "type": "deterministic", "threshold": 1.0},
{"name": "answer_grounded", "type": "llm_judge", "threshold": 0.8}
],
"budgets": {"max_latency_ms": 3500, "max_cost_usd": 0.02},
"provenance": {"source": "production_trace", "reviewed_by": "support_policy_owner"}
}
That shape maps cleanly onto current eval systems. Braintrust describes every evaluation as data, a task, and scorers or classifiers. promptfoo supports declarative test cases across prompts, providers, and assertions, and its assertion model includes types, thresholds, weights, providers, metrics, and transforms. Ragas separates RAG metrics such as context precision, context recall, response relevancy, and faithfulness from agent or tool-use metrics such as tool call accuracy and tool call F1. Those systems are not identical, but they all reward the same discipline: name the interface, name the oracle, and name the scorer meaning.
The portable contract is the important part. OpenAI’s current eval page says the Evals platform is being deprecated, with existing eval content becoming read-only on October 31, 2026 and shutdown scheduled for November 30, 2026. I read that as a reminder not to let a hosted eval object become the only place where fixture semantics live. SaaS dashboards are useful execution surfaces. The fixture contract should still be understandable in source control.
Make the harness executable in CI
A fixture contract that humans like but CI ignores is documentation, not a harness. Braintrust describes promoting a promising configuration to an immutable experiment and automating evals in CI/CD to catch regressions. Before I spend tokens on model calls, I want CI to reject malformed fixtures, missing budgets, and unowned scorers.
Here is a minimal stdlib gate I would put before any expensive eval job:
import json
import sys
from pathlib import Path
REQUIRED_TOP_LEVEL = {"id", "interface", "task", "input", "oracle", "scorers", "budgets", "provenance"}
REQUIRED_BUDGETS = {"max_latency_ms", "max_cost_usd"}
def load_fixture(path: Path) -> dict:
with path.open(encoding="utf-8") as handle:
return json.load(handle)
def validate_fixture(path: Path) -> list[str]:
fixture = load_fixture(path)
errors: list[str] = []
missing = sorted(REQUIRED_TOP_LEVEL - set(fixture))
if missing:
errors.append(f"{path}: missing fields {missing}")
budgets = fixture.get("budgets", {})
missing_budgets = sorted(REQUIRED_BUDGETS - set(budgets))
if missing_budgets:
errors.append(f"{path}: missing budgets {missing_budgets}")
for index, scorer in enumerate(fixture.get("scorers", [])):
for field in ("name", "type", "threshold"):
if field not in scorer:
errors.append(f"{path}: scorer {index} missing {field}")
if not fixture.get("provenance", {}).get("reviewed_by"):
errors.append(f"{path}: fixture has no accountable reviewer")
return errors
def main() -> int:
paths = [Path(arg) for arg in sys.argv[1:]]
errors: list[str] = []
for path in paths:
errors.extend(validate_fixture(path))
if errors:
print("fixture contract failed")
for error in errors:
print(error)
return 1
print(f"validated {len(paths)} fixture contracts")
return 0
if __name__ == "__main__":
raise SystemExit(main())
This is intentionally boring. pytest supports parametrizing test functions with @pytest.mark.parametrize and pytest_generate_tests, so the same fixture files can feed ordinary unit tests, model eval jobs, and smoke tests. If the team needs generated edge cases around contract fields, Hypothesis describes property-based tests as tests that should pass for all inputs in a described range while the library generates examples, including edge cases the developer may not have considered. I still keep the fixture contract explicit. Property generation expands a boundary; it does not replace the boundary.
The CI gate should fail fast on contract errors, then run a tiered eval plan. Tier one is deterministic: schema checks, citation ID existence, blocked phrase checks, JSON parsing, and cost ceilings. Tier two is task-specific scoring: exact match, rubric scoring, retrieval faithfulness, or tool-call accuracy. Tier three is expensive comparison: pairwise model judgments, larger judge models, or human review queues. promptfoo supports assertion thresholds and weights, which is the right primitive for turning those tiers into release gates.
Close the loop from production traces
The contract also defines how production evidence enters the harness. LangSmith separates offline evaluations for pre-deployment testing from online evaluations for production monitoring, and it says online evaluations target runs and threads from tracing. Braintrust similarly says online scoring evaluates production traces asynchronously with no latency impact. I do not promote every bad trace. I promote traces that expose a contract gap.
My promotion checklist is short. First, classify the trace by interface: retrieval, tool selection, generation, safety, routing, or formatting. Second, write or update the oracle. Third, attach reviewer provenance. Fourth, set a budget. Fifth, add a retirement condition. The retirement condition matters because a fixture can outlive the product decision it was meant to protect.
This loop prevents two opposite failures. Without online traces, offline evals become museum pieces. Without offline contracts, online scoring becomes a stream of vague judge scores. Braintrust describes the feedback loop as pulling interesting production traces into datasets to improve offline coverage. The contract is what keeps that feedback loop from becoming a junk drawer.
Trade-offs senior teams should price explicitly
Harness contracts add friction, so I price that friction directly. The first cost is coverage bias. LangSmith’s 5 to 10 manually curated examples guidance is a starting point, not a coverage claim. I label fixture sets as smoke, regression, release, or benchmark so nobody mistakes a smoke gate for a product-quality guarantee.
The second cost is scorer brittleness. Deterministic checks are cheap and stable, but they miss semantic failures. LLM judges catch more shape and reasoning issues, but Braintrust notes that online evaluation without ground truth relies on LLM-as-a-judge scorers, which means judge drift and cost become part of the system. I keep judge prompts versioned and require a small golden set for the judge itself.
The third cost is metric mismatch. Ragas lists different metrics for RAG, agent or tool use, natural language comparison, SQL, and summarization. That taxonomy is a useful warning. A faithfulness score is not a tool-call score. A tool-call F1 score is not a user-satisfaction score. The fixture contract should make the metric choice auditable.
The fourth cost is vendor coupling. OpenAI’s legacy eval page documents useful concepts like data source configuration and testing criteria, but the same page also documents the platform deprecation timeline. I want execution adapters for LangSmith, Braintrust, promptfoo, Ragas, or local pytest, but I do not want the fixture meaning trapped in any one adapter.
The payoff is boring releases. A model upgrade either satisfies the contract or fails with a named interface, named scorer, and named reviewer. That is much easier to debug than a dashboard that says quality dropped by 7 percent with no fixture provenance.
I also make fixture retirement explicit. A stale fixture is not harmless if it encodes an old product policy, an obsolete retrieval corpus, or a tool behavior that the product intentionally removed. The contract should include an owner and a review cadence because LangSmith describes evaluation as part of an iterative lifecycle across development, deployment, monitoring, and improvement. When a fixture blocks a change, the team should decide whether the product regressed, the scorer is wrong, or the fixture should retire. That decision belongs in the harness history, not in an untracked side channel.
Finally, I separate release gates from dashboards. A dashboard can trend every scorer, but a gate should be small enough that engineers understand the failure in one review. The gate should say which interface failed, which contract version failed, which budget was exceeded, and which owner can adjudicate. Braintrust describes experiments as comparable records of eval runs, and that comparability is only useful when each fixture carries stable semantics. I would rather block on 40 well-owned fixtures than wave through 4,000 unlabeled examples.
FAQ
How many fixtures should a team start with?
Start with 5 to 10 carefully reviewed examples per critical component because LangSmith recommends that range for manually curated examples of good behavior. Add volume after the contract fields are stable.
Are online evals enough if we already monitor production?
No. LangSmith says offline evaluations target datasets and examples, while online evaluations target runs and threads from tracing. They answer different questions. Online scoring finds new failures; offline fixtures decide whether a proposed change may ship.
Can one metric cover RAG and agents?
Usually no. Ragas lists RAG metrics such as context precision and faithfulness separately from agent or tool-use metrics such as tool call accuracy and tool call F1. Treat metric choice as part of the fixture contract.
Should fixture contracts live in an eval SaaS product?
They can execute there, but I keep the contract portable. Braintrust describes experiments and production scoring, promptfoo supports local declarative evals, and pytest supports parametrized tests. Source-controlled fixture semantics make adapter changes survivable.