RAG Breaks When Retrieval Has No Contract

Most broken RAG systems do not fail because the embedding model is weak. They fail because retrieval has no enforceable contract: no clear scope, freshness rule, ranking policy, budget, or evidence shape. Treat retrieval like a runtime boundary, and the model finally gets something testable to reason over.

Better embeddings are not a production strategy

The easiest RAG story is still the least useful one: split documents, embed chunks, search the vector store, paste the top results into the prompt, and ask the model to answer. That can work for a demo. It does not survive contact with permissioned data, stale documents, duplicate policy pages, user-specific filters, multi-hop questions, or incident review.

OpenAI's File Search guide describes retrieval as a tool that can search a knowledge base of uploaded files using semantic and keyword search, backed by vector stores (OpenAI File Search). That framing matters. Retrieval is not a prompt trick. It is a production tool call that decides what evidence enters the model context.

Once retrieval becomes a tool call, the question changes. The team should not ask only, "Which embedding model scored higher?" It should ask:

• Which corpus was searched?
• Which tenant, role, and permission filters were applied?
• Which query rewrite was used?
• Which ranker decided final order?
• Which freshness window was allowed?
• How many tokens were admitted into the model context?
• What should happen when evidence is missing or contradictory?

Those answers are usually scattered across app code, vector-store configuration, ingestion jobs, prompt templates, and evaluation scripts. A retrieval contract pulls them into one explicit interface.

The retrieval contract is the boundary

A retrieval contract is the agreement between the caller and the retrieval layer. It says what evidence the system may return, how it should be selected, how it should be shaped, and what metadata must travel with it.

This is not bureaucracy. It is the difference between a RAG system that can be debugged and one that can only be tuned by folklore.

flowchart LR
    U["User request"] --> Q["Query plan"]
    Q --> C["Retrieval contract"]
    C --> S["Search and rank"]
    S --> E["Evidence packet"]
    E --> M["Model answer"]
    E --> V["Eval and trace"]
    C --> V

The contract should be stable enough that evaluation, tracing, and application code can depend on it. The implementation behind it can still evolve. You can move from vector search to hybrid search, add reranking, split indexes by domain, or change chunking rules. The contract tells the rest of the system what changed and what must stay true.

LlamaIndex's retriever documentation is a good reminder that "retrieval" is not a single algorithm. It includes vector-store retrievers, BM25 retrievers, hybrid retrieval, fusion, recursive retrieval, router retrieval, and other variants (LlamaIndex retrievers). If the application treats all of those as "top_k chunks," it hides the very policy choices that determine answer quality.

What the contract should pin

A useful retrieval contract does not need to be large. It needs to pin the fields that change model behavior.

Contract field	Why it matters in production
corpus_scope	Prevents the system from mixing public docs, internal docs, customer data, and stale migrations by accident.
principal	Captures tenant, user, role, and permission filters before search.
query_plan	Records rewrite, decomposition, and whether the query was keyword, vector, hybrid, or routed.
freshness	Makes "current policy" and "historical policy" explicit instead of relying on whatever the index contains.
ranker	Versions embedding model, lexical search, reranker, fusion strategy, and thresholds.
budget	Controls result count, token count, and reserve tokens for the final answer.
evidence_shape	Defines citation IDs, snippets, scores, document dates, section titles, and redaction metadata.
abstention_rule	Tells the model and evaluator when retrieved evidence is insufficient.

Azure's RAG overview discusses chunking, vectorization, hybrid keyword and vector search, semantic ranking, query planning, and indexing freshness as part of RAG architecture (Azure AI Search RAG overview). Those are not optional tuning knobs. They are operational commitments.

If a support assistant answers from a policy document, freshness is part of correctness. If an internal engineering assistant reads incident notes, permission filters are part of correctness. If a finance assistant cites a spreadsheet, source shape and timestamp are part of correctness. If an agent retrieves tool instructions, token budget and ordering can decide whether it calls the right tool at all.

A minimal contract example

The contract can be JSON, protobuf, typed Python, TypeScript, or a database row. The format matters less than the behavior it makes explicit.

{
  "contract_version": "retrieval.v1",
  "request_id": "rag_01J8SAMPLE",
  "corpus_scope": ["product_docs", "support_policies"],
  "principal": {
    "tenant_id": "tenant_42",
    "user_role": "support_engineer"
  },
  "query_plan": {
    "original_query": "Can I refund a prepaid annual plan after renewal?",
    "rewrite": "refund policy prepaid annual renewal",
    "mode": "hybrid",
    "decompose": false
  },
  "freshness": {
    "not_before": "2026-01-01",
    "prefer_latest_per_policy_family": true
  },
  "ranker": {
    "embedding_model": "text-embedding-3-large",
    "lexical": "bm25",
    "reranker": "cross_encoder.v3",
    "min_score": 0.72
  },
  "budget": {
    "max_results": 8,
    "max_context_tokens": 6000,
    "reserve_answer_tokens": 1200
  },
  "evidence_shape": {
    "required_fields": ["doc_id", "title", "section", "updated_at", "snippet", "score", "url"],
    "citation_style": "doc_section"
  },
  "abstention_rule": "If fewer than two current policy sections agree, ask for escalation."
}

This example is intentionally boring. That is the point. Boring contracts are easier to log, diff, test, and review after an incident.

The production mistake is to bury these values in defaults. A default top_k=5 becomes product behavior. A default chunk size becomes evidence policy. A default reranker model becomes part of compliance. A default index alias becomes a source of stale answers. Defaults are fine during development, but production systems need them pinned, versioned, and visible in traces.

Evaluation should check retrieval decisions

Final-answer grading is necessary, but it is not enough. A model can produce a correct answer from the wrong evidence, or a plausible answer from incomplete evidence. It can also fail because retrieval selected the wrong document while the generation step behaved exactly as instructed.

OpenAI's evals documentation positions evals as part of the model optimization workflow (OpenAI Evals). For RAG, that workflow should include retrieval assertions before answer grading:

1. Did the query plan choose the expected corpus?
2. Were tenant and role filters applied?
3. Did the evidence include the required source family?
4. Were stale documents excluded when the question required current policy?
5. Did the ranker prefer authoritative pages over duplicates and summaries?
6. Did the model abstain when the evidence packet failed the contract?

LangChain places retrieval alongside agent runtime, context engineering, observability, and deployment concepts (LangChain retrieval concepts). That is the right mental model. Retrieval is not just a library helper. It is a runtime decision that should appear in traces and evals.

A good RAG eval set has at least four slices:

• Known-answer cases: The expected answer and expected source are both known.
• Known-source cases: The answer may vary, but the system must retrieve a particular document family.
• Permission cases: The system must not retrieve documents outside the caller's access scope.
• Abstention cases: The system should say it does not have enough evidence.

Without those slices, every evaluation collapse looks like a prompt problem. Engineers then add more instructions, more examples, and more post-processing, while the real defect sits in query planning, indexing, filtering, or ranking.

Observability: log the contract, not just the chunks

A RAG trace should include the retrieval contract, the selected evidence packet, and the final answer. Logging only the returned text is too late. You need to know why those chunks were eligible in the first place.

At minimum, log:

• contract version
• index or corpus version
• query rewrite
• filters
• ranker version
• result scores
• dropped candidates and reasons
• context-token budget
• evidence IDs used in the final answer
• abstention decision

This does not require exposing private document content to every dashboard. You can log IDs, metadata, scores, and policy decisions separately from snippets. The important part is that incident review can reconstruct the path from user request to evidence packet.

When a user reports a bad answer, the team should be able to answer three questions quickly: Was the right corpus searched? Was the right evidence retrieved? Did the model use that evidence correctly? If the trace cannot separate those questions, the system is not observable enough for production RAG.

Long context does not remove the boundary

Longer context windows reduce pressure on retrieval, but they do not remove retrieval's job. They often make the contract more important because the system can now admit more irrelevant or stale material before anyone notices.

A long context system still needs scope, freshness, permissions, ranking, and budget rules. It also needs evidence shape. A model that receives twenty pages of loosely related context may answer confidently while burying the decisive citation in the middle. More context can hide retrieval bugs by making the final answer look reasonable.

The right goal is not "retrieve as much as possible." The right goal is "retrieve the smallest evidence packet that satisfies the contract." That packet should be explainable to the model, the evaluator, and the engineer on call.

How to start this week

Do not rebuild the stack first. Add the contract at the seam you already have.

1. Pick one high-value RAG route, preferably one with user-visible risk.
2. Write down the expected corpus, freshness rule, permission filters, and evidence fields.
3. Add contract version and index version to every retrieval trace.
4. Create ten eval cases that assert documents before answers.
5. Add two abstention cases where retrieval should fail closed.
6. Pin chunking, embedding, ranker, and query-rewrite settings in configuration.
7. Review one bad answer by walking the trace from contract to evidence to generation.

That sequence is small enough to ship, but it changes the engineering posture. Retrieval stops being a pile of defaults. It becomes an interface the team can reason about.

The real payoff

The payoff is not only better answers. It is better ownership.

Application engineers can see what evidence the model was allowed to use. Search engineers can improve ranking without changing product semantics by accident. Security reviewers can inspect permission filters. Product teams can define freshness rules. Evaluation can fail the retrieval layer before the model produces a polished wrong answer.

That is what production RAG needs now: fewer mystery knobs and more explicit contracts.

FAQ

Is this just a schema for search results?

No. Search-result shape is only one part of the contract. The important fields are the eligibility, ranking, freshness, budget, and abstention rules that decide which results may appear at all.

Should every RAG app build a custom retrieval contract?

Every production RAG app should have an explicit contract, but it does not need to be exotic. Start with corpus, principal, query plan, ranker, budget, evidence shape, and abstention behavior. Use your existing search stack behind that interface.

Do long context windows make this unnecessary?

No. Long context changes the budget, not the need for scope, freshness, permissions, ranking, and evidence metadata. More context can make weak retrieval harder to notice.

What is the fastest useful first step?

Add contract version, index version, filters, query rewrite, result IDs, scores, and dropped-candidate reasons to traces for one RAG route. Then write evals that check retrieved documents before checking the final answer.

References

1. OpenAI File Search guide
2. OpenAI Evals guide
3. LlamaIndex retriever documentation
4. LangChain retriever concepts
5. Azure AI Search RAG overview