LLM Fallbacks Need Error Budgets

LLM fallback is not a backup-provider list. It is a runtime reliability policy that decides which failures are retryable, how much latency and money the request may spend, when to degrade, and what telemetry proves the user-facing operation stayed inside its contract.

Original Sieon Labs visual generated for this article. No external screenshot or third-party source image is used.

Fallback is a product contract, not a provider list

Most LLM fallback implementations start as a short config file: primary model, backup model, maybe a second provider. That looks reasonable until the first provider incident turns the backup path into a new incident surface. A fallback can change latency, token cost, model behavior, context length, safety behavior, regional routing, and downstream trace shape. Those are product and operations contracts, not just SDK options.

The tooling already hints at this. LiteLLM Router describes load balancing across deployments and providers, plus cooldowns, fallbacks, timeouts, retries, and production state through Redis (LiteLLM Router). That is useful plumbing, but plumbing is not the policy. The policy has to answer questions such as: is this user request allowed to spend another two seconds, is a cheaper model acceptable for this endpoint, and should a provider outage trigger degradation instead of another model call?

Cross-region routing makes the contract even more explicit. Amazon Bedrock cross-Region inference uses inference profiles that define the foundation model and Regions where a request can be routed (AWS Bedrock cross-Region inference). Bedrock also documents application inference profiles for tracking cost and model usage (AWS Bedrock inference profiles). That is not the same as arbitrary provider failover. It is a named invocation resource with routing, cost, and operational identity.

The engineering lesson is simple: fallback should be reviewed like an API contract. If the support chatbot can fallback from a high-reasoning model to a faster model, the product owner should know which answers are allowed to become shorter. If a code-review agent can fallback to another provider, security should know whether prompts leave a region or policy boundary. If a RAG endpoint can skip generation and return cited snippets, the UI should explain that the answer is intentionally degraded.

Classify failure before routing

The worst fallback policy is catch Exception, try next model. It hides auth failures, billing failures, prompt errors, rate limits, provider saturation, context overflow, and application bugs behind the same behavior. That creates noisy incidents and sometimes converts a cheap, obvious failure into an expensive multi-provider cascade.

Provider docs give enough signal to build a useful first taxonomy. OpenAI documents 429 rate-limit errors and says clients should pace requests and follow Retry-After when present, while spend or quota errors require changing credits or limits rather than retrying (OpenAI API error codes). Anthropic lists 429 as rate_limit_error, 500 as api_error, and 504 as timeout_error, with exponential backoff guidance for unexpected internal errors (Anthropic API errors). Azure's Retry pattern warns that retry behavior should consider exception type and idempotency because some operations are unsafe to repeat (Azure Retry pattern).

That points to a practical failure classifier:

classes:
  retryable_capacity:
    status: [429, 503, 504]
    action: retry_then_fallback
  transient_provider_error:
    status: [500]
    action: retry_with_circuit
  caller_fix_required:
    status: [400, 401, 403]
    action: fail_fast
  commercial_limit:
    codes: [insufficient_quota, organization_spend_limit_exceeded]
    action: stop_and_alert

The names matter more than the exact YAML shape. retryable_capacity means the runtime may spend a small retry budget. commercial_limit means a second provider call could hide the real billing or quota problem. caller_fix_required means the request should go back to the application team, not to another model. A fallback path that cannot explain why it fired is not production reliability. It is a roulette wheel with logs.

Put a budget in front of retries

Retries are not free. They add latency, consume tokens, create duplicate tool pressure, and can amplify a provider outage. Azure OpenAI quota guidance recommends retry logic, avoiding sharp workload changes, gradual load increases, load-pattern testing, and quota changes when necessary (Azure OpenAI quotas and limits). Google Cloud's generative AI documentation also includes quota, 429 troubleshooting, and retry strategy guidance (Google Cloud generative AI quotas). Amazon Bedrock publishes service quotas for capacity and performance (Amazon Bedrock quotas). Capacity is part of the design, not an exception handler afterthought.

A budgeted fallback policy has at least four limits:

Budget	What it protects	Typical owner
Latency budget	User experience and upstream timeouts	Product and platform
Retry budget	Provider health and queue stability	Platform
Cost budget	Token spend during incidents	Finance and platform
Quality budget	Acceptable behavior changes	Product and domain owner

Here is a minimal Python policy seam. It is intentionally boring. Reliability code should be easy to reason about during an incident.

from dataclasses import dataclass

@dataclass(frozen=True)
class Failure:
    status: int | None
    code: str | None = None

@dataclass(frozen=True)
class Budget:
    retries_left: int
    fallback_left: int
    milliseconds_left: int

STOP_CODES = {"insufficient_quota", "organization_spend_limit_exceeded"}
RETRYABLE_STATUS = {429, 500, 503, 504}
CALLER_FIX_STATUS = {400, 401, 403}

def decide_next_step(failure: Failure, budget: Budget) -> str:
    if failure.code in STOP_CODES:
        return "stop_and_alert"
    if failure.status in CALLER_FIX_STATUS:
        return "fail_fast"
    if failure.status in RETRYABLE_STATUS and budget.retries_left > 0:
        return "retry_primary"
    if failure.status in RETRYABLE_STATUS and budget.fallback_left > 0 and budget.milliseconds_left >= 800:
        return "try_fallback_model"
    return "degrade_response"

The important part is not the code. The important part is that the fallback decision is explicit, testable, and attached to a budget object. If a request is already out of time, it should not start another model call just because a backup exists. If retries are exhausted, the next step may be a degraded answer, a queued job, or a clear error. That choice should come from the endpoint contract.

Use circuits to stop cascading provider incidents

A provider incident is exactly when naive fallback looks most attractive and most dangerous. Thousands of requests fail at once, all retry, then all fallback, then the fallback provider starts throttling, then traces show every request as a long chain of partial failures. The system is busier, users are waiting longer, and operators still do not know whether the product is healthy.

Circuit breakers exist for this reason. Azure's Circuit Breaker pattern temporarily blocks access to a remote service after failures reach a threshold instead of repeatedly retrying operations likely to fail (Azure Circuit Breaker pattern). The same pattern notes that services can return 429 when throttling clients and 503 when unavailable (Azure Circuit Breaker pattern). Azure's Throttling pattern also describes outbound rate limits that reduce in-flight calls when an external dependency fails or returns errors, then restore normal flow after recovery (Azure Throttling pattern).

For LLM gateways, I like three circuit scopes:

1. Deployment circuit: trip a specific model deployment when it returns sustained 429, 503, or timeout errors.
2. Provider circuit: trip the provider when multiple deployments fail in the same window.
3. Feature circuit: degrade a product feature when fallback would violate latency, cost, or quality budgets.

That third circuit is where many teams are weak. They treat degradation as a failure page instead of a designed path. For an internal code search assistant, degradation might mean returning ranked snippets with citations and skipping synthesis. For a customer support flow, it might mean asking one clarifying question instead of producing a long answer. For an agent that writes to external systems, degradation might mean moving to review-required draft mode.

Observe logical operations, not just HTTP calls

If fallback is a runtime policy, the trace should show the policy. OpenTelemetry's GenAI semantic conventions cover LLM, agent, embeddings, retrieval, and MCP operations (OpenTelemetry GenAI conventions). Its GenAI client spans represent logical operations observed by the caller, from initiation until response, error, or cancellation (OpenTelemetry GenAI client spans). The same page says that if a transient issue is retried automatically, the corresponding span should cover the logical operation with all retries (OpenTelemetry GenAI client spans).

That is the right mental model. The user did not ask for three HTTP calls. The user asked for one answer. Your top-level span should answer: which endpoint handled the request, which provider and model were attempted, which failure class was observed, which circuit was open, how much budget was consumed, and whether the final response was full quality or degraded.

A small span vocabulary is enough to start:

fallback.telemetry:
  request_class: support_answer
  initial_provider: openai
  initial_model: primary-reasoning
  failure_class: retryable_capacity
  retry_count: 1
  fallback_provider: anthropic
  fallback_model: fast-sonnet
  degradation_mode: none
  latency_budget_ms: 4000
  latency_spent_ms: 3180
  cost_budget_usd: 0.08
  circuit_state: half_open

Do not bury these fields in unstructured logs. Put them on the operation span or attach them as structured events. The exact names can follow your internal conventions, but the shape should let an on-call engineer group by failure class, endpoint, tenant, provider, model, and degradation mode without reading prompt text.

Design degradation as a first-class path

A fallback model is only one degradation option. Azure's Throttling pattern lists graceful feature degradation, load leveling, priority-based deferral, and outbound rate limits as overload strategies (Azure Throttling pattern). Azure's Retry pattern frames transient faults as expected cloud behavior that systems should handle while minimizing impact on business tasks (Azure Retry pattern). Production AI systems need the same thinking.

The cleanest fallback designs separate response modes:

Mode	When to use it	User-facing behavior
Full answer	Primary path healthy	Normal synthesis and tool use
Equivalent fallback	Backup meets same product contract	Same UI, trace records provider swap
Reduced answer	Backup is cheaper, faster, or less capable	Shorter answer, fewer tool calls, clear caveat
Retrieval-only	Generation path unhealthy	Show cited snippets, skip synthesis
Deferred job	Latency budget gone but task still valuable	Queue work and notify later
Fail fast	Auth, billing, policy, or invalid request	Clear error and operator signal

This table is more important than a provider ranking list. It tells the runtime what kind of promise it is still allowed to make. A fallback that silently changes answer quality is product debt. A degraded mode that is labeled, measured, and reviewed is reliability engineering.

Cross-region routing deserves special care. AWS says cross-Region inference requests are logged in CloudTrail in the source Region with additionalEventData.inferenceRegion identifying where requests were processed (AWS Bedrock cross-Region inference). That is a useful operational detail, but it also reminds us that routing has audit and data-boundary meaning. If your policy can move requests across geography, record it. If your product cannot tolerate that movement, encode it as a hard constraint.

What I would ship first

I would not start by buying access to five providers. I would ship one primary model, one constrained fallback mode, one retrieval-only degradation path, and a trace schema that makes fallback visible. Then I would run load tests that force 429, 500, timeout, quota, and auth failures. The goal is not perfect availability. The goal is knowing exactly which promise the system can still keep when a model call fails.

A good first rollout checklist looks like this:

• Define endpoint-level latency, retry, cost, and quality budgets.
• Classify failures before retry or fallback.
• Add deployment, provider, and feature-level circuits.
• Make degraded modes explicit in product copy and telemetry.
• Trace the logical operation, including all retries and fallbacks.
• Review cross-region and cross-provider boundaries with security.
• Test provider saturation before the first real incident.

The uncomfortable truth is that fallback makes the system more complex before it makes it more reliable. That tradeoff is worth it only when the fallback path is governed. If the policy is explicit, budgeted, and observable, fallback can preserve user value during provider trouble. If it is just a backup list, it usually preserves hope and burns the incident budget.

FAQ

Should every 429 trigger fallback?

No. OpenAI says clients should pace requests and follow Retry-After when present for rate limits (OpenAI API error codes), and Azure recommends retry logic plus gradual workload changes for quota-related issues (Azure OpenAI quotas and limits). A small retry budget often makes sense before fallback. If the request is already out of latency budget, degradation may be better.

Is cross-region routing the same as provider fallback?

No. AWS Bedrock cross-Region inference routes through inference profiles that define the foundation model and Regions that may process requests (AWS Bedrock cross-Region inference). Provider fallback usually changes the vendor, model behavior, SDK surface, and policy boundary. Treat both as routing policies, but review them differently.

What should be traced, each HTTP call or the whole user request?

Both can exist, but the most important span is the logical operation. OpenTelemetry GenAI client spans should cover the operation from initiation until response, error, or cancellation, including automatic retries for transient issues (OpenTelemetry GenAI client spans). That is the span your SLO dashboard should use.

What is the safest fallback to ship first?

For many RAG or support systems, the safest first fallback is a retrieval-only or reduced-answer mode. Azure's Throttling pattern treats graceful feature degradation and load leveling as normal overload strategies (Azure Throttling pattern). A clear degraded answer is often safer than silently switching to a model that changes quality, cost, or compliance behavior.

References

1. OpenAI API error codes
2. Anthropic Claude API errors
3. LiteLLM Router load balancing
4. Azure OpenAI quotas and limits
5. Google Cloud generative AI quotas and system limits
6. Amazon Bedrock quotas
7. Amazon Bedrock cross-Region inference
8. Amazon Bedrock inference profiles
9. OpenTelemetry GenAI semantic conventions
10. OpenTelemetry GenAI client spans
11. Azure Circuit Breaker pattern
12. Azure Retry pattern
13. Azure Throttling pattern