Every production LLM system eventually learns the same lesson: a model call is not just a function call. It is a request against a constrained, shared capacity pool.
The mistake is to discover that pool through 429 errors in the hot path. A team ships a retrieval agent, adds streaming, gives users longer context windows, and suddenly the incident is not hallucination or latency. It is capacity shape. The application has enough requests per minute but not enough tokens per minute. Or it has enough daily quota but not enough burst headroom. Or a fallback model exists, but the fallback sits behind the same overloaded provider account.
That is why rate limits should be modeled as a capacity API, not as an exception handler. The provider response is the last signal. The runtime should already know the budget it is spending.
The Limit Is Multidimensional
OpenAI documents rate limits across dimensions such as requests per minute, requests per day, tokens per minute, tokens per day, images per minute, and audio minutes for some models. The important detail is that the first exhausted dimension wins. A workload can be under its token limit and still fail because it ran out of request slots. It can be under its request limit and still fail because a few large prompts consumed the token pool.
Anthropic exposes a similar reality for the Messages API, with request rate, input token rate, and output token rate measured separately. Its documentation also calls out acceleration limits: a sharp traffic increase can trigger 429s even if the average rate looks reasonable. Microsoft Azure OpenAI adds deployment and quota-tier concerns. Amazon Bedrock and Google Vertex AI both frame model usage through account, region, API, and feature quotas.
The pattern is consistent across providers. Capacity is not one number. It is a vector.
For an AI product, that vector usually includes:
- request slots per provider, model, region, and project;
- input tokens admitted per minute;
- output tokens reserved per minute;
- daily or monthly spend ceilings;
- per-tenant fairness budgets;
- burst limits that are stricter than the headline per-minute number;
- queue depth and maximum wait time;
- fallback capacity that may or may not be independent.
If the application only handles 429 Too Many Requests after the provider rejects the call, it has already lost the most useful design window. It can retry, but it cannot make a principled decision about priority, degradation, or fairness.
Treat Quota as Runtime State
A production AI gateway should maintain an internal view of quota state before calls leave the system. It does not need perfect provider telemetry. It needs conservative accounting that is good enough to make admission decisions.
The runtime should answer four questions before sending a request:
- Which capacity pool will this request consume?
- How many input and output tokens should we reserve?
- Is the request worth admitting now, queuing briefly, degrading, or rejecting?
- If we use a fallback, does it consume independent capacity or the same bottleneck?
That turns rate limiting from a transport concern into a scheduling concern. The gateway becomes a small operating system for model capacity.
flowchart LR
A["User request"] --> B["Admission policy"]
B --> C["Token estimate"]
C --> D{Capacity available?}
D -->|yes| E["Primary model"]
D -->|wait| F["Bounded queue"]
D -->|degrade| G["Smaller context or cheaper model"]
D -->|no| H["Typed overload response"]
E --> I["Usage ledger"]
F --> I
G --> I
This is not only about protecting providers. It protects user experience. A typed overload response is often better than a mystery spinner followed by a retry storm. A bounded queue is better than letting every worker sleep and wake at the same time. A smaller context window can preserve an interactive workflow when full retrieval would exceed the token budget.
A Minimal Admission Controller
The smallest useful implementation is a token bucket per capacity dimension. For LLMs, a single bucket is rarely enough. You usually need one bucket for requests and another for estimated tokens. For chat systems, reserve output tokens as well as input tokens, because the answer can be the expensive part.
from dataclasses import dataclass
from time import monotonic
@dataclass
class Bucket:
capacity: float
refill_per_second: float
tokens: float
updated_at: float
def refill(self) -> None:
now = monotonic()
elapsed = now - self.updated_at
self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_second)
self.updated_at = now
def try_take(self, amount: float) -> bool:
self.refill()
if self.tokens < amount:
return False
self.tokens -= amount
return True
@dataclass
class ModelPool:
requests: Bucket
input_tokens: Bucket
output_tokens: Bucket
def admit(self, estimated_input: int, reserved_output: int) -> bool:
if not self.requests.try_take(1):
return False
if not self.input_tokens.try_take(estimated_input):
return False
if not self.output_tokens.try_take(reserved_output):
return False
return True
This example is intentionally incomplete. A real implementation needs rollback when the second bucket fails after the first one succeeds. It needs distributed coordination if multiple gateway replicas share one provider account. It needs reconciliation against actual usage returned by the provider. It also needs priority classes, because background summarization and an interactive coding agent should not compete equally.
Still, the example captures the important shift. The application is deciding before the provider rejects the request.
Retries Need a Budget Too
Provider docs often mention retry guidance and Retry-After headers. Those hints are useful, but they are not a strategy by themselves. A fleet of workers that all obey the same retry delay can create synchronized pressure. The result is a retry wave that consumes the next available window before new user traffic gets a chance.
Retries should be admitted through the same capacity policy as first attempts. They also need their own budget.
A practical policy looks like this:
- only retry idempotent model tasks or tasks with a stable request key;
- obey provider
Retry-Afterwhen present; - add jitter so workers do not wake together;
- cap retries by user-facing deadline, not only by attempt count;
- lower priority for retries when fresh interactive traffic is waiting;
- record retry cause, provider, model, and capacity pool in traces.
The subtle point is deadline. If a request has three seconds of user patience left, a ten-second retry is not resilience. It is work that cannot help the user. The correct response may be a smaller model, a shorter answer, or a clear overload message.
Fallbacks Can Hide the Same Bottleneck
Many teams add a fallback model and assume capacity risk is solved. Sometimes that is true. Often it is not.
A fallback is only independent if it uses a different constrained resource. A second model in the same provider account may share account-level quota. A different deployment in the same Azure region may still share subscription limits. A model in another provider may be operationally independent but semantically weaker, slower, or more expensive.
Fallback policy should describe what changes:
| Fallback type | Helps with | Does not solve |
|---|---|---|
| Same provider, smaller model | per-model pressure, cost | account-level quota, provider outage |
| Same provider, different region | regional saturation | account-level policy, data residency constraints |
| Different provider | provider outage, independent quota | output drift, tooling differences, compliance review |
| Degraded local path | availability for narrow tasks | broad reasoning quality, maintenance cost |
This table belongs in the design doc before launch. During an incident, nobody should be debating whether a fallback can use user data, whether it supports the same tool schema, or whether its output is acceptable for the task.
Observability Should Expose Capacity Decisions
If rate limits are treated as random API failures, dashboards show only provider errors. That is too late. The control plane needs metrics and traces for decisions the provider never sees.
Track at least these signals:
- admitted, queued, degraded, rejected, and retried requests by tenant and route;
- estimated input tokens, reserved output tokens, and actual usage;
- queue wait time and deadline expiration;
- capacity pool saturation by provider, model, region, and deployment;
- fallback reason and fallback target;
- provider 429 count separated from local admission denials;
- user-visible overload responses.
This separation matters. A local admission denial can be a success if it prevents a worse user experience. A provider 429 can be a bug in local accounting. A fallback can be healthy at low volume and dangerous when it silently changes answer quality for a premium workflow.
The Engineering Contract
Rate limits become manageable when they are owned by the runtime, not scattered across SDK wrappers. The contract should be explicit:
- product decides which requests are allowed to degrade;
- platform owns provider quota configuration and capacity ledgers;
- application teams provide token estimates, deadlines, and priority;
- observability records every admission decision;
- evaluation covers fallback quality, not only primary-model accuracy;
- incident response has a playbook for raising quota, shifting traffic, and reducing demand.
This is why Azure's guidance around quota increases and Provisioned Throughput Units matters. Some workloads should not be run on best-effort shared capacity forever. If latency and availability are product commitments, reserved or provisioned capacity may be cheaper than building elaborate retry behavior around an insufficient pool.
A Launch Checklist
Before a production AI feature ships, ask these questions:
- What is the expected request rate, input token rate, and output token rate at peak?
- Which provider limits are per model, per account, per project, per region, or per deployment?
- What happens when only one dimension is exhausted?
- Are background jobs isolated from interactive traffic?
- Do retries consume a separate retry budget?
- Can the system degrade without changing regulated or user-critical behavior?
- Are fallback outputs evaluated against the primary model for the tasks they will handle?
- Can operators see local admission decisions separately from provider 429s?
- Is there a manual and automated path to reduce demand during an incident?
- Does the product copy explain temporary capacity limits without blaming the model?
The best LLM systems do not pretend capacity is infinite. They make scarcity visible, schedulable, and testable.
A 429 should be the rare confirmation that a boundary exists, not the first time your architecture learns where the boundary is.