On this page
- The decision rule: no budget, no next iteration
- Model the loop as leases, budgets, and checkpoints
- Make retries earn another attempt
- Emit pressure as typed events
- Degrade before the loop stampedes
- A minimal loop governor in Python
- FAQ
- Are provider rate limits enough?
- Should retries live inside the node or the runtime?
- What should I measure first?
- Is degradation safer than stopping?
- References
An agent loop should not ask, "What should I do next?" until the runtime can answer, "Can this system afford the next step?" Backpressure is the contract that connects reasoning, retries, queues, token budgets, and degradation before a loop turns a partial failure into an outage.
The decision rule: no budget, no next iteration
I use one rule for production agent loops: no budget, no next iteration. A loop continuation is not just a reasoning choice. It is a capacity allocation. If the runtime cannot reserve token budget, wall-clock budget, retry budget, queue capacity, and a recovery path, the correct next action is to pause, degrade, or stop.
That sounds stricter than most agent examples because most examples treat the loop as a local control structure. In production, the loop is a distributed load generator. It calls model APIs, retrievers, tools, queues, workflow engines, and storage systems. Provider limits are already multidimensional. The Claude Platform documents request-per-minute, input-token-per-minute, and output-token-per-minute limits for the Messages API, and it can return 429 responses with retry-after guidance when one of those dimensions is exceeded (Claude Platform rate limits). A loop that only counts iterations is blind to the actual bottleneck.
The same lesson shows up in service capacity. The Google SRE overload chapter warns that queries per second is often a poor capacity metric because different requests can consume very different resources, and it recommends modeling direct resources such as CPU when possible (Google SRE, Handling Overload). Agent loops need the same move. Count the resource that gets scarce: uncached input tokens, output tokens, tool concurrency, retry attempts, queue age, checkpoint growth, and human approval latency.
The decision rule I want in the runtime is simple:
| Runtime question | Continue only if |
|---|---|
| Token budget | The next model call fits reserved input and output limits |
| Time budget | The next node can finish before the run deadline |
| Retry budget | The next attempt will not exceed policy or amplify overload |
| Queue budget | The work item still owns a valid lease or acknowledgement plan |
| Observability budget | The run can emit enough events to explain pressure later |
| Degradation budget | A cheaper path exists when the expensive path is unsafe |
If any row is false, the loop does not get to improvise. It must return a typed pressure decision: wait, shrink, fallback, ask for approval, or fail closed.
Model the loop as leases, budgets, and checkpoints
Backpressure starts with state. A loop cannot make safe continuation decisions if every iteration forgets why it is running. LangGraph separates short-term thread state from long-term application data: checkpointers persist graph state snapshots for thread-scoped continuity and fault tolerance, while stores persist cross-thread memory such as user preferences or shared knowledge (LangGraph persistence). That distinction matters because pressure state belongs with the run, not only with the prompt.
For a loop, I want each iteration to carry a small runtime envelope:
{
"run_id": "run_42",
"step": 7,
"lease_expires_at": "2026-08-19T13:00:20Z",
"remaining_input_tokens": 18000,
"remaining_output_tokens": 2400,
"remaining_attempts": 2,
"queue_age_seconds": 34,
"pressure": "normal"
}
This envelope is not a log line. It is the state contract that gates the next action. The model may propose another retrieval. The planner may propose another tool call. The runtime still owns the decision to continue.
Workflow systems learned this separation earlier. Temporal documents that Workflow Execution Timeout is infinite by default, and it generally does not recommend setting Workflow Timeouts for long-running resilient workflows because timeouts can reduce the workflow's ability to handle delays; Temporal recommends timers when a specific period matters (Temporal detecting Workflow failures). The lesson is not "never time out agents." The lesson is to distinguish the durable workflow from the attempt inside it. Long-lived work needs durable state. Individual calls need bounded attempts.
In agent systems, I map that into three layers:
- The durable run can remain open while waiting for tools, people, or provider recovery.
- Each node attempt gets a hard budget for time, tokens, and retries.
- The next iteration must read the latest envelope before it spends anything.
This is why I do not put loop pressure only in prompt text. Prompt text is advisory. A lease is enforceable. A checkpoint is replayable. A budget decrement is auditable.
Make retries earn another attempt
Retries are the easiest place for a loop to become dangerous. Temporal retries Activities by default with exponential backoff, including a 1 second initial interval, a 2.0 backoff coefficient, a 100 second maximum interval, and unlimited attempts unless configured otherwise (Temporal retry policies). LangGraph's fault-tolerance docs describe node retry policies with defaults such as max_attempts 3, initial_interval 0.5 seconds, backoff_factor 2.0, max_interval 128.0 seconds, and jitter enabled (LangGraph fault tolerance). Those defaults are useful, but an agent loop also needs to know whether the next attempt is still worth buying.
AWS's guidance on timeouts, retries, backoff, and jitter is the production framing I trust: retries can make transient failures survivable, but they can also increase load on a struggling backend; idempotency matters because a timeout does not prove side effects did not happen; jitter helps avoid synchronized retry bursts (AWS Builder Center). That directly applies to tool-using agents. Retrying a search query is different from retrying a payment, a deployment, a Slack message, or a database migration.
Queue systems add another boundary. Celery documents that a task message is not removed from the queue until it is acknowledged, that messages can be redelivered if a worker is killed, and that task functions should ideally be idempotent because the worker cannot detect idempotency for you (Celery tasks). For agent loops, that means the runtime should track whether a tool call is safe to replay before it schedules another attempt.
My retry gate has four checks:
| Check | Why it exists |
|---|---|
| Retryable error | Do not retry validation, policy, or deterministic prompt failures |
| Idempotency key | Do not duplicate side effects when the previous attempt may have succeeded |
| Backoff with jitter | Do not synchronize recovering agents into the same dependency |
| Budget reserve | Do not spend the last tokens on a retry that cannot finish the run |
The important part is that retry policy is not a decorator sprinkled on every node. It is a loop-level contract. A node can ask for another attempt. The runtime should make that attempt earn capacity.
Emit pressure as typed events
Backpressure that nobody can see becomes superstition. I want every loop to emit typed pressure events because the interesting failures happen between ordinary logs: the model call was successful, the tool call was successful, but the loop needed three retries, used twice the expected output budget, and reached the human approval queue late.
LangGraph streaming already points in this direction. Its stream-mode API exposes graph execution through modes such as updates, values, messages, custom, checkpoints, tasks, and debug, and the v2 format uses a consistent shape with type, namespace, and data (LangGraph streaming). LangGraph fault tolerance also exposes execution information such as node_attempt, thread_id, run_id, checkpoint_id, and task_id inside a node (LangGraph fault tolerance). Those are the pieces a runtime can turn into a pressure contract.
A useful loop pressure event is small and boring:
{
"type": "loop.pressure",
"run_id": "run_42",
"node": "retrieve_context",
"attempt": 2,
"input_tokens_reserved": 6000,
"output_tokens_reserved": 800,
"queue_age_seconds": 34,
"decision": "degrade",
"reason": "retrieval_budget_low"
}
I care less about the exact field names than the discipline. The event must explain the decision before the next iteration happens. If the loop degrades, the event says why. If it waits, the event includes the retry-after or local backoff. If it fails closed, the event identifies the exhausted budget.
This also changes evaluation. A prompt-only eval asks whether the answer was good. A loop-pressure eval asks whether the system stopped at the right boundary. Did it retry a non-idempotent tool? Did it respect provider retry-after? Did it shrink retrieval before exhausting output tokens? Did it emit enough state to reproduce the decision?
Degrade before the loop stampedes
The best loop backpressure policy is not "fail more often." It is "spend less before failure becomes the only honest response." Google SRE describes degraded responses such as searching a smaller candidate set or relying on a cheaper local copy, and it recommends redirecting when possible, serving degraded results when necessary, and handling resource errors transparently when all else fails (Google SRE, Handling Overload). Agent loops need degradation paths that are as explicit as their happy path.
For AI engineering, I usually define three degradation levels:
| Pressure | Runtime action | Example |
|---|---|---|
| Warm | Shrink expensive context | Retrieve top 5 passages instead of top 20 |
| Hot | Switch model or tool plan | Use cached summary before a fresh crawl |
| Critical | Stop or ask | Return partial result with missing evidence called out |
Provider caching can be part of that plan, but it is not a replacement for backpressure. Claude's rate-limit docs state that for most Claude models, cache_read_input_tokens do not count toward input-token-per-minute limits, while input_tokens and cache_creation_input_tokens do count (Claude Platform rate limits). That makes prompt caching a throughput tool. It still needs a runtime that decides when cached context is acceptable and when fresh context is required.
The mistake I see is treating degradation as a UX copy problem. It is an architecture problem. The loop should know which tools are optional, which sources are authoritative, which outputs can be partial, and which actions must never be guessed. A cheaper path is only safe when the runtime can prove it preserves the user's contract.
A minimal loop governor in Python
Here is a small, runnable Python 3.13 example that captures the shape. It does not call a model. It gates loop iterations with token budget, retry budget, queue age, and pressure decisions.
from dataclasses import dataclass, replace
from enum import Enum
class Decision(str, Enum):
CONTINUE = "continue"
DEGRADE = "degrade"
WAIT = "wait"
STOP = "stop"
@dataclass(frozen=True)
class LoopEnvelope:
run_id: str
step: int
remaining_input_tokens: int
remaining_output_tokens: int
remaining_attempts: int
queue_age_seconds: int
retry_after_seconds: int | None = None
@dataclass(frozen=True)
class GateResult:
decision: Decision
reason: str
envelope: LoopEnvelope
def gate_next_iteration(env: LoopEnvelope) -> GateResult:
if env.retry_after_seconds is not None:
return GateResult(Decision.WAIT, "provider_retry_after", env)
if env.remaining_attempts <= 0:
return GateResult(Decision.STOP, "retry_budget_exhausted", env)
if env.queue_age_seconds > 120:
return GateResult(Decision.STOP, "lease_too_old", env)
if env.remaining_input_tokens < 2_000 or env.remaining_output_tokens < 400:
degraded = replace(
env,
remaining_input_tokens=max(0, env.remaining_input_tokens - 500),
remaining_output_tokens=max(0, env.remaining_output_tokens - 100),
step=env.step + 1,
)
return GateResult(Decision.DEGRADE, "budget_low_use_cached_context", degraded)
next_env = replace(
env,
remaining_input_tokens=env.remaining_input_tokens - 2_000,
remaining_output_tokens=env.remaining_output_tokens - 400,
remaining_attempts=env.remaining_attempts - 1,
step=env.step + 1,
)
return GateResult(Decision.CONTINUE, "budget_reserved", next_env)
def run_demo() -> None:
env = LoopEnvelope(
run_id="run_42",
step=0,
remaining_input_tokens=5_500,
remaining_output_tokens=1_200,
remaining_attempts=4,
queue_age_seconds=10,
)
for _ in range(5):
result = gate_next_iteration(env)
print(result.decision.value, result.reason, result.envelope)
if result.decision in {Decision.STOP, Decision.WAIT}:
break
env = result.envelope
if __name__ == "__main__":
run_demo()
The example is intentionally small. In a real runtime, the gate would also read provider headers, queue lease state, checkpoint size, tool idempotency metadata, and per-customer quota. The key design remains the same: the loop cannot spend capacity until the gate returns a typed decision.
FAQ
Are provider rate limits enough?
No. Provider rate limits are necessary external constraints, and the Claude Platform documents RPM, input-token-per-minute, and output-token-per-minute dimensions (Claude Platform rate limits). They do not know your queue age, user priority, tool idempotency, checkpoint growth, or whether a degraded answer is acceptable.
Should retries live inside the node or the runtime?
The node can describe what failed, but the runtime should own the retry decision. Temporal separates durable workflows from retryable Activities (Temporal retry policies), and LangGraph composes node retries, timeouts, and error handlers in a fixed order (LangGraph fault tolerance). Agent loops need the same separation.
What should I measure first?
Start with the budgets that can stop the loop: input tokens, output tokens, elapsed time, attempt count, queue age, and tool concurrency. Google SRE's warning about QPS as a poor capacity proxy applies here too because different loop iterations can have very different resource costs (Google SRE, Handling Overload).
Is degradation safer than stopping?
Only when the degraded path preserves the user's contract. Google SRE describes degraded responses as a way to handle overload before serving errors, but it also says systems may need to handle resource errors transparently when all else fails (Google SRE, Handling Overload). For agents, that means cached context is fine for a summary, but not for a claim that requires fresh evidence.