Agent Loops Need State Machines

An agent loop becomes production software when every turn has a named state, a permitted transition, and a terminal condition. The model can still choose actions, but the runtime should decide whether the loop may continue, retry, hand off, pause for approval, resume, or stop.

The decision rule: name every loop state

The smallest agent loop looks harmless: ask the model, run a tool if the model requests one, append the result, and ask again. Anthropic describes that tool-use cycle as the model requesting a tool, the client executing it, the client returning the result, and the model continuing with that result. OpenAI's function calling guide uses the same boundary: the model can generate structured tool calls, while the application executes those calls.

That boundary is exactly why the loop should not be treated as a casual while statement. The model proposes. The runtime disposes. If the runtime has no explicit state model, continuation becomes an accident of whatever message appeared last.

My decision rule for loop engineering is simple: if a loop turn can spend money, touch external state, wait for a human, retry after failure, or resume later, it deserves a named state. The name does not need to be fancy. It only needs to make the next legal transition reviewable.

A useful first vocabulary is:

State	Runtime question
planning	What should the model decide next?
awaiting_tool	Is this tool call allowed, bounded, and idempotent enough to run?
executing_tool	Who owns cancellation, timeout, and side effects?
observing_result	Is the result valid, late, partial, or unsafe to reuse?
awaiting_approval	Which human decision is required before continuation?
handoff	Which agent, worker, or service owns the next state?
recovering	Is retry or compensation still useful?
completed	What final artifact or answer was produced?
failed	Which invariant failed?
expired	Did the run become irrelevant before completion?

The names are less important than the discipline. A senior review should be able to ask, “what state is this run in, who may move it, and what transition is forbidden?”

Why while tool_calls hides production risk

The typical tool loop optimizes for developer speed. It is easy to read, easy to demo, and easy to extend with one more tool. The risk is that all control decisions collapse into one condition: does the model want another tool call?

That condition is too weak for production.

First, it hides retry policy. A tool failure can become a new prompt, a model apology, another tool attempt, or a final answer. Without a state transition such as executing_tool -> recovering, the retry decision becomes implicit prompt behavior instead of runtime policy.

Second, it hides approval policy. The OpenAI Agents SDK documentation separates concepts such as running agents, handoffs, guardrails, results, and human-in-the-loop workflows. That separation is a useful signal. Approval is not just another message in the chat transcript. It is a state where continuation is blocked until a particular decision exists.

Third, it hides termination. A loop can stop because the answer is complete, because the model gave up, because the budget expired, because a guardrail blocked a step, because an approval was denied, or because the run was transferred. Treating all of those as “no more tool calls” destroys the reason the loop ended.

Fourth, it hides ownership. In a multi-agent system, the OpenAI Agents SDK agent documentation lists handoffs and multi-agent design patterns as explicit design concepts. A handoff is not just a longer prompt. It is a transfer of state, authority, and accountability.

The mistake is not using a loop. The mistake is letting the loop condition become the architecture.

A minimal state model for agent loops

A practical loop state does not need to be large. I usually want five fields before anything else:

{
  "run_id": "run_93a7",
  "state": "awaiting_tool",
  "attempt": 2,
  "owner": "tool_router",
  "allowed_transitions": ["executing_tool", "awaiting_approval", "failed", "expired"],
  "deadline_at": "2026-08-12T13:01:00Z",
  "last_decision": {
    "by": "model",
    "kind": "tool_call",
    "tool": "search_docs",
    "arguments_hash": "sha256:..."
  }
}

This object is not a universal schema. It is a contract. The model can suggest a tool call, but the runtime checks whether awaiting_tool -> executing_tool is legal. The runtime can also choose awaiting_tool -> awaiting_approval when the tool has side effects, or awaiting_tool -> expired when the deadline has passed.

The state machine makes illegal moves visible. A run should not jump from planning to completed after a side-effecting tool call unless the result was observed and accepted. A human denial should not route back to planning unless the product explicitly allows a safe alternative. A late tool result should not silently re-enter the active plan if the run is already expired.

That sounds bureaucratic until the first incident review. Then the question changes from “why did the model do that?” to “which transition did the runtime allow?”

Where retries, approvals, and handoffs fit

Retries belong in the transition table, not in scattered SDK options. A failed tool call might be retryable if it is idempotent, the error is transient, and enough budget remains. A failed payment update should not be retried just because the loop wants another attempt. The transition should encode those differences.

A small transition policy can be enough:

from dataclasses import dataclass
from enum import StrEnum


class State(StrEnum):
    PLANNING = "planning"
    AWAITING_TOOL = "awaiting_tool"
    EXECUTING_TOOL = "executing_tool"
    OBSERVING_RESULT = "observing_result"
    AWAITING_APPROVAL = "awaiting_approval"
    RECOVERING = "recovering"
    COMPLETED = "completed"
    FAILED = "failed"
    EXPIRED = "expired"


TERMINAL = {State.COMPLETED, State.FAILED, State.EXPIRED}
ALLOWED = {
    State.PLANNING: {State.AWAITING_TOOL, State.AWAITING_APPROVAL, State.COMPLETED, State.FAILED, State.EXPIRED},
    State.AWAITING_TOOL: {State.EXECUTING_TOOL, State.AWAITING_APPROVAL, State.FAILED, State.EXPIRED},
    State.EXECUTING_TOOL: {State.OBSERVING_RESULT, State.RECOVERING, State.FAILED, State.EXPIRED},
    State.OBSERVING_RESULT: {State.PLANNING, State.COMPLETED, State.FAILED, State.EXPIRED},
    State.AWAITING_APPROVAL: {State.EXECUTING_TOOL, State.PLANNING, State.FAILED, State.EXPIRED},
    State.RECOVERING: {State.EXECUTING_TOOL, State.PLANNING, State.FAILED, State.EXPIRED},
}


@dataclass(frozen=True)
class LoopContext:
    state: State
    attempts: int
    remaining_ms: int
    idempotent: bool


def can_transition(ctx: LoopContext, target: State) -> bool:
    if ctx.state in TERMINAL:
        return False
    if target not in ALLOWED.get(ctx.state, set()):
        return False
    if target == State.EXECUTING_TOOL and ctx.remaining_ms <= 0:
        return False
    if target == State.RECOVERING and (ctx.attempts >= 3 or not ctx.idempotent):
        return False
    return True

The example is deliberately small. It does not replace a workflow engine. It gives the team a place to put policy before the next incident forces policy into comments, prompts, and dashboard folklore.

Human approval should fit the same model. A runtime can store awaiting_approval with the exact proposed action, argument hash, approver identity, expiry, and allowed outcomes. If the approval is denied, the transition should say whether the run fails, replans with constraints, or creates a separate remediation task. “Ask the user again” should be a named policy, not the accidental result of another model turn.

Handoffs also become cleaner. Instead of passing a transcript to another agent, pass the current state, accepted observations, pending obligations, and allowed next transitions. That keeps the receiving agent from inheriting stale authority.

Persistence changes what termination means

Persistence is where implicit loops become dangerous. LangGraph documents persistence through checkpoints that retain graph state, and its durable execution documentation frames long-running and human-in-the-loop workflows as first-class concerns. That is the right direction for real agent systems because important work often outlives a single HTTP request.

But once a loop can resume, termination must be explicit. A stored message list can tell the model what happened. It does not automatically tell the runtime whether the run is still allowed to continue.

Temporal is useful precedent here. Temporal describes a Workflow as durable, reliable, and scalable function execution, and Workflow Execution documentation distinguishes workflow IDs, run IDs, histories, events, timers, and Continue-As-New. The lesson for agents is not that every agent must run on Temporal. The lesson is that durable work needs durable control semantics.

A resumed agent should answer three questions before the next model call:

1. What state was persisted?
2. Is the next transition still legal under the current policy?
3. Is the original reason for the run still valid?

If the run was awaiting_approval and the approval expired overnight, the correct next state may be expired, not planning. If the run was executing_tool when a worker crashed, the next state may be recovering, not “call the model with the last message.” If the run was completed, a replay should reproduce or audit the result, not continue into new work.

Durability should preserve the loop's obligations, not just its memory.

Observability should show transitions, not just tokens

If loops are state machines, traces should show state transitions. OpenTelemetry describes traces as a way to represent work with spans, and context propagation keeps related execution connected across process boundaries. Agent runtimes should use that precedent for loop control.

A useful trace should not only show model latency and token count. It should show:

Transition	Useful attributes
planning -> awaiting_tool	model, tool name, argument schema version
awaiting_tool -> awaiting_approval	policy reason, risk class, approval expiry
awaiting_tool -> executing_tool	idempotency, expected latency, deadline remaining
executing_tool -> observing_result	status, result size, validation outcome
observing_result -> planning	accepted facts, rejected fields, next obligation
recovering -> failed	attempts, final error class, compensation status
any -> expired	deadline, owner, pending side effects

That table is more valuable than a screenshot of a chat transcript. It tells the on-call engineer which runtime decision moved the run forward.

This is also how evals improve. A failed run can be converted into a fixture at the transition level: “when the run is awaiting_tool with a non-idempotent action and an expired approval, the only legal states are failed or expired.” That is a stronger regression test than replaying the whole conversation and hoping a judge catches the same failure.

Implementation pattern

The implementation pattern is simple enough to start inside an existing orchestrator.

First, define states and terminal reasons in source control. Avoid letting each tool wrapper invent its own failure labels.

Second, validate every transition in one place. Even if the model emits a perfect next step, the runtime should reject transitions that violate approval, deadline, idempotency, or ownership policy.

Third, persist state separately from transcript. Messages are evidence. State is control data. They can reference each other, but they should not be the same blob.

Fourth, log transition events with stable names. That gives tracing, replay, eval promotion, and incident review the same vocabulary.

Fifth, keep the model out of terminal decisions. The model can recommend that the task is done. The runtime should mark completed only after output validation, required artifacts, and pending side effects are resolved.

The smallest production checklist looks like this:

Design review question	Why it matters
What are the terminal states?	Prevents infinite or ambiguous loops.
Which states can call tools?	Contains side effects.
Which states require human approval?	Turns consent into runtime policy.
Which transitions are retryable?	Keeps recovery from becoming repetition.
What state is persisted?	Makes resume behavior deterministic.
What transition is traced?	Makes incidents diagnosable.

This pattern can coexist with framework abstractions. If a graph framework already gives you nodes and checkpoints, use those primitives. If a workflow engine already gives you histories and timers, map agent states onto them. If an SDK gives you lifecycle hooks and handoffs, record those as transitions. The point is not to build a custom orchestration religion. The point is to stop treating loop continuation as prompt side effect.

Trade-offs senior teams should price

The first trade-off is velocity. A raw loop is faster to prototype. A state machine forces the team to name decisions earlier. I accept the friction when the loop can create side effects, run in the background, or survive process boundaries. I avoid it for a toy assistant that only answers from local context.

The second trade-off is policy surface area. Once transitions exist, teams will want to add rules. That can become a bureaucracy if every state requires a committee. Keep the initial transition table small and make terminal states strict. Expand only after incidents or product requirements prove the need.

The third trade-off is framework coupling. OpenAI Agents SDK exposes orchestration, handoffs, guardrails, lifecycle hooks, and human-in-the-loop concepts. LangGraph exposes graph persistence and durable execution concepts. Temporal exposes durable workflow execution concepts. Those tools can help, but the engineering decision should survive adapter changes. Name the states in your product language first, then map them onto the framework.

The fourth trade-off is observability cost. Recording every transition adds data volume. The answer is not to skip transition logs. The answer is to record compact, stable attributes: state, target state, policy reason, owner, deadline remaining, attempt count, and correlation IDs. That is enough to debug most loop failures without dumping private prompts or tool logs into traces.

The payoff is controlled autonomy. The model still has room to reason, plan, and propose actions. The runtime gets a reviewable contract for when the loop may continue. That separation is the difference between an impressive demo and an agent system senior engineers can operate.

FAQ

Is a state machine overkill for a simple tool-calling agent?

For a local prototype, often yes. For an agent that calls external tools, waits for approval, retries, hands off work, or resumes after interruption, no. Tool-use docs already separate model requests from client-side execution, so the application needs a control model once execution matters.

How is this different from using LangGraph or Temporal?

LangGraph and Temporal can provide useful execution primitives. LangGraph documents persistence through checkpoints, while Temporal documents durable workflow execution. The state machine described here is the product-level control contract you map onto those primitives.

Where should human approval live in the loop?

Approval should be a state, not a message. Store the proposed action, argument hash, approver, expiry, and allowed outcomes. The OpenAI Agents SDK documentation treats human-in-the-loop workflows and guardrails as explicit runtime concepts, which is the right mental model.

What should count as a terminal state?

At minimum: completed, failed, and expired. Many teams also add cancelled, denied, or compensated. The important rule is that terminal states cannot re-enter planning without a new owner, new reason, and new run identity.

References

1. OpenAI Agents SDK Runner
2. OpenAI Agents SDK Agents
3. LangGraph Persistence
4. LangGraph Durable Execution
5. Temporal Workflows
6. Temporal Workflow Execution
7. Anthropic Tool Use Overview
8. OpenTelemetry Traces
9. OpenTelemetry Context Propagation
10. OpenAI Function Calling