Source: original Sieon Labs image generated for this article and uploaded to WordPress media.
The first version of an agent tool is usually a schema and a handler. The schema says create_ticket needs a title, priority, and customer ID. The handler calls Linear, Jira, Zendesk, or an internal API. A demo works because the path is clean: model decides, tool runs, result returns.
Production breaks in the space between those verbs.
The model times out after the API committed. The orchestrator restarts after sending the request but before writing memory. A human approval arrives twice because two workers picked up the same pending action. A handoff moves the conversation to another runtime and the next runtime only sees a chat transcript that says, "I will create the ticket now."
If the tool creates one ticket, the system is fine. If it creates two, the agent did not just make a small mistake. It violated the operational contract of the business process.
That is why idempotency is not an API detail for payment companies. It is one of the missing runtime contracts for production agents.
Tool schemas describe shape, not commit semantics
Tool calling made agents useful because it moved actions out of prose and into typed calls. MCP pushes that further by standardizing how tools are described and invoked. A tool can expose input schemas, return structured results, and ask for more input through a multi-round flow.
That is necessary, but it is not enough.
A schema can tell the model that customer_id is required. It cannot tell the runtime whether a retry is safe after the network disconnects. It cannot tell a second worker whether a previous worker already committed the same side effect. It cannot prove to an auditor that the second tool response was a replay instead of a second execution.
Most teams try to patch this at the wrong layer. They add prompt rules such as "do not create duplicates" or they ask the model to check first. That helps with intent, but it does not solve the failure mode. The runtime needs a deterministic contract that survives model retries, process restarts, queue redelivery, and human approvals.
The contract should answer five questions:
| Question | Runtime answer |
|---|---|
| What operation is this? | A stable idempotency key scoped to a tenant, actor, tool, and intent |
| Is this the same request? | A normalized effect fingerprint, not raw model text |
| Did it commit? | A durable commit receipt from the downstream system or adapter |
| What should a retry return? | The stored result for the same key and fingerprint |
| What if the key is reused differently? | A conflict, not another side effect |
Stripe's idempotent request design is a useful mental model: clients send a key, the server stores the first result for that key, and later retries receive the same result. Amazon's Builders' Library makes the broader distributed-systems point: safe retries depend on caller-provided intent, not clever guessing from parameters alone.
Agent runtimes need the same idea around every side-effecting tool.
The idempotency contract for agent tools
For read-only tools, retries are usually fine. For tools that mutate the world, the runtime should require an idempotency envelope around the call.
A minimal envelope looks like this:
{
"idempotency_key": "tenant_42:conversation_9:approval_17:create_ticket",
"effect_fingerprint": "sha256:7f4c...",
"tool": "create_ticket",
"actor": "agent-runtime",
"expires_at": "2026-07-30T13:00:00Z"
}
The key is not just a UUID sprinkled into logs. It represents the business operation. If a user approves "create the incident ticket for outage X," every retry of that approved operation should carry the same key. If the user asks for a second ticket, that should be a different key.
The fingerprint is equally important. Without it, a reused key can hide a different operation. The adapter should canonicalize the fields that define the side effect, then hash them. For create_ticket, that might include tenant, project, title, severity, external customer ID, and an incident ID. It should not include volatile fields such as trace IDs, timestamps, or model wording.
The commit receipt is the durable proof that the side effect happened:
{
"idempotency_key": "tenant_42:conversation_9:approval_17:create_ticket",
"effect_fingerprint": "sha256:7f4c...",
"status": "committed",
"remote_system": "linear",
"remote_id": "LIN-1842",
"remote_url": "https://linear.example/issue/LIN-1842",
"committed_at": "2026-07-29T13:00:13Z"
}
On retry, the adapter should not call the remote API again. It should return the stored receipt. If the same key appears with a different fingerprint, it should return a conflict that the agent cannot silently resolve.
That conflict matters. It turns a dangerous duplicate side effect into a visible state transition:
{
"status": "conflict",
"reason": "idempotency_key_reused_with_different_effect",
"existing_fingerprint": "sha256:7f4c...",
"received_fingerprint": "sha256:b23a...",
"existing_remote_id": "LIN-1842"
}
A model can recover from that. A human can inspect it. An SRE can alert on it. What you cannot safely recover from is two committed actions that looked like one decision in the chat transcript.
Put the contract below the model
The right place for this logic is the tool adapter or runtime boundary, not the prompt.
A production loop should behave more like this:
- The planner proposes a side-effecting tool call.
- The policy layer classifies the call as mutating.
- The runtime derives or requires an idempotency key from the business operation.
- The adapter computes an effect fingerprint from canonicalized arguments.
- The adapter checks the idempotency store.
- It executes only if the key is new.
- It stores the commit receipt before returning success to the agent.
- It returns replay or conflict responses deterministically on later attempts.
Temporal's guidance around Activities is relevant here because Activities are the failure-prone business logic that workflows retry. If your agent system is already backed by a workflow engine, treat mutating tool calls like Activities: retries are expected, so the operation must be safe to replay.
If your agent system is a lighter queue-based runtime, the same rule still applies. Do not wait until you adopt a workflow engine to define the contract. A small database table with tenant, key, fingerprint, status, receipt JSON, timestamps, and expiry is often enough for the first version.
The implementation detail matters less than the invariant: the model never gets to decide whether a duplicate side effect is safe.
Observability should include replay decisions
A surprising number of agent traces show the prompt, the tool name, and the final text response, but not the operational state that mattered. When a duplicate ticket appears, the trace answers "what did the model say?" instead of "did the runtime replay or execute?"
OpenTelemetry gives enough primitives to fix this. A tool span should carry attributes such as:
agent.tool.name = "create_ticket"
agent.tool.side_effect = true
agent.idempotency.key = "tenant_42:conversation_9:approval_17:create_ticket"
agent.idempotency.fingerprint = "sha256:7f4c..."
agent.idempotency.decision = "execute"
agent.idempotency.receipt.remote_id = "LIN-1842"
For a replay, the same span should say agent.idempotency.decision = "replay". For a key collision, it should say conflict and record the existing receipt. That gives evaluation and incident review a real signal. You can measure how often tools are retried, which tools replay correctly, and which prompts or workers are causing conflicts.
This also changes how agent evals should be written. A test that asks, "Did the agent create a ticket?" is incomplete. A production-grade test asks:
- Does the first attempt commit exactly once?
- Does a timeout after commit return the same receipt on retry?
- Does queue redelivery replay instead of executing?
- Does a reused key with different arguments produce a conflict?
- Does the trace contain the key, fingerprint, decision, and receipt?
That is the difference between testing the conversation and testing the system.
A practical checklist
If you are adding side-effecting tools to an agent runtime, start with this checklist:
- Classify every tool as read-only, idempotent mutation, or non-idempotent mutation.
- Require idempotency keys for the mutation classes.
- Scope keys by tenant and business operation, not by process attempt.
- Canonicalize arguments before hashing the effect fingerprint.
- Store receipts durably before returning success to the agent.
- Return replayed receipts for exact retries.
- Return explicit conflicts for key reuse with different effects.
- Put key, fingerprint, decision, and receipt IDs into traces.
- Add failure tests that simulate timeout after commit and queue redelivery.
- Expire keys only after the business process can no longer be retried.
The hard part is not generating a key. The hard part is deciding what counts as the same business operation. That decision belongs close to product semantics. "Send the same approved refund" and "send another refund" are different operations even if the arguments look similar. "Create the incident ticket for outage X" and "retry creating the incident ticket for outage X" are the same operation even if the worker changed.
Make that distinction explicit, and the agent becomes easier to trust.
The real boundary
Production agent architecture is full of boundaries: model boundary, tool boundary, memory boundary, approval boundary, sandbox boundary. Idempotency is the boundary that makes side effects survivable.
Without it, every retry is a small gamble. With it, retries become ordinary distributed-systems behavior: safe, observable, and auditable.
That is the standard agent tools need to meet before they are allowed near real production workflows.
Sources
- Stripe API Reference: Idempotent requests, https://docs.stripe.com/api/idempotent_requests
- Amazon Builders' Library: Making retries safe with idempotent APIs, https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
- Temporal Docs: Activity Definition and Idempotency, https://docs.temporal.io/activity-definition#idempotency
- Model Context Protocol Docs: Tools, https://modelcontextprotocol.io/docs/concepts/tools
- OpenTelemetry Specification: Trace API, https://opentelemetry.io/docs/specs/otel/trace/api/