Agent Workers Need Leases, Not Hope

A lot of agent systems look reliable until a tool call takes longer than the chat turn that launched it.

The model emits a tool request. The runtime hands it to a worker. The user interface moves on. Somewhere else, a job is calling an API, writing a file, provisioning a sandbox, rebuilding an index, or waiting on another service. If the worker finishes quickly, everyone pretends the architecture is simple. If it crashes after committing a side effect, retries the same request, loses its network connection, or gets restarted during deploy, the missing abstraction becomes visible.

The missing abstraction is a lease.

An agent worker should not be treated as a function call with a longer timeout. It should be treated as a leased task: one worker owns it for a bounded period, the lease can be renewed while progress is still credible, the task is settled only after the outcome is durable, and every retry carries an idempotency key plus a traceable receipt.

This sounds like queue engineering, because it is. The difference is that agent systems often hide the queue behind friendly terms like background tool execution, async action, autonomous worker, or long-running run. The name does not change the failure mode.

The queue already knows the contract

AWS SQS visibility timeout is the cleanest mental model. When a consumer receives a message, the message stays in the queue but becomes temporarily invisible to other consumers. The consumer is expected to process and delete it before the timeout expires. If it does not delete the message in time, the message becomes visible again and another consumer can receive it. AWS also exposes ChangeMessageVisibility so a consumer can extend or shorten that timeout when the work takes a different amount of time than expected.

Google Cloud Pub/Sub calls the same shape lease management. A pull subscriber must process and acknowledge a message within the acknowledgment deadline, or extend that deadline. Google describes the operating tradeoff directly: a deadline that is too low increases duplicate likelihood, while a deadline that is too high delays redelivery after failure.

Azure Service Bus uses Peek-Lock and settlement language. A receiver locks a message, processes it, then completes, abandons, dead-letters, or lets the lock expire. Microsoft also calls out the uncomfortable production detail: if completion fails after minutes of processing, the receiver has to decide whether to preserve the work result and ignore the duplicate delivery later, or discard the result and allow retry.

Those are not implementation details for queue specialists. They are the runtime contract agent workers need.

Why agent workers make this sharper

A normal background job usually has a bounded input and a predictable handler. Agent work is messier.

An agent worker may execute a tool chosen by a model, fetch context, call multiple APIs, ask another agent, stream partial status, and then decide whether the original request still makes sense. It may be asked to do work that is expensive, stateful, or impossible to roll back. Some failures are cheap retry cases. Some failures are business decisions. Some retries would duplicate an external side effect.

That means the runtime needs to separate four questions that are often collapsed into one boolean called success:

Question	Bad hidden version	Better runtime version
Who owns this work right now?	Some worker process	Lease holder plus expiration time
Is progress still plausible?	No signal until timeout	Heartbeat or lease renewal with reason
Has the side effect committed?	The handler returned	Settlement receipt with idempotency key
What should retry do?	Run it again	Retry policy based on stage and receipt

The central idea is simple: a worker is not the owner of the task forever. It is only the temporary holder of a lease.

flowchart LR
    A[Agent run] --> B[Durable task]
    B --> C[Lease token]
    C --> D[Worker executes tool]
    D --> E{Done before expiry?}
    E -->|yes| F[Ack plus receipt]
    E -->|no| G[Renew or release]
    G --> H[Retry with idempotency key]
    H --> D
    F --> I[Trace links task to outcome]

Original Sieon Labs architecture diagram, rendered from the article source.

The lease should be visible in your domain model

If leases are buried inside a queue library, the product runtime cannot reason about them. Agent systems need the lease in the task record.

A minimal task record should include:

• task_id: stable identifier for the unit of work.
• agent_run_id: the conversation, plan, or run that requested it.
• lease_owner: worker identity, not just a process ID.
• lease_expires_at: when another worker is allowed to take over.
• renewal_count: how many times ownership was extended.
• stage: preflight, external_call, commit, verify, or settled.
• idempotency_key: the key used for external side effects.
• settlement_receipt: proof that the result was recorded.
• trace_id: link into logs, spans, and user-visible status.

That list looks heavier than enqueue(payload). It is also the difference between retrying safely and hoping the duplicate does not matter.

The worker loop can stay small:

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

@dataclass
class Lease:
    task_id: str
    owner: str
    expires_at: datetime
    token: str


def process_task(store, worker_id: str, now=None) -> None:
    now = now or datetime.now(timezone.utc)
    lease = store.acquire_due_task(worker_id, ttl=timedelta(minutes=2), now=now)
    if lease is None:
        return

    try:
        task = store.load_task(lease.task_id)
        receipt = run_tool_with_idempotency(
            payload=task.payload,
            idempotency_key=task.idempotency_key,
            heartbeat=lambda stage: store.renew_lease(
                token=lease.token,
                stage=stage,
                ttl=timedelta(minutes=2),
            ),
        )
        store.settle_task(lease.token, receipt=receipt)
    except RetryableToolError as exc:
        store.release_for_retry(lease.token, reason=str(exc))
    except PermanentToolError as exc:
        store.dead_letter(lease.token, reason=str(exc))

This is not a complete framework. It is the minimum shape the framework should protect. Acquisition, renewal, release, settlement, and dead-lettering are separate operations because they answer different operational questions.

Renewal is a claim, not a heartbeat for its own sake

A common mistake is renewing a lease forever because the worker process is alive. That only proves the process can still call the database. It does not prove the task is making progress.

Renewals should carry stage information. A worker that renews in external_call for ten minutes tells a different story than a worker that cycles through preflight, commit, and verify. The runtime can use that difference to alert, shorten leases, or route the retry to a human checkpoint.

Good renewal reasons are specific:

• downloaded 18 of 24 source documents
• waiting for model batch result
• committed draft, verifying readback
• index rebuild 72 percent complete

Weak renewal reasons are vague:

• still running
• heartbeat
• processing

For agent systems, renewal messages also become user experience. A user does not need every log line, but they do need a trustworthy answer to the question, "is the agent still doing useful work?"

Settlement should happen after verification

The dangerous part of a long-running agent task is often the gap between side effect and acknowledgment.

A worker may publish a post, create a GitHub issue, update a record, or send a message, then fail before it records success. If the queue redelivers the task, the next worker sees an unfinished task and may repeat the side effect. This is why Microsoft points to message IDs and idempotent handling as the typical mechanism for duplicate deliveries.

For agent workers, settlement should be a two-part act:

1. Commit the external side effect with an idempotency key or deterministic target identifier.
2. Read back enough state to create a settlement receipt.

A receipt should be boring and concrete:

{
  "task_id": "task_018",
  "stage": "settled",
  "external_system": "wordpress",
  "external_id": "2451",
  "status": "publish",
  "idempotency_key": "agent-run-8842:publish:daily-post",
  "verified_at": "2026-08-02T13:00:00Z"
}

The receipt lets a retry become a readback operation before it becomes a write operation. That is the practical line between durable automation and duplicate side effects.

Retry policy belongs to the stage

Most retry settings are too generic for agent work. Retry every error three times with exponential backoff is reasonable for a network call. It is reckless for a worker that may have already crossed a commit boundary.

A better retry table is stage-aware:

Stage	Safe retry default	Required evidence
Preflight	Retry automatically	No external side effect started
External call	Retry with idempotency key	Provider supports idempotency or deterministic resource name
Commit	Read back first	External target may already exist
Verify	Retry readback	Side effect likely completed
Settled	Do not rerun	Settlement receipt exists

This is where queue primitives and agent traces meet. The queue decides when work can be retried. The trace explains what the worker had already done. The idempotency key prevents duplicate writes. The receipt tells the next worker whether to continue, verify, or stop.

What to measure

If leases matter, they should be observable. At minimum, track:

• Lease acquisition latency.
• Lease renewal count per task.
• Tasks expired by stage.
• Duplicate delivery count.
• Retry count by error class.
• Settlement latency after external commit.
• Dead-letter rate by tool.
• Tasks with missing or stale trace links.

The metric that tends to reveal bad design fastest is expired tasks by stage. Expiration during preflight usually means capacity or scheduling trouble. Expiration during external calls may mean lease TTLs are too short or provider calls are unbounded. Expiration during commit is a correctness risk and should page someone before duplicate writes accumulate.

The decision rule

Use a simple rule when designing agent infrastructure: if a tool call can outlive the request that launched it, it needs a lease.

Not just a timeout. Not just a background worker. Not just a retry decorator. A real lease with ownership, expiration, renewal, settlement, idempotency, and trace evidence.

That does not make the system complicated. It makes the existing complexity explicit enough to operate.

Sources

• Amazon SQS visibility timeout
• Google Cloud Pub/Sub lease management
• Azure Service Bus message transfers, locks, and settlement

References

1. Amazon SQS visibility timeout
2. Google Cloud Pub/Sub lease management
3. Azure Service Bus message transfers, locks, and settlement