Graph Checkpoints Need Migration Tests

If your agent graph can resume from saved state, every graph change is a checkpoint migration until proven otherwise. LangGraph makes persisted execution state explicit through checkpointers, thread identifiers, state schemas, reducers, and time travel, so production teams should test old checkpoints against new graph code before shipping.

The checkpoint is part of the runtime contract

Most agent teams treat the graph definition as code and the checkpoint as storage. That split is too comfortable. In a production graph runtime, the checkpoint is not just a blob that happens to sit behind the graph. It is the serialized contract between yesterday's orchestration and tomorrow's deployment.

LangGraph's persistence documentation says persistence keeps useful information beyond a single graph run and gives applications short-term memory through checkpointers plus long-term memory through stores (LangGraph Persistence). That distinction matters. A store can behave like application data. A checkpointer persists graph state snapshots, which means it remembers where the graph was, what state shape existed, and which continuation path may run next (LangGraph Persistence).

The engineering rule is simple: if old state can be loaded by new graph code, the graph owns a compatibility surface. A renamed field, changed reducer, removed node, modified message shape, or new branch invariant can break a resumed run even when fresh requests pass every unit test. The problem is not that graph persistence is fragile. The problem is that persistence makes runtime history real.

This is the same family of concern that durable workflow systems have had for years. Temporal documents that workflow code must be deterministic and that changing orchestration logic can cause non-deterministic behavior during replay (Temporal Workflow Definition, Temporal Python Versioning). Agent graphs are not identical to Temporal workflows, but the deployment lesson transfers cleanly: when the runtime may replay, resume, fork, or inspect historical execution state, code changes need compatibility tests, not hope.

Where graph migrations fail in production

Graph migrations usually fail at boundaries that look harmless in code review. The first boundary is the state schema. LangGraph's Graph API makes StateGraph, state schemas, multiple schemas, reducers, messages in graph state, and graph migrations first-class concepts (LangGraph Graph API). That is a signal that state design is part of graph design, not incidental typing.

A typical failure begins with a field rename. The team changes retrieval_hits to evidence, updates the current nodes, and deploys. New runs work. Old checkpoints still contain retrieval_hits. When a human approves an interrupted run or a worker resumes a thread, the new summarizer reads evidence, sees an empty value, and produces a confident answer with no citations. Nothing crashed. The graph silently lost a safety invariant.

The second boundary is reducer behavior. Reducers define how updates are combined into state, and LangGraph documents default reducers, custom reducers, overwrite behavior, and reducers for graph state (LangGraph Graph API). A reducer change can alter the meaning of a checkpoint even when the state keys stay the same. Moving from append-only evidence accumulation to overwrite semantics may be correct for new runs, but old checkpoints may resume with partial branches that assumed append behavior.

The third boundary is node idempotency. LangGraph calls out node re-execution and idempotency in the Graph API documentation (LangGraph Graph API). Re-execution is not a theoretical corner. It appears after retries, worker restarts, interrupts, and forks. If a node writes external side effects before it records an idempotency key in state, a resumed checkpoint can send the same message twice or charge the same account twice.

The fourth boundary is time travel. LangGraph time travel lets teams replay past executions and fork alternative paths from checkpoints using state history and checkpoint identifiers (LangGraph Time Travel). That is a powerful debugging and evaluation tool, but it also expands the compatibility promise. If operators can fork a run from last week's checkpoint, the current graph should either handle that checkpoint or reject it with a clear migration error.

A migration harness for graph checkpoints

The practical fix is a checkpoint migration harness. It does not need to be large. It needs to preserve real checkpoint fixtures, load them under the candidate graph version, and assert the invariants that matter before deployment.

Start by collecting fixtures from production incidents and representative runs. Keep one fixture for an interrupted approval path, one for a tool failure path, one for a multi-branch join, one for a long conversation thread, and one for a successful happy path. LangGraph persistence uses a configurable thread_id when invoking persisted graphs, and production memory guidance recommends a database-backed checkpointer such as PostgresSaver rather than an in-memory saver (LangGraph Persistence, LangGraph Memory). That makes it natural to export fixtures by thread, checkpoint id, or state snapshot.

Then define a compatibility test matrix. For each fixture, the harness should answer four questions. Can the new graph deserialize the checkpoint state? Can it compute the next legal node or cleanly reject the checkpoint? Are required safety fields present after migration? Are external side effects protected by idempotency keys before any node can re-execute?

A minimal fixture format can stay intentionally boring:

from dataclasses import dataclass
from typing import Any, Mapping

@dataclass(frozen=True)
class GraphCheckpointFixture:
    name: str
    graph_version: str
    thread_id: str
    checkpoint_id: str
    state: Mapping[str, Any]
    expected_next: tuple[str, ...]
    required_keys: tuple[str, ...]


def validate_fixture_shape(fixture: GraphCheckpointFixture) -> None:
    if not fixture.thread_id:
        raise ValueError("thread_id is required")
    if not fixture.checkpoint_id:
        raise ValueError("checkpoint_id is required")
    missing = [key for key in fixture.required_keys if key not in fixture.state]
    if missing:
        raise ValueError(f"checkpoint fixture is missing keys: {missing}")

That code is not the whole migration system. It is the contract around the fixture. The actual harness should compile the candidate graph, load or reconstruct the fixture through the same checkpointer abstraction used in production, run one safe step, and assert the resulting state. The important part is that old checkpoints become test inputs, not archaeology.

For graph teams, the highest value fixtures come from incidents. If a replay failed because a reducer changed, keep that checkpoint. If a human approval resumed into the wrong branch, keep that checkpoint. If a fork exposed a missing field, keep that checkpoint. Evaluation prompts test model behavior. Checkpoint fixtures test runtime continuity.

Reducers, joins, and idempotency are compatibility surfaces

A graph checkpoint migration test should not only check that state can be parsed. Parsing is the easy part. The test should protect the semantics that make the graph safe.

Reducers deserve explicit assertions. Suppose a retrieval graph collects evidence from parallel workers and then joins those results before generation. If the evidence reducer changes from append to overwrite, the graph may still produce a valid Python object, but the join now observes a different set of evidence. LangGraph's reducer documentation exists because update combination is part of the Graph API, not an implementation afterthought (LangGraph Graph API). A migration test should assert that a checkpoint with two completed branches still reaches the join with two evidence bundles.

Joins deserve next-node assertions. A checkpoint that says branch A is complete and branch B is pending should not be treated like a fully joined state. When a graph uses conditional edges or commands to choose the next path, the harness should assert either the expected next node or an explicit migration rejection. Silent fallthrough is worse than a deploy failure because it hides under normal traffic.

Idempotency deserves side-effect assertions. LangGraph documents node re-execution and idempotency as graph design concerns (LangGraph Graph API). The migration harness should prove that a resumed checkpoint cannot re-send an email, re-run a payment, or re-open a ticket unless the node has a stable idempotency key. For senior teams, this is where graph engineering stops being prompt engineering. The graph is a distributed system with memory.

Time travel deserves fixture assertions too. Because LangGraph can replay and fork from checkpoint history, a production deployment should define how far back compatibility is promised (LangGraph Time Travel). One team may support all checkpoints for 30 days. Another may support only active threads and reject archived runs. Both are reasonable. The mistake is having no policy until an operator needs to fork a broken run.

Deployment decision rule

Use this decision rule: a graph deploy is safe only when every active checkpoint can be resumed, migrated, or rejected before any unsafe node executes.

That rule creates three acceptable outcomes. First, the checkpoint loads and continues under the new graph. Second, the checkpoint is migrated into the new state shape and then continues. Third, the graph refuses to continue and returns a clear operator action such as requires_manual_restart, unsupported_checkpoint_version, or migration_missing_required_field.

What is not acceptable is accidental continuation. Accidental continuation happens when a missing field becomes an empty list, an unknown branch becomes a default branch, or a changed reducer quietly drops half the evidence. These are the graph equivalent of schema drift in a database migration. The app stayed up, but the meaning changed.

The deploy pipeline should therefore include a small gate:

from typing import Iterable


def assert_checkpoint_compatibility(fixtures: Iterable[GraphCheckpointFixture]) -> None:
    failures: list[str] = []
    for fixture in fixtures:
        try:
            validate_fixture_shape(fixture)
            # In production, load this fixture through the graph checkpointer,
            # run one safe step, and assert next-node plus state invariants.
        except Exception as exc:
            failures.append(f"{fixture.name}: {exc}")
    if failures:
        raise AssertionError("checkpoint migration gate failed: " + "; ".join(failures))

This gate is intentionally smaller than a full end-to-end evaluation suite. It should run on every graph change because it protects a different failure mode. Model evals ask whether the agent gives the right answer. Checkpoint migration tests ask whether the agent can keep its promises across deploys.

The tradeoff is maintenance. Fixtures grow stale, old graph versions accumulate, and compatibility windows cost engineering time. That cost is real. The answer is not to avoid the harness. The answer is to publish the compatibility policy. If active threads are supported for 14 days, test 14 days. If only approved checkpoints are resumable, test approved checkpoints. If archived runs require restart, reject them explicitly.

Graph engineering becomes simpler when the team stops pretending checkpoints are storage internals. They are runtime contracts. Treat them like migrations, test them like migrations, and your agent graph can evolve without betraying the runs it already started.

FAQ

Why not delete old checkpoints during a deploy?

Deleting checkpoints can be valid for disposable experiments, but it is usually the wrong default for production agents. LangGraph persistence exists to keep useful information beyond a single run, and production memory guidance points teams toward durable checkpointers for real systems (LangGraph Persistence, LangGraph Memory). If the product promise includes resume, approval, audit, or time travel, deleting state is a product decision, not a cleanup task.

Is this only a LangGraph concern?

No. LangGraph makes the surface visible through checkpointers, state schemas, reducers, and time travel (LangGraph Graph API). Temporal shows the broader durable-execution principle: replayed orchestration code must remain deterministic, and versioning is needed when workflow code changes while executions are in progress (Temporal Workflow Definition, Temporal Python Versioning). Any agent runtime that resumes historical orchestration state needs a compatibility story.

What should a migration fixture contain?

A useful fixture contains the old graph version, thread id, checkpoint id, serialized state, expected next node, required safety keys, and the business invariant that made the run worth preserving. The fixture should come from a real path whenever possible, especially interrupted approvals, failed tools, parallel joins, and long-running conversations.

Where should checkpoint compatibility tests run?

Run the fast fixture harness in CI for every graph change, then run a smaller readback gate during deployment against active checkpoint versions. The CI gate catches obvious schema and reducer drift. The deployment gate catches environment-specific persistence issues, especially when production uses a database-backed checkpointer rather than an in-memory saver (LangGraph Memory).

References

1. LangGraph Persistence
2. LangGraph Graph API
3. LangGraph Time Travel
4. LangGraph Memory
5. Temporal Workflow Definition
6. Temporal Python Versioning