Durability in production

A checkpoint is not a transaction

A checkpoint records the state after a node completes. It does not make the node’s side effects atomic with that write: the email is sent, the process dies before the checkpoint lands, and the resume runs the node again. Every step that touches the world has to tolerate being executed twice.

There is no way to make a remote call and a local write happen together; that is the two-generals problem in a smaller costume. The practical answer is an idempotency key derived from the thread and the step, sent with the request, so the payment provider or mail service collapses the duplicate instead of charging twice.

For effects you own, write the intent and the result in the same database as the checkpoint, or use an outbox: record “send this” in the transaction, and let a separate worker deliver it. The graph then commits one row and the delivery is retried independently. The outbox row is the source of truth; the graph only has to write it once.

Retry the transient, not the permanent

Not every failure deserves a retry. A timeout, a rate limit and a dropped connection are transient; a validation error, a missing field and a 404 are not. Retrying the second kind burns the budget and delays the dead letter that would have told you what was wrong.

Give each node a policy: which exceptions, how many attempts, what backoff. A cap is not optional, because a node that always fails will otherwise retry until the recursion limit and leave a thread that looks stuck rather than failed.

When the cap is reached, route to a dead-letter node rather than raising into the void. Record the input, the error and the attempt count, mark the thread failed, and let a person decide. A poison message that blocks a queue is an outage; one that lands in a dead letter is a ticket.

A durable checkpointer and a retry policy

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.types import RetryPolicy

builder.add_node(
    "charge",
    charge,
    retry_policy=RetryPolicy(max_attempts=3, retry_on=ConnectionError),
)

with PostgresSaver.from_conn_string(DB_URL) as saver:
    saver.setup()
    graph = builder.compile(checkpointer=saver)

One writer per thread

A checkpoint makes a thread resumable, not exclusive. Two workers can load the same thread and both continue from the same step, which produces two copies of every side effect and a state that depends on which write landed last. LangGraph does not lock a thread across processes for you, and the second worker does not know it is second.

Take a lease before resuming: a row keyed by thread id with an owner and an expiry, claimed in one conditional update. The worker renews it while it runs, and a crashed worker’s lease expires so someone else can take over. That is the same mechanism a job queue uses, and it belongs beside the checkpointer.

Ordering matters too. A resumed run should not overtake a newer one for the same thread. The lease plus a monotonically increasing run id is enough to reject a late worker that wakes up after its lease expired.

State outlives code

State outlives code. A checkpoint written by last month’s deploy is loaded by today’s, and if the state schema changed — a key renamed, a type narrowed — the resume fails in a node that never changed. Version the state and migrate the old checkpoints, or keep the reader tolerant of missing keys.

Serialisation is part of the contract. Pick a format, keep it stable, and do not put a class instance in the state unless the deserialiser knows how to rebuild it. A dict of primitives survives a deploy; a pickled object from an old module does not. Store the serialiser version in the checkpoint metadata so a reader can refuse what it cannot parse.

Retention is a durability decision too. A finished run is worth keeping long enough to answer an incident review and no longer. Deleting checkpoints too early turns a postmortem into a guess, and keeping them too long turns a database into an archive of other people’s conversations.