Why your long-running AI agent loses work when the box restarts

Durable execution means an AI agent's multi-step run survives a process restart: every completed step is saved before the next one starts, so a crash, a deploy, or a mid-run provider outage resumes from the last checkpoint instead of starting over from zero. For a long-running agent that provisions accounts, charges cards, or sends email, that persistence is the line between exactly-once side effects and a double-charged customer. If your agent does real work over minutes, treat the run as a durable workflow, not an in-memory loop.

Most agent frameworks are built for the demo case: one request, a few tool calls, a response in a couple of seconds. The moment an agent runs long enough to span a deploy, or performs side effects that cannot be safely repeated, the in-memory loop becomes a liability. This post covers the concrete pattern for fixing it without adopting a heavy platform on day one.

What durable execution means for an agent

A durable execution engine records the outcome of each completed step in an append-only history before moving to the next step. If the process dies, the run is rehydrated from that history: finished steps are not re-run, and the agent continues from the first step that never completed. The idea is old. Payment pipelines and account provisioning have used durable workflows for years, because they mix long waits with side effects that must happen once. What is new in 2026 is applying it to agent loops, where a model decides the control flow rather than hard-coded logic.

The unit that gets checkpointed is a step: a single LLM call, a single tool call, or a single decision. An agent run is then a sequence of steps, and durability is the guarantee that the sequence can be paused and resumed at any boundary without losing state or repeating work.

Why in-memory agent loops break in production

An agent loop that keeps its state in local variables has four failure modes that only show up under real load.

A restart erases the run

Deploys, autoscaling, out-of-memory kills, and spot-instance reclaims all restart the process. An agent that is four steps into a seven-step run has that progress in memory only. When the box comes back, the run is gone. For a short request that is a retry. For a four-minute provisioning workflow, it is a customer left half-onboarded with no record of where the process stopped.

Retries repeat side effects

The naive fix for a lost run is to retry it. That is safe only if every step is idempotent, and most are not. Re-running a workflow that already created a Stripe subscription creates a second one. Re-sending a batch of welcome emails spams the customer. This is the same class of danger covered in silent failures in production AI agents: the retry looks successful, the trace is green, and the damage is a duplicate charge nobody notices for a week.

One mid-run outage wastes the whole run

A multi-minute run makes several model and tool calls. If a provider returns a burst of 429s or a tool times out on step five, an in-memory loop usually aborts the entire run. All the work in the first four steps, some of it expensive, is discarded. Client-side retry and failover, the subject of LLM API resilience for SaaS features, keep an individual call alive, but they do not save the progress of the surrounding run when the process itself goes down.

Stuck runs are invisible and unrecoverable

When state lives in memory, there is nothing to inspect. A run that hangs waiting on a slow tool cannot be listed, paused, or resumed by an operator, because it does not exist anywhere except in one process's heap. This is the operational blind spot that observability versus evals warns about, applied to run lifecycle rather than output quality.

The pattern: model the run as durable steps

The fix is to give each step a persisted record, make its side effects safe to repeat, and separate the parts that must be deterministic on replay from the parts that are not.

Checkpoint every step before the next one runs

Write the output of each step to durable storage before starting the next step. A minimal version is two tables: a runs table (run id, status, current step) and a steps table (run id, step id, status, output, idempotency key). Before executing a step, the loop checks whether that step id already has a completed row. If it does, it reads the stored output and skips execution. That single check is what makes a restart resume instead of repeat.

Make every side-effecting step idempotent

Any step that changes the outside world needs an idempotency key derived from the run id and step id, so a replay after a crash is a no-op instead of a duplicate. Stripe, most payment APIs, and well-designed internal endpoints accept an idempotency key for exactly this. Building tools that carry these keys and return machine-actionable results is the discipline described in designing tools for production AI agents. If a tool cannot be made idempotent, wrap it: record intent before the call, mark completion after, and check that record on replay.

Replay the recorded model output, never re-sample it

This is the subtlety generic durable-execution guidance misses, and the one that bites teams first. On replay, a completed LLM step must return its stored output, not call the model again. A model call is non-deterministic: re-sampling it produces different text, which sends the resumed run down a different branch than the original took, and the checkpoints stop matching reality. Treat the model output as an effect to be recorded once and read back, the same as a tool call. Deterministic orchestration (the control flow deciding which step is next) stays in code that can be replayed; non-deterministic work (model and tool calls) is checkpointed and read from history on resume.

Use durable timers for waits and backoff

Long agents wait: for a human approval, for a rate-limit window, for an external job. A durable timer lets a run sleep for minutes or hours without holding a process or a thread. When the wait ends, the run is rehydrated and continues. This is what makes human-in-the-loop steps and exponential backoff practical without a worker sitting idle for the duration.

A worked example: the onboarding agent

Consider a SaaS onboarding agent (numbers are illustrative). When an enterprise customer signs up, it runs a seven-step workflow over about four minutes: provision a workspace, import the customer's data (rate-limited, roughly ninety seconds), generate a starter configuration with an LLM call, create a Stripe subscription, send three role-specific welcome emails, post a message to the customer's Slack, and write an audit record.

Run this as an in-memory loop and a deploy at minute three does real damage. The optimistic path drops the customer after step three, provisioned but never billed or welcomed. The pessimistic path, a blanket retry, creates a second Stripe subscription and re-sends the emails, so the customer's first impression is a duplicate invoice and inbox spam.

Run it durably and the same deploy is a non-event. Each step is checkpointed, so the restart resumes at step four rather than step one. The generated configuration is read from its checkpoint instead of being regenerated, which matters because a fresh model call would produce a different config than the one the earlier steps already assumed. The Stripe call carries an idempotency key and the emails deduplicate on run id plus email type, so the resumed attempts are no-ops. The run finishes once, with exactly-once side effects, and an operator can list and inspect it.

Build versus buy

A workflow engine gives you this off the shelf. Temporal, Inngest, DBOS, and Restate all provide durable steps, retries, and timers, differing mainly in whether state lives in a dedicated service or in your own Postgres. They are the right call when you have many long workflows, need multi-region recovery, or want event-history replay without writing it yourself.

You do not have to start there. The lightweight version is the runs and steps tables described above, checked before each step, plus idempotency keys on side effects. That handles a single important workflow with a few hundred lines of code and no new infrastructure. Adopt an engine when the number of durable workflows, or the recovery guarantees you need, outgrows the hand-rolled version.

When you do not need durable execution

Durability is weight, and not every agent earns it. A single synchronous model call that returns in a couple of seconds does not need checkpointing. A read-only agent with no side effects can simply be retried end to end, because repeating it costs tokens and nothing else. A prototype you are still shaping should stay simple until the flow stabilizes.

You need durable execution when three things line up: a run spans minutes rather than seconds, it performs side effects that cannot be safely repeated, or it includes long waits for humans or external systems. That is also roughly the point where a reasoning model can no longer collapse the whole task into one call, a tradeoff covered in when reasoning models collapse agent chains. If your run clears that bar, checkpointing is not gold-plating, it is the difference between a feature you can operate and one that quietly corrupts customer data on the next deploy. Getting these guardrails in before launch is part of learning to ship AI features without breaking production, and the broader set of practices lives in our AI engineering playbook.

Frequently asked questions

Is durable execution the same as retrying a failed agent run?

No. A blanket retry re-runs the whole workflow from the start, which repeats any side effects the first attempt already committed. Durable execution resumes from the last completed step and skips finished work, so a run reaches its side effects exactly once even across a crash.

How is this different from managing agent handoffs?

Handoff reliability is about the seams between agents passing control cleanly, covered in multi-agent handoff reliability. Durable execution is about a single run surviving infrastructure failures. They are complementary: a multi-agent system still needs each run to be durable underneath the handoffs.

Do I need Temporal to get durable execution?

No. A dedicated engine is convenient at scale, but the core guarantee comes from persisting each step's output before the next runs and making side effects idempotent. A runs table and a steps table in your existing database deliver that for a single workflow without new infrastructure.

What is the hardest part of making an agent run durable?

Handling model non-determinism on replay. A completed LLM step must return its recorded output rather than call the model again, because re-sampling would send the resumed run down a different path than the original. Record model and tool outputs once and read them back on resume.