Engineering · Intermediate · 14 min read · Updated 19 November 2026

Reliable AI Agent Execution: Checkpoints and Idempotency in 42 Lines of Python

Надёжное выполнение AI-агента: чекпоинты и идемпотентность за 42 строки Python
Temporary fallback cover; replace in editorial pass.

An agent can crash in the most expensive place: after a payment succeeds, but before the workflow records that success or sends confirmation. A tiny SQLite journal plus a stable idempotency key turns that ambiguous restart into a controlled recovery path.

The core rule: record durable progress before moving on, and make every external operation safe to repeat. This article builds that rule around a concrete payment-and-confirmation workflow using only Python’s standard library.

The failure we need to survive

Imagine an agent handling an order:

  1. Validate the order.
  2. Charge the customer.
  3. Send a confirmation.

The process completes step 2 at the payment provider. Then the worker loses power before step 3. On restart, a naïve loop sees no local record of the payment and charges again.

This is the boundary between an ordinary function call and a side effect: the payment changes a system outside the worker. A retry can therefore create a duplicate charge even when the agent’s reasoning is correct.

We will use a checkpoint to record completed work and an idempotency key to give the payment provider a stable identity for the operation.

What the small design guarantees

The example has three deliberately narrow guarantees:

It does not magically make two unrelated systems commit atomically. The payment provider must support idempotent requests, and confirmation delivery must have its own retry or deduplication policy.

The 42-line Python journal

Save the following as journal_demo.py. The SQLite database is the durable local record. The payment function is intentionally a fake provider: it returns a deterministic result and stores it in the same database so the example can run without credentials or network access.

import hashlib
import sqlite3

DB = "agent_steps.db"

def connect():
    db = sqlite3.connect(DB)
    db.execute("PRAGMA journal_mode=WAL")
    db.execute("""CREATE TABLE IF NOT EXISTS steps (
        run_id TEXT, name TEXT, result TEXT,
        PRIMARY KEY (run_id, name)
    )""")
    db.execute("""CREATE TABLE IF NOT EXISTS payments (
        key TEXT PRIMARY KEY, order_id TEXT, status TEXT
    )""")
    return db

def done(db, run_id, name):
    row = db.execute(
        "SELECT result FROM steps WHERE run_id=? AND name=?",
        (run_id, name)).fetchone()
    return row[0] if row else None

def save(db, run_id, name, result):
    db.execute("INSERT OR REPLACE INTO steps VALUES (?, ?, ?)",
               (run_id, name, result))
    db.commit()

def payment(db, order_id, key):
    row = db.execute("SELECT status FROM payments WHERE key=?",
                     (key,)).fetchone()
    if row:
        return row[0]
    db.execute("INSERT INTO payments VALUES (?, ?, ?)",
               (key, order_id, "paid"))
    db.commit()
    return "paid"

def run(order_id):
    run_id = order_id
    db = connect()
    if not done(db, run_id, "validated"):
        save(db, run_id, "validated", "ok")
    key = hashlib.sha256(run_id.encode()).hexdigest()
    if not done(db, run_id, "paid"):
        status = payment(db, order_id, key)
        save(db, run_id, "paid", status)
    if not done(db, run_id, "confirmed"):
        print(f"confirmation for {order_id}")
        save(db, run_id, "confirmed", "sent")
    db.close()

if __name__ == "__main__":
    run("order-example-001")

The line count is intentionally small, but the ordering matters:

  1. The payment key is derived from the stable order identity, not from the current process.
  2. The external operation happens before its local completion checkpoint.
  3. If the process crashes after payment but before save(..., "paid", ...), the next run calls payment again with the same key.
  4. The provider-side record returns the existing result instead of creating a second payment.

Run it reproducibly

Use a fresh directory and a Python 3 interpreter:

mkdir agent-journal-demo
cd agent-journal-demo
python3 --version
python3 journal_demo.py
python3 journal_demo.py
sqlite3 agent_steps.db \
  'select run_id, name, result from steps order by name;'
sqlite3 agent_steps.db \
  'select key, order_id, status from payments;'

On the first run, the program prints a confirmation. On the second run, the journal marks all three steps complete, so the confirmation code is skipped. These are expected outcomes from reading the code; they are not a claim that this article’s environment executed the commands.

Simulating the dangerous crash window

To model the real problem, temporarily insert this line immediately after the call to payment:

raise RuntimeError("crash after external payment")

Run the program once, remove the line, and run it again. The first run creates the payment record but never writes the paid checkpoint. The second run reaches payment again, finds the same key, and receives the existing paid result before writing the missing checkpoint and sending confirmation.

This local simulation demonstrates the recovery shape, not the behavior of a real payment API. A production integration must confirm the provider’s exact idempotency semantics, key retention period, request timeout behavior, and response handling from its own documentation and tests.

Why a checkpoint is not enough

Suppose the journal contains only step names:

validated
paid
confirmed

That prevents some duplicate work, but it cannot resolve uncertainty when the worker crashes before writing paid. The journal says “not paid,” while the provider may say “paid.” The idempotency key bridges that gap by making the retry refer to the same logical operation.

For real integrations, derive the key from an immutable business identity such as order_id + operation. Do not use a random value generated on every retry. Also avoid putting personal data or payment details into the key; the example uses a one-way hash of a synthetic order identifier.

Failure cases to handle explicitly

Payment request times out

A timeout is not proof that the payment failed. Retry with the same key, then interpret the provider’s response. If the provider has a status lookup endpoint, use it with the same operation identity.

Confirmation fails

Do not charge again. Keep the paid checkpoint and retry only confirmation. In a larger system, store a confirmation delivery identifier and let the mail or messaging provider deduplicate it too.

Two workers process one order

The sample journal is intentionally minimal and does not implement leases or ownership. Concurrent workers can race between reading a missing step and executing it. Add a unique run claim, a lease with expiry, or a queue-level single-consumer guarantee before using this pattern concurrently.

The business input changes

Do not reuse an old key for a materially different payment amount or currency. Bind the operation identity to the versioned request, and reject a key if the provider reports a parameter mismatch.

The local database is lost

A SQLite file is durable only as long as its storage is durable. Back it up or place it on reliable persistent storage. If the journal disappears, recover by querying the external provider using the business operation identity rather than blindly retrying.

Production checklist

Verification without overclaiming

You can verify the design locally by inspecting the two tables before and after a simulated crash:

sqlite3 agent_steps.db \
  'select * from steps order by run_id, name;'
sqlite3 agent_steps.db \
  'select * from payments;'

The expected invariant is simple: one payment row per idempotency key, and at most one completion row for each (run_id, step) pair. Whether a real provider honors the same retry behavior must be verified against that provider’s contract; this article does not test or certify any provider.

Limitations

This pattern provides at-least-once execution of unfinished steps with deduplication at the side-effect boundary. It does not provide universal exactly-once execution. Networks can fail after a provider commits, databases can become unavailable, and a notification service may have different deduplication rules.

For a single-process workflow or a small worker, SQLite can be a practical journal. For high concurrency, multi-region execution, long-running workflows, or strict audit requirements, use a datastore and workflow system designed for those conditions. Keep the same mental model: durable progress plus an operation identity that survives retries.

Takeaway

When an agent can cause an external change, a restart must be expected—not exceptional. Checkpoint the work you have completed, retry incomplete work, and make the retry carry the same idempotency key. That combination turns “did the payment happen?” from a guess into a queryable recovery decision.

Continue with the Agent Lab Journal guides or look up related terms in the glossary.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →