34 · When a run fails: undo, retry and rewind¶

A run that writes can fail half-way: the card is charged, the CRM updated, and then the provisioning service is down. With runs: atomic, the default, the run undoes what it did, last write first, and gives one reply that says what failed, what was undone and what could not be. This chapter is how you declare an undo, what a failed run does and in which order, and what happens when nobody knows whether a write took effect, or an undo fails.

Language support

The selector understands turns written in English. Portuguese is its next grammar, and has not shipped yet; until it does, a turn written in Portuguese is not read as Portuguese.

Declaring an undo¶

A write names the tool that undoes it:

@tool(writes=True, undo="refund_card")

In agent.yaml, undo: refund_card under the tool's name says the same. The undo is a tool like any other: it must be one of the same agent's tools, and it is checked before it runs.

Each argument of the undo is bound to the write it undoes: $args.<name> is an argument of the write, and $result.<field> a field of what the write returned. Before an undo runs, the check verifies that each bound argument came from that step, that the step ran, and that it was not undone already (chapter 32). So an undo refunds the charge this run made, with its amount, and never an amount the model wrote.

Irreversible steps¶

A write nothing can undo says so: @tool(writes=True, irreversible=True), such as sending a receipt. Every write is confirmed by a person before it runs (chapter 32), and an irreversible one is confirmed with a warning:

the interface shows Nothing can undo this step once it runs. under it (chapter 35);
in a run whose earlier writes would be undone if a later step failed, the question also ends: Nothing undoes this step, and the steps after it cannot undo it.

The person approves it knowing that if the run fails later, this step stays done. A rollback leaves it in place and lists it as not undone. Before launch, QG013 finds a route where an irreversible step can come before steps that would be undone, without its confirmation saying so.

The lints¶

Registration reads every write for what recovery needs, and says what is missing. None fails anything:

Lint	What registration says (as printed)
`write_without_compensation`	a write neither compensated nor marked irreversible
`write_not_retry_safe`	a write with no idempotency, so an ambiguous failure goes to a person
`irreversible_without_confirmation`	an irreversible operation that asks for no confirmation
`compensation_unknown_tool`	the compensation names '`<undo>`', which the catalogue lacks
`compensation_not_retry_safe`	'`<undo>`' undoes it and is not retry-safe
`compensation_unbound`	`<argument>` <- `<from>`
`compensation_cycle`	its compensations return to '`<at>`'
`compensation_out_of_reach`	agent '`<agent_name>`' can call it but not '`<undo>`', which undoes it, so a run of that agent cannot undo it

A write with neither an undo nor irreversible is also QG105 at stalwart verify, on a route with another write: a failed run cannot undo it, and nothing says so.

What a failed run does¶

A run fails when a tool raises past its retries, when the check refuses a later step of a run that has already written, when a person answers cancel all, or when a person says a write of unknown outcome did not happen. Then, in this order:

The model is not called again. The run goes to its rollback; nothing more is asked of any model.
The writes are undone, last first. Each undo is checked as the agent whose write it undoes would be checked, with the step it undoes named, and is not confirmed again: the person approved the write. The check and the sending are separate steps, so what the check allowed is recorded before anything is sent.
Irreversible writes are skipped, and listed.
The run is rewound once nothing remains to undo (below), and the person gets one reply.

The failed order¶

The shop's desk grown by a CRM update and a provisioning step, each write with its undo, and a provisioning service that is down:

# The shop's desk, grown: an order is charged, marked paid in the CRM, and its account provisioned.
agents:
  desk:
    model: "anthropic:claude-sonnet-5-5"
    tools: [get_order, charge_card, refund_card, update_crm, restore_crm, provision_account, send_receipt]
    on_unread: refuse
tools:
  - python: shop.tools
rulebook: ./policy.co
runs: atomic                               # a failed run is undone, last write first

desk = Agent.from_file("agent.yaml")


def order(turn, user):
    reply = desk.chat(turn, user=user)
    while reply.waiting and reply.waiting[0].kind == "confirm":
        reply = desk.confirm(user, "approve")       # the person approves each write as it comes
    return reply


reply = order("charge order 4411 40.00, then update the crm for order 4411, "
              "then provision the account for order 4411", "u-17")
print(reply.text)
print("The tools ran in this order:")
for number, name in enumerate(CALLS, 1):
    print(f"{number}. {name}")

Then the same order with a receipt sent before provisioning:

CALLS.clear()
reply = order("charge order 4411 40.00, then send the receipt for order 4411, "
              "then provision the account for order 4411", "u-18")
print(reply.text)

This run stopped: provision_account failed (RuntimeError). Undone, latest first: update_crm (by restore_crm), charge_card (by refund_card).
The tools ran in this order:
1. charge_card
2. update_crm
3. provision_account
4. restore_crm
5. refund_card
This run stopped: provision_account failed (RuntimeError). Undone, latest first: charge_card (by refund_card). Not undoable, and left as they are: send_receipt.

In the first run the tools ran in this order, as the example printed it and as it asserts:

charge_card
update_crm
provision_account
restore_crm
refund_card

The CRM is restored before the card is refunded: last write first. The run's three steps came from the person's one turn, read by the selector; the model was never called, and the person got one reply. In the second run the charge is refunded and the receipt is listed as not undone: it was confirmed as irreversible, and it stays sent.

Skip this step and cancel all¶

A confirmation takes three answers (chapter 35):

approve: the call runs, with exactly the arguments shown;
skip: the call does not run, its result says it was declined, and the run goes on without it;
cancel all: the run fails, as a person cancelling it, and what it already did is undone as above.

Retries and their keys¶

Every write carries an idempotency key: SHA-256 over the session, the step, the tool and the arguments, so the same call at the same step of the same session always has the same key. A retry, a resumed run and a re-sent undo all carry the key the first try carried, so a tool that honours keys does the thing once.

A call is recorded in flight before it is sent. A read, or a write marked retry_safe=True, is tried again under the same key -- at most twice more -- when it fails in a way that leaves its effect in doubt, such as a timeout or a lost connection. Any other write is sent once.

An outcome nobody knows¶

A write that is not retry-safe can fail in a way that leaves its effect in doubt: the call was sent and no answer came back. Sending it again could charge twice, and assuming it failed could leave a charge nobody undoes, so the run does neither. It asks a person: Did charge_card take effect? The call was sent and no answer came back. The answers are happened, and the run goes on as if the call ran, or did not happen, and the run fails and is undone.

A process that stops mid-call finds the in-flight record when the run resumes, on any worker. A retry-safe call is sent again under its key; any other is the same question: Did charge_card take effect? It was being sent when the run stopped.

An undo that fails¶

An undo that raises is tried again under its own key, three more times, waiting longer between tries. Then the rollback stops: undoing the writes below it out of order could leave things worse than either state. The run is stuck, and a person is asked: Undoing <tool> failed after <tries> tries (<error>). <n> step(s) still need undoing. The answers:

retry: the undo is sent again, under its key;
done by hand: someone undid it outside the agent; the rollback goes on with the next write;
abandon: the rollback stops here, and the reply lists what still needs undoing.

The rewind¶

Once nothing remains to undo, the run's own messages -- everything after the person's turn that started it -- are taken out of what the model reads next, and one summary reply is added. The next turn does not read an error trace, or a marked e-mail the failed run read.

The rewind waits for every undo. While any undo remains, or the run is stuck, it is refused (rewind_with_pending_compensation): forgetting what still needs undoing is the one mistake a rewind must not make.
The record is never rewound. The session's history keeps every call, every undo and every outcome, so an order rule or a cap still sees that a charge ran and was refunded. The memory is rewound; the record is not.

An operator can undo a thread's run from outside it, with the operator token (chapter 35).

The limits¶

An irreversible step is not undone. It is confirmed with a warning, and listed as not undone.
A run compensates; it does not lock. Between a write and its undo, the charge is real: anyone who looks sees it, and anything that acts on it acts. An undo is a second write, not a step back in time.
The rewind waits for every undo, and the record is never rewound.
The undo is your function, and does what it does. The product decides when it is called and with which arguments, and checks the call; whether it really refunds the card is your code's.
A write of unknown outcome is decided by a person; the product does not guess.