36 · Seeing what happened

When a run does something you did not expect, you want to see it step by step: what the router read, what each check decided and why, which call ran, what was undone. stalwart trace shows a run as a timeline, from the storage your agent already keeps or from the spans a tracer recorded, and every decision on it names the rule that made it.

Language support

The selector understands turns written in English. Portuguese is its next grammar, and has not shipped yet; until it does, a turn written in Portuguese is not read as Portuguese.

A run as a timeline

stalwart trace has three subcommands:

• stalwart trace show: the timeline in the terminal, one block per step, each check with its answer and its reasons, each pause for a person, each undo and the step it undoes.
• stalwart trace html ... -o run.html: the same timeline as one self-contained HTML file, with no script and no request to anywhere, for a ticket or an audit.
• stalwart trace diff a.jsonl b.jsonl: where two runs of the same input went different ways: the first step whose decisions differ, and what only each run did there.

show and html read either a spans file or, with --thread and --store, your agent's storage.

The same from storage alone

A run your agent kept in storage: can be read back with nothing recorded beforehand. E6's failed order (chapter 34), with the shop's storage on disk:

desk = Agent.from_file("agent.yaml")          # storage: sqlite:///runs.db

reply = desk.chat("charge order 4411 40.00, then update the crm for order 4411, "
                  "then provision the account for order 4411", user="u-17")
while reply.waiting:
    reply = desk.confirm("u-17", "approve")   # each write approved as it comes
print(reply.text)
desk.close()

This run stopped: provision_account failed (RuntimeError). Undone, latest first: update_crm (by restore_crm), charge_card (by refund_card).

The conversation is desk/u-17: the agent's name and the person's. Read back from the storage:

$ stalwart trace show --thread desk/u-17 --store sqlite:///runs.db
Run: thread desk/u-17
step -1
  node      input
step 0
  node      run
step 1
  node      run
step 2
  node      run
  check     charge_card -> confirm
  interrupt charge_card [write] (confirm)
step 3
  node      run
  check     charge_card -> allow
step 4
  node      run
  call      charge_card
step 5
  node      run
step 6
  node      run
step 7
  node      run
  check     update_crm -> confirm
  interrupt update_crm [write] (confirm)
step 8
  node      run
  check     update_crm -> allow
step 9
  node      run
  call      update_crm
step 10
  node      run
step 11
  node      run
step 12
  node      run
  check     provision_account -> confirm
  interrupt provision_account [write] (confirm)
step 13
  node      run
  check     provision_account -> allow
step 14
  node      run
  call      provision_account
step 15
  node      run
  undo      restore_crm -> allow undoes step 3
step 16
  node      run
step 17
  node      run
  undo      refund_card -> allow undoes step 1
step 18
  node      run
step 19
  node      run
step 20
  node      run

Each write was checked and asked about (confirm, [write]), approved (allow), and run; provision_account failed at step 14, and the two undos follow, each checked, latest first: restore_crm undoes the CRM update (history step 3) and refund_card the charge (history step 1). A step with nothing under it is one that only moved the run on.

Spans for any OpenTelemetry tool

A tracer records one span around each of these, generated from the package's tracer:

Span	Opened around
stalwart.node	each step of a run
stalwart.plan	the router reading a turn
stalwart.tool_check	the tool check deciding a call
stalwart.tool	a tool running
stalwart.compensate	an undo running
stalwart.rollback	a failed run being undone
stalwart.rewind	a run rewound
stalwart.interrupt	a run pausing for a person
stalwart.resume	a run resuming on the person's answer

A tool check's span carries its decision, as these attributes and as a stalwart.decision event:

Attribute	Holds
gen_ai.operation.name	execute_tool, as OpenTelemetry's GenAI conventions name a tool call
gen_ai.tool.name	the tool
gen_ai.tool.call.id	the call's id
stalwart.outcome	the check's answer: allow, confirm or refuse
stalwart.decided_by	which step of the check decided
stalwart.rule	the rule, flow or reason behind the answer
stalwart.reasons	every reason the check gave
stalwart.labels	the marks the call carried
stalwart.args_digest	the first 16 hex characters of the arguments' digest, never their values
stalwart.history_token	a short digest of the session's history token, never the token

gen_ai.* are OpenTelemetry's GenAI conventions, so any tool that reads them shows the calls; stalwart.* are ours. With pip install "stalwart-agents[otel]", the spans go to OpenTelemetry's tracer provider, and from it to whichever exporter you configured. With no exporter at all, a JSON-lines tracer writes them to a file, which stalwart trace show and diff read.

Content is off by default

No argument value, message, result or turn is put on a span unless the tracer is made with capture_content=True. Where content is off, a 16-character digest stands in for each, so two spans can be compared without either holding the value. Turning it on adds exactly these attributes:

Attribute	Content off (the default)	Content on (capture_content=True)
stalwart.arguments	stalwart.arguments_digest	the call's arguments, as values
stalwart.result	stalwart.result_digest	the tool's result
stalwart.message	stalwart.message_digest	a message of the conversation
stalwart.turn	stalwart.turn_digest	the person's turn, as typed

They go where every other attribute goes: to the exporters of the tracer provider you gave it, or to the JSON-lines file you named. A tracer sends nothing to us, either way: a trace is yours.

The state validator

The state a run keeps -- its messages, its history, what is waiting for a person, what to undo -- must stay consistent, and a step your team writes can break it. The validator names each break:

Violation	Means
call_without_result	a call the model asked for has no result, and the conversation went on past it
result_without_call	a result answers no call in the conversation
results_out_of_order	a step's results do not follow it in the order of its calls
duplicate_message_id	two messages share an id
message_after_interrupt	a message was added while a pause for a person was still open
stack_step_not_in_history	the list of what to undo names a step the history does not have
history_broken	the history's steps are not 0, 1, 2, ..., or it has entries and no token
history_token_invalid	the history's token does not verify: an entry was edited, dropped or reordered outside the run
label_on_removed_message	a mark names a message the conversation no longer holds

stalwart trace show and html mark the first break in a run read from storage with <<< FIRST VIOLATION, at the step that caused it, so the timeline shows where the state went wrong rather than where it failed later.

Following a decision to its rule

Each check on a timeline names its answer and who decided it, and its span names the rule:

• stalwart.decided_by is the step of the tool check that decided (chapter 32);
• stalwart.rule is the rule, flow or reason: a rulebook's refusal names its flow (chapter 45), an order rule names the binding it broke;
• stalwart.reasons lists every reason a confirmation gave.

If you already use LangGraph

The spans are opened by the run a tracer is given: desk.graph(tracer=JsonLinesTracer("run.jsonl")) compiles the agent's graph with stalwart_agents.langchain.core.tracing.JsonLinesTracer, and stalwart_agents.langchain.otel.OtelTracer opens the same spans in OpenTelemetry, with NodeSpans as the callback that opens one per node. validate_state(state) and the debug wrapper validating(graph), which stops at the first violation with the node and step, are in stalwart_agents.langchain.graph. Chapter 27 has them in LangGraph's own terms.

The limits

• A timeline read from storage shows what the storage kept: the checks, the calls, the pauses and the undos, not the spans' timings, and not what happened inside your tools.
• Captured content goes wherever your exporter sends it. Turn it on only where that destination may hold your users' words and your tools' results.