46 · Checking an agent in CI: `stalwart verify`¶

stalwart verify agent.yaml is the check a build fails on. It is the proof an agent runs before its first turn (chapter 45), made into a command with an exit status, so a pull request that opens a way past a check, or adds code that turns a certificate check off, stops before it is merged. This chapter is the command in full: what it runs and where, a workflow, a passing and a failing report, and every option and exit status.

Language support

The selector understands turns written in English. Portuguese is its next grammar, and has not shipped yet; until it does, a turn written in Portuguese is not read as Portuguese.

Four steps, one on your machine¶

The code checks, on your machine. Every Python, JavaScript, TypeScript and Go file of the folder agent.yaml is in is read for network-security and credential problems, by the runtime the package carries. Any finding fails the gate. Your code never leaves the machine: only the count of findings by rule goes with the registration.
The registration. The catalogue -- every tool with its words -- is registered as a new version, with the file's rulebook: compiled into the record the tool check enforces. The rules checked here are the rules your agent runs under (chapter 45).
The hunt, on our servers. The agents' graph, the catalogue and the rulebook are sent as one hunt job (POST /v1/solves, with its quality gate on), and the bug hunter searches every route through your agents. The solver runs on Stalwart's servers and never on your machine, and nothing besides the package is installed.
The report. Each finding with its rule, its severity, the line of agent.yaml it comes from, its OWASP entries and its fix, then Gate: passed or Gate: failed. Chapter 31 reads a report line by line.

A hunt that could not run -- no route to the service, a job that ended without a findings document -- is an error with exit status 3. It is never a clean result.

What it needs¶

Your key in QLS_API_KEY, and a licence activated on the machine, as every command but stalwart licence, stalwart doctor and stalwart check needs. A CI runner is a new machine every time, so the job activates the licence first and releases its place last, whatever happened in between: a released place is free for the next run at once (chapter 38).

A GitHub Actions workflow¶

# .github/workflows/agent.yml: the shop agent checked on every pull request.
name: agent
on: [pull_request]
jobs:
  verify:
    runs-on: ubuntu-latest
    env:
      QLS_API_KEY: ${{ secrets.QLS_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - run: pip install stalwart-agents
      - run: stalwart licence activate          # a runner is a new machine every time
      - run: stalwart verify agent.yaml --replay --keep findings/
      - if: always()                            # the place is released whatever happened above
        run: stalwart licence release
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: findings
          path: findings/

QLS_API_KEY is a repository secret holding a key of your account (chapter 5). The release step and the upload carry if: always(), so they run after a failed check too: the place is freed, and the findings --keep wrote are kept with the run.

A passing report¶

The shop agent of chapter 30, checked as the workflow checks it:

$ stalwart verify agent.yaml
stalwart verify agent.yaml

Registered as catalogue version 1, with the rulebook policy.co: the tool check enforces the rules verified here.
Lints at registration (nothing fails on a lint):
  write_not_retry_safe (warning) on charge_card: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on charge_card: a write with no result schema, so its outcome could not be reported
  compensation_not_retry_safe (warning) on charge_card: 'refund_card' undoes it and is not retry-safe
  write_without_compensation (warning) on refund_card: a write neither compensated nor marked irreversible
  write_not_retry_safe (warning) on refund_card: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on refund_card: a write with no result schema, so its outcome could not be reported
  write_not_retry_safe (warning) on send_receipt: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on send_receipt: a write with no result schema, so its outcome could not be reported
  compensation_out_of_reach (warning) on charge_card: agent 'desk' can call it but not 'refund_card', which undoes it, so a run of that agent cannot undo it

Code checks: no finding in 4 file(s) (rules 4b48b0081360).

QG023 Medium on cycle:desk_model (agent.yaml:3)
  the cycle through 'desk_model' calls the model or a tool, and nothing declares a budget: the framework's recursion limit of 60 is the run's only ceiling
  OWASP LLM06:2026, ASI08:2026
  Fix: Declare a budget for the agent or the run, or add an exit the model cannot avoid -- a counter in the state, declared on the edge as exit_after.

QG105 Review on refund_card (agent.yaml:10)
  the write 'refund_card' has no compensation, is not irreversible, and runs on a path with another write
  OWASP LLM03:2026, ASI08:2026
  Fix: Declare a compensation for 'refund_card' in the catalogue, or mark it irreversible.

QG105 Review on refund_card (policy.co:19)
  the write 'refund_card' has no compensation, is not irreversible, and runs on a path with another write
  OWASP LLM03:2026, ASI08:2026
  Fix: Declare a compensation for 'refund_card' in the catalogue, or mark it irreversible.

Rules with nothing to read, so nothing raised:
  QG017: no argument is declared as code or a shell command
  QG018: no argument is declared as SQL, a query, a path, a URL or a template
  QG019: no tool is declared confidential
  QG020: no retrieval from a store declared multi-tenant
  QG021: no tool is declared to write a store declared persistent
  QG106: no tool declares a source or a server, and the catalogue declares no modules
  QG107: no agent in the catalogue declares a remote channel

Gate: passed
$ echo $?
0

The findings here are a review hotspot and a medium finding under rules that do not gate (chapter 31), so the gate passes and the command exits 0.

A failing report¶

A teammate adds a helper to the same project that fetches a carrier's tracking page, with the certificate check turned off:

"""A helper a teammate added to the shop's project: the carrier's tracking page for an order."""

import httpx


def tracking_page(order_id: str) -> str:
    """Fetch the carrier's tracking page for an order."""
    page = httpx.get(f"https://tracking.carrier.example/{order_id}", verify=False)
    return page.text

It is not a tool, and no agent calls it. The code checks read every file of the project all the same, since a tool can import any of them:

$ stalwart verify agent.yaml
stalwart verify agent.yaml

Registered as catalogue version 2, with the rulebook policy.co: the tool check enforces the rules verified here.
Lints at registration (nothing fails on a lint):
  write_not_retry_safe (warning) on charge_card: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on charge_card: a write with no result schema, so its outcome could not be reported
  compensation_not_retry_safe (warning) on charge_card: 'refund_card' undoes it and is not retry-safe
  write_without_compensation (warning) on refund_card: a write neither compensated nor marked irreversible
  write_not_retry_safe (warning) on refund_card: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on refund_card: a write with no result schema, so its outcome could not be reported
  write_not_retry_safe (warning) on send_receipt: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on send_receipt: a write with no result schema, so its outcome could not be reported
  compensation_out_of_reach (warning) on charge_card: agent 'desk' can call it but not 'refund_card', which undoes it, so a run of that agent cannot undo it

Code checks: 1 finding(s); each one fails, and only changing the code clears it.
  QC002 CWE-295 shop/tracking.py:8  This code turns TLS verification off (`verify=False`), at shop/tracking.py:8.
      shop/tracking.py:8  turns TLS verification off (`verify=False`)

QG023 Medium on cycle:desk_model (agent.yaml:3)
  the cycle through 'desk_model' calls the model or a tool, and nothing declares a budget: the framework's recursion limit of 60 is the run's only ceiling
  OWASP LLM06:2026, ASI08:2026
  Fix: Declare a budget for the agent or the run, or add an exit the model cannot avoid -- a counter in the state, declared on the edge as exit_after.

QG105 Review on refund_card (agent.yaml:10)
  the write 'refund_card' has no compensation, is not irreversible, and runs on a path with another write
  OWASP LLM03:2026, ASI08:2026
  Fix: Declare a compensation for 'refund_card' in the catalogue, or mark it irreversible.

QG105 Review on refund_card (policy.co:19)
  the write 'refund_card' has no compensation, is not irreversible, and runs on a path with another write
  OWASP LLM03:2026, ASI08:2026
  Fix: Declare a compensation for 'refund_card' in the catalogue, or mark it irreversible.

Rules with nothing to read, so nothing raised:
  QG017: no argument is declared as code or a shell command
  QG018: no argument is declared as SQL, a query, a path, a URL or a template
  QG019: no tool is declared confidential
  QG020: no retrieval from a store declared multi-tenant
  QG021: no tool is declared to write a store declared persistent
  QG106: no tool declares a source or a server, and the catalogue declares no modules
  QG107: no agent in the catalogue declares a remote channel

Gate: failed
$ echo $?
1

QC002 names the file and the line, and the gate fails with exit status 1. Removing verify=False clears it; nothing else does. A code-check finding has no warning level, no comment that waives it and no option that skips it.

The same check from Python¶

For a pipeline that is not a shell, verify is the same check, and render the same report:

from stalwart_agents import render, verify

result = verify("agent.yaml")
print(render(result), end="")
print("exit status:", result.exit_code)      # 0 the gate passed, 1 it failed

stalwart verify agent.yaml

Registered as catalogue version 3, with the rulebook policy.co: the tool check enforces the rules verified here.
Lints at registration (nothing fails on a lint):
  write_not_retry_safe (warning) on charge_card: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on charge_card: a write with no result schema, so its outcome could not be reported
  compensation_not_retry_safe (warning) on charge_card: 'refund_card' undoes it and is not retry-safe
  write_without_compensation (warning) on refund_card: a write neither compensated nor marked irreversible
  write_not_retry_safe (warning) on refund_card: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on refund_card: a write with no result schema, so its outcome could not be reported
  write_not_retry_safe (warning) on send_receipt: a write with no idempotency, so an ambiguous failure goes to a person
  write_without_result_schema (warning) on send_receipt: a write with no result schema, so its outcome could not be reported
  compensation_out_of_reach (warning) on charge_card: agent 'desk' can call it but not 'refund_card', which undoes it, so a run of that agent cannot undo it

Code checks: no finding in 4 file(s) (rules 4b48b0081360).

QG023 Medium on cycle:desk_model (agent.yaml:3)
  the cycle through 'desk_model' calls the model or a tool, and nothing declares a budget: the framework's recursion limit of 60 is the run's only ceiling
  OWASP LLM06:2026, ASI08:2026
  Fix: Declare a budget for the agent or the run, or add an exit the model cannot avoid -- a counter in the state, declared on the edge as exit_after.

QG105 Review on refund_card (agent.yaml:10)
  the write 'refund_card' has no compensation, is not irreversible, and runs on a path with another write
  OWASP LLM03:2026, ASI08:2026
  Fix: Declare a compensation for 'refund_card' in the catalogue, or mark it irreversible.

QG105 Review on refund_card (policy.co:19)
  the write 'refund_card' has no compensation, is not irreversible, and runs on a path with another write
  OWASP LLM03:2026, ASI08:2026
  Fix: Declare a compensation for 'refund_card' in the catalogue, or mark it irreversible.

Rules with nothing to read, so nothing raised:
  QG017: no argument is declared as code or a shell command
  QG018: no argument is declared as SQL, a query, a path, a URL or a template
  QG019: no tool is declared confidential
  QG020: no retrieval from a store declared multi-tenant
  QG021: no tool is declared to write a store declared persistent
  QG106: no tool declares a source or a server, and the catalogue declares no modules
  QG107: no agent in the catalogue declares a remote channel

Gate: passed
exit status: 0

result.exit_code is what the command exits with: 0 where the gate passed, 1 where it failed. A file that cannot be read, a missing key or licence, and a hunt that could not run are raised as AgentFileError, LicenceRequired and HunterUnavailable (chapter 30).

The options¶

Generated from the command's own parser:

Option	What the product then does
`--keep`	writes the export, the catalogue and the findings into the directory
`--dependencies`	the lock file of the packages you install (`requirements.txt`, `pyproject.toml`, `uv.lock` or `poetry.lock`), checked against the advisory registry
`--replay`	replays each graph finding's witness on your agents, every tool a stub; a finding that does not reproduce fails the gate
`--export-only`	writes the agents' graph, as the verifier reads it, to the file, and stops

--keep DIR writes agent.graph.json, agent.catalogue.json and agent.findings.json; stalwart redteam reads the last (chapter 33). --export-only FILE registers nothing and hunts nothing.

The exit statuses¶

Generated from the command itself:

Status	Means
`0`	done: for `verify`, the gate held; for `redteam`, nothing was realised
`1`	the gate failed, or a campaign realised an attack
`2`	the file or the command could not be read
`3`	a checker or a service needed could not be reached, so nothing was checked

For verify, 2 is a file that cannot be read, or no key or no licence; 3 is a hunt or a code check that could not run, so nothing was checked.

The limits¶

Chapter 45's limits hold here (chapter 45): the model is read as choosing anything, your own functions as able to do anything, and a clean hunt holds to the depth it states (chapter 31).
A hunt that could not run is never a clean result. The hunt is the service's, so a build with no route to api.stalwart.it cannot pass the gate: it exits 3, never 0.
The code checks read network security and credential handling, in Python, JavaScript, TypeScript and Go, and nothing else about what your code does.

46 · Checking an agent in CI: stalwart verify¶