51 · The bug hunter as a quality gate in CI¶

A quality gate in CI works the way a team that uses SonarQube expects: a project per repository, a baseline on the default branch, a profile that says what fails, every pull request scanned and decorated, the merge blocked while the gate fails, and a person's decisions kept from one scan to the next. This chapter sets it up from the start, for GitHub, GitLab and Jenkins. Chapter 28 is the gate's conditions in full, and chapter 46 the agent check in full.

Language support

The selector reads turns written in English and in Brazilian Portuguese (pt-BR). The language is chosen, never guessed from the turn: the console's Settings page (Language the selector reads), language on a plan request, or the settings.language of your vocabulary file; with none of them, English. A Portuguese turn may be typed with its accents or without them. European Portuguese is not a grammar the selector has, and a turn in any other language is unread.

Two gates¶

	An agent: `stalwart verify`	A policy or a router: a project's scans
Checks	the agents' graph, the rulebook and your tool code (`QC`)	a guardrail policy, a rule sheet, a router built from your catalogue
Compared with	nothing: every finding is read as new	the project's baseline, the last scan of its default branch
Fails on	any `QC` finding, and the gate's conditions over the graph's findings	the gate's conditions over what is new, by the project's profile
Remembers	nothing	each finding's history and a person's decisions
Runs from	any CI, with the `stalwart` command	the GitHub Action, or the HTTP API from any CI

An agent declared in agent.yaml is checked with stalwart verify. A team that keeps a guardrail policy or a router in a repository, and wants a baseline and a review trail, keeps it in a project. A repository can have both.

1. A project and its baseline¶

POST /v1/projects with a name and the default_branch (main where none is given) makes a project. The first scan on the default branch is its baseline: every finding in it is existing, so none of them fails the gate. From then on:

a scan on the default branch moves the baseline forward, and the findings it no longer has are fixed;
a scan on any other branch reports against the baseline, as new, existing, fixed or reintroduced, and moves nothing;
a finding that was fixed and comes back is reintroduced, and the gate reads it as new.

So the default branch is scanned on every push to it, and every pull request is scanned against it.

2. The quality profile¶

Without a profile, the gate has its default conditions (chapter 28): it fails on a new Critical finding, a broken row of your own tables, a new policy that cannot ship, a finding not reproduced and an engine disagreement. A profile adds conditions, or removes them:

{
  "gate": {
    "add": ["no_new_severity_at_least:High", "no_new_rule:QG101"]
  }
}

no_new_severity_at_least:High fails on a new finding of High severity or more, and no_new_rule:QG101 on a new finding under that rule. PUT /v1/projects/{id}/profile with {"profile": ...} sets it, and {"profile": null} goes back to the default; the next scan reads it. A profile needs a key with the hunt_details entitlement (chapter 40).

3. Every pull request¶

On GitHub, the Guardrail Bug Hunter Action does every step on each run:

uploads the files its inputs name -- the policy, its rails configuration, a host's extracted model;
submits a scan of the project on the pull request's branch and commit, and follows it to the end;
uploads the scan's SARIF to code scanning, so each finding shows on the line it is about;
posts one comment on the pull request -- new, fixed and existing findings, the gate's result and a link to the project -- and updates that comment on a re-run rather than adding another;
fails the check when the gate fails.

Make that check required in the branch's protection rules, and a pull request whose gate fails cannot be merged. Run the same workflow on pushes to the default branch, so the baseline stays current.

On any other CI, the same scan is three requests to the HTTP API (chapter 18):

POST /v1/uploads for each file;
POST /v1/projects/{id}/scans with the branch, the commit and a hunt request as POST /v1/solves takes one, with the operand gate: true;
GET /v1/solves/{job_id} until the job ends: the findings document's gate holds passed, the conditions it read and the ones that failed.

Exit with a failure where passed is false, and make the job a required check of the merge request. The SARIF is among the job's artefacts, for a code-scanning view or SonarQube (below).

4. Agents on every pull request¶

stalwart verify agent.yaml exits 1 where the gate fails, 2 where the file cannot be read or the key or licence is missing, and 3 where the hunt or the code checks could not run (chapter 46). Every CI blocks a merge on a failed step, so the command is the gate. The runner is a new machine every time, so the licence is activated first and released last, whatever happened in between.

GitHub Actions:

# .github/workflows/agent.yml: the shop agent checked on every pull request.
name: agent
on: [pull_request]
jobs:
  verify:
    runs-on: ubuntu-latest
    env:
      QLS_API_KEY: ${{ secrets.QLS_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - run: pip install stalwart-agents
      - run: stalwart licence activate          # a runner is a new machine every time
      - run: stalwart verify agent.yaml --replay --keep findings/
      - if: always()                            # the place is released whatever happened above
        run: stalwart licence release
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: findings
          path: findings/

GitLab CI, as .gitlab-ci.yml, with the key in a masked CI/CD variable:

# .gitlab-ci.yml: the shop agent checked on every merge request.
verify:
  image: python:3.13
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  variables:
    QLS_API_KEY: $QLS_API_KEY                 # a masked CI/CD variable holding a key of your account
  script:
    - pip install stalwart-agents
    - stalwart licence activate               # a runner is a new machine every time
    - stalwart verify agent.yaml --replay --keep findings/
  after_script:
    - stalwart licence release                # the place is released whatever happened above
  artifacts:
    when: always
    paths: [findings/]

Under Settings > Merge requests, Pipelines must succeed blocks the merge while the job fails.

Jenkins, as a Jenkinsfile, with the key in a secret text credential named qls-api-key:

// Jenkinsfile: the shop agent checked on every change request.
pipeline {
    agent { docker { image 'python:3.13' } }
    environment {
        QLS_API_KEY = credentials('qls-api-key')   // a secret text credential holding a key of your account
    }
    stages {
        stage('verify') {
            when { changeRequest() }
            steps {
                sh 'python -m venv .venv && .venv/bin/pip install stalwart-agents'
                sh '.venv/bin/stalwart licence activate'   // an agent is a new machine every time
                sh '.venv/bin/stalwart verify agent.yaml --replay --keep findings/'
            }
        }
    }
    post {
        always {
            sh '.venv/bin/stalwart licence release || true'   // the place is released whatever happened above
            archiveArtifacts artifacts: 'findings/**', allowEmptyArchive: true
        }
    }
}

A multibranch pipeline reports the stage's result on the change request, and the repository's branch protection makes it required.

5. Triage¶

A finding's lifecycle is a person's decision about it, kept in the project and read by every later scan:

Lifecycle	Means	What the gate does
`open`	nobody has decided yet	reads it by its conditions
`confirmed`	it is a real problem, to be fixed	reads it by its conditions
`false_positive`	it is not a problem	leaves it out of the conditions a decision settles: a finding not reproduced, an engine disagreement, a broken row of your tables. A reproduced finding is not a false positive: it stays open
`accepted_risk`	it is a problem the team accepts until a date, with a justification	as for a false positive, until the date; then the finding is `open` again

A decision settles a finding where it is; it does not hide one that comes back. A finding fixed and then reintroduced is new again, and fails the conditions on new findings -- a Critical one, a floor the profile sets -- whatever was decided about it before.

PATCH /v1/projects/{id}/findings/{fingerprint} with the lifecycle, a justification and, for an accepted risk, expires as YYYY-MM-DD records the decision; an accepted risk with no date lasts 90 days. The console's Findings pages show each project's scans and findings and record the same decisions, and GET /v1/projects/{id}/findings/{fingerprint} reads one finding's whole history: when it appeared, each decision and who made it. Recording a decision and reading a finding's history need hunt_details, as the profile does. A QC finding has no lifecycle: only changing the code clears it (chapter 50).

SonarQube¶

A team that already reads its findings in SonarQube imports the scan's SARIF as external issues (sonar.sarifReportPaths), with the same rules, severities, locations and fingerprints. Keep the decisions in the project: the gate reads the project's baseline and lifecycles, not SonarQube's.

The limits¶

A gate is only as good as what it is given. A scan of a policy without the catalogue classes every action High; send the catalogue or the extracted model with it (chapter 28).
stalwart verify has no baseline. Every finding of an agent is read as new on every run; a Critical one fails the gate until it is fixed.
A hunt that could not run is never a pass. No route to the service is exit status 3, not 0.