QG017 Model writes code that runs¶
| Type | Vulnerability |
| Severity | Critical; High where the tool declares sandboxed |
| Front end | a Colang 1 policy with a catalogue, a Colang 2 policy, an agent graph, and a router's plan |
| Guardrail category (Vassilev 2026) | E. Action & Execution Guardrails |
| AI RMF functions (NIST AI 100-1) | MAP, MEASURE, MANAGE |
| Why these functions | the catalogue's assignment, not the article's chain (GOVERN, MANAGE): the flow is mapped from the team's declarations, measured by a reachability question and managed by the fix; no governance rule is in question |
| OWASP | LLM10:2026 Improper Output Handling, LLM01:2026 Prompt Injection, ASI05:2026 Unexpected Code Execution (RCE) |
| Gates by default | yes |
Raised when¶
An argument the catalogue declares as code or a shell command can take a value the model wrote, or one carrying a trust label, on a route with no confirmation.
Why it matters¶
The tool runs what it is given, and what it is given here is the model's text: an instruction planted in anything the model read becomes a program the tool executes. The hunter decides whether a model-written value reaches the tool unconfirmed, not what the tool does with it.
How to fix it¶
Bound the argument in the tool's schema with an enum, a const or a pattern; or take it from the user's typed text, the host's context or an earlier tool's result; or put the call behind a confirmation -- the guarded tool node, an interrupt(), or in Colang a guard on the confirmation context. Where the tool runs code in isolation, declare it sandboxed. The fix is re-scanned.
OWASP LLM10:2026 and ASI05:2026, require the user's approval: put a person's confirmation between a model-written value and the tool that interprets it, after the bound or the origin the fix names.
OWASP entries¶
LLM10:2026 Improper Output Handling, LLM01:2026 Prompt Injection and ASI05 Unexpected Code Execution, from the OWASP Top 10 for LLM Applications 2026 and for Agentic Applications 2026. The
mapping holds because a model-written program reaching a tool that runs it is output handled as code, and the model's value is the route an injected instruction takes. The coverage is at the policy level: the rule decides the part of the risk the policy and its catalogue control, and a scan with no finding under it says that and no more.
What the finding carries¶
A location at the call site; a fingerprint over the rule, the tool, the argument or label and a digest of the flow or node; the witness as a path, naming the origin chain or the label and the absent confirmation; and a fix that is re-scanned and marked verified. A Colang finding is replayed against NeMo Guardrails; a graph finding in the team's environment with stub tools. The replay confirms the tool is called on the witness's path; the origin, label or scope that makes the call a finding is the static model's, and the finding says so. The rule fires only on a declaration the team made in the catalogue, agent.yaml or the configuration; a missing declaration raises nothing, and the scan summary lists the rule as having had nothing to read. See the catalogue for the other rules.