Auditing my coding agent's guardrails: the holes I found

TL;DR: After setting up layered guardrails for my coding agent, I audited them twice the way an attacker would and found holes that every layer had missed. The delete protection covered the git command but not the same operation sent as a raw API call. A misplaced case-insensitive regex flag made the safety hook crash and silently allow everything. Server-side branch protection on private repositories turned out to need a paid plan. And months of clicking "always allow" had grown the permission allowlist to close to 800 rules, a handful of them with credentials written inline. The fixes: block every spelling of a dangerous operation, a self-test that runs in both directions after every hook change, a written map of which boundary rests on infrastructure and which on my laptop, a scripted allowlist cleanup, rotated credentials, and secret scanning on every push.

Why audit guardrails you already trust?

In an earlier post I described the four layers I run around an autonomous coding agent: a safety-checked permission mode, hard deny rules, a pre-execution hook, and a behavior instruction. Together they form the agent harness, the runtime around the model that decides which of its tool calls actually execute and with what permissions. This post is what happened when I stopped describing that harness and started trying to break it.

Two audits, about seven weeks apart, each started by something small. The first was a plain question I put to the agent: make sure you cannot delete my repository or my main branch. Answering that honestly meant listing every path to deletion, not reading the config and nodding. The second started as an annoyance: every new session opened with a dozen warnings about allow rules containing a wildcard in the middle of a command. Chasing the warnings turned into the more uncomfortable finding.

The reason to audit at all is that guardrail config drifts quietly. Rules get added by clicking a button, hook patterns get edited in a hurry, and nothing tells you when a layer stops working. A guard that fails open looks exactly like a guard that works, right up to the moment it matters.

What does each boundary actually rest on?

The most useful artifact of the first audit was a table. For every destructive outcome I cared about, it names what stops it and where that protection lives:

The best row is the first one. It relies on least privilege: the token was issued without the capability, so no local process, however clever, can do what the token cannot express. Three rows rest on nothing but a script on my machine.

I considered three ways to harden the local-only rows. Server-side rulesets that forbid deletion and non-fast-forward pushes are the only layer immune to anything running locally, and the obvious answer. On private repositories on my plan, the API answered with an upgrade prompt: it is a paid feature. The second option was to take credentials away from the agent entirely, which would also take away pushing branches, a large part of what makes it useful. The third was to keep the hook, widen it and test it relentlessly. I chose the third for now and wrote the paid tier into the notes as the real fix whenever the threat model demands a guarantee. The table stays, so I never mistake a local rule for a server-side one.

The API form that walked past the delete protection

The hook blocked deleting a remote branch and force-pushing through git. The first audit found that the host's CLI has a generic API subcommand, and a DELETE request against a branch ref went straight through. I verified it empirically: a branch was deleted that way. I had asked for that particular deletion, but nothing in the harness would have stopped it if I had not.

# illustrative shape: one operation, three spellings
git push origin --delete feature-x        # blocked from day one
gh api -X DELETE .../git/refs/heads/feature-x   # walked through
curl -X DELETE <host API> + raw token     # needs the token first

The lesson generalizes. A destructive operation has many spellings: the friendly command, the API call behind it, and a raw HTTP request carrying the token. Guarding one spelling guards nothing. The fix was to enumerate the spellings for each outcome in the table: the API subcommand with a DELETE method, any mutation of refs through the API, commands that print the token or request additional scopes, and HTTP clients sending DELETE to the host's or my cloud provider's API. Blocking the token printout matters more than it looks, because a token in hand turns every other rule into a suggestion. The hook's self-test grew by fourteen cases.

A regex flag that switched the hook off

The earlier post mentions this in one sentence; the mechanism deserves more. The hook matches commands against regular expressions in Python. An inline case-insensitive flag is allowed only at the very start of a pattern. I had placed one in the middle, after an alternation, and on recent Python versions that raises an error when the pattern compiles. The hook's error path exits with success, and success means "allow". So one edit turned the whole protection off, with no message, no failed command, nothing visible.

This is the difference between failing open (an error lets everything through) and failing closed (an error blocks everything). Failing closed sounds safer, and for some controls it is the right call; I use it for secret scanning below. For a hook that sits in front of every shell command, a bug that blocks all work creates immediate pressure to rip the hook out. Whichever way a guard fails, you need to know which, and you need something that notices.

That something is a self-test run after every change to the hook, in both directions. One set of cases must be blocked: the known bypasses, including the interpreter-wrapped command from the first post and the API form above. Another set must pass: removing a build cache directory, ordinary git work. Both directions are needed because each alone is satisfied by a broken guard. A hook that blocks everything passes the "does it block" test; a hook that blocks nothing passes the "does normal work flow" test. Only the pair tells you the guard is doing its job.

The audit also surfaced a limit I decided to keep. The hook scans the raw command string, including the body of a heredoc. Appending a paragraph of documentation that mentions a force push through a shell heredoc gets blocked. That is a false positive, but a heredoc is also a real vector, so I kept the behavior and changed the habit: file edits go through the agent's edit tool, and git commands run in their own clean call.

How do secrets end up in an agent's config?

Back to the startup warnings. The root cause was mundane. Every time you answer a permission prompt with "always allow", the harness writes the full command, verbatim, into the allowlist in the agent's settings file. After months of that, my allowlist held close to 800 rules. A few dozen contained an asterisk in the middle of a command, which came from a file glob in a path. The matcher reads that asterisk as a rule wildcard, so it also matches any options inserted at that position. Those were the warnings.

The worse part: a handful of rules contained credentials inline. A database connection string with the password in it, an API key in a request header. Each had been a one-off command I approved with the wrong button. And because I back up my agent configuration to a private repository, one of those credentials had also landed in git history. The affected credentials were rotated and the rules removed. A private repository changes nothing about that rule: a secret that reached history gets rotated at the provider.

I weighed three ways to clean up. Editing by hand across hundreds of rules invites exactly the mistakes the audit was meant to remove. Wiping the allowlist would throw away months of reasonable approvals and bring back prompt fatigue, which is how people start clicking without reading. I chose a script: load the settings as JSON, drop every rule with a mid-command wildcard and every rule matching credential shapes (a password inside a URL, key headers, token parameters, bearer strings), check that the output still parses, assert that deny rules and hooks are untouched, and confirm that the diff contains only deletions.

One detail made me smile. The settings file is on the agent's own deny list for edits, deliberately, so the agent could not apply its own cleanup. It wrote the cleaned file to another location and I copied it into place myself. The guard held against the person who built it, which is the point. The result: about 750 rules, zero startup warnings, zero credentials. The habits that keep it that way: a command that needs a secret reads it silently into an environment variable and references the variable, never "always allow" on a command with a literal secret; rules containing a path glob are one-off commands, deleted rather than fixed; any config backup gets grepped for secret shapes before commit; the same script runs once a quarter.

Where does secret scanning on push stop helping?

The first audit had already found a sibling problem: a package registry auth token committed for months in a package manager config file in two private repositories. Config files that do not look like code slip past review because nobody reads them as code. The pattern I moved to keeps only the registry mapping in the repository and the token only in user-level config.

Discipline was not going to be enough, so I added secret scanning: an automated check of outgoing commits for strings that look like credentials. A global pre-push hook runs gitleaks on everything about to leave the machine, set through a global hooks path so it covers every repository, including ones cloned later. Unlike the command hook, this one fails closed: if the scanner is missing, the push is blocked. False positives go into an ignore file inside the repository, where they are visible in review. There is an emergency skip for a single push, meant to stay rare. One consequence I had to write down: per-repository hook managers that set their own hooks path silently disable the global scanner in that repository, so I do not install them.

Here is the limit. The scanner did not flag the database password that went into history through the config backup. Scanning is a net, and nets have holes shaped like whatever the rule set does not recognize. The fix that actually removes the class of problem sits upstream: secrets never get typed into commands in the first place, so they never reach an allowlist, a backup, or a commit.

What I'd tell someone auditing their harness tomorrow

• Start from outcomes, then enumerate spellings. Ask "what could delete this" and list the CLI form, the API form and the raw-token form. Guard all of them or none of them counts.
• Write down what each boundary rests on. Token scope and host rules are guarantees; a local hook is a script you maintain. Prefer tokens that simply lack the dangerous capability.
• Know how each guard fails, and test both directions. Run must-block and must-pass cases after every change. A guard that fails open without a test is indistinguishable from no guard.
• Treat the allowlist as config that rots. Review it on a schedule with a script that asserts what it must not touch.
• Never "always allow" a command with a secret in it. That button is how credentials end up in config files and backups.
• Scan on push, block when the scanner is missing, rotate on any exposure. And accept that the scanner will miss things, which is why the upstream habit matters more.