A green build is not proof: how I review AI-generated changes

TL;DR: An automated bulk refactor that moved hard-coded strings into translation calls left one React hook call at module level, just past a component's closing brace. The type checker passed, the production build passed, it deployed, and the entire admin panel of a product I run stopped opening. The linter would have flagged it in a second, but nothing ran the linter. The gates I added: lint on staged files in a global pre-commit hook, a narrow prebuild guard for rules whose violation crashes at runtime, tested in both directions, and a habit of running the app and opening every changed screen after any bulk automated change. The same stance covers other ways AI-generated changes go wrong: package names get audited before install, "this does not exist" needs two independent checks, and a second agent reviews the diff with fresh context.

A green build that took the admin panel down

A lot of the code in my products is now written or transformed by coding agents, and much of it arrives as bulk changes: hundreds of small, mechanical edits applied by a script. That is where agents shine, and it is also where one wrong insertion hides best. This one was an internationalization pass. The script replaced literal strings with calls to a translation hook, and in one file the inserted line landed after the closing brace of the component instead of inside it:

// illustrative shape
export function DemandPanel() {
  const t = useTranslations();
  return <Section title={t.demand.title} />;
}
const labels = useTranslations().demand; // module level: builds fine, crashes on render

Everything automated said yes. The type check was clean, the production build was clean, and the deploy went out. Then the admin panel would not open at all, because the hook call ran outside any component and threw as soon as the module loaded.

Why didn't the type checker or the build catch it?

Because neither was asked to. A type checker verifies types and syntax, and that line is well typed. The rules of hooks, React's requirement that hooks are called only at the top level of a component or another hook, are enforced by a lint rule, not by the compiler. The build in this setup does not run the linter. The lint output would have said it plainly: a hook cannot be called at the top level of a module. Nobody ran it.

That is the general trap with AI-generated changes. A green build tells you the code is shaped correctly. It says nothing about whether it behaves correctly, and the failures that slip through are exactly the ones where shape and behavior diverge: runtime rules, rendering, ordering, anything that only exists when the code runs.

Where should the gate go?

I weighed four places to catch this class of bug.

• In CI. Works, but it charges you for server minutes and a round trip for something a laptop can answer in a second, and the deploy you have to roll back may already be out.
• A per-repository hook manager. The usual answer. In my setup it would have replaced the global hooks path, which silently disables the secret scanner I run on every push. Not an option.
• Full lint as a build gate. Tempting and wrong for a codebase with existing lint debt. With around seventy pre-existing errors of a harmless kind, the gate would be red from day one, and a gate that is always red gets disabled within a week. Red has to mean one thing: do not deploy.
• A local pre-commit hook plus a narrow build guard. Fast, free, and it fires at the moment the broken file is being committed.

I chose the last one, in two layers. The first is a global pre-commit hook, a script git runs before recording a commit, that lints only the staged files and blocks the commit on errors. It is quick because it touches only what changed, and the error is always in a file you are working on. In a monorepo it groups files by the nearest lint config and runs the linter from that directory, because a flat config is looked up from the working directory and running from the root simply does not find the app configs. It formats only in repositories that have a formatter config; imposing formatting elsewhere would reformat the whole project on the first commit and bury the real change in thousands of lines of diff.

The second layer is a prebuild guard that blocks the build on a short list of rules whose violation takes a page down. The list started with one rule, the rules of hooks, and a rule joins it only when breaking it crashes something. It catches commits that bypassed the hook, from another machine or with the emergency skip. The guard is tested in both directions: on clean code it must pass, and on a deliberately broken file it must fail. An untested gate is a prop. I apply the same both-directions test to the safety hooks around my coding agent.

The first version of the pre-commit hook did not work, and the way it failed is worth knowing. It used a shell builtin that exists in modern bash but not in the old bash that macOS ships as the system shell, which is the one git invokes. Run by hand from my terminal it passed. Run by a real commit, it failed every time and blocked perfectly clean work. Hooks now get written for the old shell and tested through an actual commit, never by invoking the script directly, because a manual run uses a different shell, a different path and a different working directory.

Why run the app after a bulk automated refactor?

Lint catches one rule. The broader lesson is that a script inserting code in hundreds of places will be wrong somewhere you did not predict, and the build cannot tell you where. So after any bulk automated change I run the app and open every changed screen before deploying. It takes minutes, and it is the only check that would have shown the broken panel before users did.

The same applies to visual claims. Markup that reads correctly can render badly. Once, the HTML of a feature looked fine, and on screen it showed only a small fraction of the content it was supposed to show, with a button sitting on top of a table header. Neither problem was visible in the source. And the reverse also bites: before reporting a visual bug from a screenshot, I confirm it with computed styles and element geometry, because a screenshot tool that scrolls inside a clipped container can manufacture a bug that is not there.

How do you vet a package an AI agent suggests?

Models hallucinate package names, and attackers register the hallucinated names in advance. That is slopsquatting, the AI-era cousin of typosquatting, where a malicious package sits one typo away from a popular one. Both are supply chain attacks: the dangerous code arrives as a dependency, with the full permissions of your process, on your machine and in production.

My rule is that no package goes in until it is audited, and the audit happens before the install, not after. It takes about a minute: maintainers, repository link, license, deprecation status, last publish date, and weekly downloads from the registry. Red flags are a name suspiciously close to a well-known package, a single anonymous maintainer, no repository, and install scripts, which run code before you have built anything. The first question is always whether the package is needed at all; a dozen lines of my own code beat a new dependency with a large tree.

I learned the ordering the easy way. I once added a markdown renderer and its tables plugin and audited them only when asked. The result was clean, but the order was wrong: with a vulnerable or abandoned package I would have found out after it was already in. The audit also has to cover the package's security model as well as its CVEs. That renderer escapes raw HTML by default, and a separate plugin turns that off. When the input is LLM output, that default is the whole point, so there is a test that fails if anyone removes it. For the other half of dependency hygiene, patching what you already have, see the transitive CVE I fixed with npm overrides.

Why is "it doesn't exist" the most expensive claim?

An agent saying "this is not implemented yet" sounds harmless, and it leads straight to planning work that is already done. It happened to me: a session searched for a structured-data type in double quotes, the code used single quotes, zero hits came back, and the conclusion was that the feature needed building. It existed on four levels of pages and had been committed two days earlier.

Verifying absence takes more than one grep. Search the bare keyword without quotes or punctuation first, then narrow down. Check git history, because commit messages usually say plainly when something was added. And when a person says a thing exists and the search does not find it, assume the search is wrong and keep looking before correcting them.

A second agent as reviewer, and what I'd tell you tomorrow

The agent that wrote a change shares its own blind spots: it knows what it meant, so it reads what it meant. The review I trust more comes from a second agent in a fresh session that sees only the diff and the spec, ideally a different model, because a different model makes different mistakes and notices different ones. Its checklist is short: tests pass in the main working tree, the scope matches the spec and nothing else changed, and the project's conventions held. The limit is that review is still reading. A reviewer might have spotted a hook outside a component; only running the linter or the app would have made it certain. More on how that fits a daily workflow in eight months of running a product with agents as co-engineers.

Why do I still read every pull request myself?

Agents open pull requests; they never merge them. The last gate is a human, and for my products that human is me. This is what people mean by human in the loop: the agent does the volume, and a person who owns the outcome decides what reaches production. Checks turning green is a precondition for my review, never a substitute for it.

My read of the diff is not a second pass for typos. I check it against the written standards the agents were given: does the route validate and delegate, or has business logic leaked into it; is a new helper a third copy of something that already exists; does any function no longer fit on a screen, or any file grow past a few hundred lines; are the business rules pure functions with the side effects pushed to the edges. Then the tests themselves: do they check behavior a user would notice and the edge cases that break money or data, or do they pin the implementation so the next refactor breaks them for nothing. A test that asserts a getter returns what was set is noise; a test that reproduces the bug before the fix is the one I want.

Tests come in layers, and each catches a different class of mistake: unit tests on rules and edge cases, integration tests on the paths that touch the database and third parties, and end-to-end tests in Playwright on the flows users depend on, the ones that only fail when the pieces are put together. The E2E suite is the closest thing to a user clicking through the product before every release, and a flow that opens the admin panel catches exactly the kind of crash from the top of this post. Only when the diff reads right, the tests test the right things, and the changed screens work in a browser do I merge.

• Never let an agent merge. Green checks start the human review; they do not replace it.
• Treat a green build as a shape check. It proves types and syntax, not behavior.
• Put the first gate where the mistake is made. Lint staged files before the commit, then keep a narrow build guard as a second line.
• Keep gates meaningful. Only rules that crash things, so red always means stop. Test every gate in both directions, through the real trigger.
• After bulk automated changes, run the app. Open every changed screen before deploying.
• Audit a package before installing it. Especially one a model suggested.
• Demand two independent checks for "it does not exist". A false negative costs more than a minute of extra searching.