Twice last week I pointed an external LLM auditor at my own system, and twice it came back with real defects. The part worth writing down is what a clean run would have meant: almost nothing. A clean audit run from an LLM is one data point from a stochastic process, not a verdict, and treating it as a verdict is how you end up confident and wrong.
Some context, briefly. I run a multi-agent dev framework: agents build and review code in fresh contexts, and accounting machinery underneath tracks token usage, cost attribution, and run confidence for every run. That accounting layer is what I audited. The auditor was OpenAI's Codex, driven by an adversarial prompt stored in the framework's own docs and written to be re-run unchanged after any significant script, hook, or gating change. The prompt tells Codex to hunt for silent false-positives, fabricated "exact" confidence stamps, ownership gaps in the hooks, bugs reintroduced by fixes, and end-to-end trace failures.
The baseline going in was clean: 159 test fixtures passing, none failing, and the framework's CI check green. Two audit rounds later I had eight verified defects and a much more specific opinion about what "done" means when your auditor is a language model.
What a clean run means
An LLM auditor is a sampler. It reads what fits in a context window, follows whatever threads its sampling happens to pull, and reports what it hit. Run the identical prompt twice and you get two different context windows and two different walks through the code. "Found nothing" means nothing got sampled, not that nothing is there.
This is not a complaint about Codex, which did well here, as the numbers below show. It is a property of the tool class. A stochastic auditor over a finite context window works like a fuzzer: every fresh run buys more coverage, and no number of runs adds up to a proof. When a fuzzer goes a day without a crash, nobody announces the program correct. They say the fuzzer found nothing, which is a different and much weaker claim.
Hold onto that: a clean fuzzer run is evidence about the run, not about the program. So the useful question was never "does the auditor come back clean?" It was: what has to be true before a clean run stops mattering? It took two rounds of real findings to answer that.
Two rounds, eight real findings
Round 1: Codex returned 5 findings, 2 rated BLOCKER and 3 MAJOR. Before any fix was written, an independent verifier reproduced all 5 cold. Every one was real. Zero were false.
The cold verify paid for itself in the other direction too, because Codex had over-rated two severities and mis-stated two blast radii. F1 came in as a BLOCKER and verified down to MAJOR: the trigger was same-run only, not the wild foreign-session attack the report implied. F4 dropped from MAJOR to MINOR because it needs a near-impossible zero-everything state. F3's blast radius was wrong outright: Codex cited "1776 tokens across four roles," but the field it was reading is computed at build time, not runtime, so the figure was meaningless. Fixes got scoped against verified reality instead of the raw report. If you copy one habit from this piece, copy that one.
One more detail from round 1: 4 of the 5 findings sat directly on the edges of fixes I had shipped earlier the same day. A fresh adversary with full context gravitates to the newest seams, which is exactly what you want from it.
Fixes went in and the suite grew from 159 fixtures to 165, all mutation-verified, happy paths cold-verified byte-clean. Then two deeper follow-ups from the round-1 report went in on a separate run, taking the suite to 171, and that hardened tree is what round 2 audited.
Round 2 was the identical prompt, re-run unchanged. Codex returned 3 findings: 1 BLOCKER, 2 MAJOR. Cold verify: 3 of 3 real, 0 false, 0 already covered by the round-1 fixes. Three genuinely new defects in a tree that had just absorbed a full hardening pass.
This is where the fuzzer framing earns its keep. If round 2 had come back empty, I would have called it finished while reasoning from a clean run about a tree with three live defects in it. The seed just hadn't landed on those branches yet. What those 3 findings actually were is why a clean round would have been luck, not closure.
The same class, again
Round 2 found no new failure classes. Every one of its 3 findings was a fresh instance of a class round 1 had already surfaced, sitting in an adjacent branch the round-1 fixtures never reached. That is the hinge: an audit that samples instances keeps handing you the same class from a new address, and you never fixed the class by fixing the instance.
Class one: ownership-by-mention. Token attribution keyed on the presence of a session identifier in a log rather than on proven session identity. The round-1 fix gated one branch of the session-stop handler. A second branch, added later for subagent-spawned work, re-implemented the same concern by hand and never got gated. Its own comment claimed it "already proved ownership." It had not.
Class two: silent exact-wash on degraded data. The rollup summed token buckets for three roles without carrying each event's degraded-data note upward, so a zeroed bucket, tokens missing but note present, got stamped exact. The round-1 fix had closed the exact-wash on a different code path; the rollup's own summing logic had the same gap and never got touched.
Neither class was new to the codebase either. The first ownership-by-mention instance predates the Codex audit entirely: a foreign session once read the run identifier out of a shared log it had been echoed into, and the content check waved it through. And an earlier evaluation of the framework found that 9 of 10 after-run review blockers mapped to failure classes already in the defect taxonomy, classes with checklist items written specifically to prevent them. The classes were known. And the instances kept arriving anyway.
The mechanism is mundane. Every new branch that re-implements a concern by hand gets a fresh chance to omit the guard, and fixing the instance in front of you leaves that chance intact everywhere else. That is why a round 3 could still turn up a MINOR somewhere, and why the line I wrote in the framework's log that evening is the one I keep coming back to:
An LLM auditor is stochastic and context-bounded — re-running with fresh context is more coverage (like re-seeding a fuzzer), so a single empty run isn't proof.
What "done" looks like
After round 2 the obvious move was to fix the two instances and queue a round 3. I fixed the instances (6 more fixtures, mutation-verified, suite at 177), but the real decision was to stop patching instances at all. If every new branch can re-open the class, the class has to be closed somewhere no branch can route around. So: two structural guards, shipped in one run, zero logic change to the tested emitter or rollup scripts. Comment markers, a lint check, two meta-tests. Nothing else.
Guard one is an ownership chokepoint. All four known token-event emit sites now carry an OWNERSHIP-GATE: marker comment, and the CI check gained a count-parity lint: markers per known emitter must equal or exceed emit sites, and any token emit found in a shell script outside the known set gets flagged. That closes the "new file escapes the guard" gap. And because a marker by itself proves nothing, a meta-test (fixture 172) drives all three hook emitters with a foreign session and asserts zero attribution. The lint counts markers; the fixture proves the guard.
Guard two is a rollup invariant. A second meta-test (fixture 173) walks the final accounting output structurally, with no reference to any specific role block: no object anywhere in the output may carry an exact confidence stamp while having zero processed tokens and a degraded-data note present. A role block added next year can exact-wash a dropped note and this fixture fails without ever having heard of it.
The two follow-ups from round 1 push the same way, away from patched instances and toward guards a new branch can't route around. The first moved identity checking to capture time: the run's nonce goes into the specialist agent's spawn prompt, so it shows up in that agent's transcript and never in any shared log, and the session-stop hook verifies the nonce before attributing tokens. No nonce means reject, not accept. The one backward-compatibility carve-out keys on whether the run's pointer file exists, never on transcript content, since a content-based fallback would reopen the hole. The second made the gate's callers consume its exit code: the run watcher and the manager persona now escalate on a non-zero exit instead of trusting file presence.
Suite progression across the whole thread: 159 to 165 to 171 to 177 to 179. Zero regressions at any step.
The honest limits
A closure argument that pretends there are no open edges is not a closure argument, so here are mine, stated flat.
The lint has a named adjacency gap. It counts markers; it does not bind each marker to its emit site. A decoy marker placed elsewhere could mask an ungated emit and the count would still come out even. This is acknowledged in the spec. Marker-adjacency window logic was considered and rejected as brittle and forgeable, so the backstop is fixture 172: whatever the markers claim, a foreign session gets zero attribution or CI fails.
Three tempting refactors were declined. Two shared-helper refactors and a source-level lint on the rollup were proposed during the structural-close run. All three would have touched the most-tested scripts in the framework for DRY improvements without adding any catching power. Risk without catch power is gate theater.
One residual was accepted rather than patched. The session-stop handler inherits an old fallback: if the payload carries no session identifier, it derives one from the transcript's basename. A payload that omitted its identifier and named its transcript to match a real conductor's would pass. In practice the payload fields are assigned by the runtime, not echoed from transcript content, so a real foreign session arrives carrying its own identifier; the bypass requires a forged payload, not just a hostile session. The residual is bounded by that platform assumption, documented, and flagged for whenever a real ownership primitive gets built. Accepted is not fixed, and the docs say which one this is.
Convergence, and the real finish line
The log names two signals for a converging audit loop. First, severities dropping across rounds. That one is moving: round 1's F1 arrived as a BLOCKER and verified to MAJOR; round 2's F1 arrived as a MAJOR, and its blast radius was blunted further because the delta math now zeroes a single foreign fire. Same class, shrinking damage. Second, the cold verify starting to reject findings as false, which is the sign an auditor's signal is depleting. Across both rounds, every finding was real, so the auditor is not out of material. Both signals have to land before the round-loop itself is done. By that standard, this loop is not done.
It also matters less than it did. The two dominant classes are now handled structurally: a future emitter that forgets its ownership gate fails CI, and a future rollup block that exact-washes a degraded bucket fails CI, whether or not any auditor ever reads the code. That is the finish line I trust: the next omission fails automatically, whether or not any auditor ever ran. At that point, what any single run found stops being the interesting question.
The suite stands at 179 fixtures. The prompt is still in the docs, unchanged, waiting for the next significant change to the hooks. If it comes back clean, that will still be one data point.