Does an AI code reviewer catch real problems?

What was measured

Four sets of records log the review stages inside one framework: 85 runs across 9 repos, all first-party, all written by the pipeline they describe, with 135 coded findings from the original 8 runs, 31 marked BLOCKER and 92 FIXED.

The Challenger findings taxonomy (architect-challenger-patterns.md, refreshed 2026-07-09) is the closest thing to a defect list: each finding gets a pattern, severity, action taken, and artifact. It has two views that do not share scope: the frequency counts are cumulative through the 2026-07-09 refresh, over 10 runs, while the severity, action, and artifact counts cover the original 8 runs only.

The other three are counters: critic-loop records say whether a stage was sent back for another pass; cold-review ledgers and verdict files record a gatekeeper agent decision of proceed, revise, or escalate; proofreader verdict events on content pages record CLEAR, HOLD, or DROP.

What these records cannot see is the larger half of the question. A miss is invisible by construction: only catches get written down, so nothing here divides problems caught by problems present. There is no control group either, because no task was run twice, once with the review stage and once without. And this is one framework, 9 repos, one operator, with the models changing between June and September 2026, so the 85 runs are not one system measured repeatedly.

The severity and action codes are self-assigned: the same pipeline produced the artifact, ran the reviewer, coded the finding as BLOCKER or WARNING, and decided whether to fix it. The taxonomy doc does not reproduce finding text, so no reader can re-judge a call, which is who checks the checker in plain form, and counting does not solve it.

Coverage is uneven, and one field is counted under two scopes: the corpus table reports 67 runs with a critic-loop record, the coverage table 84 with the field and 1 without; the first scoped to non-content workflows, the second not at all, though both are correct for what they count.

What it stands on

The corpus table shows total runs and how many carried each record type, not whether those runs form one consistent system.

The pattern frequency table below is cumulative through the 2026-07-09 refresh, over 10 runs, and mixes scopes.

The artifact table below covers only the 8 runs that produced the findings, not the full 85.

The Delegate ledger table mixes two scopes: the proceed, revise, and escalate counts come from the 20 runs that have a ledger, not the 65 without, while the four verdict-file columns come from 20 verdict files found in 8 runs.

Proofreader verdict events on content pages, one per page per pass: the table covers the 6 runs with proofread verdicts, not the other 79.

The five holds, one line each, checkable against the page that produced it.

How much of the corpus carries each record at all, though presence does not mean a missing record would have changed any outcome.

The answer

Read plainly: things got caught, and the runs acted on most of them.

Of 135 coded findings, 92 were FIXED, 42 ACKNOWLEDGED, and 1 OVERRIDDEN, the sole override the only recorded case of a run rejecting a finding outright. Severity splits 31 BLOCKER to 104 WARNING, so most of what the reviewer raised was not stop-the-run material by the framework's own coding.

The top five patterns are missing-edge-case 30, under-specified 26, wrong-api-shape 25, internal-contradiction 24, and wrong-call-site 15. A few categories point the other way, toward reviewer error: false-positive-risk 1, false-reuse 2, symbol-inverted-reuse 1, and an uncertain concurrency/race 1–3.

The artifact column is the sharpest limit on the question: findings landed mostly on prompt 56 and spec-body 34, well ahead of architecture-section, acceptance-criteria, and plan, so in this corpus the reviewer was mostly reading instructions and specs, not code diffs. Calling that an AI code reviewer stretches what the records cover; a clean pass would prove little by itself, as a clean audit run proves nothing argues.

The gatekeeper agent ledgers show 30 proceed, 4 revise, and 25 escalate. Of the 20 verdict files inspected, 6 named a required change and 6 escalations cited signal 5 (public shipping) only, firing on what was being shipped and not something found in the work; 6 were placeholder hashes, discarded instead of counted.

On content pages the proofreader returned 21 CLEAR, 5 HOLD, and 0 DROP; those five holds are the most legible catches in the corpus, each naming a specific, checkable defect, from a failing build to a contradiction against the site's own published pages. Those are the kind of catches a critic agent can be checked on, since a reader can look.

So: the review stages logged problems that the runs then acted on, 92 of 135 coded findings fixed and 5 pages held before shipping, but the records do not show what share of the real problems that was, and they never could, because nobody wrote down the ones that got through.

Why this grade

Limited means the records support one narrow claim well and cannot address the question's plain reading at all. The narrow claim is that findings were raised and acted on. The plain reading, what fraction of real problems an AI reviewer catches, has no denominator anywhere in this corpus.

Four things hold it down. The severity-coded sample is 135 findings across 8 of 85 runs, and reviewer verdict files exist for only 8 runs. The corpus is first-party end to end, so "real problem" is a label the framework assigned to its own output, with the finding text unpublished. The models changed between June and September 2026, so a rate across the corpus would mix systems. And the findings sit mostly on prompts and specs, the shift-left end of the pipeline rather than the code-diff end most readers mean.

What keeps it from dropping to insufficient is that the actions and the five holds are concrete and independently checkable. A build that fails on a missing dataset file is not a matter of taste. The token cost of loops and rework is reported separately on run costs, so the spend side is visible even where the catch rate is not.

What would change the grade

A held-out set of known-injected defects, human-verified, run through the same reviewer, supplying the missing denominator and turning catches into a rate, up or down.

A controlled comparison: the same task and model, run once with the review stage and once without, scored by someone who ran neither.

Independent re-coding of the 135 findings by a reader outside the framework, with the finding text published, testing whether "real problem" survives a second opinion; a re-coding that reclassified a large share of the 31 BLOCKERs as false positives would move this grade down.

A stable single-model corpus, so the rate describes one system rather than four months of model churn.

Fuller coverage of the records themselves: reviewer verdict files exist for 8 of 85 runs today, and proofread verdicts for 6.