How often does a critic loop change the output?

What was measured

Four records, all produced by the same framework whose review practice is in question, none of them scoring correctness.

The critic-loop counter is the primary one. It records, per run and per stage, how many times a stage was sent back after a critic agent read its output. The stages counted are analyst, architect, and prompts.

The rework ratio is a spend proxy: the share of a run's tokens spent on rework, where the run recorded one. It measures tokens, not defects, and it is reported in full on run costs.

Cold-review ledgers record a decision of proceed, revise, or escalate: revise sends work back, escalate hands the call to a human. Proofreader verdict events on content pages record CLEAR, HOLD, or DROP, one per page per pass, and a HOLD stops a page before it ships.

What none of these can see:

Whether the output that came back was better. No ground truth on correctness exists anywhere in the corpus, so "changed" is the most this supports, never "improved."

A loop that should have fired and did not. Like a missed defect, a missed loop leaves no record.

The counterfactual. No task was run twice, once with the critic stage and once without, so nothing separates what the loop changed from what the run would have produced anyway.

One scoping trap is worth naming before reading the numbers. The corpus table reports 67 runs with a critic-loop record while the coverage table reports 84 runs carrying the field and 1 without. The distribution table below is explicitly scoped to non-content workflows; the coverage row counts every run that carries the field. Both are correct for what they count, and the 73% share belongs to the 67, not the 84. The per-run table shows content runs carrying 0 in that column, so the wider scope would not read the same way.

Finally, the corpus is 85 runs across 9 repos from one operator, with the models changing between June and September 2026, so it is not one system measured repeatedly. The Challenger findings taxonomy is not used here: it codes what reviewers found, not how often a stage looped.

What it stands on

The corpus:

Critic loops, for non-content workflows with a record:

Rework ratio, the share of tokens spent on rework where recorded:

Cold-review decisions from the Delegate ledgers:

Proofreader verdict events on content pages, one event per page per pass:

The five holds, one line each:

How much of the corpus carries each record at all:

The answer

Loops fire often. Of 67 runs with a record, 49 had at least one loop, a share of 73%. The distribution: 18 runs looped 0 times, 21 looped once, 15 twice, 11 three times, and 2 looped 4 or more times.

The looping concentrates at two stages. Mean architect loops is 0.67 and mean prompts loops is 0.49, against mean analyst loops of 0.09. Whatever the critic is catching, it is almost never catching it at the analyst stage. That is the shape argued in gate the flow, not the judgment: the loop lands where the artifact is specific enough to be wrong in a checkable way.

The spend tells a quieter story. Of 61 runs that recorded a rework ratio, 23 recorded any rework above zero. The median across all 61 runs is 0.0. The 75th percentile is 0.086 and the maximum single run is 0.562. Among the 23 runs that did record rework, the median is 0.139. So the typical run spent no measurable share of its tokens on rework.

Those two readings sit together awkwardly, and the gap is the honest finding. A loop fired in 73% of recorded runs while most runs recorded 0.0 rework spend. Either loops usually cost too little to register in the ratio, or the two records are measuring different runs, and the corpus does not settle which.

Downstream, the gate decisions are rarer. Ledgers show 30 proceed, 4 revise, and 25 escalate. Revise, the decision that sends work back, is the least common of the three. Escalate at 25 moves the call to a human instead of changing the output, and 6 of those cited signal 5 (public shipping) only, so the trigger was the shipping surface, not something found in the work. Of 20 verdict files inspected, 6 named a required change and 6 were placeholder hashes, discarded.

On content pages the proofreader returned 21 CLEAR, 5 HOLD, and 0 DROP. Each hold forced a change before the page shipped, and each names something checkable, from a dataset file that does not exist to a dek disagreeing with its own table. This is where a changed output is easiest to see, and it is also the smallest slice, covering 6 runs.

A critic loop fired at least once in 49 of the 67 runs with a record, a 73% share, most often at the architect and prompts stages, while 23 of 61 runs recorded any rework spend and the median run recorded 0.0. How often a loop fires is on the record. Whether the output that came back was better is not.

Why this grade

Mixed means the records answer one half of the question well and the other half not at all.

The frequency half is well recorded. The loop counter is mechanical, written per run and per stage, and 84 of 85 runs carry the field. A count of loops is not a judgment the framework could flatter itself with; it either looped or it did not.

The change half is not measured. The rework ratio is spend, not correctness, and it is missing for 24 of 85 runs. Nothing in the corpus records what a loop actually did to the artifact, so a loop that rewrote a section and a loop that changed a heading are one increment each, which makes "changed the output" inferred rather than observed.

Three confounds sit on top. Loop counts partly reflect workflow shape: more critic stages means more chances to loop, independent of anything the reviewer found. The corpus spans four months of model churn, so the 73% describes a moving system. And it is first-party throughout, the pipeline reporting on itself: the same orchestrator decided when to loop and then wrote down that it looped.

What would change the grade

A per-loop diff record, so a loop that changed nothing is distinguishable from a loop that rewrote a section. That is the cheapest of these.

A paired corpus: the same task run with the critic loop and without it, the two outputs scored blind by someone outside the run. That is the only thing here that addresses "better" rather than "different."

Rework ratio recorded on every run, not 61 of 85, so the spend distribution stops being a partial view. The same holds for the proofread verdicts, which cover 6 runs.

A stable single-model corpus, so a share describes one system, not four months of model changes.

In the other direction: recorded diffs showing most loops produced cosmetic changes would move this grade down, and so would a paired comparison finding no difference.