engineering

Shift-left quality in an agentic dev pipeline: five structural changes that move Critic catches earlier

What one structured multi-agent pipeline learned from iterating on when its reviewer fires, not how thorough it is.

By Mark Down · Jul 2026 · 5 min read

In a multi-agent pipeline, the Critic will catch things. The question is when.

Picture the chain. An Analyst produces requirements. An Architect turns them into a plan. A Scribe turns the plan into prompts. Coders execute the prompts. A Critic reviews the outputs. Somewhere in there, a defect exists, and the Critic finds it. If the defect is a bug in the spec, and the Architect builds the spec faithfully, and the Scribe documents it faithfully, and a builder implements it faithfully, then the Critic is catching a spec bug after three stages of faithful work built on top of it. That costs more to fix than the same bug caught in the spec before anyone built anything.

The instinctive response is to run the Critic more often, or make it more thorough. That is the wrong lever. A more thorough Critic still fires at the end of a chain, and the thing it catches still has to be unwound through every stage that trusted the broken artifact. The better question is narrower: what is the Critic actually catching? And which of those catches should have happened upstream?

The pattern that compounds the cost

The Critic in this kind of pipeline is cold by design. It hasn't seen the run's prior context, so its objections are real rather than shaped by the reasoning that produced the artifact. If the Architect talked itself into a bad decision, a warm reviewer inherits the same talking-into. A cold reviewer doesn't. That coldness is the point, and it is worth keeping.

But cold review at the end of a long chain has a cost that scales with how deep the chain is. An Architect re-spec is more expensive than a pre-spec grill, because the re-spec has to undo a plan and everything downstream that trusted it. A build-phase catch of a wrong API is more expensive than a prompt-review catch of a wrong contract, because by build time the wrong contract is already code. The Critic didn't fail in either case. The timing did. So the fix is not a better Critic. It is redundant checks earlier, at the points where a catch is still cheap to act on.

Five structural changes

Pre-spec decision screen

Before the Analyst writes requirements, it does a pass over the brief and asks one question: what would get guessed here if we just proceeded? Material decisions have a specific property. Timezone. Locale. Auth model. Whether an action triggers an external send. Whether it spends money. These are cheap to ask about before the spec exists and expensive to change after the Architect has built a plan around a silent assumption.

The discipline is a scan for three signals: a new surface, unsourced facts about users, and high-blast-radius calls like auth, money, or external sends. If three or more material questions would otherwise get guessed, the Analyst raises one batched checkpoint with recommended defaults attached. One question set, not a live interview. The point is to surface the guesses, not to slow the run to a crawl.

The spec exists only after those answers are in. A spec built on a guessed timezone reads as clean and correct right up until someone in the wrong timezone uses it, and by then it has propagated through three stages. The pre-spec screen catches that while it is still a question and not yet a bug.

ADR decision records

On any project that runs more than once, the Architect eventually re-argues a decision it already made. Not out of laziness. Each run starts fresh, so a settled question comes back open, and sometimes it gets settled the other way, and now two runs disagree with each other.

The fix is to commit architecture decisions to docs/adr/ in the target repo. The Analyst reads them before writing requirements. The Architect reads them before designing. The Critic reviews specs against them cold: it has the committed project facts but not the current run's history. An accepted ADR is ground truth. The Critic blocks a spec that re-opens a settled question unless the spec includes a superseding ADR that says so on purpose.

This does not make cold review warm, and it isn't supposed to. It gives the cold reviewer a way to flag drift without knowing the run that produced the drift. The record carries the memory the reviewer doesn't have.

Seam-first testing

Every build prompt has to declare the seam it tests. The seam is the public interface the feature actually crosses: an HTTP request and response, CLI output, a reducer transition, a persisted data invariant, a job side effect. Or the prompt declares none with a reason, for cases where there is no seam to cross, like a migration, a visual-only change, or generated sync code. Declaring the seam forces you to name what the feature does before you write a test for it.

Then, mutation verification. Break or invert the guarded behavior in a throwaway edit. Confirm the test goes red. Restore the code. Run it green again. If the test doesn't go red when you break the behavior, you don't have a test. You have code that asserts the implementation runs without throwing.

The anti-pattern this targets is common and quiet: a test that covers the code path but not the observable behavior. It calls the function, the function doesn't error, the test passes. It would also pass if the function returned the wrong answer. A tautological test is green while the behavior is broken, which is worse than no test, because it reads as coverage.

Two-axis build-diff review

Build-diff review used to produce a single list of findings: genuine spec bugs mixed with code-standards issues. A cosmetic note about variable naming could sit at the same apparent weight as "this diff doesn't implement what the approved spec required." A reader scanning the list has to re-sort it every time.

The two-axis split separates them at the source. Axis one is spec-fidelity: does the diff build what the approved plan required? Axis two is code standards: does it meet baseline hygiene? The axes don't cross-rank. A standards warning can't mask a spec bug, and a spec-faithful diff can still carry maintainability notes without those notes blocking it.

The practical effect is that a reviewer can look at the spec-fidelity axis and see only "builds the wrong thing" findings, undiluted by style. The genuine problems stay visible because the cosmetic ones are in a different column.

Bug-diagnosis discipline

Bugs get fixed before they're diagnosed. Someone reads the report, forms a theory, changes the code, checks whether the symptom went away. If it did, the ticket closes. This is guessing with extra steps.

The correct sequence is the other way around. Build a red-capable feedback loop first: something that exercises the actual bug path and can fail. Minimize the repro by stripping everything that isn't load-bearing. Locate the cause. Then fix. A fix that addresses a plausible cause without first confirming the symptom is a guess. Guesses can be right. They can also mask the real bug while leaving it in place, or introduce a new one next to it, and you won't know which, because you never had a loop that fails on the real thing.

The regression test closes the loop. The committed test has to fail on pre-fix code and pass after the fix. That is the evidence that the fix addressed the actual bug and not a plausible neighbor. It lives in the run record, not just in the commit message, because the commit message is a claim and the failing-then-passing test is a check.

One data point

One build went through the changed pipeline. The target was a feature on novadiem.com: an assessment-intake endpoint, public-facing, able to trigger a spend.

The Critic caught that endpoint in the pre-build review, before any code was written. The finding was specific: a public-facing path with no rate-limit, budget-cap, or length-guard decisions pinned. An unguarded endpoint that can trigger a spend reads as fine in a spec and turns into an incident in production. Catching it before the spec turned into three stages of code that all trusted an ungated endpoint is the right catch at the right time.

Prompt review came back clean. Build-diff review came back with warnings and no blockers. Live verification covered the spending branches.

One data point. That is what one run produces: evidence of shape, not evidence of scale. It tells you the pieces fit together on a real feature. It does not tell you they hold up across different failure modes, different projects, or a hundred more runs. Those are separate claims and this run doesn't make them.

What the changes share

Each of the five moves a catch to before it costs something to unwind. The pre-spec screen moves product-decision catches to before the spec exists. ADRs move architectural-drift catches to before the Architect re-argues them. Seam-first testing moves test-validity catches to before the test is written. Two-axis review makes spec bugs visible separately from standards issues, so the expensive ones don't hide behind the cheap ones. Diagnosis discipline moves "we don't actually know what's wrong" to before the code changes.

None of these remove the cold Critic. It still runs, cold, at the end. But by the time it runs, the most expensive catches have already happened upstream, where they were cheap. Whether that holds at scale is a measurement question, not a structural one. The structure is in place. The measurement isn't in yet.