Gauntlet Loops

How One Round Actually Runs

The loop is small. Most of the engineering is in the setup, and almost every failure comes from skipping step one.

1. Write the bar before anything exists. A bar written afterwards is a description of the artifact. Point at something concrete and inspectable: a product you want to beat, a spec, a screenshot, a golden file, a baseline measured before the change.
2. The builder produces a real artifact. Not a plan, not a diff summary. Something that runs, renders or fails on its own.
3. Package the evidence. Pixels, logs, test output, a git range, a recording of a failure being induced and recovered from.
4. Hand a fresh critic the artifact and the bar, and nothing else. No builder history, no rationale, no list of known limitations. Those are the inputs that talk a critic out of a rejection.
5. The critic reproduces, then rules. A claim it cannot reproduce did not happen. The verdict is a binary pick against the bar plus one named largest gap.
6. The largest gap goes back. One gap, to a builder, with the artifact. Smaller gaps get recorded and deliberately left alone.
7. Stop on a rule written in advance. Not on a feeling that it is probably fine now.

Judge The Artifact, Not The Story About It

This rule carries the most weight and gets abandoned first, because reading a summary is fast and running the artifact is slow. A build report is a compressed, self-authored account written by the party with the most reason to be generous. It is a useful pointer. It is never the evidence. The practical test is whether the critic could reach the same verdict with the report deleted. If not, the critic is grading prose.

Writing A Bar You Can Actually Lose Against

Most gauntlet loops fail here. The bar says make it excellent, or match industry standards, or ship production quality. None of those can be lost against, so the critic approves, and the loop degrades into an expensive way of agreeing with yourself.

A losable bar has three properties. It names something that exists outside the loop. The critic can inspect it without asking the builder for anything. And there is a plausible world in which the work fails it. If you cannot describe what failure would look like, you have written an aspiration.

Blind comparison is the sharpest form. Put the artifact and the reference side by side with the labels stripped and ask which is better. A binary pick is much harder to fudge than a score out of ten, because a score can be nudged and a pick cannot. That framing comes from Matt Shumer’s public write-up of the technique, and it is a genuine improvement on what we were doing before we read it.

Daniel Kahneman’s adversarial collaboration supplies the other half. Opposing researchers agree in advance on what result each would accept as evidence against their own position. Do the same before the builder starts: write down what would make you reject, while you have nothing invested in passing. A bar renegotiated after the artifact exists is not a bar, and the urge to soften it is the most common way these loops die.

The bar is itself a prompt artifact, so it benefits from the prompt engineering rung: explicit acceptance criteria, worked examples of pass and fail, and reuse across rounds so the standard does not drift.

When The Critic And The Builder Are Wrong Together

Fresh context stops the critic inheriting the builder’s reasoning. It does nothing about a wrong assumption they both hold because they were trained on the same internet. If both parties believe an API works a certain way, the artifact will implement that belief and the critic will confirm it. The loop converges, confidently, on a bug.

Adding more critics is the obvious move and it is weaker than it looks. A recent Apple study of nine frontier judges drawn from seven model families found their errors correlated enough that the panel carried roughly two independent votes’ worth of information, and its accuracy barely beat the best single judge. Nine opinions, two votes. Scaling up similar reviewers buys agreement, not independence.

What works is critics that differ in what they are looking for, not just in weights. Give each one a lens and a mandate to find the worst instance of its own concern. A security critic and a performance critic disagree about the same artifact in useful ways. Six copies of a general reviewer produce the same paragraph six times.

Two mechanisms keep a panel honest. First, majority-refute rather than majority-approve: if any critic produces a reproducible failure, the artifact loses even if the rest passed it. Approval needs consensus, rejection needs one. The asymmetry is deliberate, because a demonstrated failure is evidence while an approval is only the absence of one. Second, anchor the panel to something that cannot be talked around. A test suite, a compiler, a device, a real measurement. Model consensus is an opinion. A failing test is a fact.

Gauntlet Loops Versus RALPH Loops

These are not rivals, and picking the wrong one is expensive in both directions. RALPH, from Geoffrey Huntley, reruns the same prompt against a fresh context in a shell loop and lets the filesystem and git history carry the state. There is no adversary. The pushback comes from the compiler, the type system and the test suite, which Huntley calls deterministic back pressure. It is cheap, unglamorous, and on well-defined work it is effective in a way that annoys people expecting something more sophisticated.

A gauntlet loop adds a party whose job is to say no. That costs real tokens and real wall time. You are buying judgment, and you only need judgment when the definition of done and the definition of good have come apart.

Shoulders We Stand On

We run this as our standing build doctrine. We did not invent adversarial review, and we did not publish the name first. Both facts belong on a page whose whole argument is that claims should match reality.

The name is Matt Shumer’s. He named the Gauntlet Loop publicly in mid-2026 while building a browser-based first-person shooter from a single prompt, and his formulation contributed the blind comparison against a named, fetchable reference and the binary pick instead of a score. We had been running the same shape internally against DCENT_ADE. We are not going to overstate that in either direction. Credit lands where it lands.

The method is older than either of us. Michael Fagan formalized design and code inspection at IBM in 1976, and what made it stick was the finding that structured inspection by people who did not write the code caught defects unit testing missed. Ian Goodfellow’s generative adversarial networks put a generator and a discriminator in a minimax game in 2014, the same architecture with gradients instead of prompts. Karl Cobbe and colleagues at OpenAI trained verifiers in 2021 to rank candidate solutions, establishing that a separate judge can beat a bigger generator. Ethan Perez and colleagues published red teaming language models with language models in 2022, and red teaming itself is older still, borrowed from military and security practice.

The agent-era work is worth reading directly. Self-Refine established the generate, critique, refine cycle. Reflexion turned a verbal self-critique into episodic memory across attempts. Du and colleagues showed multi-agent debate improving factuality, and later work catalogued how debate degrades into premature consensus. OpenAI’s CriticGPT trained a model specifically to find bugs in another model’s code, and reviewers helped by it outperformed reviewers working alone in a clear majority of cases. Verga and colleagues showed panels of smaller diverse judges beating a single large one, and the correlated-errors work since showed how fast that advantage collapses when judges resemble each other. Daniel Kahneman and Anne Treisman contributed adversarial collaboration, the source of the discipline of agreeing on the evidence before the argument.

What we add is operational rather than conceptual: running this every day against a real product, with the evidence rules enforced instead of aspirational, and reporting what breaks. The tooling we build for it is free and open, as-is, with no paid tier and nothing held back. If it saves you real time or you use it commercially, a voluntary subscription or donation is expected and never enforced. That lives at d-central.tech/fund.