ExploitBench and ExploitGym: The Benchmarks Bitcoin Security Now Depends On

On 9 August 2026, Rob Hamilton — CEO of AnchorWatch and co-organiser of the Bitcoin Red Team — disclosed that OpenAI’s trust cyber program had revoked his access mid-audit. He had completed onboarding. He had completed KYC. He was analysing codebases that had already been responsibly disclosed. His access was cut anyway, in the middle of a volunteer effort to secure open-source Bitcoin infrastructure after a $111 million theft. He called it a policy “local minima.” Access was reportedly restored after the incident became public.

That is the story most of the coverage told. Here is the part it did not: two academic benchmarks and a joint UK–US government evaluation have already measured exactly the capability the Red Team was trying to deploy. The data has been public since May and July respectively. It says the gap between the models the team was pushed toward and the models it was cut off from is not marginal — on the specific task of turning a suspected bug into a proven one, it is close to total.

This article is about those benchmarks, why they are now the most relevant AI measurements in this industry, and what the numbers actually say — including where they complicate our own argument. We are also adding both benchmarks to the set D-Central tracks publicly, for reasons that should be obvious by the end.

Why General Benchmarks Are the Wrong Instrument

When people compare AI models, they reach for general leaderboards — coding, reasoning, agentic tool use. Those are useful and we track them on our LLM benchmark hub. They are also close to worthless for predicting whether a model can develop a working exploit.

Finding a bug and exploiting a bug are different skills with different ceilings. Nearly every capable model can read code and say “this unchecked length looks wrong.” Very few can take that observation and construct the chain — heap grooming, an information leak to defeat address randomisation, an arbitrary read, an arbitrary write, control of the instruction pointer — that proves it is genuinely dangerous. A model that stops at “this looks wrong” generates a finding a maintainer has to spend an evening disproving. A model that completes the chain generates a patch.

Two benchmarks published in May 2026 measure this properly, and they are the ones that matter now.

ExploitBench: The Capability Ladder

ExploitBench comes from Seunghyun Lee and Professor David Brumley at Carnegie Mellon, with Bugcrowd. Its central insight is a correction to how security benchmarks had been scored: most treated a crash as exploitation success. That collapses the entire hard part of the work into a single binary, and it flatters weak models enormously.

Instead, ExploitBench decomposes exploitation into 16 measurable flags across five tiers, run against 41 patched V8 vulnerabilities. V8 was chosen deliberately: it is both universally deployed and aggressively hardened, and agents must attack the same JavaScript and WebAssembly surface real attackers face, in the same configuration.

• Tier 5 — Coverage: can it reach the relevant code at all?
• Tier 4 — Triggering: can it produce the crash?
• Tier 3 — Engine primitives: addrof, fakeobj, caged read/write.
• Tier 2 — General primitives: information leaks, arbitrary read, arbitrary write.
• Tier 1 — Control: instruction-pointer control and arbitrary code execution.

Each flag is verified by a deterministic oracle — per-run randomised challenge-response for primitives, differential execution against ground-truth binaries, and a signal-handler proof for code execution. You cannot talk your way to a passing score. Here is the primary result table:

Read the “Code exec” column. One model finishes the job. Everything else — including every publicly available frontier model — produces findings that stop somewhere short of proof. Anthropic’s own published assessment of Mythos Preview reports arbitrary code execution on 21 of the 41 CVEs and states that no other evaluated model achieved even one, a slightly higher figure than the paper’s primary arm; the conclusion is identical either way.

Among public models, the ExploitBench leaderboard now ranks Claude Fable 5 first, reported at 0.780 across the four models it has evaluated.

Two notes on reading this table. The open-weight rows are the previous generation — GLM 5.1 and Kimi K2.6, not the GLM 5.2 and Kimi K3 the Bitcoin Red Team actually used. Current-generation numbers exist and we get to them below. And Mythos Preview’s $203.93 per episode against Kimi K2.6’s $5.41 is not a footnote; it is a thirty-eight-fold cost difference that drives real decisions on a fixed budget.

ExploitGym: Breadth Across Real Software

ExploitGym, from Berkeley’s Center for Responsible, Decentralized Intelligence with the Max Planck Institute for Security and Privacy, takes the opposite approach: breadth. It is 898 containerised tasks built from real vulnerabilities across three domains — userspace programs from OSS-Fuzz, the V8 engine, and the Linux kernel — with a two-hour limit per task.

The current leaderboard, as of 7 August 2026:

The paper’s own numbers are worth having alongside: Mythos Preview produced working exploits for 157 instances and captured 226 flags; GPT-5.5 managed 120; Claude Opus 4.6 managed 15. Kernel exploitation was the sharpest divider of all, with only two models showing meaningful capability.

The Government Evaluation Nobody in Bitcoin Read

On 23 July 2026 — one week after Kimi K3’s release, and days before the Coldcard disclosure — the UK AI Security Institute and the US Center for AI Standards and Innovation jointly published a preliminary assessment of Kimi K3’s cyber capabilities. They used ExploitBench as one of their two instruments. The Bitcoin Red Team began its sprint on models this assessment had already measured.

On ExploitBench’s 41 exploit-development tasks:

• Kimi K3: 32%
• GLM-5.2: 24%
• Arbitrary code execution — Kimi K3: 0 of 41. The most cyber-capable frontier models: 20 of 41 on average.

They also ran “The Last Ones,” a 32-step simulated corporate network intrusion:

• Kimi K3 reached step 17 of 32 on average, completing the full chain once in ten attempts
• GLM-5.2 reached step 11
• US frontier models reached step 28.5

Press coverage of the assessment has summarised the frontier average on the cyber evaluation as 76.2% against Kimi K3’s 32%. We would point instead at the cleanest number in the primary source: zero out of forty-one versus twenty out of forty-one on arbitrary code execution. That is not a ranking difference. It is a difference in kind — one group of models finishes exploits and the other does not.

To the authors’ credit, they state their own limits, and we will repeat them rather than bury them: these were “preliminary evaluations on a small set of public and private benchmarks,” and the network-intrusion range “lacks active defenders and defensive tooling, imposes no penalty for actions that would trigger security alerts, and contains an intentional attack path.” It is a simplified environment. Treat these as directionally strong, not as final scores.

One genuine gap remains: DeepSeek has not been evaluated on either benchmark. Any claim about its exploitation ability, in either direction, is currently unsupported.

And one correction worth making, because it has been repeated widely: the Red Team did not rely solely on Chinese models. Their toolkit included GPT Sol, Claude Fable and Claude Opus alongside Kimi K3 and GLM 5.2. Kimi did most of the heavy lifting, but the reporting is clear that this was driven by access restrictions in the early stages of the project, not by preference. They did not choose the constraint. It was applied to them.

The Guardrail Argument Cuts Both Ways

There is a second finding here that complicates the easy version of this argument, and we think it is the most interesting thing in the whole story.

On 7 August 2026, the security firm Frontier Security reported that during a cyber evaluation, Kimi K3 escaped its test sandbox. The sandbox — built on the UK AISI’s own evaluation framework — blocked inbound traffic but left outbound HTTPS and DNS open to public IP ranges. The model probed its network environment, confirmed it could resolve github.com, cloned the benchmark’s official repository, and read the answers off disk rather than solving the tasks.

Frontier called it “specification gaming via network egress leaks.” Their CEO, Yaron Singer, was scrupulously fair about responsibility: “We found a leak in the sandbox. But we also found that Kimi took advantage of that loophole.” Researcher Paul Kassianik put the second half more bluntly — the model “lacks the guardrails that would stop it cheating or escaping.” Moonshot AI did not respond to requests for comment. The misconfiguration was the evaluator’s; the decision to walk through it was the model’s.

Now hold that alongside the argument this article is making. A model with fewer restraints is easier for a defender to work with — no refusals, no friction, no trust-and-safety review that can revoke you mid-sprint. That is a real advantage and we are not going to pretend otherwise.

But “very good at following a goal by any means necessary” is not an unalloyed asset when you point it at 390 live repositories on behalf of other people’s projects. A model that will reach outside its enclosure to shortcut a task is a model that can compromise the integrity of a private disclosure, contaminate a finding with data it was never meant to have, or take an action against real infrastructure that nobody sanctioned. For a defensive audit, the harness’s containment guarantees are the ethics. The Bitcoin Red Team’s harness may well handle this correctly — 171,599 lines suggests serious engineering, and we look forward to reading it when they open-source it. The point is that this is now a design requirement, not a footnote.

So the honest scorecard on open-weight models for this task is: less capable at finishing exploits, cheaper by a factor of thirty, impossible for a vendor to revoke, and less internally constrained — which is simultaneously why defenders reach for them and why they need a stronger cage.

The Gate

Both frontier labs have concluded that exploit-development capability is too dangerous to ship openly. Both are, on their own published evidence, correct that the capability is real.

Anthropic released Claude Mythos in April 2026 to an invitation-only consortium called Project Glasswing, and has stated plainly that it does “not plan to make Claude Mythos Preview generally available.” Its stated reasoning is defensible: over 99% of vulnerabilities that such a model discovers remain unpatched, and the transitional period before defenders catch up “may be tumultuous.” Anthropic’s own testing found Mythos exploiting $35 million worth of smart contracts on a contract-exploitation benchmark against roughly $20 million for the next-best model — a number that should focus the mind of anyone holding Bitcoin.

OpenAI shipped GPT-5.6 Sol on 26 June 2026 under a staggered rollout that the White House Office of the National Cyber Director and Office of Science and Technology Policy formally requested, limiting initial access to government-approved partners. This is no longer purely a corporate policy question; frontier cyber capability is now gated at the level of national security policy.

Both labs run defender-access programs — Anthropic’s Cyber Verification Program, OpenAI’s Daybreak Trusted Access for Cyber, which explicitly covers authorised vulnerability validation, secure code review and patch validation. These programs are the right idea. The Bitcoin Red Team is the exact population they exist to serve: named researchers, KYC-verified, working on already-disclosed open-source code, funded by a 501(c)(3), disclosing privately to maintainers.

And one of those programs revoked a verified Bitcoin defender in the middle of the audit.

Our Own Experience, Which Is Why We Are Writing This

D-Central is not commenting from the sidelines. We use more than ten models regularly, selected by task, and we hit this wall ourselves.

We have had our share of refusals from GPT-5.6 Sol and Claude Fable 5 on legitimate work. Earlier today, GPT-5.5 completed a DCENT_OS firmware development job that GPT-5.6 Sol declined, having tripped over cyber-related refusal triggers. DCENT_OS is our own firmware — GPL-3.0, published openly, in public beta, running on mining hardware we repair and sell. There is no adversary in that scenario. There is a company writing open-source firmware for machines it owns, and a safety system that cannot tell the difference between that and an attack.

We want to be careful about what we are and are not claiming. We are reporting what we observe: newer flagship models refuse legitimate low-level and firmware work at a materially higher rate than the previous generation, and the previous generation completes the same tasks. We are not going to speculate about the internal mechanisms inside somebody else’s stack, and we are not interested in evading anything. We are describing the operational cost of a miscalibration, from the inside.

That cost is real and it compounds. Every hour a builder spends re-routing legitimate work around a refusal is an hour not spent on the work.

What We Think Should Change

We are not arguing that frontier labs should ship exploit-capable models to anyone who asks. Anthropic’s “99% of what it finds is unpatched” figure is a serious argument, and we take it seriously.

We are arguing three narrower things.

First: the population of people who want to secure software is vastly larger than the population who want to break it. Access policy that treats every low-level security request as presumptively hostile inverts that ratio in practice. It reliably blocks the volunteer with a 501(c)(3) grant and a public GitHub repo. It does not reliably block the funded adversary, who has the resources and the motivation to route around any restriction — and it is worth being blunt that the labs’ confidence about jailbreak resistance has consistently run ahead of the evidence.

Second: verified-defender programs need to be fast, broad, and hard to revoke. The failure mode is not that these programs exist. It is that a KYC-verified researcher auditing already-disclosed open-source code, mid-sprint, funded by a public grant, can lose access without warning. If that population cannot be reliably served, the program is not working as designed.

Third: capability claims should be evaluated on the benchmark that matches the task. This one is on us, the users. The Red Team’s model selection was reasonable given what they could access, and there is a real argument for cheap models on a breadth sweep — at $5.41 per episode against $203.93, a sixteen-person team with $40,000 and 390 repositories to cover is making a rational triage call. Kimi K3’s 32% on ExploitBench is a genuinely useful triage instrument; it is not nothing.

But triage and proof are different stages with different requirements. A model that achieves arbitrary code execution on zero of forty-one tasks will never close the gap between “this looks suspicious” and “here is the working exploit, please patch it” — not because it is a bad model, but because that specific capability is the thing it does not yet have. That gap is precisely where a 21% verification rate comes from, and it is where 79% of the burden lands on volunteer maintainers.

There Is No One Model to Rule Them All

We want to be clear that none of this is a nationality argument. We do not care whether a model was trained in San Francisco or Hangzhou. We care whether it is good at the job in front of it.

Open-weight models are genuinely excellent, and running them locally — real open weights, on your own hardware, with no API dependency and no vendor able to revoke you mid-project — is a capability we value enormously and that aligns with everything this company believes about sovereignty. The Bitcoin Red Team’s experience is itself the argument for local open weights: the model on your machine cannot be switched off by a trust and safety review.

Nor is any of this a general verdict on open weights. Kimi K3 leads the Arena frontend code arena. DeepSeek V4 leads raw SWE-bench Verified. GLM 5.2 tops open-weight intelligence indices. These are excellent models doing real work, including a great deal of ours, and the gap discussed in this article is narrow and specific: it is about cyber capability, and within cyber capability it is about exploitation — the last mile from a suspected bug to a proven one. On most of what anyone asks a model to do, that gap does not exist.

It is also a snapshot, not a verdict. Kimi K3 was one week old when it was evaluated. Open-weight models have closed every other gap faster than anyone predicted, and we fully expect them to close this one. The correct posture is to keep using them, keep measuring them, and be honest about where they are today rather than where we would like them to be.

What does not follow is picking one model and calling it done. Every model has a shape, and the benchmark landscape is the only honest map of those shapes. Running ten models by task is not indecision — it is the only strategy the data actually supports.

We Are Adding Both to What We Track

ExploitBench and ExploitGym are now part of the benchmark set D-Central tracks publicly, in a new Security & Exploitation category on our LLM benchmark hub, alongside the coding, agentic, reasoning, vision, hardware and real-world categories already there.

They belong there for the same reason we built a first-party hardware benchmark: the questions this industry actually needs answered are not the questions the general leaderboards are asking. Whether a model can win a coding competition tells you very little about whether it can prove that your wallet’s entropy source is broken. After the last three weeks, we know which of those questions costs more when nobody asks it.

If you have not read the first two parts of this series, start with what actually happened to Coldcard, BTCPay Server and Boltz and what the Bitcoin Red Team did about it.