{ "meta": { "title": "D-Central — LLM Benchmark Hub (by capability)", "description": "A capability-organized map of 37 AI-model benchmarks across six categories (Coding, Agentic, Reasoning & Knowledge, Vision, Hardware & PCB Design, Real-World Tasks). A taxonomy that links each benchmark's live leaderboard rather than freezing scores that rot monthly — anchored by D-Central's own first-party Konduit KiCadBench hardware-design leaderboard.", "generated": "2026-07-18T20:43:29+00:00", "as_of": "2026-07-18", "version": "1.0", "license": "https://creativecommons.org/licenses/by/4.0/", "license_name": "CC BY 4.0", "source": "https://d-central.tech/llm-benchmarks/", "method": "Benchmark descriptions are authoritative statements of what each eval measures and who maintains it, with a link to the live leaderboard as the always-current source of scores. Frontier per-model scores are deliberately NOT frozen into this dataset for the four general categories, because they change monthly and the public boards are canonical. Hard scores appear only in the Hardware category: (a) D-Central's first-party Konduit KiCadBench + EE replica leaderboard (preview, dated), and (b) published, dated, sourced scores from the EE-benchmark papers. The emerging OSRSBench is listed but carries no numbers pending its own citation.", "provenance": "Konduit KiCadBench = first-party (DCENT_Konduit KiCad MCP server, evaluator v2, 2026-07-09). EE scores from arXiv:2606.23704 (PCB-QA), arXiv:2602.00510 (PCBBench), arXiv:2507.10639 (SMPS). General-category benchmark descriptions from the maintainers' own leaderboards/papers.", "disclaimer": "Konduit preview scores are NOT fab-ready claims — human EE review required before manufacturing. External leaderboard links are the source of truth for current scores; this page does not freeze them. OSRSBench is an emerging benchmark listed without verified metrics." }, "categories": [ { "id": "coding", "name": "Coding", "blurb": "Can the model write and, more importantly, FIX real code — repo-level bug fixes whose patches must pass the project's own tests, competitive-programming under contamination control, and correct diff-format editing." }, { "id": "agentic", "name": "Agentic", "blurb": "Can it reliably DRIVE tools and multi-step workflows — call functions correctly, recover from failure, complete tasks across real websites and APIs. Reliability across repeated attempts (pass^k) matters more here than a single lucky run." }, { "id": "reasoning", "name": "Reasoning & Knowledge", "blurb": "Graduate-level, Google-proof questions and competition math — probing genuine reasoning and expert knowledge rather than retrieval. The hardest of these are designed to sit near the ceiling of what frontier models can do." }, { "id": "vision", "name": "Vision", "blurb": "Multimodal understanding — reading diagrams, charts, documents and figures, and reasoning over what it sees, not just captioning it. Harder variants strip the text shortcuts so the model must actually look." }, { "id": "hardware", "name": "Hardware & PCB Design", "blurb": "The frontier almost nobody benchmarks: can a model design electronics — read a datasheet, generate a schematic, build and route a board, pass ERC/DRC? This is D-Central's home turf, and the one category on this page where we publish our OWN first-party leaderboard." }, { "id": "realworld", "name": "Real-World Tasks", "blurb": "Long-horizon, open-ended tasks from the actual world — operating a desktop, a phone, or a video game; doing a real freelance job priced in real money. The messiest, least-saturated, most honest test of whether an \"agent\" is actually an agent." } ], "rows": [ { "cat": "coding", "name": "SWE-bench Verified", "org": "SWE-bench team (Princeton)", "measures": "500 human-validated real GitHub issues; scores the % where the model's patch makes the repository's own test suite pass. The default \"can it fix real code\" bar.", "url": "https://www.swebench.com/", "tag": "public", "note": "The industry's headline agentic-coding number. Live leaderboard — scores move with every model release." }, { "cat": "coding", "name": "LiveCodeBench", "org": "LiveCodeBench", "measures": "Contamination-resistant competitive-programming tasks collected continuously over time: code generation, self-repair, and test-output prediction on problems published after a model's training cut.", "url": "https://livecodebench.github.io/", "tag": "public", "note": "Designed so a model can't have seen the problems in training — the honesty control for coding scores." }, { "cat": "coding", "name": "Terminal-Bench", "org": "Terminal-Bench", "measures": "Agentic command-line tasks in a sandboxed shell — build, debug, sysadmin — where the model must operate a real terminal to completion.", "url": "https://www.tbench.ai/", "tag": "public", "note": "On the Kimi K3 card as \"Terminal-Bench 2.1.\" Measures tool-driving, not just code authoring." }, { "cat": "coding", "name": "Aider Polyglot", "org": "Aider", "measures": "225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust; tests correct DIFF-format code editing, not greenfield generation.", "url": "https://aider.chat/docs/leaderboards/", "tag": "public", "note": "Edits must apply cleanly to existing files — a stricter, more real-world bar than write-from-scratch." }, { "cat": "coding", "name": "SciCode", "org": "SciCode", "measures": "Research-grade scientific coding problems drawn from real papers across physics, chemistry, biology and materials science.", "url": "https://scicode-bench.github.io/", "tag": "public", "note": "Used as a component of Artificial Analysis's Intelligence Index." }, { "cat": "agentic", "name": "τ-bench (tau-bench)", "org": "Sierra", "measures": "Tool-agent-user dialogue in retail and airline domains; reports pass^k — reliability across k repeated attempts — not just a single pass@1. Consistency is the whole point.", "url": "https://github.com/sierra-research/tau-bench", "tag": "public", "note": "The benchmark that made \"does it work EVERY time\" a first-class metric for agents." }, { "cat": "agentic", "name": "τ²-bench (tau2-bench)", "org": "Sierra", "measures": "Dual-control extension of τ-bench where the USER also takes actions in the environment, forcing genuine coordination rather than one-sided tool use.", "url": "https://github.com/sierra-research/tau2-bench", "tag": "public", "note": "Harder successor; the telecom split (τ2-Telecom) is a common heavyweight agentic probe." }, { "cat": "agentic", "name": "BFCL — Berkeley Function-Calling Leaderboard", "org": "UC Berkeley (Gorilla)", "measures": "Accuracy of function/tool calling across single, parallel, and multi-turn calls plus irrelevance detection (knowing when NOT to call a tool).", "url": "https://gorilla.cs.berkeley.edu/leaderboard.html", "tag": "public", "note": "The benchmark of record for raw tool-calling. See our companion Agent-Capability Database for the open-weight cut." }, { "cat": "agentic", "name": "GAIA", "org": "Meta / Hugging Face", "measures": "466 real-world assistant questions that require web search and tool use, across three difficulty levels; easy for humans, hard for models.", "url": "https://huggingface.co/spaces/gaia-benchmark/leaderboard", "tag": "public", "note": "Tests whether an assistant can actually go and find things, not just recall them." }, { "cat": "agentic", "name": "WebArena", "org": "Carnegie Mellon", "measures": "Self-hosted, fully functional replicas of real websites (e-commerce, forum, CMS, GitLab); scores task success on multi-step web actions.", "url": "https://webarena.dev/", "tag": "public", "note": "Runs against real running sites, so success means the task actually completed end to end." }, { "cat": "agentic", "name": "BrowseComp", "org": "OpenAI", "measures": "Hard web-browsing research questions whose answers are buried and require persistent, multi-hop browsing to locate.", "url": "https://openai.com/index/browsecomp/", "tag": "public", "note": "On the Kimi K3 card (reported 90.4). A deep-research stress test." }, { "cat": "reasoning", "name": "GPQA Diamond", "org": "GPQA authors", "measures": "198 \"Google-proof\" graduate-level physics, chemistry and biology multiple-choice questions written by domain PhDs to resist quick web lookup.", "url": "https://github.com/idavidrein/gpqa", "tag": "public", "note": "A staple hard-knowledge probe on every serious 2026 leaderboard." }, { "cat": "reasoning", "name": "MMLU-Pro", "org": "TIGER-Lab", "measures": "Harder successor to MMLU: ~12,000 questions with 10 answer options and a heavier reasoning weighting to reduce guess-luck.", "url": "https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro", "tag": "public", "note": "The reasoning-weighted broad-knowledge baseline once plain MMLU saturated." }, { "cat": "reasoning", "name": "Humanity's Last Exam (HLE)", "org": "CAIS + Scale AI", "measures": "~2,500 extremely hard expert questions across ~100 subjects, deliberately built to sit near the ceiling of frontier-model capability.", "url": "https://lastexam.ai/", "tag": "public", "note": "Named for the ambition: the hardest closed-ended exam that still has verifiable answers." }, { "cat": "reasoning", "name": "AIME 2025", "org": "MAA (AMC)", "measures": "American Invitational Mathematics Examination — 15 integer-answer problems; the standard hard competition-math probe. Use the current year's set to avoid contamination.", "url": "https://artofproblemsolving.com/wiki/index.php/2025_AIME_I", "tag": "public", "note": "Cheap to state, brutal to solve; a clean signal of step-by-step math reasoning." }, { "cat": "reasoning", "name": "ARC-AGI (ARC-AGI-1 / ARC-AGI-2)", "org": "ARC Prize Foundation", "measures": "Novel abstract grid puzzles testing fluid reasoning and generalization to unseen patterns — knowledge doesn't help, only reasoning does.", "url": "https://arcprize.org/leaderboard", "tag": "public", "note": "The benchmark specifically built to be easy for humans and hard for pattern-matchers." }, { "cat": "vision", "name": "MMMU", "org": "MMMU team", "measures": "College-level multimodal questions across 30 subjects, mixing diagrams, figures, tables and text — the broad multimodal-understanding baseline.", "url": "https://mmmu-benchmark.github.io/", "tag": "public", "note": "Reasoning over images at expert breadth, not object recognition." }, { "cat": "vision", "name": "MMMU-Pro", "org": "MMMU team", "measures": "Harder MMMU variant that adds vision-only-input questions, defeating models that shortcut through the text and ignore the image.", "url": "https://mmmu-benchmark.github.io/", "tag": "public", "note": "On the Kimi K3 multimodal card. Forces the model to actually look." }, { "cat": "vision", "name": "MathVista", "org": "MathVista team", "measures": "Visual mathematical reasoning over figures, plots and geometry — reading a chart or diagram and doing the math it implies.", "url": "https://mathvista.github.io/", "tag": "public", "note": "Where vision meets quantitative reasoning; a frequent separator between strong and weak VLMs." }, { "cat": "vision", "name": "DocVQA", "org": "DocVQA", "measures": "Question answering over scanned document images, requiring layout understanding and OCR-grounded reasoning.", "url": "https://www.docvqa.org/", "tag": "public", "note": "The practical \"read this paperwork and answer\" test." }, { "cat": "vision", "name": "ChartQA", "org": "ChartQA authors", "measures": "Question answering over charts and graphs — visual plus numeric reasoning to extract values and trends from plotted data.", "url": "", "tag": "public", "note": "Directly relevant to reading the kind of performance charts these very benchmarks produce." }, { "cat": "vision", "name": "ZeroBench", "org": "ZeroBench", "measures": "A deliberately near-impossible hard-perception benchmark where current frontier VLMs score at or near zero — a headroom probe for the next generation.", "url": "https://zerobench.github.io/", "tag": "public", "note": "Built to NOT be saturated; useful precisely because nobody is acing it." }, { "cat": "hardware", "name": "Konduit KiCadBench", "org": "D-Central (first-party)", "measures": "End-to-end autonomous PCB and schematic design through a LIVE KiCad MCP server: the model authors the board with real tools, the harness executes it on a real KiCad 10 engine, and ERC/DRC plus structural checks (nets, components, outline, netclass, hierarchy) gate the pass. 4-tier difficulty from schematic-only to a full fabrication package.", "url": "", "tag": "first-party", "note": "The only benchmark on this page that measures AGENTIC hardware design — building a board with tools, not answering trivia about one. Our leaderboard is below. Grok 4.5 currently leads at 0.973 composite; see the full table." }, { "cat": "hardware", "name": "PCB-QA", "org": "Srinivasan, Tan, Turnbull & Pearce (arXiv:2606.23704)", "measures": "480 question-answer pairs from 8 open-source hardware projects, covering component connections, datasheet reading and SPICE simulation. The first PCB question-answering benchmark.", "url": "https://arxiv.org/abs/2606.23704", "tag": "research", "note": "Verified scores (Jun 2026): Gemini 3 Flash 93% (JSON format) · Claude Sonnet 4.6 90% · GPT-5.4 Nano 88% · Llama 3.3 70B 84%. JSON prompting beats native-KiCad format." }, { "cat": "hardware", "name": "PCBBench / PCBSchemaGen", "org": "HZou9 et al. (arXiv:2602.00510)", "measures": "227 real-IC schematic-generation tasks across 22 circuit domains, verified by a 5-layer deterministic structural checker (PCBs have no unit tests, so correctness is proven structurally).", "url": "https://arxiv.org/abs/2602.00510", "tag": "research", "note": "Verified scores (Jan 2026): GPT-5.4 94.0% · Gemini 3.1 Pro 94.3% · DeepSeek V3.2 81.6% · Gemma-4-31B 81.3% (best local) · MiniMax M2.5 76%. The \"hard\" tier is the real separator." }, { "cat": "hardware", "name": "SMPS Benchmark", "org": "arXiv:2507.10639", "measures": "269 switched-mode power-supply design tasks (parameter tuning, topology adaptation, optimization) with SPICE simulation feedback available in the loop.", "url": "https://arxiv.org/abs/2507.10639", "tag": "research", "note": "Headline result (Jul 2025): SPICE feedback + reasoning lifts solve rate from 15% to 91% (o3 + RAG + SPICE 90.7%). Tool-in-the-loop beats raw model size for EE work." }, { "cat": "hardware", "name": "CircuChain", "org": "arXiv:2602.15037", "measures": "Diagnostic circuit-analysis benchmark: 100 Control/Trap problem pairs across 5 canonical topologies, built to separate genuine competence from pattern-matched compliance.", "url": "https://arxiv.org/abs/2602.15037", "tag": "research", "note": "Trap pairs expose models that recite the textbook answer without actually analysing the circuit in front of them." }, { "cat": "hardware", "name": "PowerAgentBench-Dyn", "org": "arXiv:2606.20401", "measures": "Agentic AI on power-system dynamic-analysis tasks — grid and power engineering rather than PCB design.", "url": "https://arxiv.org/abs/2606.20401", "tag": "research", "note": "One of the five EE replica suites in our Konduit composite (poweragentbench_dyn)." }, { "cat": "hardware", "name": "RadioBench", "org": "arXiv:2606.01862", "measures": "The first benchmark for LLM-driven radio-signal generation — RF and communications design.", "url": "https://arxiv.org/abs/2606.01862", "tag": "research", "note": "One of the five EE replica suites in our Konduit composite (radiobench)." }, { "cat": "hardware", "name": "CVDP (Verilog)", "org": "NVIDIA", "measures": "Digital-hardware / Verilog design — the RTL side of hardware, adjacent to our analog and PCB focus.", "url": "", "tag": "public", "note": "Included for completeness: the digital-design family, distinct from schematic/PCB work." }, { "cat": "realworld", "name": "OSRSBench", "org": "Community / emerging", "measures": "An agent's ability to play Old School RuneScape — a long-horizon, GUI-driven video game demanding navigation, resource management and multi-step questing over extended play.", "url": "", "tag": "emerging", "note": "🔴 EMERGING benchmark: we list it because it exemplifies the real-world direction, but treat its methodology and any scores as unconfirmed until its own repo/site is cited. No numbers published here." }, { "cat": "realworld", "name": "OSWorld", "org": "OSWorld team", "measures": "369 real computer-desktop tasks across Ubuntu and Windows applications; open-ended GUI computer-use with execution-based scoring on the actual resulting system state.", "url": "https://os-world.github.io/", "tag": "public", "note": "The leading general computer-use benchmark — operate a real desktop, get graded on the result." }, { "cat": "realworld", "name": "AndroidWorld", "org": "Google Research", "measures": "116 tasks across 20 real Android apps; mobile GUI agents rewarded on the app's actual resulting state.", "url": "https://github.com/google-research/android_world", "tag": "public", "note": "The phone equivalent of OSWorld: real apps, real state-based reward." }, { "cat": "realworld", "name": "Voyager / MineDojo", "org": "MineDojo", "measures": "Open-ended embodied agents in Minecraft — skill acquisition, exploration and long-horizon goal pursuit across thousands of open tasks.", "url": "https://voyager.minedojo.org/", "tag": "public", "note": "The canonical open-ended embodied-learning environment; lifelong skill-building, not a fixed score." }, { "cat": "realworld", "name": "TheAgentCompany", "org": "TheAgentCompany", "measures": "An agent completes tasks inside a simulated software company — coding, browsing, internal chat, HR workflows — scored on the fraction of real job workflows finished.", "url": "https://the-agent-company.com/", "tag": "public", "note": "\"Can it do a real knowledge-worker's job\" — the most economically-framed of the real-world suites." }, { "cat": "realworld", "name": "Factorio Learning Environment", "org": "Hopkins et al.", "measures": "Long-horizon automation and planning in the game Factorio — open-ended factory throughput requiring deep sequential planning.", "url": "https://github.com/JackHopkins/factorio-learning-environment", "tag": "public", "note": "Perhaps the purest long-horizon planning stress test: build a self-expanding factory." }, { "cat": "realworld", "name": "SWE-Lancer", "org": "OpenAI", "measures": "1,400+ real freelance software-engineering gigs priced in real US dollars; models are scored on earnings — did the delivered work pass the client's acceptance test.", "url": "https://github.com/openai/SWELancer-Benchmark", "tag": "public", "note": "Real jobs, real money as the yardstick — the sharpest bridge from benchmark to economic value." } ], "konduit_leaderboard": { "benchmark": "Konduit KiCadBench + EE Replica Suites (evaluator v2)", "tool": "DCENT_Konduit (first-party KiCad MCP server)", "as_of": "2026-07-09", "weights": "composite = 0.6 KiCadBench (live MCP board-build, ERC/DRC-gated) + 0.4 EE replica suites (PCB-QA, SMPS, CircuChain, PowerAgentBench-Dyn, RadioBench)", "rows": [ { "model": "Grok 4.5", "org": "xAI", "open": false, "ctx": "2M", "composite": "0.973", "kicad": "5/5", "replicas": "0.93", "grade": "A", "note": "The only model to clear all five live KiCad build tasks under v2 structural gates AND score 0.93 on the EE suites. Full coverage." }, { "model": "Claude Opus 4.8", "org": "Anthropic", "open": false, "ctx": "1M", "composite": "0.680", "kicad": "3/5", "replicas": "0.80", "grade": "B", "note": "Strongest Claude on live board-build and the most tool-efficient (19 calls/task avg); the USB-C power-entry task fails its required VBUS net labels." }, { "model": "Claude Haiku 4.5", "org": "Anthropic", "open": false, "ctx": "200K", "composite": "0.587", "kicad": "2/5", "replicas": "0.87", "grade": "C", "note": "Punches above its weight on the EE suites (0.87) for a small, fast model; struggles on multi-constraint board builds." }, { "model": "Claude Sonnet 5", "org": "Anthropic", "open": false, "ctx": "1M", "composite": "0.373", "kicad": "0/5", "replicas": "0.93", "grade": "C", "note": "Tied for top on the EE replica suites (0.93) but scored 0/5 on live builds — incomplete plan rounds under a subscription session limit plus stricter v2 gates. Reads as a harness-coverage gap, not a capability floor; flagged for re-run." }, { "model": "Claude Fable 5", "org": "Anthropic", "open": false, "ctx": "200K+", "composite": "—", "kicad": "n/a", "replicas": "1.00", "grade": "—", "note": "Perfect 5/5 (100%) on the EE replica suites but UNRANKED — the live board-build leg never ran (session limit hit mid-run). Partial coverage, not comparable to full-coverage rows until it completes." } ] }, "context": { "why_taxonomy": "This page is a MAP of how AI models are measured, not a frozen scoreboard. Frontier scores change with almost every model release, and the public leaderboards are the authoritative, always-current source — so for the Coding, Agentic, Reasoning and Vision categories we link each benchmark's LIVE leaderboard rather than freeze a number that would be wrong within weeks. Publishing stale scores as if current is exactly the kind of thing this site refuses to do.", "hardware_edge": "The exception is Hardware & PCB design. There we publish hard numbers, because the data is ours: D-Central built both the benchmark (Konduit KiCadBench) and the tool it runs on (DCENT_Konduit, our KiCad MCP server). No general LLM leaderboard measures whether a model can actually design a circuit board — read a datasheet, generate a schematic, place and route it, pass ERC/DRC. We do, and the results below exist nowhere else.", "konduit_caveat": "🔴 The Konduit leaderboard is an early PREVIEW: five live KiCad build tasks plus five clean-room EE suites, run through subscription-based harnesses. Evaluator v2 gates a pass on the board's declared structural checks (nets, components, outline, netclass, hierarchy, ERC/DRC), not bare exit codes. Preview scores are NOT fab-ready claims — every board still needs human EE review before manufacturing. The task set is small and growing; treat rankings as directional and expect movement as we add tasks and re-run incomplete rows.", "verdict": "Hardware design is the least-saturated, least-measured frontier in AI evaluation — and the one D-Central is built to own. Two findings already stand out from our data: Grok 4.5 leads our Konduit board (0.973) as the only model to clear every live board-build task, and across the EE literature the biggest lever is not model size but tool-in-the-loop feedback — SPICE and ERC signals lift solve rates from ~15% to ~91%. The map matters more than any single cell in it: it shows where the industry measures heavily (coding, knowledge) and where it barely measures at all (hardware, long-horizon real-world work)." } }