HumanEval

HumanEval is a code-generation benchmark released by OpenAI in 2021 alongside the Codex paper. It contains 164 hand-written Python programming problems, each consisting of a function signature, a natural-language docstring describing the task, and a set of hidden unit tests. Rather than comparing the model's text to a reference answer, HumanEval executes the generated code and checks whether it passes every test, making it a measure of functional correctness rather than surface similarity — does the code actually work, not does it look right.

The pass@k metric

HumanEval popularized the pass@k metric. The model generates k independent solutions for each problem, and a problem counts as solved if at least one sample passes all its tests. Common reports are pass@1 (a single attempt), pass@10, and pass@100. Pass@1 reflects how reliably a model writes correct code on the first try — the number that matters most for an assistant you actually work with — while higher k values reveal whether a correct solution exists somewhere in the model's distribution even if it is not the most likely output. The paper also introduced an unbiased estimator for pass@k, since naively sampling exactly k solutions gives a noisy picture. Each problem ships with an average of roughly 7.7 unit tests, so passing is meaningfully harder than producing something that superficially resembles a solution.

Strengths and limits

Because it runs real tests, HumanEval rewards genuinely working code, which makes it far harder to game than overlap-based text metrics — and its execution-based design became the template for nearly every coding benchmark that followed. Its limits are equally clear. First, scope: 164 short, self-contained Python functions say nothing about navigating large codebases, multi-file changes, debugging, or languages other than Python. Second, contamination: the problems have circulated publicly for years and are widely present in training data, so a high score may partly reflect memorization; treat small score differences between models as noise. Successor suites address both problems — HumanEval+ adds far more rigorous test cases (exposing solutions that passed the original's thin tests), MBPP broadens the problem pool, and later benchmarks move toward realistic repository-level tasks. By the mid-2020s, frontier models were scoring high enough that HumanEval stopped discriminating between them, pushing serious evaluation toward those harder suites.

Using it when choosing a local model

For a sovereign operator picking a coding model to run on their own hardware, HumanEval-family scores remain a useful first filter, read with care. Compare pass@1 under the same conditions (sampling settings and prompts materially change results), prefer the stricter HumanEval+ variant where reported, and remember that quantization can shave real points off coding accuracy — a benchmark run on full-precision weights may flatter the GGUF you actually deploy through Ollama or llama.cpp. Then do what the benchmark itself teaches: test functional correctness on your own problems, because your workflow is the only benchmark with zero contamination. Read HumanEval next to general-knowledge tests like the MMLU benchmark and human-preference rankings such as Chatbot Arena Elo for a rounded picture of any model you plan to make part of your local inference stack.

Building your own micro-benchmark

The most durable lesson HumanEval offers self-hosters is methodological: evaluation means executing code against tests, and you can apply that at kitchen-table scale. Collect ten or fifteen real tasks from your own work — the shell one-liners, the config parsers, the API calls you actually write — pair each with a couple of assertions, and run every candidate model through them at the same settings, quantization, and temperature you will use in production. Score pass@1, nothing fancier. An afternoon of this produces a private benchmark with zero training-data contamination and perfect relevance to your workload, which is more than can be said for any public leaderboard. Keep the suite; rerun it whenever a new model or a new fine-tuning tempts you to switch.