Chen et al. (2021) — ARXIV-2107.03374
What it measures
Functional correctness: a generated program is right if it passes the unit tests, not if it resembles a reference solution. The paper is explicit that this is the point of departure from “standard match-based metrics”, and it is why HumanEval displaced BLEU-style code evaluation.
164 hand-written problems, each a docstring plus tests. Hand-written deliberately — the paper built them rather than scraping, because a scraped Python benchmark is contaminated by construction against a model trained on GitHub.
What pass@k is, and why the estimator matters
pass@k is the probability that at least one of k samples passes. The naive
measurement — draw k, check, repeat — is what the paper argues against:
While the top expression may look correct, it underestimates the true value by a considerable margin. The unbiased estimator may have a slightly higher variance initially but allows for a fair comparison across different numbers of samples.
So they generate n ≥ k samples (n=200, k≤100), count the c that pass, and
compute 1 − C(n−c, k) / C(n, k).
This is the estimator SOTA-210 calls “the standard one” without saying whose. That practice also compresses this paper’s variance argument into a clause. The convention it recommends is defined here.
What it is insensitive to
- Single functions from docstrings. Not architecture, not multi-file change, not debugging existing code.
- The paper names its own model’s failure modes, which double as the benchmark’s blind spots at the time: “difficulty with docstrings describing long chains of operations and with binding operations to variables.”
- Tests certify behaviour, not quality. A solution that passes can be unreadable, quadratic, or unsafe, and the metric cannot tell.
- 164 problems is small, so differences of a point or two are noise.
And tests do not always certify behaviour either
The bullets above are this paper’s own account of its limits. One more was measured from outside it, and it is sharper than any of them.
Li et al. (LIT-650) took fifty problems their model solved on
HumanEval and examined one passing solution for each by hand. Thirty per
cent were false positives — programs that pass every test and are not
correct. HumanEval carries 7.77 tests per problem; the same measurement
puts APPS at 60% on 20.99 tests, and the dataset they built to avoid the
problem at 4% on 203.7.
That comparison is LIT-650’s, run by one procedure across three datasets, and HumanEval sits inside it at half APPS’s false-positive rate on about a third as many tests — the direction the paper’s own caveat, fewer samples on simpler problems, would push it.
The cause is not carelessness in this benchmark specifically. It is that
“input/output examples are an under-specification of program behavior”, so
test count is the binding constraint and 7.77 is a small number. A passing
pass@k is the probability that at least one sample passes the tests, and
the tests are what the measurement above is about.
Three qualifications, because this number travels badly. It is fifty
problems, checked once, in 2022, against one 1B model’s outputs — a
measurement of test coverage, not a correction to any published score. The
comparison was run unfavourably to their own dataset: 200 samples per
HumanEval problem against a million per CodeContests problem, and fewer
samples on simpler problems both push the rate down. And 30% of solved
problems being false positives does not mean a reported pass@k is 30% too
high, because the arithmetic depends on how the errors distribute over
problems — what it means is that the benchmark’s ceiling is lower than its
tests suggest, by an amount nobody has bounded tightly.
Standing in the anthology
Filed under ADR-032, which admits benchmarks and says a benchmark note will usually carry no practice. This one is admitted for a stronger reason than the usual: the record recommends its metric. HumanEval appears in five practices and twelve documents, and until now the anthology had the convention and not the paper that defines it — the trunk shape DP-007 describes, where what everyone agrees on has no arrival event.
Carries no practice of its own; SOTA-210 is where the recommendation lives, and this note is what it should have been citing.