HumanEval Benchmark

HumanEval is a benchmark for evaluating the functional correctness of program code generated by language models from a natural language description and a function signature. It was introduced in July 2021 by OpenAI researchers led by Mark Chen and has become one of the key standards for measuring the functional correctness of generated programs.

The development of HumanEval was driven by the need for a reliable way to evaluate code generation. Earlier, the quality of code produced by language models was often assessed with indirect metrics such as BLEU or by manual inspection, neither of which reliably reflected whether a program actually worked. HumanEval addresses this by focusing on functional correctness: generated code is judged not by its textual similarity to a reference solution, but by whether it passes a set of automated unit tests.

Structure of the Task Set

The benchmark consists of 164 programming problems, hand-written for this dataset to reduce the likelihood of overlap with existing code corpora — although hand-writing does not guarantee that the problems are absent from a model's training data, especially for models trained after the benchmark's public release. All problems are written in Python and presented as code with a natural language specification.

In the released dataset, each item is stored with the fields task_id, prompt, canonical_solution, test, and entry_point. From the model's point of view, each problem provides:

• Function signature: the function's name and parameters.
• Text description: an English-language docstring describing the required behaviour.
• Function body: the empty space that the model must fill with generated code.

The following elements are hidden from the model and used only for grading:

• Canonical solution: a correct reference implementation.
• Unit tests: used to automatically verify the generated code; each problem has an average of about 7.7 tests, covering both typical and edge cases.

The problems span a range of topics — basic language constructs, algorithms, and simple mathematics — which makes the dataset varied without being exhaustive.

Model Evaluation Methodology

The primary success metric on HumanEval is pass@k, which measures the fraction of problems for which a model produces at least one correct solution within k generated samples. A problem counts as solved if the generated code passes all of its unit tests.

• pass@1: the most commonly reported figure — the fraction of problems solved by a single generated sample (under fixed decoding parameters). It is often described informally as solving a problem "on the first attempt", but strictly it refers to one sample rather than one of several tries.
• pass@k: the fraction of problems for which at least one of k generated samples passes all tests. For example, pass@10 reflects the model's success when allowed up to ten samples per problem.

Because a naive estimate of pass@k from the observed success rate is biased, the authors proposed an unbiased estimator: for each problem they generate n ≥ k samples, count the c that pass all tests, and average the quantity below over all problems:

$1-\frac{(\genfrac{}{}{0ex}{}{n-c}{k})}{(\genfrac{}{}{0ex}{}{n}{k})}$

Evaluation is performed by executing the generated code against the hidden tests. Because running model-generated code is inherently unsafe, the official evaluation harness disables code execution by default; the user must enable it and is responsible for providing a suitable sandbox (for example, a container) — a security precaution rather than a mere implementation detail.

Results and Impact on the Industry

The first experiments on HumanEval revealed a large gap between general-purpose models and models trained specifically on code.

• In 2021, OpenAI's Codex model (12 billion parameters, trained on GitHub code) solved about 28.8 % of the problems on the first sample (pass@1), compared with 11.4 % for GPT-J.
• The much larger general-purpose model GPT-3 (175 billion parameters), which was not trained on code, solved 0 %.

These results underscored how important specialised training on code is for successful generation, and HumanEval quickly became a standard reference for comparing new models. When citing later results, however, it is essential to state the exact model version, decoding setup, and evaluation harness, because published figures for the same model vary widely.

• In the GPT-4 Technical Report (March 2023), OpenAI reported 48.1 % pass@1 for GPT-3.5 and 67.0 % for GPT-4 on HumanEval in a 0-shot setting.
• Later third-party evaluations of newer API snapshots reported higher numbers — for example, an August 2023 re-test measured about 72 % for the updated GPT-3.5-Turbo and about 82 % for GPT-4 — but these are not directly comparable to the official figures, as they use different snapshots, prompts, and evaluation harnesses.
• Open-source code models proved competitive with early proprietary ones: Code Llama-Python 34B reached about 53.7 % pass@1, and WizardCoder-15B about 57.3 % (with the later WizardCoder-Python-34B reaching 73.2 %), all clearly surpassing the original Codex-12B.

By the mid-2020s, the original HumanEval had become largely saturated for frontier and code-specialised models, with the strongest systems clustering near the top of the scale (exact figures vary by model snapshot and evaluation harness). Evaluation increasingly shifted toward stricter variants such as HumanEval+ and more realistic software-engineering benchmarks, and the original HumanEval now serves mainly as a historical reference point rather than a discriminating one.

Extensions and Variants of the Benchmark

The success of HumanEval inspired several derivative benchmarks that test models under broader conditions.

• HumanEval+ (EvalPlus): an extension (2023) that augments each HumanEval problem with far more tests (on the order of 80× the original). It was designed to catch incorrect solutions that pass the sparse original tests. Adding these tests can substantially lower reported scores and even change the relative ranking of models, with the magnitude of the drop depending on the model and evaluation setup, which exposes the weakness of the original test coverage; HumanEval+ has since become a de-facto standard for stricter evaluation alongside the original.
• HumanEval-X: a multilingual benchmark (2023) built for CodeGeeX, extending the original problems with hand-written solutions and tests in C++, Java, JavaScript, and Go (in addition to Python).
• Multilingual HumanEval: part of a broader effort (Athiwaratkun et al., 2022) that introduced the MBXP, Multilingual HumanEval, and MathQA-X benchmarks. It uses an automated conversion framework to transpile the original Python problems into more than ten programming languages.
• MultiPL-E: a scalable, test-driven framework (Cassano et al., 2022) that translates HumanEval (and MBPP) into up to about 19 programming languages while keeping prompts and test suites parallel.
• CL-HumanEval: a cross-lingual variant (2024) in which the original problems are adapted to test a model's ability to understand descriptions in languages other than English while still generating Python code.
• HumanEval-XL: a large-scale benchmark (2024) that combines both directions. It links 23 natural languages (including Russian, Chinese, and Arabic) with 12 programming languages, for a total of 22,080 "description–code" prompts with an average of 8.33 tests each.
• Even broader multilingual extensions have followed, such as mHumanEval (Raihan et al., 2025), which scales the benchmark to over 200 natural languages.

Limitations

Despite its influence, HumanEval has well-documented limitations and is no longer considered sufficient on its own.

• Weak test coverage. Passing the original tests does not prove that a program is correct. As HumanEval+ showed, adding more tests reveals faulty solutions that slip through the sparse original suite.
• Data contamination. Because the benchmark was published openly on GitHub, its problems may have been absorbed into the training data of later models, which can inflate their scores.
• Narrow format. The tasks are short, standalone Python functions synthesised from a docstring. They do not involve large codebases, multi-file reasoning, dependency management, bug fixing, or production constraints — the kinds of tasks targeted by later benchmarks such as SWE-bench.
• Small size and single language. With only 164 Python problems, the benchmark covers a limited slice of programming, which motivated the multilingual and enlarged variants above.
• Saturation. Because the strongest current models cluster near the top of the scale, the original HumanEval no longer discriminates well between them, which is why newer work relies on stricter variants such as HumanEval+.

External links

• Official HumanEval repository (OpenAI)
• HumanEval dataset on Hugging Face
• HumanEval page on Papers with Code

See also

• SWE-bench (benchmark)
• MMLU Benchmark
• LLM benchmarks
• LLM evaluation

Literature

• Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374.
• Austin, J. et al. (2021). Program Synthesis with Large Language Models (MBPP). arXiv:2108.07732.
• Liu, J. et al. (2023). Is Your Code Generated by ChatGPT Really Correct? (EvalPlus / HumanEval+). arXiv:2305.01210.
• Athiwaratkun, B. et al. (2022). Multi-lingual Evaluation of Code Generation Models. arXiv:2210.14868.
• Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
• Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.
• Ni, S. et al. (2025). A Survey on Large Language Model Benchmarks. arXiv:2508.15361.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.