SWE-bench Verified

SWE-bench Verified is a human-validated subset of 500 tasks drawn from the SWE-bench software engineering benchmark, created to more reliably evaluate the ability of large language models (LLMs) and coding agents to resolve real-world software issues. It was released on August 13, 2024 by the OpenAI Preparedness team in collaboration with the original Princeton authors of SWE-bench, and was designed to remove tasks that in the original benchmark were under-specified, incorrectly graded, or unsolvable as posed. From late 2024 through early 2026 it became the de-facto standard for reporting coding-agent performance, superseding the original SWE-bench and SWE-bench Lite test sets. In February 2026, however, OpenAI stopped reporting SWE-bench Verified scores, citing widespread training-data contamination and a high proportion of flawed test cases, and recommended that other developers move to harder, contamination-resistant successors such as the public split of SWE-bench Pro.

Background and Motivation

The original SWE-bench, introduced by Princeton researchers in 2023, evaluates models on 2,294 real GitHub issues paired with their fixing pull requests, drawn from 12 popular open-source Python repositories. Each task gives the model an issue description and a repository snapshot and asks it to produce a patch that makes the project's hidden tests pass. As models began to be evaluated on it, OpenAI and others found that a substantial share of its tasks were problematic: some hidden unit tests were so specific that they rejected otherwise-correct patches, some issue descriptions were too vague to solve unambiguously, and some tasks could not be solved because of environment-setup problems that made valid solutions fail.

OpenAI's interest in the benchmark came from its Preparedness Framework, under which the ability to autonomously complete software-engineering tasks is treated as a leading indicator of model autonomy and a component of the framework's model-autonomy risk category. Because an over- or under-stated autonomy score could distort risk assessments, the team concluded that the evaluation itself needed to be cleaned up, which motivated the creation of SWE-bench Verified.

Construction and Methodology

To build Verified, OpenAI ran a human-annotation campaign with 93 professional software developers experienced in Python, who screened 1,699 randomly sampled SWE-bench tasks — each reviewed independently by three annotators — for well-specified problem statements and fairly scoped tests. Samples flagged as under-specified, unfairly tested, or otherwise problematic were discarded (about 68% of those reviewed), leaving a curated set of 500 tasks, still drawn from the same 12 Python repositories, for which a passing patch can be interpreted with substantially higher confidence than in the original benchmark — though a later OpenAI audit showed that some flawed tests remained. Alongside the subset, OpenAI released human annotations for the full SWE-bench test set, allowing tasks to be sliced by difficulty: the "easy" tier comprises 196 tasks estimated to take a developer under 15 minutes, while the "hard" tier comprises 45 tasks estimated to take over an hour. The team also collaborated with the SWE-bench authors on a new Docker-based evaluation harness that runs each task in a containerized environment to make results easier to reproduce.

Evaluation follows the same pattern as SWE-bench: a model or agent generates a patch, which is applied and tested against the task's fail-to-pass tests (which must newly pass) and pass-to-pass regression tests (which must keep passing). The standard metric is pass@1; because model outputs are stochastic, it is generally computed by sampling several candidate patches per task and averaging the per-instance success rate, with OpenAI's system cards reporting an average over four to five attempts. OpenAI's initial implementation used the open-source Agentless scaffold because its early models lacked code-execution tools; because a reported score reflects both the underlying model and its surrounding agentic scaffold, different laboratories evaluate under different harnesses, which complicates direct comparison.

Adoption and Results

SWE-bench Verified quickly became the dominant reporting target for frontier coding models. At launch in August 2024, OpenAI's GPT-4o resolved 33.2% of the tasks using the best available open-source scaffold, roughly double its score on the original SWE-bench. Scores then climbed steeply, but by early 2026 progress at the top had slowed: OpenAI reported that state-of-the-art results had risen only from about 75% to 81% over the preceding six months, a sign that the benchmark was saturating at the frontier. On tracked leaderboards several leading models from different laboratories clustered within about a point of one another near 80%, so the metric no longer clearly separated the strongest systems. Higher vendor-reported figures — into the high-80% to roughly 90% range — also circulated, but because they used different scaffolds, tools, and evaluation budgets they were not directly comparable with the tracked standardized results.

Contamination, Test Flaws, and Deprecation

Two structural problems ultimately undermined the benchmark. The first was defective tests. In early 2026, OpenAI audited 138 problems that its o3 model had failed to solve consistently across 64 independent runs (a 27.6% subset of the set), with each case reviewed by at least six engineers, and reported that 59.4% contained material flaws in their tests or problem statements. These fell into two main types: "narrow" tests (35.5% of the audited tasks) that enforce specific implementation details and reject otherwise-correct solutions, and "wide" tests (18.8%) that check for behavior never mentioned in the problem description; the remaining 5.1% had miscellaneous issues.

The second problem was training-data contamination. Because Verified's tasks come from public Python repositories whose issues and fixes predate the benchmark, the corresponding code entered model training data. Using an automated red-teaming setup — in which a prober model supplied task identifiers and, in some cases, hints drawn from the description — OpenAI found that every frontier model it tested (including GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash) could reproduce verbatim gold patches or problem-specific details for some tasks; in the most extreme case a model output an exact reference diff from the task identifier alone. OpenAI concluded that gains on the benchmark increasingly reflected exposure at training time rather than real ability, and that models which had seen a task during training were more likely to pass its under-specified tests.

On February 23, 2026, OpenAI announced that it had stopped reporting SWE-bench Verified scores and recommended that other developers do the same, pointing them instead to the public split of SWE-bench Pro and to privately authored evaluations such as GDPval. The large per-model gap between the two benchmarks — where the same frontier models score far lower on Pro's contamination-resistant public split, often by 30 or more percentage points — was cited as direct evidence of the distortion.

Significance

Despite its retirement as a frontier metric, SWE-bench Verified had a lasting influence on how AI coding ability is measured. It established human validation of tasks and containerized, reproducible harnesses as expectations for serious code benchmarks, and for over a year it was the single number most often cited to compare coding agents. Its decline also became an instructive case study in benchmark decay: OpenAI framed the episode as a lesson that benchmarks built from public code carry inherent contamination risk and that automated test-based grading is difficult to get right, and it accelerated the field's move toward contamination-resistant and privately authored evaluations.

External links

• Introducing SWE-bench Verified — OpenAI
• Why SWE-bench Verified no longer measures frontier coding capabilities — OpenAI
• SWE-bench Verified page and leaderboard
• SWE-bench Verified dataset on Hugging Face

Literature

• Jimenez, C. E. et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. arXiv:2310.06770.
• Deng, X. et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?. arXiv:2509.16941.