---
title: "SWE-bench Verified"
source: "https://systems-analysis.info/eng/SWE-bench_Verified"
wiki: "systems-analysis.info/eng"
article: "SWE-bench_Verified"
language: "en"
categories:
  - "Category:English"
  - "Category:Large language models"
  - "Category:LLM benchmarks"
  - "Category:Machine learning"
  - "Category:Technology"
revision_id: 335
wiki_created_at: 2026-09-06T22:22:15Z
wiki_modified_at: 2026-09-06T22:22:15Z
downloaded_at: 2026-09-07T22:22:46Z
---

# SWE-bench Verified

**SWE-bench Verified** is a **human-validated** subset of **500 tasks** drawn from the **[SWE-bench](https://systems-analysis.info/eng/SWE-bench_(benchmark) "SWE-bench (benchmark)")** software engineering benchmark, created to more reliably evaluate the ability of [**large language models** (**LLMs**](https://systems-analysis.info/eng/Large_language_model "Large language model")) and **coding agents** to resolve real-world software issues<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)[\[2\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-swebench-verified-2)</sup>. It was released on **August 13, 2024** by the **OpenAI** Preparedness team in collaboration with the original **Princeton** authors of SWE-bench, and was designed to remove tasks that in the original benchmark were **under-specified**, **incorrectly graded**, or **unsolvable as posed**<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)[\[3\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-o1-systemcard-3)</sup>. From late 2024 through early 2026 it became the **de-facto standard** for reporting coding-agent performance, superseding the original SWE-bench and SWE-bench Lite test sets<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)</sup>. In **February 2026**, however, OpenAI stopped reporting SWE-bench Verified scores, citing widespread **training-data contamination** and a high proportion of **flawed test cases**, and recommended that other developers move to harder, contamination-resistant successors such as the public split of [SWE-bench Pro](https://systems-analysis.info/eng/SWE-bench_Pro "SWE-bench Pro")<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>.

## Background and Motivation

The original **[SWE-bench](https://systems-analysis.info/eng/SWE-bench_(benchmark) "SWE-bench (benchmark)")**, introduced by Princeton researchers in 2023, evaluates models on **2,294 real GitHub issues** paired with their fixing pull requests, drawn from **12 popular open-source Python repositories**<sup>[\[5\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-jimenez-swebench-5)</sup>. Each task gives the model an issue description and a repository snapshot and asks it to produce a **patch** that makes the project's hidden tests pass<sup>[\[5\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-jimenez-swebench-5)</sup>. As models began to be evaluated on it, OpenAI and others found that a substantial share of its tasks were **problematic**: some hidden unit tests were so specific that they rejected otherwise-correct patches, some issue descriptions were too vague to solve unambiguously, and some tasks could not be solved because of environment-setup problems that made valid solutions fail<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)[\[3\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-o1-systemcard-3)</sup>.

OpenAI's interest in the benchmark came from its **Preparedness Framework**, under which the ability to autonomously complete software-engineering tasks is treated as a **leading indicator of model autonomy** and a component of the framework's model-autonomy risk category<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)</sup>. Because an over- or under-stated autonomy score could distort risk assessments, the team concluded that the evaluation itself needed to be cleaned up, which motivated the creation of SWE-bench Verified<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)</sup>.

## Construction and Methodology

To build Verified, OpenAI ran a human-annotation campaign with **93 professional software developers** experienced in Python, who screened **1,699** randomly sampled SWE-bench tasks — each reviewed independently by **three annotators** — for well-specified problem statements and fairly scoped tests<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)[\[2\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-swebench-verified-2)</sup>. Samples flagged as under-specified, unfairly tested, or otherwise problematic were discarded (about **68%** of those reviewed), leaving a curated set of **500 tasks**, still drawn from the same 12 Python repositories, for which a passing patch can be interpreted with **substantially higher confidence** than in the original benchmark — though a later OpenAI audit showed that some flawed tests remained<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>. Alongside the subset, OpenAI released **human annotations** for the full SWE-bench test set, allowing tasks to be sliced by difficulty: the **"easy"** tier comprises 196 tasks estimated to take a developer under 15 minutes, while the **"hard"** tier comprises 45 tasks estimated to take over an hour<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)</sup>. The team also collaborated with the SWE-bench authors on a new **Docker-based evaluation harness** that runs each task in a containerized environment to make results easier to reproduce<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)</sup>.

Evaluation follows the same pattern as SWE-bench: a model or agent generates a patch, which is applied and tested against the task's **fail-to-pass** tests (which must newly pass) and **pass-to-pass** regression tests (which must keep passing)<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)</sup>. The standard metric is **pass@1**; because model outputs are stochastic, it is generally computed by sampling several candidate patches per task and averaging the per-instance success rate, with OpenAI's system cards reporting an average over four to five attempts<sup>[\[3\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-o1-systemcard-3)</sup>. OpenAI's initial implementation used the open-source **Agentless** scaffold because its early models lacked code-execution tools; because a reported score reflects both the underlying model and its surrounding **agentic scaffold**, different laboratories evaluate under different harnesses, which complicates direct comparison<sup>[\[3\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-o1-systemcard-3)[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)</sup>.

## Adoption and Results

SWE-bench Verified quickly became the **dominant reporting target** for frontier coding models. At launch in August 2024, OpenAI's **GPT-4o** resolved **33.2%** of the tasks using the best available open-source scaffold, roughly double its score on the original SWE-bench<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)</sup>. Scores then climbed steeply, but by early 2026 progress at the top had **slowed**: OpenAI reported that state-of-the-art results had risen only from about **75% to 81%** over the preceding six months, a sign that the benchmark was **saturating** at the frontier<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>. On tracked leaderboards several leading models from different laboratories clustered within about a point of one another near **80%**, so the metric no longer clearly separated the strongest systems<sup>[\[6\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-morph-swebench-6)</sup>. Higher **vendor-reported** figures — into the high-80% to roughly 90% range — also circulated, but because they used different scaffolds, tools, and evaluation budgets they were not directly comparable with the tracked standardized results<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)[\[6\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-morph-swebench-6)</sup>.

## Contamination, Test Flaws, and Deprecation

Two structural problems ultimately undermined the benchmark. The first was **defective tests**. In early 2026, OpenAI audited **138 problems** that its o3 model had failed to solve consistently across 64 independent runs (a **27.6%** subset of the set), with each case reviewed by at least six engineers, and reported that **59.4%** contained material flaws in their tests or problem statements<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>. These fell into two main types: **"narrow" tests** (35.5% of the audited tasks) that enforce specific implementation details and reject otherwise-correct solutions, and **"wide" tests** (18.8%) that check for behavior never mentioned in the problem description; the remaining 5.1% had miscellaneous issues<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>.

The second problem was **training-data contamination**. Because Verified's tasks come from public Python repositories whose issues and fixes predate the benchmark, the corresponding code entered model training data. Using an automated **red-teaming** setup — in which a prober model supplied task identifiers and, in some cases, hints drawn from the description — OpenAI found that every frontier model it tested (including **GPT-5.2**, **Claude Opus 4.5**, and **Gemini 3 Flash**) could reproduce **verbatim gold patches** or problem-specific details for some tasks; in the most extreme case a model output an exact reference diff from the task identifier alone<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>. OpenAI concluded that gains on the benchmark increasingly reflected **exposure at training time** rather than real ability, and that models which had seen a task during training were more likely to pass its under-specified tests<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>.

On **February 23, 2026**, OpenAI announced that it had **stopped reporting** SWE-bench Verified scores and recommended that other developers do the same, pointing them instead to the public split of **[SWE-bench Pro](https://systems-analysis.info/eng/SWE-bench_Pro "SWE-bench Pro")** and to privately authored evaluations such as **GDPval**<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>. The large per-model gap between the two benchmarks — where the same frontier models score far lower on Pro's contamination-resistant public split, often by 30 or more percentage points — was cited as direct evidence of the distortion<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)[\[6\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-morph-swebench-6)</sup>.

## Significance

Despite its retirement as a frontier metric, SWE-bench Verified had a lasting influence on how AI coding ability is measured. It established **human validation** of tasks and **containerized, reproducible harnesses** as expectations for serious code benchmarks, and for over a year it was the single number most often cited to compare coding agents<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-verified-1)[\[6\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-morph-swebench-6)</sup>. Its decline also became an instructive **case study** in benchmark decay: OpenAI framed the episode as a lesson that benchmarks built from public code carry inherent **contamination risk** and that automated test-based grading is difficult to get right, and it accelerated the field's move toward **contamination-resistant** and **privately authored** evaluations<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Verified#cite_note-openai-deprecation-4)</sup>.

## External links

- <a href="https://openai.com/index/introducing-swe-bench-verified/" class="external text" rel="nofollow">Introducing SWE-bench Verified — OpenAI</a>
- <a href="https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/" class="external text" rel="nofollow">Why SWE-bench Verified no longer measures frontier coding capabilities — OpenAI</a>
- <a href="https://www.swebench.com/verified.html" class="external text" rel="nofollow">SWE-bench Verified page and leaderboard</a>
- <a href="https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified" class="external text" rel="nofollow">SWE-bench Verified dataset on Hugging Face</a>

## Literature

- Jimenez, C. E. et al. (2023). *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?*. <a href="https://arxiv.org/abs/2310.06770" class="external text" rel="nofollow">arXiv:2310.06770</a>.
- Deng, X. et al. (2025). *SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?*. <a href="https://arxiv.org/abs/2509.16941" class="external text" rel="nofollow">arXiv:2509.16941</a>.

## References

1.  <span id="cite_note-openai-verified-1">↑ <sup>[1.00](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-0)</sup> <sup>[1.01](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-1)</sup> <sup>[1.02](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-2)</sup> <sup>[1.03](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-3)</sup> <sup>[1.04](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-4)</sup> <sup>[1.05](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-5)</sup> <sup>[1.06](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-6)</sup> <sup>[1.07](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-7)</sup> <sup>[1.08](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-8)</sup> <sup>[1.09](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-9)</sup> <sup>[1.10](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-10)</sup> <sup>[1.11](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-11)</sup> <sup>[1.12](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-12)</sup> <sup>[1.13](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-verified_1-13)</sup> "Introducing SWE-bench Verified". *OpenAI*. <a href="https://openai.com/index/introducing-swe-bench-verified/" class="external autonumber" rel="nofollow">[1]</a></span>
2.  <span id="cite_note-swebench-verified-2">↑ <sup>[2.0](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-swebench-verified_2-0)</sup> <sup>[2.1](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-swebench-verified_2-1)</sup> "SWE-bench Verified". *SWE-bench*. <a href="https://www.swebench.com/verified.html" class="external autonumber" rel="nofollow">[2]</a></span>
3.  <span id="cite_note-o1-systemcard-3">↑ <sup>[3.0](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-o1-systemcard_3-0)</sup> <sup>[3.1](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-o1-systemcard_3-1)</sup> <sup>[3.2](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-o1-systemcard_3-2)</sup> <sup>[3.3](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-o1-systemcard_3-3)</sup> "OpenAI o1 System Card". *arXiv*. <a href="https://arxiv.org/abs/2412.16720" class="external autonumber" rel="nofollow">[3]</a></span>
4.  <span id="cite_note-openai-deprecation-4">↑ <sup>[4.00](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-0)</sup> <sup>[4.01](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-1)</sup> <sup>[4.02](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-2)</sup> <sup>[4.03](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-3)</sup> <sup>[4.04](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-4)</sup> <sup>[4.05](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-5)</sup> <sup>[4.06](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-6)</sup> <sup>[4.07](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-7)</sup> <sup>[4.08](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-8)</sup> <sup>[4.09](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-9)</sup> <sup>[4.10](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-openai-deprecation_4-10)</sup> "Why SWE-bench Verified no longer measures frontier coding capabilities". *OpenAI*. <a href="https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/" class="external autonumber" rel="nofollow">[4]</a></span>
5.  <span id="cite_note-jimenez-swebench-5">↑ <sup>[5.0](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-jimenez-swebench_5-0)</sup> <sup>[5.1](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-jimenez-swebench_5-1)</sup> Jimenez, Carlos E. et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?". *arXiv*. <a href="https://arxiv.org/abs/2310.06770" class="external autonumber" rel="nofollow">[5]</a></span>
6.  <span id="cite_note-morph-swebench-6">↑ <sup>[6.0](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-morph-swebench_6-0)</sup> <sup>[6.1](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-morph-swebench_6-1)</sup> <sup>[6.2](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-morph-swebench_6-2)</sup> <sup>[6.3](https://systems-analysis.info/eng/SWE-bench_Verified#cite_ref-morph-swebench_6-3)</sup> "SWE-bench Pro Leaderboard (2026)". *Morph LLM*. <a href="https://www.morphllm.com/swe-bench-pro" class="external autonumber" rel="nofollow">[6]</a></span>