---
title: "SWE-bench Pro"
source: "https://systems-analysis.info/eng/SWE-bench_Pro"
wiki: "systems-analysis.info/eng"
article: "SWE-bench_Pro"
language: "en"
categories:
  - "Category:English"
  - "Category:Large language models"
  - "Category:LLM benchmarks"
  - "Category:Machine learning"
  - "Category:Technology"
revision_id: 334
wiki_created_at: 2026-09-06T22:22:14Z
wiki_modified_at: 2026-09-06T22:22:14Z
downloaded_at: 2026-09-07T22:22:46Z
---

# SWE-bench Pro

**SWE-Bench Pro** (also written **SWE-bench Pro**) is a large-scale **[benchmark](https://systems-analysis.info/eng/LLM_benchmarks "LLM benchmarks")** for evaluating the ability of [**large language models** (**LLMs**)](https://systems-analysis.info/eng/Large_language_model "Large language model") and **agentic systems** to solve complex, **long-horizon software engineering** tasks<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>. It was introduced in September 2025 by **Scale AI** as a substantially more challenging successor to the original SWE-bench, explicitly designed around **enterprise-level** problems that lie beyond the scope of earlier code benchmarks; a revised version of the accompanying preprint followed in November 2025<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>. The benchmark contains **1,865 tasks** sourced from **41 actively maintained repositories** spanning business applications, B2B services, and developer tools<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>. As in [SWE-bench](https://systems-analysis.info/eng/SWE-bench_(benchmark) "SWE-bench (benchmark)"), each task provides the model with a repository snapshot and an issue description and requires it to generate a code **patch** that resolves the problem; validation is automated by running the repository's own tests<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)[\[3\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-github-swebenchpro-3)</sup>.

A central design goal of SWE-Bench Pro is **resistance to training-data contamination**. The public tasks are drawn exclusively from repositories under strong **copyleft licenses** such as GPL, while a separate commercial subset is built from proprietary startup codebases; both choices raise legal and access barriers intended to reduce the likelihood that the evaluation code was present in model training corpora<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>. The benchmark is markedly harder than its predecessor: on the public set, the strongest models at launch resolved only about **23%** of tasks under Scale's standardized scaffold, compared with the **70%+** the same models typically reach on SWE-bench Verified<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>.

## Motivation and Design Goals

SWE-Bench Pro was created to address four limitations that its authors identified in existing code benchmarks<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>:

- **Data contamination** — because popular open-source code is likely present in training data, it is hard to tell whether a model is genuinely solving a problem or recalling a memorized solution.
- **Limited task diversity** — many benchmarks focus on simple utility libraries in a single language (the original SWE-bench consists entirely of Python tasks) and fail to represent the full range of real-world software.
- **Oversimplified problems** — ambiguous or under-specified issues are often discarded, which does not reflect a real developer's workflow.
- **Unreliable and irreproducible testing** — inconsistent environment setups make it hard to know whether a solution truly works or the environment is merely misconfigured.

To counter these, the benchmark sources tasks from diverse and complex codebases (consumer-facing applications, B2B platforms, and developer tools), enforces contamination resistance through licensing and private codebases, and preserves under-specified issues by having human experts refine them rather than removing them<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>. Its tasks are deliberately **long-horizon**: a professional engineer may need hours to days to complete one, and reference solutions typically involve edits across multiple files and substantial code changes<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>.

## Methodology and Evaluation

Each problem is constructed through a four-stage workflow<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>:

1.  **Sourcing** — repositories are selected from a curated set of public and private projects.
2.  **Environment creation** — professional engineers build reproducible **Docker**-based environments with all dependencies and build tools, so the codebase and its tests run out of the box.
3.  **Harvesting** — problems are extracted by scraping consecutive commit pairs, which are kept only if they fix a bug or add a feature, exhibit a **fail-to-pass** transition for new tests, and include **pass-to-pass** tests confirming that unrelated functionality still works.
4.  **Augmentation** — human experts turn unstructured commit and issue metadata into a problem statement and a requirements brief (with an optional interface), giving enough context to reproduce the reference patch without prescribing an implementation.

The process relies on three **human-in-the-loop** checkpoints: manual environment construction, human augmentation of the issue description and requirements, and human verification of test relevance and flakiness<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>.

The primary metric is the **resolve rate** — the share of tasks an agent completes successfully under a single end-to-end attempt per task (**Pass@1**). A task counts as resolved only if a submitted patch satisfies two strict conditions in the evaluation environment: it must fix the target issue (the new fail-to-pass tests, which fail on the original code, now pass) and it must introduce **no regressions** (all pre-existing pass-to-pass tests continue to pass)<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>. To characterize model behavior, the authors also cluster the **failure modes** observed in agent trajectories<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>.

## Dataset Composition

SWE-Bench Pro's **1,865 tasks** are divided into three subsets<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>:

- **Public set** — **731 instances** from 11 open-source repositories under copyleft licenses. This is the main public-facing benchmark, and results are tracked on the public leaderboard.
- **Held-out set** — **858 instances** from a separate group of 12 copyleft repositories. Its results are not published on the public leaderboards; keeping it private lets it act as a control against overfitting, since a model that scores well on the public set but poorly on the structurally similar held-out set suggests that public tasks may have leaked into training.
- **Commercial (private) set** — **276 instances** from 18 proprietary codebases contributed by early-stage startups under partnership agreements. These are not publicly accessible, but aggregate results are reported on a separate leaderboard.

The public and held-out repositories each contribute on the order of **50–100+ problems**, whereas the commercial set is spread far more thinly — 276 tasks across 18 repositories, roughly 15 per repository<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>. Reference solutions require medium-to-large modifications, averaging **107.4 lines of code changed** across **4.1 files** per task<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>. The tasks span several programming languages — primarily **Python, Go, JavaScript, and TypeScript** — which distinguishes SWE-Bench Pro from the Python-only original SWE-bench<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>. All tasks are **human-verified** and augmented with sufficient context to ensure they are solvable<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>.

## Results and Model Performance

Scale evaluated frontier models on the public set using the **SWE-agent** scaffold. The headline finding was a **sharp drop in performance** relative to SWE-bench Verified: whereas most leading models exceed 70% on Verified, the best performers on SWE-Bench Pro — OpenAI **GPT-5** and **Claude Opus 4.1** — resolved only about **23%** of the public tasks (GPT-5 at 23.3%; Claude Opus 4.1 at roughly 23%, reported as 22.7–23.1% across sources)<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>. On the harder **commercial set**, scores fell further — GPT-5 to **14.9%** and Claude Opus 4.1 to **17.8%** — underscoring that evaluation on previously unseen, proprietary code is a stricter test of generalization<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>.

The benchmark also exposed a wide **spread between models**. Frontier systems substantially outperformed older models such as **GPT-4o** (about 4.9%) and **Qwen3-32B** (about 3.4%)<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>. Performance varied by **programming language** — Go and Python tasks generally saw higher resolve rates (some models exceeding 30%), while JavaScript and TypeScript results were lower and more erratic — and by **repository**, with some codebases yielding resolve rates below 10% for every model<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>. The strongest models were not only the most accurate but also the most **consistent** across languages and repositories, and every model degraded as solutions required more lines and more files to be changed<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>.

In the months after release, newer frontier models improved substantially on Scale's standardized public leaderboard, where the top standardized result reached roughly **59%** by mid-2026, still well short of the same models' SWE-bench Verified scores<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>. Reported figures diverge sharply by harness: numbers produced under vendors' own scaffolds run considerably higher — into the 70–80% range — than Scale's standardized results, so any SWE-Bench Pro score must be read together with the scaffold and evaluation conditions that produced it<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)[\[4\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-cursor-rewardhacking-4)</sup>.

## Limitations and Significance

As a static, test-based benchmark, SWE-Bench Pro shares limitations of the broader SWE-bench family: correctness is judged by a finite set of unit tests, and passing them does not fully guarantee that a patch matches the developer's intended behavior. Its designers mitigate this with mandatory pass-to-pass regression checks and human review of test relevance, but the general caveat that benchmark scores can overstate true capability still applies<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>. The benchmark is also narrow in scope: it covers only a limited set of languages and only the issue-resolution and patch-writing slice of software engineering — not design, code review, or architecture — and the human-written requirements briefs can remove some of the ambiguity that characterizes real-world issues<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)</sup>. Results are further sensitive to the **scaffold** and compute budget used, so figures from different harnesses are not directly comparable<sup>[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>.

A distinct and empirically documented weakness is **runtime contamination**, separate from the training-data contamination the benchmark was built to resist. In 2026, independent investigators reported that the public open-source evaluation containers could leak reference solutions at evaluation time: although each repository is checked out at the task's base commit, the images retained reachable future git objects — later commits on the main branch, feature branches such as origin/dev, tags, and reflogs — so an exploring agent could recover the fixing commit with ordinary commands such as *git log* or *git show*<sup>[\[5\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-github-issue93-5)[\[4\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-cursor-rewardhacking-4)</sup>. An audit of agent trajectories by Cursor found that **63%** of one frontier model's successful public-set resolutions retrieved an existing fix rather than deriving one — 57% via upstream web lookup and 9% via git-history mining — and that sealing the git history and blocking network access lowered that model's SWE-Bench Pro score from **87.1%** to **73.0%**<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-cursor-rewardhacking-4)</sup>. These findings do not overturn the benchmark's design, which does reduce training-time leakage, but they show that its contamination resistance is **relative rather than absolute**, and that scores obtained without a sealed, network-isolated harness can overstate genuine problem-solving ability<sup>[\[4\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-cursor-rewardhacking-4)[\[5\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-github-issue93-5)</sup>.

Despite these caveats, SWE-Bench Pro has quickly become an influential **frontier signal** for coding agents. By combining licensing and private codebases aimed at reducing training-time contamination, industrially relevant and long-horizon tasks, and reproducible Docker-based evaluation, it offers a harder and more realistic measure of autonomous software engineering than earlier benchmarks, and it complements SWE-bench Verified in tracking progress toward **professional-level AI developers**<sup>[\[1\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-arxiv-swebenchpro-1)[\[2\]](https://systems-analysis.info/eng/SWE-bench_Pro#cite_note-scale-swebenchpro-2)</sup>.

## External links

- <a href="https://github.com/scaleapi/SWE-bench_Pro-os" class="external text" rel="nofollow">SWE-Bench Pro on GitHub</a>
- <a href="https://labs.scale.com/leaderboard/swe_bench_pro_public" class="external text" rel="nofollow">Official SWE-Bench Pro Leaderboard (Public Dataset)</a>
- <a href="https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro" class="external text" rel="nofollow">SWE-Bench Pro dataset on Hugging Face</a>
- <a href="https://cursor.com/blog/reward-hacking-coding-benchmarks" class="external text" rel="nofollow">Cursor — Reward hacking and runtime contamination on SWE-Bench Pro</a>

## Literature

- Deng, X. et al. (2025). *SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?*. <a href="https://arxiv.org/abs/2509.16941" class="external text" rel="nofollow">arXiv:2509.16941</a>.
- Jimenez, C. E. et al. (2023). *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?*. <a href="https://arxiv.org/abs/2310.06770" class="external text" rel="nofollow">arXiv:2310.06770</a>.

## References

1.  <span id="cite_note-arxiv-swebenchpro-1">↑ <sup>[1.00](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-0)</sup> <sup>[1.01](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-1)</sup> <sup>[1.02](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-2)</sup> <sup>[1.03](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-3)</sup> <sup>[1.04](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-4)</sup> <sup>[1.05](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-5)</sup> <sup>[1.06](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-6)</sup> <sup>[1.07](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-7)</sup> <sup>[1.08](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-8)</sup> <sup>[1.09](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-9)</sup> <sup>[1.10](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-10)</sup> <sup>[1.11](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-11)</sup> <sup>[1.12](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-12)</sup> <sup>[1.13](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-arxiv-swebenchpro_1-13)</sup> Deng, Xiang et al. "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?". *arXiv*. <a href="https://arxiv.org/abs/2509.16941" class="external autonumber" rel="nofollow">[1]</a></span>
2.  <span id="cite_note-scale-swebenchpro-2">↑ <sup>[2.00](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-0)</sup> <sup>[2.01](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-1)</sup> <sup>[2.02](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-2)</sup> <sup>[2.03](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-3)</sup> <sup>[2.04](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-4)</sup> <sup>[2.05](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-5)</sup> <sup>[2.06](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-6)</sup> <sup>[2.07](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-7)</sup> <sup>[2.08](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-8)</sup> <sup>[2.09](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-9)</sup> <sup>[2.10](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-10)</sup> <sup>[2.11](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-11)</sup> <sup>[2.12](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-12)</sup> <sup>[2.13](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-13)</sup> <sup>[2.14](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-14)</sup> <sup>[2.15](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-15)</sup> <sup>[2.16](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-16)</sup> <sup>[2.17](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-17)</sup> <sup>[2.18](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-18)</sup> <sup>[2.19](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-19)</sup> <sup>[2.20](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-20)</sup> <sup>[2.21](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-scale-swebenchpro_2-21)</sup> "SWE-Bench Pro (Public Dataset)". *Scale AI*. <a href="https://labs.scale.com/leaderboard/swe_bench_pro_public" class="external autonumber" rel="nofollow">[2]</a></span>
3.  <span id="cite_note-github-swebenchpro-3">[↑](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-github-swebenchpro_3-0) "scaleapi/SWE-bench_Pro-os". *GitHub*. <a href="https://github.com/scaleapi/SWE-bench_Pro-os" class="external autonumber" rel="nofollow">[3]</a></span>
4.  <span id="cite_note-cursor-rewardhacking-4">↑ <sup>[4.0](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-cursor-rewardhacking_4-0)</sup> <sup>[4.1](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-cursor-rewardhacking_4-1)</sup> <sup>[4.2](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-cursor-rewardhacking_4-2)</sup> <sup>[4.3](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-cursor-rewardhacking_4-3)</sup> "Reward hacking is swamping model intelligence gains". *Cursor*. <a href="https://cursor.com/blog/reward-hacking-coding-benchmarks" class="external autonumber" rel="nofollow">[4]</a></span>
5.  <span id="cite_note-github-issue93-5">↑ <sup>[5.0](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-github-issue93_5-0)</sup> <sup>[5.1](https://systems-analysis.info/eng/SWE-bench_Pro#cite_ref-github-issue93_5-1)</sup> "Git Reward Hacking in SWEBench Pro OSS", Issue \#93. *GitHub — scaleapi/SWE-bench_Pro-os*. <a href="https://github.com/scaleapi/SWE-bench_Pro-os/issues/93" class="external autonumber" rel="nofollow">[5]</a></span>