GPQA Diamond Benchmark

GPQA Diamond (also GPQA-D) is the hardest and most heavily vetted subset of GPQA (Graduate-Level Google-Proof Q&A), a benchmark of expert-written multiple-choice questions in biology, physics, and chemistry designed to measure the scientific reasoning of advanced artificial intelligence (AI) systems. The parent dataset was introduced in November 2023 by researchers from New York University, the AI company Cohere, and Anthropic, led by David Rein. The work was later accepted to the First Conference on Language Modeling (COLM 2024).

The questions are deliberately "Google-proof": they are written so that the answer cannot be found through a simple web search and instead require genuine graduate-level understanding. The Diamond subset consists of the 198 questions that both domain-expert validators answered correctly but that the majority of skilled non-expert validators answered incorrectly, making it the cleanest measure of expert-level reasoning in the dataset.

Because of its compact size, high expert agreement, and resistance to search, GPQA Diamond became one of the most widely cited reasoning benchmarks for frontier language models. By 2026, however, the strongest models were answering more than 93% of its questions correctly, and the benchmark is now widely regarded as largely saturated.

Background

GPQA was conceived not primarily as a difficulty ranking but as a tool for scalable oversight research: the study of how human supervisors can reliably verify the outputs of AI systems that may eventually exceed human expertise. To make such experiments realistic, the authors needed questions that were hard both for skilled non-experts and for frontier AI, so that the gap between a knowledgeable-but-non-expert supervisor and a specialist could be studied directly.

By late 2023, popular knowledge tests such as MMLU were approaching saturation for the leading models, reducing their value as a measure of progress. GPQA was designed as a harder, contamination-resistant alternative rooted in genuine domain expertise rather than broad factual recall. At release, even the strongest baseline built on GPT-4 answered only about 39% of the questions correctly—well below the human expert level and only modestly above the non-expert baseline.

Creation of the benchmark

The dataset was written by nearly a hundred contributors holding or pursuing PhDs in the relevant fields. Each question passed through a multi-stage validation pipeline intended to guarantee both objective difficulty and a single, defensible answer:

• Expert validation: two independent domain experts attempted each question; agreement between them was required for a question to be considered clean.
• Non-expert validation: highly skilled non-experts in adjacent fields attempted each question with over 30 minutes of unrestricted web access, establishing the "Google-proof" property.
• Revision: authors could revise questions in response to validator feedback before a question entered the final sets.

From this process the authors defined three nested sets of increasing quality:

• The extended set (546 questions), containing all retained items.
• The main set (448 questions).
• The Diamond set (198 questions)—those for which both expert validators were correct and the majority of non-expert validators were wrong.

The Diamond subset is therefore the intersection of two conditions—high expert agreement and high non-expert difficulty—which is why it is used as the headline evaluation set by most independent evaluators.

Structure and content of the benchmark

GPQA Diamond contains 198 four-option multiple-choice questions drawn from the natural sciences—biology, physics, and chemistry—and their subfields, including organic chemistry, quantum mechanics, molecular biology, genetics, and astrophysics. Because there are four options, the random-guessing baseline is 25%.

Human performance provides the key reference points. On the full GPQA set, domain experts (holding or pursuing a PhD) reached about 65% accuracy—or 74% after discounting clear mistakes they later identified—while skilled non-experts reached only about 34%, despite spending on average more than half an hour per question with full internet access. For GPQA Diamond specifically, OpenAI later recruited PhD-level experts and reported a 69.7% human baseline; because Diamond is defined partly by non-experts answering incorrectly, non-expert accuracy on this subset is lower still by construction.

The gap between the expert and non-expert baselines is the core signal the benchmark is built around: it quantifies how much of the difficulty comes from genuine expertise rather than from information that can simply be looked up.

For grading, most evaluators use a zero-shot chain-of-thought prompt in which the model reasons step by step and then emits its choice in a fixed format; some also report consensus (majority-vote) variants. Answer parsing is strict—responses that do not match the required format receive no credit, which can occasionally push a model's measured score below the 25% random baseline if it formats answers poorly.

Model performance results

The defining feature of GPQA Diamond is the speed with which model performance moved from below the non-expert baseline to well above the expert baseline. Because scores depend on the evaluator, the prompt, and whether consensus voting is used, the figures below are best read as a trajectory rather than as a fixed ranking.

• Launch (November 2023). The strongest GPT-4 baseline scored about 39%, below both human baselines.
• Early-to-mid 2024. Reported zero-shot chain-of-thought scores on Diamond climbed steadily—Claude 3 Sonnet 40.4%, GPT-4 Turbo 48.0%, Claude 3 Opus 50.4%, GPT-4o 53.6%—moving the frontier into the band between the non-expert and expert baselines.
• Mid-2024. Claude 3.5 Sonnet led public Diamond evaluations at 59.4% zero-shot chain-of-thought.
• September 2024 — the inflection point. OpenAI's o1 scored 77.3% (zero-shot pass@1) and 78.0% with consensus. OpenAI reported that o1 was the first model to surpass the PhD-expert baseline of 69.7% on GPQA Diamond—a result widely credited with helping establish the "reasoning model" category.
• 2025. Reasoning models pushed scores into the 80s: xAI's Grok 4 was independently measured by Epoch AI at about 87% (±2%), with other frontier systems (e.g. OpenAI's o3-class models and Google's Gemini 2.5 Pro) reported in the low-to-mid 80s. Several vendor-reported figures reached the low 90s by mid-year.
• 2026. The top cluster compressed above 93%, and progress on the benchmark approached an asymptote. Leading systems—Google's Gemini 3.1 Pro Preview, OpenAI's GPT-5.5, and Anthropic's Claude Fable 5—now sit near the practical ceiling of the test.

The progression is summarized below.

Date	Model / baseline	Accuracy, %	Note
Nov 2023	GPT-4 (zero-shot CoT)	~39	strongest baseline in the original paper
—	Non-expert humans (30+ min web)	34	the "Google-proof" floor (full set)
—	PhD-level experts (Diamond)	69.7	re-measured by OpenAI for the o1 comparison
Early 2024	Claude 3 Opus	50.4	0-shot CoT (Anthropic model card)
Mid 2024	Claude 3.5 Sonnet	59.4	0-shot CoT; leading public score at the time
Sep 2024	OpenAI o1	77.3 (78.0 consensus)	surpassed the expert baseline (OpenAI-reported first)
2025	Grok 4 (Epoch AI)	~87	reasoning models climb into the 80s
2026	Gemini 3.1 Pro Preview / GPT-5.5 / Claude Fable 5	~93–95	benchmark largely saturated

Criticism and limitations

Despite its influence, GPQA Diamond has significant, widely acknowledged limitations.

• Saturation. The most prominent limitation is that the benchmark has been effectively solved at the top. With frontier models clustered above 93%, incremental gains are small and often within noise, and the test no longer discriminates well among the strongest systems—contrary to the original expectation that it would remain challenging for years.
• Small size and sampling uncertainty. At only 198 questions, GPQA Diamond carries substantial sampling uncertainty: one item is worth about 0.5 percentage points, and binomial confidence intervals span several percentage points—roughly ±3 points near the 94% ceiling and wider (around ±7 points) in the middle of the range. A handful of items can therefore reorder closely spaced models, which is one reason different leaderboards disagree on the exact ranking even when they agree on the overall picture.
• Training-time contamination. Because the questions are public multiple-choice items, there is a persistent concern that they leak into model training corpora. GPQA's release is notable for its explicit anti-contamination framing: the dataset ships with a canary string, and the authors ask that examples not be reposted in plain text or images, since public exposure raises the risk of items entering future training sets. The closed-form format nonetheless remains more vulnerable to memorization and guessing than open-ended tasks, and once a public benchmark becomes popular its value as a clean measure of frontier capability tends to decay.
• Search-time contamination. A newer concern is specific to tool-augmented, search-enabled evaluations rather than the closed-book scores reported above. Even though the questions are designed to be "Google-proof", a 2025 study found that for roughly 3% of questions across Humanity's Last Exam, SimpleQA, and GPQA, search-based agents could retrieve public dataset mirrors with ground-truth labels—copying the answer rather than reasoning to it. The measured effect on accuracy was clearest on HLE and SimpleQA; for GPQA it was smaller, but the finding still complicates the interpretation of any search-augmented result on a benchmark whose premise is that search should not help.
• Methodology sensitivity. Scores depend on prompting (zero-shot versus consensus voting), answer parsing, and harness details. As an illustration, the same flagship model can be reported at 94.1% by one independent evaluator and 95.45% by another.
• Expert disagreement and label quality. Because even qualified experts top out around 65–74%, some questions involve genuine expert disagreement or contestable answer keys, so a portion of any measured gap reflects annotation difficulty rather than pure capability.
• Multiple-choice ceiling. The four-option format likely accelerated saturation: even an uncertain model can gain credit by eliminating implausible distractors, so partial knowledge converts into correct answers more readily than in open-ended tasks. This is a contributing factor rather than a full explanation—the clustering above 93% mostly reflects genuine capability plus the inherent measurement ceiling of a small closed-form test—but it does make GPQA Diamond less demanding than open-ended expert evaluation.
• Narrow construct. By design the benchmark measures closed-ended, verifiable scientific knowledge and reasoning. It does not assess experimental design, open-ended research, or creativity, so surpassing the expert baseline on multiple-choice questions does not imply that a model matches a scientist in practice—and it partly undercuts the original scalable-oversight motivation once models exceed the experts meant to supervise them.

Significance and outlook

GPQA Diamond became the de facto standard for reporting scientific reasoning in frontier model releases, and its trajectory has served as a compact record of a broader shift in AI capability.

• A catalyst for reasoning models. o1's crossing of the expert baseline on GPQA Diamond was among the most cited early demonstrations that inference-time reasoning could deliver large gains on hard problems.
• A common reference point. The benchmark gave researchers and vendors a shared, independently trackable measure of graduate-level scientific reasoning, widely reproduced in model cards and third-party leaderboards.
• A benchmark past its frontier. With the top cluster near the ceiling, GPQA Diamond now functions mainly as a regression check and a floor rather than as a frontier differentiator; harder successors have taken over that role. These are not simply "next versions" of GPQA: where GPQA Diamond is small, STEM-only, and four-choice, Humanity's Last Exam is far broader, spanning over a hundred disciplines with a mix of exact-match and multiple-choice items and a multimodal component. Its arc—from a test expected to last years to near-saturation in roughly two—illustrates how quickly benchmark headroom is now consumed.

Model results by source

GPQA Diamond scores are not absolute: they depend on the evaluation date, the prompt and parsing rules, and whether consensus voting is used. Different leaderboards therefore report different numbers for the same model, and figures are best compared within a single source rather than across sources. Both independent boards below agree that Gemini 3.1 Pro Preview currently leads, yet they disagree on its exact score (94.1% versus 95.45%) and on the precise ordering of the tightly clustered runners-up; a third aggregator (LLM Stats) reports Anthropic's Claude Mythos Preview at about 94.6%. Both tables are point-in-time snapshots that change frequently, and only the confidently sourced top of each board is reproduced here.

Artificial Analysis — independent leaderboard

An independent evaluator that runs its own harness on the 198-question Diamond set. The leading cluster below is read from the Artificial Analysis leaderboard as of 1 July 2026; standings change frequently.

#	Model	Accuracy, %
1	Gemini 3.1 Pro Preview	94.1
2	GPT-5.5 (xhigh)	93.5
3	GPT-5.5 (high)	93.2
4	Qwen3.7 Max	92.3

Note: the top of this board is extremely compressed—several further systems (including MiniMax M3 and Claude Fable 5) also cluster around 92–93%—so the fourth-place gap is smaller than the reported sampling uncertainty.

Vals AI — independent leaderboard

A separate independent evaluator that runs zero-shot and few-shot chain-of-thought on the Diamond subset. The top of the board is dated 17 June 2026 and, like the others, changes frequently.

#	Model	Accuracy, %
1	Gemini 3.1 Pro Preview (02/26)	95.45
2	Claude Fable 5	93.18
2	GPT-5.5	93.18

External links

• Research paper presenting GPQA (arXiv)
• GPQA dataset — GitHub
• GPQA dataset — Hugging Face
• GPQA Diamond Leaderboard — Artificial Analysis
• GPQA Diamond — Vals AI
• Learning to reason with LLMs — OpenAI (o1)

Literature

• Rein, D. et al. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv:2311.12022.
• Hendrycks, D. et al. (2021). Measuring Massive Multitask Language Understanding (MMLU). arXiv:2009.03300.
• Phan, L. et al. (2025). Humanity's Last Exam. arXiv:2501.14249.
• Bowman, S. R. et al. (2022). Measuring Progress on Scalable Oversight for Large Language Models. arXiv:2211.03540.
• Han, Z. et al. (2025). Search-Time Data Contamination. arXiv:2508.13180.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.