--- title: "MMLU-Pro Benchmark" source: "https://systems-analysis.info/eng/MMLU-Pro_Benchmark" wiki: "systems-analysis.info/eng" article: "MMLU-Pro_Benchmark" language: "en" categories: - "Category:English" - "Category:Large language models" - "Category:LLM benchmarks" - "Category:Machine learning" - "Category:Technology" revision_id: 227 wiki_created_at: 2026-09-06T22:19:12Z wiki_modified_at: 2026-09-06T22:19:12Z downloaded_at: 2026-09-07T22:22:02Z --- # MMLU-Pro Benchmark **MMLU-Pro** is a [benchmark](https://systems-analysis.info/eng/LLM_benchmarks "LLM benchmarks") test designed to evaluate the knowledge and reasoning capabilities of [large language models (LLMs)](https://systems-analysis.info/eng/Large_language_model "Large language model"). It was introduced in June 2024 in the paper *MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark*, whose designated core contributors are **Yubo Wang**, **Xueguang Ma**, and **Wenhu Chen**; the authors are drawn from the University of Waterloo (home of the TIGER-Lab group), the University of Toronto, and Carnegie Mellon University. The paper was published at the NeurIPS 2024 conference in the Datasets and Benchmarks track^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. MMLU-Pro was created as a direct successor to the original [MMLU benchmark](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark"), whose usefulness had declined as leading models approached its ceiling. It addresses this "saturation" by adding more reasoning-intensive questions, expanding the number of answer choices from four to ten, and removing trivial or mislabeled items inherited from the original set^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. ## Background By 2024, [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark") had become the de facto standard for evaluating LLMs, but progress on it had stalled. Since GPT-4 reached **86.4%** in March 2023, frontier models such as GPT-4-Turbo, Gemini 1.5 Pro, Claude 3, and Llama 3 had all settled into a narrow band of roughly **86–87%**^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. This clustering left the benchmark unable to distinguish between the strongest systems. The authors noted that GPT-4o, despite large gains on other tests, improved its MMLU score by only about 1% (to 87.4%), which called into question the benchmark's ability to track further progress^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. The original [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark") also had two structural weaknesses that MMLU-Pro set out to fix: a four-option format that allowed a 25% success rate from random guessing, and a portion of noisy or unanswerable questions that imposed an artificial ceiling on scores^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. ## Description and Content MMLU-Pro consists of **12,032 multiple-choice questions** distributed across **14 disciplines**^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)[\[2\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-split_note-2)}. The subjects are Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Mathematics, Philosophy, Physics, Psychology, and a catch-all "Other" category^{[\[3\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_github-3)}. Construction began by merging MMLU's 57 original subjects into these 14 broader categories and removing questions that were too easy: eight smaller models were run on MMLU, and any question answered correctly by more than four of them was discarded, eliminating **5,886 questions** (about 42% of the original set). New questions were then drawn from three further sources — an online STEM problem collection, *TheoremQA*, and *SciBench* — and GPT-4-Turbo was used both to convert these into multiple-choice form and to expand every question from four options to ten by generating six additional distractors. Finally, the dataset passed through two rounds of expert review: in the first, human experts checked answer correctness and removed unsuitable questions; in the second, the Gemini-1.5-Pro model flagged options that might in fact be additional correct answers ("false negatives"), which human annotators then re-examined^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. As a result, about **57%** of the final questions come from the original MMLU, **34%** from the STEM collection, and the remainder from TheoremQA (roughly 5%) and SciBench (roughly 4.5%)^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. The mix varies sharply by discipline. Reasoning-heavy STEM fields were expanded with new material — only about a third of the Physics questions and a sixth of the Chemistry questions are drawn from MMLU, the rest coming from the added sources — whereas knowledge-heavy disciplines such as Law and History rely entirely on MMLU and were reduced rather than expanded, because many of their questions were removed as too easy (History, for example, fell from 930 MMLU questions to 381)^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. ## Key Differences from MMLU MMLU-Pro differs from the original benchmark in three main respects^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}: - **More answer choices**. Questions were expanded from four options to ten; in the final set, 83% of questions have ten options and the rest have fewer, for an average of **9.47** options per question. This lowers the expected accuracy from random guessing from 25% toward roughly **10%** and makes elimination strategies less effective. - **Emphasis on reasoning**. The share of challenging college-level problems was increased, so that strong performance requires multi-step reasoning rather than recall alone. - **Cleaner data**. Trivial, ambiguous, and mislabeled questions from the original MMLU were filtered out to raise the effective ceiling of the benchmark. A notable consequence is the changed role of *chain-of-thought* (CoT) prompting. On the original MMLU, many models scored higher with direct answer prediction and CoT could even reduce accuracy; on MMLU-Pro, CoT is necessary for good results — it raises GPT-4o's score from **53.5%** with direct answering to **72.6%** with CoT, a gain of about 19 percentage points^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. MMLU-Pro is also more robust to prompt variation. Across **24 different prompt styles**, the sensitivity of model scores to prompt wording fell from 4–5% on MMLU to about **2%** on MMLU-Pro, making results more reproducible^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. ## Results The paper evaluated more than 50 models using a standard protocol of 5-shot chain-of-thought prompting (the two Gemini 1.5 models were run zero-shot), extracting answers from the generated text with regular expressions and falling back to a random choice when extraction failed^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. Under this setup the benchmark caused a sharp drop in reported accuracy: scores fell by **16% to 33%** compared with MMLU, depending on the model^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. In the paper's evaluation, **GPT-4o** was the strongest model tested, with an overall accuracy of **72.6%** — well below the 86–87% range that leading models reached on the original MMLU^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. Among open-source models, **Llama-3-70B-Instruct** (56.2%) and **DeepSeek-V2-Chat** (54.8%) performed best, approaching Claude 3 Sonnet but still trailing GPT-4o and Claude 3 Opus^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. The benchmark also proved more discriminative than its predecessor. The gap between GPT-4o and GPT-4-Turbo, only about 1% on [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark"), widened to roughly 9% on MMLU-Pro, illustrating a greater ability to separate closely performing systems^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. Since 2024, MMLU-Pro has been widely adopted and is tracked by independent public leaderboards^{[\[4\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-artificial_analysis-4)}. Frontier models improved rapidly: by mid-2026, the strongest systems (such as Gemini 3 Pro and Claude Opus 4.5) had reached roughly **90%** on these leaderboards and were clustered within about two points of one another — reviving the same saturation concern that had motivated MMLU-Pro's creation^{[\[4\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-artificial_analysis-4)}. ## Limitations Even as it corrects several problems of the original [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark"), MMLU-Pro carries limitations that its own authors emphasize. Chief among them is the multiple-choice format itself, which cannot capture depth of comprehension or open-ended generation as well as free-form answers, together with the benchmark's exclusive focus on text and its lack of any multimodal assessment^{[\[1\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_paper-1)}. It also shares weaknesses common to public benchmarks. Test questions remain vulnerable to *data contamination*, in which items appear in a model's training data and inflate its apparent performance^{[\[5\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-contamination_survey-5)}. The dataset is not entirely free of labeling errors, and its maintainers continue to accept community corrections^{[\[6\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-mmlu_pro_hf-6)}. Finally, published figures are often self-reported and depend on whether chain-of-thought prompting is used and on the specific evaluation harness, so scores from different sources are not always directly comparable^{[\[7\]](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_note-leaderboard_note-7)}. ## External links - Official MMLU-Pro repository on GitHub - MMLU-Pro leaderboard on Hugging Face - MMLU-Pro dataset on Hugging Face - MMLU-Pro paper on arXiv ## Literature - Wang, Y. et al. (2024). *MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark*. arXiv:2406.01574. - Hendrycks, D. et al. (2021). *Measuring Massive Multitask Language Understanding*. arXiv:2009.03300. - Xu, C. et al. (2024). *Benchmark Data Contamination of Large Language Models: A Survey*. arXiv:2406.04244. - Chang, Y. et al. (2023). *A Survey on Evaluation of Large Language Models*. arXiv:2307.03109. - Ni, S. et al. (2025). *A Survey on Large Language Model Benchmarks*. arXiv:2508.15361. ## References 1. ↑ ^{[1.00](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-0)} ^{[1.01](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-1)} ^{[1.02](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-2)} ^{[1.03](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-3)} ^{[1.04](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-4)} ^{[1.05](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-5)} ^{[1.06](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-6)} ^{[1.07](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-7)} ^{[1.08](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-8)} ^{[1.09](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-9)} ^{[1.10](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-10)} ^{[1.11](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-11)} ^{[1.12](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-12)} ^{[1.13](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-13)} ^{[1.14](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-14)} ^{[1.15](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-15)} ^{[1.16](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-16)} ^{[1.17](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_paper_1-17)} Wang, Y. et al. "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark". *arXiv:2406.01574*, 2024 (NeurIPS 2024, Datasets and Benchmarks Track). [1] 2. [↑](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-split_note_2-0) The figure of 12,032 refers to the main test set. The public release on Hugging Face also includes a small validation split of 70 questions — five demonstration examples for each of the 14 disciplines, used for few-shot prompting — giving 12,102 rows in total. [2] 3. [↑](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_github_3-0) "TIGER-AI-Lab/MMLU-Pro". *GitHub*. [3] 4. ↑ ^{[4.0](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-artificial_analysis_4-0)} ^{[4.1](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-artificial_analysis_4-1)} "MMLU-Pro Benchmark Leaderboard". *Artificial Analysis*. [4] 5. [↑](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-contamination_survey_5-0) Xu, C. et al. "Benchmark Data Contamination of Large Language Models: A Survey". *arXiv:2406.04244*, 2024. [5] 6. [↑](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-mmlu_pro_hf_6-0) "TIGER-Lab/MMLU-Pro". *Hugging Face Datasets*. [6] 7. [↑](https://systems-analysis.info/eng/MMLU-Pro_Benchmark#cite_ref-leaderboard_note_7-0) The dataset's maintainers note that reported results come from a mix of the authors and third parties, typically using 5-shot evaluation, though some models (such as Gemini 1.5) are scored zero-shot; results drawn from different sources should therefore be compared with caution. [7]