---
title: "MMLU Benchmark"
source: "https://systems-analysis.info/eng/MMLU_Benchmark"
wiki: "systems-analysis.info/eng"
article: "MMLU_Benchmark"
language: "en"
categories:
  - "Category:English"
  - "Category:Large language models"
  - "Category:LLM benchmarks"
  - "Category:Machine learning"
  - "Category:Technology"
revision_id: 228
wiki_created_at: 2026-09-06T22:19:13Z
wiki_modified_at: 2026-09-06T22:19:13Z
downloaded_at: 2026-09-07T22:22:03Z
---

# MMLU Benchmark

**MMLU** (**Massive Multitask Language Understanding**) is a benchmark test designed to evaluate the capabilities of [large language models](https://systems-analysis.info/eng/Large_language_model "Large language model") (LLMs) across a wide range of subject areas. It was introduced in the 2020 paper *Measuring Massive Multitask Language Understanding* by a team of researchers led by **Dan Hendrycks** of UC Berkeley, and was published at the ICLR conference in 2021<sup>[\[1\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_paper-1)</sup>.

The goal of MMLU is to test how well a model has absorbed the diverse knowledge and skills acquired during its [pre-training](https://systems-analysis.info/eng/Pre-training_of_large_language_models "Pre-training of large language models") phase by evaluating it in a *zero-shot* or *[few-shot](https://systems-analysis.info/eng/Few-Shot_and_Zero-Shot_Learning "Few-Shot and Zero-Shot Learning")* setting without additional [fine-tuning](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)"). MMLU was created as a more challenging alternative to earlier tests (such as [GLUE](https://systems-analysis.info/eng/GLUE_Benchmark "GLUE Benchmark") and [SuperGLUE](https://systems-analysis.info/eng/SuperGLUE_(benchmark) "SuperGLUE (benchmark)")), on which state-of-the-art models had rapidly approached or surpassed human baselines<sup>[\[1\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_paper-1)</sup>.

## Description and Content

MMLU consists of **15,908 multiple-choice questions** covering **57 different disciplines**. The dataset is divided into a few-shot development set (5 questions per subject), a validation set of 1,540 questions, and a test set of 14,079 questions; models are evaluated on the test split<sup>[\[1\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_paper-1)[\[2\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-split_note-2)</sup>. The subjects include:

- STEM subjects (mathematics, physics, biology, computer science, engineering).
- Humanities and social sciences (history, philosophy, law, economics).
- Applied and professional fields (medicine, business, accounting)<sup>[\[1\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_paper-1)</sup>.

The difficulty ranges from elementary school level to advanced professional level. The questions are drawn from real exam materials from schools, universities, and professional tests, such as the GRE and USMLE<sup>[\[1\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_paper-1)</sup>. Each question has four possible answers, so random guessing yields an accuracy of 25%. To achieve a high score, a model must possess extensive encyclopedic knowledge and reasoning abilities.

## Results and Development

When MMLU was released in 2020, most of the evaluated models performed at or near random-chance levels. Among them, [GPT](https://systems-analysis.info/eng/GPT_(OpenAI) "GPT (OpenAI)")-3 (175B parameters) achieved the best result for a purely few-shot model, at **~43.9%** (and 37.7% in the zero-shot setting); the highest overall score in the original paper belonged to **UnifiedQA** — a T5-based model fine-tuned on question-answering datasets and evaluated by transfer — at **~48.9%**<sup>[\[1\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_paper-1)</sup>. Both remained far below the authors' estimate of expert-level human performance, which they placed at **~89.8%** (based on the 95th percentile of human test-takers); non-expert human annotators scored only about 34.5%<sup>[\[1\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_paper-1)</sup>. This gap confirmed the difficulty and high standard of the new benchmark.

Progress on the benchmark was rapid. By March 2023, [GPT](https://systems-analysis.info/eng/GPT_(OpenAI) "GPT (OpenAI)")-4 reached **86.4%** in the standard 5-shot setting<sup>[\[3\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-gpt4_report-3)</sup>. In late 2023, Google's **[Gemini](https://systems-analysis.info/eng/Gemini_(Google) "Gemini (Google)") Ultra** reported **90.04%**, becoming the first model to exceed the authors' 89.8% human-expert estimate — although this figure used a chain-of-thought prompting protocol with self-consistency (CoT@32) rather than the standard 5-shot setting, in which it scored 83.7%<sup>[\[4\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-gemini_report-4)</sup>. By mid-2024, several leading models — including GPT-4o, Anthropic's **[Claude](https://systems-analysis.info/eng/Claude_(Anthropic) "Claude (Anthropic)") 3.5 Sonnet**, and Llama 3.1 405B — were clustered in the high 80s (roughly **86–88%**), though such figures are not always directly comparable, since evaluation harnesses and prompting protocols differ<sup>[\[5\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_pro_paper-5)</sup>.

This clustering near the top of the scale led to a gradual "saturation" of the benchmark: leading models began to achieve near-maximum scores, which diminished MMLU's ability to differentiate between their capabilities. The authors of MMLU-Pro noted that scores had reached a plateau, with little significant progress after GPT-4's 86.4%<sup>[\[5\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_pro_paper-5)</sup>. This spurred the community to develop new, more difficult tests.

## Limitations and Criticism

Despite its widespread use, MMLU has several significant limitations.

### Data Quality and Correctness

In 2024, researchers conducted a manual analysis of a sample of 5,700 MMLU questions and found a significant number of errors<sup>[\[6\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-done_with_mmlu_2024-6)</sup>.

- Approximately **6.5%** of the analysed questions contain errors in their labeling or wording.
- In certain categories, the proportion is very high. For example, in the "Virology" section, **57%** of the analysed questions contained errors (multiple correct answers, unclear wording, or the wrong reference answer).

This means that even a perfect model cannot score 100% on the original dataset, and some of the apparent improvements in metrics may reflect models memorizing systematic errors in the set<sup>[\[6\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-done_with_mmlu_2024-6)</sup>.

### Evaluation Methodology and Data Leakage

- **Sensitivity to the evaluation protocol**. Scores depend heavily on how a model is evaluated — zero-shot, five-shot, or chain-of-thought — as well as on the specific prompt wording, scoring method, and evaluation harness. Results are therefore not directly comparable unless the protocol is specified, and the original MMLU is known to be particularly sensitive to prompt formatting<sup>[\[5\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_pro_paper-5)</sup>.
- **Data contamination**. There is a risk that questions and answers from public benchmarks are included in the training datasets of LLMs. In such cases, a model effectively "knows" the correct answers, making the evaluation unfair<sup>[\[7\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-contamination_survey-7)</sup>.
- **Multiple-choice format**. With only four options, a model can sometimes reach the correct answer through elimination or shortcuts rather than genuine understanding, which can overestimate its true ability<sup>[\[5\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_pro_paper-5)</sup>.

## Derivative Versions and Extensions

To address the problems of the original MMLU, several variants have been created.

- **MMLU-Redux**. A corrected and re-annotated subset of the dataset, introduced in June 2024. The initial version contained 3,000 relabeled questions across 30 subjects<sup>[\[6\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-done_with_mmlu_2024-6)</sup>; it was later expanded in **MMLU-Redux 2.0** to 5,700 questions covering all 57 subjects<sup>[\[8\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-done_with_mmlu_naacl-8)</sup>. It is designed for more reliable evaluation, free of the distortions caused by data errors.
- **[MMLU-Pro](https://systems-analysis.info/eng/MMLU-Pro_Benchmark "MMLU-Pro Benchmark")**. An expanded and more difficult version of the test, introduced in June 2024 and published at NeurIPS 2024. It contains over 12,000 questions consolidated into 14 broader disciplines, each with **10 answer choices** instead of four. This reduces the probability of random guessing to 10% and lowers the benchmark's sensitivity to prompt formatting. The questions were expert-reviewed and include additional, more challenging tasks<sup>[\[5\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmlu_pro_paper-5)</sup>.
- **MMMLU** (*Multilingual MMLU*). A multilingual version released by OpenAI in September 2024. The MMLU **test set** was professionally translated by human translators into 14 languages, including widely spoken ones (Spanish, Chinese) and low-resource ones (e.g., Yoruba), allowing model capabilities to be evaluated and compared across languages; such multilingual evaluations typically reveal a marked drop in accuracy on non-English and especially low-resource languages. The release does not include Russian, and the original English test set is not part of the multilingual package<sup>[\[9\]](https://systems-analysis.info/eng/MMLU_Benchmark#cite_note-mmmlu_hf-9)</sup>.

## External links

- <a href="https://github.com/hendrycks/test" class="external text" rel="nofollow">Official MMLU repository on GitHub</a>
- <a href="https://paperswithcode.com/dataset/mmlu" class="external text" rel="nofollow">MMLU page on Papers with Code with model results</a>
- <a href="https://en.wikipedia.org/wiki/MMLU" class="external text" rel="nofollow">MMLU — Wikipedia</a>

## See also

- [Large language model](https://systems-analysis.info/eng/Large_language_model "Large language model")
- [GLUE Benchmark](https://systems-analysis.info/eng/GLUE_Benchmark "GLUE Benchmark")
- [SuperGLUE (benchmark)](https://systems-analysis.info/eng/SuperGLUE_(benchmark) "SuperGLUE (benchmark)")
- [Few-Shot and Zero-Shot Learning](https://systems-analysis.info/eng/Few-Shot_and_Zero-Shot_Learning "Few-Shot and Zero-Shot Learning")
- [Fine-tuning (deep learning)](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)")
- [Pre-training of large language models](https://systems-analysis.info/eng/Pre-training_of_large_language_models "Pre-training of large language models")

## Literature

- Liang, P. et al. (2022). *Holistic Evaluation of Language Models (HELM)*. <a href="https://arxiv.org/abs/2211.09110" class="external text" rel="nofollow">arXiv:2211.09110</a>.
- Chang, Y. et al. (2023). *A Survey on Evaluation of Large Language Models*. <a href="https://arxiv.org/abs/2307.03109" class="external text" rel="nofollow">arXiv:2307.03109</a>.
- Ni, S. et al. (2025). *A Survey on Large Language Model Benchmarks*. <a href="https://arxiv.org/abs/2508.15361" class="external text" rel="nofollow">arXiv:2508.15361</a>.
- Biderman, S. et al. (2024). *The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned*. <a href="https://arxiv.org/abs/2405.14782" class="external text" rel="nofollow">arXiv:2405.14782</a>.
- Kiela, D. et al. (2021). *Dynabench: Rethinking Benchmarking in NLP*. <a href="https://arxiv.org/abs/2104.14337" class="external text" rel="nofollow">arXiv:2104.14337</a>.
- Xu, C. et al. (2024). *Benchmark Data Contamination of Large Language Models: A Survey*. <a href="https://arxiv.org/abs/2406.04244" class="external text" rel="nofollow">arXiv:2406.04244</a>.
- Chiang, W.-L. et al. (2024). *Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference*. <a href="https://arxiv.org/abs/2403.04132" class="external text" rel="nofollow">arXiv:2403.04132</a>.

## References

1.  <span id="cite_note-mmlu_paper-1">↑ <sup>[1.0](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_paper_1-0)</sup> <sup>[1.1](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_paper_1-1)</sup> <sup>[1.2](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_paper_1-2)</sup> <sup>[1.3](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_paper_1-3)</sup> <sup>[1.4](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_paper_1-4)</sup> <sup>[1.5](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_paper_1-5)</sup> <sup>[1.6](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_paper_1-6)</sup> Hendrycks, D. et al. "Measuring Massive Multitask Language Understanding". *arXiv:2009.03300*, 2021. <a href="https://arxiv.org/abs/2009.03300" class="external autonumber" rel="nofollow">[1]</a></span>
2.  <span id="cite_note-split_note-2">[↑](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-split_note_2-0) The figure of 15,908 is the total reported in the original paper. Some later public mirrors of the dataset report slightly lower counts after deduplication; for example, the *cais/mmlu* release on Hugging Face lists 285 development, 1,531 validation, and 14,042 test examples. <a href="https://huggingface.co/datasets/cais/mmlu" class="external autonumber" rel="nofollow">[2]</a></span>
3.  <span id="cite_note-gpt4_report-3">[↑](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-gpt4_report_3-0) OpenAI. "GPT-4 Technical Report". *arXiv:2303.08774*, 2023. <a href="https://arxiv.org/abs/2303.08774" class="external autonumber" rel="nofollow">[3]</a></span>
4.  <span id="cite_note-gemini_report-4">[↑](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-gemini_report_4-0) Gemini Team, Google. "Gemini: A Family of Highly Capable Multimodal Models". *arXiv:2312.11805*, 2023. <a href="https://arxiv.org/abs/2312.11805" class="external autonumber" rel="nofollow">[4]</a></span>
5.  <span id="cite_note-mmlu_pro_paper-5">↑ <sup>[5.0](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_pro_paper_5-0)</sup> <sup>[5.1](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_pro_paper_5-1)</sup> <sup>[5.2](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_pro_paper_5-2)</sup> <sup>[5.3](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_pro_paper_5-3)</sup> <sup>[5.4](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmlu_pro_paper_5-4)</sup> Wang, Y. et al. "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark". *arXiv:2406.01574*, 2024 (NeurIPS 2024). <a href="https://arxiv.org/abs/2406.01574" class="external autonumber" rel="nofollow">[5]</a></span>
6.  <span id="cite_note-done_with_mmlu_2024-6">↑ <sup>[6.0](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-done_with_mmlu_2024_6-0)</sup> <sup>[6.1](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-done_with_mmlu_2024_6-1)</sup> <sup>[6.2](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-done_with_mmlu_2024_6-2)</sup> Gema, A. P. et al. "Are We Done with MMLU?". *arXiv:2406.04127*, 2024 (NAACL 2025). <a href="https://arxiv.org/abs/2406.04127" class="external autonumber" rel="nofollow">[6]</a></span>
7.  <span id="cite_note-contamination_survey-7">[↑](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-contamination_survey_7-0) Xu, C. et al. "Benchmark Data Contamination of Large Language Models: A Survey". *arXiv:2406.04244*, 2024. <a href="https://arxiv.org/abs/2406.04244" class="external autonumber" rel="nofollow">[7]</a></span>
8.  <span id="cite_note-done_with_mmlu_naacl-8">[↑](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-done_with_mmlu_naacl_8-0) Gema, A. P. et al. "Are We Done with MMLU?". *NAACL 2025*. <a href="https://aclanthology.org/2025.naacl-long.262/" class="external autonumber" rel="nofollow">[8]</a></span>
9.  <span id="cite_note-mmmlu_hf-9">[↑](https://systems-analysis.info/eng/MMLU_Benchmark#cite_ref-mmmlu_hf_9-0) "openai/MMMLU". *Hugging Face Datasets*. <a href="https://huggingface.co/datasets/openai/MMMLU" class="external autonumber" rel="nofollow">[9]</a></span>