---
title: "MATH Benchmark"
source: "https://systems-analysis.info/eng/MATH_Benchmark"
wiki: "systems-analysis.info/eng"
article: "MATH_Benchmark"
language: "en"
categories:
  - "Category:English"
  - "Category:Large language models"
  - "Category:LLM benchmarks"
  - "Category:Machine learning"
  - "Category:Technology"
revision_id: 223
wiki_created_at: 2026-09-06T22:19:09Z
wiki_modified_at: 2026-09-06T22:19:09Z
downloaded_at: 2026-09-07T22:22:00Z
---

# MATH Benchmark

**MATH** (an acronym for **Mathematics Aptitude Test of Heuristics**) is a large dataset and benchmark for evaluating the mathematical reasoning and problem-solving skills of [large language models](https://systems-analysis.info/eng/Large_language_model "Large language model") (LLMs). The dataset was introduced in 2021 by a group of researchers led by **Dan Hendrycks** and contains **12,500 problems** sourced from American high school mathematics competitions, such as the AMC 10, AMC 12, and AIME<sup>[\[1\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-hendrycks2021-1)</sup>.

The problems cover a wide range of subjects (algebra, geometry, number theory, combinatorics, etc.) and are graded by difficulty level. Unlike standard textbook problems, they often require creative approaches and heuristic methods rather than the direct application of formulas. Each problem is accompanied by a complete step-by-step solution and a final answer, making MATH a valuable resource for both training and testing models<sup>[\[2\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-llm_eval_datasets-2)</sup>.

## Structure and Features of the Dataset

The MATH benchmark has several key features that make it a challenging and reliable evaluation tool.

### Problem Format

All problems and solutions are presented in LaTeX format, and the **Asymptote** language is used to describe geometric diagrams. This allows all conditions, including images, to be represented in a text-based format that can be processed by a language model. Each problem is tagged with one of seven mathematical subjects and one of five difficulty levels<sup>[\[1\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-hendrycks2021-1)</sup>.

### Automated Evaluation

The final answers in the dataset are enclosed in the special \`\boxed{...}\` format and adhere to a strict standard (e.g., fractions are in their simplest form). This enables automated evaluation of models using the **exact match** metric, which eliminates subjectivity and ambiguity when checking results. A model must produce the strictly correct answer for the problem to be considered solved<sup>[\[1\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-hendrycks2021-1)</sup>.

## Problem Difficulty and Human Performance

MATH is one of the most challenging mathematical tests for AI. The problems are difficult even for individuals with strong mathematical backgrounds.

- In the study accompanying the dataset, a group of **university students** was tested, with scores ranging from **~40%** to **~90%** for Olympiad winners.
- Even a three-time gold medalist of the International Mathematical Olympiad could not solve all the problems without errors<sup>[\[1\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-hendrycks2021-1)</sup>.

This demonstrates that successfully solving MATH problems requires not only knowledge but also high precision and mathematical intuition.

## Model Results and Progress in Problem Solving

### Initial Results (2021)

When the benchmark was launched in 2021, even the largest models achieved extremely low scores.

- The **[GPT](https://systems-analysis.info/eng/GPT_(OpenAI) "GPT (OpenAI)")-3** model (175 billion parameters) was only able to solve about **5%** of the problems correctly.
- [Fine-tuned](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)") versions of **GPT-2** showed an accuracy of 6–7%<sup>[\[1\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-hendrycks2021-1)</sup>.

The authors concluded that simply scaling up models had little effect on performance and that new algorithmic approaches were needed for progress<sup>[\[3\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-lang_models_surprised-3)</sup>.

### Breakthroughs with Minerva and GPT-4 (2022–2023)

A breakthrough occurred with the advent of models specifically trained on scientific texts and new problem-solving methods.

- In 2022, the **Google Minerva** model achieved an accuracy of about **50%**, demonstrating that a combination of scale and specialized training could drastically improve solution quality<sup>[\[3\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-lang_models_surprised-3)</sup>.
- In 2023, OpenAI's **GPT-4** showed another leap forward. By using tools, the model was able to significantly improve its results:
  - With **Code Interpreter** (executing code to verify calculations), its accuracy reached nearly **70%**.
  - Using a *code-based self-verification* method (self-checking and correcting errors with code), it set a record of **84.3%** of problems solved<sup>[\[4\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-decoder_gpt4-4)</sup>.

This result is comparable to the performance of strong human competitors and approaches an expert threshold.

## Significance and Impact

The MATH benchmark has played a key role in the development of LLMs' mathematical abilities. It clearly demonstrated that solving complex problems requires more than simple scaling, necessitating new approaches such as:

- Training on complete step-by-step solutions.
- Specialized training on scientific data.
- Using external tools for calculation and verification.

Despite significant progress, MATH remains an important and difficult challenge. It continues to serve as an indicator of the level of mathematical reasoning in LLMs and stimulates research into robustly solving problems that require multi-step reasoning<sup>[\[1\]](https://systems-analysis.info/eng/MATH_Benchmark#cite_note-hendrycks2021-1)</sup>.

## External links

- <a href="https://github.com/hendrycks/math" class="external text" rel="nofollow">Official MATH dataset repository on GitHub</a>
- <a href="https://paperswithcode.com/dataset/math" class="external text" rel="nofollow">Dataset page on Papers With Code</a>
- <a href="https://en.wikipedia.org/wiki/Language_model_benchmark" class="external text" rel="nofollow">Language model benchmark — Wikipedia</a>

## See also

- [GSM8K (Grade School Math 8K)](https://systems-analysis.info/eng/GSM8K_(Grade_School_Math_8K) "GSM8K (Grade School Math 8K)")
- [Chain-of-Thought Prompting](https://systems-analysis.info/eng/Chain-of-Thought_Prompting "Chain-of-Thought Prompting")
- [MMLU Benchmark](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark")
- [LLM benchmarks](https://systems-analysis.info/eng/LLM_benchmarks "LLM benchmarks")
- [LLM evaluation](https://systems-analysis.info/eng/LLM_evaluation "LLM evaluation")

## Literature

- Liang, P. et al. (2022). *Holistic Evaluation of Language Models (HELM)*. <a href="https://arxiv.org/abs/2211.09110" class="external text" rel="nofollow">arXiv:2211.09110</a>.
- Chang, Y. et al. (2023). *A Survey on Evaluation of Large Language Models*. <a href="https://arxiv.org/abs/2307.03109" class="external text" rel="nofollow">arXiv:2307.03109</a>.
- Ni, S. et al. (2025). *A Survey on Large Language Model Benchmarks*. <a href="https://arxiv.org/abs/2508.15361" class="external text" rel="nofollow">arXiv:2508.15361</a>.
- Biderman, S. et al. (2024). *The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned*. <a href="https://arxiv.org/abs/2405.14782" class="external text" rel="nofollow">arXiv:2405.14782</a>.
- Kiela, D. et al. (2021). *Dynabench: Rethinking Benchmarking in NLP*. <a href="https://arxiv.org/abs/2104.14337" class="external text" rel="nofollow">arXiv:2104.14337</a>.
- Ma, Z. et al. (2021). *Dynaboard: An Evaluation‑As‑A‑Service Platform for Holistic Next‑Generation Benchmarking*. <a href="https://arxiv.org/abs/2106.06052" class="external text" rel="nofollow">arXiv:2106.06052</a>.
- Goel, K. et al. (2021). *Robustness Gym: Unifying the NLP Evaluation Landscape*. <a href="https://arxiv.org/abs/2101.04840" class="external text" rel="nofollow">arXiv:2101.04840</a>.
- Xu, C. et al. (2024). *Benchmark Data Contamination of Large Language Models: A Survey*. <a href="https://arxiv.org/abs/2406.04244" class="external text" rel="nofollow">arXiv:2406.04244</a>.
- Liu, S. et al. (2025). *A Comprehensive Survey on Safety Evaluation of LLMs*. <a href="https://arxiv.org/abs/2506.11094" class="external text" rel="nofollow">arXiv:2506.11094</a>.
- Chiang, W.-L. et al. (2024). *Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference*. <a href="https://arxiv.org/abs/2403.04132" class="external text" rel="nofollow">arXiv:2403.04132</a>.
- Boubdir, M. et al. (2023). *Elo Uncovered: Robustness and Best Practices in Language Model Evaluation*. <a href="https://arxiv.org/abs/2311.17295" class="external text" rel="nofollow">arXiv:2311.17295</a>.
- Huang, L. et al. (2023). *A Survey on Hallucination in Large Language Models*. <a href="https://arxiv.org/abs/2311.05232" class="external text" rel="nofollow">arXiv:2311.05232</a>.

## References

1.  <span id="cite_note-hendrycks2021-1">↑ <sup>[1.0](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-hendrycks2021_1-0)</sup> <sup>[1.1](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-hendrycks2021_1-1)</sup> <sup>[1.2](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-hendrycks2021_1-2)</sup> <sup>[1.3](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-hendrycks2021_1-3)</sup> <sup>[1.4](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-hendrycks2021_1-4)</sup> <sup>[1.5](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-hendrycks2021_1-5)</sup> Hendrycks, D., et al. "Measuring Mathematical Problem Solving With the MATH Dataset." *arXiv:2103.03874*. <a href="https://arxiv.org/abs/2103.03874" class="external autonumber" rel="nofollow">[1]</a></span>
2.  <span id="cite_note-llm_eval_datasets-2">[↑](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-llm_eval_datasets_2-0) "AI Benchmarks and Datasets for LLM Evaluation." *arXiv:2412.01020*. <a href="https://arxiv.org/html/2412.01020v1" class="external autonumber" rel="nofollow">[2]</a></span>
3.  <span id="cite_note-lang_models_surprised-3">↑ <sup>[3.0](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-lang_models_surprised_3-0)</sup> <sup>[3.1](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-lang_models_surprised_3-1)</sup> "Language models surprised us." *Planned-Obsolescence.org*. <a href="https://www.planned-obsolescence.org/language-models-surprised-us/" class="external autonumber" rel="nofollow">[3]</a></span>
4.  <span id="cite_note-decoder_gpt4-4">[↑](https://systems-analysis.info/eng/MATH_Benchmark#cite_ref-decoder_gpt4_4-0) "GPT-4 Code Interpreter smashes maths benchmarks, hits new SOTA." *The Decoder*. <a href="https://the-decoder.com/gpt-4-code-interpreter-smashes-maths-benchmarks-hits-new-sota/" class="external autonumber" rel="nofollow">[4]</a></span>