LLM benchmarks

From Systems analysis Wiki
Jump to navigation Jump to search

Large language model benchmarks are standardized sets of tests designed to measure, compare, and evaluate the quality and capabilities of Large Language Models (LLMs)[1]. Typically, each benchmark consists of a fixed set of tasks (e.g., questions, texts, or instructions) for which the correct answers or evaluation criteria are known in advance. This approach ensures an objective comparison of different models under identical conditions, allowing for tracking progress in the field and identifying the strengths and weaknesses of the models[2].

The regular use of benchmarks plays a key role in the development of LLMs, encouraging developers to improve their models and ensuring transparency and comparability of results within the scientific community. The evolution of benchmarks reflects the development of LLMs themselves: from simple language understanding tasks to complex tests that assess multi-step reasoning, common sense, ethics, and safety. By the mid-2020s this arms race had accelerated sharply: as leading models saturated the classic benchmarks, the field shifted toward expert-level knowledge tests, multimodal and instruction-following evaluation, contamination-resistant datasets built from unpublished problems, and evaluations of autonomous, tool-using agents.

Main Categories and Examples

LLM benchmarks cover a diverse range of skills and application areas. Below are the main categories and the most well-known task sets in each.

General Language Understanding

This category evaluates a model's basic abilities to understand and interpret natural language.

  • GLUE (General Language Understanding Evaluation, 2018) — one of the first comprehensive benchmarks, including a range of diverse tasks from sentiment analysis to assessing the logical coherence of text. The results from all tasks are aggregated into a single score, which allowed for the comparison of early models based on their overall effectiveness[3].
  • SuperGLUE (2019) — an "enhanced" successor to GLUE, developed in response to models quickly achieving near-human performance on it. SuperGLUE includes more difficult tasks that require a deep understanding of context and the ability to draw inferences[4].
  • WinoGrande (2019) — an expanded version of the Winograd Schema Challenge. It contains 44,000 tasks on resolving ambiguous pronouns in sentences, requiring common sense to choose the correct interpretation[5].

Multitask and Complex Benchmarks

These datasets test models on a wide spectrum of knowledge and skills, going beyond purely linguistic tasks.

  • MMLU (Massive Multitask Language Understanding, 2020) — a collection of multiple-choice questions covering 57 subject areas, from school-level disciplines to highly specialized professional knowledge (law, medicine). MMLU measures the breadth of a model's knowledge[6]. By 2024–2025 the benchmark had become saturated (see Limitations).
  • MMLU-Pro (2024) — a harder, more robust successor to MMLU created in response to its saturation. It comprises over 12,000 reasoning-focused questions across 14 domains and increases the number of answer options from four to ten, reducing the payoff from random guessing and rewarding chain-of-thought reasoning[7].
  • GPQA (Graduate-Level Google-Proof Q&A, 2023) — a set of PhD-level multiple-choice questions in biology, physics, and chemistry, written so that they are difficult to answer even with unrestricted web access. On the widely used "Diamond" subset of 198 questions, domain PhDs reach roughly 65% accuracy while skilled non-experts with internet access reach only about 34%[8].
  • BIG-bench (Beyond the Imitation Game Benchmark, 2022) — one of the largest collaborative benchmarks of its time, developed by over 400 authors across 132 institutions. It includes more than 200 tasks on a wide variety of topics, from linguistics to physics, to test models beyond pattern matching and identify their limitations in non-standard situations[9].
  • Humanity's Last Exam (HLE, 2025) — a deliberately extreme "final exam" for AI, developed by the Center for AI Safety and Scale AI to counter benchmark saturation. It consists of 2,500 expert-authored questions spanning over 100 subjects (with a portion held back as a private set to detect overfitting); roughly 14% of tasks are multimodal. At release in early 2025 the best models scored only single-digit percentages against expert accuracy of around 90%[10].
  • LiveBench (2024) — a general-purpose benchmark explicitly designed to resist contamination: it releases new questions on a regular schedule (often derived from recently published sources) and scores every answer automatically against objective ground truth. It spans math, coding, reasoning, language, instruction following, and data analysis[11].

Multimodal Understanding

As models gained the ability to process images alongside text, evaluation expanded beyond text-only tasks.

  • MMMU (Massive Multi-discipline Multimodal Understanding, 2024) — a benchmark for multimodal models built from about 11,500 college-level questions drawn from exams, quizzes, and textbooks, spanning six disciplines, 30 subjects, and 183 subfields, and incorporating diverse image types such as charts, diagrams, maps, and chemical structures. It requires integrating visual perception with domain knowledge and deliberate reasoning. As with MMLU, a harder successor (MMMU-Pro) was later released to remove questions solvable without the image and to counter early saturation[12][13].

Common Sense and Truthfulness

These benchmarks assess a model's ability to make logical inferences about everyday situations and to avoid spreading false information.

  • HellaSwag (2019) — tests common sense through the task of choosing the most plausible completion for a given situation. A key feature of the benchmark is the presence of "traps": incorrect answers are generated automatically and appear very plausible, requiring the model to have a deep understanding of the context[14].
  • TruthfulQA (2021) — measures a model's tendency to propagate popular myths and misconceptions. It contains 817 questions across 38 categories where the common answer found online is incorrect (e.g., "Do vaccines cause autism?"). The model is required to avoid false stereotypes and provide a factually correct answer[15].

Mathematical Problems

  • GSM8K (2021) — includes 8,500 grade school-level math word problems. Each problem requires performing a sequence of 2–8 arithmetic steps to arrive at the answer, which tests the model's multi-step reasoning ability[16].
  • MATH (2021) — a more challenging dataset of 12,500 problems drawn from mathematics competitions. It includes sections on algebra, geometry, and number theory, requiring the model to possess non-trivial problem-solving methods[17]. Both GSM8K and MATH are now largely saturated; the American Invitational Mathematics Examination (AIME) is often used in their place to stress-test reasoning models.
  • FrontierMath (2024) — a contamination-resistant benchmark developed by Epoch AI together with more than 60 mathematicians (including IMO gold medalists and Fields medalists). It comprises hundreds of original, unpublished research-level problems across modern mathematics, organized into difficulty tiers. At launch, leading models solved under 3% of problems; by late 2024 OpenAI's o3 reported roughly 25%, but the hardest tier remained essentially unsolved[18].

Code Generation

  • HumanEval (2021) — a standard test for evaluating an LLM's ability to write code. It contains 164 programming problems where the model must generate correct Python code based on a given description. Correctness is evaluated using unit tests[19]. By the mid-2020s top models exceed ~99% on it, and it is now considered saturated.
  • SWE-bench (2023) — a more realistic benchmark that collects descriptions of real problems (issues) from GitHub. The model must generate a patch (a code snippet) that fixes the problem. This requires understanding a large volume of existing code and complex, step-by-step reasoning[20].
  • SWE-bench Verified (2024) — a 500-task, human-validated subset of SWE-bench released by OpenAI's Preparedness team in collaboration with the SWE-bench authors, filtering out ambiguous or unsolvable problems. It became the headline "autonomous software engineering" number in frontier model reports, though by 2026 concerns about data contamination and residual flawed tasks led some evaluators to move on to harder agentic tests[21].

Instruction Following

  • IFEval (Instruction-Following Evaluation, 2023) — measures how precisely a model follows explicit, programmatically verifiable instructions (e.g., "write at least 300 words", "mention keyword X three times", "respond in JSON"). It contains around 500 prompts built from 25 types of verifiable instructions, which allows automatic, objective scoring without relying on a judge model[22].

Agentic and Tool Use

As models became capable of using external tools and completing multi-step tasks, evaluation shifted toward autonomous agents.

  • GAIA (2023) — a benchmark of real-world assistant questions that require reasoning, web browsing, multimodality, and tool use. The questions are conceptually simple for humans but demand chaining several capabilities together, exposing a large gap between assistants and human performance[23].
  • τ-bench (2024) — evaluates agents in realistic tool-agent-user interactions (e.g., retail and airline customer service), measuring whether an agent can follow domain rules and complete tasks reliably across repeated trials rather than in a single turn[24].

Evaluation of Conversational Models

  • LMArena (originally Chatbot Arena, launched 2023) — an open online platform where two anonymous models engage in a pairwise dialogue with a user. After the conversation, the user votes on which response was better. Based on millions of these "duels," a rating of user preferences is formed (originally an Elo system, later computed with the Bradley–Terry model), which reflects the quality of models in live interaction[25]. Launched by the LMSYS research group at UC Berkeley, it "graduated" to its own site (lmarena.ai) in September 2024, spun out as an independent company, and was rebranded again to Arena (arena.ai) in early 2026.
  • MT-Bench (2023) — an automated benchmark for stress-testing conversational abilities. It contains 80 multi-turn questions (each with a follow-up turn) that simulate a two-turn dialogue. The models' responses are evaluated by another, more powerful LLM ("LLM-as-a-judge," such as GPT-4) on a predefined scale; the authors also document judge biases such as position bias, verbosity bias, and self-enhancement bias[26].

Safety and Reliability

  • AgentHarm (2024) — a benchmark that assesses the propensity of LLM agents to follow harmful instructions. It includes 110 explicitly malicious agent tasks (440 with augmentations) across 11 harm categories (such as fraud, cybercrime, and harassment). A good model should refuse to execute such requests, and a robust one should retain that refusal even under jailbreak attempts[27].
  • SafetyBench (2023) — a bilingual multiple-choice benchmark of 11,435 Chinese and English questions across 7 categories of safety concerns (such as offensiveness, bias, physical and mental health, illegal activities, and ethics). It primarily measures a model's safety understanding rather than free-form generation; the authors report that these scores correlate with models' safety generation abilities[28].

Limitations and Current Issues

  • Data Contamination: The main threat to evaluation validity is the leakage of test data into training sets. The model might simply memorize the answers, which artificially inflates its score[2]. This concern has driven a wave of contamination-resistant designs: FrontierMath is built entirely from unpublished problems, HLE withholds a private question set, and benchmarks such as LiveBench refresh their questions on a rolling basis. Contamination has also been documented in established tests — audits of SWE-bench Verified found that a substantial share of "solved" tasks involved solution or file-path leakage from training data[21].
  • Benchmark Saturation: As models advance, their performance on older benchmarks reaches a ceiling, and the test ceases to be useful for distinguishing newer, more powerful models[2]. By 2024–2025 this had happened to most first-generation benchmarks: MMLU was effectively saturated around September 2024, and GSM8K, HumanEval, and HellaSwag now see top models clustered above 90–99%. As a symptom of this shift, Hugging Face retired its influential Open LLM Leaderboard in March 2025, noting that its underlying benchmarks had lost the ability to separate frontier models[29]. The response has been the constant development of harder standards (MMLU-Pro, GPQA, HLE, FrontierMath).
  • Leaderboard Integrity: Preference-based leaderboards face their own pressures. The 2025 study "The Leaderboard Illusion" argued that large labs can gain an advantage on Chatbot Arena/LMArena through selective disclosure of privately tested model variants and asymmetric data access; the platform disputed the framing and introduced policy changes, but the debate highlighted the difficulty of keeping open leaderboards fair[30].
  • Gap with Reality: High scores on benchmarks do not always guarantee reliable model performance in real-world, unstructured scenarios. The real environment is often richer and more unpredictable than any fixed set of tasks[1].

See also

References

  1. ↑ 1.0 1.1 "What Are LLM Benchmarks?". IBM. [1]
  2. ↑ 2.0 2.1 2.2 "20 LLM evaluation benchmarks and how they work". Evidently AI. [2]
  3. ↑ Wang, Alex; Singh, Amanpreet; Michael, Julian; et al. "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding". arXiv. [3]
  4. ↑ Wang, Alex; Pruksachatkun, Yada; Nangia, Nikita; et al. "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems". arXiv. [4]
  5. ↑ Sakaguchi, Keisuke; Le Bras, Ronan; Bhagavatula, Chandra; Choi, Yejin. "WinoGrande: An Adversarial Winograd Schema Challenge at Scale". arXiv. [5]
  6. ↑ Hendrycks, Dan; Burns, Collin; Basart, Steven; et al. "Measuring Massive Multitask Language Understanding". arXiv. [6]
  7. ↑ Wang, Yubo; Ma, Xueguang; Zhang, Ge; et al. "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark". NeurIPS 2024, Datasets and Benchmarks Track. [7]
  8. ↑ Rein, David; Hou, Betty Li; Stickland, Asa Cooper; et al. "GPQA: A Graduate-Level Google-Proof Q&A Benchmark". arXiv. [8]
  9. ↑ Srivastava, Aarohi; et al. "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models". arXiv. [9]
  10. ↑ Phan, Long; Gatti, Alice; Han, Ziwen; et al. "Humanity's Last Exam". arXiv. [10]
  11. ↑ White, Colin; Dooley, Samuel; Roberts, Manley; et al. "LiveBench: A Challenging, Contamination-Free LLM Benchmark". arXiv. [11]
  12. ↑ Yue, Xiang; Ni, Yuansheng; Zhang, Kai; et al. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI". arXiv. [12]
  13. ↑ Yue, Xiang; et al. "MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark". arXiv. [13]
  14. ↑ Zellers, Rowan; Holtzman, Ari; Bisk, Yonatan; et al. "HellaSwag: Can a Machine Really Finish Your Sentence?". arXiv. [14]
  15. ↑ Lin, Stephanie; Hilton, Jacob; Evans, Owain. "TruthfulQA: Measuring How Models Mimic Human Falsehoods". arXiv. [15]
  16. ↑ Cobbe, Karl; Kosaraju, Vineet; Bavarian, Mohammad; et al. "Training Verifiers to Solve Math Word Problems". arXiv. [16]
  17. ↑ Hendrycks, Dan; Burns, Collin; Kadavath, Saurav; et al. "Measuring Mathematical Problem Solving With the MATH Dataset". arXiv. [17]
  18. ↑ Glazer, Elliot; et al. "FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI". arXiv. [18]
  19. ↑ Chen, Mark; Tworek, Jerry; Jun, Heewoo; et al. "Evaluating Large Language Models Trained on Code". arXiv. [19]
  20. ↑ Jimenez, Carlos E.; et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?". arXiv. [20]
  21. ↑ 21.0 21.1 "Introducing SWE-bench Verified". OpenAI. [21]
  22. ↑ Zhou, Jeffrey; Lu, Tianjian; Mishra, Swaroop; et al. "Instruction-Following Evaluation for Large Language Models". arXiv. [22]
  23. ↑ Mialon, Grégoire; Fourrier, Clémentine; Swift, Craig; et al. "GAIA: A Benchmark for General AI Assistants". arXiv. [23]
  24. ↑ Yao, Shunyu; Shinn, Noah; Razavi, Pedram; Narasimhan, Karthik. "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains". arXiv. [24]
  25. ↑ Chiang, Wei-Lin; et al. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference". arXiv. [25]. Launch announcement: LMSYS, May 3, 2023. [26]
  26. ↑ Zheng, Lianmin; et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena". arXiv. [27]
  27. ↑ Andriushchenko, Maksym; Souly, Alexandra; Dziemian, Mateusz; et al. "AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents". arXiv. [28]
  28. ↑ Zhang, Zhexin; et al. "SafetyBench: A Comprehensive Benchmark for Evaluating the Safety of Large Language Models". arXiv. [29]
  29. ↑ "It's been a wild ride, folks :) (end of the Open LLM Leaderboard)". Hugging Face, March 2025. [30]
  30. ↑ Singh, Shivalika; et al. "The Leaderboard Illusion". arXiv. [31]
Retrieved from "https://systems-analysis.info/eng/index.php?title=LLM_benchmarks&oldid=199"