SuperGLUE (benchmark)

SuperGLUE is a comprehensive benchmark (a set of test tasks) for evaluating natural language processing systems, particularly large language models (LLMs). It was introduced in 2019 by a group of researchers led by Alex Wang from New York University with participation from Facebook AI Research and other organizations.

The creation of SuperGLUE was driven by the fact that by mid-2019, the preceding GLUE benchmark had become an "easy task" for modern models: the aggregate score of the best models on GLUE reached 88.4, exceeding the average human level (87.1). Thus, the room for further progress had narrowed. In response, the authors developed SuperGLUE as a more challenging alternative capable of providing more rigorous verification of language understanding by models. The goal of SuperGLUE is to provide a neutral and difficult-to-"game" measure of progress in general language understanding for English. It was expected that significant improvement in SuperGLUE results would require substantial innovations in machine learning methods — for example, more efficient learning from small samples, multi-task and self-supervised learning. In other words, SuperGLUE includes tasks that are simple for humans but challenging for machine intelligence, to stimulate the development of models with truly deep language understanding.

Features and Differences from GLUE

SuperGLUE largely follows the format of GLUE — it offers a single aggregate quality score across tasks, a public leaderboard, and tools for model analysis. However, SuperGLUE introduces several improvements and innovations compared to its predecessor:

• More challenging tasks: SuperGLUE selected eight of the most difficult tasks. Two of them are inherited from GLUE (among the most difficult there), while the rest were chosen from new candidates based on their difficulty for modern NLP models. Thus, the benchmark focuses on those aspects of understanding where models previously showed the worst results.
• Diversity of formats: while in GLUE all tasks were reduced to sentence or sentence pair classification, SuperGLUE includes a wider range of formats. In addition to classification, tasks for coreference resolution and question answering have been added, requiring the model to understand coherent text and logical inference.
• Human evaluation on all tasks: for each SuperGLUE task, a baseline human performance level (non-expert) is calculated, confirming that even strong models like BERT significantly lagged behind humans at the time of the benchmark's launch. The presence of a human baseline (~90% overall) provides "headroom" for model growth and serves as a target benchmark.
• Transparent rules and tools: the rules for posting results on the leaderboard have been revised (to ensure fair comparison and acknowledge dataset authors' contributions). A new open-source toolkit has also been published for convenient fine-tuning and multi-task training of models on SuperGLUE data.

Together, these measures make SuperGLUE a more reliable test for generalized language abilities of models, preventing high results through narrow cheating or overfitting to specific formats of the previous GLUE.

SuperGLUE Task Set

SuperGLUE consists of eight tasks covering different aspects of text understanding.

• BoolQ (Boolean Questions): a question-answering (QA) task where each example is given a short text (Wikipedia excerpt) and a question that must be answered "yes" or "no". Questions are formulated by users (from Google search queries) and require extracting explicit or implicit facts from the text; the quality metric is the proportion of correct answers (accuracy).
• CB (CommitmentBank): a textual entailment task with three classes. The dataset consists of short texts containing complex sentences; it is required to determine to what extent the author of the text is committed to the truth of the embedded statement. Essentially, this tests whether a statement can be inferred from the given context. The task is difficult due to small sample sizes (about 250 examples) and class imbalance; quality is assessed by accuracy and class-averaged F1 score.
• COPA (Choice of Plausible Alternatives): a task on causal reasoning. The model is given a premise (one sentence) and must choose the correct cause or effect from two options. All COPA examples are manually formulated and require common sense to establish causal relationships. Topics include situations from blogs and specialized encyclopedias; the metric is accuracy (proportion of correct choices). Example: given the sentence "the child developed immunity to the disease" and the question "what is the cause?" — a human immediately understands that the correct answer is "he received a vaccine," while models have to guess the causal connection.
• MultiRC (Multi-Sentence Reading Comprehension): a task on multi-sentence text comprehension with multiple-choice elements. The model receives a paragraph of text, a question about the paragraph's content, and a list of possible answers; it must determine which answers are correct (each question may have multiple correct answers). Features: answering the question typically requires combining information from several sentences of the text, which tests the model's ability to connect facts. Quality is measured by two metrics: F1 on answers (accounting for partially correct sets) and Exact Match — the proportion of questions for which completely correct answer sets are given.
• ReCoRD (Reading Comprehension with Commonsense Reasoning Dataset): a task on reading comprehension and knowledge use. It is a modified Cloze test: given a news text (CNN/Daily Mail article) and a sentence with a missing entity word; the model must choose which entity from the text fits in the gap. Answer options are given as all entities mentioned in the article, and they may be semantically equivalent. Successful solution requires context understanding and common sense. Metrics are maximum token-level F1 and Exact Match for predicted answers.
• RTE (Recognizing Textual Entailment): a binary classification task on textual entailment (entailment vs. not entailment). The dataset combines examples from several textual inference recognition competitions (RTE 1-5 series). Each task contains a pair of text fragments (premise-hypothesis); the model must determine whether the hypothesis follows from the text. Unlike many large datasets, RTE is quite small (about 2,500 training examples), but showed significant gains from transfer learning: accuracy rose from ~56% (random guessing level) to ~86% with the advent of models like BERT. Nevertheless, at the time of SuperGLUE's launch, model accuracy still lagged behind humans by about 8 percentage points, so RTE was included as one of the tasks that maintained a gap to human level.
• WiC (Word-in-Context): a task on resolving word sense ambiguity in context (WSD). Two independent sentences are given, each containing the same polysemous word; it is necessary to determine whether this word is used in the same sense in both cases. Data is taken from dictionary resources (WordNet, VerbNet, Wiktionary), thus covering a wide range of words and meanings. The task is formalized as binary classification and assessed by the proportion of correct answers. WiC requires the model to understand subtle semantic differences, essentially testing lexical semantics.
• WSC (Winograd Schema Challenge): a task on coreference resolution using common sense. Each task consists of one sentence containing a pronoun and a list of two entities (nouns) from the same sentence. It is required to determine which of the proposed nouns the given pronoun refers to. Example of a classic Winograd sentence: "The trophy didn't fit in the suitcase because it was too small" — a human understands that "it" refers to the suitcase (the suitcase was too small). Such examples cannot be solved without everyday knowledge and context. GLUE already contained a simplified version of this task (WNLI), but models could not surpass even random level on it for a long time. Only special techniques, such as adding external data with similar examples, raised model quality on WSC to ~90% by 2019. However, humans solve WSC tasks virtually without errors (~96-100% correct answers). SuperGLUE includes the original version of WSC in binary classification format (for each "pronoun-entity" pair, the model answers whether they match by reference). This task remains one of the most difficult tests requiring commonsense reasoning.

All SuperGLUE tests have closed test sets with answers unknown to developers. Models submit their predictions to a server where an aggregate score is calculated — task-averaged accuracy (for tasks with multiple metrics, the internal metric is averaged first). This single SuperGLUE score simplifies comparing models by overall language intelligence level.

Results and Model Progress

At SuperGLUE's launch, the authors provided as a reference the results of a strong baseline model (enhanced BERT) — and they turned out to be significantly below human on all tasks. On average, the best model at that time scored approximately 20 points lower than humans on the aggregate metric. On individual tasks, the gap was particularly large: for example, on the WSC task, the model barely reached ~65% accuracy versus 100% for humans (a gap of ~35 points). Even on "simpler" looking tasks (BoolQ, CB, RTE, WiC), automatic systems lagged ~10 points behind human level. These differences confirmed that SuperGLUE truly poses a serious challenge to current technologies and cannot be solved trivially.

Nevertheless, just a few months after SuperGLUE's appearance, rapid progress began. In late 2019, Google researchers presented the T5 model (Text-To-Text Transfer Transformer) with 11 billion parameters, which achieved an aggregate result of 88.9, closely approaching the human level of ~89.8. In fact, T5 improved the previous SuperGLUE record by 4.3 points at once and reduced the error rate by almost a third, leaving only a minimal gap of 0.9 points to the human score. Developers noted that SuperGLUE was deliberately selected so that tasks are easy for humans, so the model reaching ~89% became an important milestone.

The first to surpass average human quality was Microsoft's DeBERTa model (Decoding-enhanced BERT with disentangled attention). In January 2021, researchers reported that the DeBERTa version with 1.5 billion parameters scored 89.9 points, slightly above the human baseline of 89.8. This was the first case when a single model exceeded humans on the SuperGLUE metric. Additionally, an ensemble of several DeBERTa models raised the record to ~90.3 points. The DeBERTa model outperformed the previous leader (Google T5) by approximately 0.6% and demonstrated the effectiveness of new ideas in Transformer architecture (separate representation of word content and position, improved mask decoder, etc.).

Progress did not stop there: as the size and complexity of language models grew, SuperGLUE results continued to improve. By the end of 2021, Microsoft's T-NLRv5 model (Microsoft Turing NLR family) topped the leaderboard — it further increased the gap above human level. The last unresolved tasks for machines on GLUE (e.g., NLI subtleties) were "closed" by this model, which came very close to full parity with humans even on the most difficult subtasks.

By 2022-2023, the human-level threshold on SuperGLUE was confidently surpassed by several independent large models. For example, Google's PaLM model (540 billion parameters) when fine-tuned on SuperGLUE tasks achieved around 90.4 points, and OpenAI's GPT-4 model showed an even slightly higher result. By mid-2023, the SuperGLUE leaderboard had several models with scores above 90 (i.e., exceeding the average human level). One can say that the benchmark is practically solved by modern systems: the scores of the best models are so high that they exceed the capabilities of most non-expert humans. This success demonstrates enormous progress in NLP in a short time, but simultaneously points to the need for new, even more challenging tests for the latest models. Subsequent benchmarks are already appearing (e.g., MMLU, BIG-Bench, etc.), designed to test models on broader understanding and erudition beyond SuperGLUE tasks.

Impact and Further Research

SuperGLUE has thus established itself as an important milestone in the development of evaluation methodologies in language processing. In enthusiast and scientific circles, its results have become a kind of "litmus test" for new LLM architectures: achieving or exceeding human level on SuperGLUE is perceived as a sign of an advanced model with deep language understanding. This has also been reflected in practice — many modern language models that achieved high results on SuperGLUE have formed the basis of applied question-answering systems, dialogue agents, text summarization systems, etc.. SuperGLUE continues to be used by researchers for fine-tuning and algorithm comparison, although the cutting edge is now gradually shifting to new frontiers of artificial intelligence evaluation.

External links

• Official SuperGLUE website
• Original SuperGLUE paper (NeurIPS)
• Microsoft article on DeBERTa achieving human level
• SuperGLUE dataset page on Papers With Code
• Language model benchmark — Wikipedia

Bibliography

• Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
• Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.
• Ni, S. et al. (2025). A Survey on Large Language Model Benchmarks. arXiv:2508.15361.
• Biderman, S. et al. (2024). The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned. arXiv:2405.14782.
• Kiela, D. et al. (2021). Dynabench: Rethinking Benchmarking in NLP. arXiv:2104.14337.
• Ma, Z. et al. (2021). Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking. arXiv:2106.06052.
• Goel, K. et al. (2021). Robustness Gym: Unifying the NLP Evaluation Landscape. arXiv:2101.04840.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.
• Liu, S. et al. (2025). A Comprehensive Survey on Safety Evaluation of LLMs. arXiv:2506.11094.
• Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
• Boubdir, M. et al. (2023). Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv:2311.17295.
• Huang, L. et al. (2023). A Survey on Hallucination in Large Language Models. arXiv:2311.05232.