--- title: "SuperGLUE (benchmark)" source: "https://systems-analysis.info/eng/SuperGLUE_(benchmark)" wiki: "systems-analysis.info/eng" article: "SuperGLUE_(benchmark)" language: "en" categories: - "Category:English" - "Category:Large language models" - "Category:LLM benchmarks" - "Category:Machine learning" - "Category:Technology" revision_id: 352 wiki_created_at: 2026-09-06T22:22:29Z wiki_modified_at: 2026-09-06T22:22:29Z downloaded_at: 2026-09-07T22:22:54Z --- # SuperGLUE (benchmark) **SuperGLUE** is a comprehensive **benchmark** (a set of test tasks) for evaluating natural language processing systems, particularly **large language models** (LLMs)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. It was introduced in 2019 by a group of researchers led by Alex Wang from New York University with participation from Facebook AI Research and other organizations^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. The creation of SuperGLUE was driven by the fact that by mid-2019, the preceding GLUE benchmark had become an "easy task" for modern models: the aggregate score of the best models on GLUE reached 88.4, exceeding the average human level (87.1)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Thus, the room for further progress had narrowed^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. In response, the authors developed SuperGLUE as a more challenging alternative capable of providing more rigorous verification of language understanding by models^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. The goal of SuperGLUE is to provide a neutral and difficult-to-"game" measure of progress in general language understanding for English^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. It was expected that significant improvement in SuperGLUE results would require substantial innovations in machine learning methods — for example, more efficient learning from small samples, multi-task and self-supervised learning^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. In other words, SuperGLUE includes tasks that are simple for humans but challenging for machine intelligence^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}, to stimulate the development of models with truly deep language understanding. ## Features and Differences from GLUE SuperGLUE largely follows the format of GLUE — it offers a single **aggregate quality score** across tasks, a public **leaderboard**, and **tools** for model analysis^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. However, SuperGLUE introduces several improvements and innovations compared to its predecessor^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}: - **More challenging tasks**: SuperGLUE selected **eight of the most difficult tasks**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Two of them are inherited from GLUE (among the most difficult there), while the rest were chosen from new candidates based on their difficulty for modern NLP models^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Thus, the benchmark focuses on those aspects of understanding where models previously showed the worst results. - **Diversity of formats**: while in GLUE all tasks were reduced to sentence or sentence pair classification, SuperGLUE includes a **wider range of formats**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. In addition to classification, tasks for **coreference resolution** and **question answering** have been added, requiring the model to understand coherent text and **logical inference**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. - **Human evaluation on all tasks**: for each SuperGLUE task, a **baseline human performance level** (non-expert) is calculated^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}, confirming that even strong models like BERT significantly lagged behind humans at the time of the benchmark's launch^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. The presence of a **human baseline** (~90% overall) provides "headroom" for model growth and serves as a target benchmark^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. - **Transparent rules and tools**: the rules for posting results on the leaderboard have been revised (to ensure fair comparison and acknowledge dataset authors' contributions)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. A new open-source toolkit has also been published for convenient fine-tuning and multi-task training of models on SuperGLUE data^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Together, these measures make SuperGLUE a more reliable test for **generalized language abilities** of models, preventing high results through narrow cheating or overfitting to specific formats of the previous GLUE^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. ## SuperGLUE Task Set SuperGLUE consists of **eight tasks** covering different aspects of text understanding. - **BoolQ** (Boolean Questions): a **question-answering (QA)** task where each example is given a short text (Wikipedia excerpt) and a question that must be answered "yes" or "no"^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Questions are formulated by users (from Google search queries) and require extracting explicit or implicit facts from the text; the quality metric is the proportion of correct answers (accuracy)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. - **CB** (CommitmentBank): a **textual entailment** task with three classes^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. The dataset consists of short texts containing complex sentences; it is required to determine to what extent the author of the text is **committed to the truth** of the embedded statement^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Essentially, this tests whether a statement can be inferred from the given context. The task is difficult due to small sample sizes (about 250 examples) and class imbalance; quality is assessed by accuracy and class-averaged F1 score^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. - **COPA** (Choice of Plausible Alternatives): a task on **causal reasoning**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. The model is given a premise (one sentence) and must choose the correct cause or effect from two options^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. All COPA examples are manually formulated and require **common sense** to establish causal relationships. Topics include situations from blogs and specialized encyclopedias; the metric is accuracy (proportion of correct choices)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Example: given the sentence "the child developed immunity to the disease" and the question "what is the cause?" — a human immediately understands that the correct answer is "he received a vaccine," while models have to guess the causal connection^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. - **MultiRC** (Multi-Sentence Reading Comprehension): a task on **multi-sentence text comprehension** with multiple-choice elements^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. The model receives a paragraph of text, a question about the paragraph's content, and a list of possible answers; it must determine which answers are correct (each question may have multiple correct answers)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Features: answering the question typically requires combining information from several sentences of the text, which tests the model's ability to **connect facts**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Quality is measured by two metrics: F1 on answers (accounting for partially correct sets) and Exact Match — the proportion of questions for which completely correct answer sets are given^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. - **ReCoRD** (Reading Comprehension with Commonsense Reasoning Dataset): a task on **reading comprehension and knowledge use**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. It is a modified Cloze test: given a news text (CNN/Daily Mail article) and a sentence with a missing entity word; the model must choose which entity from the text fits in the gap^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Answer options are given as all entities mentioned in the article, and they may be semantically equivalent^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Successful solution requires context understanding and common sense. Metrics are maximum token-level F1 and Exact Match for predicted answers^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. - **RTE** (Recognizing Textual Entailment): a binary classification task on **textual entailment** (entailment vs. not entailment)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. The dataset combines examples from several textual inference recognition competitions (RTE 1-5 series)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Each task contains a pair of text fragments (premise-hypothesis); the model must determine whether the hypothesis follows from the text. Unlike many large datasets, RTE is quite small (about 2,500 training examples), but showed significant gains from transfer learning: accuracy rose from ~56% (random guessing level) to ~86% with the advent of models like [BERT](https://systems-analysis.info/eng/BERT_(language_model) "BERT (language model)")^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Nevertheless, at the time of SuperGLUE's launch, model accuracy still lagged behind humans by about 8 percentage points^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}, so RTE was included as one of the tasks that maintained a gap to human level. - **WiC** (Word-in-Context): a task on **resolving word sense ambiguity in context** (WSD)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Two independent sentences are given, each containing the same polysemous word; it is necessary to determine whether this word is used **in the same sense** in both cases^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Data is taken from dictionary resources (WordNet, VerbNet, Wiktionary), thus covering a wide range of words and meanings^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. The task is formalized as binary classification and assessed by the proportion of correct answers. WiC requires the model to understand subtle semantic differences, essentially testing **lexical semantics**. - **WSC** (Winograd Schema Challenge): a task on **coreference resolution using common sense**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Each task consists of one sentence containing a pronoun and a list of two entities (nouns) from the same sentence^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. It is required to determine which of the proposed nouns **the given pronoun refers to**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Example of a classic Winograd sentence: "The trophy didn't fit in the suitcase because it was too small" — a human understands that "it" refers to the suitcase (the suitcase was too small). Such examples cannot be solved without **everyday knowledge and context**^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. GLUE already contained a simplified version of this task (WNLI), but models could not surpass even random level on it for a long time^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Only special techniques, such as adding external data with similar examples, raised model quality on WSC to ~90% by 2019^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. However, humans solve WSC tasks virtually without errors (~96-100% correct answers)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. SuperGLUE includes the original version of WSC in binary classification format (for each "pronoun-entity" pair, the model answers whether they match by reference)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. This task remains one of the most difficult tests requiring commonsense reasoning. All SuperGLUE tests have **closed test sets** with answers unknown to developers^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Models submit their predictions to a server where an aggregate score is calculated — task-averaged accuracy (for tasks with multiple metrics, the internal metric is averaged first)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. This single **SuperGLUE score** simplifies comparing models by overall language intelligence level. ## Results and Model Progress At SuperGLUE's launch, the authors provided as a reference the results of a strong baseline model (enhanced BERT) — and they turned out to be **significantly below human** on all tasks^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. On average, the best model at that time scored approximately **20 points lower** than humans on the aggregate metric^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. On individual tasks, the gap was particularly large: for example, on the WSC task, the model barely reached ~65% accuracy versus 100% for humans (a gap of ~35 points)^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. Even on "simpler" looking tasks (BoolQ, CB, RTE, WiC), automatic systems lagged ~10 points behind human level^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. These differences confirmed that SuperGLUE truly poses a serious challenge to current technologies and cannot be solved trivially. Nevertheless, just a few months after SuperGLUE's appearance, **rapid progress** began^{[\[1\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-neurips-main-1)}. In late 2019, Google researchers presented the **[T5](https://systems-analysis.info/eng/T5_(Text-to-Text_Transfer_Transformer) "T5 (Text-to-Text Transfer Transformer)")** model (Text-To-Text Transfer Transformer) with 11 billion parameters, which achieved an aggregate result of 88.9, closely approaching the human level of ~89.8^{[\[2\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-reddit-t5-2)}. In fact, T5 improved the previous SuperGLUE record by 4.3 points at once and reduced the error rate by almost a third^{[\[2\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-reddit-t5-2)}, leaving only a minimal gap of **0.9 points** to the human score^{[\[2\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-reddit-t5-2)}. Developers noted that SuperGLUE was deliberately selected so that tasks are easy for humans, so the model reaching ~89% became an important milestone^{[\[2\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-reddit-t5-2)}. The first to **surpass average human quality** was Microsoft's **DeBERTa** model (Decoding-enhanced BERT with disentangled attention)^{[\[3\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-deberta-3)}. In January 2021, researchers reported that the DeBERTa version with 1.5 billion parameters scored **89.9 points**, slightly above the human baseline of 89.8^{[\[3\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-deberta-3)}. This was the **first case** when a single model exceeded humans on the SuperGLUE metric^{[\[3\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-deberta-3)}. Additionally, an ensemble of several DeBERTa models raised the record to ~90.3 points^{[\[3\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-deberta-3)}. The DeBERTa model outperformed the previous leader (Google T5) by approximately 0.6% and demonstrated the effectiveness of new ideas in Transformer architecture (separate representation of word content and position, improved mask decoder, etc.)^{[\[4\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-syncedreview-deberta-4)}. Progress did not stop there: as the size and complexity of language models grew, SuperGLUE results continued to improve^{[\[5\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-scaling-5)}. By the end of 2021, Microsoft's **T-NLRv5** model (Microsoft Turing NLR family) topped the leaderboard — it further increased the gap above human level^{[\[5\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-scaling-5)}. The last unresolved tasks for machines on GLUE (e.g., NLI subtleties) were "closed" by this model, which came very close to **full parity with humans** even on the most difficult subtasks^{[\[5\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-scaling-5)}. By 2022-2023, the human-level threshold on SuperGLUE was confidently surpassed by several independent large models^{[\[6\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-ainavigator-benchmarks-6)}. For example, Google's **PaLM** model (540 billion parameters) when fine-tuned on SuperGLUE tasks achieved around 90.4 points, and OpenAI's **GPT-4** model showed an even slightly higher result^{[\[6\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-ainavigator-benchmarks-6)}. By mid-2023, the SuperGLUE leaderboard had several models with scores above 90 (i.e., exceeding the average human level)^{[\[6\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-ainavigator-benchmarks-6)}. One can say that the benchmark is **practically solved** by modern systems^{[\[6\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-ainavigator-benchmarks-6)}: the scores of the best models are so high that they exceed the capabilities of most non-expert humans^{[\[6\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-ainavigator-benchmarks-6)}. This success demonstrates enormous progress in NLP in a short time, but simultaneously points to the need for new, even more challenging tests for the latest models^{[\[6\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-ainavigator-benchmarks-6)}. Subsequent benchmarks are already appearing (e.g., MMLU, BIG-Bench, etc.), designed to test models on broader understanding and erudition beyond SuperGLUE tasks^{[\[6\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-ainavigator-benchmarks-6)}. ## Impact and Further Research SuperGLUE has thus established itself as an **important milestone in the development of evaluation methodologies** in language processing^{[\[3\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-deberta-3)}. In enthusiast and scientific circles, its results have become a kind of "litmus test" for new LLM architectures: achieving or exceeding human level on SuperGLUE is perceived as a sign of an advanced model with deep language understanding^{[\[3\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-deberta-3)}. This has also been reflected in practice — many modern language models that achieved high results on SuperGLUE have formed the basis of applied question-answering systems, dialogue agents, text summarization systems, etc.^{[\[3\]](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_note-microsoft-deberta-3)}. SuperGLUE continues to be used by researchers for fine-tuning and algorithm comparison, although the cutting edge is now gradually shifting to new frontiers of artificial intelligence evaluation. ## External links - Official SuperGLUE website - Original SuperGLUE paper (NeurIPS) - Microsoft article on DeBERTa achieving human level - SuperGLUE dataset page on Papers With Code - Language model benchmark — Wikipedia ## Bibliography - Liang, P. et al. (2022). *Holistic Evaluation of Language Models (HELM)*. arXiv:2211.09110. - Chang, Y. et al. (2023). *A Survey on Evaluation of Large Language Models*. arXiv:2307.03109. - Ni, S. et al. (2025). *A Survey on Large Language Model Benchmarks*. arXiv:2508.15361. - Biderman, S. et al. (2024). *The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned*. arXiv:2405.14782. - Kiela, D. et al. (2021). *Dynabench: Rethinking Benchmarking in NLP*. arXiv:2104.14337. - Ma, Z. et al. (2021). *Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking*. arXiv:2106.06052. - Goel, K. et al. (2021). *Robustness Gym: Unifying the NLP Evaluation Landscape*. arXiv:2101.04840. - Xu, C. et al. (2024). *Benchmark Data Contamination of Large Language Models: A Survey*. arXiv:2406.04244. - Liu, S. et al. (2025). *A Comprehensive Survey on Safety Evaluation of LLMs*. arXiv:2506.11094. - Chiang, W.-L. et al. (2024). *Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference*. arXiv:2403.04132. - Boubdir, M. et al. (2023). *Elo Uncovered: Robustness and Best Practices in Language Model Evaluation*. arXiv:2311.17295. - Huang, L. et al. (2023). *A Survey on Hallucination in Large Language Models*. arXiv:2311.05232. ## References 1. ↑ ^{[1.00](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-0)} ^{[1.01](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-1)} ^{[1.02](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-2)} ^{[1.03](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-3)} ^{[1.04](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-4)} ^{[1.05](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-5)} ^{[1.06](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-6)} ^{[1.07](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-7)} ^{[1.08](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-8)} ^{[1.09](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-9)} ^{[1.10](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-10)} ^{[1.11](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-11)} ^{[1.12](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-12)} ^{[1.13](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-13)} ^{[1.14](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-14)} ^{[1.15](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-15)} ^{[1.16](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-16)} ^{[1.17](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-17)} ^{[1.18](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-18)} ^{[1.19](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-19)} ^{[1.20](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-20)} ^{[1.21](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-21)} ^{[1.22](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-22)} ^{[1.23](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-23)} ^{[1.24](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-24)} ^{[1.25](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-25)} ^{[1.26](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-26)} ^{[1.27](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-27)} ^{[1.28](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-28)} ^{[1.29](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-29)} ^{[1.30](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-30)} ^{[1.31](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-31)} ^{[1.32](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-32)} ^{[1.33](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-33)} ^{[1.34](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-34)} ^{[1.35](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-35)} ^{[1.36](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-36)} ^{[1.37](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-37)} ^{[1.38](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-38)} ^{[1.39](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-39)} ^{[1.40](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-40)} ^{[1.41](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-41)} ^{[1.42](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-42)} ^{[1.43](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-43)} ^{[1.44](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-44)} ^{[1.45](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-45)} ^{[1.46](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-46)} ^{[1.47](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-47)} ^{[1.48](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-48)} ^{[1.49](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-49)} ^{[1.50](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-50)} ^{[1.51](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-51)} ^{[1.52](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-52)} ^{[1.53](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-53)} ^{[1.54](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-54)} ^{[1.55](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-55)} ^{[1.56](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-56)} ^{[1.57](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-57)} ^{[1.58](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-neurips-main_1-58)} Wang, Alex et al. (2019). "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems". *NeurIPS*. [1] 2. ↑ ^{[2.0](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-reddit-t5_2-0)} ^{[2.1](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-reddit-t5_2-1)} ^{[2.2](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-reddit-t5_2-2)} ^{[2.3](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-reddit-t5_2-3)} "Google T5 algorithm scores 88.9 on SuperGLUE language benchmark, compared to 89.8 human baseline". *Reddit /r/linguistics*. [2] 3. ↑ ^{[3.0](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-deberta_3-0)} ^{[3.1](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-deberta_3-1)} ^{[3.2](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-deberta_3-2)} ^{[3.3](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-deberta_3-3)} ^{[3.4](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-deberta_3-4)} ^{[3.5](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-deberta_3-5)} ^{[3.6](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-deberta_3-6)} "Microsoft DeBERTa surpasses human performance on the SuperGLUE benchmark". *Microsoft Research Blog*. [3] 4. [↑](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-syncedreview-deberta_4-0) "Microsoft DeBERTa Tops Human Performance on SuperGLUE NLU Benchmark". *Synced Review*. [4] 5. ↑ ^{[5.0](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-scaling_5-0)} ^{[5.1](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-scaling_5-1)} ^{[5.2](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-microsoft-scaling_5-2)} "Efficiently and effectively scaling up language model pretraining for best language representation model on GLUE and SuperGLUE". *Microsoft Research Blog*. [5] 6. ↑ ^{[6.0](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-ainavigator-benchmarks_6-0)} ^{[6.1](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-ainavigator-benchmarks_6-1)} ^{[6.2](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-ainavigator-benchmarks_6-2)} ^{[6.3](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-ainavigator-benchmarks_6-3)} ^{[6.4](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-ainavigator-benchmarks_6-4)} ^{[6.5](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-ainavigator-benchmarks_6-5)} ^{[6.6](https://systems-analysis.info/eng/SuperGLUE_(benchmark)#cite_ref-ainavigator-benchmarks_6-6)} "The Ultimate Guide to AI Benchmarks". *The AI Navigator*. [6]