MT-Bench (benchmark)

MT-Bench (short for Multi-Turn Benchmark) is a benchmark for evaluating large language models (LLMs) in multi-turn conversational settings. It was introduced in 2023 by researchers from LMSYS (first author Lianmin Zheng) in the same study that systematically analyzed the LLM-as-a-Judge approach — a scalable, automated way to approximate human-preference judgments when comparing chatbots. In this framing, MT-Bench is the benchmark, while LLM-as-a-Judge is the evaluation method applied to it.

Unlike traditional single-turn tests (such as MMLU), MT-Bench evaluates a model's ability to handle a follow-up turn that builds on its previous response, process new inputs in context, and accurately follow user instructions. The goal is a more realistic evaluation of chatbot performance in open-ended scenarios, focusing on alignment with human preferences and the practical requirements of conversational systems.

Rationale

The development of conversational LLMs such as ChatGPT, GPT-4, and Vicuna revealed a gap between traditional quality metrics and how users actually perceived model responses. Improving a model's alignment with human instructions (for example, through RLHF) did not always translate into higher scores on older, single-turn benchmarks: a small amount of high-quality conversational fine-tuning could teach a model a style strongly preferred by human (and GPT-4) judges while barely moving a knowledge benchmark such as MMLU. Benchmarks such as MMLU and HELM therefore could not reliably distinguish these aligned chat models from the base models they were built on, nor capture the quality of multi-turn interactions and open-ended instruction following.

MT-Bench was created in response to this problem, offering a set of open-ended, dialogue-based questions that focus on two aspects:

1. the model's ability to stay coherent across a two-turn exchange (an initial prompt and one follow-up); and
2. the model's ability to accurately follow complex user instructions.

Benchmark Structure and Content

MT-Bench consists of 80 carefully designed two-turn scenarios covering a range of task types. In each scenario the user poses an initial prompt and then a single predefined follow-up, so every model is scored across exactly two turns; this tests its ability to keep context and adapt to a new instruction rather than to sustain a long, open-ended dialogue. The questions are grouped into 8 task categories:

• Writing — creative tasks (e.g., writing a blog post).
• Roleplay — dialogues in specific roles.
• Extraction — extracting facts from a given context.
• Reasoning — logical-thinking problems.
• Math — mathematical problems.
• Coding — writing or debugging code.
• STEM — science, technology, engineering, and mathematics (called "knowledge I" in the paper).
• Humanities — history, literature, and social sciences (called "knowledge II" in the paper).

Each category contains 10 questions, for 80 in total. The follow-ups are deliberately challenging (for example, sudden clarifying questions) to probe the model in a more realistic conversational setting.

Evaluation Method: LLM-as-a-Judge

A key feature of MT-Bench is the use of a powerful language model as a judge for automated response evaluation (LLM-as-a-Judge). In the original paper GPT-4 served in this role. Two judging modes are used: pairwise comparison, where the judge picks the better of two answers, and single-answer grading, where it rates a response on a 1–10 scale; a reference-guided variant supplies a solution to help grade math and reasoning. The default MT-Bench score reported on the leaderboard is the single-answer grade averaged over both turns.

Automated judging replaces laborious manual annotation. In the study, the human side consisted of 58 expert-level annotators, mostly graduate students. GPT-4 agreed with them about 85% of the time when ties were excluded — slightly higher than the 81% humans agreed with one another — although agreement dropped to roughly 66% once ties and position-inconsistent votes were included. The paper summarizes this as agreement above 80%, comparable to the human–human level.

Several biases of the judge were identified and only partly mitigated: positional bias (favoring the first response), verbosity bias (favoring longer responses), and self-enhancement bias (favoring answers in its own style). To limit positional bias, the judge is called twice with the answer order swapped and a win is counted only if the same answer wins both times, otherwise the result is recorded as a tie; positions may also be randomized at scale. This swapping applies to pairwise judging; the default leaderboard score uses single-answer grading, which rates each answer on its own and so is not subject to answer-order bias, though its absolute scores are less stable and can still reflect verbosity or self-enhancement effects. Chain-of-thought and reference-guided judging are used to improve grading of math and reasoning questions.

Results and Application

MT-Bench cleanly separated models. In the original study GPT-4 led with an average score of 8.99 out of 10, versus 7.94 for GPT-3.5 and about 6.4 for Vicuna-13B. In the math and coding categories the win-rate gap between GPT-3.5 and GPT-4 was small, because both stumbled on the hardest items, so GPT-4's advantage showed up more clearly in single-answer grading than in raw win rates. The authors stress that no single benchmark determines model quality and that MT-Bench should complement, not replace, capability benchmarks.

The dataset and the code to run it were released publicly, letting independent developers evaluate their own models. In its early leaderboard the LMSYS team reported MT-Bench scores alongside Chatbot Arena Elo and MMLU; the Arena side later grew into a broader standalone crowdsourced leaderboard. Although presented together, MT-Bench and Chatbot Arena are different evaluation modes: MT-Bench is a fixed set of 80 questions scored by an LLM (or human) judge, whereas Chatbot Arena collects live, crowdsourced pairwise votes on user-supplied prompts.

Limitations and Criticism

Despite its wide adoption, MT-Bench and the LLM-as-a-Judge approach have several limitations:

• Imperfect judge. The judge model (e.g., GPT-4) is not infallible and cannot always recognize factual errors or hallucinations in the responses being evaluated.
• Difficulty with logic and math. An LLM judge may fail to follow a complex line of reasoning or verify a proof; in the paper the judge was sometimes misled by the supplied answers even on problems it could solve on its own.
• Residual bias. Mitigation reduces but does not remove the judge's bias toward a particular style or format.
• Narrow focus. MT-Bench chiefly measures helpfulness and answer quality; safety, honesty, and harmlessness require separate benchmarks.
• Small size and saturation. With only 80 questions (160 turns), MT-Bench is small: strong models now cluster near the top of the scale, and static question sets are vulnerable to training-data contamination. These pressures motivated harder successors such as MT-Bench-101.

For mission-critical applications, human oversight or combined evaluation methods therefore remain desirable.

Development and Extensions

The success of MT-Bench has spurred extended versions. In 2024 the MT-Bench-101 benchmark was introduced for a finer-grained analysis of conversational ability. Its authors built a three-tier ability taxonomy — perceptivity, adaptability, and interactivity — spanning 13 tasks, with 4,208 turns across 1,388 multi-turn dialogues, which exposed subtle differences in how models behave at different points in a conversation.

External links

• Official MT-Bench data and code (FastChat)
• Original paper: "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"
• LMSYS blog: introducing MT-Bench and the leaderboard
• Language model benchmark — Wikipedia

See also

• LLM-as-a-Judge
• LMArena (Chatbot Arena)
• Elo ranking of language models
• MMLU Benchmark
• LLM quality metrics

Literature

• Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
• Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.
• Ni, S. et al. (2025). A Survey on Large Language Model Benchmarks. arXiv:2508.15361.
• Biderman, S. et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. arXiv:2405.14782.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.
• Liu, S. et al. (2025). A Comprehensive Survey on Safety Evaluation of LLMs. arXiv:2506.11094.
• Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
• Boubdir, M. et al. (2023). Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv:2311.17295.
• Huang, L. et al. (2023). A Survey on Hallucination in Large Language Models. arXiv:2311.05232.