--- title: "Elo ranking of language models" source: "https://systems-analysis.info/eng/Elo_ranking_of_language_models" wiki: "systems-analysis.info/eng" article: "Elo_ranking_of_language_models" language: "en" categories: - "Category:English" - "Category:Large language models" - "Category:LLM benchmarks" - "Category:Machine learning" - "Category:Technology" revision_id: 133 wiki_created_at: 2026-09-06T22:17:49Z wiki_modified_at: 2026-09-06T22:17:49Z downloaded_at: 2026-09-07T22:21:14Z --- # Elo ranking of language models **Elo ranking of language models** is a family of methods for evaluating and comparing large language models (LLMs) through pairwise, human-preference comparisons, using rating systems originally developed for competitive games such as chess. Rather than measuring absolute performance on fixed tasks, these methods estimate the *relative* strength of models from direct comparisons of their outputs. The best-known implementation is the platform now called **[Arena](https://systems-analysis.info/eng/LMArena_(Chatbot_Arena) "LMArena (Chatbot Arena)")** (originally **LMSYS Chatbot Arena**, later **LMArena**), where users compare the responses of two anonymous models to the same prompt and vote for the one they prefer.^{[\[1\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_arena_2023-1)} Although the approach is popularly described as "Elo" ranking — after the Elo rating system used in its first version — the leaderboard has since moved to a Bradley–Terry maximum-likelihood formulation, and the operators renamed the published metric **Arena Score**. The word "Elo" nonetheless persists as an informal label for the resulting scores.^{[\[2\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_bt_2023-2)[\[3\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_arenascore_2024-3)} Unlike traditional [benchmarks](https://systems-analysis.info/eng/LLM_benchmarks "LLM benchmarks"), which measure absolute performance on specific tasks, preference-based ranking determines relative ability from votes cast by human evaluators. A higher score reflects stronger aggregated human preference over the platform's particular population of users, prompts, and competing models — not a universal, task-independent measure of model ability.^{[\[1\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_arena_2023-1)} ## History ### Origin of the Elo rating system The Elo rating system is named after **Arpad Elo** (Arpad Emmerich Elo; Hungarian: *Élő Árpád Imre*; 1903–1992), a Hungarian-American physicist and chess master who taught physics at Marquette University in Milwaukee. Because "Elo" is a surname, it is written in mixed case rather than as an acronym.^{[\[4\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-fide_elo-4)} Elo developed the system in the late 1950s as a more statistically grounded replacement for the earlier Harkness system, whose ratings many observers considered inaccurate; he first described it in a series of papers from 1961 onward.^{[\[5\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-elo1978-5)} - The United States Chess Federation (USCF) adopted Elo's system in 1960. - The World Chess Federation (FIDE) adopted it in 1970.^{[\[5\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-elo1978-5)} ### Adaptation for language models The use of preference ratings to evaluate LLMs began with **LMSYS Chatbot Arena**, publicly introduced in a blog post on 3 May 2023; the first published leaderboard already covered voting data collected from about 24 April 2023.^{[\[1\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_arena_2023-1)} The platform was developed by members of LMSYS and UC Berkeley SkyLab; LMSYS itself grew out of a multi-university collaboration that included UC Berkeley, Stanford, UC San Diego, Carnegie Mellon University, and MBZUAI.^{[\[6\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_policy_2024-6)} The platform was subsequently rebranded and spun out: it moved to a dedicated site, lmarena.ai (LMArena), in September 2024; the team incorporated as an independent company in April 2025; and on 28 January 2026 it was renamed **Arena** and moved to the domain arena.ai. Older links, such as the LMSYS leaderboard formerly hosted on Hugging Face, are now largely historical. A detailed timeline is given in the article on [LMArena (Chatbot Arena)](https://systems-analysis.info/eng/LMArena_(Chatbot_Arena) "LMArena (Chatbot Arena)").^{[\[7\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_newsite_2024-7)} ## Methodology ### Mathematical foundations #### Classical Elo system (iterative) In the classical Elo system, a rating is updated *incrementally* after each individual match. The expected probability that model A defeats model B is given by a logistic curve: $$ P(A > B) = \frac{1}{1 + 10^{(R_{B} - R_{A})/400}} $$ where $R_{A}$ and $R_{B}$ are the current ratings before the comparison. After an outcome is observed, the rating of model A is updated as: $$ R'_{A} = R_{A} + K \times (S_{A} - E_{A}) $$ where $K$ is the development coefficient (K-factor), $S_{A}$ is the actual result (1 for a win, 0.5 for a draw, 0 for a loss), and $E_{A}$ is the expected result.^{[\[5\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-elo1978-5)} The main drawback of this approach for evaluating LLMs is its *dependence on the order of matches*: because ratings are recomputed step by step, the final result can shift depending on the sequence in which games are processed. This behaviour suits human players, whose skill changes over time, but is less appropriate for models whose behaviour is essentially fixed. (Open-weight models are effectively static, whereas hosted proprietary models may change silently, which is why the operators try to pin specific API versions.)^{[\[2\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_bt_2023-2)} #### Bradley–Terry model and maximum-likelihood estimation To remove the dependence on match order, modern versions of the platform adopted the **Bradley–Terry model**. The probability that the response of model $i$ is preferred over that of model $j$ is: $$ P(i > j) = \frac{e^{\beta_{i}}}{e^{\beta_{i}} + e^{\beta_{j}}} $$ where $\beta_{i}$ and $\beta_{j}$ are the model coefficients (ratings). Rather than updating scores after each match, the system estimates all coefficients *simultaneously* from the entire set of collected votes using maximum-likelihood estimation (MLE) — equivalent to fitting a logistic regression over all pairwise comparisons. This makes the computation global and independent of chronological order.^{[\[2\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_bt_2023-2)[\[8\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-boubdir2023-8)} For readability, the platform rescales the fitted coefficients — by convention multiplying them by 400 and adding 1000 — so that scores fall in a familiar, chess-like range. This rescaling does not affect predictions; and because the Bradley–Terry model uses a different logistic base than the classical Elo formula, the resulting figures are not numerically identical to chess Elo ratings, which is one further reason the operators adopted the name "Arena Score." Confidence intervals are obtained by bootstrapping, and ties are counted as half a win and half a loss.^{[\[2\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_bt_2023-2)} The transition from the incremental online Elo system to the Bradley–Terry formulation was announced on 7 December 2023, motivated by the observations that the full comparison history is available and that most models are static, so a centralized estimate is both feasible and more stable.^{[\[2\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_bt_2023-2)} In June 2024 the operators renamed the "Elo rating" column to "Arena Score," noting that the quantity had in fact been a Bradley–Terry coefficient rather than a classical Elo rating.^{[\[3\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_arenascore_2024-3)} #### Style control In August 2024 the platform introduced *style control*. Stylistic attributes — such as response length and the use of Markdown headers, bold text, and lists — are added to the Bradley–Terry regression as explicit covariates, and their estimated effect is factored out when computing a model's score. The aim is to isolate a model's substantive quality from confounding stylistic preferences, so that a model is not rewarded merely for producing longer or more heavily formatted answers.^{[\[9\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_style_2024-9)} ## Evaluation process 1. **Anonymous comparison:** the user interacts with two anonymous models in parallel for the same prompt. 2. **Voting:** after reading both responses, the user selects the preferred one (or declares a tie). 3. **Identity reveal:** the model names are shown only after the vote. 4. **Rating update:** scores are recomputed from the accumulated votes, typically in batch, to improve stability.^{[\[1\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_arena_2023-1)} ## Advantages and limitations ### Advantages - **Interpretability:** the underlying idea — who beats whom, and by how much — is easy to understand. - **Scalability:** models can be compared without an exhaustive set of pairwise matchups. - **Alignment with human preference:** scores reflect actual user choices on open-ended tasks rather than fixed multiple-choice benchmarks. ### Limitations - **Volatility.** Individual Elo computations can be unstable; ratings become reliable only after many comparisons.^{[\[8\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-boubdir2023-8)} - **Non-transitivity.** Preferences need not satisfy the condition that A \> B and B \> C imply A \> C, which conflicts with a core assumption of the rating models.^{[\[8\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-boubdir2023-8)} - **Sensitivity to sampling and order.** Robust ratings require large amounts of data, and, under the classical scheme, results are sensitive to the order in which comparisons are aggregated.^{[\[8\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-boubdir2023-8)} - **Style and verbosity bias.** Users may favour longer or more heavily formatted answers; the style-control mechanism was introduced specifically to counteract this.^{[\[9\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_style_2024-9)} - **Prompt-distribution bias.** A score reflects the distribution of prompts submitted on the platform, not uniform coverage of all possible tasks.^{[\[10\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-chiang2024-10)} - **Non-stationarity of hosted models.** Proprietary models served through APIs can change without notice, so the platform attempts to fix specific versions.^{[\[2\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-lmsys_bt_2023-2)} - **Fragility of top rankings.** Because all ratings are coupled, top positions can be sensitive to tiny amounts of data: one study found that dropping as little as 0.003% of human preferences can change the top-ranked model.^{[\[11\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-huang2025-11)} - **Vulnerability to vote rigging.** The interconnected rating mechanism enables "omnipresent" manipulation — influencing a target model's ranking through votes in battles where the target does not even appear — using only hundreds of crafted votes.^{[\[12\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-min2025-12)} - **Institutional and disclosure bias.** Some researchers have argued that private pre-release testing, selective disclosure of results, and asymmetric access to battle data can distort the leaderboard — for example, by letting large providers test many private variants and reveal only the best, while open-weight models are deprecated more often. The platform's operators have disputed the magnitude of these effects while acknowledging the need for clearer policies and greater transparency.^{[\[13\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-singh2025-13)[\[14\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-arena_response-14)} - **Annotator reliability.** Standard Elo-style methods weight every vote equally, ignoring differences in the reliability of the people (or models) casting them. Frameworks such as am-ELO jointly estimate, by maximum likelihood, both model strength and each annotator's reliability, allowing the influence of unreliable or adversarial voters to be down-weighted.^{[\[15\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-amelo2025-15)} - **Non-transitivity in automated (LLM-as-a-judge) variants.** Chatbot Arena itself relies on human votes, but many derived benchmarks that scale the approach — such as AlpacaEval and MT-Bench — use an LLM as the judge. In that setting, judges exhibit position bias and non-transitive preferences, so the resulting rankings become sensitive to the choice of baseline model.^{[\[16\]](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_note-xu2025-16)} ## Conclusion Preference-based (Elo-style) ranking has become an important part of the LLM evaluation ecosystem, offering an intuitive way to compare models on open-ended tasks according to human judgement. The move from the classical Elo update to the Bradley–Terry/MLE formulation, together with style control and bootstrapped confidence intervals, has made rankings substantially more stable and better calibrated. At the same time, documented issues — volatility, non-transitivity, sampling and style biases, fragility, disclosure asymmetries, and susceptibility to manipulation — show that no single preference score fully captures model capability. Ongoing work, including annotator-aware models such as am-ELO, points toward combining multiple evaluation approaches for a more complete picture. ## External links - Arena (LMArena) text leaderboard — the current official leaderboard. - "Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings" — the original LMSYS announcement. ## See also - [LMArena (Chatbot Arena)](https://systems-analysis.info/eng/LMArena_(Chatbot_Arena) "LMArena (Chatbot Arena)") - [LLM benchmarks](https://systems-analysis.info/eng/LLM_benchmarks "LLM benchmarks") ## Literature **Core references** - Chiang, W.-L. et al. (2024). *Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference*. arXiv:2403.04132. - Boubdir, M. et al. (2023). *Elo Uncovered: Robustness and Best Practices in Language Model Evaluation*. arXiv:2311.17295. - Daynauth, R. et al. (2025). *Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat*. arXiv:2411.14483. - Huang, J. Y. et al. (2025). *Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings*. arXiv:2508.11847. - Min, R. et al. (2025). *Improving Your Model Ranking on Chatbot Arena by Vote Rigging*. arXiv:2501.17858. - Liu, Z. et al. (2025). *am-ELO: A Stable Framework for Arena-based LLM Evaluation*. arXiv:2505.03475. - Xu, Y. et al. (2025). *Investigating Non-Transitivity in LLM-as-a-Judge*. arXiv:2502.14074. - Singh, S. et al. (2025). *The Leaderboard Illusion*. arXiv:2504.20879. - Ameli, S. et al. (2024). *A Statistical Framework for Ranking LLM-Based Chatbots*. arXiv:2412.18407. **Technical background** - Elo, A. E. (1978). *The Rating of Chessplayers, Past and Present*. Arco Publishing. archive.org. - Bradley, R. A.; Terry, M. E. (1952). *Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons*. *Biometrika* 39(3/4): 324–345. - Chatzi, I. et al. (2024). *Prediction-Powered Ranking of Large Language Models*. arXiv:2402.17826. - Liu, Y. et al. (2024). *Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators*. arXiv:2403.16950. - Sun, H.; Shen, Y.; Ton, J.-F. (2024). *Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives*. arXiv:2411.04991. - Tang, S.; Wang, Y.; Jin, C. (2025). *Is Elo Rating Reliable? A Study Under Model Misspecification*. arXiv:2502.10985. - Glickman, M. E. (2025). *Paired Comparison Models with Strength-Dependent Ties and Order Effects*. arXiv:2505.24783. - Glickman, M. E. (2025). *Rating Competitors in Games with Strength-Dependent Tie Probabilities*. arXiv:2506.11354. - Hua, H.-F.; Dong, J.; Liu, Z. (2023). *Rating of Players by Laplace Approximation and Dynamic Bradley–Terry Model*. arXiv:2310.10386. **Elo-style ratings outside model leaderboards** - Nair, A. et al. (2025). *Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings*. arXiv:2506.00178. ## References 1. ↑ ^{[1.0](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_arena_2023_1-0)} ^{[1.1](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_arena_2023_1-1)} ^{[1.2](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_arena_2023_1-2)} ^{[1.3](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_arena_2023_1-3)} “Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings”. *LMSYS Org*, 3 May 2023. [1] 2. ↑ ^{[2.0](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_bt_2023_2-0)} ^{[2.1](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_bt_2023_2-1)} ^{[2.2](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_bt_2023_2-2)} ^{[2.3](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_bt_2023_2-3)} ^{[2.4](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_bt_2023_2-4)} ^{[2.5](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_bt_2023_2-5)} “Chatbot Arena: New models & Elo system update”. *LMSYS Org*, 7 December 2023. [2] 3. ↑ ^{[3.0](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_arenascore_2024_3-0)} ^{[3.1](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_arenascore_2024_3-1)} “The Multimodal Arena is Here!”. *LMSYS Org*, 27 June 2024. [3] 4. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-fide_elo_4-0) “Anniversary of Arpad Elo – rating system that changed chess world”. *International Chess Federation (FIDE)*. [4] 5. ↑ ^{[5.0](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-elo1978_5-0)} ^{[5.1](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-elo1978_5-1)} ^{[5.2](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-elo1978_5-2)} Elo, A. E. (1978). *The Rating of Chessplayers, Past and Present*. Arco Publishing. (Preface: the system was used by the USCF from 1960 and adopted by FIDE in 1970.) 6. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_policy_2024_6-0) “LMSYS Chatbot Arena: Live and Community-Driven LLM Evaluation”. *LMSYS Org*, 1 March 2024. [5] 7. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_newsite_2024_7-0) “Announcing a New Site for Chatbot Arena”. *LMSYS Org*, 20 September 2024. [6] 8. ↑ ^{[8.0](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-boubdir2023_8-0)} ^{[8.1](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-boubdir2023_8-1)} ^{[8.2](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-boubdir2023_8-2)} ^{[8.3](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-boubdir2023_8-3)} Boubdir, M. et al. “Elo Uncovered: Robustness and Best Practices in Language Model Evaluation”. *arXiv:2311.17295*. [7] 9. ↑ ^{[9.0](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_style_2024_9-0)} ^{[9.1](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-lmsys_style_2024_9-1)} “Does style matter? Disentangling style and substance in Chatbot Arena”. *LMSYS Org*, 28 August 2024. [8] 10. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-chiang2024_10-0) Chiang, W.-L. et al. “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference”. *arXiv:2403.04132*. [9] 11. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-huang2025_11-0) Huang, J. Y. et al. “Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings”. *arXiv:2508.11847*. [10] 12. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-min2025_12-0) Min, R. et al. “Improving Your Model Ranking on Chatbot Arena by Vote Rigging”. *arXiv:2501.17858*. [11] 13. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-singh2025_13-0) Singh, S. et al. (2025). “The Leaderboard Illusion”. *arXiv:2504.20879*. [12] 14. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-arena_response_14-0) “LMArena Response to 'The Leaderboard Illusion' Writeup”. *Arena / LMArena Blog*. [13] 15. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-amelo2025_15-0) Liu, Z. et al. “am-ELO: A Stable Framework for Arena-based LLM Evaluation”. *arXiv:2505.03475*. [14] 16. [↑](https://systems-analysis.info/eng/Elo_ranking_of_language_models#cite_ref-xu2025_16-0) Xu, Y. et al. “Investigating Non-Transitivity in LLM-as-a-Judge”. *arXiv:2502.14074*. [15]