--- title: "Chinchilla (language model)" source: "https://systems-analysis.info/eng/Chinchilla_(language_model)" wiki: "systems-analysis.info/eng" article: "Chinchilla_(language_model)" language: "en" categories: - "Category:English" - "Category:Google" - "Category:Large language models" - "Category:LLM families" - "Category:Machine learning" - "Category:Technology" revision_id: 85 wiki_created_at: 2026-09-06T22:17:13Z wiki_modified_at: 2026-09-06T22:17:13Z downloaded_at: 2026-09-07T22:20:53Z --- # Chinchilla (language model) **Chinchilla** is a [large language model](https://systems-analysis.info/eng/Large_language_model "Large language model") (LLM) developed by the DeepMind research team and introduced in March 2022[\[1\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-hoffmann2022-1). The model contains approximately **70 billion** parameters and was trained on a text corpus of **1.4 trillion** [tokens](https://systems-analysis.info/eng/Token_(LLM) "Token (LLM)"). The key feature of Chinchilla is its compute-optimal approach to training. Unlike previous models where the main focus was on increasing the number of parameters, Chinchilla was created based on the hypothesis that both model size and the volume of training data must be scaled proportionally. Thanks to this approach, Chinchilla demonstrated superior performance over significantly larger models, such as **[Gopher](https://systems-analysis.info/eng/Gopher_(Google) "Gopher (Google)")** (280 billion parameters) and **GPT-3** (175 billion), across a wide range of language tasks[\[2\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-wali2022-2). ## Background and History The development of Chinchilla was the result of research into LLM scaling conducted at DeepMind, building upon the **[Gopher](https://systems-analysis.info/eng/Gopher_(Google) "Gopher (Google)")** family of models[\[3\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-gopher2022-3). The Gopher model, introduced in 2021, had 280 billion parameters but was trained on a relatively small corpus of 300 billion tokens. At the time, the prevailing approach in the industry was that model performance grew primarily by increasing their size (number of parameters), while the amount of data remained relatively constant. This view drew on earlier scaling laws by Kaplan et al. (2020), which Chinchilla would later revise by showing that training data had been substantially undervalued[\[4\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-kaplan2020-4). ### The Compute-Optimal Training Hypothesis DeepMind researchers hypothesized that many large models, including Gopher, were **undertrained** relative to their size. They did not achieve the maximum possible quality for a given computational budget because they lacked sufficient training data[\[2\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-wali2022-2). The core of the hypothesis was that for optimal use of computational resources, model size and the volume of training data should be increased **proportionally** to each other. In other words, doubling the number of model parameters requires approximately doubling the number of training tokens[\[1\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-hoffmann2022-1). This conclusion contradicted previous research that had overestimated the value of increasing model size, as those studies were conducted with a fixed amount of data. To test this hypothesis, the DeepMind team conducted extensive experiments, training over 400 models ranging from 70 million to over 16 billion parameters on datasets of 5 to 500 billion tokens. The results confirmed that parallel scaling is the optimal strategy. Based on these findings, the Chinchilla model was developed as a practical test of this new paradigm[\[5\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-neurips_proc-5). ## Architecture and Training ### Architectural Features Chinchilla belongs to the family of autoregressive [transformers](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture") and uses essentially the same architecture as Gopher: a decoder-only transformer broadly similar to GPT-2/GPT-3, but adopting RMSNorm instead of LayerNorm and relative positional encodings instead of absolute ones[\[3\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-gopher2022-3). Its main differences from Gopher are aimed at reducing size while maintaining network depth: - **Parameters**: ~70 billion parameters, distributed across 80 layers. - **Model Width**: The number of self-attention heads was reduced to 64 (compared to 128 in Gopher), and the internal layer dimension to 8192 (compared to ~16384 in Gopher). - **Optimizer**: It uses **AdamW** instead of Adam, which improves convergence on large datasets[\[3\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-gopher2022-3). This architecture allowed Chinchilla to maintain the same network depth as Gopher but with a significantly smaller number of parameters, reducing memory and computational requirements. ### Scaling and Training Data To validate the hypothesis, Chinchilla was trained with the same computational budget as Gopher, but with resources reallocated in favor of data. The 70-billion-parameter model was trained on a corpus of **1.4 trillion tokens**, which is approximately four times the amount of data used for Gopher[\[1\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-hoffmann2022-1). This ratio, approximately **20 tokens for every parameter**, became known as the **Chinchilla point** (or "Chinchilla-optimal") and serves as a benchmark for the compute-optimal training of modern LLMs[\[6\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-legalgenie-6). The experiment confirmed that Chinchilla, being trained closer to this optimal limit, was able to realize its potential more fully than undertrained, albeit larger, models. ## Results and Performance Across a wide range of standard benchmarks, Chinchilla demonstrated a significant advantage over previous models. It confidently outperformed not only Gopher but also other state-of-the-art LLMs of its time, including OpenAI's GPT-3 (175 billion parameters) and Megatron-Turing NLG (530 billion parameters)[\[1\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-hoffmann2022-1). The most indicative result came from the comprehensive **[MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark")** (*Measuring Massive Multitask Language Understanding*) benchmark, which evaluates knowledge and reasoning across hundreds of diverse tasks. Chinchilla achieved an average 5-shot accuracy of **67.6%**, setting a new record for models of its class and improving on Gopher's 60.0% by 7.6 percentage points; on the same benchmark GPT-3 scored 43.9%[\[1\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-hoffmann2022-1). In addition to its high performance, Chinchilla also proved to be **economical** to use. Its smaller size (70 billion vs. 175+ billion for its counterparts) means that it requires significantly fewer computational resources for *inference* and *[fine-tuning](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)")*, simplifying its practical application. ## Significance and Impact The Chinchilla research has had a fundamental impact on approaches to [training large language models](https://systems-analysis.info/eng/Training_large_language_models "Training large language models"). - **Chinchilla scaling laws**: The identified optimal ratio between model size and data volume became a de facto standard and a guide for subsequent developments in the industry, later refined to also account for inference costs (see below). - **Shift in focus from size to data**: The work encouraged the industry to pay more attention to creating, cleaning, and expanding training corpora, rather than just indiscriminately increasing the number of parameters. - **Application in [multimodal](https://systems-analysis.info/eng/Multimodal_large_language_models "Multimodal large language models") systems**: Chinchilla was used as the core language component in DeepMind's multimodal model **Flamingo**, which is capable of understanding images and text[\[7\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-wiki_eng-7). Although the Chinchilla model itself was not publicly released, its concepts and the results published in the research paper changed the development trajectory of the entire LLM field, charting a path toward more efficient and balanced growth in artificial intelligence capabilities. ## Later Developments and Critiques ### Inference-aware scaling ("beyond Chinchilla") A widely noted limitation of the Chinchilla scaling laws is that they optimize only for the compute cost of *training* and ignore the cost of *inference*. Because a smaller model is cheaper to run, a model that will serve many requests can be worth training well beyond the 20-tokens-per-parameter ratio. Meta's **LLaMA** models (2023) illustrated this in practice: their smaller models were deliberately trained on far more data than Chinchilla would deem optimal — the 7B model on about 1 trillion tokens (~140 tokens per parameter) — trading extra training compute for lower inference cost[\[8\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-llama2023-8). This idea was formalized by Sardana et al. (2024), who modified the Chinchilla laws to account for inference demand and argued that models expected to serve large request volumes should be trained "smaller and longer than Chinchilla-optimal"; they also observed that model quality continued to improve as the token-to-parameter ratio was pushed to extreme values[\[9\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-sardana2024-9). Later models pushed the ratio much further still — LLaMA 3's 8-billion-parameter model was trained on 15 trillion tokens — reflecting a broad shift toward "over-training" relatively small models. ### Replication of the scaling law (2024) In April 2024, researchers at Epoch AI (Besiroglu, Erdil, Barnett, and You) published a replication of the third of the three methods Hoffmann et al. used to estimate their scaling law — the parametric fit to the loss surface. They reported that the paper's published coefficients fit the reconstructed data poorly and came with implausibly narrow confidence intervals, so narrow that they would require far more experiments than were actually run[\[10\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-besiroglu2024-10). One of the original authors subsequently acknowledged an optimizer-configuration error that explained the overly tight intervals. Importantly, the replication did not overturn the central conclusion: the re-derivation reconciled the third method with the other two, and the finding that parameters and training tokens should be scaled at roughly equal rates (about 20 tokens per parameter) has held up in later work, though the exact optimum is now understood to depend on the quality of the training data[\[10\]](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_note-besiroglu2024-10). ## External links - Chinchilla (language model) — Wikipedia ## Literature - Hendrycks, D.; Gimpel, K. (2016). *Gaussian Error Linear Units (GELUs)*. arXiv:1606.08415. - Loshchilov, I.; Hutter, F. (2017). *Decoupled Weight Decay Regularization*. arXiv:1711.05101. - Shoeybi, M.; et al. (2019). *Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism*. arXiv:1909.08053. - Kaplan, J.; et al. (2020). *Scaling Laws for Neural Language Models*. arXiv:2001.08361. - Brown, T. B.; et al. (2020). *Language Models are Few‑Shot Learners*. arXiv:2005.14165. - Rajbhandari, S.; et al. (2020). *ZeRO: Memory Optimizations Toward Training Trillion Parameter Models*. arXiv:1910.02054. - Press, O.; et al. (2021). *Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation*. arXiv:2108.12409. - Rae, J.; et al. (2021). *Scaling Language Models: Methods, Analysis & Insights from Training Gopher*. arXiv:2112.11446. - Hoffmann, J.; et al. (2022). *Training Compute‑Optimal Large Language Models*. arXiv:2203.15556. - Alayrac, J.‑B.; et al. (2022). *Flamingo: A Visual Language Model for Few‑Shot Learning*. arXiv:2204.14198. - Hendrycks, D.; et al. (2020). *Measuring Massive Multitask Language Understanding*. arXiv:2009.03300. - Touvron, H.; et al. (2023). *LLaMA: Open and Efficient Foundation Language Models*. arXiv:2302.13971. - Sardana, N.; et al. (2024). *Beyond Chinchilla‑Optimal: Accounting for Inference in Language Model Scaling Laws*. arXiv:2401.00448. - Besiroglu, T.; et al. (2024). *Chinchilla Scaling: A Replication Attempt*. arXiv:2404.10102. ## References 1. ↑ [1.0](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-hoffmann2022_1-0) [1.1](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-hoffmann2022_1-1) [1.2](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-hoffmann2022_1-2) [1.3](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-hoffmann2022_1-3) [1.4](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-hoffmann2022_1-4) Hoffmann, J. et al. (2022). "Training Compute-Optimal Large Language Models". *NeurIPS 2022*. [1] 2. ↑ [2.0](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-wali2022_2-0) [2.1](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-wali2022_2-1) Wali, K. (2022). "DeepMind launches GPT-3 rival, Chinchilla". *Analytics India Magazine*. [2] 3. ↑ [3.0](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-gopher2022_3-0) [3.1](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-gopher2022_3-1) [3.2](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-gopher2022_3-2) Rae, J. et al. (2022). "Scaling Language Models: Methods, Analysis & Insights from Training Gopher". *arXiv:2112.11446*. 4. [↑](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-kaplan2020_4-0) Kaplan, J. et al. (2020). "Scaling Laws for Neural Language Models". *arXiv:2001.08361*. [3] 5. [↑](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-neurips_proc_5-0) "Training Compute-Optimal Large Language Models". *proceedings.neurips.cc*. 6. [↑](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-legalgenie_6-0) "What is the Chinchilla Point (\\Chinchilla Optimal\\)?". *Legal Genie*. 7. [↑](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-wiki_eng_7-0) "Chinchilla (language model)". *Wikipedia*. 8. [↑](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-llama2023_8-0) Touvron, H. et al. (2023). "LLaMA: Open and Efficient Foundation Language Models". *arXiv:2302.13971*. [4] 9. [↑](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-sardana2024_9-0) Sardana, N.; Portes, J.; Doubov, S.; Frankle, J. (2024). "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws". *ICML 2024, arXiv:2401.00448*. [5] 10. ↑ [10.0](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-besiroglu2024_10-0) [10.1](https://systems-analysis.info/eng/Chinchilla_(language_model)#cite_ref-besiroglu2024_10-1) Besiroglu, T.; Erdil, E.; Barnett, M.; You, J. (2024). "Chinchilla Scaling: A replication attempt". *arXiv:2404.10102*. [6]