---
title: "BLOOM (language model)"
source: "https://systems-analysis.info/eng/BLOOM_(language_model)"
wiki: "systems-analysis.info/eng"
article: "BLOOM_(language_model)"
language: "en"
categories:
  - "Category:English"
  - "Category:Large language models"
  - "Category:LLM families"
  - "Category:Machine learning"
  - "Category:Technology"
revision_id: 75
wiki_created_at: 2026-09-06T22:17:05Z
wiki_modified_at: 2026-09-06T22:17:05Z
downloaded_at: 2026-09-07T22:20:49Z
---

# BLOOM (language model)

**BLOOM** (**B**igScience **L**arge **O**pen-science **O**pen-access **M**ultilingual Language Model) is a large, open-access language model (LLM) containing **176 billion** parameters. It was developed in 2022 as part of the **BigScience** project—an international collaboration of over 1,000 researchers from 70 countries, spearheaded by [Hugging Face](https://systems-analysis.info/eng/Hugging_Face "Hugging Face")<sup>[\[1\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-bloom_blog-1)</sup>.

BLOOM is an autoregressive transformer model capable of generating coherent text in **46 natural languages** and **13 programming languages**. The model was trained on the Jean Zay supercomputer in France and became one of the first truly open-access alternatives to closed models like [GPT](https://systems-analysis.info/eng/GPT_(OpenAI) "GPT (OpenAI)")-3 from OpenAI<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>.

## Background and Development

The **BigScience** initiative was launched in May 2021 as a one-year research workshop, with the goal of democratizing AI research by collaboratively creating a large, open-source language model<sup>[\[1\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-bloom_blog-1)[\[3\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-akiki2022-3)</sup>. At that time, state-of-the-art LLMs like GPT-3 were developed in a closed-source manner by large corporations that did not disclose their architecture, training data, or source code. The BigScience project brought together over a thousand volunteer researchers from around the world to create a competitive and fully open model<sup>[\[4\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-infoq_2022-4)</sup>.

The project received a grant for computing resources on the French supercomputer **Jean Zay** (grant 2021-A0101012475), owned by GENCI and operated at IDRIS, the computing centre of the French National Centre for Scientific Research (CNRS)<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>. The model's training took place from March 11 to July 6, 2022<sup>[\[5\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-siliconangle_2022-5)</sup>. The development was conducted with maximum transparency: the team published information about data selection, training configurations, and held public discussions, following the project's adopted ethical charter<sup>[\[6\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-oecd_rail-6)</sup>.

## Architecture and Training

### Model Architecture

BLOOM is built on a [decoder-only](https://systems-analysis.info/eng/Decoder-only_models_(architecture) "Decoder-only models (architecture)") autoregressive [transformer architecture](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture"), similar to the GPT-3 model<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>.

| Parameter               | Value                                                                      |
|-------------------------|----------------------------------------------------------------------------|
| **Type**                | Decoder-only transformer                                                   |
| **Parameters**          | 176,247,271,424                                                            |
| **Layers**              | 70                                                                         |
| **Attention heads**     | 112                                                                        |
| **Hidden size**         | 14,336                                                                     |
| **Sequence length**     | 2048 [tokens](https://systems-analysis.info/eng/Token_(LLM) "Token (LLM)") |
| **Activation function** | GELU                                                                       |
| **Positional encoding** | ALiBi (Attention with Linear Biases)                                       |
| **Vocabulary**          | 250,680 tokens (byte-level BPE)                                            |
| **Training tokens**     | ≈366 billion                                                               |

Architectural specifications of BLOOM<sup>[\[7\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-huggingface_model_card-7)</sup>

Two architectural choices distinguish BLOOM from a standard GPT-style model, both selected after systematic ablation experiments<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)[\[8\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-millionhours-8)</sup>:

- **ALiBi positional embeddings**: instead of adding positional information at the embedding layer, ALiBi attenuates attention scores according to the distance between keys and queries. Although originally motivated by extrapolation to longer sequences, it also produced smoother training and better downstream results than both learned and rotary embeddings<sup>[\[9\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-alibi-9)</sup>.
- **Embedding LayerNorm**: an additional normalization layer placed immediately after the embedding layer, which markedly improved training stability at 176B scale (at some cost in performance)<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>.

The tokenizer is a byte-level BPE model with a deliberately large vocabulary of 250,680 tokens, chosen so that low-resource languages would not be excessively over-segmented; because it operates on bytes, tokenization never produces unknown tokens<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>.

Besides the flagship 176B model, the family includes smaller checkpoints of **560M, 1.1B, 1.7B, 3B, and 7.1B** parameters, trained on the same corpus<sup>[\[7\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-huggingface_model_card-7)</sup>.

### Training Data

BLOOM was trained on the specially created **ROOTS** (The **R**esponsible **O**pen-science **O**pen-collaboration **T**ext **S**ources) text corpus. The total data volume was **1.6 terabytes** of cleaned and deduplicated text (≈366 billion tokens seen during training)<sup>[\[10\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-roots_paper-10)</sup>.

The corpus includes texts in **59 languages**<sup>[\[10\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-roots_paper-10)[\[7\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-huggingface_model_card-7)</sup>:

- **46 natural languages**, including English (30% of tokens), Chinese, French, Spanish, Arabic, as well as many low-resource languages (e.g., Chi Tumbuka — 0.00002% of tokens).
- **13 programming languages**, including Python, Java, JavaScript, and C++.

This multilingual and multi-domain dataset was intentionally compiled to make the model useful for a wide range of language communities. The corpus was documented in unusual detail, with a published datasheet describing sources, preprocessing, and known limitations<sup>[\[10\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-roots_paper-10)</sup>.

### Training Infrastructure

Training ran for about 3.5 months and consumed **1,082,990 GPU-hours** on 48 nodes of 8 NVIDIA A100 80 GB GPUs each (384 GPUs in total, with spare nodes held in reserve against hardware failures)<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>. The model was implemented using the **Megatron-LM**<sup>[\[11\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-megatron-11)</sup> and **DeepSpeed**<sup>[\[12\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-zero-12)</sup> frameworks, developed by Nvidia and Microsoft, respectively, combined into a **Megatron-DeepSpeed** fork that applied 3D parallelism (data, tensor, and pipeline parallelism) and bfloat16 mixed-precision training<sup>[\[13\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-tech_behind_bloom-13)</sup>.

### Carbon Footprint

BLOOM was the first LLM of its scale for which a full life-cycle carbon assessment was published. Its final training run was estimated to have emitted approximately **24.7 tonnes of CO₂eq** when counting only dynamic power consumption, and **50.5 tonnes** when equipment manufacturing and idle consumption are included — a comparatively low figure for a model of this size, partly because the Jean Zay supercomputer runs on France's largely low-carbon (nuclear) electricity grid<sup>[\[14\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-carbon-14)</sup>.

## Performance and Application

BLOOM's results depend strongly on the type of task<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>:

- On **multilingual** tasks — machine translation on Flores-101 and multilingual summarization on WikiLingua — BLOOM performs well and outperforms the comparably sized OPT-175B from Meta, with translation quality for several low-resource languages approaching that of dedicated supervised systems.
- On **English-only** benchmarks, the picture is more modest. On the HELM evaluation suite the authors report that BLOOM is roughly on par with previous-generation English-only models such as GPT-3 davinci v1, but behind more recent models including InstructGPT davinci v2 and OPT; it does, however, rank among the best models evaluated for fairness<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)[\[15\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-helm-15)</sup>.
- In the **zero-shot** setting on SuperGLUE, performance is close to the random baseline on many tasks; a single in-context example yields only marginal improvement<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>.

The model is capable of performing a wide range of tasks in a *zero-shot* setting (without additional training), including:

- Generating text in a given style.
- Summarizing documents.
- Answering questions based on context.
- Translating between languages.
- Generating simple program code (though well below the level of specialized models such as Codex)<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>.

To improve its practical utility, the BigScience team later conducted additional multilingual multitask [fine-tuning](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)") on the **xP3** prompt corpus, creating the **BLOOMZ** version, which follows user instructions far more accurately and substantially improves zero-shot task generalization<sup>[\[16\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-bloomz-16)</sup>.

## Limitations and Criticism

Two limitations shaped BLOOM's practical fate<sup>[\[2\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-lescao2022-2)</sup>:

- **Not compute-optimal.** BLOOM was trained on roughly 366 billion tokens for 176 billion parameters — about 2 tokens per parameter. The Chinchilla scaling study, published while BLOOM was already being trained, argued that compute-optimal training requires on the order of 20 tokens per parameter, meaning BLOOM (like GPT-3 and OPT-175B) was substantially **undertrained** relative to its size<sup>[\[17\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-chinchilla-17)</sup>. Subsequent open models — LLaMA, Mistral, Falcon and their successors — achieved comparable or better quality with far fewer parameters trained on far more data, and quickly displaced BLOOM in practical use.
- **Deployment cost.** Inference with the full 176B model requires more than 350 GB of accelerator memory, placing it out of reach of most users. This motivated **Petals**, a BigScience-affiliated system for running BLOOM-scale models collaboratively over the internet, BitTorrent-style, by distributing layers across volunteer machines<sup>[\[18\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-petals-18)</sup>.

## Licensing and Open Access

The full 176-billion-parameter BLOOM model, its source code, and data were released in July 2022. The model is distributed under the specially developed **BigScience BLOOM RAIL (Responsible AI License) v1.0** (identifier \`bigscience-bloom-rail-1.0\`), dated 19 May 2022<sup>[\[19\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-rail_license-19)</sup>.

This license permits free use, modification, and redistribution of the model but imposes a series of use-based restrictions. In particular, it is forbidden to use BLOOM for purposes that contradict the BigScience ethical norms, such as<sup>[\[19\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-rail_license-19)[\[6\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-oecd_rail-6)</sup>:

- Mass surveillance.
- Algorithmic discrimination.
- Spreading disinformation.
- Controlling lethal autonomous weapons.

According to the RAIL Initiative, the BLOOM RAIL license was the first of its kind: the first open license devoted specifically to licensing a machine learning model, combining a permissive intellectual-property grant with behavioural use restrictions<sup>[\[20\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-openrail_faq-20)</sup>. Models released in parallel, such as Meta's OPT-175B and SEER, also carried behavioural-use clauses, but permitted research use only and were therefore not open in the same sense<sup>[\[21\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-openrail_blog-21)</sup>.

In August 2022 the license was generalized into the **BigScience OpenRAIL-M** license, applicable to any AI model, which in turn became the template for **CreativeML OpenRAIL-M** (used by Stable Diffusion) and later **BigCode OpenRAIL-M** (used by StarCoder)<sup>[\[21\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-openrail_blog-21)[\[19\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-rail_license-19)</sup>.

## Legacy

Although BLOOM itself has been superseded as a practical tool, its main contribution proved to be procedural rather than architectural: the project established a template for open, documented model releases — a detailed model card, a corpus datasheet, an ethical charter, published training logs, and a full carbon-footprint accounting — and produced the OpenRAIL licensing family now used well beyond BigScience<sup>[\[3\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-akiki2022-3)[\[14\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-carbon-14)[\[21\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-openrail_blog-21)</sup>. BLOOM also remains a reference point in research on multilingual models, including efforts to extend it to further languages<sup>[\[22\]](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_note-bloomplus1-22)</sup>.

## External links

- <a href="https://huggingface.co/bigscience/bloom" class="external text" rel="nofollow">BLOOM model card on Hugging Face</a>
- <a href="https://bigscience.huggingface.co/blog/bloom" class="external text" rel="nofollow">Official BLOOM announcement — BigScience Blog</a>
- <a href="https://en.wikipedia.org/wiki/BLOOM_(language_model)" class="external text" rel="nofollow">BLOOM (language model) — Wikipedia</a>

## Literature

- Hendrycks, D.; Gimpel, K. (2016). *Gaussian Error Linear Units (GELUs)*. <a href="https://arxiv.org/abs/1606.08415" class="external text" rel="nofollow">arXiv:1606.08415</a>.
- Shoeybi, M.; et al. (2019). *Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism*. <a href="https://arxiv.org/abs/1909.08053" class="external text" rel="nofollow">arXiv:1909.08053</a>.
- Rajbhandari, S.; et al. (2020). *ZeRO: Memory Optimizations Toward Training Trillion Parameter Models*. <a href="https://arxiv.org/abs/1910.02054" class="external text" rel="nofollow">arXiv:1910.02054</a>.
- Press, O.; et al. (2021). *Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation*. <a href="https://arxiv.org/abs/2108.12409" class="external text" rel="nofollow">arXiv:2108.12409</a>.
- Hoffmann, J.; et al. (2022). *Training Compute‑Optimal Large Language Models*. <a href="https://arxiv.org/abs/2203.15556" class="external text" rel="nofollow">arXiv:2203.15556</a>.
- Le Scao, T.; et al. (2022). *What Language Model to Train if You Have One Million GPU Hours?*. <a href="https://arxiv.org/abs/2210.15424" class="external text" rel="nofollow">arXiv:2210.15424</a>.
- Le Scao, T.; et al. (2022). *BLOOM: A 176B‑Parameter Open‑Access Multilingual Language Model*. <a href="https://arxiv.org/abs/2211.05100" class="external text" rel="nofollow">arXiv:2211.05100</a>.
- Luccioni, A. S.; Viguier, S.; Ligozat, A.-L. (2023). *Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model*. *JMLR* 24. <a href="https://arxiv.org/abs/2211.02001" class="external text" rel="nofollow">arXiv:2211.02001</a>.
- Muennighoff, N.; et al. (2022). *Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0)*. <a href="https://arxiv.org/abs/2211.01786" class="external text" rel="nofollow">arXiv:2211.01786</a>.
- BigScience Workshop (2022). *BigScience OpenRAIL‑M License v1.0*. <a href="https://bigscience.huggingface.co/blog/bigscience-openrail-m" class="external text" rel="nofollow">Online specification</a>.
- Akiki, C.; et al. (2022). *BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model*. <a href="https://arxiv.org/abs/2212.04960" class="external text" rel="nofollow">arXiv:2212.04960</a>.
- Yong, Z.‑X.; et al. (2022). *BLOOM+1: Adding Language Support to BLOOM for Zero‑Shot Prompting*. <a href="https://arxiv.org/abs/2212.09535" class="external text" rel="nofollow">arXiv:2212.09535</a>.
- Laurençon, H.; et al. (2022). *The BigScience ROOTS Corpus: A 1.6 TB Composite Multilingual Dataset*. <a href="https://arxiv.org/abs/2303.03915" class="external text" rel="nofollow">arXiv:2303.03915</a>.
- Borzunov, A.; et al. (2023). *Distributed Inference and Fine-tuning of Large Language Models Over The Internet (Petals)*. <a href="https://arxiv.org/abs/2312.08361" class="external text" rel="nofollow">arXiv:2312.08361</a>.
- Liang, P.; et al. (2022). *Holistic Evaluation of Language Models (HELM)*. <a href="https://arxiv.org/abs/2211.09110" class="external text" rel="nofollow">arXiv:2211.09110</a>.

## References

1.  <span id="cite_note-bloom_blog-1">↑ <sup>[1.0](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-bloom_blog_1-0)</sup> <sup>[1.1](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-bloom_blog_1-1)</sup> "BLOOM". *BigScience Blog*. <a href="https://bigscience.huggingface.co/blog/bloom" class="external autonumber" rel="nofollow">[1]</a></span>
2.  <span id="cite_note-lescao2022-2">↑ <sup>[2.00](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-0)</sup> <sup>[2.01](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-1)</sup> <sup>[2.02](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-2)</sup> <sup>[2.03](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-3)</sup> <sup>[2.04](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-4)</sup> <sup>[2.05](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-5)</sup> <sup>[2.06](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-6)</sup> <sup>[2.07](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-7)</sup> <sup>[2.08](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-8)</sup> <sup>[2.09](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-9)</sup> <sup>[2.10](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-10)</sup> <sup>[2.11](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-lescao2022_2-11)</sup> Le Scao, T., et al. (2022). "BLOOM: A 176B-Parameter Open-Access Multilingual Language Model". *arXiv:2211.05100*. <a href="https://arxiv.org/abs/2211.05100" class="external autonumber" rel="nofollow">[2]</a></span>
3.  <span id="cite_note-akiki2022-3">↑ <sup>[3.0](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-akiki2022_3-0)</sup> <sup>[3.1](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-akiki2022_3-1)</sup> Akiki, C.; et al. (2022). "BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model". *arXiv:2212.04960*. <a href="https://arxiv.org/abs/2212.04960" class="external autonumber" rel="nofollow">[3]</a></span>
4.  <span id="cite_note-infoq_2022-4">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-infoq_2022_4-0) "BigScience Releases 176B Parameter AI Language Model BLOOM". *InfoQ*. 26 Jul 2022. <a href="https://www.infoq.com/news/2022/07/bigscience-bloom-nlp-ai/" class="external autonumber" rel="nofollow">[4]</a></span>
5.  <span id="cite_note-siliconangle_2022-5">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-siliconangle_2022_5-0) "Researchers open-source neural network with 176B parameters". *SiliconANGLE*. <a href="https://siliconangle.com/2022/07/12/researchers-open-source-neural-network-176b-parameters/" class="external autonumber" rel="nofollow">[5]</a></span>
6.  <span id="cite_note-oecd_rail-6">↑ <sup>[6.0](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-oecd_rail_6-0)</sup> <sup>[6.1](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-oecd_rail_6-1)</sup> "BigScience BLOOM Responsible AI License (RAIL) 1.0". *OECD.AI Catalogue of Tools & Metrics*. <a href="https://oecd.ai/en/catalogue/tools/bigscience-bloom-responsible-ai-license-rail-10" class="external autonumber" rel="nofollow">[6]</a></span>
7.  <span id="cite_note-huggingface_model_card-7">↑ <sup>[7.0](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-huggingface_model_card_7-0)</sup> <sup>[7.1](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-huggingface_model_card_7-1)</sup> <sup>[7.2](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-huggingface_model_card_7-2)</sup> "bigscience/bloom". *Hugging Face*. <a href="https://huggingface.co/bigscience/bloom" class="external autonumber" rel="nofollow">[7]</a></span>
8.  <span id="cite_note-millionhours-8">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-millionhours_8-0) Le Scao, T.; et al. (2022). "What Language Model to Train if You Have One Million GPU Hours?". *Findings of EMNLP 2022, arXiv:2210.15424*. <a href="https://arxiv.org/abs/2210.15424" class="external autonumber" rel="nofollow">[8]</a></span>
9.  <span id="cite_note-alibi-9">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-alibi_9-0) Press, O.; Smith, N. A.; Lewis, M. (2022). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation". *ICLR 2022, arXiv:2108.12409*. <a href="https://arxiv.org/abs/2108.12409" class="external autonumber" rel="nofollow">[9]</a></span>
10. <span id="cite_note-roots_paper-10">↑ <sup>[10.0](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-roots_paper_10-0)</sup> <sup>[10.1](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-roots_paper_10-1)</sup> <sup>[10.2](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-roots_paper_10-2)</sup> Laurençon, H.; Saulnier, L.; Wang, T.; et al. (2022). "The BigScience ROOTS Corpus: A 1.6 TB Composite Multilingual Dataset". *NeurIPS 2022 Datasets and Benchmarks Track, arXiv:2303.03915*. <a href="https://arxiv.org/abs/2303.03915" class="external autonumber" rel="nofollow">[10]</a></span>
11. <span id="cite_note-megatron-11">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-megatron_11-0) Shoeybi, M.; et al. (2019). "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism". *arXiv:1909.08053*. <a href="https://arxiv.org/abs/1909.08053" class="external autonumber" rel="nofollow">[11]</a></span>
12. <span id="cite_note-zero-12">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-zero_12-0) Rajbhandari, S.; et al. (2020). "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models". *arXiv:1910.02054*. <a href="https://arxiv.org/abs/1910.02054" class="external autonumber" rel="nofollow">[12]</a></span>
13. <span id="cite_note-tech_behind_bloom-13">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-tech_behind_bloom_13-0) "The Technology Behind BLOOM Training". *Hugging Face Blog*. <a href="https://huggingface.co/blog/bloom-megatron-deepspeed" class="external autonumber" rel="nofollow">[13]</a></span>
14. <span id="cite_note-carbon-14">↑ <sup>[14.0](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-carbon_14-0)</sup> <sup>[14.1](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-carbon_14-1)</sup> Luccioni, A. S.; Viguier, S.; Ligozat, A.-L. (2023). "Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model". *Journal of Machine Learning Research*, 24, *arXiv:2211.02001*. <a href="https://arxiv.org/abs/2211.02001" class="external autonumber" rel="nofollow">[14]</a></span>
15. <span id="cite_note-helm-15">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-helm_15-0) Liang, P.; et al. (2022). "Holistic Evaluation of Language Models (HELM)". *arXiv:2211.09110*. <a href="https://arxiv.org/abs/2211.09110" class="external autonumber" rel="nofollow">[15]</a></span>
16. <span id="cite_note-bloomz-16">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-bloomz_16-0) Muennighoff, N.; et al. (2023). "Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0)". *ACL 2023, arXiv:2211.01786*. <a href="https://arxiv.org/abs/2211.01786" class="external autonumber" rel="nofollow">[16]</a></span>
17. <span id="cite_note-chinchilla-17">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-chinchilla_17-0) Hoffmann, J.; et al. (2022). "Training Compute-Optimal Large Language Models". *arXiv:2203.15556*. <a href="https://arxiv.org/abs/2203.15556" class="external autonumber" rel="nofollow">[17]</a></span>
18. <span id="cite_note-petals-18">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-petals_18-0) Borzunov, A.; et al. (2023). "Distributed Inference and Fine-tuning of Large Language Models Over The Internet". *NeurIPS 2023, arXiv:2312.08361*. <a href="https://arxiv.org/abs/2312.08361" class="external autonumber" rel="nofollow">[18]</a></span>
19. <span id="cite_note-rail_license-19">↑ <sup>[19.0](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-rail_license_19-0)</sup> <sup>[19.1](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-rail_license_19-1)</sup> <sup>[19.2](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-rail_license_19-2)</sup> "The BigScience OpenRAIL-M License". *Responsible AI Licenses (RAIL) Initiative*. <a href="https://www.licenses.ai/blog/2022/8/26/bigscience-open-rail-m-license" class="external autonumber" rel="nofollow">[19]</a></span>
20. <span id="cite_note-openrail_faq-20">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-openrail_faq_20-0) "FAQ". *Responsible AI Licenses (RAIL)*. <a href="https://www.licenses.ai/faq-2" class="external autonumber" rel="nofollow">[20]</a></span>
21. <span id="cite_note-openrail_blog-21">↑ <sup>[21.0](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-openrail_blog_21-0)</sup> <sup>[21.1](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-openrail_blog_21-1)</sup> <sup>[21.2](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-openrail_blog_21-2)</sup> "OpenRAIL: Towards open and responsible AI licensing frameworks". *Hugging Face Blog*. <a href="https://huggingface.co/blog/open_rail" class="external autonumber" rel="nofollow">[21]</a></span>
22. <span id="cite_note-bloomplus1-22">[↑](https://systems-analysis.info/eng/BLOOM_(language_model)#cite_ref-bloomplus1_22-0) Yong, Z.-X.; et al. (2023). "BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting". *ACL 2023, arXiv:2212.09535*. <a href="https://arxiv.org/abs/2212.09535" class="external autonumber" rel="nofollow">[22]</a></span>
