--- title: "Gopher (Google)" source: "https://systems-analysis.info/eng/Gopher_(Google)" wiki: "systems-analysis.info/eng" article: "Gopher_(Google)" language: "en" categories: - "Category:English" - "Category:Google" - "Category:Large language models" - "Category:LLM families" - "Category:Machine learning" - "Category:Technology" revision_id: 170 wiki_created_at: 2026-09-06T22:18:24Z wiki_modified_at: 2026-09-06T22:18:24Z downloaded_at: 2026-09-07T22:21:33Z --- # Gopher (Google) **Gopher** is a family of dense autoregressive [large language models](https://systems-analysis.info/eng/Large_language_model "Large language model") developed by DeepMind, now part of Google DeepMind. The name most commonly refers to the largest model in the family, a 280-billion-parameter [decoder-only](https://systems-analysis.info/eng/Decoder-only_models_(architecture) "Decoder-only models (architecture)") [Transformer](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture") announced on 8 December 2021.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1)[\[2\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherBlog-2) The research programme trained six models ranging from 44 million to 280 billion parameters. All six were trained on 300 billion tokens sampled from **MassiveText**, a 2.35-trillion-token English-dominated corpus assembled from web pages, books, news, source code, Wikipedia, and the C4 dataset. The models used a context length of 2,048 tokens.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) Gopher was a dense text-only base language model. It was not a sparse [Mixture-of-Experts](https://systems-analysis.info/eng/Mixture-of-Experts_(MoE) "Mixture-of-Experts (MoE)") model, was not natively multimodal, and did not contain built-in web search, code execution, external tools, or contemporary instruction-following alignment. Dialogue behaviour was demonstrated by prompting the base model with conversation examples rather than by deploying a separately released consumer chatbot. DeepMind did not release Gopher's weights, inference code, or a public API. The model card stated that Gopher was intended for internal research and would not be made publicly available.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) MassiveText was likewise not distributed to third parties. As of 10 July 2026, Gopher is best understood as a historical research model rather than a current Google product. Google DeepMind's active public model catalogue instead centres on model families including Gemini, Gemma, Veo, Imagen, Lyria, Genie, and Gemini Robotics.[\[3\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-CurrentModels-3) Gopher was scientifically important for its unusually broad evaluation across 152 tasks, its detailed discussion of training data and environmental cost, and its analysis of how model scale affected factual knowledge, reading comprehension, toxicity, bias, dialogue, and reasoning. Its most influential consequence was the subsequent development of **[Chinchilla](https://systems-analysis.info/eng/Chinchilla_(language_model) "Chinchilla (language model)")**. That work showed that Gopher was substantially undertrained for its parameter count: a 70-billion-parameter model trained on approximately 1.3 trillion tokens outperformed Gopher under approximately the same training-compute budget.[\[4\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-Chinchilla-4)[\[5\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-ChinchillaBlog-5) Gopher also served as the base model for **GopherCite**, a retrieval-assisted question-answering system trained to support its responses with quoted evidence and abstain when uncertain.[\[6\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherCite-6) ## History and institutional context ### Development at DeepMind Gopher was developed before the creation of the present Google DeepMind organization. In April 2023, Google combined the original DeepMind laboratory with the Google Brain team to form Google DeepMind.[\[7\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GoogleDeepMindFormation-7) It is therefore historically precise to say that Gopher was developed by DeepMind, while describing the institution's current name as Google DeepMind. The flagship Gopher training run took place principally in November and December 2020. The resulting model was dated December 2020 in the accompanying model card, although the technical paper and public announcement appeared approximately one year later, on 8 December 2021.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The public release was not a model deployment. DeepMind simultaneously published three related research directions:[\[2\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherBlog-2) - the Gopher scaling and evaluation paper; - a taxonomy of ethical and social risks from large language models; - the Retrieval-Enhanced Transformer, or RETRO, which used an external text database to reduce dependence on parameter scale. This grouping reflected an attempt to study model capability, social risk, and alternative architectures together rather than presenting parameter scaling as the only route to improved language modelling. ### Research chronology | Date | Publication or system | Significance | |----------------------------|------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **November–December 2020** | Gopher training | The 280B model was trained on 4096 TPU v3 chips. The model card dates the checkpoint to December 2020. | | **8 December 2021** | **Gopher paper and public announcement** | Introduced a six-model scaling family from 44M to 280B parameters, MassiveText, evaluation across 152 tasks, and detailed bias, toxicity, and resource analysis. | | **8 December 2021** | **Ethical and social risk taxonomy** | Classified 21 language-model risks across six broad areas. | | **8 December 2021** | **RETRO** | Demonstrated retrieval-enhanced language modelling with a two-trillion-token external database and a substantially smaller neural model. | | **February 2022** | **Language-model red teaming study** | Used language models to generate adversarial prompts for the 280B dialogue-prompted Gopher system. | | **March 2022** | **GopherCite** | Fine-tuned Gopher with supervised learning and reinforcement learning from human preferences to retrieve and quote evidence. | | **April 2022** | **Chinchilla** | Demonstrated that a 70B model trained on substantially more tokens could outperform Gopher with approximately the same compute. | | **April 2023** | **Formation of Google DeepMind** | Google Brain and DeepMind were combined into one organization. | | **10 July 2026** | Current status | Gopher remains an internal historical research model without public weights or a supported API. | ### Naming and scope The term **Gopher** can refer either to the 280B flagship or, less strictly, to the complete scaling family. The technical paper uses the name specifically for the largest model while referring to the smaller systems as members of the same model series. The following related systems should not be treated as ordinary Gopher checkpoints: - **Dialogue-Prompted Gopher** — the base model prompted with examples of conversation; - **GopherCite** — a fine-tuned retrieval-and-citation system based on the 280B model; - **RETRO** — a separate retrieval-enhanced architecture, not a Gopher derivative; - **Chinchilla** — a separately trained compute-optimal model; - **Gemini** and **Gemma** — much later Google DeepMind model families with different architectures, modalities, training programmes, and distribution models. ## Architecture ### Decoder-only Transformer Gopher is a dense decoder-only Transformer trained with a standard autoregressive next-token objective. For a sequence of tokens $x_{1},\ldots,x_{T}$, the model factorizes the sequence probability as: $$ P_{\theta}(x_{1},\ldots,x_{T}) = \prod\limits_{t = 1}^{T}P_{\theta}\left( x_{t} \mid x_{1},\ldots,x_{t - 1} \right), $$ where $\theta$ denotes the model parameters. A conventional scaled dot-product attention operation is: $$ \operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \operatorname{softmax}\left( \frac{\mathbf{Q}\mathbf{K}^{\mathsf{T}}}{\sqrt{d_{k}}} \right)\mathbf{V}, $$ where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ are the query, key, and value matrices and $d_{k}$ is the head dimension.[\[8\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-Attention-8) Unlike later sparse expert models, every Gopher Transformer block applies its dense feed-forward network to every token. Its 280 billion parameters must therefore participate in ordinary inference rather than being selectively activated by a router. ### Architectural modifications The Gopher family used several modifications relative to the original Transformer:[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) - **RMSNorm** instead of LayerNorm; - relative positional encodings derived from Transformer-XL; - a SentencePiece tokenizer with a 32,000-token vocabulary; - byte-level fallback for text that could not be represented through ordinary vocabulary pieces; - a fixed maximum context of 2,048 tokens; - dense multi-head self-attention; - dense feed-forward layers with an intermediate width four times the main hidden-state dimension. RMSNorm scales a representation according to its root-mean-square magnitude without subtracting the mean. In simplified form: $$ \operatorname{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\sqrt{\frac{1}{d}\sum\limits_{i = 1}^{d}x_{i}^{2} + \epsilon}} \odot \mathbf{g}, $$ where $\mathbf{g}$ is a learned scale vector.[\[9\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-RMSNorm-9) The relative-position mechanism was based on Transformer-XL, which represents token relationships through relative distance rather than relying only on a fixed absolute position embedding.[\[10\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-TransformerXL-10) ### Model-family configurations All six models used the same 2,048-token context and were each trained on 300 billion tokens. The principal architectural configurations were:[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) | Model size | Layers | Attention heads | Head dimension | Hidden dimension | Feed-forward dimension | Maximum learning rate | Global batch, tokens | |-----------------|--------|-----------------|----------------|------------------|------------------------|-----------------------|---------------------------| | **44M** | 8 | 16 | 32 | 512 | 2,048 | $6 \times 10^{- 4}$ | 0.25M | | **117M** | 12 | 12 | 64 | 768 | 3,072 | $6 \times 10^{- 4}$ | 0.25M | | **417M** | 12 | 12 | 128 | 1,536 | 6,144 | $2 \times 10^{- 4}$ | 0.25M | | **1.4B** | 24 | 16 | 128 | 2,048 | 8,192 | $2 \times 10^{- 4}$ | 0.25M | | **7.1B** | 32 | 32 | 128 | 4,096 | 16,384 | $1.2 \times 10^{- 4}$ | 2M | | **Gopher 280B** | 80 | 128 | 128 | 16,384 | 65,536 | $4 \times 10^{- 5}$ | 3M, later increased to 6M | The family was designed to study scale under a broadly fixed data budget. This made the comparison scientifically useful but produced an increasingly poor parameter-to-token ratio at the largest scales. ### Tokenizer The tokenizer was trained with SentencePiece and contained approximately 32,000 subword units.[\[11\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-SentencePiece-11) Byte-level fallback prevented arbitrary text from becoming completely unrepresentable, but a vocabulary and corpus optimized overwhelmingly for English did not produce equal token efficiency or model quality across languages. The comparatively small 32K vocabulary differs from the much larger token inventories used by many later multilingual and code-oriented models. ## Training ### Optimization The models were trained with the Adam optimizer. The learning rate was warmed up during the first 1,500 optimization steps and then reduced with a cosine schedule to approximately one tenth of the maximum value.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) For the 7.1B and 280B models, DeepMind applied global gradient-norm clipping at 0.25. Parameters and activations were represented in bfloat16, with stochastic rounding used to reduce numerical error when converting updated values back to reduced precision. The training programme was implemented in JAX with DeepMind's Haiku neural-network library. ### Distributed training The 280B model was too large to fit on one accelerator. DeepMind combined several forms of distributed execution:[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) - data parallelism; - model and tensor parallelism; - pipeline parallelism; - partitioning of optimizer states; - activation rematerialization, also called gradient checkpointing; - communication–computation overlap. The paper reports that half-precision parameters together with single-precision Adam optimizer state occupied approximately 2.5 TiB during training. This figure excludes some additional activation, buffer, communication, and runtime memory. Activation rematerialization reduced stored activation memory by recalculating intermediate values during backpropagation. It therefore reduced memory requirements at the cost of additional floating-point computation. ### Hardware and duration The principal Gopher run used 4,096 Google TPU v3 chips and lasted approximately 920 hours in November and December 2020.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The paper reports approximately: $$ 6.31 \times 10^{23} $$ floating-point operations for the practical training run. This estimate includes overhead from implementation choices such as rematerialization and is larger than a simplified theoretical parameter–token calculation. The reported training-time breakdown was approximately: | Component | Share of training time | |------------------------------|------------------------| | Dense linear operations | 51% | | Attention | 8% | | Optimizer | 3% | | Model-parallel communication | 7% | | Activation rematerialization | 17% | | Pipeline overhead | 9% | | Other operations | 5% | These values describe DeepMind's specific TPU v3 implementation and should not be treated as universal proportions for dense Transformers. ### Environmental estimate DeepMind reported that the run took place in a Google data centre in Georgia, United States, with an estimated power-usage effectiveness of 1.08. Under the paper's accounting assumptions, Gopher training produced approximately 380 metric tons of net carbon-dioxide-equivalent emissions.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The estimate used an assumed average power draw of approximately 283 watts per TPU chip and Google-specific energy accounting. It excluded much of the broader research and development process, such as failed experiments, earlier scaling runs, data processing, evaluation, storage, and subsequent use of the model. The figure is therefore a developer-provided estimate for the principal training run rather than a complete lifecycle assessment. ## MassiveText training corpus ### Composition MassiveText contained approximately 10.5 TB of filtered text, 2.35 billion documents, and 2.35 trillion tokenizer tokens.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) | Source | Filtered size | Documents | Corpus tokens | Sampling share in Gopher training | |----------------|---------------|-----------|---------------|-----------------------------------| | **MassiveWeb** | 1.9 TB | 604M | 506B | 48% | | **Books** | 2.1 TB | 4M | 560B | 27% | | **C4** | 0.75 TB | 361M | 182B | 10% | | **News** | 2.7 TB | 1.1B | 676B | 10% | | **GitHub** | 3.1 TB | 142M | 422B | 3% | | **Wikipedia** | 0.001 TB | 6M | 4B | 2% | | **Total** | 10.5 TB | 2.35B | 2.35T | 100% | Gopher was trained on 300 billion sampled tokens, equivalent to approximately 12.8% of the corpus's listed token count. Because the sources were sampled with deliberately chosen mixture weights, the training sequence was not simply a uniform 300-billion-token slice of MassiveText. Books were heavily upweighted relative to their document count, while news and GitHub data were downweighted relative to their raw token volume. ### MassiveWeb MassiveWeb was a curated web-text collection created from publicly accessible pages. Processing included: - language identification; - SafeSearch filtering; - heuristic document-quality rules; - removal of repeated text; - exact-document deduplication; - near-duplicate detection; - filtering against evaluation datasets. Near-duplicate filtering used MinHash-style estimates of document similarity, including 13-gram overlap and a high Jaccard-similarity threshold.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The pipeline reduced obvious duplication and low-quality web content but did not establish the truthfulness, neutrality, copyright status, or social representativeness of every retained document. ### Books, news, and Wikipedia The books component contained material with publication dates spanning approximately 1500 to 2008. The news, Wikipedia, MassiveWeb, and GitHub components were principally collected or finalized during 2020. This means that Gopher's parametric knowledge was already historically bounded when the model was announced in December 2021. The model cannot know later events unless they are supplied directly in the prompt. ### Source code The GitHub component was restricted to repositories carrying selected permissive licences, including Apache 2.0, MIT, BSD, ISC, Unlicense, CC0, and Artistic 2.0.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) Licence filtering reduces some legal risk but does not guarantee that every file in a repository was correctly licensed, free of copied material, secure, or representative of high-quality software engineering. Code accounted for only 3% of the Gopher training mixture. Gopher was therefore not designed as a specialized coding model. ### Language distribution The corpus was overwhelmingly English. The paper's language-identification analysis classified approximately 99% of sampled training text as English.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) Small amounts of other languages appeared through multilingual web pages, quotations, names, code, and classification error. Their presence did not make Gopher a balanced multilingual model. The data card explicitly warned against treating MassiveText as appropriate for training a multilingual system. ### Toxic and harmful material DeepMind filtered explicit content and some unsafe pages, but the dataset intentionally retained a broad range of naturally occurring internet language. The paper reported that approximately 0.8% of sampled spans exceeded a 0.5 toxicity threshold under one Perspective API classifier configuration.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) This number is dependent on the classifier, threshold, sampling strategy, and demographic biases of automated toxicity detection. It is not a complete measurement of harmful content. The data documentation acknowledged that MassiveText contained offensive, misleading, stereotypical, or otherwise harmful material and that filtering could not reliably remove all such content without also removing useful text. ### Dataset availability MassiveText was not publicly released. DeepMind also did not release the complete preprocessing implementation or document-level provenance records needed to reconstruct it exactly.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The dataset card states that no formal external ethical-review process was conducted for the original collection. This and the absence of a redistributable corpus limit independent auditing and reproduction. ## Evaluation methodology ### Breadth of evaluation The Gopher family was evaluated on 152 tasks covering:[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) - language modelling; - reading comprehension; - factual and general knowledge; - question answering; - fact verification; - common-sense reasoning; - logical reasoning; - mathematics; - science; - medicine; - humanities and social science; - toxicity and bias; - dialogue; - memorization. Of the tasks for which comparable prior results were available, Gopher exceeded the previous language-model state of the art on 100 of 124, or approximately 81%. This was a broad result across a heterogeneous benchmark collection rather than a single standardized aggregate score. Most evaluations used zero-shot or few-shot prompting. Gopher was not instruction-tuned in the modern sense, and benchmark prompts were often task-specific. ### Selected benchmark results | Benchmark | Setting | Gopher result | Comparison or qualification | |-------------------------------------------------------------------------------|-------------------------|---------------|------------------------------------------------------------------------------| | **[MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark")** | 5-shot | 60.0% | GPT-3: 43.9%; UnifiedQA: 48.9%; estimated human expert: 89.8% | | **RACE-high** | Few-shot paper protocol | 71.6% | GPT-3 comparison in the paper: 46.8% | | **RACE-middle** | Few-shot paper protocol | 75.1% | GPT-3 comparison in the paper: 58.1% | | **FEVER** | Fact verification | 77.5% | Reported for the paper's two-way and three-way settings | | **Natural Questions** | 0-shot | 10.1% | GPT-3 comparison: 14.6% | | **Natural Questions** | 5-shot | 24.5% | Performance increased strongly with demonstrations | | **Natural Questions** | 64-shot | 28.2% | GPT-3 comparison: 29.9% | | **Known Unknowns** | Paper protocol | 63.6% | Tests whether the model distinguishes answerable from unanswerable questions | | **Misconceptions** | Paper protocol | 61.7% | Tests resistance to common false beliefs | [\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The table illustrates that Gopher did not dominate every benchmark. Its largest improvements appeared in several knowledge-intensive and reading tasks, while GPT-3 remained competitive or superior in some open-domain question-answering settings. ### [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark") Gopher's 60.0% five-shot score on Massive Multitask Language Understanding represented a large improvement over the paper's GPT-3 and UnifiedQA baselines.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The gains varied by subject. Performance improved substantially in humanities, social science, medicine, general knowledge, and science and technology, while mathematics remained comparatively weak. The human-expert estimate of 89.8% was far above Gopher, demonstrating that the model's improvement did not amount to expert-level reliability. ### Reading comprehension Reading-comprehension tasks showed some of the clearest benefits from parameter scaling. The paper reported large improvements on RACE, a benchmark based on English examination passages. The gains are consistent with a model's ability to use increased capacity to represent lexical, factual, and discourse patterns. They do not prove that it possesses a robust human-like model of the underlying situation. ### Fact checking and knowledge Gopher performed strongly on FEVER and several knowledge benchmarks. It could reproduce a large amount of information encoded in MassiveText and often answer factual questions from demonstrations alone. However, benchmark accuracy and factual reliability in unrestricted generation are different properties. The dialogue analysis showed that the model could state incorrect information confidently even when prompted to express uncertainty. ### Mathematical and logical reasoning DeepMind found that increased parameter scale produced less consistent improvement in mathematical, logical, and some common-sense tasks than in knowledge and reading comprehension.[\[2\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherBlog-2) Some tasks remained close to chance despite large increases in parameter count. The results helped motivate later work on improved data allocation, chain-of-thought prompting, external tools, executable verification, and specialized post-training. ### Test-set contamination The MassiveText pipeline attempted to remove documents overlapping with evaluation test sets. Filtering used long n-gram matches and related similarity checks.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) No automated method can guarantee complete removal of benchmark contamination. Limitations include: - paraphrased or reformatted test examples; - benchmark questions quoted without their original formatting; - answers present in explanatory documents; - contamination through related training examples; - incomplete knowledge of benchmark provenance. For some evaluations, DeepMind compared results before and after contamination filtering and reported that major conclusions were generally unchanged. Nevertheless, benchmark contamination remains a source of uncertainty. ## Effects of model scale ### Capabilities that improved strongly The Gopher study found relatively strong scaling gains in: - reading comprehension; - factual knowledge; - fact verification; - some question-answering tasks; - toxicity identification; - language modelling; - several academic knowledge domains. Larger models also needed fewer demonstrations than smaller models on some tasks, indicating improved in-context learning. ### Capabilities with weaker scaling Scaling alone produced limited or inconsistent gains in: - formal logical reasoning; - arithmetic and mathematics; - some common-sense tasks; - calibration; - truthfulness under open-ended dialogue; - elimination of social stereotypes; - resistance to toxic prompting. The study therefore did not support the claim that all capabilities automatically improve at the same rate as parameter count. ### Emergent and discontinuous behaviour Some evaluations showed little progress through the smaller models followed by larger improvements near the top of the scale. Such patterns have subsequently been discussed as emergent abilities. Interpretation requires caution. Apparent discontinuities can depend on: - the benchmark metric; - the threshold used to count success; - prompt selection; - sampling noise; - the coarse spacing between model sizes; - contamination; - whether performance is measured continuously or as exact accuracy. The Gopher paper primarily documented empirical behaviour rather than claiming a complete theory of emergence. ## Dialogue behaviour ### Dialogue prompting Gopher was not initially trained as a conversational assistant. DeepMind created a dialogue format by placing examples of user and model turns in the prompt and asking the model to continue the sequence.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) In some conversations, Gopher produced coherent explanations, maintained local context, and gave relevant factual information. These demonstrations showed that sufficiently large base models could imitate a chat format through in-context learning. Dialogue-Prompted Gopher was not equivalent to a contemporary assistant with: - instruction tuning; - preference optimization; - system-level safety policies; - tool access; - retrieval; - persistent memory; - product moderation; - structured action interfaces. ### Failure modes The dialogue analysis identified: - repetition; - confident factual errors; - contradictions; - fabricated explanations; - social stereotypes; - failure to express appropriate uncertainty; - sensitivity to leading or toxic prompts. The official announcement displayed examples in which Gopher confidently gave incorrect answers and denied uncertainty.[\[2\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherBlog-2) A small dialogue-fine-tuning experiment did not produce a clearly preferred system under the limited human study reported in the paper. This highlighted that ordinary supervised dialogue data alone was insufficient to solve alignment and factuality. ## GopherCite ### Architecture and training GopherCite was a 280-billion-parameter question-answering system obtained by fine-tuning Gopher with: - supervised demonstrations; - reinforcement learning from human preferences; - search-engine retrieval; - evidence selection; - explicit abstention. Given a question, the system retrieved relevant pages through Google Search or used a user-provided document. It generated an answer interleaved with short quotations intended to support individual claims.[\[6\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherCite-6) GopherCite was not a new base-model architecture. It combined the existing Gopher checkpoint with retrieval, fine-tuning, reward modelling, and an evidence-oriented interface. ### Reported results Human evaluators judged GopherCite's answers and supporting evidence to be high quality approximately:[\[6\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherCite-6) - 80% of the time on a subset of Natural Questions; - 67% of the time on a subset of ELI5. When the system abstained from the one third of questions for which it estimated the greatest uncertainty, these values increased to approximately: - 90% on Natural Questions; - 80% on ELI5. The results demonstrated the value of selective answering and evidence retrieval. They did not establish that every cited answer was true. ### Limits of citation The TruthfulQA analysis showed that a quotation can genuinely support a claim that is itself misleading, incomplete, or false. Search results may contain errors, satire, advertising, outdated statements, or widely repeated misconceptions. Citation therefore helps users inspect provenance but does not replace: - source-quality assessment; - cross-source verification; - temporal checking; - domain expertise; - calibrated uncertainty; - independent fact checking. GopherCite was a research prototype and was not released as a general public search service. ## Red teaming A 2022 DeepMind study used language models to generate adversarial prompts for Dialogue-Prompted Gopher.[\[12\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-RedTeaming-12) The method combined: - an attacker model that generated candidate prompts; - diversity-enhancing prompt-generation strategies; - classifiers that estimated whether a Gopher response was harmful; - human inspection of selected failures. This approach found tens of thousands of prompts that could elicit offensive or otherwise undesirable responses. The study also investigated memorization and outputs resembling private information, including telephone-number-like strings and other personal data. Automated red teaming increased scale and diversity relative to manually writing every attack, but it inherited limitations from the attacker model and the harm classifier. It could miss behaviours outside their distributions and produce false positives. The work helped establish model-assisted red teaming as a common technique for later frontier-model evaluations. ## Bias, toxicity, and social risk ### Stereotypes and representational bias The Gopher paper evaluated associations involving gender, occupation, race, religion, nationality, and other social categories.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The model reproduced occupational gender stereotypes and generated different sentiment distributions for different demographic groups. Increasing model size did not reliably eliminate these patterns. The analysis did not cover every identity or intersection among identities. Absence of a measured disparity should therefore not be interpreted as evidence of equal treatment. ### Dialect performance The model card reported unequal performance across dialects, including weaker behaviour on some African American English evaluations.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) This reflects both corpus imbalance and limitations in benchmarks and tokenization. A model trained on predominantly standardized English may treat legitimate dialect features as errors, low-quality text, or toxicity. ### Toxic generation Gopher could generate toxic material, particularly when prompted with toxic text. Larger size did not guarantee safer output. Automated toxicity classifiers used in the analysis also displayed demographic bias. Text referring to some religious or identity groups was more likely to receive a high toxicity score even when the surrounding sentence was not abusive.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) Classifier-based safety results must therefore be interpreted alongside human review and subgroup analysis. ### Memorization and information hazards Like other large autoregressive models, Gopher could memorize fragments of training data. Memorization risk depends on: - duplication frequency; - sequence uniqueness; - model capacity; - prompt construction; - decoding strategy; - whether a document appeared repeatedly across sources. MassiveText deduplication reduced but did not eliminate this risk. Because the underlying dataset was not publicly released, independent investigators could not fully audit whether a generated passage came from training data. ### Risk taxonomy A companion DeepMind paper organized language-model risks into six areas and 21 more specific risks:[\[13\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-RiskTaxonomy-13) 1. discrimination, exclusion, and toxicity; 2. information hazards; 3. misinformation harms; 4. malicious uses; 5. human–computer interaction harms; 6. automation, access, and environmental harms. The taxonomy emphasized that optimizing one risk metric can worsen another. For example, aggressive filtering can reduce explicit toxicity while disproportionately censoring discussion by or about minority groups. The paper also stressed that model-level evaluation is insufficient for many risks. Real-world harm depends on deployment context, user expectations, institutional incentives, interface design, access controls, and social power. ## Chinchilla and compute-optimal scaling ### Gopher's parameter–data imbalance All Gopher-family models were trained on 300 billion tokens. For the 44M and 117M systems this represented many passes of data relative to model capacity; for the 280B system it was comparatively little data per parameter. The flagship ratio was approximately: $$ \frac{300\ \text{billion training tokens}}{280\ \text{billion parameters}} \approx 1.07 $$ training tokens per parameter. Later work found this allocation to be far from compute-optimal. ### Chinchilla experiment DeepMind trained Chinchilla with approximately:[\[4\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-Chinchilla-4)[\[5\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-ChinchillaBlog-5) - 70 billion parameters; - 1.3 trillion training tokens; - approximately the same total training compute as Gopher. Chinchilla was therefore one quarter of Gopher's parameter count but used more than four times as many training tokens. | Property | Gopher | Chinchilla | |------------------------------|--------------|----------------------------------| | Parameters | 280B | 70B | | Training tokens | 300B | Approximately 1.3T | | Context | 2,048 | 2,048 in the original comparison | | Approximate training compute | Similar | Similar | | MMLU | 60.0% | 67.5% | | Public weights | Not released | Not generally released | Chinchilla significantly outperformed Gopher across nearly every evaluated task while being cheaper to serve because of its smaller parameter count. ### Scaling-law conclusion The compute-optimal analysis concluded that model parameters and training tokens should increase at broadly similar rates when total compute grows. Earlier scaling practice had disproportionately allocated compute to larger models while holding training data near 300 billion tokens. A simplified compute relation for dense Transformers is often approximated as: $$ C \propto ND, $$ where $C$ is training compute, $N$ is the number of parameters, and $D$ is the number of training tokens. At a fixed value of $C$, increasing $N$ necessarily reduces the token budget $D$. Chinchilla demonstrated that Gopher's point on this trade-off used too many parameters and too few tokens. The resulting “Chinchilla scaling laws” influenced later model development across the industry. Many subsequent systems were trained on substantially more tokens per parameter than Gopher. ### Interpretation Chinchilla did not make Gopher's research obsolete. Gopher provided: - the large controlled family used to observe scaling behaviour; - an extensive benchmark suite; - a documented data pipeline; - a practical large-scale training case; - evidence needed to identify the parameter–data imbalance. Gopher and Chinchilla should therefore be viewed as successive stages of one research programme rather than merely as competing products. ## Relationship to RETRO The Retrieval-Enhanced Transformer was published alongside Gopher but used a different approach to scaling.[\[14\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-RETRO-14) RETRO conditioned generation on passages retrieved from a database containing approximately two trillion tokens. Its largest reported model had approximately 7.5 billion neural parameters, roughly 25 times fewer than GPT-3, while achieving competitive language-modelling performance on several evaluations. The system contained: - a frozen BERT-style retriever; - a large passage database; - a differentiable passage encoder; - chunked cross-attention between retrieved passages and the main language model. RETRO illustrated that some factual information can be stored outside model parameters. This can provide: - cheaper model updates; - lower parameter count; - more inspectable evidence; - access to more text than the model processes during gradient training. It also adds retrieval-specific risks, including irrelevant passages, malicious documents, stale indexes, and dependence on the retriever's similarity function. RETRO is not a smaller Gopher checkpoint. It was a separate architecture investigated as an alternative to pure parameter scaling. ## Availability and licensing ### Model access DeepMind did not make Gopher publicly available.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) As of 10 July 2026, there is no official: - Gopher weight download; - Gopher API; - Google AI Studio endpoint; - Vertex AI deployment; - self-hosting package; - model licence for third-party inference; - supported consumer chatbot using the Gopher name. The paper, model card, data documentation, and related research publications are public, but these materials do not contain sufficient artefacts to reproduce the original model exactly. ### MassiveText access MassiveText was not released to third parties. Some components, such as C4, Wikipedia, and public GitHub repositories, are independently accessible, but DeepMind's exact versions, filters, deduplication decisions, sampling weights, and proprietary or licensed book and news collections are not reproducible from the paper alone. A newly assembled dataset using similarly named sources would not be identical to MassiveText. ### Intended use The model card described Gopher as an internal research model intended to study large-scale language modelling. It excluded unsupervised downstream deployment and harmful or deceptive generation from intended use.[\[1\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-GopherPaper-1) The lack of public deployment reduced direct misuse risk but also limited independent safety evaluation. ## Ethical and governance considerations ### Data provenance MassiveText drew on publicly accessible and licensed material, but public accessibility does not by itself resolve: - copyright; - author consent; - privacy; - cultural representation; - accuracy; - later deletion requests; - labour involved in producing the source material. The absence of a document-level public provenance map makes individual claims difficult to audit. ### Personal information Large web corpora can contain names, addresses, telephone numbers, email addresses, usernames, medical details, and other personal information. Deduplication does not remove all such data and can sometimes preserve rare sequences precisely because they are unique. A language model may memorize and reproduce some of this information under specially constructed prompts. ### Overreliance Gopher's fluent dialogue can encourage users to attribute understanding, expertise, or confidence to the model beyond what its accuracy supports. The risk is especially significant for: - health; - law; - finance; - education; - politics; - historical interpretation; - scientific claims. The model card did not recommend deployment in such applications. ### Unequal access The model's computational scale and non-public status restricted access to DeepMind researchers. This reduced open misuse but also prevented broad independent investigation of claims and failure modes. The tension between safety through restricted access and accountability through external evaluation remains a central governance issue for frontier models. ### Automation and labour The companion risk taxonomy included possible economic and labour effects from language-model automation. Gopher itself was not deployed widely enough to produce a direct product-level labour impact, but it contributed to research demonstrating that large models could perform increasingly broad knowledge tasks. ### Environmental responsibility The paper's explicit compute and carbon reporting was unusually detailed for its time. However, the reported total did not constitute a complete environmental assessment. Broader impacts include: - accelerator manufacturing; - data-centre construction; - water use; - network and storage infrastructure; - discarded hardware; - evaluation and inference; - repeated experiments. ## Research impact and legacy ### Broad capability evaluation Gopher established one of the broadest early evaluation suites for a frontier language model. Its 152-task analysis demonstrated that a single average benchmark can conceal substantial differences among knowledge, reading, reasoning, toxicity, and dialogue. This encouraged later developers to publish category-specific model cards and safety evaluations. ### Compute-optimal training The most direct legacy is Chinchilla. The finding that Gopher was overparameterized and undertrained changed the standard approach to allocating language-model training compute. Later developers increasingly emphasized: - more training tokens; - higher-quality data; - repeated data where necessary; - smaller models for a fixed compute budget; - lower inference cost; - data–parameter scaling rather than parameter count alone. ### Retrieval and attribution RETRO and GopherCite explored two complementary ways of reducing reliance on inaccessible parametric memory: - retrieving passages during language-model prediction; - retrieving sources and citing evidence during question answering. These ideas contributed to the later development of retrieval-augmented generation, search-grounded assistants, inline citations, and source-aware agents. ### Safety evaluation The Gopher programme combined capability evaluation with: - bias analysis; - toxicity analysis; - memorization studies; - dialogue failure analysis; - automated red teaming; - a general risk taxonomy; - environmental reporting. The methods were incomplete but helped establish safety evaluation as a normal part of frontier-model research rather than an optional product-stage addition. ### Model and data documentation The paper included both model-card and data-card information. It documented intended use, excluded uses, data sources, filtering, language distribution, training compute, emissions, and known limitations. This disclosure was broader than many contemporary language-model releases, although the absence of weights and training data restricted verification. ### Historical position Gopher occupies an intermediate position in large-language-model history: - after GPT-3 demonstrated broad few-shot learning; - alongside dense scaling systems such as Jurassic-1 and Megatron-Turing NLG; - before instruction-tuned assistants became dominant; - immediately before Chinchilla changed compute-optimal scaling practice; - before retrieval-augmented and tool-using systems became mainstream; - before Google's Gemini and Gemma model families. Its importance lies less in current practical use than in the evidence it provided about what parameter scale could and could not accomplish. ## Current status As of 10 July 2026: - Gopher remains a historical internal research model; - no public weights or supported API have been released; - MassiveText remains unavailable as a complete dataset; - GopherCite remains a research prototype; - Chinchilla is regarded as the compute-efficient successor to the Gopher scaling experiment; - Google DeepMind's active public model portfolio uses other brands, principally Gemini and Gemma for general language-model work.[\[3\]](https://systems-analysis.info/eng/Gopher_(Google)#cite_note-CurrentModels-3) There is no official announcement of a new production model called Gopher 2, Gopher Pro, or Gopher Ultra. Such names should not be inferred from later Google model releases. ## External links - Gopher technical paper - Official Google DeepMind overview - Chinchilla compute-optimal scaling paper - GopherCite paper - RETRO paper - Ethical and social risk taxonomy - Red Teaming Language Models with Language Models - Current Google DeepMind model catalogue ## See also - [Large language model](https://systems-analysis.info/eng/Large_language_model "Large language model") - [Transformer (deep learning architecture)](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture") - [Chinchilla (language model)](https://systems-analysis.info/eng/Chinchilla_(language_model) "Chinchilla (language model)") ## Literature - Rae, J. W. et al. *Scaling Language Models: Methods, Analysis & Insights from Training Gopher*. arXiv:2112.11446, 2021. - Hoffmann, J. et al. *Training Compute-Optimal Large Language Models*. arXiv:2203.15556, 2022. - Borgeaud, S. et al. *Improving Language Models by Retrieving from Trillions of Tokens*. ICML, 2022. - Menick, J. et al. *Teaching Language Models to Support Answers with Verified Quotes*. arXiv:2203.11147, 2022. - Weidinger, L. et al. *Ethical and Social Risks of Harm from Language Models*. arXiv:2112.04359, 2021. - Perez, E. et al. *Red Teaming Language Models with Language Models*. arXiv:2202.03286, 2022. - Vaswani, A. et al. *Attention Is All You Need*. Advances in Neural Information Processing Systems 30, 2017. - Dai, Z. et al. *Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context*. ACL, 2019. - Zhang, B.; Sennrich, R. *Root Mean Square Layer Normalization*. arXiv:1910.07467, 2019. ## References 1. ↑ [1.00](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-0) [1.01](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-1) [1.02](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-2) [1.03](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-3) [1.04](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-4) [1.05](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-5) [1.06](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-6) [1.07](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-7) [1.08](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-8) [1.09](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-9) [1.10](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-10) [1.11](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-11) [1.12](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-12) [1.13](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-13) [1.14](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-14) [1.15](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-15) [1.16](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-16) [1.17](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-17) [1.18](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-18) [1.19](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-19) [1.20](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-20) [1.21](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-21) [1.22](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-22) [1.23](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-23) [1.24](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-24) [1.25](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherPaper_1-25) Rae, J. W. et al. (2021). *Scaling Language Models: Methods, Analysis & Insights from Training Gopher*. arXiv:2112.11446, submitted 8 December 2021; revised 21 January 2022. https://arxiv.org/abs/2112.11446 2. ↑ [2.0](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherBlog_2-0) [2.1](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherBlog_2-1) [2.2](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherBlog_2-2) [2.3](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherBlog_2-3) Google DeepMind (2021). *Language modelling at scale: Gopher, ethical considerations, and retrieval*, 8 December 2021. https://deepmind.google/blog/language-modelling-at-scale-gopher-ethical-considerations-and-retrieval/ 3. ↑ [3.0](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-CurrentModels_3-0) [3.1](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-CurrentModels_3-1) Google DeepMind. *Models*. Accessed 10 July 2026. https://deepmind.google/models/ 4. ↑ [4.0](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-Chinchilla_4-0) [4.1](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-Chinchilla_4-1) Hoffmann, J. et al. (2022). *Training Compute-Optimal Large Language Models*. arXiv:2203.15556. https://arxiv.org/abs/2203.15556 5. ↑ [5.0](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-ChinchillaBlog_5-0) [5.1](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-ChinchillaBlog_5-1) Google DeepMind (2022). *An empirical analysis of compute-optimal large language model training*, 12 April 2022. https://deepmind.google/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/ 6. ↑ [6.0](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherCite_6-0) [6.1](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherCite_6-1) [6.2](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GopherCite_6-2) Menick, J. et al. (2022). *Teaching Language Models to Support Answers with Verified Quotes*. arXiv:2203.11147. https://arxiv.org/abs/2203.11147 7. [↑](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-GoogleDeepMindFormation_7-0) Pichai, S. (2023). *Google DeepMind: Bringing together two world-class AI teams*. Google, 20 April 2023. https://blog.google/innovation-and-ai/technology/ai/april-ai-update/ 8. [↑](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-Attention_8-0) Vaswani, A. et al. (2017). *Attention Is All You Need*. Advances in Neural Information Processing Systems 30. https://arxiv.org/abs/1706.03762 9. [↑](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-RMSNorm_9-0) Zhang, B.; Sennrich, R. (2019). *Root Mean Square Layer Normalization*. arXiv:1910.07467. https://arxiv.org/abs/1910.07467 10. [↑](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-TransformerXL_10-0) Dai, Z. et al. (2019). *Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context*. Proceedings of ACL 2019. https://arxiv.org/abs/1901.02860 11. [↑](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-SentencePiece_11-0) Kudo, T.; Richardson, J. (2018). *SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing*. Proceedings of EMNLP 2018. https://aclanthology.org/D18-2012/ 12. [↑](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-RedTeaming_12-0) Perez, E. et al. (2022). *Red Teaming Language Models with Language Models*. arXiv:2202.03286. https://arxiv.org/abs/2202.03286 13. [↑](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-RiskTaxonomy_13-0) Weidinger, L. et al. (2021). *Ethical and Social Risks of Harm from Language Models*. arXiv:2112.04359. https://arxiv.org/abs/2112.04359 14. [↑](https://systems-analysis.info/eng/Gopher_(Google)#cite_ref-RETRO_14-0) Borgeaud, S. et al. (2021). *Improving Language Models by Retrieving from Trillions of Tokens*. arXiv:2112.04426; ICML 2022. https://arxiv.org/abs/2112.04426