BERT (language model)

BERT (Bidirectional Encoder Representations from Transformers) is a large language model (LLM) for natural language understanding, developed by researchers at Google and introduced in 2018. BERT marked a new era in natural language processing (NLP) by demonstrating unprecedented performance across a wide range of tasks and establishing the "pre-train and fine-tune" paradigm as an industry standard.

The key innovation of BERT is its deeply bidirectional architecture, which allows the model to consider the context of a word from both the left and the right simultaneously across all layers of the network. This is achieved through a new pre-training task: Masked Language Modeling (MLM).

Name and Operating Principle

The acronym BERT stands for Bidirectional Encoder Representations from Transformers.

• Bidirectional: This points to the model's core feature—the ability to process a word's context in both directions (left-to-right and right-to-left) simultaneously. Unlike unidirectional models (like GPT), which only see the preceding context when processing a word, BERT sees the entire sequence at once, allowing it to form a deeper and more accurate understanding of the word's meaning.
• Encoder: This signifies that BERT uses only the encoder part of the Transformer architecture. The encoder's task is to read an input sequence of text and create a rich contextual representation (vector) for each token. BERT is not designed for free-form text generation like decoder-based models.
• Representations: The model is trained to create high-quality numerical representations (vectors or embeddings) for words and sentences, which can then be used to solve various NLP tasks.
• from Transformers: This indicates that the model's architecture is based entirely on the Transformer.

History

The development of BERT was the result of several key breakthroughs in NLP:

1. Contextual Embeddings: Models like Word2vec and GloVe created static vectors for words, disregarding context. The ELMo model (2018) was a step forward, generating context-dependent representations using bidirectional LSTMs, but this bidirectionality was "shallow" (a concatenation of two unidirectional models).
2. Transfer Learning and GPT: In mid-2018, OpenAI introduced the GPT model, which demonstrated the effectiveness of pre-training a large Transformer model on unlabeled data followed by fine-tuning on specific tasks. However, GPT was strictly unidirectional (left-to-right), limiting its capabilities in tasks that require understanding the full context. A parallel line of work, ULMFiT, had shown that fine-tuning a pre-trained language model could transfer effectively to downstream classification tasks.

Recognizing these limitations, Google researchers led by Jacob Devlin (with Ming-Wei Chang, Kenton Lee, and Kristina Toutanova) developed BERT to create a truly deeply bidirectional model. The paper was first published on arXiv in October 2018 and later presented at NAACL-HLT 2019, where it won the Best Long Paper Award; the code and pre-trained models were released open-source, sparking an explosion of interest in the research community. BERT set new state-of-the-art results on 11 key NLP benchmarks, including GLUE and SQuAD, and became the emblem of what Sebastian Ruder had shortly before termed "NLP's ImageNet moment" — the shift, begun with ELMo, ULMFiT, and GPT, away from pre-trained word vectors toward pre-training entire models that could then be adapted to a multitude of tasks.

Architecture

BERT is based entirely on the encoder part of the Transformer architecture. It consists of several identical layers stacked on top of each other. There are two main versions:

• BERT-Base: 12 layers, 12 attention heads, hidden state size of 768, ~110 million total parameters.
• BERT-Large: 24 layers, 16 attention heads, hidden state size of 1024, ~340 million total parameters.

Each layer contains two main sub-layers:

1. Multi-Head Self-Attention mechanism: Allows each token in the input sequence to "attend" to all other tokens, weighing their importance to determine its own contextual meaning.
2. Feed-Forward Network: Applied to each token individually.

Both versions accept input sequences of at most 512 tokens; this limit, tied to BERT's learned absolute positional embeddings, restricts its direct use on longer texts and was one of the motivations for later models such as ModernBERT.

Input Data

For proper operation, BERT requires a specific input data format. The token sequence fed into the model always begins with a special `[CLS]` (classification) token, which is used for text classification tasks. If a pair of sentences is provided as input (e.g., in question-answering tasks), they are separated by a `[SEP]` (separator) token.

The final representation of each input token is the sum of three embeddings:

• Token Embedding: A vector corresponding to a specific token from the vocabulary (BERT uses WordPiece tokenization, a subword scheme introduced for Google's neural machine translation system).
• Segment Embedding: Indicates which sentence (the first or second) the token belongs to.
• Positional Embedding: Indicates the token's position in the sequence, as the Transformer architecture itself does not account for word order.

Pre-training Tasks

To achieve deep bidirectionality, BERT is trained on two unique tasks simultaneously.

Masked Language Modeling (MLM)

This is the key innovation of BERT. Instead of predicting the next word, as in standard language models, BERT predicts randomly "masked" words within a sentence. The process is as follows:

• 15% of the tokens from the input sequence are randomly selected.
• Of these 15%:
  • 80% are replaced with a special `[MASK]` token.
  • 10% are replaced with a random token from the vocabulary.
  • 10% remain unchanged.

• The model's task is to predict the original values of these 15% of tokens based on their surrounding (left and right) context.

This scheme forces the model to learn deep semantic and syntactic relationships between words and allows it to be truly bidirectional. A structural drawback is its sample inefficiency: the training signal comes only from the masked 15% of tokens, which later motivated alternative objectives such as ELECTRA's replaced-token detection.

Next Sentence Prediction (NSP)

This task was designed to teach BERT to understand relationships between sentences, which is critical for tasks like question-answering or natural language inference (NLI). The model is given a pair of sentences (A and B) and must predict whether sentence B is a logical continuation of sentence A.

• In 50% of cases, B is indeed the next sentence from the original text.
• In 50% of cases, B is a random sentence taken from elsewhere in the corpus.

Later research (e.g., in the RoBERTa model) showed that the NSP task is less important than MLM and can be abandoned in favor of more efficient training schemes, but it played a significant role in the original BERT.

Application and Fine-Tuning

The power of BERT lies in the transfer learning paradigm. After large-scale and costly pre-training on massive corpora — English Wikipedia and BooksCorpus, together about 3.3 billion words — the pre-trained model can be easily and quickly fine-tuned for a specific application.

The fine-tuning process typically looks like this: 1. A small, untrained task-specific layer (e.g., a classifier for sentiment analysis) is added to the pre-trained BERT architecture. 2. The entire model (including BERT's weights and the new layer) is trained on a small, labeled dataset for that specific task.

Examples of tasks for which BERT is adapted:

• Text classification (sentiment analysis, spam filters): A classifier is added to the output of the `[CLS]` token.
• Question-answering systems (e.g., SQuAD): The model is trained to predict the start and end tokens of the answer within a given text.
• Named Entity Recognition (NER): A classifier is added to the output of each token to determine if it is part of a name, organization, date, etc.

Variants and Derivative Models

The success of BERT has led to the emergence of a whole family of models based on its ideas:

• RoBERTa (from Facebook AI): "A Robustly Optimized BERT." It is not a new architecture but rather the result of more thorough and prolonged training of BERT: on more data, without the NSP task, and with dynamic masking. RoBERTa demonstrated that the original BERT was "undertrained" and surpassed it on all major benchmarks.
• DistilBERT (from Hugging Face): A smaller version of BERT created using knowledge distillation. DistilBERT is 40% smaller, 60% faster, and retains 97% of BERT's performance, making it ideal for use in production and on resource-constrained devices.
• ALBERT (A Lite BERT, from Google): A version optimized for reducing the number of parameters. It uses two key techniques — embedding factorization and cross-layer parameter sharing — and replaces NSP with a sentence-order prediction (SOP) objective. This allows for the creation of much larger models with fewer parameters.
• ELECTRA (from Google/Stanford): Replaces masked language modeling with replaced token detection: a small generator network substitutes plausible tokens, and the main model (the discriminator) must decide for every position whether the token is original or replaced. Because the learning signal comes from all tokens rather than only the masked 15%, ELECTRA is substantially more compute-efficient than BERT at equal quality.
• DeBERTa (from Microsoft): Introduces disentangled attention, in which each word is represented by two separate vectors — one for content and one for relative position — with attention weights computed from disentangled matrices, plus an enhanced mask decoder. Scaled to 1.5 billion parameters, DeBERTa became in January 2021 the first model to surpass the human baseline on the SuperGLUE benchmark (89.9 versus 89.8), outperforming the eight-times-larger T5-11B. Its successor, DeBERTaV3, combines disentangled attention with ELECTRA-style pre-training and gradient-disentangled embedding sharing, and remained for years the strongest encoder below one billion parameters — and the principal baseline against which ModernBERT was later measured.
• mBERT (Multilingual BERT): A version of BERT pre-trained on 104 languages simultaneously. It has shown a surprising ability for cross-lingual knowledge transfer, generalizing to languages it was never explicitly aligned on.
• Domain-specific models: Numerous models pre-trained or fine-tuned on data from specific fields, such as BioBERT (biomedicine), SciBERT (scientific texts), and FinBERT (finance; several independent models share this name).
• ModernBERT (2024): Released in December 2024 by Answer.AI and LightOn (with collaborators from Hugging Face, NVIDIA, and others), ModernBERT is a ground-up modernization of the encoder, offered in base (≈149 million) and large (≈395 million) parameter sizes and trained on about 2 trillion tokens of text and code. It extends the context window to 8192 tokens (16× the original's 512), replaces absolute positional embeddings with RoPE (Rotary Position Embeddings), alternates global and local attention, adopts GeGLU activations and Flash Attention, drops the NSP objective, and raises the masking rate to 30% — while retaining BERT's encoder-only, fine-tuning-friendly design. A subsequent controlled study noted that, because the training corpora differ, it is not fully settled how much of ModernBERT's advantage stems from architecture rather than data — DeBERTaV3 remains competitive under matched conditions.
• Newer encoders (2025–2026): ModernBERT triggered a broader renewal of interest in encoder-only models, with several successors appearing in 2025 — including NeoBERT (a 250-million-parameter English encoder with a 4,096-token context), the multilingual EuroBERT family (210M–2.1B parameters, ~15 languages), and mmBERT (a ModernBERT-style multilingual model trained on over 1,800 languages). The trend has continued into 2026 with a steady stream of language-specific and domain-specific ModernBERT-style encoders (for Japanese, German, Turkish, Portuguese, and others).

Comparison with Other Models

Comparison of BERT with Other Key Architectures

Model	Developer	Architecture	Context Direction	Primary Task
BERT	Google	Encoder	Bidirectional	Text understanding, classification, extraction
GPT	OpenAI	Decoder	Unidirectional (left-to-right)	Text generation, sequence continuation
XLNet	Google / CMU	Autoregressive (permutation-based)	Bidirectional (in theory)	Text understanding (alternative to MLM)
T5	Google	Encoder-Decoder	Bidirectional (encoder) + Unidirectional (decoder)	Universal "text-to-text" transformation

Impact

BERT brought about a true revolution in NLP and laid the foundation for many subsequent developments:

1. Solidified the "pre-train and fine-tune" paradigm as the dominant approach in NLP.
2. Proved the importance of deep bidirectional context for language understanding.
3. Lowered the barrier to entry for creating high-performance NLP systems, as researchers and developers no longer needed to build complex architectures from scratch for each task.
4. Was integrated into Google Search (from October 2019): the rollout initially affected about one in ten English-language queries in the US and was later extended to more than 70 languages; it was described as one of the biggest updates to the search engine since RankBrain. BERT is still listed among the operational ranking systems in Google's own documentation.
5. Spawned an entire ecosystem of derivative models, tools, and research — a distinct subfield of analysis papers known as "BERTology"; a 2020 survey already synthesized more than 150 separate studies of how BERT works, making it one of the most examined and most cited works in the field of AI.

Although newer and larger models like GPT-3 and GPT-4 have surpassed BERT on many benchmarks (especially in generative tasks), BERT and its variants remain powerful and widely used tools for tasks requiring deep text understanding: encoders are still the default choice for retrieval (including RAG pipelines), classification, content moderation, and entity extraction, where their small size and low inference cost matter more than generation. As late as December 2024 — six years after its release — the original BERT was still the second most downloaded model on the Hugging Face hub, with more than 68 million downloads per month.

External links

• Official BERT repository on GitHub
• BERT announcement on the Google AI blog
• The Illustrated BERT — a visual explanation of the BERT architecture
• "NLP's ImageNet Moment Has Arrived" — S. Ruder, The Gradient
• BERT (language model) — Wikipedia

Literature

• Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.
• Vaswani, A. et al. (2017). Attention Is All You Need. arXiv:1706.03762.
• Mikolov, T. et al. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781.
• Pennington, J. et al. (2014). GloVe: Global Vectors for Word Representation. EMNLP 2014, pp. 1532–1543.
• Peters, M. E. et al. (2018). Deep Contextualized Word Representations. arXiv:1802.05365.
• Howard, J.; Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. arXiv:1801.06146.
• Radford, A. et al. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI.
• Wu, Y. et al. (2016). Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv:1609.08144.
• Wang, A. et al. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arXiv:1804.07461.
• Rajpurkar, P. et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250.
• Liu, Y. et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692.
• Lan, Z. et al. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv:1909.11942.
• Sanh, V. et al. (2020). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.
• Clark, K. et al. (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. arXiv:2003.10555.
• He, P. et al. (2021). DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv:2006.03654.
• He, P. et al. (2023). DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. arXiv:2111.09543.
• Pires, T. et al. (2019). How Multilingual is Multilingual BERT?. arXiv:1906.01502.
• Yang, Z. et al. (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv:1906.08237.
• Raffel, C. et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683.
• Lee, J. et al. (2020). BioBERT: a pre-trained biomedical language representation model for biomedical text mining. arXiv:1901.08746.
• Beltagy, I. et al. (2019). SciBERT: A Pretrained Language Model for Scientific Text. arXiv:1903.10676.
• Araci, D. (2019). FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. arXiv:1908.10063.
• Rogers, A.; Kovaleva, O.; Rumshisky, A. (2020). A Primer in BERTology: What We Know About How BERT Works. arXiv:2002.12327.
• Warner, B. et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663.
• Antoun, W. et al. (2025). ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance. arXiv:2504.08716.
• Le Breton, L. et al. (2025). NeoBERT: A Next-Generation BERT. arXiv:2502.19587.
• Boizard, N. et al. (2025). EuroBERT: Scaling Multilingual Encoders for European Languages. arXiv:2503.05500.
• Marone, M. et al. (2025). mmBERT: A Modern Multilingual Encoder with Annealed Language Learning. arXiv:2509.06888.