--- title: "Encoder-only models" source: "https://systems-analysis.info/eng/Encoder-only_models" wiki: "systems-analysis.info/eng" article: "Encoder-only_models" language: "en" categories: - "Category:English" - "Category:Large language models" - "Category:Machine learning" - "Category:Technology" revision_id: 136 wiki_created_at: 2026-09-06T22:17:51Z wiki_modified_at: 2026-09-06T22:17:51Z downloaded_at: 2026-09-07T22:21:16Z --- # Encoder-only models **Encoder-only models** are a class of large language model (LLM) architectures based exclusively on the **encoder** part of the **[Transformer](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture")** architecture. Unlike models that use a decoder or a full encoder-decoder architecture, these models specialize in **Natural Language Understanding (NLU)** tasks. The flagship model and pioneer of this approach is **BERT** (Bidirectional Encoder Representations from Transformers), developed by Google in 2018. ## Concept and Architecture The core idea behind encoder-only models is to create deep, **contextualized representations** (embeddings) for each token in an input sequence. Thanks to the **self-attention** mechanism in the Transformer, each token can "see" and interact with all other tokens in the sequence, allowing the model to capture rich context. A key feature is **bidirectionality**: the representation of each token is formed based on both the left and right context simultaneously. This fundamentally distinguishes them from autoregressive decoder-only models (like GPT), which are inherently unidirectional. Architecturally, the model is a stack of $N$ identical encoder layers. Each layer consists of two main sub-layers: 1. **Multi-Head Self-Attention:** Computes a contextualized representation for each token. 2. **Feed-Forward Network:** Applies a non-linear transformation to each token's representation. At the output, the model generates a sequence of vectors of the same length as the input sequence, where each vector is a rich representation of the corresponding input token. ## Pre-training Tasks To teach the model to understand language in a bidirectional context, special self-supervised pre-training tasks are used: ### Masked Language Modeling (MLM) This is the primary and most important task for encoder-only models, first introduced in BERT. - **How it works:** A small percentage of tokens (usually 15%) in the input sequence is randomly hidden (masked). The model's task is to predict the original values of these masked tokens using their surrounding bidirectional context. - **Objective:** This task forces the model to learn deep semantic and syntactic relationships between words. ### Next Sentence Prediction (NSP) This task (also from the original BERT) was designed to teach the model to understand relationships between sentences. - **How it works:** The model is given a pair of sentences and must determine whether the second sentence is a logical continuation of the first in the original text. - **Status:** Later research (e.g., in the RoBERTa model) showed that NSP is less effective than MLM, and it is often replaced by other tasks or removed entirely. ## Applications Encoder-only models are not designed for free-form text generation, as they lack an autoregressive decoder. Their strength lies in text analysis and understanding. The model's output vector representations are used to solve a wide range of NLU tasks: - **Text Classification:** For tasks like sentiment analysis or topic detection, the representation of the special \`\[CLS\]\` token, which is added to the beginning of each sequence, is used. Its final vector aggregates information about the entire sequence. - **Token Classification:** For tasks like Named Entity Recognition (NER) or Part-of-Speech (POS) tagging, the vector representations of each individual token are used. - **Question Answering:** In tasks where the answer is a snippet from a given text (extractive QA), the model is trained to predict the start and end tokens of the answer. - **Embedding Extraction:** Encoder models are often used as general-purpose text encoders to obtain high-quality embeddings for sentences or documents, which can then be used in search engines or for semantic similarity tasks. ## Key Models and Their Evolution - **BERT** (2018): The pioneer of the architecture, setting new records on numerous NLP benchmarks. - **RoBERTa** (2019): "A Robustly Optimized BERT Pretraining Approach." Showed that BERT's performance could be significantly improved through longer training on more data and by abandoning the NSP task. - **ALBERT** (2019): "A Lite BERT." A model with significantly fewer parameters due to techniques like embedding factorization and cross-layer parameter sharing. - **DistilBERT** (2019): A smaller version of BERT created using knowledge distillation, which is faster and lighter while retaining most of the original's performance. - **ELECTRA** (2020): Introduced a more efficient pre-training task, **replaced token detection**, where the model learns to distinguish original tokens from "fake" ones generated by a small generator model. - **DeBERTa** (2020): Introduced a "disentangled attention" mechanism that separately encodes content and the relative positions of tokens. ## External links - Transformer (deep learning) — Wikipedia ## See also - [BERT](https://systems-analysis.info/eng/BERT_(language_model) "BERT (language model)")