---
title: "Large language model architectures"
source: "https://systems-analysis.info/eng/Large_language_model_architectures"
wiki: "systems-analysis.info/eng"
article: "Large_language_model_architectures"
language: "en"
categories:
  - "Category:English"
  - "Category:Large language models"
  - "Category:Machine learning"
  - "Category:Technology"
revision_id: 212
wiki_created_at: 2026-09-06T22:18:58Z
wiki_modified_at: 2026-09-06T22:18:58Z
downloaded_at: 2026-09-07T22:21:54Z
---

# Large language model architectures

**Large Language Model (LLM) architectures** are the fundamental principles and structures that define how large language models are built, trained, and operated. Modern LLMs, capable of understanding and generating human language, are almost entirely based on the **[Transformer](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture")** architecture<sup>[\[1\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Vaswani2017-1)</sup>, but incorporate numerous enhancements and different approaches aimed at improving efficiency, scalability, and capabilities.

## Families of LLM Architectures (Transformers)

Modern large language models are based on the [Transformer architecture](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture")<sup>[\[1\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Vaswani2017-1)</sup>, but they utilize it differently depending on the objective: to understand text, generate a continuation, or transform one text into another. In practice, three main families are distinguished, all while preserving the core principles of the Transformer<sup>[\[2\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-2)[\[3\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-3)[\[4\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-4)</sup>.

### 1. [Encoder-only](https://systems-analysis.info/eng/Encoder-only_models "Encoder-only models")

The model uses only a stack of [encoders](https://systems-analysis.info/eng/Encoder_(Transformer) "Encoder (Transformer)") and processes the entire input text bidirectionally. Pre-training is typically based on masked-language modeling (MLM): some tokens are masked, and the model learns to reconstruct them from their context. Thanks to this bidirectional context, such models excel at understanding and scoring tasks, such as classification, named entity recognition, document re-ranking, and extractive QA. They are not designed for autoregressive generation from scratch.

Additionally, alternative pre-training objectives are used for the encoder-only family in practice: *replaced token detection (RTD) in ELECTRA* (where a discriminator model identifies replaced tokens) and *contrastive learning* for bi-encoders in semantic search/retrieval (InfoNCE/softmax loss on "query-document" pairs, as in Dense Passage Retrieval). When used in RAG, encoder-only models act either as a *bi-encoder* (separate encoding of query and document for fast ANN search) or as a *cross-encoder* (joint encoding of the pair for precise re-ranking).

**Advantages:**

- High-quality text understanding due to bidirectional context: classification, NER, fact extraction, re-ranking, extractive QA.
- Parallel processing and high throughput: a single forward pass without auto-regression; convenient for batching mass scoring tasks.
- Natural integration with search and RAG: as a bi-encoder for fast semantic search; as a cross-encoder for precise re-ranking.
- Effective adaptation: relatively compact variants (≈100–300M parameters; BERT-base ≈110M) achieve high quality after task-specific fine-tuning.
- Stable latency, independent of the generated response length (no step-by-step decoding); well-suited for offline scoring of large collections.
- Ability to extend the context window using relative/rotary position embeddings and/or local-sparse attention (e.g., Longformer/BigBird), which is useful for long documents.

**Disadvantages:**

- No native generative capabilities: requires a decoder or an external generative module for dialogues and detailed answers.
- Limited in interactive scenarios: no step-by-step generation with state preservation.
- Mismatch between pre-training objective and free-form generation tasks: MLM is less aligned with generation compared to causal language modeling.
- Historically limited context window (often 512 tokens in base configurations with absolute positions); extension requires special position/attention schemes and/or fine-tuning.
- For retrieval tasks, separate contrastive fine-tuning of a bi-encoder and/or cross-encoder is required; without it, search/re-ranking quality is typically lower than that of specially trained models.

**Representative models:** [BERT](https://systems-analysis.info/eng/BERT_(language_model) "BERT (language model)") and its derivatives, as well as RoBERTa and DeBERTa (advanced encoder-only variants); from alternative pre-training objectives, ELECTRA (RTD). <sup>[\[5\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-5)[\[6\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-6)[\[7\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-7)[\[8\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-8)[\[9\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-9)[\[10\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-10)</sup>

### 2. [Decoder-only](https://systems-analysis.info/eng/Decoder-only_models_(architecture) "Decoder-only models (architecture)")

This architecture uses only a stack of decoders with causal (left-to-right) attention: the model predicts the next token based on a given prefix. This training mode—causal language modeling (CLM)—makes these models a natural choice for generation tasks: dialogues, detailed answers, creative text, and code. The trade-off is an increase in latency and KV cache size with long prompts. In practice, various engineering techniques are widely used for decoder-only models: reducing the KV cache with MQA and GQA, accelerating inference with speculative decoding, and server-side optimizations (PagedAttention/vLLM, continuous batching, chunked prefill).<sup>[\[11\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-11)[\[12\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-12)[\[13\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-13)[\[14\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-14)[\[15\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-15)</sup>

**Advantages:**

- Natural text generation (CLM): strong zero-shot and few-shot capabilities; scales well.<sup>[\[16\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-16)</sup>
- Versatility: a single model can solve many tasks through instructions and examples in the prompt; naturally combines with RAG and tool use.
- Mature ecosystem: established practices for instruction fine-tuning and alignment ([RLHF](https://systems-analysis.info/eng/Reinforcement_learning_from_human_feedback_(RLHF) "Reinforcement learning from human feedback (RLHF)"), DPO); open-source and commercial implementations are available.<sup>[\[17\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-17)[\[18\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-18)</sup>
- Rich stack of inference optimizations: MQA/GQA reduce KV cache size and increase throughput; speculative decoding accelerates inference without changing the output distribution; PagedAttention/vLLM with continuous batching and chunked prefill improve end-to-end GPU utilization.<sup>[\[19\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-19)[\[20\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-20)[\[21\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-21)[\[22\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-22)[\[23\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-23)</sup>
- Support for structured generation for strict output formats (JSON/SQL/DSL), which simplifies integration with information systems and APIs.<sup>[\[24\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-24)[\[25\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-25)</sup>

**Disadvantages:**

- Increased generation latency: sequential output; the cost of a new token grows with the length of the already processed context (KV cache).
- Less efficient for "long input–short output" profiles (summarization, translation) compared to encoder-decoder models, where the input is encoded only once.
- Limited by unidirectional context: in understanding tasks, it can sometimes be inferior to models with bidirectional representations (encoder-only / encoder-decoder).
- Memory for the KV cache can be a bottleneck with long prompts and large batches. <sup>[\[26\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-26)</sup>
- Quantization of activations/KV (INT8/FP8) speeds up inference but can degrade quality on long contexts/code; requires careful validation (especially with strict SLAs).

**Representative models:** [GPT-3](https://systems-analysis.info/eng/GPT_(OpenAI) "GPT (OpenAI)"), [GPT-4](https://systems-analysis.info/eng/GPT_(OpenAI) "GPT (OpenAI)") (architecture and dataset details are not publicly disclosed), [LLaMA](https://systems-analysis.info/eng/LLaMA_(Meta_AI) "LLaMA (Meta AI)") and *Llama 3* (8B/70B, 2024).<sup>[\[27\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-27)[\[28\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-28)[\[29\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-29)[\[30\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-30)</sup>

### 3. [Encoder-decoder](https://systems-analysis.info/eng/Encoder%E2%80%93decoder_architecture "Encoder–decoder architecture")

This architecture combines both components. The encoder operates bidirectionally, while the decoder operates causally. The encoder analyzes the input once to form its representation; the decoder then generates the output by attending to this representation via cross-attention. This separate approach is particularly useful for tasks that require transforming a long input text into a short output, such as machine translation, summarization, and document-based question answering. Although this method requires greater overall computational resources (due to two stacks and cross-attention), its advantage is controlled generation based on a complete analysis of the source text. The encoding is performed only once and is reused throughout the entire inference process.

**Advantages:**

- Conditional generation: the decoder uses cross-attention to the input representation. <sup>[\[31\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-31)</sup>
- Efficient in "long input → short output" scenarios: the input is encoded only once.
- Convenient for the "text-to-text" format and controlled output (task-specific prefixes, special instructions). <sup>[\[32\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-32)</sup>
- Stability and efficiency with long source texts: during the decoding phase, only the self-attention over the output grows, while cross-attention reuses fixed keys/values from the encoder (the input is not "re-read" at each step).

**Disadvantages:**

- Two stacks increase memory and computational requirements for training and deployment.
- On extremely long sequences, the total latency is comparable to decoder-only models; auto-regression remains the bottleneck.
- Fewer universal chat models compared to the decoder-only family; more often used as a high-quality seq2seq engine for specific tasks.
- With very long inputs, the memory for cross-attention keys/values in each decoder layer increases (across the entire source), requiring careful serving planning.

**Representative models:** T5 (including T5 v1.1 and the instruction fine-tuning practice in *FLAN-T5*) and BART. <sup>[\[33\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-33)[\[34\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-34)[\[35\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-35)</sup>

## Dense Transformers

The classic and most common LLM architecture, where nearly the entire set of model parameters is engaged in processing each token. Unlike sparse approaches (e.g., Mixture-of-Experts), there is no selective activation of subnetworks—every block operates on every token. <sup>[\[1\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Vaswani2017-1)</sup>

### Principle of Operation and Architecture

**Basic Structure.** The model is a stack of N identical Transformer blocks. Each block includes:

1.  **Multi-Head Self-Attention.** For each token, three vectors are computed: Q (query), K (key), and V (value); attention is defined as $\operatorname{softmax}\!\left( \frac{QK^{\top} + M}{\sqrt{d_{k}}} \right) \cdot V$, where $M$ is a mask (causal and/or padding mask) that excludes invalid positions. Multiple attention "heads" run in parallel to consider different aspects of the context (H heads, usually $d_{head} = \frac{d_{model}}{H}$); their number grows with the model's scale. <sup>[\[1\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Vaswani2017-1)</sup>
2.  **Feed-Forward Network (FFN).** Two linear layers with a non-linearity between them (typically GELU/SiLU; in several modern models, SwiGLU). The intermediate dimensionality is typically $\approx 4\, d_{model}$; when using SwiGLU, it is often set to $\approx \frac{8}{3}\, d_{model}$ to maintain a comparable number of parameters. The FFN contains a significant portion of the parameters. <sup>[\[1\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Vaswani2017-1)[\[36\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-36)</sup>

**Additional Components.** Residual connections and layer normalization are used; modern LLMs more often apply Pre-LN (normalization before sub-blocks), which improves training stability at greater depths. In addition to classic LayerNorm, **RMSNorm** is increasingly used (it reduces computational overhead and performs well in large models); some families also apply normalization in the attention space (e.g., normalizing Q/K before softmax). Positional representations can be absolute or relative; for long contexts, RoPE has become the de facto standard.

#### Examples of Models and Scale

- [BERT](https://systems-analysis.info/eng/BERT_(language_model) "BERT (language model)")-Large: 24 layers, 1024 hidden size, 16 attention heads, ≈340M parameters. <sup>[\[37\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-37)</sup>
- [GPT-3](https://systems-analysis.info/eng/GPT_(OpenAI) "GPT (OpenAI)") (175B): 96 layers, 12288 hidden size, 96 attention heads, ≈175B parameters. <sup>[\[38\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-38)</sup>
- [LLaMA](https://systems-analysis.info/eng/LLaMA_(Meta_AI) "LLaMA (Meta AI)")-65B: 80 layers, 8192 hidden size, 64 attention heads, ≈65B parameters. <sup>[\[39\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-39)</sup>
- [PaLM](https://systems-analysis.info/eng/PaLM_(Pathways_Language_Model) "PaLM (Pathways Language Model)")-540B: 118 layers, hidden size of around 18432, ≈540B parameters. <sup>[\[40\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-40)</sup>

#### Advantages

- Uniform blocks, well-studied training regimes, and predictable scaling behavior.
- Quality improves as a power law with the growth of parameters and data; the compute-optimal regime involves increasing both model size and the volume of training tokens. <sup>[\[41\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-41)[\[42\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-42)</sup>
- The same architecture can cover a wide range of tasks after fine-tuning, without changes at the layer level.

#### Disadvantages

- Full self-attention has quadratic complexity with respect to sequence length ($O(n^{2})$), which limits the context window. <sup>[\[1\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Vaswani2017-1)</sup>
- Full parameter activation at each generation step: in a non-MoE decoder, the inference cost per token grows approximately proportionally to the number of parameters.
- Bottlenecked by memory bandwidth (memory-bound): loading weights from HBM often limits inference speed.

#### Scaling and Context Limitations

- Memory for parameters grows linearly with model size; training memory increases due to gradients and optimizer states.
- Base configurations were historically limited to 2,000–4,000 tokens. Modern positional schemes (RoPE) and extension techniques (Position Interpolation, YaRN, etc.) can increase the window by an order of magnitude or more, but at the cost of additional computational/memory load. <sup>[\[43\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-43)[\[44\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-44)</sup>

### Modern Optimizations

- **FlashAttention.** An exact attention implementation that is aware of the GPU memory hierarchy; it reduces memory costs and accelerates training/inference for long sequences. <sup>[\[45\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-45)</sup>
- **KV Cache Reduction and Management.** Multi-Query Attention and Grouped-Query Attention reduce cache size and memory traffic; at the server level, PagedAttention (vLLM) increases throughput via page-based cache management. <sup>[\[46\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-46)[\[47\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-47)[\[48\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-48)</sup>
- **Speculative Decoding.** A draft model proposes a continuation, which the main model quickly verifies; this achieves acceleration without changing the output distribution. <sup>[\[49\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-49)</sup>

## Sparse Models and Mixture-of-Experts (MoE)

MoE is a method for increasing a model's capacity without a proportional increase in computation per token. Instead of a single large FFN block, each layer uses a set of parallel "experts" (multiple independent FFNs), and a trainable router (gating network) selects the top-k most relevant experts for each token (typically k=1–2; in some models, k=4). Only the selected experts are activated; their outputs are weighted and summed. This allows the total number of parameters to reach hundreds of billions or even trillions, while only a small fraction is engaged at each step. <sup>[\[50\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Switch-50)[\[51\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-GLAM-51)</sup>

#### Examples of Models and Scale

- **Switch Transformer (Google)**: up to ~1.6T parameters; top-1 routing (one expert per token). Showed that MoE allows for a dramatic increase in capacity at comparable per-token costs. <sup>[\[50\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Switch-50)</sup>
- **GLaM (Google)**: 1.2T parameters, 64 experts per layer, top-2; ≈96.6B parameters (≈8%) are activated for each token. <sup>[\[51\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-GLAM-51)</sup>
- **Mixtral 8×7B (Mistral AI)**: ~46.7B total parameters, ≈12.9B active per token, top-2. <sup>[\[52\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Mixtral8x7-52)[\[53\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Mixtral8x7_paper-53)</sup>
- **[Mixtral 8×22B](https://systems-analysis.info/eng/Mixtral_(Mistral_AI) "Mixtral (Mistral AI)")**: ~141B total parameters, ≈39B active per token, top-2. <sup>[\[54\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Mixtral8x22-54)</sup>
- **[DBRX](https://systems-analysis.info/eng/DBRX_(language_model) "DBRX (language model)") (Databricks)**: 132B total parameters, ≈36B active per token; 16 experts and top-4 routing (fine-grained MoE). <sup>[\[55\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-DBRX-55)</sup>

##### Advantages

- Computational cost is determined by the number of active experts (k), not the total number of parameters: trillion-scale models can be trained and used with costs comparable to much smaller dense models. <sup>[\[51\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-GLAM-51)</sup>
- Specialization: experts automatically "adapt" to languages/domains/patterns, improving quality in multi-domain tasks.
- Flexible deployment: frequently used experts can be kept in memory, while rare ones are loaded on demand (with appropriate infrastructure).

##### Limitations

- Load balancing: without regularization, the router can "stick" to a subset of experts (router collapse). Auxiliary losses (load-balancing) and improved routing schemes are needed. <sup>[\[50\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Switch-50)</sup>
- Complexity of distributed computing: requires expert parallelism and all-to-all communication; communication overhead and memory management become bottlenecks. <sup>[\[56\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-NVIDIA_MoE-56)</sup>
- Training stability: router settings and capacity limits are crucial; otherwise, quality/convergence degradation is possible.

#### Modern Improvements

- **Expert-Choice routing**: experts "choose" tokens, which improves load balancing and convergence at comparable costs. <sup>[\[57\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-ExpertChoice-57)</sup>
- **Fine-grained MoE**: a larger number of smaller experts (as in DBRX) provides finer granularity of specialization. <sup>[\[55\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-DBRX-55)</sup>
- **Sparse Upcycling**: converting a dense model into an MoE from its checkpoint can significantly improve quality at a moderate cost. <sup>[\[58\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-SparseUpcycling-58)</sup>

#### When to Use MoE

- Large multi-domain assistants with a limited compute budget.
- Training on vast corpora where specialization offers an advantage.
- Scenarios with advanced distributed infrastructure (many GPUs/TPUs and fast networks).

**When dense models are better**: limited infrastructure (1–2 GPUs), strict requirements for predictable latency, and simplicity of deployment.

## Retrieval-Augmented Generation (RAG)

RAG is an architectural **system pattern** built around an LLM, rather than an internal architecture of the model itself. It combines an LLM (the generative component) with an external knowledge base (the retrieval component), which helps compensate for the limitations of the model's "parametric memory".

- **Principle of Operation:** Before generating a response, the LLM retrieves relevant documents from an external source (a wiki, a corporate knowledge base, the web) and relies on them to formulate the answer. <sup>[\[59\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-RAG-59)</sup>
- **Advantages:**
  - Reduced hallucinations and improved factual accuracy. <sup>[\[59\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-RAG-59)[\[60\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-60)</sup>
  - Up-to-date information without fully retraining the model. <sup>[\[59\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-RAG-59)</sup>
  - Citable and traceable responses.
- **Application:** The de facto standard for enterprise assistants and systems requiring verifiable facts and operation on private/specialized data. <sup>[\[59\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-RAG-59)</sup>

## Attention Mechanisms and Context Handling

Basic self-attention has quadratic complexity with respect to sequence length ($O(n^{2})$), leading to the development of optimizations.

- **Sparse Attention:** Restricts attention to local windows/patterns. Examples: **Longformer**<sup>[\[61\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-61)</sup>, **BigBird**<sup>[\[62\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-62)</sup>.
- **[FlashAttention](https://systems-analysis.info/eng/FlashAttention "FlashAttention"):** Reorders computations to account for the GPU memory hierarchy; it provides significant gains in speed and memory usage and has become the de facto standard for training LLMs with long contexts<sup>[\[63\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-63)[\[64\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-64)[\[65\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-65)</sup>.
- **MQA/GQA (decoding acceleration):** *Multi-Query Attention* (shared keys/values for all heads) reduces KV cache traffic<sup>[\[66\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-66)</sup>. *Grouped-Query Attention* balances quality and speed<sup>[\[67\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-67)</sup>.
- **Improved positional representations:**
  - **ALiBi (Attention with Linear Biases):** Linear biases added to attention scores improve generalization to longer lengths. <sup>[\[68\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-68)</sup>
  - **RoPE (Rotary Position Embeddings):** Relative positional information via rotation of Q/K; widely used in modern models (e.g., LLaMA). <sup>[\[69\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-69)[\[70\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-70)</sup>
  - **Context extension for RoPE models:** *Position Interpolation* <sup>[\[71\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-71)</sup>, *YaRN* <sup>[\[72\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-72)</sup>, and NTK-aware modifications allow for efficient context window extension without architectural changes.

<!-- -->

- **Other approaches for long sequences:**
  - **Transformer-XL:** Recurrent memory between segments to model long-range dependencies. <sup>[\[73\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-73)</sup>
  - **Reformer:** LSH-attention and reversible residual blocks to save memory. <sup>[\[74\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-74)</sup>
  - **Performer:** Linear approximation of softmax-attention (FAVOR+). <sup>[\[75\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-75)</sup>
  - **Linformer:** Low-rank approximation of the attention matrix. <sup>[\[76\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-76)</sup>

## Model Optimization and Training Infrastructure

Specialized techniques and frameworks are used for training and deploying LLMs.

- **Quantization:** Reducing the bit precision of weights decreases memory usage and accelerates inference. **QLoRA** enables efficient fine-tuning of 4-bit models (including 65B models) with quality close to full precision<sup>[\[77\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-77)</sup>.
- **Knowledge Distillation:** *Teacher→Student* training for compact models<sup>[\[78\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-78)</sup>; an example is **DistilBERT**<sup>[\[79\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-79)</sup>.
- **Distributed Training:**
  - **DeepSpeed** and **ZeRO** distribute parameters/gradients/optimizer states to train trillion-parameter models<sup>[\[80\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-80)</sup>.
  - **Megatron-LM** uses tensor and pipeline parallelism for very large transformers<sup>[\[81\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-81)</sup>.
- **Ecosystem and Tools:** **Hugging Face Transformers** and **Accelerate** provide standard model implementations and integration with DeepSpeed/FSDP for training and inference<sup>[\[82\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-82)[\[83\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-83)</sup>.

## Scaling Laws and Compute-Optimal Training

Empirical **scaling laws** demonstrate that cross-entropy loss decreases as a power law with increases in parameters, data, and computation. <sup>[\[84\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-84)</sup> The **Chinchilla** paper refined the **compute-optimal** regimes: for optimal efficiency, the model size and the number of training tokens should be scaled together (e.g., a 70B model trained on ~1.4T tokens outperforms larger, under-trained models). <sup>[\[85\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-85)</sup>

## State Space Models (SSM)

**State Space Models (SSM)** are an alternative architecture to Transformers for processing long sequences. They borrow ideas from control theory and digital signal processing and address the main problem of self-attention: the quadratic growth of computation with increasing text length.

### The Core Problem and Solution

**The Transformer Problem.** The main issue with traditional transformers is the quadratic complexity of attention: text that is 10 times longer requires about 100 times more computation.

**The SSM Approach.** Instead of "simultaneous attention to all words," the model processes the text sequentially and maintains a compact internal **memory state** that is updated at each step. As a result, time and memory consumption grow approximately linearly with text length. At the same time, training can be performed in parallel through a convolutional representation of the kernel (high throughput on long sequences). <sup>[\[86\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-S4-86)</sup>

### Principle of Operation

A discrete SSM is described by the state and output equations:

$x_{t} = Ax_{t - 1} + Bu_{t},\quad y_{t} = Cx_{t} + Du_{t}$

where $x_{t}$ is the memory state, $u_{t}$ is the input (token), and $y_{t}$ is the output. In deep SSMs, the matrices $A,B,C,D$ are parameterized to ensure stability and efficient computation on long sequences. The same layer can be viewed as:

- recurrent (scanning step-by-step) — for memory-efficient inference without a KV cache;
- convolutional — for parallel training with a pre-computed kernel. <sup>[\[86\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-S4-86)</sup>

### Main Architectures and Hybrids

- **S4 (Structured State Spaces).** The baseline SSM with a stable parameterization of the state matrix; demonstrates efficiency on very long sequences. <sup>[\[86\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-S4-86)</sup>
- **Mamba.** *Selective* SSMs: the memory update rules depend on the current input (the model decides what to "keep in memory" and what to "forget"). The implementation is optimized for the GPU memory hierarchy; according to the authors, it achieves a multiple-fold increase in inference throughput with linear complexity in sequence length. <sup>[\[87\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Mamba-87)</sup>
- **RetNet.** A *retention* mechanism with three modes: parallel training, recurrent inference, and chunk-wise recurrent inference. The goal is to combine fast training (like Transformers) with efficient streaming inference (O(1) memory per token). <sup>[\[88\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-RetNet-88)</sup>
- **Attention+SSM Hybrids.** An example is **[Jamba](https://systems-analysis.info/eng/Jamba_(language_model) "Jamba (language model)")** (alternating Transformer and Mamba layers, plus MoE): it reports support for contexts of around ~256K tokens with significantly lower memory requirements compared to pure transformer models of a similar class. <sup>[\[89\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Jamba-89)</sup>

#### Advantages

- Linear complexity and memory efficiency during inference. No global self-attention or KV cache; only a compact state is stored. <sup>[\[87\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Mamba-87)[\[88\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-RetNet-88)</sup>
- Parallel training on long sequences. The convolutional mode increases training throughput. <sup>[\[86\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-S4-86)</sup>
- Hardware efficiency. Implementations are optimized for the modern memory hierarchy (HBM/SRAM). <sup>[\[87\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Mamba-87)</sup>
- Long contexts and streaming. SSM+Attention hybrids are practical for hundreds of thousands of tokens with moderate resources. <sup>[\[89\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Jamba-89)</sup>

#### Limitations and Current Practice

- Ecosystem maturity. Tools and "recipes" for scaling (instruction tuning, RLHF/DPO) are not yet as developed as the Transformer stack. <sup>[\[87\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Mamba-87)</sup>
- Quality and stability. On some tasks, hybrids (Attention+SSM) show a more stable trade-off between quality, speed, and memory than "pure" SSMs. <sup>[\[89\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Jamba-89)</sup>

#### Comparison of Approaches (Generalized)

| Characteristic               | Transformers                   | SSM                       | Hybrids (Attention+SSM) |
|------------------------------|--------------------------------|---------------------------|-------------------------|
| Complexity by length         | Quadratic (self-attention)     | Linear (scan/convolution) | Close to linear         |
| Memory per token (inference) | KV cache grows with context    | O(1) state                | Moderate growth         |
| Long contexts                | Requires special optimizations | Natively supported        | Practical up to ~256K   |
| Ecosystem maturity           | High                           | Developing                | Developing              |

#### Practical Applications

- Analysis of very long documents (books, reports, scientific reviews).
- Stream processing and chat scenarios with long histories without increased memory costs.
- Environments with limited resources (mobile/edge devices).
- Time series and other sequential data.

**Representative models:** S4, Mamba, RetNet; Attention+SSM hybrids (Jamba). <sup>[\[86\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-S4-86)[\[87\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Mamba-87)[\[88\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-RetNet-88)[\[89\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Jamba-89)</sup>

## Evolution of Architectures

- 2017 — The paper "Attention Is All You Need" is published. It introduces the Transformer architecture: multi-head self-attention and positional encodings allow models to be trained without recurrence or convolutions; however, attention has quadratic complexity with respect to context length.<sup>[\[1\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-Vaswani2017-1)</sup>

<!-- -->

- 2018 — GPT-1 and BERT are introduced. GPT-1 uses a decoder-only stack with causal attention for generation and subsequent fine-tuning; BERT introduces a bidirectional encoder and MLM pre-training for text understanding tasks. <sup>[\[90\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-90)[\[91\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-91)</sup>

<!-- -->

- 2019 — Methods for handling long sequences are proposed, and the decoder-only approach is scaled up. Transformer-XL adds "memory" and relative positions to extend beyond a fixed window; GPT-2 demonstrates the growth of zero-shot capabilities with increased scale; BART shows the effectiveness of denoising pre-training for seq2seq. <sup>[\[92\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-92)[\[93\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-93)[\[94\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-94)</sup>

<!-- -->

- 2020 — The "text-to-text" format is unified, and methods for long documents are shown. T5 formulates a unified encoder-decoder approach for various tasks; Longformer and BigBird use sparse/structured attention for long texts; GPT-3 confirms the effectiveness of scaling dense decoder-only models. <sup>[\[95\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-95)[\[96\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-96)[\[97\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-97)[\[98\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-98)</sup>

<!-- -->

- 2021 — Positional representations are improved, and parameter sparsity (MoE) is demonstrated. RoPE and ALiBi improve generalization to longer lengths; Switch Transformer and GLaM activate only a subset of experts per token, increasing capacity without a proportional increase in inference cost. <sup>[\[99\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-99)[\[100\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-100)[\[101\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-101)[\[102\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-102)</sup>

<!-- -->

- 2022 — The compute-optimal regime is refined, and inference on long prompts is accelerated. Chinchilla shows the benefit of more training tokens with a moderate model size; PaLM with Multi-Query Attention reduces KV cache size; FlashAttention speeds up attention on GPUs. <sup>[\[103\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-103)[\[104\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-104)[\[105\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-105)[\[106\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-106)</sup>

<!-- -->

- 2023 — Context windows are extended without layer modifications, and server-side delivery is improved. The LLaMA series solidifies best practices (RMSNorm, SwiGLU, RoPE); Position Interpolation and YaRN extend context; vLLM/PagedAttention more efficiently manages the KV cache. <sup>[\[107\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-107)[\[108\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-108)[\[109\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-109)[\[110\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-110)[\[111\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-111)[\[112\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-112)</sup>

<!-- -->

- 2023 — GPT-4 and Gemini demonstrate multi-modal processing and generation within a single family of models. <sup>[\[113\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-113)[\[114\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-114)</sup>

<!-- -->

- 2023 — State Space Models (SSM) are proposed. Mamba and RetNet bring back sequential processing with a compact state instead of a KV cache, laying the groundwork for hybrid architectures. <sup>[\[115\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-115)[\[116\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-116)</sup>

<!-- -->

- 2024 — Open-source MoE models and Attention+SSM hybrids are published; attention is accelerated on new GPUs. Mixtral 8×7B/8×22B and DBRX confirm the practicality of MoE; Jamba combines Transformer and Mamba for very long contexts; FlashAttention-3 increases throughput. <sup>[\[117\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-117)[\[118\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-118)[\[119\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-119)[\[120\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-120)[\[121\]](https://systems-analysis.info/eng/Large_language_model_architectures#cite_note-121)</sup>

## External links

- <a href="https://jalammar.github.io/illustrated-transformer/" class="external free" rel="nofollow">https://jalammar.github.io/illustrated-transformer/</a> The Illustrated Transformer — a visual explanation
- <a href="https://en.wikipedia.org/wiki/Large_language_model" class="external text" rel="nofollow">Large language model — Wikipedia</a>

## See also

- [Theoretical foundations of large language models](https://systems-analysis.info/eng/Theoretical_foundations_of_large_language_models "Theoretical foundations of large language models")
- [BLOOM (language model)](https://systems-analysis.info/eng/BLOOM_(language_model) "BLOOM (language model)")
- [IBM Granite (language model)](https://systems-analysis.info/eng/IBM_Granite_(language_model) "IBM Granite (language model)")
- [Jais (language model)](https://systems-analysis.info/eng/Jais_(language_model) "Jais (language model)")
- [PanGu (Huawei)](https://systems-analysis.info/eng/PanGu_(Huawei) "PanGu (Huawei)")

## Literature

- Vaswani, A. et al. (2017). *Attention Is All You Need*. NIPS. <a href="https://arxiv.org/abs/1706.03762" class="external free" rel="nofollow">https://arxiv.org/abs/1706.03762</a>
- Devlin, J. et al. (2019). *BERT*. NAACL. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a>
- Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. NeurIPS. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a>
- Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer (T5)*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a>
- Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a>
- Touvron, H. et al. (2023). *LLaMA*. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a>
- Chowdhery, A. et al. (2022). *PaLM: Scaling Language Modeling with Pathways*. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a>
- Dao, T. et al. (2022–2024). *FlashAttention (1/2/3)*. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a> ; <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a> ; <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a>
- Shazeer, N. (2019). *MQA*. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a>
- Ainslie, J. et al. (2023). *GQA*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a>
- Kwon, W. et al. (2023). *PagedAttention / vLLM*. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a>
- Leviathan, Y. et al. (2023). *Speculative Decoding*. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a>
- Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). *Switch Transformers*. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a>
- Du, N. et al. (2022). *GLaM*. <a href="https://proceedings.mlr.press/v162/du22c/du22c.pdf" class="external free" rel="nofollow">https://proceedings.mlr.press/v162/du22c/du22c.pdf</a>
- Jiang, A.Q. et al. (2024). *Mixtral of Experts*. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a>
- Databricks (2024). *Introducing DBRX*. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a>
- NVIDIA (2024). *Applying Mixture of Experts in LLM Architectures*. <a href="https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/" class="external free" rel="nofollow">https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/</a>
- Zhou, Y. et al. (2022). *Expert Choice Routing*. <a href="https://arxiv.org/abs/2202.09368" class="external free" rel="nofollow">https://arxiv.org/abs/2202.09368</a>
- Komatsuzaki, A. et al. (2022). *Sparse Upcycling*. <a href="https://arxiv.org/abs/2212.05055" class="external free" rel="nofollow">https://arxiv.org/abs/2212.05055</a>
- Lewis, P. et al. (2020). *RAG*. <a href="https://arxiv.org/abs/2005.11401" class="external free" rel="nofollow">https://arxiv.org/abs/2005.11401</a>
- Beltagy, I. et al. (2020). *Longformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a>
- Zaheer, M. et al. (2020). *BigBird*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a>
- Press, O. et al. (2022). *ALiBi*. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a>
- Su, J. et al. (2021). *RoFormer (RoPE)*. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a>
- Chen, S. et al. (2023). *Position Interpolation*. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a>
- Peng, B. et al. (2023). *YaRN*. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a>
- Dettmers, T. et al. (2023). *QLoRA*. <a href="https://arxiv.org/abs/2305.14314" class="external free" rel="nofollow">https://arxiv.org/abs/2305.14314</a>
- Rajbhandari, S. et al. (2020). *ZeRO*. <a href="https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/" class="external free" rel="nofollow">https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/</a>
- Shoeybi, M. et al. (2019). *Megatron‑LM*. <a href="https://arxiv.org/abs/1909.08053" class="external free" rel="nofollow">https://arxiv.org/abs/1909.08053</a>
- Kaplan, J. et al. (2020). *Scaling Laws*. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a>
- Hoffmann, J. et al. (2022). *Chinchilla / Compute‑Optimal*. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a>
- Gemini Team (2023). *Gemini*. <a href="https://arxiv.org/abs/2312.11805" class="external free" rel="nofollow">https://arxiv.org/abs/2312.11805</a>
- Bai, Y. et al. (2022). *Constitutional AI*. <a href="https://arxiv.org/abs/2212.08073" class="external free" rel="nofollow">https://arxiv.org/abs/2212.08073</a>
- OpenAI (2023). *GPT‑4 Technical Report*. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a>
- OpenAI (2023). *DevDay: GPT‑4 Turbo 128k*. <a href="https://openai.com/index/new-models-and-developer-products-announced-at-devday/" class="external free" rel="nofollow">https://openai.com/index/new-models-and-developer-products-announced-at-devday/</a>
- Zhang, B.; Sennrich, R. (2019). *RMSNorm*. <a href="https://arxiv.org/abs/1910.07467" class="external free" rel="nofollow">https://arxiv.org/abs/1910.07467</a>
- Shazeer, N. (2020). *GLU Variants / SwiGLU*. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a>
- Gu, A.; Goel, K.; Ré, C. (2021). *S4: Structured State Spaces*. <a href="https://arxiv.org/abs/2111.00396" class="external free" rel="nofollow">https://arxiv.org/abs/2111.00396</a>
- Gu, A.; Dao, T. (2023/2024). *Mamba: Selective State Spaces*. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a>
- Sun, Y. et al. (2023). *RetNet*. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a>
- Lieber, O. et al. (2024). *Jamba: Hybrid Transformer‑Mamba*. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a>
- Dai, Z. et al. (2019). *Transformer‑XL*. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a>
- Kitaev, N.; Kaiser, L.; Levskaya, A. (2020). *Reformer*. <a href="https://arxiv.org/abs/2001.04451" class="external free" rel="nofollow">https://arxiv.org/abs/2001.04451</a>
- Choromanski, K. et al. (2021). *Performer*. <a href="https://arxiv.org/abs/2009.14794" class="external free" rel="nofollow">https://arxiv.org/abs/2009.14794</a>
- Wang, S. et al. (2020). *Linformer*. <a href="https://arxiv.org/abs/2006.04768" class="external free" rel="nofollow">https://arxiv.org/abs/2006.04768</a>

## References

1.  <span id="cite_note-Vaswani2017-1">↑ <sup>[1.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Vaswani2017_1-0)</sup> <sup>[1.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Vaswani2017_1-1)</sup> <sup>[1.2](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Vaswani2017_1-2)</sup> <sup>[1.3](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Vaswani2017_1-3)</sup> <sup>[1.4](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Vaswani2017_1-4)</sup> <sup>[1.5](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Vaswani2017_1-5)</sup> <sup>[1.6](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Vaswani2017_1-6)</sup> Vaswani, A. et al. (2017). *Attention Is All You Need*. <a href="https://arxiv.org/abs/1706.03762" class="external free" rel="nofollow">https://arxiv.org/abs/1706.03762</a></span>
2.  <span id="cite_note-2">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-2) Devlin, J. et al. (2019). *BERT*. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
3.  <span id="cite_note-3">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-3) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
4.  <span id="cite_note-4">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-4) Raffel, C. et al. (2020). *T5*. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
5.  <span id="cite_note-5">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-5) Devlin, J. et al. (2019). *BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding*. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
6.  <span id="cite_note-6">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-6) Liu, Y. et al. (2019). *RoBERTa: A Robustly Optimized BERT Pretraining Approach*. <a href="https://arxiv.org/abs/1907.11692" class="external free" rel="nofollow">https://arxiv.org/abs/1907.11692</a></span>
7.  <span id="cite_note-7">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-7) He, P. et al. (2021). *DeBERTa: Decoding‑enhanced BERT with Disentangled Attention*. <a href="https://arxiv.org/abs/2006.03654" class="external free" rel="nofollow">https://arxiv.org/abs/2006.03654</a></span>
8.  <span id="cite_note-8">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-8) Clark, K. et al. (2020). *ELECTRA: Pre‑training Text Encoders as Discriminators Rather Than Generators*. <a href="https://arxiv.org/abs/2003.10555" class="external free" rel="nofollow">https://arxiv.org/abs/2003.10555</a></span>
9.  <span id="cite_note-9">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-9) Zaheer, M. et al. (2020). *Big Bird: Transformers for Longer Sequences*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
10. <span id="cite_note-10">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-10) Beltagy, I. et al. (2020). *Longformer: The Long‑Document Transformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
11. <span id="cite_note-11">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-11) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need* (Multi‑Query Attention). <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
12. <span id="cite_note-12">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-12) Ainslie, J. et al. (2023). *GQA: Training Generalized Multi‑Query Transformer Models from Multi‑Head Checkpoints*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
13. <span id="cite_note-13">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-13) Leviathan, Y. et al. (2023). *Fast Inference from Transformers via Speculative Decoding*. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
14. <span id="cite_note-14">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-14) Kwon, W. et al. (2023). *Efficient Memory Management for LLM Serving with PagedAttention (vLLM)*. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
15. <span id="cite_note-15">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-15) vLLM Docs (2024–2025). *Continuous batching, Chunked prefill, Structured outputs*. <a href="https://docs.vllm.ai/" class="external free" rel="nofollow">https://docs.vllm.ai/</a></span>
16. <span id="cite_note-16">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-16) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
17. <span id="cite_note-17">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-17) Ouyang, L. et al. (2022). *InstructGPT (RLHF)*. <a href="https://arxiv.org/abs/2203.02155" class="external free" rel="nofollow">https://arxiv.org/abs/2203.02155</a></span>
18. <span id="cite_note-18">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-18) Rafailov, R. et al. (2023). *Direct Preference Optimization*. <a href="https://arxiv.org/abs/2305.18290" class="external free" rel="nofollow">https://arxiv.org/abs/2305.18290</a></span>
19. <span id="cite_note-19">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-19) Shazeer, 2019. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
20. <span id="cite_note-20">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-20) Ainslie, 2023. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
21. <span id="cite_note-21">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-21) Leviathan, 2023. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
22. <span id="cite_note-22">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-22) Kwon, 2023. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
23. <span id="cite_note-23">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-23) vLLM Docs. <a href="https://docs.vllm.ai/" class="external free" rel="nofollow">https://docs.vllm.ai/</a></span>
24. <span id="cite_note-24">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-24) OpenAI (2024). *Structured Outputs*. <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/" class="external free" rel="nofollow">https://openai.com/index/introducing-structured-outputs-in-the-api/</a></span>
25. <span id="cite_note-25">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-25) vLLM Docs — Structured outputs. <a href="https://docs.vllm.ai/en/v0.9.2/features/structured_outputs.html" class="external free" rel="nofollow">https://docs.vllm.ai/en/v0.9.2/features/structured_outputs.html</a></span>
26. <span id="cite_note-26">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-26) Kwon, 2023. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
27. <span id="cite_note-27">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-27) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
28. <span id="cite_note-28">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-28) Touvron, H. et al. (2023). *LLaMA: Open and Efficient Foundation Language Models*. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
29. <span id="cite_note-29">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-29) Achiam, J. et al. (2023). *GPT‑4 Technical Report*. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a></span>
30. <span id="cite_note-30">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-30) Meta AI (2024). *Introducing Meta Llama 3*. <a href="https://ai.meta.com/blog/meta-llama-3/" class="external free" rel="nofollow">https://ai.meta.com/blog/meta-llama-3/</a></span>
31. <span id="cite_note-31">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-31) Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer (T5)*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
32. <span id="cite_note-32">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-32) Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
33. <span id="cite_note-33">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-33) Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
34. <span id="cite_note-34">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-34) Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training for NLG, Translation, and Comprehension*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
35. <span id="cite_note-35">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-35) Chung, H. W. et al. (2022). *Scaling Instruction‑Finetuned Language Models (FLAN‑T5)*. <a href="https://arxiv.org/abs/2210.11416" class="external free" rel="nofollow">https://arxiv.org/abs/2210.11416</a></span>
36. <span id="cite_note-36">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-36) Shazeer, N. (2020). GLU Variants Improve Transformer. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a></span>
37. <span id="cite_note-37">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-37) Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
38. <span id="cite_note-38">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-38) Brown, T. et al. (2020). Language Models are Few‑Shot Learners. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
39. <span id="cite_note-39">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-39) Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
40. <span id="cite_note-40">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-40) Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a></span>
41. <span id="cite_note-41">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-41) Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a></span>
42. <span id="cite_note-42">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-42) Hoffmann, J. et al. (2022). Training Compute‑Optimal Large Language Models. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
43. <span id="cite_note-43">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-43) Chen, S. et al. (2023). Extending Context Window via Positional Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
44. <span id="cite_note-44">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-44) Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of LLMs. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
45. <span id="cite_note-45">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-45) Dao, T. et al. (2022–2024). FlashAttention (1/2/3). <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a> ; <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a> ; <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>
46. <span id="cite_note-46">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-46) Shazeer, N. (2019). Fast Transformer Decoding: One Write‑Head is All You Need. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
47. <span id="cite_note-47">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-47) Ainslie, J. et al. (2023). GQA. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
48. <span id="cite_note-48">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-48) Kwon, W. et al. (2023). Efficient Memory Management for LLM Serving with PagedAttention. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
49. <span id="cite_note-49">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-49) Leviathan, Y. et al. (2023). Fast Inference from Transformers via Speculative Decoding. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
50. <span id="cite_note-Switch-50">↑ <sup>[50.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Switch_50-0)</sup> <sup>[50.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Switch_50-1)</sup> <sup>[50.2](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Switch_50-2)</sup> Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). *Switch Transformers*. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a></span>
51. <span id="cite_note-GLAM-51">↑ <sup>[51.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-GLAM_51-0)</sup> <sup>[51.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-GLAM_51-1)</sup> <sup>[51.2](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-GLAM_51-2)</sup> Du, N. et al. (2021). *GLaM: Efficient Scaling of Language Models with Mixture‑of‑Experts*. <a href="https://arxiv.org/pdf/2112.06905.pdf" class="external free" rel="nofollow">https://arxiv.org/pdf/2112.06905.pdf</a></span>
52. <span id="cite_note-Mixtral8x7-52">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Mixtral8x7_52-0) Mistral AI (2023). *Mixtral of Experts*. <a href="https://mistral.ai/news/mixtral-of-experts/" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-of-experts/</a></span>
53. <span id="cite_note-Mixtral8x7_paper-53">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Mixtral8x7_paper_53-0) Jiang, A.Q. et al. (2024). *Mixtral of Experts*. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a></span>
54. <span id="cite_note-Mixtral8x22-54">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Mixtral8x22_54-0) Mistral AI (2024). *Mixtral 8x22B*. <a href="https://mistral.ai/news/mixtral-8x22b" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-8x22b</a></span>
55. <span id="cite_note-DBRX-55">↑ <sup>[55.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-DBRX_55-0)</sup> <sup>[55.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-DBRX_55-1)</sup> Databricks (2024). *Introducing DBRX*. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a></span>
56. <span id="cite_note-NVIDIA_MoE-56">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-NVIDIA_MoE_56-0) NVIDIA (2024). *Applying Mixture of Experts in LLM Architectures*. <a href="https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/" class="external free" rel="nofollow">https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/</a></span>
57. <span id="cite_note-ExpertChoice-57">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-ExpertChoice_57-0) Zhou, Y. et al. (2022). *Mixture‑of‑Experts with Expert Choice Routing*. <a href="https://arxiv.org/abs/2202.09368" class="external free" rel="nofollow">https://arxiv.org/abs/2202.09368</a></span>
58. <span id="cite_note-SparseUpcycling-58">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-SparseUpcycling_58-0) Komatsuzaki, A. et al. (2022). *Sparse Upcycling: Training Mixture‑of‑Experts from Dense Checkpoints*. <a href="https://arxiv.org/abs/2212.05055" class="external free" rel="nofollow">https://arxiv.org/abs/2212.05055</a></span>
59. <span id="cite_note-RAG-59">↑ <sup>[59.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-RAG_59-0)</sup> <sup>[59.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-RAG_59-1)</sup> <sup>[59.2](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-RAG_59-2)</sup> <sup>[59.3](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-RAG_59-3)</sup> Lewis, P. et al. (2020). *Retrieval‑Augmented Generation for Knowledge‑Intensive NLP Tasks*. <a href="https://arxiv.org/abs/2005.11401" class="external free" rel="nofollow">https://arxiv.org/abs/2005.11401</a></span>
60. <span id="cite_note-60">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-60) NVIDIA Blog (2025). *What is Retrieval‑Augmented Generation (RAG)*. <a href="https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/" class="external free" rel="nofollow">https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/</a></span>
61. <span id="cite_note-61">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-61) Beltagy, I. et al. (2020). *Longformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
62. <span id="cite_note-62">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-62) Zaheer, M. et al. (2020). *Big Bird*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
63. <span id="cite_note-63">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-63) Dao, T. et al. (2022). *FlashAttention*. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a></span>
64. <span id="cite_note-64">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-64) Dao, T. et al. (2023). *FlashAttention‑2*. <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a></span>
65. <span id="cite_note-65">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-65) Shah, M. et al. (2024). *FlashAttention‑3*. <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>
66. <span id="cite_note-66">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-66) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need*. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
67. <span id="cite_note-67">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-67) Ainslie, J. et al. (2023). *GQA*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
68. <span id="cite_note-68">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-68) Press, O. et al. (2022). ALiBi. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a></span>
69. <span id="cite_note-69">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-69) Su, J. et al. (2021). RoFormer: Rotary Position Embedding. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a></span>
70. <span id="cite_note-70">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-70) Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
71. <span id="cite_note-71">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-71) Chen, S. et al. (2023). Extending Context Window via Positional Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
72. <span id="cite_note-72">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-72) Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of LLMs. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
73. <span id="cite_note-73">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-73) Dai, Z. et al. (2019). *Transformer‑XL: Attentive Language Models Beyond a Fixed‑Length Context*. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a></span>
74. <span id="cite_note-74">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-74) Kitaev, N.; Kaiser, L.; Levskaya, A. (2020). *Reformer: The Efficient Transformer*. <a href="https://arxiv.org/abs/2001.04451" class="external free" rel="nofollow">https://arxiv.org/abs/2001.04451</a></span>
75. <span id="cite_note-75">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-75) Choromanski, K. et al. (2021). *Rethinking Attention with Performers*. <a href="https://arxiv.org/abs/2009.14794" class="external free" rel="nofollow">https://arxiv.org/abs/2009.14794</a></span>
76. <span id="cite_note-76">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-76) Wang, S. et al. (2020). *Linformer: Self‑Attention with Linear Complexity*. <a href="https://arxiv.org/abs/2006.04768" class="external free" rel="nofollow">https://arxiv.org/abs/2006.04768</a></span>
77. <span id="cite_note-77">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-77) Dettmers, T. et al. (2023). *QLoRA: Efficient Finetuning of Quantized LLMs*. <a href="https://arxiv.org/abs/2305.14314" class="external free" rel="nofollow">https://arxiv.org/abs/2305.14314</a></span>
78. <span id="cite_note-78">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-78) Hinton, G. et al. (2015). *Distilling the Knowledge in a Neural Network*. <a href="https://arxiv.org/abs/1503.02531" class="external free" rel="nofollow">https://arxiv.org/abs/1503.02531</a></span>
79. <span id="cite_note-79">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-79) Sanh, V. et al. (2019). *DistilBERT*. <a href="https://arxiv.org/abs/1910.01108" class="external free" rel="nofollow">https://arxiv.org/abs/1910.01108</a></span>
80. <span id="cite_note-80">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-80) Rajbhandari, S. et al. (2020). *ZeRO: Memory Optimizations Toward Training Trillion‑Parameter Models*. <a href="https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/" class="external free" rel="nofollow">https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/</a></span>
81. <span id="cite_note-81">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-81) Shoeybi, M. et al. (2019). *Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism*. <a href="https://arxiv.org/abs/1909.08053" class="external free" rel="nofollow">https://arxiv.org/abs/1909.08053</a></span>
82. <span id="cite_note-82">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-82) Hugging Face. *Transformers Documentation*. <a href="https://huggingface.co/docs/transformers" class="external free" rel="nofollow">https://huggingface.co/docs/transformers</a></span>
83. <span id="cite_note-83">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-83) Hugging Face. *Accelerate Documentation*. <a href="https://huggingface.co/docs/accelerate" class="external free" rel="nofollow">https://huggingface.co/docs/accelerate</a></span>
84. <span id="cite_note-84">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-84) Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a></span>
85. <span id="cite_note-85">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-85) Hoffmann, J. et al. (2022). Training Compute‑Optimal Large Language Models. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
86. <span id="cite_note-S4-86">↑ <sup>[86.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-S4_86-0)</sup> <sup>[86.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-S4_86-1)</sup> <sup>[86.2](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-S4_86-2)</sup> <sup>[86.3](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-S4_86-3)</sup> <sup>[86.4](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-S4_86-4)</sup> Gu, A.; Goel, K.; Ré, C. (2021). *Efficiently Modeling Long Sequences with Structured State Spaces (S4)*. <a href="https://arxiv.org/abs/2111.00396" class="external free" rel="nofollow">https://arxiv.org/abs/2111.00396</a></span>
87. <span id="cite_note-Mamba-87">↑ <sup>[87.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Mamba_87-0)</sup> <sup>[87.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Mamba_87-1)</sup> <sup>[87.2](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Mamba_87-2)</sup> <sup>[87.3](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Mamba_87-3)</sup> <sup>[87.4](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Mamba_87-4)</sup> Gu, A.; Dao, T. (2023/2024). *Mamba: Linear‑Time Sequence Modeling with Selective State Spaces*. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a></span>
88. <span id="cite_note-RetNet-88">↑ <sup>[88.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-RetNet_88-0)</sup> <sup>[88.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-RetNet_88-1)</sup> <sup>[88.2](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-RetNet_88-2)</sup> Sun, Y. et al. (2023). *Retentive Network: A Successor to Transformer for Large Language Models*. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a></span>
89. <span id="cite_note-Jamba-89">↑ <sup>[89.0](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Jamba_89-0)</sup> <sup>[89.1](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Jamba_89-1)</sup> <sup>[89.2](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Jamba_89-2)</sup> <sup>[89.3](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-Jamba_89-3)</sup> Lieber, O. et al. (2024). *Jamba: A Hybrid Transformer‑Mamba Language Model*. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a></span>
90. <span id="cite_note-90">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-90) Radford, A. et al. (2018). Improving Language Understanding by Generative Pre‑Training. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" class="external free" rel="nofollow">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a></span>
91. <span id="cite_note-91">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-91) Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
92. <span id="cite_note-92">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-92) Dai, Z. et al. (2019). Transformer‑XL. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a></span>
93. <span id="cite_note-93">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-93) Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners. <a href="https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" class="external free" rel="nofollow">https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf</a></span>
94. <span id="cite_note-94">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-94) Lewis, M. et al. (2019). BART. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
95. <span id="cite_note-95">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-95) Raffel, C. et al. (2020). T5. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
96. <span id="cite_note-96">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-96) Beltagy, I. et al. (2020). Longformer. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
97. <span id="cite_note-97">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-97) Zaheer, M. et al. (2020). BigBird. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
98. <span id="cite_note-98">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-98) Brown, T. et al. (2020). Language Models are Few‑Shot Learners. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
99. <span id="cite_note-99">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-99) Su, J. et al. (2021). RoPE. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a></span>
100. <span id="cite_note-100">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-100) Press, O. et al. (2021/2022). ALiBi. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a></span>
101. <span id="cite_note-101">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-101) Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). Switch Transformers. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a></span>
102. <span id="cite_note-102">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-102) Du, N. et al. (2021). GLaM. <a href="https://arxiv.org/pdf/2112.06905.pdf" class="external free" rel="nofollow">https://arxiv.org/pdf/2112.06905.pdf</a></span>
103. <span id="cite_note-103">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-103) Hoffmann, J. et al. (2022). Chinchilla. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
104. <span id="cite_note-104">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-104) Chowdhery, A. et al. (2022). PaLM. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a></span>
105. <span id="cite_note-105">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-105) Shazeer, N. (2019). Fast Transformer Decoding. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
106. <span id="cite_note-106">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-106) Dao, T. et al. (2022). FlashAttention. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a></span>
107. <span id="cite_note-107">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-107) Touvron, H. et al. (2023). LLaMA. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
108. <span id="cite_note-108">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-108) Zhang, B.; Sennrich, R. (2019). RMSNorm. <a href="https://arxiv.org/abs/1910.07467" class="external free" rel="nofollow">https://arxiv.org/abs/1910.07467</a></span>
109. <span id="cite_note-109">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-109) Shazeer, N. (2020). GLU Variants. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a></span>
110. <span id="cite_note-110">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-110) Chen, S. et al. (2023). Position Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
111. <span id="cite_note-111">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-111) Peng, B. et al. (2023). YaRN. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
112. <span id="cite_note-112">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-112) Kwon, W. et al. (2023). vLLM/PagedAttention. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
113. <span id="cite_note-113">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-113) OpenAI (2023). GPT‑4 Technical Report. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a></span>
114. <span id="cite_note-114">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-114) Gemini Team (2023). Gemini. <a href="https://arxiv.org/abs/2312.11805" class="external free" rel="nofollow">https://arxiv.org/abs/2312.11805</a></span>
115. <span id="cite_note-115">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-115) Gu, A.; Dao, T. (2023). Mamba. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a></span>
116. <span id="cite_note-116">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-116) Sun, Y. et al. (2023). RetNet. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a></span>
117. <span id="cite_note-117">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-117) Jiang, A.Q. et al. (2024). Mixtral of Experts. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a></span>
118. <span id="cite_note-118">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-118) Mistral AI (2024). Mixtral 8x22B. <a href="https://mistral.ai/news/mixtral-8x22b" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-8x22b</a></span>
119. <span id="cite_note-119">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-119) Databricks (2024). Introducing DBRX. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a></span>
120. <span id="cite_note-120">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-120) Lieber, O. et al. (2024). Jamba. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a></span>
121. <span id="cite_note-121">[↑](https://systems-analysis.info/eng/Large_language_model_architectures#cite_ref-121) Shah, M. et al. (2024). FlashAttention‑3. <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>