--- title: "Large language model architectures — LLM 架构" source: "https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84" wiki: "systems-analysis.info/int" article: "Large_language_model_architectures_—_LLM_架构" language: "zh" categories: - "Category:Chinese" - "Category:Large language models" - "Category:Machine learning" revision_id: 3783 wiki_created_at: 2026-09-06T23:25:58Z wiki_modified_at: 2026-09-06T23:25:58Z downloaded_at: 2026-09-07T22:58:56Z --- # Large language model architectures — LLM 架构 **大型语言模型 (LLM) 架构**是定义大型语言模型如何构建、训练和运行的基本原则和结构。能够理解和生成人类语言的现代 LLM 几乎完全基于**Transformer**架构^{[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Vaswani2017-1)}，但它们也包含了许多旨在提高效率、可扩展性和能力的改进和不同方法。 ## LLM 架构家族 (Transformer) 现代大型语言模型基于Transformer 架构^{[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Vaswani2017-1)}，但根据目标（理解文本、生成续写或将一种文本转换为另一种）以不同方式使用它。在实践中，在保留 Transformer 基本原则的基础上，主要分为三个家族^{[\[2\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-2)[\[3\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-3)[\[4\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-4)}。 ### Encoder-only - 仅编码器该模型仅使用编码器堆栈，并双向处理整个输入文本。预训练通常构建为掩码语言建模（masked-language modeling, MLM）：部分词元（token）被隐藏，模型学习根据上下文恢复它们。由于具有双向上下文，这类模型在理解和评分任务中表现出色：分类、实体识别、文档重排和抽取式问答（extractive QA）。它们不适用于“从零开始”的自回归生成。在实践中，仅编码器家族还采用了其他预训练目标：ELECTRA 中的*替换词元检测 (replaced token detection, RTD)*（判别器模型识别被替换的词元）和用于语义搜索/检索器的双编码器*对比学习*（在“查询-文档”对上使用 InfoNCE/softmax-loss，如 Dense Passage Retrieval）。在 RAG 中使用时，仅编码器模型可作为*双编码器*（bi-encoder，为快速 ANN 搜索分别编码查询和文档）或*交叉编码器*（cross-encoder，为精确重排联合编码查询对）。 **优势:** - 凭借双向上下文实现高质量文本理解：分类、命名实体识别 (NER)、事实提取、重排、抽取式问答。 - 并行处理和高吞吐量：单次前向传播，无自回归；便于批量处理大规模评分任务。 - 与搜索和 RAG 自然集成：作为双编码器，可实现快速语义搜索；作为交叉编码器，可实现精确重排。 - 高效适配：相对紧凑的版本（约 1-3 亿参数；BERT-base 约 1.1 亿）经过目标微调后能达到高质量。 - 延迟稳定，不依赖于生成答案的长度（无逐词解码）；非常适合对大型数据集进行离线评分。 - 可通过相对/旋转位置编码和/或局部稀疏注意力（如 Longformer/BigBird）扩展编码器的上下文窗口，这对于处理长文档很有用。 **劣势:** - 自身不具备生成能力：对话和详细回答需要解码器或外部生成模块。 - 在交互式场景中受限：无法保存状态进行逐步生成。 - 预训练目标与自由生成任务不匹配：与因果建模相比，MLM 与生成任务的契合度较差。 - 上下文窗口在历史上受限（基础绝对位置编码配置中通常为 512 个词元）；扩展需要特殊的位置/注意力方案和/或微调。 - 对于检索任务，需要对双编码器和/或交叉编码器进行单独的对比微调；否则，搜索/重排的质量通常低于专门训练的模型。 **代表模型:** BERT 及其衍生模型，以及 RoBERTa 和 DeBERTa（增强型仅编码器变体）；采用替代预训练目标的模型有 ELECTRA (RTD)。^{[\[5\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-5)[\[6\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-6)[\[7\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-7)[\[8\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-8)[\[9\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-9)[\[10\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-10)} ### Decoder-only - 仅解码器该架构仅使用带有因果（从左到右）注意力的解码器堆栈：模型根据给定的前缀预测下一个词元。这种训练模式——因果语言建模（causal language modeling, CLM）——使这些模型成为生成任务的自然选择：对话、详细回答、创意写作、代码生成。其代价是长提示（prompt）会增加延迟和 KV 缓存的体积。在实践中，针对仅解码器模型广泛应用了多种工程技术：通过 MQA 和 GQA 减少 KV 缓存，通过推测解码（speculative decoding）和服务器端优化（PagedAttention/vLLM、连续批处理 (continuous batching)、分块预填充 (chunked prefill)）来加速推理。^{[\[11\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-11)[\[12\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-12)[\[13\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-13)[\[14\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-14)[\[15\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-15)} **优势:** - 自然的文本生成 (CLM)：强大的零样本（zero-shot）和少样本（few-shot）能力；易于扩展。^{[\[16\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-16)} - 应用广泛：单个模型通过提示中的指令和示例解决多种任务；与 RAG 和工具调用 (tool use) 自然结合。 - 成熟的生态系统：指令微调和行为对齐（RLHF、DPO）的最佳实践；提供开源和商业实现。^{[\[17\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-17)[\[18\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-18)} - 丰富的推理优化技术栈：MQA/GQA 减少 KV 缓存体积并提高吞吐量；推测解码在不改变输出分布的情况下加速推理；PagedAttention/vLLM 结合连续批处理和分块预填充提高了端到端的 GPU 利用率。^{[\[19\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-19)[\[20\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-20)[\[21\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-21)[\[22\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-22)[\[23\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-23)} - 支持结构化生成，以获得严格格式的回答（JSON/SQL/DSL），简化了与信息系统和 API 的集成。^{[\[24\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-24)[\[25\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-25)} **劣势:** - 生成延迟较高：顺序输出；新词元的成本随已“读取”上下文（KV 缓存）的长度增加而增加。 - 在“长输入-短输出”场景（如摘要、翻译）中，与编码器-解码器模型相比效率较低，后者只需对输入编码一次。 - 受单向上下文限制：在理解任务中，有时不如具有双向表示的模型（仅编码器/编码器-解码器）。 - 在长提示和大批量处理时，KV 缓存的内存可能成为瓶颈。^{[\[26\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-26)} - 激活/KV 缓存的量化（INT8/FP8）可加速推理，但可能降低在长上下文/代码上的质量；需要仔细验证（尤其是在严格的 SLA 要求下）。 **代表模型:** GPT-3、GPT-4（架构和数据集细节未公开）、LLaMA 和 *Llama 3* (8B/70B, 2024)。^{[\[27\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-27)[\[28\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-28)[\[29\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-29)[\[30\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-30)} ### Encoder-decoder - 编码器-解码器该架构结合了两个组件。编码器以双向模式工作，而解码器以因果模式工作。编码器一次性分析输入并形成其表示；解码器通过交叉注意力（cross-attention）访问该表示来生成输出。这种分离的方法在需要将长输入文本转换为短输出的场景中特别有用，例如机器翻译、摘要和基于文档的问答。尽管该方法需要更多的总计算开销（两个堆栈和交叉注意力），但其优势在于基于对源文本的完整分析进行可控生成；同时，编码只执行一次，并在整个推理过程中重复使用。 **优势:** - 条件生成：解码器使用对输入表示的交叉注意力。^{[\[31\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-31)} - 在“长输入→短输出”场景中效率高：输入只编码一次。 - 便于“文本到文本”（text-to-text）格式和可控输出（任务前缀、特殊指令）。^{[\[32\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-32)} - 在长源文本下稳定高效：在解码阶段，只有输出的自注意力会增长，而交叉注意力重复使用来自编码器的固定键/值（每一步都不需要“重新读取”输入）。 **劣势:** - 两个堆栈增加了训练和应用中的内存和计算需求。 - 在超长序列上，总延迟与仅解码器模型相当；自回归仍然是瓶颈。 - 通用聊天模型少于仅解码器模型；更常被用作针对特定任务的高质量 seq2seq 引擎。 - 对于非常长的输入，每个解码器层中交叉注意力的键/值内存会增加（覆盖整个源文本），这需要仔细规划服务部署。 **代表模型:** T5（包括 T5 v1.1 和 *FLAN-T5* 中的指令微调实践）和 BART。^{[\[33\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-33)[\[34\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-34)[\[35\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-35)} ## 密集 (Dense) Transformer 这是最经典且最普遍的 LLM 架构：在处理每个词元时，几乎整个模型的参数集都参与计算。与稀疏方法（如 Mixture-of-Experts）不同，这里没有子网络的选择性激活——每个模块都为每个词元工作。^{[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Vaswani2017-1)} ### 工作原理与架构 **基本结构。**模型由 N 个相同类型的 Transformer 模块堆叠而成。每个模块包括： 1. **多头自注意力 (Multi-Head Self-Attention)。** 对每个词元计算三个向量：Q (query)、K (key)、V (value)；注意力计算公式为 $\operatorname{softmax}\!\left( \frac{QK^{\top} + M}{\sqrt{d_{k}}} \right) \cdot V$，其中 $M$ 是掩码（因果和/或填充掩码），用于排除无效位置。多个注意力“头”并行地关注上下文的不同方面（H 个头，通常 $d_{head} = \frac{d_{model}}{H}$）；其数量随模型规模增长。^{[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Vaswani2017-1)} 2. **前馈网络 (Feed-Forward Network, FFN)。** 两个线性层，中间有一个非线性激活函数（通常是 GELU/SiLU；在一些现代模型中为 SwiGLU）。中间维度通常约为 $\approx 4\, d_{model}$；使用 SwiGLU 时通常取 $\approx \frac{8}{3}\, d_{model}$ 以保持相似的参数数量。FFN 包含了模型参数的很大一部分。^{[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Vaswani2017-1)[\[36\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-36)} **附加组件。** 使用残差连接和层归一化；在现代 LLM 中，更常用的是 Pre-LN（在子模块前进行归一化）——这能提高在深层网络中的训练稳定性。除了经典的 LayerNorm，**RMSNorm** 也越来越广泛地被使用（它减少了计算开销，并且在大型模型中表现良好）；此外，在某些模型家族中还应用了注意力空间内的归一化（例如，在 softmax 之前对 Q/K 进行归一化）。位置表示可以是绝对的或相对的；对于长上下文，RoPE 已成为事实上的标准。 #### 模型示例与规模 - BERT-Large: 24 层，维度 1024，16 个注意力头，约 3.4 亿参数。^{[\[37\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-37)} - GPT-3 (175B): 96 层，维度 12288，96 个注意力头，约 1750 亿参数。^{[\[38\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-38)} - LLaMA-65B: 80 层，维度 8192，64 个注意力头，约 650 亿参数。^{[\[39\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-39)} - PaLM-540B: 118 层，维度约 18432，约 5400 亿参数。^{[\[40\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-40)} #### 优势 - 模块统一，训练模式经过充分研究，扩展行为可预测。 - 随着参数和数据的增长，质量呈幂律提升；计算最优（compute-optimal）模式要求模型大小和训练词元量同步增加。^{[\[41\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-41)[\[42\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-42)} - 相同的架构经过微调后，无需修改层级结构即可覆盖广泛的任务。 #### 劣势 - 完全自注意力的复杂度与序列长度呈二次方关系 ($O(n^{2})$)，这限制了上下文窗口的大小。^{[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Vaswani2017-1)} - 生成步骤中参数完全激活：在没有 MoE 的解码器中，每个词元的推理成本大致与参数数量成正比。 - 内存带宽成为瓶颈 (memory-bound)：从 HBM 加载权重的速度常常限制了推理速度。 #### 扩展与上下文的限制 - 参数所需的内存随模型大小线性增长；由于梯度和优化器状态，训练内存增加得更多。 - 基础配置在历史上被限制在 2000-4000 个词元。现代位置编码方案（RoPE）和扩展技术（Position Interpolation, YaRN 等）可以将窗口扩大一个数量级以上，但代价是额外的计算/内存开销。^{[\[43\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-43)[\[44\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-44)} ### 现代优化技术 - **FlashAttention.** 考虑到 GPU 内存层次结构的精确注意力机制；在处理长序列时，降低了内存消耗并加速了训练/推理。^{[\[45\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-45)} - **KV 缓存的缩减与管理。** Multi-Query Attention 和 Grouped-Query Attention 减少了缓存大小和内存流量；在服务器层面，PagedAttention (vLLM) 通过页式缓存管理提高了吞吐量。^{[\[46\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-46)[\[47\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-47)[\[48\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-48)} - **推测解码 (Speculative Decoding)。** 一个草稿（draft）模型提出续写，主模型快速验证；在不改变输出分布的情况下实现加速。^{[\[49\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-49)} ## Sparse Models and Mixture-of-Experts (MoE) - 稀疏模型与专家混合 (MoE) MoE 是一种在不按比例增加每词元计算量的情况下，提升模型容量的方法。它不在层中使用一个大型 FFN 模块，而是使用一组并行的“专家”（多个独立的 FFN），并通过一个可训练的路由器（gating network）为每个词元选择 top-k 个最相关的专家（通常 k=1-2；在某些模型中 k=4）。只有被选中的专家会被激活；它们的输出经过加权求和。这样，总参数可以达到数千亿甚至数万亿，但每一步只使用其中一小部分。^{[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Switch-50)[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-GLAM-51)} #### 模型示例与规模 - **Switch Transformer (Google)**: 高达约 1.6T 参数；top-1 路由（每个词元一个专家）。证明了 MoE 可以在相似的每词元成本下大幅增加模型容量。^{[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Switch-50)} - **GLaM (Google)**: 1.2T 参数，每层 64 个专家，top-2；每个词元激活约 96.6B 参数（约 8%）。^{[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-GLAM-51)} - **Mixtral 8x7B (Mistral AI)**: 总参数约 46.7B，每个词元激活约 12.9B，top-2。^{[\[52\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Mixtral8x7-52)[\[53\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Mixtral8x7_paper-53)} - **Mixtral 8x22B**: 总参数约 141B，每个词元激活约 39B，top-2。^{[\[54\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Mixtral8x22-54)} - **DBRX (Databricks)**: 总参数 132B，每个词元激活约 36B；16 个专家和 top-4 路由（细粒度 MoE）。^{[\[55\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-DBRX-55)} ##### 优势 - 计算成本由激活的专家数量 k 决定，而不是总参数数量：可以在与小得多的密集模型相当的成本下，训练和使用万亿级规模的模型。^{[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-GLAM-51)} - 专业化：专家自动“适应”不同的语言/领域/模式，从而提高在多领域任务中的质量。 - 灵活部署：可以只在内存中保留常用专家，并按需加载不常用的专家（需要相应的基础设施支持）。 ##### 局限性 - 负载均衡：若无正则化，路由器可能“固化”于部分专家（路由器塌陷）。需要辅助损失（负载均衡）和改进的路由方案。^{[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Switch-50)} - 分布式计算的复杂性：需要专家并行和 all-to-all 通信；通信开销和内存管理成为瓶颈。^{[\[56\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-NVIDIA_MoE-56)} - 训练稳定性：路由器的设置和容量限制至关重要，否则可能导致质量/收敛性下降。 #### 最新改进 - **Expert-Choice routing**: 专家“选择”词元，从而在相当的成本下改善负载均衡和收敛性。^{[\[57\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-ExpertChoice-57)} - **细粒度 MoE (Fine-grained MoE)**: 更多、更小的专家（如 DBRX）提供了更精细的专业化粒度。^{[\[55\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-DBRX-55)} - **稀疏升级 (Sparse Upcycling)**: 将密集模型从其检查点转换为 MoE 模型，可以在适度的成本下显著提升质量。^{[\[58\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-SparseUpcycling-58)} #### MoE 的适用性 - 在有限的计算预算下构建大型多领域助手。 - 在广泛的语料库上进行训练，专业化能带来优势。 - 拥有发达的分布式基础设施（大量 GPU/TPU 和高速网络）的场景。 **何时密集模型更优**: 基础设施有限（1-2 个 GPU）、对可预测延迟和部署简便性有严格要求的场景。 ## Retrieval-Augmented Generation (RAG) - 检索增强生成 (RAG) RAG 是一个围绕 LLM 的**系统架构模式**，而非模型本身的内部架构。它将 LLM（生成组件）与外部知识库（检索组件）相结合，以弥补模型“参数化记忆”的局限性。 - **工作原理:** 在生成之前，LLM 从外部来源（维基百科、企业知识库、网络）检索相关文档，并依据这些文档来构建答案。^{[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-RAG-59)} - **优势:** - 减少幻觉并提高事实准确性。^{[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-RAG-59)[\[60\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-60)} - 无需完全重新训练模型即可保持信息时效性。^{[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-RAG-59)} - 答案可引用、可追溯。 - **应用:** 对于需要可验证事实并处理私有/高度专业化数据的企业助手和系统，RAG 已成为事实上的标准。^{[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-RAG-59)} ## 注意力机制与上下文处理基础自注意力的复杂度与序列长度呈二次方关系 ($O(n^{2})$)，因此出现了一些优化方法。 - **稀疏注意力 (Sparse Attention):** 将注意力限制在局部窗口/模式。例如：**Longformer**^{[\[61\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-61)}、**BigBird**^{[\[62\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-62)}。 - **FlashAttention:** 重新组织计算顺序以适应 GPU 内存层次结构；在训练长上下文 LLM 时显著节省时间和内存，已成为事实上的标准^{[\[63\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-63)[\[64\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-64)[\[65\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-65)}。 - **MQA/GQA (加速解码):** *Multi-Query Attention*（所有头共享键/值）减少了 KV 缓存的流量^{[\[66\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-66)}。*Grouped-Query Attention* 平衡了质量和速度^{[\[67\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-67)}。 - **改进的位置表示:** - **ALiBi (Attention with Linear Biases):** 对注意力分数添加线性偏置，改善了对更长序列的泛化能力。^{[\[68\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-68)} - **RoPE (Rotary Position Embeddings):** 通过旋转 Q/K 向量引入相对位置信息；在现代模型（如 LLaMA）中广泛使用。^{[\[69\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-69)[\[70\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-70)} - **扩展 RoPE 模型的上下文:** *Position Interpolation*^{[\[71\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-71)}、*YaRN*^{[\[72\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-72)} 以及 NTK-aware 修改等技术，可以在不改变架构的情况下有效增加上下文窗口。 - **处理长序列的其他方法:** - **Transformer-XL:** 在段之间使用循环记忆来建模远距离依赖。^{[\[73\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-73)} - **Reformer:** 使用 LSH 注意力和可逆残差块来节省内存。^{[\[74\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-74)} - **Performer:** 对 softmax 注意力进行线性逼近 (FAVOR+)。^{[\[75\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-75)} - **Linformer:** 对注意力矩阵进行低秩逼近。^{[\[76\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-76)} ## 模型优化与训练基础设施训练和部署 LLM 使用了专门的技术和框架。 - **量化 (Quantization):** 降低权重精度可减少内存占用并加速推理。**QLoRA** 允许在接近全精度质量的情况下高效微调 4 位模型（包括 65B 模型）^{[\[77\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-77)}。 - **知识蒸馏 (Knowledge Distillation):** 用于紧凑模型的“教师→学生”式学习^{[\[78\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-78)}；例如 **DistilBERT**^{[\[79\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-79)}。 - **分布式训练:** - **DeepSpeed** 和 **ZeRO** — 分布参数/梯度/优化器状态，以训练万亿参数模型^{[\[80\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-80)}。 - **Megatron-LM** — 用于超大型 Transformer 的张量和流水线并行^{[\[81\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-81)}。 - **生态系统与工具:** **Hugging Face Transformers** 和 **Accelerate** 提供了模型的标准实现，并与 DeepSpeed/FSDP 集成以进行训练和推理^{[\[82\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-82)[\[83\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-83)}。 ## Scaling laws and compute-optimal training - 缩放定律与计算最优训练经验性的**缩放定律**表明，随着参数、数据和计算量的增长，交叉熵损失呈幂律下降。^{[\[84\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-84)} **Chinchilla** 的研究进一步明确了**计算最优**模式：为达到最佳效率，模型大小和训练词元数量应协同扩展（例如，一个在约 1.4T 词元上训练的 70B 模型，其表现优于训练不足的更大模型）。^{[\[85\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-85)} ## State Space Models (SSM) - 状态空间模型 (SSM) **状态空间模型 (State Space Models, SSM)** 是一种处理长序列的替代 Transformer 的架构。它借鉴了控制理论和数字信号处理的思想，解决了自注意力的主要问题：计算量随文本长度呈二次方增长。 ### 主要问题与解决方案 **Transformer 的问题。** 传统 Transformer 的主要问题是注意力的二次复杂度：文本长度增加 10 倍，计算量大约增加 100 倍。 **SSM 的方法。** 模型不是“同时关注所有词”，而是顺序地处理文本，并维护一个紧凑的内部**记忆状态**，该状态在每一步都会更新。因此，时间和内存消耗随文本长度大致呈线性增长。同时，训练可以并行执行——通过卷积核表示（在长序列上具有高吞吐量）。^{[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-S4-86)} ### 工作原理离散 SSM 由状态方程和输出方程描述： $x_{t} = Ax_{t - 1} + Bu_{t},\quad y_{t} = Cx_{t} + Du_{t}$ 其中 $x_{t}$ 是记忆状态，$u_{t}$ 是输入（词元），$y_{t}$ 是输出。在深度 SSM 中，矩阵 $A,B,C,D$ 的参数化方式旨在确保在长序列上的稳定性和高效计算。同一层可以被视为： - 循环的（按步扫描）——推理时内存高效，无需 KV 缓存； - 卷积的（并行计算）——使用预计算的核进行并行训练。^{[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-S4-86)} ### 主要架构与混合模型 - **S4 (Structured State Spaces)。** 具有稳定状态矩阵参数化的 SSM 基线；在非常长的序列上表现出高效率。^{[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-S4-86)} - **Mamba。** *选择性* SSM：记忆更新规则取决于当前输入（模型自行决定“记住”什么，“忘记”什么）。其实现针对 GPU 内存层次结构进行了优化；据作者称，在序列长度上实现了线性复杂度的同时，推理吞吐量提高了数倍。^{[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Mamba-87)} - **RetNet。** *retention* 机制具有三种模式：并行训练、循环推理和分块循环推理。目标是结合 Transformer 的快速训练和推理时的经济内存流（每个词元 O(1) 内存）。^{[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-RetNet-88)} - **Attention+SSM 混合模型。** 例如 **Jamba**（交替使用 Transformer 和 Mamba 层，并加入 MoE）：据报道，在内存需求远低于同类纯 Transformer 模型的情况下，支持约 256K 词元的上下文。^{[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Jamba-89)} #### 优势 - 推理时具有线性复杂度和内存经济性。没有全局自注意力和 KV 缓存；只存储一个紧凑的状态。^{[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Mamba-87)[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-RetNet-88)} - 可在长序列上并行训练。卷积模式提高了训练吞吐量。^{[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-S4-86)} - 硬件效率高。实现针对现代内存层次结构（HBM/SRAM）进行了优化。^{[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Mamba-87)} - 长上下文和流式处理。在适度资源下，SSM+Attention 混合模型对于数十万词元的处理非常实用。^{[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Jamba-89)} #### 局限性与当前实践 - 生态系统成熟度。在工具和扩展“配方”（指令、RLHF/DPO）方面，仍落后于 Transformer 技术栈。^{[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Mamba-87)} - 质量与稳定性。在某些任务上，混合模型 (Attention+SSM) 在“质量/速度/内存”的权衡上表现比“纯”SSM 更稳定。^{[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Jamba-89)} #### 方法比较（概述） | 特性 | Transformer | SSM | 混合模型 (Attention+SSM) | |-------------------|---------------------|------------------|--------------------------| | 序列长度复杂度 | 二次方 (自注意力) | 线性 (扫描/卷积) | 接近线性 | | 每词元内存 (推理) | KV 缓存随上下文增长 | O(1) 状态 | 适度增长 | | 长上下文 | 需要专门优化 | 自然支持 | 在 ~256K 词元下实用 | | 生态系统成熟度 | 高 | 发展中 | 发展中 | #### 实际应用 - 分析超长文档（书籍、报告、科学评论）。 - 流式处理和具有长历史的聊天场景，内存成本不增加。 - 资源受限的环境（移动/边缘设备）。 - 时间序列和其他序列数据。 **代表模型:** S4, Mamba, RetNet; 混合模型 Attention+SSM (Jamba)。^{[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-S4-86)[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Mamba-87)[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-RetNet-88)[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Jamba-89)} ## 架构演进 - 2017 — 论文《Attention Is All You Need》发表。提出了 Transformer 架构：多头自注意力和位置编码使得模型无需循环和卷积即可训练；但注意力的复杂度与上下文长度呈二次方关系。^{[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-Vaswani2017-1)} - 2018 — GPT-1 和 BERT 问世。GPT-1 仅使用解码器堆栈和因果注意力进行生成和后续微调；BERT 引入了双向编码器和 MLM 预训练，用于文本理解任务。^{[\[90\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-90)[\[91\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-91)} - 2019 — 提出了处理长序列的方法，并扩展了仅解码器模型。Transformer-XL 增加了“记忆”和相对位置编码，以超越固定窗口的限制；GPT-2 显示了随着规模扩大，零样本能力的增长；BART 证明了去噪预训练对 seq2seq 的有效性。^{[\[92\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-92)[\[93\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-93)[\[94\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-94)} - 2020 — 统一了“文本到文本”格式，并展示了处理长文档的方法。T5 为不同任务制定了统一的编码器-解码器方法；Longformer 和 BigBird 使用稀疏/结构化注意力处理长文本；GPT-3 证实了扩展密集仅解码器模型的有效性。^{[\[95\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-95)[\[96\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-96)[\[97\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-97)[\[98\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-98)} - 2021 — 改进了位置表示，并展示了参数稀疏化 (MoE)。RoPE 和 ALiBi 改善了对长序列的泛化能力；Switch Transformer 和 GLaM 每个词元只激活部分专家，在不大幅增加推理成本的情况下提高了模型容量。^{[\[99\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-99)[\[100\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-100)[\[101\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-101)[\[102\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-102)} - 2022 — 明确了计算最优模式，并加速了长提示的推理。Chinchilla 显示，在模型规模适中的情况下，增加训练词元数量更有益；PaLM 通过 Multi-Query Attention 减少了 KV 缓存的体积；FlashAttention 加速了 GPU 上的注意力计算。^{[\[103\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-103)[\[104\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-104)[\[105\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-105)[\[106\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-106)} - 2023 — 在不改变层结构的情况下扩展了上下文窗口，并改进了服务器端服务。LLaMA 系列巩固了最佳实践（RMSNorm、SwiGLU、RoPE）；Position Interpolation 和 YaRN 扩展了上下文；vLLM/PagedAttention 更有效地管理 KV 缓存。^{[\[107\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-107)[\[108\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-108)[\[109\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-109)[\[110\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-110)[\[111\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-111)[\[112\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-112)} - 2023 — GPT-4 和 Gemini 在同一模型家族内展示了多模态处理和生成能力。^{[\[113\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-113)[\[114\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-114)} - 2023 — 提出了状态空间模型 (SSM)。Mamba 和 RetNet 重新引入了使用紧凑状态而非 KV 缓存的顺序处理，为混合架构奠定了基础。^{[\[115\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-115)[\[116\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-116)} - 2024 — 开源 MoE 模型和 Attention+SSM 混合模型发布；在新 GPU 上加速了注意力计算。Mixtral 8x7B/8x22B 和 DBRX 证实了 MoE 的实用性；Jamba 结合 Transformer 和 Mamba 以处理超长上下文；FlashAttention-3 提高了吞吐量。^{[\[117\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-117)[\[118\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-118)[\[119\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-119)[\[120\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-120)[\[121\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_note-121)} ## 外部链接 - https://jalammar.github.io/illustrated-transformer/ The Illustrated Transformer — 图解说明 ## 参考文献 - Vaswani, A. et al. (2017). *Attention Is All You Need*. NIPS. https://arxiv.org/abs/1706.03762 - Devlin, J. et al. (2019). *BERT*. NAACL. https://arxiv.org/abs/1810.04805 - Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. NeurIPS. https://arxiv.org/abs/2005.14165 - Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer (T5)*. JMLR. https://jmlr.org/papers/volume21/20-074/20-074.pdf - Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training*. https://arxiv.org/abs/1910.13461 - Touvron, H. et al. (2023). *LLaMA*. https://arxiv.org/abs/2302.13971 - Chowdhery, A. et al. (2022). *PaLM: Scaling Language Modeling with Pathways*. https://arxiv.org/abs/2204.02311 - Dao, T. et al. (2022–2024). *FlashAttention (1/2/3)*. https://arxiv.org/abs/2205.14135 ; https://arxiv.org/abs/2307.08691 ; https://arxiv.org/abs/2407.08608 - Shazeer, N. (2019). *MQA*. https://arxiv.org/abs/1911.02150 - Ainslie, J. et al. (2023). *GQA*. https://arxiv.org/abs/2305.13245 - Kwon, W. et al. (2023). *PagedAttention / vLLM*. https://arxiv.org/abs/2309.06180 - Leviathan, Y. et al. (2023). *Speculative Decoding*. https://arxiv.org/abs/2211.17192 - Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). *Switch Transformers*. https://arxiv.org/abs/2101.03961 - Du, N. et al. (2022). *GLaM*. https://proceedings.mlr.press/v162/du22c/du22c.pdf - Jiang, A.Q. et al. (2024). *Mixtral of Experts*. https://arxiv.org/abs/2401.04088 - Databricks (2024). *Introducing DBRX*. https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm - NVIDIA (2024). *Applying Mixture of Experts in LLM Architectures*. https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/ - Zhou, Y. et al. (2022). *Expert Choice Routing*. https://arxiv.org/abs/2202.09368 - Komatsuzaki, A. et al. (2022). *Sparse Upcycling*. https://arxiv.org/abs/2212.05055 - Lewis, P. et al. (2020). *RAG*. https://arxiv.org/abs/2005.11401 - Beltagy, I. et al. (2020). *Longformer*. https://arxiv.org/abs/2004.05150 - Zaheer, M. et al. (2020). *BigBird*. https://arxiv.org/abs/2007.14062 - Press, O. et al. (2022). *ALiBi*. https://arxiv.org/abs/2108.12409 - Su, J. et al. (2021). *RoFormer (RoPE)*. https://arxiv.org/abs/2104.09864 - Chen, S. et al. (2023). *Position Interpolation*. https://arxiv.org/abs/2306.15595 - Peng, B. et al. (2023). *YaRN*. https://arxiv.org/abs/2309.00071 - Dettmers, T. et al. (2023). *QLoRA*. https://arxiv.org/abs/2305.14314 - Rajbhandari, S. et al. (2020). *ZeRO*. https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/ - Shoeybi, M. et al. (2019). *Megatron‑LM*. https://arxiv.org/abs/1909.08053 - Kaplan, J. et al. (2020). *Scaling Laws*. https://arxiv.org/abs/2001.08361 - Hoffmann, J. et al. (2022). *Chinchilla / Compute‑Optimal*. https://arxiv.org/abs/2203.15556 - Gemini Team (2023). *Gemini*. https://arxiv.org/abs/2312.11805 - Bai, Y. et al. (2022). *Constitutional AI*. https://arxiv.org/abs/2212.08073 - OpenAI (2023). *GPT‑4 Technical Report*. https://arxiv.org/abs/2303.08774 - OpenAI (2023). *DevDay: GPT‑4 Turbo 128k*. https://openai.com/index/new-models-and-developer-products-announced-at-devday/ - Zhang, B.; Sennrich, R. (2019). *RMSNorm*. https://arxiv.org/abs/1910.07467 - Shazeer, N. (2020). *GLU Variants / SwiGLU*. https://arxiv.org/abs/2002.05202 - Gu, A.; Goel, K.; Ré, C. (2021). *S4: Structured State Spaces*. https://arxiv.org/abs/2111.00396 - Gu, A.; Dao, T. (2023/2024). *Mamba: Selective State Spaces*. https://arxiv.org/abs/2312.00752 - Sun, Y. et al. (2023). *RetNet*. https://arxiv.org/abs/2307.08621 - Lieber, O. et al. (2024). *Jamba: Hybrid Transformer‑Mamba*. https://arxiv.org/abs/2403.19887 - Dai, Z. et al. (2019). *Transformer‑XL*. https://arxiv.org/abs/1901.02860 - Kitaev, N.; Kaiser, L.; Levskaya, A. (2020). *Reformer*. https://arxiv.org/abs/2001.04451 - Choromanski, K. et al. (2021). *Performer*. https://arxiv.org/abs/2009.14794 - Wang, S. et al. (2020). *Linformer*. https://arxiv.org/abs/2006.04768 ## 注释 1. ↑ ^{[1.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Vaswani2017_1-0)} ^{[1.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Vaswani2017_1-1)} ^{[1.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Vaswani2017_1-2)} ^{[1.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Vaswani2017_1-3)} ^{[1.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Vaswani2017_1-4)} ^{[1.5](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Vaswani2017_1-5)} ^{[1.6](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Vaswani2017_1-6)} Vaswani, A. et al. (2017). *Attention Is All You Need*. https://arxiv.org/abs/1706.03762 2. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-2) Devlin, J. et al. (2019). *BERT*. https://arxiv.org/abs/1810.04805 3. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-3) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. https://arxiv.org/abs/2005.14165 4. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-4) Raffel, C. et al. (2020). *T5*. https://jmlr.org/papers/volume21/20-074/20-074.pdf 5. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-5) Devlin, J. et al. (2019). *BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding*. https://arxiv.org/abs/1810.04805 6. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-6) Liu, Y. et al. (2019). *RoBERTa: A Robustly Optimized BERT Pretraining Approach*. https://arxiv.org/abs/1907.11692 7. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-7) He, P. et al. (2021). *DeBERTa: Decoding‑enhanced BERT with Disentangled Attention*. https://arxiv.org/abs/2006.03654 8. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-8) Clark, K. et al. (2020). *ELECTRA: Pre‑training Text Encoders as Discriminators Rather Than Generators*. https://arxiv.org/abs/2003.10555 9. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-9) Zaheer, M. et al. (2020). *Big Bird: Transformers for Longer Sequences*. https://arxiv.org/abs/2007.14062 10. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-10) Beltagy, I. et al. (2020). *Longformer: The Long‑Document Transformer*. https://arxiv.org/abs/2004.05150 11. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-11) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need* (Multi‑Query Attention). https://arxiv.org/abs/1911.02150 12. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-12) Ainslie, J. et al. (2023). *GQA: Training Generalized Multi‑Query Transformer Models from Multi‑Head Checkpoints*. https://arxiv.org/abs/2305.13245 13. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-13) Leviathan, Y. et al. (2023). *Fast Inference from Transformers via Speculative Decoding*. https://arxiv.org/abs/2211.17192 14. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-14) Kwon, W. et al. (2023). *Efficient Memory Management for LLM Serving with PagedAttention (vLLM)*. https://arxiv.org/abs/2309.06180 15. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-15) vLLM Docs (2024–2025). *Continuous batching, Chunked prefill, Structured outputs*. https://docs.vllm.ai/ 16. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-16) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. https://arxiv.org/abs/2005.14165 17. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-17) Ouyang, L. et al. (2022). *InstructGPT (RLHF)*. https://arxiv.org/abs/2203.02155 18. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-18) Rafailov, R. et al. (2023). *Direct Preference Optimization*. https://arxiv.org/abs/2305.18290 19. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-19) Shazeer, 2019. https://arxiv.org/abs/1911.02150 20. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-20) Ainslie, 2023. https://arxiv.org/abs/2305.13245 21. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-21) Leviathan, 2023. https://arxiv.org/abs/2211.17192 22. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-22) Kwon, 2023. https://arxiv.org/abs/2309.06180 23. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-23) vLLM Docs. https://docs.vllm.ai/ 24. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-24) OpenAI (2024). *Structured Outputs*. https://openai.com/index/introducing-structured-outputs-in-the-api/ 25. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-25) vLLM Docs — Structured outputs. https://docs.vllm.ai/en/v0.9.2/features/structured_outputs.html 26. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-26) Kwon, 2023. https://arxiv.org/abs/2309.06180 27. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-27) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. https://arxiv.org/abs/2005.14165 28. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-28) Touvron, H. et al. (2023). *LLaMA: Open and Efficient Foundation Language Models*. https://arxiv.org/abs/2302.13971 29. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-29) Achiam, J. et al. (2023). *GPT‑4 Technical Report*. https://arxiv.org/abs/2303.08774 30. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-30) Meta AI (2024). *Introducing Meta Llama 3*. https://ai.meta.com/blog/meta-llama-3/ 31. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-31) Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer (T5)*. JMLR. https://jmlr.org/papers/volume21/20-074/20-074.pdf 32. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-32) Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training*. https://arxiv.org/abs/1910.13461 33. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-33) Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer*. JMLR. https://jmlr.org/papers/volume21/20-074/20-074.pdf 34. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-34) Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training for NLG, Translation, and Comprehension*. https://arxiv.org/abs/1910.13461 35. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-35) Chung, H. W. et al. (2022). *Scaling Instruction‑Finetuned Language Models (FLAN‑T5)*. https://arxiv.org/abs/2210.11416 36. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-36) Shazeer, N. (2020). GLU Variants Improve Transformer. https://arxiv.org/abs/2002.05202 37. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-37) Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805 38. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-38) Brown, T. et al. (2020). Language Models are Few‑Shot Learners. https://arxiv.org/abs/2005.14165 39. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-39) Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. https://arxiv.org/abs/2302.13971 40. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-40) Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. https://arxiv.org/abs/2204.02311 41. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-41) Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361 42. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-42) Hoffmann, J. et al. (2022). Training Compute‑Optimal Large Language Models. https://arxiv.org/abs/2203.15556 43. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-43) Chen, S. et al. (2023). Extending Context Window via Positional Interpolation. https://arxiv.org/abs/2306.15595 44. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-44) Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of LLMs. https://arxiv.org/abs/2309.00071 45. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-45) Dao, T. et al. (2022–2024). FlashAttention (1/2/3). https://arxiv.org/abs/2205.14135 ; https://arxiv.org/abs/2307.08691 ; https://arxiv.org/abs/2407.08608 46. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-46) Shazeer, N. (2019). Fast Transformer Decoding: One Write‑Head is All You Need. https://arxiv.org/abs/1911.02150 47. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-47) Ainslie, J. et al. (2023). GQA. https://arxiv.org/abs/2305.13245 48. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-48) Kwon, W. et al. (2023). Efficient Memory Management for LLM Serving with PagedAttention. https://arxiv.org/abs/2309.06180 49. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-49) Leviathan, Y. et al. (2023). Fast Inference from Transformers via Speculative Decoding. https://arxiv.org/abs/2211.17192 50. ↑ ^{[50.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Switch_50-0)} ^{[50.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Switch_50-1)} ^{[50.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Switch_50-2)} Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). *Switch Transformers*. https://arxiv.org/abs/2101.03961 51. ↑ ^{[51.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-GLAM_51-0)} ^{[51.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-GLAM_51-1)} ^{[51.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-GLAM_51-2)} Du, N. et al. (2021). *GLaM: Efficient Scaling of Language Models with Mixture‑of‑Experts*. https://arxiv.org/pdf/2112.06905.pdf 52. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Mixtral8x7_52-0) Mistral AI (2023). *Mixtral of Experts*. https://mistral.ai/news/mixtral-of-experts/ 53. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Mixtral8x7_paper_53-0) Jiang, A.Q. et al. (2024). *Mixtral of Experts*. https://arxiv.org/abs/2401.04088 54. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Mixtral8x22_54-0) Mistral AI (2024). *Mixtral 8x22B*. https://mistral.ai/news/mixtral-8x22b 55. ↑ ^{[55.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-DBRX_55-0)} ^{[55.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-DBRX_55-1)} Databricks (2024). *Introducing DBRX*. https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm 56. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-NVIDIA_MoE_56-0) NVIDIA (2024). *Applying Mixture of Experts in LLM Architectures*. https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/ 57. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-ExpertChoice_57-0) Zhou, Y. et al. (2022). *Mixture‑of‑Experts with Expert Choice Routing*. https://arxiv.org/abs/2202.09368 58. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-SparseUpcycling_58-0) Komatsuzaki, A. et al. (2022). *Sparse Upcycling: Training Mixture‑of‑Experts from Dense Checkpoints*. https://arxiv.org/abs/2212.05055 59. ↑ ^{[59.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-RAG_59-0)} ^{[59.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-RAG_59-1)} ^{[59.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-RAG_59-2)} ^{[59.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-RAG_59-3)} Lewis, P. et al. (2020). *Retrieval‑Augmented Generation for Knowledge‑Intensive NLP Tasks*. https://arxiv.org/abs/2005.11401 60. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-60) NVIDIA Blog (2025). *What is Retrieval‑Augmented Generation (RAG)*. https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/ 61. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-61) Beltagy, I. et al. (2020). *Longformer*. https://arxiv.org/abs/2004.05150 62. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-62) Zaheer, M. et al. (2020). *Big Bird*. https://arxiv.org/abs/2007.14062 63. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-63) Dao, T. et al. (2022). *FlashAttention*. https://arxiv.org/abs/2205.14135 64. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-64) Dao, T. et al. (2023). *FlashAttention‑2*. https://arxiv.org/abs/2307.08691 65. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-65) Shah, M. et al. (2024). *FlashAttention‑3*. https://arxiv.org/abs/2407.08608 66. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-66) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need*. https://arxiv.org/abs/1911.02150 67. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-67) Ainslie, J. et al. (2023). *GQA*. https://arxiv.org/abs/2305.13245 68. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-68) Press, O. et al. (2022). ALiBi. https://arxiv.org/abs/2108.12409 69. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-69) Su, J. et al. (2021). RoFormer: Rotary Position Embedding. https://arxiv.org/abs/2104.09864 70. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-70) Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. https://arxiv.org/abs/2302.13971 71. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-71) Chen, S. et al. (2023). Extending Context Window via Positional Interpolation. https://arxiv.org/abs/2306.15595 72. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-72) Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of LLMs. https://arxiv.org/abs/2309.00071 73. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-73) Dai, Z. et al. (2019). *Transformer‑XL: Attentive Language Models Beyond a Fixed‑Length Context*. https://arxiv.org/abs/1901.02860 74. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-74) Kitaev, N.; Kaiser, L.; Levskaya, A. (2020). *Reformer: The Efficient Transformer*. https://arxiv.org/abs/2001.04451 75. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-75) Choromanski, K. et al. (2021). *Rethinking Attention with Performers*. https://arxiv.org/abs/2009.14794 76. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-76) Wang, S. et al. (2020). *Linformer: Self‑Attention with Linear Complexity*. https://arxiv.org/abs/2006.04768 77. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-77) Dettmers, T. et al. (2023). *QLoRA: Efficient Finetuning of Quantized LLMs*. https://arxiv.org/abs/2305.14314 78. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-78) Hinton, G. et al. (2015). *Distilling the Knowledge in a Neural Network*. https://arxiv.org/abs/1503.02531 79. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-79) Sanh, V. et al. (2019). *DistilBERT*. https://arxiv.org/abs/1910.01108 80. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-80) Rajbhandari, S. et al. (2020). *ZeRO: Memory Optimizations Toward Training Trillion‑Parameter Models*. https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/ 81. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-81) Shoeybi, M. et al. (2019). *Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism*. https://arxiv.org/abs/1909.08053 82. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-82) Hugging Face. *Transformers Documentation*. https://huggingface.co/docs/transformers 83. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-83) Hugging Face. *Accelerate Documentation*. https://huggingface.co/docs/accelerate 84. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-84) Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361 85. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-85) Hoffmann, J. et al. (2022). Training Compute‑Optimal Large Language Models. https://arxiv.org/abs/2203.15556 86. ↑ ^{[86.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-S4_86-0)} ^{[86.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-S4_86-1)} ^{[86.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-S4_86-2)} ^{[86.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-S4_86-3)} ^{[86.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-S4_86-4)} Gu, A.; Goel, K.; Ré, C. (2021). *Efficiently Modeling Long Sequences with Structured State Spaces (S4)*. https://arxiv.org/abs/2111.00396 87. ↑ ^{[87.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Mamba_87-0)} ^{[87.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Mamba_87-1)} ^{[87.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Mamba_87-2)} ^{[87.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Mamba_87-3)} ^{[87.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Mamba_87-4)} Gu, A.; Dao, T. (2023/2024). *Mamba: Linear‑Time Sequence Modeling with Selective State Spaces*. https://arxiv.org/abs/2312.00752 88. ↑ ^{[88.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-RetNet_88-0)} ^{[88.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-RetNet_88-1)} ^{[88.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-RetNet_88-2)} Sun, Y. et al. (2023). *Retentive Network: A Successor to Transformer for Large Language Models*. https://arxiv.org/abs/2307.08621 89. ↑ ^{[89.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Jamba_89-0)} ^{[89.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Jamba_89-1)} ^{[89.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Jamba_89-2)} ^{[89.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-Jamba_89-3)} Lieber, O. et al. (2024). *Jamba: A Hybrid Transformer‑Mamba Language Model*. https://arxiv.org/abs/2403.19887 90. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-90) Radford, A. et al. (2018). Improving Language Understanding by Generative Pre‑Training. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf 91. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-91) Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805 92. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-92) Dai, Z. et al. (2019). Transformer‑XL. https://arxiv.org/abs/1901.02860 93. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-93) Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf 94. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-94) Lewis, M. et al. (2019). BART. https://arxiv.org/abs/1910.13461 95. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-95) Raffel, C. et al. (2020). T5. https://jmlr.org/papers/volume21/20-074/20-074.pdf 96. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-96) Beltagy, I. et al. (2020). Longformer. https://arxiv.org/abs/2004.05150 97. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-97) Zaheer, M. et al. (2020). BigBird. https://arxiv.org/abs/2007.14062 98. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-98) Brown, T. et al. (2020). Language Models are Few‑Shot Learners. https://arxiv.org/abs/2005.14165 99. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-99) Su, J. et al. (2021). RoPE. https://arxiv.org/abs/2104.09864 100. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-100) Press, O. et al. (2021/2022). ALiBi. https://arxiv.org/abs/2108.12409 101. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-101) Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). Switch Transformers. https://arxiv.org/abs/2101.03961 102. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-102) Du, N. et al. (2021). GLaM. https://arxiv.org/pdf/2112.06905.pdf 103. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-103) Hoffmann, J. et al. (2022). Chinchilla. https://arxiv.org/abs/2203.15556 104. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-104) Chowdhery, A. et al. (2022). PaLM. https://arxiv.org/abs/2204.02311 105. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-105) Shazeer, N. (2019). Fast Transformer Decoding. https://arxiv.org/abs/1911.02150 106. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-106) Dao, T. et al. (2022). FlashAttention. https://arxiv.org/abs/2205.14135 107. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-107) Touvron, H. et al. (2023). LLaMA. https://arxiv.org/abs/2302.13971 108. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-108) Zhang, B.; Sennrich, R. (2019). RMSNorm. https://arxiv.org/abs/1910.07467 109. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-109) Shazeer, N. (2020). GLU Variants. https://arxiv.org/abs/2002.05202 110. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-110) Chen, S. et al. (2023). Position Interpolation. https://arxiv.org/abs/2306.15595 111. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-111) Peng, B. et al. (2023). YaRN. https://arxiv.org/abs/2309.00071 112. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-112) Kwon, W. et al. (2023). vLLM/PagedAttention. https://arxiv.org/abs/2309.06180 113. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-113) OpenAI (2023). GPT‑4 Technical Report. https://arxiv.org/abs/2303.08774 114. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-114) Gemini Team (2023). Gemini. https://arxiv.org/abs/2312.11805 115. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-115) Gu, A.; Dao, T. (2023). Mamba. https://arxiv.org/abs/2312.00752 116. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-116) Sun, Y. et al. (2023). RetNet. https://arxiv.org/abs/2307.08621 117. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-117) Jiang, A.Q. et al. (2024). Mixtral of Experts. https://arxiv.org/abs/2401.04088 118. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-118) Mistral AI (2024). Mixtral 8x22B. https://mistral.ai/news/mixtral-8x22b 119. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-119) Databricks (2024). Introducing DBRX. https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm 120. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-120) Lieber, O. et al. (2024). Jamba. https://arxiv.org/abs/2403.19887 121. [↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_LLM_%E6%9E%B6%E6%9E%84#cite_ref-121) Shah, M. et al. (2024). FlashAttention‑3. https://arxiv.org/abs/2407.08608