--- title: "Transformer architecture — Transformer 架构" source: "https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84" wiki: "systems-analysis.info/int" article: "Transformer_architecture_—_Transformer_架构" language: "zh" categories: - "Category:Chinese" - "Category:Large language models" - "Category:LLM core concepts" - "Category:Machine learning" - "Category:Systems analysis terminology" revision_id: 8254 wiki_created_at: 2026-09-07T01:14:36Z wiki_modified_at: 2026-09-07T01:14:36Z downloaded_at: 2026-09-07T23:24:21Z --- # Transformer architecture — Transformer 架构 **Transformer 架构**是一种神经网络架构，由谷歌研究人员在 2017 年的论文《Attention Is All You Need》中提出^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}。它彻底改变了自然语言处理 (NLP) 领域，并成为大多数现代大型语言模型 (LLM) 的基础，例如 BERT、GPT 和 Gemini。Transformer 的关键创新是**自注意力 (self‑attention)** 机制，该机制允许模型权衡输入数据不同部分的重要性，并并行处理序列，从而摒弃了 RNN 和 LSTM 所特有的循环结构。 ## 历史背景与前提在 2017 年之前，处理文本等序列数据的主流架构是循环神经网络 (RNN) 及其改进版本——长短期记忆网络 (LSTM)。 ### Transformer 解决的 RNN/LSTM 问题 - **顺序处理的局限性：** RNN 和 LSTM 逐个 token 处理数据，这排除了序列内的并行计算，减慢了在大量数据上的训练速度。 - **梯度消失与梯度爆炸问题：** 在长序列中，通过多个时间步反向传播的梯度可能会衰减或激增，从而难以学习长期依赖关系。 - **长期依赖：** 序列开头的信息可能会在传播到末尾时丢失。 Transformer 的方法是**完全摒弃循环结构**，转而采用注意力机制。它确保了任意位置之间的**依赖路径长度恒为常量** ($O(1)$)，这使得对远距离依赖的建模更加容易，尽管自注意力的基础实现具有关于序列长度的**二次方计算复杂度** ($O(n^{2})$)^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)[\[2\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-tay2020-2)}。梯度消失/爆炸问题并非“消失”，而是通过残差连接、LayerNorm 和训练策略得以**缓解**；在现代实现中，通常采用**Pre‑LayerNorm (Pre‑LN)** 方案，因为它在训练时更稳定^{[\[3\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-xiong2020-3)}。 ## 架构与关键组件原始的 Transformer 架构由两个主要部分组成：**编码器 (encoder)** 和 **解码器 (decoder)**。两个组件都是由 N 个相同层堆叠而成（在原始论文中 $N = 6$）^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}。 - **编码器层结构：** (1) 多头自注意力 (MHA)，(2) 按位置前馈网络 (FFN)；每个子层都包裹着一个残差连接和层归一化 (LayerNorm)^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}。 - **解码器层结构：** (1) **掩码**自注意力（因果掩码禁止访问未来的位置），(2) 对编码器输出的**交叉注意力 (cross‑attention)**，(3) FFN——同样带有残差连接和层归一化^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}。 ### Attention mechanism and Self-Attention - 注意力机制与自注意力注意力机制计算值向量 (Value) 的加权和，其中权重由键 (Key) 与查询 (Query) 的兼容性程度决定。Transformer 使用的是**缩放点积注意力 (Scaled Dot-Product Attention)**： ${Attention}(Q,K,V) = {softmax}\!\left( \frac{QK^{\top}}{\sqrt{d_{k}}} \right)V$ 其中 $d_{k}$ 是键/查询的维度；除以 $\sqrt{d_{k}}$ 是为了防止 softmax 函数饱和^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}。当 $Q$、$K$ 和 $V$ 由同一序列生成时，该机制称为**自注意力 (self‑attention)**。 ### Multi-Head Attention - 多头注意力该机制不使用单一一组矩阵 $(W_{Q},W_{K},W_{V})$，而是采用 $h$ 个并行的“头”，每个头将 $Q,K,V$ 投影到较低维度的子空间中，独立计算注意力，然后将结果拼接并再次投影^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}： ${MultiHead}(Q,K,V) = {Concat}(\text{head}_{1},\ldots,\text{head}_{h})W^{O},\quad\text{其中~}\text{head}_{i} = {Attention}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}).$ **用于加速推理的变体：** - **MQA (Multi‑Query Attention):** 所有头共享同一对键/值 → 在解码过程中显著减少 KV 缓存的大小和带宽需求^{[\[4\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-shazeer2019-4)}。 - **GQA (Grouped‑Query Attention):** MHA 和 MQA 之间的折衷方案——几组头共享 K/V；在接近 MQA 的速度下，其质量接近 MHA^{[\[5\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-ainslie2023-5)}。 ### Positional Encoding - 位置编码由于自注意力机制对 token 的顺序不敏感，因此需要在输入嵌入中加入**位置编码**。 - **原始的正弦位置编码** (PE)，源自^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}： $\text{PE}(\text{pos},2i) = \sin\!\left( \text{pos}/10000^{2i/d_{\text{model}}} \right),\quad\text{PE}(\text{pos},2i + 1) = \cos\!\left( \text{pos}/10000^{2i/d_{\text{model}}} \right).$ - **现代的相对/旋转式变体：** - **RoPE (Rotary Position Embeddings)** 通过旋转 $Q$/$K$ 向量来编码相对位置偏移；已在多种现代 LLM 中应用^{[\[6\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-su2021-6)}。 - **ALiBi** 在注意力分数中引入线性偏置惩罚，从而改善了模型对超过训练长度的序列的外推能力^{[\[7\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-press2021-7)}。 ### Position-wise FFN, Residuals, and Normalization - 按位置前馈网络、残差连接与归一化除了注意力子层，编码器和解码器的每一层还包含一个**按位置前馈网络 (FFN)**： ${FFN}(x) = \max(0,xW_{1} + b_{1})W_{2} + b_{2}.$ 每个子层的周围都使用了**残差连接 (residual connections)**和**层归一化 (Layer Normalization)**：${LayerNorm}(x + {Sublayer}(x))$。原始论文中采用的是**Post‑LN**方案^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}; 现代 LLM 中常使用**Pre‑LN**，因为它能提供更好的训练稳定性，并减少对长时间预热 (warm-up) 的依赖^{[\[3\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-xiong2020-3)}。 ## 演进与现代变体早期基于循环神经网络 (RNN) 及其高级变体（如 LSTM）的方法是按顺序逐个 token 处理文本的。虽然这种方法直观上符合语言结构，但它也带来了显著的局限性：难以进行并行计算，并且难以捕捉文本中相距较远的元素之间的依赖关系。2017年，一组来自谷歌的研究人员发表了一篇题为《Attention Is All You Need》的论文。他们在文中描述了一种全新的架构——“Transformer”。该模型首次完全摒弃了循环神经网络，并以“注意力”(attention) 机制取而代之。其核心创新在于，注意力机制允许 Transformer 在生成输出序列中的某个词时，评估输入序列中每个词的重要性。同时，模型能够并行处理所有词。这种并行处理能力使得在海量数据上训练规模大得多的模型成为可能。最终催生了现代的大型语言模型 (LLM)。 Transformer 架构为众多模型奠定了基础，这些模型大致可分为三类。 ### 1. 仅编码器模型 (Encoder-only) - **示例：** BERT (及 RoBERTa, ALBERT)^{[\[8\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-devlin2019-8)}。 - **原理：** 通过带有双向上下文的**掩码语言建模 (MLM)**任务进行预训练。 - **应用：** 理解类任务（如分类、命名实体识别 NER 等）。 ### 2. 仅解码器模型 (Decoder-only) - **示例：** GPT 系列 (GPT‑1/2/3)^{[\[9\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-radford2018-9)[\[10\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-brown2020-10)}、LLaMA^{[\[11\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-llama2023-11)}、Claude。 - **原理：** **因果语言建模 (CLM)**——预测下一个 token；注意力机制应用了因果掩码^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}。 - **应用：** 文本生成、对话、代码生成。 ### 3. 编码器-解码器模型 (Encoder-decoder) - **示例：** 原始 Transformer、T5、BART^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)[\[12\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-raffel2019-12)}。 - **原理：** 编码器构建输入的表示，解码器生成输出，并利用交叉注意力机制关注编码器的特征^{[\[1\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-vaswani2017-1)}。 - **应用：** seq2seq 任务（如翻译、摘要等）。 ### 4. 多模态与替代架构 - **Vision Transformer (ViT)**——将图像分割成块（patch）以适应 Transformer 架构^{[\[13\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-dosovitskiy2020-13)}；**Swin Transformer**——一种采用“移位窗口”(shifted windows) 的分层模型^{[\[14\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-liu2021-swin-14)}。 - **长序列替代方案：** - **Mamba**——具有线性复杂度的选择性状态空间模型 (SSM)^{[\[15\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-gu2023-mamba-15)}。 - **RWKV**——一种类似 RNN 的架构，支持并行训练且推理复杂度为线性^{[\[16\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-peng2023-rwkv-16)}。 - **混合模型**（例如 **Jamba**）：交替使用 Transformer 和 Mamba 块；有时会辅以 MoE（专家混合）^{[\[17\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-lieber2024-jamba-17)}。 ## 训练与优化技术 Transformer 的效率与训练技术和基础设施密切相关。 - **预训练策略：** CLM 和 MLM；以及对比学习和去噪目标（如 ELECTRA, T5）^{[\[12\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-raffel2019-12)}。 - **微调技术：** - **全量微调**所有参数。 - **参数高效微调 (PEFT)：** **LoRA** 在冻结基础模型权重的同时，引入低秩适配器^{[\[18\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-hu2021-lora-18)}。 - **行为对齐：** **RLHF**——基于人类反馈的强化学习^{[\[19\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-ouyang2022-rlhf-19)}。 - **推理系统优化：** **PagedAttention/vLLM** 通过对 KV 缓存进行分页管理来提高服务吞吐量；在处理长序列和大批量数据时尤其有效^{[\[20\]](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_note-kwon2023-vllm-20)}。 ## 外部链接 - The Illustrated Transformer — Transformer 架构图解 ## 参考文献 - Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). *Attention Is All You Need*. NeurIPS. arXiv:1706.03762. - Devlin, J., Chang, M.‑W., Lee, K., Toutanova, K. (2019). *BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding*. arXiv:1810.04805. - Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. (2018). *Improving Language Understanding by Generative Pre‑Training*. OpenAI Technical Report. - Brown, T. B., Mann, B., Ryder, N., et al. (2020). *Language Models Are Few‑Shot Learners*. NeurIPS. arXiv:2005.14165. - Raffel, C., Shazeer, N., Roberts, A., et al. (2019). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer*. arXiv:1910.10683. - Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2020). *An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale*. arXiv:2010.11929. - Liu, Z., Lin, Y., Cao, Y., et al. (2021). *Swin Transformer: Hierarchical Vision Transformer using Shifted Windows*. arXiv:2103.14030. - Tay, Y., Dehghani, M., Bahri, D., Metzler, D. (2020). *Efficient Transformers: A Survey*. arXiv:2009.06732. - Xiong, R., Yang, Y., He, D., et al. (2020). *On Layer Normalization in the Transformer Architecture*. ICML. arXiv:2002.04745. - Su, J., Lu, Y., Pan, S., et al. (2021). *RoFormer: Rotary Position Embedding*. arXiv:2104.09864. - Press, O., Smith, N. A., Lewis, M. (2021). *Train Short, Test Long: Attention with Linear Biases (ALiBi)*. arXiv:2108.12409. - Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need* (MQA). arXiv:1911.02150. - Ainslie, J., Lee‑Thorp, J., de Jong, M., et al. (2023). *GQA: Training Generalized Multi‑Query Transformer Models from Multi‑Head Checkpoints*. EMNLP. arXiv:2305.13245. - Kwon, W., Li, Z., Zhuang, S., et al. (2023). *Efficient Memory Management for LLM Serving with PagedAttention* (vLLM). arXiv:2309.06180. - Touvron, H., Lavril, T., Izacard, G., et al. (2023). *LLaMA: Open and Efficient Foundation Language Models*. arXiv:2302.13971. - Gu, A., Dao, T. (2023). *Mamba: Linear‑Time Sequence Modeling with Selective State Spaces*. arXiv:2312.00752. - Peng, B., et al. (2023). *RWKV: Reinventing RNNs for the Transformer Era*. arXiv:2305.13048. - Lieber, O., Lenz, B., Bata, H., et al. (2024). *Jamba: A Hybrid Transformer‑Mamba Language Model*. arXiv:2403.19887. - Hu, E. J., Shen, Y., Wallis, P., et al. (2021). *LoRA: Low‑Rank Adaptation of Large Language Models*. arXiv:2106.09685. - Ouyang, L., Wu, J., Jiang, X., et al. (2022). *Training language models to follow instructions with human feedback*. OpenReview. ## 注释 1. ↑ ^{[1.00](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-0)} ^{[1.01](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-1)} ^{[1.02](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-2)} ^{[1.03](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-3)} ^{[1.04](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-4)} ^{[1.05](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-5)} ^{[1.06](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-6)} ^{[1.07](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-7)} ^{[1.08](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-8)} ^{[1.09](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-9)} ^{[1.10](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-10)} ^{[1.11](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-vaswani2017_1-11)} Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). *Attention Is All You Need*. NeurIPS. arXiv:1706.03762. 2. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-tay2020_2-0) Tay, Y., Dehghani, M., Bahri, D., Metzler, D. (2020). *Efficient Transformers: A Survey*. arXiv:2009.06732. 3. ↑ ^{[3.0](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-xiong2020_3-0)} ^{[3.1](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-xiong2020_3-1)} Xiong, R., Yang, Y., He, D., et al. (2020). *On Layer Normalization in the Transformer Architecture*. ICML. arXiv:2002.04745. 4. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-shazeer2019_4-0) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need*. arXiv:1911.02150. 5. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-ainslie2023_5-0) Ainslie, J., Lee‑Thorp, J., de Jong, M., et al. (2023). *GQA: Training Generalized Multi‑Query Transformer Models from Multi‑Head Checkpoints*. EMNLP. arXiv:2305.13245. 6. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-su2021_6-0) Su, J., Lu, Y., Pan, S., et al. (2021). *RoFormer: Rotary Position Embedding*. arXiv:2104.09864. 7. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-press2021_7-0) Press, O., Smith, N. A., Lewis, M. (2021). *Train Short, Test Long: Attention with Linear Biases (ALiBi)*. arXiv:2108.12409. 8. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-devlin2019_8-0) Devlin, J., Chang, M.‑W., Lee, K., Toutanova, K. (2019). *BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding*. arXiv:1810.04805. 9. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-radford2018_9-0) Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. (2018). *Improving Language Understanding by Generative Pre‑Training*. OpenAI. 10. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-brown2020_10-0) Brown, T. B., Mann, B., Ryder, N., et al. (2020). *Language Models Are Few‑Shot Learners*. NeurIPS. arXiv:2005.14165. 11. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-llama2023_11-0) Touvron, H., Lavril, T., Izacard, G., et al. (2023). *LLaMA: Open and Efficient Foundation Language Models*. arXiv:2302.13971. 12. ↑ ^{[12.0](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-raffel2019_12-0)} ^{[12.1](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-raffel2019_12-1)} Raffel, C., Shazeer, N., Roberts, A., et al. (2019). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer*. JMLR. arXiv:1910.10683. 13. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-dosovitskiy2020_13-0) Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2020). *An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale*. ICLR. arXiv:2010.11929. 14. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-liu2021-swin_14-0) Liu, Z., Lin, Y., Cao, Y., et al. (2021). *Swin Transformer: Hierarchical Vision Transformer using Shifted Windows*. ICCV. arXiv:2103.14030. 15. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-gu2023-mamba_15-0) Gu, A., Dao, T. (2023). *Mamba: Linear‑Time Sequence Modeling with Selective State Spaces*. arXiv:2312.00752. 16. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-peng2023-rwkv_16-0) Peng, B., et al. (2023). *RWKV: Reinventing RNNs for the Transformer Era*. arXiv:2305.13048. 17. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-lieber2024-jamba_17-0) Lieber, O., Lenz, B., Bata, H., et al. (2024). *Jamba: A Hybrid Transformer‑Mamba Language Model*. arXiv:2403.19887. 18. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-hu2021-lora_18-0) Hu, E. J., Shen, Y., Wallis, P., et al. (2021). *LoRA: Low‑Rank Adaptation of Large Language Models*. arXiv:2106.09685. 19. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-ouyang2022-rlhf_19-0) Ouyang, L., Wu, J., Jiang, X., et al. (2022). *Training language models to follow instructions with human feedback*. OpenReview. 20. [↑](https://systems-analysis.info/int/Transformer_architecture_%E2%80%94_Transformer_%E6%9E%B6%E6%9E%84#cite_ref-kwon2023-vllm_20-0) Kwon, W., Li, Z., Zhuang, S., et al. (2023). *Efficient Memory Management for LLM Serving with PagedAttention*. arXiv:2309.06180.