---
title: "Large language model architectures — สถาปัตยกรรมของโมเดลภาษาขนาดใหญ่"
source: "https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88"
wiki: "systems-analysis.info/int"
article: "Large_language_model_architectures_—_สถาปัตยกรรมของโมเดลภาษาขนาดใหญ่"
language: "th"
categories:
  - "Category:Large language models"
  - "Category:Machine learning"
  - "Category:Thai"
revision_id: 3791
wiki_created_at: 2026-09-06T23:26:11Z
wiki_modified_at: 2026-09-06T23:26:11Z
downloaded_at: 2026-09-07T22:59:05Z
---

# Large language model architectures — สถาปัตยกรรมของโมเดลภาษาขนาดใหญ่

**สถาปัตยกรรมของโมเดลภาษาขนาดใหญ่ (LLM)** — คือหลักการและโครงสร้างพื้นฐานที่กำหนดวิธีการสร้าง การฝึก และการทำงานของโมเดลภาษาขนาดใหญ่ LLM สมัยใหม่ที่สามารถเข้าใจและสร้างภาษามนุษย์ได้นั้น เกือบทั้งหมดอิงบนสถาปัตยกรรม **Transformer**<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Vaswani2017-1)</sup> แต่รวมถึงการปรับปรุงและแนวทางต่าง ๆ มากมายที่มุ่งเพิ่มประสิทธิภาพ ความสามารถในการปรับขนาด และขีดความสามารถ

## ตระกูลสถาปัตยกรรม LLM (transformer)

โมเดลภาษาขนาดใหญ่สมัยใหม่อิงบนสถาปัตยกรรม transformer<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Vaswani2017-1)</sup> แต่ใช้งานในรูปแบบที่แตกต่างกันตามวัตถุประสงค์ ไม่ว่าจะเป็นการทำความเข้าใจข้อความ การสร้างข้อความต่อเนื่อง หรือการแปลงข้อความหนึ่งเป็นอีกข้อความหนึ่ง ในทางปฏิบัติมีการแบ่งออกเป็นสามตระกูลโดยยังคงหลักการพื้นฐานของ transformer<sup>[\[2\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-2)[\[3\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-3)[\[4\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-4)</sup>

### 1. Encoder‑only (เฉพาะตัวเข้ารหัส)

โมเดลใช้เฉพาะกลุ่ม encoder และพิจารณาข้อความขาเข้าทั้งหมดในแบบสองทิศทาง การฝึกเบื้องต้นมักสร้างบน masked‑language modeling (MLM หรือการสร้างแบบจำลองภาษาด้วยการซ่อน) โดยซ่อน token บางส่วนแล้วให้โมเดลเรียนรู้การกู้คืนจากบริบทโดยรอบ ด้วยบริบทสองทิศทาง โมเดลเหล่านี้จึงเก่งในงานด้านความเข้าใจและการให้คะแนน ได้แก่ การจำแนกประเภท การระบุเอนทิตี การจัดอันดับเอกสาร และ extractive QA อย่างไรก็ตาม โมเดลเหล่านี้ไม่ได้ออกแบบมาสำหรับการสร้างข้อความแบบ autoregressive ตั้งแต่ต้น

นอกจากนี้ในทางปฏิบัติยังใช้วัตถุประสงค์การฝึกเบื้องต้นทางเลือกสำหรับตระกูล encoder‑only อีกด้วย ได้แก่ *replaced token detection (RTD) ใน ELECTRA* (โมเดล discriminator ตรวจจับ token ที่ถูกแทนที่) และ *การเรียนรู้เชิงเปรียบเทียบ* ของ bi-encoder สำหรับการค้นหาเชิงความหมาย/retriever (InfoNCE/softmax‑loss บนคู่ "คำถาม-เอกสาร" เช่นใน Dense Passage Retrieval) เมื่อใช้ใน RAG encoder‑only ทำหน้าที่เป็น *bi‑encoder* (เข้ารหัสคำถามและเอกสารแยกกันสำหรับการค้นหา ANN ที่รวดเร็ว) หรือเป็น *cross‑encoder* (เข้ารหัสคู่ร่วมกันเพื่อการจัดอันดับใหม่อย่างแม่นยำ)

**ข้อดี:**

- คุณภาพสูงในการทำความเข้าใจข้อความด้วยบริบทสองทิศทาง ได้แก่ การจำแนกประเภท NER การดึงข้อเท็จจริง การจัดอันดับใหม่ และ extractive QA
- การประมวลผลแบบขนานและปริมาณงานสูง: ผ่านเดียวโดยไม่มี auto‑regression เหมาะสำหรับการ batch การให้คะแนนขนาดใหญ่
- การผสานรวมกับการค้นหาและ RAG ได้อย่างเป็นธรรมชาติ: ในบทบาท bi‑encoder — การค้นหาเชิงความหมายที่รวดเร็ว; ในบทบาท cross‑encoder — การจัดอันดับใหม่อย่างแม่นยำ
- การปรับตัวที่มีประสิทธิภาพ: รูปแบบที่ค่อนข้างกะทัดรัด (≈100–300 ล้านพารามิเตอร์; BERT‑base ≈110 ล้าน) ให้คุณภาพสูงหลังจาก fine-tuning ตามเป้าหมาย
- Latency ที่มั่นคงไม่ขึ้นอยู่กับความยาวของคำตอบที่สร้าง (ไม่มีการถอดรหัสทีละขั้นตอน) เหมาะสำหรับการให้คะแนนแบบออฟไลน์ของชุดเอกสารขนาดใหญ่
- ความสามารถในการขยายหน้าต่างบริบทของ encoder ด้วยตำแหน่งสัมพัทธ์/เชิงหมุนและ/หรือ attention แบบกระจัดกระจายในพื้นที่ (เช่น Longformer/BigBird) ซึ่งมีประโยชน์สำหรับเอกสารยาว

**ข้อเสีย:**

- ไม่มีความสามารถในการสร้างข้อความในตัว: สำหรับบทสนทนาและคำตอบโดยละเอียดจำเป็นต้องมี decoder หรือโมดูลสร้างภายนอก
- ข้อจำกัดในสถานการณ์แบบโต้ตอบ: ไม่มีการสร้างทีละขั้นตอนพร้อมการรักษาสถานะ
- ความไม่สอดคล้องของวัตถุประสงค์การฝึกเบื้องต้นกับงานการสร้างแบบอิสระ: MLM สอดคล้องกับการสร้างได้น้อยกว่าเมื่อเทียบกับ causal modeling
- หน้าต่างบริบทที่จำกัดในอดีต (มักเป็น 512 token ในการกำหนดค่าพื้นฐานของตำแหน่งสัมบูรณ์) การขยายต้องใช้รูปแบบตำแหน่ง/attention พิเศษและ/หรือ fine-tuning
- สำหรับงาน retrieval จำเป็นต้องมี fine-tuning เชิงเปรียบเทียบแยกต่างหากของ bi‑encoder และ/หรือ cross‑encoder หากไม่มีสิ่งนี้ คุณภาพการค้นหา/การจัดอันดับใหม่มักจะต่ำกว่าโมเดลที่ฝึกมาโดยเฉพาะ

**โมเดลตัวอย่าง:** BERT และรูปแบบที่ได้รับการพัฒนา รวมถึง RoBERTa และ DeBERTa (รูปแบบ encoder‑only ขั้นสูง) สำหรับวัตถุประสงค์การฝึกทางเลือก — ELECTRA (RTD) <sup>[\[5\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-5)[\[6\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-6)[\[7\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-7)[\[8\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-8)[\[9\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-9)[\[10\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-10)</sup>

### 2. Decoder‑only (เฉพาะตัวถอดรหัส)

ใช้เฉพาะกลุ่ม decoder ที่มี attention แบบ causal (ซ้ายไปขวา): โมเดลทำนาย token ถัดไปจาก prefix ที่กำหนดไว้แล้ว โหมดการฝึกนี้ — causal language modeling (CLM) — ทำให้โมเดลเหล่านี้เป็นตัวเลือกที่เป็นธรรมชาติสำหรับการสร้างข้อความ ได้แก่ บทสนทนา คำตอบโดยละเอียด ข้อความสร้างสรรค์ และโค้ดโปรแกรม ข้อแลกเปลี่ยนคือความล่าช้าที่เพิ่มขึ้นและขนาด KV‑cache ที่ใหญ่ขึ้นเมื่อมี prompt ยาว ในทางปฏิบัติสำหรับ decoder‑only มีการใช้เทคนิควิศวกรรมอย่างแพร่หลาย ได้แก่ การลด KV‑cache ด้วย MQA และ GQA การเร่งการ inference ผ่าน speculative decoding และการเพิ่มประสิทธิภาพฝั่งเซิร์ฟเวอร์ (PagedAttention/vLLM, continuous batching, chunked prefill)<sup>[\[11\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-11)[\[12\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-12)[\[13\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-13)[\[14\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-14)[\[15\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-15)</sup>

**ข้อดี:**

- การสร้างข้อความแบบธรรมชาติ (CLM): ความสามารถ zero‑shot และ few‑shot ที่แข็งแกร่ง ปรับขนาดได้ดี<sup>[\[16\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-16)</sup>
- ความเอนกประสงค์: โมเดลเดียวแก้ปัญหาหลายอย่างด้วยคำสั่งและตัวอย่างใน prompt รวมเข้ากับ RAG และการเรียกใช้เครื่องมือ (tool use) ได้อย่างเป็นธรรมชาติ
- ระบบนิเวศที่เติบโตเต็มที่: แนวปฏิบัติของ fine-tuning เชิงคำสั่งและการปรับพฤติกรรม (RLHF, DPO) มีการใช้งานทั้งแบบเปิดและเชิงพาณิชย์<sup>[\[17\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-17)[\[18\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-18)</sup>
- ชุดการเพิ่มประสิทธิภาพ inference ที่สมบูรณ์: MQA/GQA ลดขนาด KV‑cache และเพิ่มปริมาณงาน; speculative decoding เร่งการ inference โดยไม่เปลี่ยนการกระจาย; PagedAttention/vLLM พร้อม continuous batching และ chunked prefill เพิ่มการใช้งาน GPU แบบ end-to-end<sup>[\[19\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-19)[\[20\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-20)[\[21\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-21)[\[22\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-22)[\[23\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-23)</sup>
- รองรับการสร้างโครงสร้างสำหรับรูปแบบคำตอบที่เข้มงวด (JSON/SQL/DSL) ซึ่งช่วยให้การผสานรวมกับระบบสารสนเทศและ API ง่ายขึ้น<sup>[\[24\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-24)[\[25\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-25)</sup>

**ข้อเสีย:**

- Latency การสร้างที่สูงขึ้น: การ inference แบบลำดับ; ต้นทุนของ token ใหม่เติบโตตามความยาวของบริบทที่ "อ่านไปแล้ว" (KV‑cache)
- ประสิทธิภาพน้อยกว่าในโปรไฟล์ "input ยาว – output สั้น" (การสรุป การแปล) เมื่อเทียบกับ encoder–decoder ที่เข้ารหัส input เพียงครั้งเดียว
- ข้อจำกัดของบริบททิศทางเดียว: ในงานด้านความเข้าใจบางครั้งด้อยกว่าโมเดลที่มีการแสดงผลสองทิศทาง (encoder‑only / encoder–decoder)
- หน่วยความจำสำหรับ KV‑cache อาจเป็น "คอขวด" เมื่อมี prompt ยาวและ batch ขนาดใหญ่<sup>[\[26\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-26)</sup>
- การ quantize activation/KV (INT8/FP8) เร่ง inference แต่อาจลดคุณภาพในบริบทยาว/โค้ด จำเป็นต้องมีการตรวจสอบความถูกต้องอย่างละเอียด (โดยเฉพาะเมื่อมี SLA ที่เข้มงวด)

**โมเดลตัวอย่าง:** GPT‑3, GPT‑4 (รายละเอียดสถาปัตยกรรมและชุดข้อมูลไม่เปิดเผยต่อสาธารณะ), LLaMA และ *Llama 3* (8B/70B, 2024)<sup>[\[27\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-27)[\[28\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-28)[\[29\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-29)[\[30\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-30)</sup>

### 3. Encoder–decoder (ตัวเข้ารหัส–ตัวถอดรหัส)

สถาปัตยกรรมนี้รวมส่วนประกอบทั้งสองไว้ด้วยกัน Encoder ทำงานในโหมดสองทิศทาง ส่วน decoder — แบบ causal Encoder วิเคราะห์ input ครั้งเดียวและสร้างการแทนค่า; decoder สร้าง output โดยอ้างอิงการแทนค่านั้นผ่าน cross‑attention แนวทางแยกส่วนนี้มีประโยชน์อย่างยิ่งในกรณีที่ต้องแปลง input ข้อความยาวให้เป็น output สั้น: การแปลด้วยเครื่อง การสรุป และการตอบคำถามโดยอิงเอกสาร แม้วิธีนี้ต้องการต้นทุนการคำนวณรวมที่สูงกว่า (สองกลุ่มและ cross‑attention) แต่ข้อได้เปรียบคือการสร้างที่ควบคุมได้บนพื้นฐานการวิเคราะห์ข้อความต้นฉบับอย่างสมบูรณ์ โดยการเข้ารหัสทำเพียงครั้งเดียวและนำมาใช้ซ้ำตลอดกระบวนการ inference

**ข้อดี:**

- การสร้างแบบมีเงื่อนไข: decoder ใช้ cross‑attention กับการแทนค่าของ input<sup>[\[31\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-31)</sup>
- มีประสิทธิภาพในสถานการณ์ "input ยาว → output สั้น": input ถูกเข้ารหัสเพียงครั้งเดียว
- เหมาะสำหรับรูปแบบ "text‑to‑text" และการสร้าง output แบบควบคุม (prefix ของงาน คำสั่งพิเศษ)<sup>[\[32\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-32)</sup>
- ความเสถียรและประสิทธิภาพเมื่อมีแหล่งที่มายาว: ในระยะถอดรหัส self‑attention ของ output เติบโต แต่ cross‑attention นำ key/value ที่คงที่จาก encoder มาใช้ซ้ำ (input ไม่ถูก "อ่านใหม่" ในทุกขั้นตอน)

**ข้อเสีย:**

- สองกลุ่มเพิ่มความต้องการหน่วยความจำและการคำนวณในระหว่างการฝึกและการใช้งาน
- สำหรับลำดับที่ยาวมาก latency รวมเทียบได้กับ decoder‑only; auto‑regression ยังคงเป็นคอขวด
- โมเดลแชทแบบเอนกประสงค์มีน้อยกว่าในกลุ่ม decoder‑only; มักใช้เป็น seq2seq engine คุณภาพสูงสำหรับงานเฉพาะ
- เมื่อ input ยาวมาก หน่วยความจำสำหรับ key/value ของ cross‑attention ในแต่ละเลเยอร์ของ decoder (ตลอดแหล่งที่มา) จะเพิ่มขึ้น ต้องวางแผนการให้บริการอย่างระมัดระวัง

**โมเดลตัวอย่าง:** T5 (รวมถึง T5 v1.1 และแนวปฏิบัติ fine-tuning เชิงคำสั่งใน *FLAN‑T5*) และ BART<sup>[\[33\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-33)[\[34\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-34)[\[35\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-35)</sup>

## Dense Transformer - Transformer แบบหนาแน่น

สถาปัตยกรรม LLM แบบคลาสสิกและแพร่หลายที่สุด: ในการประมวลผล token แต่ละตัวจะมีพารามิเตอร์ของโมเดลเกือบทั้งหมดเข้าร่วม ต่างจากแนวทาง sparse (เช่น Mixture‑of‑Experts) ตรงที่ไม่มีการเปิดใช้งาน subnetwork แบบเลือกสรร — แต่ละบล็อกทำงานสำหรับ token ทุกตัว<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Vaswani2017-1)</sup>

### หลักการทำงานและสถาปัตยกรรม

**โครงสร้างพื้นฐาน.** โมเดลคือกลุ่มของบล็อก transformer N บล็อกที่มีรูปแบบเหมือนกัน แต่ละบล็อกประกอบด้วย:

1.  **Multi‑Head Self‑Attention (การใส่ใจตนเองหลายหัว).** สำหรับ token แต่ละตัวจะคำนวณเวกเตอร์สามตัว: Q (query), K (key), V (value); attention ถูกกำหนดเป็น $\operatorname{softmax}\!\left( \frac{QK^{o}p + M}{\sqrt{d_{k}}} \right) \cdot V$ โดยที่ $M$ คือ mask (causal และ/หรือ padding mask) ที่ยกเว้นตำแหน่งที่ไม่อนุญาต "หัว" attention หลายหัวพิจารณาด้านต่าง ๆ ของบริบทแบบขนาน (H หัว มักเป็น $d_{head} = \frac{d_{model}}{H}$) จำนวนหัวเพิ่มขึ้นตามขนาดของโมเดล<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Vaswani2017-1)</sup>
2.  **Feed‑Forward Network (FFN หรือเครือข่ายไปข้างหน้าแบบเชื่อมต่อเต็มรูปแบบ).** สองเลเยอร์เชิงเส้นที่มี nonlinearity ระหว่างกัน (มักเป็น GELU/SiLU; ในโมเดลสมัยใหม่บางรุ่น — SwiGLU) มิติตรงกลางมักเป็น $\approx 4\, d_{model}$; เมื่อใช้ SwiGLU มักใช้ $\approx frac83\, d_{model}$ เพื่อรักษาจำนวนพารามิเตอร์ที่ใกล้เคียงกัน FFN มีพารามิเตอร์เป็นสัดส่วนที่มีนัยสำคัญ<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Vaswani2017-1)[\[36\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-36)</sup>

**ส่วนประกอบเพิ่มเติม.** ใช้ residual connection และการ normalize เลเยอร์; ใน LLM สมัยใหม่มักใช้ Pre‑LN (การ normalize ก่อน subblock) — ซึ่งช่วยเพิ่มเสถียรภาพการฝึกที่ความลึกมาก นอกจาก LayerNorm แบบคลาสสิก ยังใช้ **RMSNorm** อย่างแพร่หลายมากขึ้น (ลดต้นทุนการคำนวณและทำงานได้ดีในโมเดลขนาดใหญ่) นอกจากนี้ในบางตระกูลยังใช้การ normalize ในพื้นที่ attention (เช่น การ normalize Q/K ก่อน softmax) การแสดงตำแหน่งอาจเป็นแบบสัมบูรณ์หรือสัมพัทธ์; สำหรับบริบทยาว RoPE ได้กลายเป็น de facto standard

##### ตัวอย่างโมเดลและขนาด

- BERT‑Large: 24 เลเยอร์, มิติ 1024, 16 หัว attention, ≈340 ล้านพารามิเตอร์<sup>[\[37\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-37)</sup>
- GPT‑3 (175B): 96 เลเยอร์, มิติ 12288, 96 หัว attention, ≈175 พันล้านพารามิเตอร์<sup>[\[38\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-38)</sup>
- LLaMA‑65B: 80 เลเยอร์, มิติ 8192, 64 หัว attention, ≈65 พันล้านพารามิเตอร์<sup>[\[39\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-39)</sup>
- PaLM‑540B: 118 เลเยอร์, มิติประมาณ 18432, ≈540 พันล้านพารามิเตอร์<sup>[\[40\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-40)</sup>

##### ข้อดี

- บล็อกที่สม่ำเสมอ โหมดการฝึกที่ศึกษาดีแล้ว และพฤติกรรมที่คาดเดาได้เมื่อปรับขนาด
- คุณภาพดีขึ้นตามกฎกำลังเมื่อพารามิเตอร์และข้อมูลเพิ่มขึ้น; โหมด compute‑optimal ต้องการการเพิ่มขนาดโมเดลและจำนวน token สำหรับการฝึกพร้อมกัน<sup>[\[41\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-41)[\[42\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-42)</sup>
- สถาปัตยกรรมเดียวกันหลัง fine-tuning ครอบคลุมงานหลากหลายโดยไม่ต้องเปลี่ยนแปลงระดับเลเยอร์

##### ข้อเสีย

- Self‑attention แบบเต็มรูปแบบมีความซับซ้อน quadratic ตามความยาวลำดับ ($O(n^{2})$) ซึ่งจำกัดหน้าต่างบริบท<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Vaswani2017-1)</sup>
- การเปิดใช้งานพารามิเตอร์เต็มรูปแบบในขั้นตอนการสร้าง: ใน decoder ที่ไม่มี MoE ต้นทุนการ inference ต่อ token เพิ่มขึ้นตามสัดส่วนของจำนวนพารามิเตอร์โดยประมาณ
- คอขวดคือ bandwidth ของหน่วยความจำ (memory‑bound): การโหลด weight จาก HBM มักจำกัดความเร็ว inference

##### ข้อจำกัดของการปรับขนาดและบริบท

- หน่วยความจำสำหรับพารามิเตอร์เพิ่มขึ้นเชิงเส้นตามขนาดโมเดล; หน่วยความจำสำหรับการฝึกเพิ่มขึ้นเนื่องจาก gradient และสถานะ optimizer
- การกำหนดค่าพื้นฐานในอดีตมักจำกัดอยู่ที่ 2–4 พัน token รูปแบบตำแหน่งสมัยใหม่ (RoPE) และเทคนิคการขยาย (Position Interpolation, YaRN และอื่น ๆ) ช่วยให้ขยายหน้าต่างได้อีกหลายเท่า แต่แลกกับภาระการคำนวณ/หน่วยความจำเพิ่มเติม<sup>[\[43\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-43)[\[44\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-44)</sup>

### การเพิ่มประสิทธิภาพสมัยใหม่

- **FlashAttention.** Attention แบบแม่นยำที่คำนึงถึงลำดับชั้นหน่วยความจำ GPU; ลดการใช้หน่วยความจำและเร่งการฝึก/inference สำหรับลำดับยาว<sup>[\[45\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-45)</sup>
- **การลดและจัดการ KV‑cache.** Multi‑Query Attention และ Grouped‑Query Attention ลดขนาด cache และ traffic หน่วยความจำ; ในระดับเซิร์ฟเวอร์ PagedAttention (vLLM) เพิ่ม throughput ด้วยการจัดการ cache แบบ paging<sup>[\[46\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-46)[\[47\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-47)[\[48\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-48)</sup>
- **Speculative Decoding.** โมเดล draft เสนอข้อความต่อ แล้วโมเดลหลักตรวจสอบอย่างรวดเร็ว ได้ความเร็วเพิ่มขึ้นโดยไม่เปลี่ยนการกระจายของ output<sup>[\[49\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-49)</sup>

## Sparse Models และ Mixture‑of‑Experts (MoE)

MoE คือวิธีเพิ่มความจุของโมเดลโดยไม่ต้องเพิ่มการคำนวณต่อ token แบบสัดส่วน แทนที่จะใช้บล็อก FFN ขนาดใหญ่บล็อกเดียวในเลเยอร์ จะใช้กลุ่ม "experts" แบบขนาน (FFN อิสระหลายตัว) และ gating network ที่ฝึกได้จะเลือก expert ที่เกี่ยวข้องที่สุด top‑k สำหรับแต่ละ token (มักเป็น k=1–2; ในบางโมเดลเป็น k=4) เฉพาะ expert ที่เลือกเท่านั้นที่ถูกเปิดใช้งาน; output ของพวกมันถูกถ่วงน้ำหนักและรวมกัน ดังนั้นพารามิเตอร์ทั้งหมดอาจมีหลายร้อยพันล้านหรือแม้แต่ล้านล้าน แต่ในแต่ละขั้นตอนมีเพียงส่วนน้อยเท่านั้นที่ถูกใช้งาน<sup>[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Switch-50)[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-GLAM-51)</sup>

#### ตัวอย่างโมเดลและขนาด

- **Switch Transformer (Google)**: ถึง ~1.6T พารามิเตอร์; การ routing แบบ top‑1 (หนึ่ง expert ต่อหนึ่ง token) แสดงให้เห็นว่า MoE ช่วยเพิ่มความจุได้อย่างมากด้วยต้นทุนต่อ token ที่ใกล้เคียงกัน<sup>[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Switch-50)</sup>
- **GLaM (Google)**: 1.2T พารามิเตอร์, 64 expert ต่อเลเยอร์, top‑2; สำหรับแต่ละ token มีการเปิดใช้งาน ≈96.6B พารามิเตอร์ (≈8%)<sup>[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-GLAM-51)</sup>
- **Mixtral 8×7B (Mistral AI)**: ~46.7B พารามิเตอร์ทั้งหมด, ≈12.9B ที่ active ต่อ token, top‑2<sup>[\[52\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Mixtral8x7-52)[\[53\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Mixtral8x7_paper-53)</sup>
- **Mixtral 8×22B**: ~141B พารามิเตอร์ทั้งหมด, ≈39B ที่ active ต่อ token, top‑2<sup>[\[54\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Mixtral8x22-54)</sup>
- **DBRX (Databricks)**: 132B พารามิเตอร์ทั้งหมด, ≈36B ที่ active ต่อ token; 16 expert และการ routing แบบ top‑4 (fine‑grained MoE)<sup>[\[55\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-DBRX-55)</sup>

##### ข้อดี

- ต้นทุนการคำนวณถูกกำหนดโดยจำนวน expert ที่ active k ไม่ใช่จำนวนพารามิเตอร์รวม: สามารถฝึกและใช้โมเดลขนาดล้านล้านพารามิเตอร์ด้วยต้นทุนที่เทียบได้กับโมเดล dense ขนาดเล็กกว่าอย่างเห็นได้ชัด<sup>[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-GLAM-51)</sup>
- การเชี่ยวชาญเฉพาะด้าน: expert ปรับตัวโดยอัตโนมัติตามภาษา/โดเมน/รูปแบบ ช่วยเพิ่มคุณภาพในงานหลายโดเมน
- การ deploy ที่ยืดหยุ่น: สามารถเก็บ expert ที่ใช้บ่อยไว้ในหน่วยความจำและโหลด expert ที่ใช้น้อยตามต้องการ (พร้อม infrastructure ที่เหมาะสม)

##### ข้อจำกัด

- การ balance load: หากไม่มี regularization router อาจ "ติด" กับ expert บางส่วน (router collapse) จำเป็นต้องมี auxiliary losses (load‑balancing) และรูปแบบ routing ที่ปรับปรุงแล้ว<sup>[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Switch-50)</sup>
- ความซับซ้อนของการคำนวณแบบกระจาย: ต้องการ expert parallelism และการแลกเปลี่ยน all‑to‑all; ค่าใช้จ่ายด้านการสื่อสารและการจัดการหน่วยความจำกลายเป็นคอขวด<sup>[\[56\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-NVIDIA_MoE-56)</sup>
- เสถียรภาพการฝึก: การตั้งค่า router และข้อจำกัด capacity สำคัญมาก มิฉะนั้นอาจเกิดการลดลงของคุณภาพ/การ convergence

#### การปรับปรุงสมัยใหม่

- **Expert‑Choice routing**: expert "เลือก" token ซึ่งช่วยปรับปรุงการ balance และ convergence ด้วยต้นทุนที่ใกล้เคียงกัน<sup>[\[57\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-ExpertChoice-57)</sup>
- **Fine‑grained MoE**: จำนวน expert ที่มากขึ้นและขนาดเล็กลง (เช่นใน DBRX) ให้ความเชี่ยวชาญเฉพาะที่ละเอียดขึ้น<sup>[\[55\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-DBRX-55)</sup>
- **Sparse Upcycling**: การแปลงโมเดล dense เป็น MoE จาก checkpoint ช่วยเพิ่มคุณภาพได้อย่างมีนัยสำคัญด้วยต้นทุนปานกลาง<sup>[\[58\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-SparseUpcycling-58)</sup>

#### ความเหมาะสมของการใช้ MoE

- ผู้ช่วยหลายโดเมนขนาดใหญ่ที่มีงบประมาณ compute จำกัด
- การฝึกบน corpus ขนาดใหญ่ที่การเชี่ยวชาญเฉพาะด้านให้ประโยชน์
- สถานการณ์ที่มี infrastructure แบบกระจายขั้นสูง (GPU/TPU จำนวนมากและเครือข่ายความเร็วสูง)

**เมื่อใดที่โมเดล dense ดีกว่า**: infrastructure จำกัด (1–2 GPU), ข้อกำหนด latency ที่คาดเดาได้อย่างเข้มงวด และความเรียบง่ายในการ deploy

## Retrieval‑Augmented Generation (RAG)

RAG คือ **pattern ของระบบ** รอบ LLM ไม่ใช่สถาปัตยกรรมภายในของโมเดลเอง มันรวม LLM (ส่วนประกอบเชิงสร้างสรรค์) กับฐานความรู้ภายนอก (ส่วนประกอบการดึงข้อมูล) ซึ่งช่วยชดเชยข้อจำกัดของ "parametric memory" ของโมเดล

- **หลักการทำงาน:** ก่อนการสร้าง LLM ดึงเอกสารที่เกี่ยวข้องจากแหล่งภายนอก (wiki, ฐานความรู้องค์กร, เว็บ) และอิงพวกมันในการสร้างคำตอบ<sup>[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-RAG-59)</sup>
- **ข้อดี:**
  - การลดภาพหลอนและปรับปรุงความถูกต้องเชิงข้อเท็จจริง<sup>[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-RAG-59)[\[60\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-60)</sup>
  - ความทันสมัยโดยไม่ต้องฝึกโมเดลใหม่ทั้งหมด<sup>[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-RAG-59)</sup>
  - การอ้างอิงได้และการติดตามคำตอบได้
- **การประยุกต์ใช้:** De facto standard สำหรับผู้ช่วยองค์กรและระบบที่ต้องการข้อเท็จจริงที่ตรวจสอบได้และการทำงานกับข้อมูลส่วนตัว/เฉพาะทาง<sup>[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-RAG-59)</sup>

## กลไก Attention และการทำงานกับบริบท

Self‑attention พื้นฐานมีความซับซ้อน quadratic ตามความยาวลำดับ ($O(n^{2})$) ดังนั้นจึงมีการเพิ่มประสิทธิภาพต่าง ๆ เกิดขึ้น

- **Sparse Attention (การใส่ใจแบบกระจัดกระจาย):** การจำกัด attention ด้วยหน้าต่างท้องถิ่น/รูปแบบ ตัวอย่าง: **Longformer**<sup>[\[61\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-61)</sup>, **BigBird**<sup>[\[62\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-62)</sup>
- **FlashAttention:** การจัดลำดับการคำนวณใหม่โดยคำนึงถึงลำดับชั้นหน่วยความจำ GPU; ให้ประโยชน์ด้านเวลาและหน่วยความจำอย่างมีนัยสำคัญ และกลายเป็น de facto standard ในการฝึก LLM ที่มีบริบทยาว<sup>[\[63\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-63)[\[64\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-64)[\[65\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-65)</sup>
- **MQA/GQA (การเร่งการถอดรหัส):** *Multi‑Query Attention* (key/value ที่ใช้ร่วมกันสำหรับทุกหัว) ลด traffic ของ KV‑cache<sup>[\[66\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-66)</sup> *Grouped‑Query Attention* สร้างสมดุลระหว่างคุณภาพ/ความเร็ว<sup>[\[67\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-67)</sup>
- **การแสดงตำแหน่งที่ปรับปรุงแล้ว:**
  - **ALiBi (Attention with Linear Biases):** การเลื่อนเชิงเส้นในคะแนน attention ช่วยปรับปรุงการ generalize บนความยาวที่มากขึ้น<sup>[\[68\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-68)</sup>
  - **RoPE (Rotary Position Embeddings):** ข้อมูลตำแหน่งสัมพัทธ์ผ่านการหมุน Q/K; ใช้กันอย่างแพร่หลายในโมเดลสมัยใหม่ (เช่น LLaMA)<sup>[\[69\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-69)[\[70\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-70)</sup>
  - **การขยายบริบทสำหรับโมเดล RoPE:** *Position Interpolation*<sup>[\[71\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-71)</sup>, *YaRN*<sup>[\[72\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-72)</sup> รวมถึงการปรับเปลี่ยนแบบ NTK‑aware ช่วยให้ขยายหน้าต่างบริบทได้อย่างมีประสิทธิภาพโดยไม่ต้องเปลี่ยนสถาปัตยกรรม

<!-- -->

- **แนวทางอื่น ๆ สำหรับลำดับยาว:**
  - **Transformer‑XL:** หน่วยความจำ recurrent ระหว่าง segment สำหรับการสร้างแบบจำลอง long-range dependency<sup>[\[73\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-73)</sup>
  - **Reformer:** LSH‑attention และ reversible residual block เพื่อประหยัดหน่วยความจำ<sup>[\[74\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-74)</sup>
  - **Performer:** การประมาณเชิงเส้นของ softmax‑attention (FAVOR+)<sup>[\[75\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-75)</sup>
  - **Linformer:** การประมาณ low-rank ของ matrix attention<sup>[\[76\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-76)</sup>

## การเพิ่มประสิทธิภาพโมเดลและ Infrastructure สำหรับการฝึก

สำหรับการฝึกและ deploy LLM ใช้เทคนิคและ framework เฉพาะทาง

- **Quantization (การลดบิต):** การลดจำนวนบิตของ weight ลดหน่วยความจำและเร่ง inference **QLoRA** ช่วยให้ fine-tune โมเดล 4‑bit (รวมถึง 65B) ได้อย่างมีประสิทธิภาพด้วยคุณภาพใกล้เคียงกับความแม่นยำเต็ม<sup>[\[77\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-77)</sup>
- **Knowledge Distillation (การกลั่นความรู้):** การฝึกแบบ *Teacher→Student* สำหรับโมเดลขนาดกะทัดรัด<sup>[\[78\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-78)</sup>; ตัวอย่าง — **DistilBERT**<sup>[\[79\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-79)</sup>
- **การฝึกแบบกระจาย:**
  - **DeepSpeed** และ **ZeRO** — การกระจายพารามิเตอร์/gradient/สถานะ optimizer สำหรับการฝึกโมเดลขนาดล้านล้านพารามิเตอร์<sup>[\[80\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-80)</sup>
  - **Megatron‑LM** — tensor parallelism และ pipeline parallelism สำหรับ transformer ขนาดใหญ่มาก<sup>[\[81\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-81)</sup>
- **ระบบนิเวศและเครื่องมือ:** **Hugging Face Transformers** และ **Accelerate** ให้การ implement โมเดลมาตรฐานและการผสานรวมกับ DeepSpeed/FSDP สำหรับการฝึกและ inference<sup>[\[82\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-82)[\[83\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-83)</sup>

## Scaling Laws และ Compute‑Optimal Training

**กฎการปรับขนาด** เชิงประจักษ์แสดงให้เห็นว่า cross-entropy error ลดลงตามกฎกำลังเมื่อพารามิเตอร์ ข้อมูล และการคำนวณเพิ่มขึ้น<sup>[\[84\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-84)</sup> งาน **Chinchilla** ได้ระบุโหมด **compute‑optimal** ให้ชัดเจนขึ้น: เพื่อประสิทธิภาพที่ดีที่สุด ขนาดโมเดลและจำนวน token สำหรับการฝึกควรปรับขนาดร่วมกัน (ตัวอย่าง — โมเดล 70B ที่ฝึกบน ~1.4T token แสดงประสิทธิภาพเหนือกว่าโมเดลขนาดใหญ่กว่าที่ฝึกไม่เพียงพอ)<sup>[\[85\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-85)</sup>

## State Space Models (SSM) - โมเดลกับปริภูมิสถานะ

**State Space Models (SSM)** คือสถาปัตยกรรมทางเลือกแทน transformer สำหรับการทำงานกับลำดับยาว มันยืมแนวคิดจากทฤษฎีการควบคุมและการประมวลผลสัญญาณดิจิทัล และแก้ปัญหาหลักของ self‑attention: การเติบโต quadratic ของการคำนวณตามความยาวข้อความที่เพิ่มขึ้น

### ปัญหาหลักและแนวทางแก้ไข

**ปัญหาของ transformer.** ปัญหาหลักของ transformer แบบดั้งเดิมคือความซับซ้อน quadratic ของ attention: ข้อความที่ยาวขึ้น 10 เท่าต้องการการคำนวณมากขึ้นประมาณ 100 เท่า

**แนวทาง SSM.** แทนที่จะ "ใส่ใจทุกคำพร้อมกัน" โมเดลผ่านข้อความตามลำดับและรักษา **สถานะหน่วยความจำ** ภายในที่กะทัดรัดซึ่งอัปเดตในแต่ละขั้นตอน ผลลัพธ์คือเวลาและการใช้หน่วยความจำเติบโตแบบเชิงเส้นโดยประมาณตามความยาวข้อความ ในขณะเดียวกัน การฝึกสามารถทำแบบขนาน — ผ่านการแสดงผลแบบ convolution ของ kernel (throughput สูงบนลำดับยาว)<sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-S4-86)</sup>

### หลักการทำงาน

SSM แบบ discrete อธิบายด้วยสมการสถานะและผลลัพธ์:

$x_{t} = Ax_{t - 1} + Bu_{t},\quad y_{t} = Cx_{t} + Du_{t}$

โดยที่ $x_{t}$ คือสถานะหน่วยความจำ, $u_{t}$ คือ input (token), $y_{t}$ คือ output ใน SSM แบบลึก matrix $A,B,C,D$ ถูก parameterize เพื่อให้มีเสถียรภาพและการคำนวณที่มีประสิทธิภาพบนลำดับยาว เลเยอร์เดียวกันสามารถมองได้ว่าเป็น:

- recurrent (การสแกนทีละขั้นตอน) — inference ที่ประหยัดหน่วยความจำโดยไม่มี KV‑cache;
- convolution — การฝึกแบบขนานด้วย kernel ที่คำนวณล่วงหน้า<sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-S4-86)</sup>

### สถาปัตยกรรมหลักและ hybrid

- **S4 (Structured State Spaces).** Baseline ของ SSM พร้อม parameterization ที่มั่นคงของ matrix สถานะ; แสดงให้เห็นประสิทธิภาพบนลำดับที่ยาวมาก<sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-S4-86)</sup>
- **Mamba.** SSM แบบ *selective*: กฎการอัปเดตหน่วยความจำขึ้นอยู่กับ input ปัจจุบัน (โมเดลตัดสินใจเองว่าจะ "เก็บไว้ในหน่วยความจำ" หรือ "ลืม") การ implement มุ่งเน้นที่ลำดับชั้นหน่วยความจำ GPU; ตามข้อมูลของผู้เขียน บรรลุการเพิ่ม throughput inference หลายเท่าด้วยความซับซ้อนเชิงเส้นตามความยาว<sup>[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Mamba-87)</sup>
- **RetNet.** กลไก *retention* สามโหมด: การฝึกแบบขนาน, การ inference แบบ recurrent และ block‑recurrent ด้วยวัตถุประสงค์เพื่อรวมการฝึกที่รวดเร็ว (เช่น transformer) กับ throughput ที่ประหยัดในการ inference (O(1) หน่วยความจำต่อ token)<sup>[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-RetNet-88)</sup>
- **Hybrid Attention+SSM.** ตัวอย่าง — **Jamba** (การสลับเลเยอร์ Transformer และ Mamba บวก MoE): รายงานการรองรับบริบทประมาณ ~256K token ด้วยความต้องการหน่วยความจำที่น้อยกว่าอย่างมีนัยสำคัญเมื่อเทียบกับโมเดลที่เป็น transformer ล้วน ๆ ในคลาสที่ใกล้เคียงกัน<sup>[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Jamba-89)</sup>

#### ข้อดี

- ความซับซ้อนเชิงเส้นและการประหยัดหน่วยความจำในการ inference ไม่มี global self‑attention และ KV‑cache; เก็บเพียงสถานะที่กะทัดรัด<sup>[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Mamba-87)[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-RetNet-88)</sup>
- การฝึกแบบขนานบนลำดับยาว โหมด convolution เพิ่ม throughput การฝึก<sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-S4-86)</sup>
- ประสิทธิภาพด้านฮาร์ดแวร์ การ implement มุ่งเน้นที่ลำดับชั้นหน่วยความจำสมัยใหม่ (HBM/SRAM)<sup>[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Mamba-87)</sup>
- บริบทยาวและการ streaming Hybrid SSM+Attention ใช้งานได้จริงสำหรับหลายแสน token ด้วยทรัพยากรปานกลาง<sup>[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Jamba-89)</sup>

#### ข้อจำกัดและการปฏิบัติในปัจจุบัน

- ความเติบโตของระบบนิเวศ เครื่องมือและ "สูตร" การปรับขนาด (คำสั่ง, RLHF/DPO) ยังด้อยกว่า transformer stack<sup>[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Mamba-87)</sup>
- คุณภาพและความเสถียร ในบางงาน hybrid (Attention+SSM) แสดงการประนีประนอม "คุณภาพ/ความเร็ว/หน่วยความจำ" ที่เสถียรกว่า SSM "บริสุทธิ์"<sup>[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Jamba-89)</sup>

#### การเปรียบเทียบแนวทาง (โดยสรุป)

| ลักษณะ                          | Transformer                | SSM                       | Hybrid (Attention+SSM) |
|--------------------------------|----------------------------|---------------------------|------------------------|
| ความซับซ้อนตามความยาว            | Quadratic (self‑attention) | เชิงเส้น (สแกน/convolution) | ใกล้เคียงเชิงเส้น          |
| หน่วยความจำต่อ token (inference) | KV‑cache เติบโตตามบริบท      | สถานะ O(1)                | การเติบโตปานกลาง        |
| บริบทยาว                        | ต้องการการเพิ่มประสิทธิภาพพิเศษ  | รองรับโดยธรรมชาติ           | ใช้งานได้จริงถึง ~256K     |
| ความเติบโตของระบบนิเวศ           | สูง                         | กำลังพัฒนา                  | กำลังพัฒนา               |

#### การประยุกต์ใช้งานจริง

- การวิเคราะห์เอกสารยาวมาก (หนังสือ รายงาน บทวิจารณ์ทางวิทยาศาสตร์)
- การประมวลผลแบบ streaming และสถานการณ์แชทที่มีประวัติยาวโดยไม่เพิ่มต้นทุนหน่วยความจำ
- สภาพแวดล้อมที่มีทรัพยากรจำกัด (อุปกรณ์มือถือ/edge)
- อนุกรมเวลาและข้อมูลแบบลำดับอื่น ๆ

**โมเดลตัวอย่าง:** S4, Mamba, RetNet; Hybrid Attention+SSM (Jamba)<sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-S4-86)[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Mamba-87)[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-RetNet-88)[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Jamba-89)</sup>

## วิวัฒนาการของสถาปัตยกรรม

- 2017 — ตีพิมพ์บทความ \\Attention Is All You Need\\ เสนอสถาปัตยกรรม transformer: multi-head self-attention และ positional encoding ช่วยให้ฝึกโมเดลได้โดยไม่ใช้ recurrence และ convolution; ในขณะเดียวกัน attention มีความซับซ้อน quadratic ตามความยาวบริบท<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-Vaswani2017-1)</sup>

<!-- -->

- 2018 — เสนอ GPT‑1 และ BERT GPT‑1 ใช้กลุ่ม decoder เท่านั้นที่มี causal attention สำหรับการสร้างและ fine-tuning ต่อมา; BERT นำเสนอ encoder สองทิศทางและการฝึกเบื้องต้น MLM สำหรับงานด้านความเข้าใจข้อความ<sup>[\[90\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-90)[\[91\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-91)</sup>

<!-- -->

- 2019 — เสนอวิธีทำงานกับลำดับยาวและปรับขนาด decoder‑only Transformer‑XL เพิ่ม "หน่วยความจำ" และตำแหน่งสัมพัทธ์เพื่อขยายออกไปจากหน้าต่างคงที่; GPT‑2 แสดงการเพิ่มขึ้นของความสามารถ zero‑shot เมื่อปรับขนาด; BART แสดงประสิทธิภาพของการฝึก denoising สำหรับ seq2seq<sup>[\[92\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-92)[\[93\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-93)[\[94\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-94)</sup>

<!-- -->

- 2020 — ทำให้รูปแบบ "text‑to‑text" เป็นมาตรฐานและแสดงวิธีการสำหรับเอกสารยาว T5 กำหนดแนวทาง encoder–decoder แบบรวมสำหรับงานต่าง ๆ; Longformer และ BigBird ใช้ attention แบบ sparse/structured สำหรับข้อความยาว; GPT‑3 ยืนยันประสิทธิภาพของการปรับขนาด dense decoder‑only<sup>[\[95\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-95)[\[96\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-96)[\[97\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-97)[\[98\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-98)</sup>

<!-- -->

- 2021 — ปรับปรุงการแสดงตำแหน่งและแสดง sparsity ของพารามิเตอร์ (MoE) RoPE และ ALiBi ปรับปรุงการ generalize บนความยาวที่มากขึ้น; Switch Transformer และ GLaM เปิดใช้งานเฉพาะ expert บางส่วนต่อ token เพิ่มความจุโดยไม่เพิ่มราคา inference แบบสัดส่วน<sup>[\[99\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-99)[\[100\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-100)[\[101\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-101)[\[102\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-102)</sup>

<!-- -->

- 2022 — ระบุโหมด compute‑optimal และเร่ง inference บน prompt ยาว Chinchilla แสดงประโยชน์ของ token การฝึกที่มากขึ้นด้วยขนาดโมเดลปานกลาง; PaLM ด้วย Multi‑Query Attention ลดขนาด KV‑cache; FlashAttention เร่ง attention บน GPU<sup>[\[103\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-103)[\[104\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-104)[\[105\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-105)[\[106\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-106)</sup>

<!-- -->

- 2023 — ขยายหน้าต่างบริบทโดยไม่เปลี่ยนเลเยอร์และปรับปรุงการให้บริการ LLaMA ยืนยันแนวปฏิบัติ (RMSNorm, SwiGLU, RoPE); Position Interpolation และ YaRN ขยายบริบท; vLLM/PagedAttention จัดการ KV‑cache ได้อย่างมีประสิทธิภาพมากขึ้น<sup>[\[107\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-107)[\[108\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-108)[\[109\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-109)[\[110\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-110)[\[111\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-111)[\[112\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-112)</sup>

<!-- -->

- 2023 — GPT‑4 และ Gemini แสดงการประมวลผลและการสร้างในหลาย modality ภายในตระกูลโมเดลเดียว<sup>[\[113\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-113)[\[114\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-114)</sup>

<!-- -->

- 2023 — เสนอโมเดลกับปริภูมิสถานะ (SSM) Mamba และ RetNet กลับมาใช้การประมวลผลตามลำดับด้วยสถานะที่กะทัดรัดแทน KV‑cache และวางรากฐานสำหรับสถาปัตยกรรม hybrid<sup>[\[115\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-115)[\[116\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-116)</sup>

<!-- -->

- 2024 — ตีพิมพ์โมเดล MoE แบบเปิดและ hybrid Attention+SSM; เร่ง attention บน GPU ใหม่ Mixtral 8×7B/8×22B และ DBRX ยืนยันความใช้ได้จริงของ MoE; Jamba รวม Transformer และ Mamba สำหรับบริบทยาวมาก; FlashAttention‑3 เพิ่ม throughput<sup>[\[117\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-117)[\[118\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-118)[\[119\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-119)[\[120\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-120)[\[121\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_note-121)</sup>

## ลิงก์

- <a href="https://jalammar.github.io/illustrated-transformer/" class="external free" rel="nofollow">https://jalammar.github.io/illustrated-transformer/</a> The Illustrated Transformer — คำอธิบายเชิงภาพ

## บรรณานุกรม

- Vaswani, A. et al. (2017). *Attention Is All You Need*. NIPS. <a href="https://arxiv.org/abs/1706.03762" class="external free" rel="nofollow">https://arxiv.org/abs/1706.03762</a>
- Devlin, J. et al. (2019). *BERT*. NAACL. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a>
- Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. NeurIPS. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a>
- Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer (T5)*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a>
- Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a>
- Touvron, H. et al. (2023). *LLaMA*. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a>
- Chowdhery, A. et al. (2022). *PaLM: Scaling Language Modeling with Pathways*. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a>
- Dao, T. et al. (2022–2024). *FlashAttention (1/2/3)*. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a> ; <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a> ; <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a>
- Shazeer, N. (2019). *MQA*. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a>
- Ainslie, J. et al. (2023). *GQA*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a>
- Kwon, W. et al. (2023). *PagedAttention / vLLM*. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a>
- Leviathan, Y. et al. (2023). *Speculative Decoding*. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a>
- Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). *Switch Transformers*. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a>
- Du, N. et al. (2022). *GLaM*. <a href="https://proceedings.mlr.press/v162/du22c/du22c.pdf" class="external free" rel="nofollow">https://proceedings.mlr.press/v162/du22c/du22c.pdf</a>
- Jiang, A.Q. et al. (2024). *Mixtral of Experts*. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a>
- Databricks (2024). *Introducing DBRX*. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a>
- NVIDIA (2024). *Applying Mixture of Experts in LLM Architectures*. <a href="https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/" class="external free" rel="nofollow">https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/</a>
- Zhou, Y. et al. (2022). *Expert Choice Routing*. <a href="https://arxiv.org/abs/2202.09368" class="external free" rel="nofollow">https://arxiv.org/abs/2202.09368</a>
- Komatsuzaki, A. et al. (2022). *Sparse Upcycling*. <a href="https://arxiv.org/abs/2212.05055" class="external free" rel="nofollow">https://arxiv.org/abs/2212.05055</a>
- Lewis, P. et al. (2020). *RAG*. <a href="https://arxiv.org/abs/2005.11401" class="external free" rel="nofollow">https://arxiv.org/abs/2005.11401</a>
- Beltagy, I. et al. (2020). *Longformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a>
- Zaheer, M. et al. (2020). *BigBird*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a>
- Press, O. et al. (2022). *ALiBi*. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a>
- Su, J. et al. (2021). *RoFormer (RoPE)*. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a>
- Chen, S. et al. (2023). *Position Interpolation*. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a>
- Peng, B. et al. (2023). *YaRN*. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a>
- Dettmers, T. et al. (2023). *QLoRA*. <a href="https://arxiv.org/abs/2305.14314" class="external free" rel="nofollow">https://arxiv.org/abs/2305.14314</a>
- Rajbhandari, S. et al. (2020). *ZeRO*. <a href="https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/" class="external free" rel="nofollow">https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/</a>
- Shoeybi, M. et al. (2019). *Megatron‑LM*. <a href="https://arxiv.org/abs/1909.08053" class="external free" rel="nofollow">https://arxiv.org/abs/1909.08053</a>
- Kaplan, J. et al. (2020). *Scaling Laws*. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a>
- Hoffmann, J. et al. (2022). *Chinchilla / Compute‑Optimal*. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a>
- Gemini Team (2023). *Gemini*. <a href="https://arxiv.org/abs/2312.11805" class="external free" rel="nofollow">https://arxiv.org/abs/2312.11805</a>
- Bai, Y. et al. (2022). *Constitutional AI*. <a href="https://arxiv.org/abs/2212.08073" class="external free" rel="nofollow">https://arxiv.org/abs/2212.08073</a>
- OpenAI (2023). *GPT‑4 Technical Report*. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a>
- OpenAI (2023). *DevDay: GPT‑4 Turbo 128k*. <a href="https://openai.com/index/new-models-and-developer-products-announced-at-devday/" class="external free" rel="nofollow">https://openai.com/index/new-models-and-developer-products-announced-at-devday/</a>
- Zhang, B.; Sennrich, R. (2019). *RMSNorm*. <a href="https://arxiv.org/abs/1910.07467" class="external free" rel="nofollow">https://arxiv.org/abs/1910.07467</a>
- Shazeer, N. (2020). *GLU Variants / SwiGLU*. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a>
- Gu, A.; Goel, K.; Ré, C. (2021). *S4: Structured State Spaces*. <a href="https://arxiv.org/abs/2111.00396" class="external free" rel="nofollow">https://arxiv.org/abs/2111.00396</a>
- Gu, A.; Dao, T. (2023/2024). *Mamba: Selective State Spaces*. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a>
- Sun, Y. et al. (2023). *RetNet*. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a>
- Lieber, O. et al. (2024). *Jamba: Hybrid Transformer‑Mamba*. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a>
- Dai, Z. et al. (2019). *Transformer‑XL*. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a>
- Kitaev, N.; Kaiser, L.; Levskaya, A. (2020). *Reformer*. <a href="https://arxiv.org/abs/2001.04451" class="external free" rel="nofollow">https://arxiv.org/abs/2001.04451</a>
- Choromanski, K. et al. (2021). *Performer*. <a href="https://arxiv.org/abs/2009.14794" class="external free" rel="nofollow">https://arxiv.org/abs/2009.14794</a>
- Wang, S. et al. (2020). *Linformer*. <a href="https://arxiv.org/abs/2006.04768" class="external free" rel="nofollow">https://arxiv.org/abs/2006.04768</a>

## หมายเหตุ

1.  <span id="cite_note-Vaswani2017-1">↑ <sup>[1.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Vaswani2017_1-0)</sup> <sup>[1.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Vaswani2017_1-1)</sup> <sup>[1.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Vaswani2017_1-2)</sup> <sup>[1.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Vaswani2017_1-3)</sup> <sup>[1.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Vaswani2017_1-4)</sup> <sup>[1.5](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Vaswani2017_1-5)</sup> <sup>[1.6](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Vaswani2017_1-6)</sup> Vaswani, A. et al. (2017). *Attention Is All You Need*. <a href="https://arxiv.org/abs/1706.03762" class="external free" rel="nofollow">https://arxiv.org/abs/1706.03762</a></span>
2.  <span id="cite_note-2">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-2) Devlin, J. et al. (2019). *BERT*. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
3.  <span id="cite_note-3">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-3) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
4.  <span id="cite_note-4">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-4) Raffel, C. et al. (2020). *T5*. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
5.  <span id="cite_note-5">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-5) Devlin, J. et al. (2019). *BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding*. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
6.  <span id="cite_note-6">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-6) Liu, Y. et al. (2019). *RoBERTa: A Robustly Optimized BERT Pretraining Approach*. <a href="https://arxiv.org/abs/1907.11692" class="external free" rel="nofollow">https://arxiv.org/abs/1907.11692</a></span>
7.  <span id="cite_note-7">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-7) He, P. et al. (2021). *DeBERTa: Decoding‑enhanced BERT with Disentangled Attention*. <a href="https://arxiv.org/abs/2006.03654" class="external free" rel="nofollow">https://arxiv.org/abs/2006.03654</a></span>
8.  <span id="cite_note-8">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-8) Clark, K. et al. (2020). *ELECTRA: Pre‑training Text Encoders as Discriminators Rather Than Generators*. <a href="https://arxiv.org/abs/2003.10555" class="external free" rel="nofollow">https://arxiv.org/abs/2003.10555</a></span>
9.  <span id="cite_note-9">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-9) Zaheer, M. et al. (2020). *Big Bird: Transformers for Longer Sequences*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
10. <span id="cite_note-10">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-10) Beltagy, I. et al. (2020). *Longformer: The Long‑Document Transformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
11. <span id="cite_note-11">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-11) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need* (Multi‑Query Attention). <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
12. <span id="cite_note-12">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-12) Ainslie, J. et al. (2023). *GQA: Training Generalized Multi‑Query Transformer Models from Multi‑Head Checkpoints*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
13. <span id="cite_note-13">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-13) Leviathan, Y. et al. (2023). *Fast Inference from Transformers via Speculative Decoding*. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
14. <span id="cite_note-14">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-14) Kwon, W. et al. (2023). *Efficient Memory Management for LLM Serving with PagedAttention (vLLM)*. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
15. <span id="cite_note-15">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-15) vLLM Docs (2024–2025). *Continuous batching, Chunked prefill, Structured outputs*. <a href="https://docs.vllm.ai/" class="external free" rel="nofollow">https://docs.vllm.ai/</a></span>
16. <span id="cite_note-16">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-16) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
17. <span id="cite_note-17">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-17) Ouyang, L. et al. (2022). *InstructGPT (RLHF)*. <a href="https://arxiv.org/abs/2203.02155" class="external free" rel="nofollow">https://arxiv.org/abs/2203.02155</a></span>
18. <span id="cite_note-18">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-18) Rafailov, R. et al. (2023). *Direct Preference Optimization*. <a href="https://arxiv.org/abs/2305.18290" class="external free" rel="nofollow">https://arxiv.org/abs/2305.18290</a></span>
19. <span id="cite_note-19">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-19) Shazeer, 2019. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
20. <span id="cite_note-20">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-20) Ainslie, 2023. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
21. <span id="cite_note-21">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-21) Leviathan, 2023. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
22. <span id="cite_note-22">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-22) Kwon, 2023. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
23. <span id="cite_note-23">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-23) vLLM Docs. <a href="https://docs.vllm.ai/" class="external free" rel="nofollow">https://docs.vllm.ai/</a></span>
24. <span id="cite_note-24">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-24) OpenAI (2024). *Structured Outputs*. <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/" class="external free" rel="nofollow">https://openai.com/index/introducing-structured-outputs-in-the-api/</a></span>
25. <span id="cite_note-25">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-25) vLLM Docs — Structured outputs. <a href="https://docs.vllm.ai/en/v0.9.2/features/structured_outputs.html" class="external free" rel="nofollow">https://docs.vllm.ai/en/v0.9.2/features/structured_outputs.html</a></span>
26. <span id="cite_note-26">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-26) Kwon, 2023. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
27. <span id="cite_note-27">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-27) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
28. <span id="cite_note-28">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-28) Touvron, H. et al. (2023). *LLaMA: Open and Efficient Foundation Language Models*. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
29. <span id="cite_note-29">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-29) Achiam, J. et al. (2023). *GPT‑4 Technical Report*. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a></span>
30. <span id="cite_note-30">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-30) Meta AI (2024). *Introducing Meta Llama 3*. <a href="https://ai.meta.com/blog/meta-llama-3/" class="external free" rel="nofollow">https://ai.meta.com/blog/meta-llama-3/</a></span>
31. <span id="cite_note-31">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-31) Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer (T5)*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
32. <span id="cite_note-32">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-32) Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
33. <span id="cite_note-33">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-33) Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
34. <span id="cite_note-34">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-34) Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training for NLG, Translation, and Comprehension*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
35. <span id="cite_note-35">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-35) Chung, H. W. et al. (2022). *Scaling Instruction‑Finetuned Language Models (FLAN‑T5)*. <a href="https://arxiv.org/abs/2210.11416" class="external free" rel="nofollow">https://arxiv.org/abs/2210.11416</a></span>
36. <span id="cite_note-36">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-36) Shazeer, N. (2020). GLU Variants Improve Transformer. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a></span>
37. <span id="cite_note-37">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-37) Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
38. <span id="cite_note-38">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-38) Brown, T. et al. (2020). Language Models are Few‑Shot Learners. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
39. <span id="cite_note-39">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-39) Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
40. <span id="cite_note-40">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-40) Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a></span>
41. <span id="cite_note-41">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-41) Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a></span>
42. <span id="cite_note-42">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-42) Hoffmann, J. et al. (2022). Training Compute‑Optimal Large Language Models. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
43. <span id="cite_note-43">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-43) Chen, S. et al. (2023). Extending Context Window via Positional Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
44. <span id="cite_note-44">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-44) Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of LLMs. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
45. <span id="cite_note-45">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-45) Dao, T. et al. (2022–2024). FlashAttention (1/2/3). <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a> ; <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a> ; <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>
46. <span id="cite_note-46">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-46) Shazeer, N. (2019). Fast Transformer Decoding: One Write‑Head is All You Need. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
47. <span id="cite_note-47">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-47) Ainslie, J. et al. (2023). GQA. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
48. <span id="cite_note-48">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-48) Kwon, W. et al. (2023). Efficient Memory Management for LLM Serving with PagedAttention. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
49. <span id="cite_note-49">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-49) Leviathan, Y. et al. (2023). Fast Inference from Transformers via Speculative Decoding. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
50. <span id="cite_note-Switch-50">↑ <sup>[50.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Switch_50-0)</sup> <sup>[50.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Switch_50-1)</sup> <sup>[50.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Switch_50-2)</sup> Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). *Switch Transformers*. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a></span>
51. <span id="cite_note-GLAM-51">↑ <sup>[51.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-GLAM_51-0)</sup> <sup>[51.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-GLAM_51-1)</sup> <sup>[51.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-GLAM_51-2)</sup> Du, N. et al. (2021). *GLaM: Efficient Scaling of Language Models with Mixture‑of‑Experts*. <a href="https://arxiv.org/pdf/2112.06905.pdf" class="external free" rel="nofollow">https://arxiv.org/pdf/2112.06905.pdf</a></span>
52. <span id="cite_note-Mixtral8x7-52">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Mixtral8x7_52-0) Mistral AI (2023). *Mixtral of Experts*. <a href="https://mistral.ai/news/mixtral-of-experts/" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-of-experts/</a></span>
53. <span id="cite_note-Mixtral8x7_paper-53">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Mixtral8x7_paper_53-0) Jiang, A.Q. et al. (2024). *Mixtral of Experts*. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a></span>
54. <span id="cite_note-Mixtral8x22-54">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Mixtral8x22_54-0) Mistral AI (2024). *Mixtral 8x22B*. <a href="https://mistral.ai/news/mixtral-8x22b" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-8x22b</a></span>
55. <span id="cite_note-DBRX-55">↑ <sup>[55.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-DBRX_55-0)</sup> <sup>[55.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-DBRX_55-1)</sup> Databricks (2024). *Introducing DBRX*. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a></span>
56. <span id="cite_note-NVIDIA_MoE-56">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-NVIDIA_MoE_56-0) NVIDIA (2024). *Applying Mixture of Experts in LLM Architectures*. <a href="https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/" class="external free" rel="nofollow">https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/</a></span>
57. <span id="cite_note-ExpertChoice-57">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-ExpertChoice_57-0) Zhou, Y. et al. (2022). *Mixture‑of‑Experts with Expert Choice Routing*. <a href="https://arxiv.org/abs/2202.09368" class="external free" rel="nofollow">https://arxiv.org/abs/2202.09368</a></span>
58. <span id="cite_note-SparseUpcycling-58">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-SparseUpcycling_58-0) Komatsuzaki, A. et al. (2022). *Sparse Upcycling: Training Mixture‑of‑Experts from Dense Checkpoints*. <a href="https://arxiv.org/abs/2212.05055" class="external free" rel="nofollow">https://arxiv.org/abs/2212.05055</a></span>
59. <span id="cite_note-RAG-59">↑ <sup>[59.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-RAG_59-0)</sup> <sup>[59.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-RAG_59-1)</sup> <sup>[59.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-RAG_59-2)</sup> <sup>[59.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-RAG_59-3)</sup> Lewis, P. et al. (2020). *Retrieval‑Augmented Generation for Knowledge‑Intensive NLP Tasks*. <a href="https://arxiv.org/abs/2005.11401" class="external free" rel="nofollow">https://arxiv.org/abs/2005.11401</a></span>
60. <span id="cite_note-60">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-60) NVIDIA Blog (2025). *What is Retrieval‑Augmented Generation (RAG)*. <a href="https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/" class="external free" rel="nofollow">https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/</a></span>
61. <span id="cite_note-61">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-61) Beltagy, I. et al. (2020). *Longformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
62. <span id="cite_note-62">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-62) Zaheer, M. et al. (2020). *Big Bird*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
63. <span id="cite_note-63">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-63) Dao, T. et al. (2022). *FlashAttention*. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a></span>
64. <span id="cite_note-64">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-64) Dao, T. et al. (2023). *FlashAttention‑2*. <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a></span>
65. <span id="cite_note-65">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-65) Shah, M. et al. (2024). *FlashAttention‑3*. <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>
66. <span id="cite_note-66">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-66) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need*. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
67. <span id="cite_note-67">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-67) Ainslie, J. et al. (2023). *GQA*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
68. <span id="cite_note-68">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-68) Press, O. et al. (2022). ALiBi. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a></span>
69. <span id="cite_note-69">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-69) Su, J. et al. (2021). RoFormer: Rotary Position Embedding. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a></span>
70. <span id="cite_note-70">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-70) Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
71. <span id="cite_note-71">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-71) Chen, S. et al. (2023). Extending Context Window via Positional Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
72. <span id="cite_note-72">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-72) Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of LLMs. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
73. <span id="cite_note-73">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-73) Dai, Z. et al. (2019). *Transformer‑XL: Attentive Language Models Beyond a Fixed‑Length Context*. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a></span>
74. <span id="cite_note-74">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-74) Kitaev, N.; Kaiser, L.; Levskaya, A. (2020). *Reformer: The Efficient Transformer*. <a href="https://arxiv.org/abs/2001.04451" class="external free" rel="nofollow">https://arxiv.org/abs/2001.04451</a></span>
75. <span id="cite_note-75">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-75) Choromanski, K. et al. (2021). *Rethinking Attention with Performers*. <a href="https://arxiv.org/abs/2009.14794" class="external free" rel="nofollow">https://arxiv.org/abs/2009.14794</a></span>
76. <span id="cite_note-76">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-76) Wang, S. et al. (2020). *Linformer: Self‑Attention with Linear Complexity*. <a href="https://arxiv.org/abs/2006.04768" class="external free" rel="nofollow">https://arxiv.org/abs/2006.04768</a></span>
77. <span id="cite_note-77">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-77) Dettmers, T. et al. (2023). *QLoRA: Efficient Finetuning of Quantized LLMs*. <a href="https://arxiv.org/abs/2305.14314" class="external free" rel="nofollow">https://arxiv.org/abs/2305.14314</a></span>
78. <span id="cite_note-78">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-78) Hinton, G. et al. (2015). *Distilling the Knowledge in a Neural Network*. <a href="https://arxiv.org/abs/1503.02531" class="external free" rel="nofollow">https://arxiv.org/abs/1503.02531</a></span>
79. <span id="cite_note-79">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-79) Sanh, V. et al. (2019). *DistilBERT*. <a href="https://arxiv.org/abs/1910.01108" class="external free" rel="nofollow">https://arxiv.org/abs/1910.01108</a></span>
80. <span id="cite_note-80">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-80) Rajbhandari, S. et al. (2020). *ZeRO: Memory Optimizations Toward Training Trillion‑Parameter Models*. <a href="https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/" class="external free" rel="nofollow">https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/</a></span>
81. <span id="cite_note-81">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-81) Shoeybi, M. et al. (2019). *Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism*. <a href="https://arxiv.org/abs/1909.08053" class="external free" rel="nofollow">https://arxiv.org/abs/1909.08053</a></span>
82. <span id="cite_note-82">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-82) Hugging Face. *Transformers Documentation*. <a href="https://huggingface.co/docs/transformers" class="external free" rel="nofollow">https://huggingface.co/docs/transformers</a></span>
83. <span id="cite_note-83">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-83) Hugging Face. *Accelerate Documentation*. <a href="https://huggingface.co/docs/accelerate" class="external free" rel="nofollow">https://huggingface.co/docs/accelerate</a></span>
84. <span id="cite_note-84">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-84) Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a></span>
85. <span id="cite_note-85">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-85) Hoffmann, J. et al. (2022). Training Compute‑Optimal Large Language Models. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
86. <span id="cite_note-S4-86">↑ <sup>[86.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-S4_86-0)</sup> <sup>[86.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-S4_86-1)</sup> <sup>[86.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-S4_86-2)</sup> <sup>[86.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-S4_86-3)</sup> <sup>[86.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-S4_86-4)</sup> Gu, A.; Goel, K.; Ré, C. (2021). *Efficiently Modeling Long Sequences with Structured State Spaces (S4)*. <a href="https://arxiv.org/abs/2111.00396" class="external free" rel="nofollow">https://arxiv.org/abs/2111.00396</a></span>
87. <span id="cite_note-Mamba-87">↑ <sup>[87.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Mamba_87-0)</sup> <sup>[87.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Mamba_87-1)</sup> <sup>[87.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Mamba_87-2)</sup> <sup>[87.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Mamba_87-3)</sup> <sup>[87.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Mamba_87-4)</sup> Gu, A.; Dao, T. (2023/2024). *Mamba: Linear‑Time Sequence Modeling with Selective State Spaces*. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a></span>
88. <span id="cite_note-RetNet-88">↑ <sup>[88.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-RetNet_88-0)</sup> <sup>[88.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-RetNet_88-1)</sup> <sup>[88.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-RetNet_88-2)</sup> Sun, Y. et al. (2023). *Retentive Network: A Successor to Transformer for Large Language Models*. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a></span>
89. <span id="cite_note-Jamba-89">↑ <sup>[89.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Jamba_89-0)</sup> <sup>[89.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Jamba_89-1)</sup> <sup>[89.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Jamba_89-2)</sup> <sup>[89.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-Jamba_89-3)</sup> Lieber, O. et al. (2024). *Jamba: A Hybrid Transformer‑Mamba Language Model*. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a></span>
90. <span id="cite_note-90">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-90) Radford, A. et al. (2018). Improving Language Understanding by Generative Pre‑Training. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" class="external free" rel="nofollow">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a></span>
91. <span id="cite_note-91">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-91) Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
92. <span id="cite_note-92">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-92) Dai, Z. et al. (2019). Transformer‑XL. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a></span>
93. <span id="cite_note-93">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-93) Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners. <a href="https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" class="external free" rel="nofollow">https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf</a></span>
94. <span id="cite_note-94">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-94) Lewis, M. et al. (2019). BART. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
95. <span id="cite_note-95">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-95) Raffel, C. et al. (2020). T5. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
96. <span id="cite_note-96">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-96) Beltagy, I. et al. (2020). Longformer. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
97. <span id="cite_note-97">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-97) Zaheer, M. et al. (2020). BigBird. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
98. <span id="cite_note-98">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-98) Brown, T. et al. (2020). Language Models are Few‑Shot Learners. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
99. <span id="cite_note-99">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-99) Su, J. et al. (2021). RoPE. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a></span>
100. <span id="cite_note-100">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-100) Press, O. et al. (2021/2022). ALiBi. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a></span>
101. <span id="cite_note-101">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-101) Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). Switch Transformers. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a></span>
102. <span id="cite_note-102">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-102) Du, N. et al. (2021). GLaM. <a href="https://arxiv.org/pdf/2112.06905.pdf" class="external free" rel="nofollow">https://arxiv.org/pdf/2112.06905.pdf</a></span>
103. <span id="cite_note-103">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-103) Hoffmann, J. et al. (2022). Chinchilla. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
104. <span id="cite_note-104">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-104) Chowdhery, A. et al. (2022). PaLM. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a></span>
105. <span id="cite_note-105">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-105) Shazeer, N. (2019). Fast Transformer Decoding. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
106. <span id="cite_note-106">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-106) Dao, T. et al. (2022). FlashAttention. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a></span>
107. <span id="cite_note-107">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-107) Touvron, H. et al. (2023). LLaMA. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
108. <span id="cite_note-108">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-108) Zhang, B.; Sennrich, R. (2019). RMSNorm. <a href="https://arxiv.org/abs/1910.07467" class="external free" rel="nofollow">https://arxiv.org/abs/1910.07467</a></span>
109. <span id="cite_note-109">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-109) Shazeer, N. (2020). GLU Variants. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a></span>
110. <span id="cite_note-110">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-110) Chen, S. et al. (2023). Position Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
111. <span id="cite_note-111">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-111) Peng, B. et al. (2023). YaRN. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
112. <span id="cite_note-112">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-112) Kwon, W. et al. (2023). vLLM/PagedAttention. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
113. <span id="cite_note-113">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-113) OpenAI (2023). GPT‑4 Technical Report. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a></span>
114. <span id="cite_note-114">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-114) Gemini Team (2023). Gemini. <a href="https://arxiv.org/abs/2312.11805" class="external free" rel="nofollow">https://arxiv.org/abs/2312.11805</a></span>
115. <span id="cite_note-115">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-115) Gu, A.; Dao, T. (2023). Mamba. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a></span>
116. <span id="cite_note-116">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-116) Sun, Y. et al. (2023). RetNet. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a></span>
117. <span id="cite_note-117">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-117) Jiang, A.Q. et al. (2024). Mixtral of Experts. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a></span>
118. <span id="cite_note-118">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-118) Mistral AI (2024). Mixtral 8x22B. <a href="https://mistral.ai/news/mixtral-8x22b" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-8x22b</a></span>
119. <span id="cite_note-119">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-119) Databricks (2024). Introducing DBRX. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a></span>
120. <span id="cite_note-120">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-120) Lieber, O. et al. (2024). Jamba. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a></span>
121. <span id="cite_note-121">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%E0%B8%AA%E0%B8%96%E0%B8%B2%E0%B8%9B%E0%B8%B1%E0%B8%95%E0%B8%A2%E0%B8%81%E0%B8%A3%E0%B8%A3%E0%B8%A1%E0%B8%82%E0%B8%AD%E0%B8%87%E0%B9%82%E0%B8%A1%E0%B9%80%E0%B8%94%E0%B8%A5%E0%B8%A0%E0%B8%B2%E0%B8%A9%E0%B8%B2%E0%B8%82%E0%B8%99%E0%B8%B2%E0%B8%94%E0%B9%83%E0%B8%AB%E0%B8%8D%E0%B9%88#cite_ref-121) Shah, M. et al. (2024). FlashAttention‑3. <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>