---
title: "Large language model architectures — 대형 언어 모델 아키텍처"
source: "https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98"
wiki: "systems-analysis.info/int"
article: "Large_language_model_architectures_—_대형_언어_모델_아키텍처"
language: "ko"
categories:
  - "Category:Korean"
  - "Category:Large language models"
  - "Category:Machine learning"
revision_id: 3792
wiki_created_at: 2026-09-06T23:26:12Z
wiki_modified_at: 2026-09-06T23:26:12Z
downloaded_at: 2026-09-07T22:59:06Z
---

# Large language model architectures — 대형 언어 모델 아키텍처

**대형 언어 모델(LLM) 아키텍처**란 대형 언어 모델이 어떻게 구성되고, 학습되며, 작동하는지를 결정하는 근본적인 원리와 구조를 말한다. 현재 인간의 언어를 이해하고 생성할 수 있는 LLM은 거의 전적으로 **Transformer** 아키텍처<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Vaswani2017-1)</sup>를 기반으로 하지만, 효율성·확장성·기능 향상을 목적으로 한 수많은 개선 사항과 다양한 접근 방식을 포함하고 있다.

## LLM 아키텍처 패밀리 (트랜스포머)

현대의 대형 언어 모델은 트랜스포머 아키텍처<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Vaswani2017-1)</sup>를 기반으로 하지만, 목적에 따라—텍스트 이해, 이어쓰기 생성, 혹은 한 텍스트를 다른 텍스트로 변환—각기 다르게 활용한다. 실용적으로는 트랜스포머의 기본 원리를 유지하면서 세 가지 패밀리로 구분된다<sup>[\[2\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-2)[\[3\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-3)[\[4\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-4)</sup>.

### 1. Encoder‑only (인코더만 사용)

모델은 인코더 스택만 사용하며 전체 입력 텍스트를 양방향으로 처리한다. 사전학습은 일반적으로 masked‑language modeling(MLM, 마스킹 언어 모델링) 방식으로 구성된다: 일부 토큰을 숨기고 모델이 주변 맥락을 토대로 이를 복원하도록 학습한다. 양방향 맥락 덕분에 이러한 모델은 이해 및 스코어링 작업—분류, 개체명 인식, 문서 재순위화, 추출형 QA—에 강하다. 그러나 자기회귀적 생성('처음부터' 생성)에는 적합하지 않다.

실용적으로는 encoder‑only 패밀리에 대한 대안적 사전학습 목표도 적용된다: *ELECTRA의 replaced token detection(RTD)*(판별자 모델이 교체된 토큰을 식별) 및 의미 검색/리트리버를 위한 *바이인코더의 대조 학습*(Dense Passage Retrieval에서처럼 '쿼리-문서' 쌍에 대한 InfoNCE/softmax-loss). RAG에서 사용될 때 encoder‑only 모델은 *바이인코더*(빠른 ANN 검색을 위해 쿼리와 문서를 별도로 인코딩)나 *크로스인코더*(정밀한 재순위화를 위해 쌍을 함께 인코딩) 역할을 한다.

**장점:**

- 양방향 맥락 덕분에 높은 텍스트 이해 품질: 분류, NER, 사실 추출, 재순위화, 추출형 QA.
- 병렬 처리 및 높은 처리량: 자기회귀 없이 단일 순전파 통과; 대량 스코어링 배치 처리에 적합.
- 검색 및 RAG와의 자연스러운 통합: 바이인코더 역할로 빠른 의미 검색, 크로스인코더 역할로 정밀 재순위화.
- 효율적인 적응: 목표 파인튜닝 후 상대적으로 소형 변형(≈1억~3억 파라미터; BERT‑base ≈1억1천만)이 높은 품질을 달성.
- 생성 응답 길이에 무관한 안정적인 지연 시간(단계적 디코딩 없음); 대규모 컬렉션의 오프라인 스코어링에 적합.
- 상대/회전 위치 표현 및/또는 국소 희소 어텐션(예: Longformer/BigBird)을 통해 인코더의 맥락 창 확장 가능, 긴 문서에 유용.

**단점:**

- 자체 생성 능력 없음: 대화나 장문 응답에는 디코더 또는 외부 생성 모듈이 필요.
- 대화형 시나리오에서의 한계: 상태 유지 단계별 생성 없음.
- 사전학습 목표와 자유 생성 작업 간 불일치: MLM은 인과적 모델링에 비해 생성과 잘 맞지 않음.
- 역사적으로 제한된 맥락 창(절대 위치의 기본 설정에서는 종종 512 토큰); 확장에는 특수한 위치/어텐션 방식 및/또는 파인튜닝이 필요.
- 검색 작업에는 바이인코더 및/또는 크로스인코더의 별도 대조 파인튜닝이 필요; 이 없이는 검색/재순위화 품질이 전용 학습 모델보다 낮은 경우가 많음.

**대표 모델:** BERT 및 파생 모델, RoBERTa, DeBERTa(확장된 encoder‑only 변형); 대안적 사전학습 목표로는 ELECTRA(RTD). <sup>[\[5\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-5)[\[6\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-6)[\[7\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-7)[\[8\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-8)[\[9\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-9)[\[10\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-10)</sup>

### 2. Decoder‑only (디코더만 사용)

인과적(왼쪽 방향) 어텐션을 사용하는 디코더 스택만 활용: 모델은 이미 주어진 접두사를 기반으로 다음 토큰을 예측한다. 이러한 학습 방식—causal language modeling(CLM)—은 이 모델들을 생성에 있어 자연스러운 선택으로 만든다: 대화, 장문 응답, 창의적 텍스트, 프로그램 코드. 트레이드오프는 긴 프롬프트에서 지연 시간과 KV 캐시 크기의 증가이다. 실용적으로 decoder‑only에는 다양한 엔지니어링 기법이 널리 적용된다: MQA 및 GQA를 통한 KV 캐시 감소, 추측 디코딩을 통한 추론 가속, 서버 최적화(PagedAttention/vLLM, continuous batching, chunked prefill).<sup>[\[11\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-11)[\[12\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-12)[\[13\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-13)[\[14\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-14)[\[15\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-15)</sup>

**장점:**

- 자연스러운 텍스트 생성(CLM): 강력한 zero‑shot 및 few‑shot 능력; 우수한 확장성.<sup>[\[16\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-16)</sup>
- 범용성: 하나의 모델이 프롬프트 내 지시문과 예시를 통해 다양한 작업을 수행; RAG 및 tool use와 자연스럽게 결합.
- 성숙한 생태계: 지시 파인튜닝 및 행동 정렬 방법론(RLHF, DPO); 오픈 소스 및 상용 구현 이용 가능.<sup>[\[17\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-17)[\[18\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-18)</sup>
- 풍부한 추론 최적화 스택: MQA/GQA로 KV 캐시 크기 감소 및 처리량 향상; 추측 디코딩으로 분포 변경 없이 추론 가속; PagedAttention/vLLM과 continuous batching, chunked prefill로 GPU 전체 활용도 향상.<sup>[\[19\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-19)[\[20\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-20)[\[21\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-21)[\[22\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-22)[\[23\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-23)</sup>
- 엄격한 응답 형식(JSON/SQL/DSL)을 위한 구조화된 생성 지원으로 정보 시스템 및 API와의 통합 용이.<sup>[\[24\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-24)[\[25\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-25)</sup>

**단점:**

- 높은 생성 지연: 순차적 추론; 새 토큰의 비용은 이미 '읽힌' 맥락 길이에 따라 증가(KV 캐시).
- '긴 입력 – 짧은 출력' 프로파일(요약, 번역)에서는 입력을 한 번만 인코딩하는 encoder–decoder에 비해 불리.
- 단방향 맥락 제약: 이해 작업에서 양방향 표현(encoder‑only / encoder–decoder) 모델에 뒤처지는 경우 있음.
- 긴 프롬프트 및 대형 배치에서 KV 캐시용 메모리가 '병목'이 될 수 있음; <sup>[\[26\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-26)</sup>
- 활성화/KV 양자화(INT8/FP8)는 추론을 가속하지만 긴 맥락/코드에서 품질을 저하시킬 수 있음; 신중한 검증 필요(특히 엄격한 SLA 환경).

**대표 모델:** GPT‑3, GPT‑4(아키텍처 및 데이터셋 세부 정보 비공개), LLaMA 및 *Llama 3*(8B/70B, 2024).<sup>[\[27\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-27)[\[28\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-28)[\[29\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-29)[\[30\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-30)</sup>

### 3. Encoder–decoder (인코더-디코더)

이 아키텍처는 두 구성 요소를 결합한다. 인코더는 양방향 모드로 작동하고, 디코더는 인과적으로 작동한다. 인코더는 입력을 한 번 분석하여 표현을 형성하고, 디코더는 cross‑attention을 통해 이 표현을 참조하면서 출력을 생성한다. 이러한 분리된 접근 방식은 긴 입력 텍스트를 짧은 출력으로 변환해야 하는 경우—기계 번역, 요약, 문서 기반 답변—에 특히 유용하다. 이 방법은 더 큰 전체 계산 비용(두 스택과 크로스 어텐션)이 필요하지만, 전체 원문 분석을 기반으로 한 제어된 생성이 장점이며, 인코딩은 한 번만 수행되어 전체 추론 과정에서 재사용된다.

**장점:**

- 조건부 생성: 디코더가 입력 표현에 cross‑attention을 사용. <sup>[\[31\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-31)</sup>
- '긴 입력 → 짧은 출력' 시나리오에서 효율적: 입력이 한 번만 인코딩됨.
- 'text‑to‑text' 형식 및 제어된 출력(작업 접두사, 특수 지시문)에 적합. <sup>[\[32\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-32)</sup>
- 긴 소스에서의 안정성과 효율성: 디코딩 단계에서는 출력에 대한 self‑attention만 증가하고, cross‑attention은 인코더의 고정된 키/값을 재사용(각 단계에서 입력을 '재읽지' 않음).

**단점:**

- 두 스택으로 인해 학습 및 적용 시 메모리 및 계산 요구량 증가.
- 매우 긴 시퀀스에서 전체 지연은 decoder‑only와 비슷; 자기회귀가 병목으로 남음.
- decoder‑only에 비해 범용 채팅 모델이 적음; 주로 특정 작업을 위한 고품질 seq2seq 엔진으로 사용됨.
- 매우 긴 입력에서는 디코더 각 레이어의 cross‑attention 키/값(전체 소스) 메모리가 증가하여 신중한 서빙 계획이 필요.

**대표 모델:** T5(T5 v1.1 포함, *FLAN‑T5*의 지시 파인튜닝 방법론 포함) 및 BART. <sup>[\[33\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-33)[\[34\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-34)[\[35\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-35)</sup>

## 밀집(Dense) 트랜스포머

가장 고전적이고 널리 사용되는 LLM 아키텍처: 각 토큰 처리 시 모델 파라미터의 거의 전체가 참여한다. 희소 접근 방식(예: Mixture‑of‑Experts)과 달리 서브네트워크의 선택적 활성화가 없으며—각 블록이 모든 토큰에 대해 작동한다. <sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Vaswani2017-1)</sup>

### 작동 원리 및 아키텍처

**기본 구조.** 모델은 N개의 동일한 트랜스포머 블록으로 구성된 스택이다. 각 블록은 다음을 포함한다:

1.  **다중 헤드 자기 어텐션(Multi‑Head Self‑Attention).** 각 토큰에 대해 세 벡터가 계산된다: Q(query), K(key), V(value); 어텐션은 $\operatorname{softmax}\!\left( \frac{QK^{\top} + M}{\sqrt{d_{k}}} \right) \cdot V$로 정의되며, $M$는 허용되지 않는 위치를 제외하는 마스크(인과적 및/또는 패딩 마스크)이다. 여러 '어텐션 헤드'가 병렬로 맥락의 다양한 측면을 고려한다(H개 헤드, 일반적으로 $d_{head} = \frac{d_{model}}{H}$); 헤드 수는 모델 규모와 함께 증가한다. <sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Vaswani2017-1)</sup>
2.  **완전 연결 네트워크(Feed‑Forward Network, FFN).** 두 선형 레이어 사이에 비선형성이 있다(일반적으로 GELU/SiLU; 일부 현대 모델에서는 SwiGLU). 중간 차원은 일반적으로 $\approx 4\, d_{model}$; SwiGLU 사용 시 비교 가능한 파라미터 수를 유지하기 위해 종종 $\approx \frac{8}{3}\, d_{model}$를 사용. FFN은 파라미터의 상당 부분을 차지한다. <sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Vaswani2017-1)[\[36\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-36)</sup>

**추가 구성 요소.** 잔차(residual) 연결과 레이어 정규화가 사용되며; 현대 LLM에서는 Pre‑LN(서브블록 이전 정규화)이 더 자주 적용된다—이는 큰 깊이에서 학습 안정성을 향상시킨다. 고전적 LayerNorm 외에도 **RMSNorm**이 점점 더 널리 사용되며(계산 비용 감소, 대형 모델에서 잘 작동); 일부 패밀리에서는 어텐션 공간의 정규화도 사용된다(예: softmax 이전 Q/K 정규화). 위치 표현은 절대적이거나 상대적일 수 있으며; 긴 맥락에서의 사실상 표준은 RoPE가 되었다.

##### 모델 예시 및 규모

- BERT‑Large: 24개 레이어, 차원 1024, 16개 어텐션 헤드, ≈3억4천만 파라미터. <sup>[\[37\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-37)</sup>
- GPT‑3(175B): 96개 레이어, 차원 12288, 96개 어텐션 헤드, ≈1750억 파라미터. <sup>[\[38\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-38)</sup>
- LLaMA‑65B: 80개 레이어, 차원 8192, 64개 어텐션 헤드, ≈650억 파라미터. <sup>[\[39\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-39)</sup>
- PaLM‑540B: 118개 레이어, 차원 약 18432, ≈5400억 파라미터. <sup>[\[40\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-40)</sup>

##### 장점

- 균일한 블록, 잘 연구된 학습 방식, 확장 시 예측 가능한 동작.
- 파라미터와 데이터 증가에 따라 품질이 거듭제곱 법칙으로 향상; compute‑optimal 방식은 모델 크기와 학습 토큰 수의 동시 증가를 제안. <sup>[\[41\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-41)[\[42\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-42)</sup>
- 동일한 아키텍처가 레이어 수준 변경 없이 파인튜닝 후 광범위한 작업을 커버.

##### 단점

- 완전 자기 어텐션은 시퀀스 길이에 대해 이차 복잡도($O(n^{2})$)를 가지며 맥락 창을 제한. <sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Vaswani2017-1)</sup>
- 생성 단계에서 파라미터 완전 활성화: MoE 없는 디코더에서 토큰당 추론 비용은 파라미터 수에 대략 비례하여 증가.
- 메모리 대역폭 병목(memory‑bound): HBM에서 가중치 로딩이 추론 속도를 제한하는 경우가 많음.

##### 확장 및 맥락 한계

- 파라미터용 메모리는 모델 크기에 비례하여 선형 증가; 학습 메모리는 그래디언트와 옵티마이저 상태로 인해 증가.
- 기본 구성은 역사적으로 2,000~4,000 토큰으로 제한되었다. 현대 위치 방식(RoPE)과 확장 기술(Position Interpolation, YaRN 등)은 창을 10배 이상 늘릴 수 있으나 추가적인 계산/메모리 부담이 따른다. <sup>[\[43\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-43)[\[44\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-44)</sup>

### 현대적 최적화

- **FlashAttention.** GPU 메모리 계층 구조를 고려한 정확한 어텐션; 긴 시퀀스에서 메모리 비용을 줄이고 학습/추론을 가속. <sup>[\[45\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-45)</sup>
- **KV 캐시 축소 및 관리.** Multi‑Query Attention과 Grouped‑Query Attention은 캐시 크기와 메모리 트래픽을 줄이고; 서버 수준에서 PagedAttention(vLLM)은 페이지 방식 캐시 관리로 처리량을 향상. <sup>[\[46\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-46)[\[47\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-47)[\[48\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-48)</sup>
- **추측 디코딩(Speculative Decoding).** 초안(draft) 모델이 이어지는 내용을 제안하고 주 모델이 빠르게 검증; 출력 분포 변경 없이 가속 달성. <sup>[\[49\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-49)</sup>

## 희소 모델(Sparse Models)과 Mixture‑of‑Experts(MoE)

MoE는 토큰당 계산 비용을 비례적으로 늘리지 않고 모델 용량을 증가시키는 방법이다. 레이어에 하나의 큰 FFN 블록 대신 병렬 '전문가' 집합(여러 독립적인 FFN)을 사용하고, 학습 가능한 라우터(gating network)가 각 토큰에 대해 가장 관련성 높은 상위 k개의 전문가를 선택한다(일반적으로 k=1~2; 일부 모델에서는 k=4). 선택된 전문가만 활성화되며, 그 출력은 가중치를 적용하여 합산된다. 이렇게 하면 전체 파라미터가 수천억, 심지어 수조 개에 달할 수 있지만 각 단계에서는 소수만 사용된다. <sup>[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Switch-50)[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-GLAM-51)</sup>

#### 모델 예시 및 규모

- **Switch Transformer(Google)**: 최대 ~1.6T 파라미터; top‑1 라우팅(토큰당 하나의 전문가). MoE가 토큰당 비교 가능한 비용으로 용량을 급격히 늘릴 수 있음을 보였다. <sup>[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Switch-50)</sup>
- **GLaM(Google)**: 1.2T 파라미터, 레이어당 64개 전문가, top‑2; 각 토큰에 대해 ≈96.6B 파라미터(≈8%) 활성화. <sup>[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-GLAM-51)</sup>
- **Mixtral 8×7B(Mistral AI)**: 전체 ~46.7B 파라미터, 토큰당 ≈12.9B 활성, top‑2. <sup>[\[52\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Mixtral8x7-52)[\[53\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Mixtral8x7_paper-53)</sup>
- **Mixtral 8×22B**: 전체 ~141B 파라미터, 토큰당 ≈39B 활성, top‑2. <sup>[\[54\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Mixtral8x22-54)</sup>
- **DBRX(Databricks)**: 전체 132B 파라미터, 토큰당 ≈36B 활성; 16개 전문가와 top‑4 라우팅(세분화된 MoE). <sup>[\[55\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-DBRX-55)</sup>

##### 장점

- 계산 비용은 전체 파라미터 수가 아닌 활성 전문가 수 k에 의해 결정: 훨씬 작은 밀집 모델과 유사한 비용으로 조 단위 규모의 모델을 학습하고 사용할 수 있다. <sup>[\[51\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-GLAM-51)</sup>
- 특화: 전문가들이 언어/도메인/패턴에 자동으로 '적응'하여 멀티도메인 작업의 품질을 향상.
- 유연한 배포: 자주 사용되는 전문가를 메모리에 유지하고 드물게 사용되는 것은 필요 시 로드 가능(적절한 인프라 필요).

##### 한계

- 부하 균형: 정규화 없이 라우터가 일부 전문가에 '고착'될 수 있음(라우터 붕괴). 보조 손실(load‑balancing)과 개선된 라우팅 방식이 필요. <sup>[\[50\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Switch-50)</sup>
- 분산 컴퓨팅 복잡성: expert parallelism과 all‑to‑all 통신이 필요; 통신 오버헤드와 메모리 관리가 병목이 됨. <sup>[\[56\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-NVIDIA_MoE-56)</sup>
- 학습 안정성: 라우터 설정과 용량 제약이 중요하며, 그렇지 않으면 품질/수렴 저하가 발생할 수 있음.

#### 현대적 개선

- **Expert‑Choice 라우팅**: 전문가가 토큰을 '선택'하여 비교 가능한 비용으로 부하 균형 및 수렴을 개선. <sup>[\[57\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-ExpertChoice-57)</sup>
- **세분화된 MoE(Fine‑grained MoE)**: 더 많고 작은 전문가(DBRX처럼)로 세밀한 특화 세분성 달성. <sup>[\[55\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-DBRX-55)</sup>
- **Sparse Upcycling**: 밀집 모델의 체크포인트에서 MoE로 전환하여 적절한 비용으로 품질을 크게 향상. <sup>[\[58\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-SparseUpcycling-58)</sup>

#### MoE 적용의 적절성

- 제한된 compute 예산의 대규모 멀티도메인 어시스턴트.
- 특화가 이점을 제공하는 광범위한 코퍼스 학습.
- 발전된 분산 인프라(다수의 GPU/TPU 및 고속 네트워크) 환경.

**밀집 모델이 더 적합한 경우**: 제한된 인프라(GPU 1~2개), 예측 가능한 지연 시간 및 간단한 배포에 대한 엄격한 요구 사항.

## Retrieval‑Augmented Generation(RAG)

RAG는 모델 자체의 내부 아키텍처가 아니라 LLM을 중심으로 한 **시스템 패턴**이다. LLM(생성 구성 요소)과 외부 지식 베이스(검색 구성 요소)를 결합하여 모델의 '파라미터 메모리' 한계를 보완한다.

- **작동 원리:** 생성 전에 LLM이 외부 소스(위키, 기업 지식 베이스, 웹)에서 관련 문서를 검색하고 응답 생성 시 이를 참조한다. <sup>[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-RAG-59)</sup>
- **장점:**
  - 환각 감소 및 사실 정확성 향상. <sup>[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-RAG-59)[\[60\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-60)</sup>
  - 모델 완전 재학습 없이 최신성 유지. <sup>[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-RAG-59)</sup>
  - 응답의 인용 가능성 및 추적 가능성.
- **활용:** 검증 가능한 사실과 비공개/전문화된 데이터 처리가 요구되는 기업 어시스턴트 및 시스템의 사실상 표준. <sup>[\[59\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-RAG-59)</sup>

## 어텐션 메커니즘과 맥락 처리

기본 자기 어텐션은 시퀀스 길이에 대해 이차 복잡도($O(n^{2})$)를 가지므로 다양한 최적화가 등장했다.

- **희소 어텐션(Sparse Attention):** 로컬 창/패턴으로 어텐션 제한. 예시: **Longformer**<sup>[\[61\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-61)</sup>, **BigBird**<sup>[\[62\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-62)</sup>.
- **FlashAttention:** GPU 메모리 계층 구조를 고려한 계산 순서 재구성; 시간과 메모리에서 상당한 이점을 제공하며 긴 맥락의 LLM 학습에서 사실상 표준이 됨<sup>[\[63\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-63)[\[64\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-64)[\[65\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-65)</sup>.
- **MQA/GQA(디코딩 가속):** *Multi‑Query Attention*(모든 헤드에 공유된 키/값)으로 KV 캐시 트래픽 감소<sup>[\[66\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-66)</sup>. *Grouped‑Query Attention*으로 품질/속도 균형 조정<sup>[\[67\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-67)</sup>.
- **개선된 위치 표현:**
  - **ALiBi(Attention with Linear Biases):** 어텐션 점수에 선형 편향을 더해 더 긴 길이로의 일반화 향상. <sup>[\[68\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-68)</sup>
  - **RoPE(Rotary Position Embeddings):** Q/K 회전을 통한 상대적 위치 정보; 현대 모델(예: LLaMA)에서 널리 사용. <sup>[\[69\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-69)[\[70\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-70)</sup>
  - **RoPE 모델의 맥락 확장:** *Position Interpolation* <sup>[\[71\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-71)</sup>, *YaRN* <sup>[\[72\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-72)</sup>, NTK‑aware 수정을 통해 아키텍처 변경 없이 맥락 창을 효과적으로 확장.

<!-- -->

- **긴 시퀀스를 위한 다른 접근 방식:**
  - **Transformer‑XL:** 장거리 의존성 모델링을 위한 세그먼트 간 순환 메모리. <sup>[\[73\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-73)</sup>
  - **Reformer:** 메모리 절약을 위한 LSH 어텐션 및 가역 잔차 블록. <sup>[\[74\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-74)</sup>
  - **Performer:** 소프트맥스 어텐션의 선형 근사(FAVOR+). <sup>[\[75\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-75)</sup>
  - **Linformer:** 어텐션 행렬의 저랭크 근사. <sup>[\[76\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-76)</sup>

## 모델 최적화 및 학습 인프라

LLM 학습과 배포에는 특화된 기술과 프레임워크가 사용된다.

- **양자화(Quantization):** 가중치의 비트 수 감소로 메모리 절약 및 추론 가속. **QLoRA**는 완전 정밀도에 근접한 품질로 4비트 모델(65B 포함)을 효율적으로 파인튜닝할 수 있게 함<sup>[\[77\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-77)</sup>.
- **지식 증류(Knowledge Distillation):** 소형 모델을 위한 *Teacher→Student* 학습<sup>[\[78\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-78)</sup>; 예시—**DistilBERT**<sup>[\[79\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-79)</sup>.
- **분산 학습:**
  - **DeepSpeed**와 **ZeRO**—조 단위 모델 학습을 위한 파라미터/그래디언트/옵티마이저 상태 분산<sup>[\[80\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-80)</sup>.
  - **Megatron‑LM**—매우 큰 트랜스포머를 위한 텐서 및 파이프라인 병렬화<sup>[\[81\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-81)</sup>.
- **생태계 및 도구:** **Hugging Face Transformers**와 **Accelerate**는 표준 모델 구현과 학습 및 추론을 위한 DeepSpeed/FSDP 통합을 제공<sup>[\[82\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-82)[\[83\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-83)</sup>.

## 확장 법칙과 Compute‑Optimal 학습

경험적 **확장 법칙**은 파라미터, 데이터, 계산 증가에 따라 교차 엔트로피 오류가 거듭제곱 법칙으로 감소함을 보여준다. <sup>[\[84\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-84)</sup> **Chinchilla** 연구는 **compute‑optimal** 방식을 구체화했다: 최적 효율을 위해 모델 크기와 학습 토큰 수를 함께 확장해야 한다(예—약 1.4T 토큰으로 학습된 70B 모델이 더 큰 과소학습 모델보다 우수). <sup>[\[85\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-85)</sup>

## 상태 공간 모델(State Space Models, SSM)

**State Space Models(SSM)**은 긴 시퀀스 처리를 위한 트랜스포머의 대안 아키텍처이다. 제어 이론과 디지털 신호 처리의 아이디어를 차용하여 self‑attention의 주요 문제인 텍스트 길이 증가에 따른 이차적 계산 증가를 해결한다.

### 핵심 문제와 해결책

**트랜스포머의 문제.** 전통적인 트랜스포머의 주요 문제는 어텐션의 이차 복잡도이다: 10배 긴 텍스트에는 약 100배 더 많은 계산이 필요하다.

**SSM 접근 방식.** '모든 단어에 동시에 주의를 기울이는' 대신, 모델은 텍스트를 순차적으로 처리하면서 각 단계에서 업데이트되는 컴팩트한 내부 **메모리 상태**를 유지한다. 결과적으로 시간과 메모리 소비는 텍스트 길이에 대략 선형으로 증가한다. 한편 학습은 합성곱 커널 표현을 통해 병렬로 수행될 수 있다(긴 시퀀스에서 높은 처리량). <sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-S4-86)</sup>

### 작동 원리

이산 SSM은 상태 방정식과 출력 방정식으로 기술된다:

$x_{t} = Ax_{t - 1} + Bu_{t},\quad y_{t} = Cx_{t} + Du_{t}$

여기서 $x_{t}$는 메모리 상태, $u_{t}$은 입력(토큰), $y_{t}$은 출력이다. 깊은 SSM에서 행렬 $A,B,C,D$는 긴 시퀀스에서 안정성과 효율적인 계산을 보장하도록 파라미터화된다. 동일한 레이어를 다음과 같이 볼 수 있다:

- 순환적(단계별 스캔)—KV 캐시 없이 메모리 효율적인 추론;
- 합성곱적—사전 계산된 커널로 병렬 학습. <sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-S4-86)</sup>

### 주요 아키텍처 및 하이브리드

- **S4(Structured State Spaces).** 상태 행렬의 안정적 파라미터화를 가진 SSM의 기준선; 매우 긴 시퀀스에서 효율성을 보여준다. <sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-S4-86)</sup>
- **Mamba.** *선택적* SSM: 메모리 업데이트 규칙이 현재 입력에 의존한다(모델 스스로 무엇을 '메모리에 유지'하고 무엇을 '잊을지' 결정). 구현은 GPU 메모리 계층 구조에 최적화되어 있으며; 저자들에 따르면 선형 복잡도로 추론 처리량이 몇 배 증가한다. <sup>[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Mamba-87)</sup>
- **RetNet.** 세 가지 모드를 가진 *retention* 메커니즘: 병렬 학습, 순환 및 블록 순환 추론. 목표는 빠른 학습(트랜스포머처럼)과 경제적인 추론(토큰당 O(1) 메모리)을 결합하는 것이다. <sup>[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-RetNet-88)</sup>
- **Attention+SSM 하이브리드.** 예시—**Jamba**(Transformer와 Mamba 레이어 교차 및 MoE): 유사 클래스의 순수 트랜스포머 모델에 비해 메모리 요구량이 크게 낮으면서 ~256K 토큰 수준의 맥락 지원을 보고한다. <sup>[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Jamba-89)</sup>

#### 장점

- 추론에서 선형 복잡도 및 메모리 절약. 전역 self‑attention 및 KV 캐시 없음; 컴팩트한 상태만 저장. <sup>[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Mamba-87)[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-RetNet-88)</sup>
- 긴 시퀀스에서 병렬 학습. 합성곱 모드로 학습 처리량 향상. <sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-S4-86)</sup>
- 하드웨어 효율성. 구현이 현대 메모리 계층 구조(HBM/SRAM)에 최적화. <sup>[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Mamba-87)</sup>
- 긴 맥락 및 스트리밍. SSM+Attention 하이브리드는 적절한 자원으로 수십만 토큰에서 실용적. <sup>[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Jamba-89)</sup>

#### 한계 및 현재 실제 적용

- 생태계 성숙도. 도구와 확장 '레시피'(지시, RLHF/DPO)는 아직 트랜스포머 스택에 비해 부족. <sup>[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Mamba-87)</sup>
- 품질 및 안정성. 일부 작업에서 하이브리드(Attention+SSM)가 '순수' SSM보다 '품질/속도/메모리' 균형이 더 안정적. <sup>[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Jamba-89)</sup>

#### 접근 방식 비교 (요약)

| 특성                | 트랜스포머                 | SSM               | 하이브리드(Attention+SSM) |
|---------------------|----------------------------|-------------------|---------------------------|
| 길이에 따른 복잡도  | 이차(self‑attention)       | 선형(스캔/합성곱) | 선형에 가까움             |
| 토큰당 메모리(추론) | KV 캐시가 맥락과 함께 증가 | O(1) 상태         | 완만한 증가               |
| 긴 맥락             | 특수 최적화 필요           | 자연스러운 지원   | ~256K까지 실용적          |
| 생태계 성숙도       | 높음                       | 발전 중           | 발전 중                   |

#### 실용적 활용

- 매우 긴 문서 분석(책, 보고서, 학술 리뷰).
- 메모리 비용 증가 없이 긴 이력을 가진 스트리밍 처리 및 대화 시나리오.
- 제한된 자원 환경(모바일/엣지 기기).
- 시계열 및 기타 순차 데이터.

**대표 모델:** S4, Mamba, RetNet; Attention+SSM 하이브리드(Jamba). <sup>[\[86\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-S4-86)[\[87\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Mamba-87)[\[88\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-RetNet-88)[\[89\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Jamba-89)</sup>

## 아키텍처의 진화

- 2017년 — 'Attention Is All You Need' 논문 발표. 트랜스포머 아키텍처 소개: 다중 헤드 자기 어텐션과 위치 인코딩을 통해 순환이나 합성곱 없이 모델 학습이 가능해졌으나, 어텐션은 맥락 길이에 대해 이차 복잡도를 가진다.<sup>[\[1\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-Vaswani2017-1)</sup>

<!-- -->

- 2018년 — GPT‑1과 BERT 소개. GPT‑1은 생성 및 이후 파인튜닝을 위해 인과적 어텐션의 디코더 전용 스택을 사용; BERT는 텍스트 이해 작업을 위한 양방향 인코더와 MLM 사전학습을 도입. <sup>[\[90\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-90)[\[91\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-91)</sup>

<!-- -->

- 2019년 — 긴 시퀀스 처리 방법 제안 및 decoder‑only 확장. Transformer‑XL은 고정 창 너머를 처리하기 위해 '메모리'와 상대적 위치를 추가; GPT‑2는 규모 증가에 따른 zero‑shot 능력 향상을 보임; BART는 seq2seq에 대한 잡음 제거 사전학습의 효과를 보여줌. <sup>[\[92\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-92)[\[93\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-93)[\[94\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-94)</sup>

<!-- -->

- 2020년 — 'text‑to‑text' 형식 통일 및 긴 문서 처리 방법 제시. T5는 다양한 작업을 위한 통합 encoder–decoder 접근 방식을 공식화; Longformer와 BigBird는 긴 텍스트를 위해 희소/구조적 어텐션을 사용; GPT‑3는 밀집 decoder‑only 확장의 효과를 확인. <sup>[\[95\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-95)[\[96\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-96)[\[97\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-97)[\[98\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-98)</sup>

<!-- -->

- 2021년 — 위치 표현 개선 및 파라미터 희소성(MoE) 시연. RoPE와 ALiBi는 더 긴 길이에서의 일반화를 향상; Switch Transformer와 GLaM은 토큰당 일부 전문가만 활성화하여 추론 비용의 배수 증가 없이 용량을 향상. <sup>[\[99\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-99)[\[100\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-100)[\[101\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-101)[\[102\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-102)</sup>

<!-- -->

- 2022년 — compute‑optimal 방식 구체화 및 긴 프롬프트에서의 추론 가속. Chinchilla는 적절한 모델 크기로 더 많은 학습 토큰의 이점을 보여줌; Multi‑Query Attention을 갖춘 PaLM은 KV 캐시 크기를 줄임; FlashAttention은 GPU에서 어텐션을 가속. <sup>[\[103\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-103)[\[104\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-104)[\[105\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-105)[\[106\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-106)</sup>

<!-- -->

- 2023년 — 레이어 변경 없이 맥락 창 확장 및 서버 측 처리 개선. LLaMA 시리즈는 방법론(RMSNorm, SwiGLU, RoPE)을 정착시킴; Position Interpolation과 YaRN은 맥락을 확장; vLLM/PagedAttention은 KV 캐시를 더 효율적으로 관리. <sup>[\[107\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-107)[\[108\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-108)[\[109\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-109)[\[110\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-110)[\[111\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-111)[\[112\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-112)</sup>

<!-- -->

- 2023년 — GPT‑4와 Gemini가 단일 모델 패밀리 내에서 여러 모달리티의 처리 및 생성을 보여줌. <sup>[\[113\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-113)[\[114\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-114)</sup>

<!-- -->

- 2023년 — 상태 공간 모델(SSM) 제안. Mamba와 RetNet은 KV 캐시 대신 컴팩트한 상태를 사용한 순차 처리를 부활시키고 하이브리드 아키텍처의 기반을 마련. <sup>[\[115\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-115)[\[116\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-116)</sup>

<!-- -->

- 2024년 — 오픈 MoE 모델과 Attention+SSM 하이브리드 발표; 새 GPU에서 어텐션 가속. Mixtral 8×7B/8×22B와 DBRX는 MoE의 실용성을 확인; Jamba는 매우 긴 맥락을 위해 Transformer와 Mamba를 결합; FlashAttention‑3은 처리량을 향상. <sup>[\[117\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-117)[\[118\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-118)[\[119\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-119)[\[120\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-120)[\[121\]](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_note-121)</sup>

## 참조 링크

- <a href="https://jalammar.github.io/illustrated-transformer/" class="external free" rel="nofollow">https://jalammar.github.io/illustrated-transformer/</a> The Illustrated Transformer — 시각적 설명

## 참고 문헌

- Vaswani, A. et al. (2017). *Attention Is All You Need*. NIPS. <a href="https://arxiv.org/abs/1706.03762" class="external free" rel="nofollow">https://arxiv.org/abs/1706.03762</a>
- Devlin, J. et al. (2019). *BERT*. NAACL. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a>
- Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. NeurIPS. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a>
- Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer (T5)*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a>
- Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a>
- Touvron, H. et al. (2023). *LLaMA*. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a>
- Chowdhery, A. et al. (2022). *PaLM: Scaling Language Modeling with Pathways*. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a>
- Dao, T. et al. (2022–2024). *FlashAttention (1/2/3)*. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a> ; <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a> ; <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a>
- Shazeer, N. (2019). *MQA*. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a>
- Ainslie, J. et al. (2023). *GQA*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a>
- Kwon, W. et al. (2023). *PagedAttention / vLLM*. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a>
- Leviathan, Y. et al. (2023). *Speculative Decoding*. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a>
- Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). *Switch Transformers*. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a>
- Du, N. et al. (2022). *GLaM*. <a href="https://proceedings.mlr.press/v162/du22c/du22c.pdf" class="external free" rel="nofollow">https://proceedings.mlr.press/v162/du22c/du22c.pdf</a>
- Jiang, A.Q. et al. (2024). *Mixtral of Experts*. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a>
- Databricks (2024). *Introducing DBRX*. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a>
- NVIDIA (2024). *Applying Mixture of Experts in LLM Architectures*. <a href="https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/" class="external free" rel="nofollow">https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/</a>
- Zhou, Y. et al. (2022). *Expert Choice Routing*. <a href="https://arxiv.org/abs/2202.09368" class="external free" rel="nofollow">https://arxiv.org/abs/2202.09368</a>
- Komatsuzaki, A. et al. (2022). *Sparse Upcycling*. <a href="https://arxiv.org/abs/2212.05055" class="external free" rel="nofollow">https://arxiv.org/abs/2212.05055</a>
- Lewis, P. et al. (2020). *RAG*. <a href="https://arxiv.org/abs/2005.11401" class="external free" rel="nofollow">https://arxiv.org/abs/2005.11401</a>
- Beltagy, I. et al. (2020). *Longformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a>
- Zaheer, M. et al. (2020). *BigBird*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a>
- Press, O. et al. (2022). *ALiBi*. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a>
- Su, J. et al. (2021). *RoFormer (RoPE)*. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a>
- Chen, S. et al. (2023). *Position Interpolation*. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a>
- Peng, B. et al. (2023). *YaRN*. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a>
- Dettmers, T. et al. (2023). *QLoRA*. <a href="https://arxiv.org/abs/2305.14314" class="external free" rel="nofollow">https://arxiv.org/abs/2305.14314</a>
- Rajbhandari, S. et al. (2020). *ZeRO*. <a href="https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/" class="external free" rel="nofollow">https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/</a>
- Shoeybi, M. et al. (2019). *Megatron‑LM*. <a href="https://arxiv.org/abs/1909.08053" class="external free" rel="nofollow">https://arxiv.org/abs/1909.08053</a>
- Kaplan, J. et al. (2020). *Scaling Laws*. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a>
- Hoffmann, J. et al. (2022). *Chinchilla / Compute‑Optimal*. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a>
- Gemini Team (2023). *Gemini*. <a href="https://arxiv.org/abs/2312.11805" class="external free" rel="nofollow">https://arxiv.org/abs/2312.11805</a>
- Bai, Y. et al. (2022). *Constitutional AI*. <a href="https://arxiv.org/abs/2212.08073" class="external free" rel="nofollow">https://arxiv.org/abs/2212.08073</a>
- OpenAI (2023). *GPT‑4 Technical Report*. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a>
- OpenAI (2023). *DevDay: GPT‑4 Turbo 128k*. <a href="https://openai.com/index/new-models-and-developer-products-announced-at-devday/" class="external free" rel="nofollow">https://openai.com/index/new-models-and-developer-products-announced-at-devday/</a>
- Zhang, B.; Sennrich, R. (2019). *RMSNorm*. <a href="https://arxiv.org/abs/1910.07467" class="external free" rel="nofollow">https://arxiv.org/abs/1910.07467</a>
- Shazeer, N. (2020). *GLU Variants / SwiGLU*. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a>
- Gu, A.; Goel, K.; Ré, C. (2021). *S4: Structured State Spaces*. <a href="https://arxiv.org/abs/2111.00396" class="external free" rel="nofollow">https://arxiv.org/abs/2111.00396</a>
- Gu, A.; Dao, T. (2023/2024). *Mamba: Selective State Spaces*. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a>
- Sun, Y. et al. (2023). *RetNet*. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a>
- Lieber, O. et al. (2024). *Jamba: Hybrid Transformer‑Mamba*. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a>
- Dai, Z. et al. (2019). *Transformer‑XL*. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a>
- Kitaev, N.; Kaiser, L.; Levskaya, A. (2020). *Reformer*. <a href="https://arxiv.org/abs/2001.04451" class="external free" rel="nofollow">https://arxiv.org/abs/2001.04451</a>
- Choromanski, K. et al. (2021). *Performer*. <a href="https://arxiv.org/abs/2009.14794" class="external free" rel="nofollow">https://arxiv.org/abs/2009.14794</a>
- Wang, S. et al. (2020). *Linformer*. <a href="https://arxiv.org/abs/2006.04768" class="external free" rel="nofollow">https://arxiv.org/abs/2006.04768</a>

## 주석

1.  <span id="cite_note-Vaswani2017-1">↑ <sup>[1.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Vaswani2017_1-0)</sup> <sup>[1.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Vaswani2017_1-1)</sup> <sup>[1.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Vaswani2017_1-2)</sup> <sup>[1.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Vaswani2017_1-3)</sup> <sup>[1.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Vaswani2017_1-4)</sup> <sup>[1.5](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Vaswani2017_1-5)</sup> <sup>[1.6](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Vaswani2017_1-6)</sup> Vaswani, A. et al. (2017). *Attention Is All You Need*. <a href="https://arxiv.org/abs/1706.03762" class="external free" rel="nofollow">https://arxiv.org/abs/1706.03762</a></span>
2.  <span id="cite_note-2">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-2) Devlin, J. et al. (2019). *BERT*. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
3.  <span id="cite_note-3">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-3) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
4.  <span id="cite_note-4">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-4) Raffel, C. et al. (2020). *T5*. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
5.  <span id="cite_note-5">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-5) Devlin, J. et al. (2019). *BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding*. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
6.  <span id="cite_note-6">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-6) Liu, Y. et al. (2019). *RoBERTa: A Robustly Optimized BERT Pretraining Approach*. <a href="https://arxiv.org/abs/1907.11692" class="external free" rel="nofollow">https://arxiv.org/abs/1907.11692</a></span>
7.  <span id="cite_note-7">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-7) He, P. et al. (2021). *DeBERTa: Decoding‑enhanced BERT with Disentangled Attention*. <a href="https://arxiv.org/abs/2006.03654" class="external free" rel="nofollow">https://arxiv.org/abs/2006.03654</a></span>
8.  <span id="cite_note-8">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-8) Clark, K. et al. (2020). *ELECTRA: Pre‑training Text Encoders as Discriminators Rather Than Generators*. <a href="https://arxiv.org/abs/2003.10555" class="external free" rel="nofollow">https://arxiv.org/abs/2003.10555</a></span>
9.  <span id="cite_note-9">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-9) Zaheer, M. et al. (2020). *Big Bird: Transformers for Longer Sequences*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
10. <span id="cite_note-10">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-10) Beltagy, I. et al. (2020). *Longformer: The Long‑Document Transformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
11. <span id="cite_note-11">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-11) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need* (Multi‑Query Attention). <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
12. <span id="cite_note-12">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-12) Ainslie, J. et al. (2023). *GQA: Training Generalized Multi‑Query Transformer Models from Multi‑Head Checkpoints*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
13. <span id="cite_note-13">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-13) Leviathan, Y. et al. (2023). *Fast Inference from Transformers via Speculative Decoding*. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
14. <span id="cite_note-14">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-14) Kwon, W. et al. (2023). *Efficient Memory Management for LLM Serving with PagedAttention (vLLM)*. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
15. <span id="cite_note-15">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-15) vLLM Docs (2024–2025). *Continuous batching, Chunked prefill, Structured outputs*. <a href="https://docs.vllm.ai/" class="external free" rel="nofollow">https://docs.vllm.ai/</a></span>
16. <span id="cite_note-16">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-16) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
17. <span id="cite_note-17">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-17) Ouyang, L. et al. (2022). *InstructGPT (RLHF)*. <a href="https://arxiv.org/abs/2203.02155" class="external free" rel="nofollow">https://arxiv.org/abs/2203.02155</a></span>
18. <span id="cite_note-18">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-18) Rafailov, R. et al. (2023). *Direct Preference Optimization*. <a href="https://arxiv.org/abs/2305.18290" class="external free" rel="nofollow">https://arxiv.org/abs/2305.18290</a></span>
19. <span id="cite_note-19">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-19) Shazeer, 2019. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
20. <span id="cite_note-20">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-20) Ainslie, 2023. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
21. <span id="cite_note-21">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-21) Leviathan, 2023. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
22. <span id="cite_note-22">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-22) Kwon, 2023. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
23. <span id="cite_note-23">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-23) vLLM Docs. <a href="https://docs.vllm.ai/" class="external free" rel="nofollow">https://docs.vllm.ai/</a></span>
24. <span id="cite_note-24">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-24) OpenAI (2024). *Structured Outputs*. <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/" class="external free" rel="nofollow">https://openai.com/index/introducing-structured-outputs-in-the-api/</a></span>
25. <span id="cite_note-25">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-25) vLLM Docs — Structured outputs. <a href="https://docs.vllm.ai/en/v0.9.2/features/structured_outputs.html" class="external free" rel="nofollow">https://docs.vllm.ai/en/v0.9.2/features/structured_outputs.html</a></span>
26. <span id="cite_note-26">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-26) Kwon, 2023. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
27. <span id="cite_note-27">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-27) Brown, T. et al. (2020). *Language Models are Few‑Shot Learners*. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
28. <span id="cite_note-28">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-28) Touvron, H. et al. (2023). *LLaMA: Open and Efficient Foundation Language Models*. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
29. <span id="cite_note-29">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-29) Achiam, J. et al. (2023). *GPT‑4 Technical Report*. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a></span>
30. <span id="cite_note-30">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-30) Meta AI (2024). *Introducing Meta Llama 3*. <a href="https://ai.meta.com/blog/meta-llama-3/" class="external free" rel="nofollow">https://ai.meta.com/blog/meta-llama-3/</a></span>
31. <span id="cite_note-31">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-31) Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer (T5)*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
32. <span id="cite_note-32">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-32) Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
33. <span id="cite_note-33">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-33) Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text‑to‑Text Transformer*. JMLR. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
34. <span id="cite_note-34">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-34) Lewis, M. et al. (2019). *BART: Denoising Sequence‑to‑Sequence Pre‑training for NLG, Translation, and Comprehension*. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
35. <span id="cite_note-35">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-35) Chung, H. W. et al. (2022). *Scaling Instruction‑Finetuned Language Models (FLAN‑T5)*. <a href="https://arxiv.org/abs/2210.11416" class="external free" rel="nofollow">https://arxiv.org/abs/2210.11416</a></span>
36. <span id="cite_note-36">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-36) Shazeer, N. (2020). GLU Variants Improve Transformer. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a></span>
37. <span id="cite_note-37">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-37) Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
38. <span id="cite_note-38">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-38) Brown, T. et al. (2020). Language Models are Few‑Shot Learners. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
39. <span id="cite_note-39">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-39) Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
40. <span id="cite_note-40">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-40) Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a></span>
41. <span id="cite_note-41">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-41) Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a></span>
42. <span id="cite_note-42">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-42) Hoffmann, J. et al. (2022). Training Compute‑Optimal Large Language Models. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
43. <span id="cite_note-43">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-43) Chen, S. et al. (2023). Extending Context Window via Positional Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
44. <span id="cite_note-44">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-44) Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of LLMs. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
45. <span id="cite_note-45">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-45) Dao, T. et al. (2022–2024). FlashAttention (1/2/3). <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a> ; <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a> ; <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>
46. <span id="cite_note-46">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-46) Shazeer, N. (2019). Fast Transformer Decoding: One Write‑Head is All You Need. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
47. <span id="cite_note-47">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-47) Ainslie, J. et al. (2023). GQA. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
48. <span id="cite_note-48">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-48) Kwon, W. et al. (2023). Efficient Memory Management for LLM Serving with PagedAttention. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
49. <span id="cite_note-49">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-49) Leviathan, Y. et al. (2023). Fast Inference from Transformers via Speculative Decoding. <a href="https://arxiv.org/abs/2211.17192" class="external free" rel="nofollow">https://arxiv.org/abs/2211.17192</a></span>
50. <span id="cite_note-Switch-50">↑ <sup>[50.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Switch_50-0)</sup> <sup>[50.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Switch_50-1)</sup> <sup>[50.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Switch_50-2)</sup> Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). *Switch Transformers*. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a></span>
51. <span id="cite_note-GLAM-51">↑ <sup>[51.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-GLAM_51-0)</sup> <sup>[51.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-GLAM_51-1)</sup> <sup>[51.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-GLAM_51-2)</sup> Du, N. et al. (2021). *GLaM: Efficient Scaling of Language Models with Mixture‑of‑Experts*. <a href="https://arxiv.org/pdf/2112.06905.pdf" class="external free" rel="nofollow">https://arxiv.org/pdf/2112.06905.pdf</a></span>
52. <span id="cite_note-Mixtral8x7-52">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Mixtral8x7_52-0) Mistral AI (2023). *Mixtral of Experts*. <a href="https://mistral.ai/news/mixtral-of-experts/" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-of-experts/</a></span>
53. <span id="cite_note-Mixtral8x7_paper-53">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Mixtral8x7_paper_53-0) Jiang, A.Q. et al. (2024). *Mixtral of Experts*. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a></span>
54. <span id="cite_note-Mixtral8x22-54">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Mixtral8x22_54-0) Mistral AI (2024). *Mixtral 8x22B*. <a href="https://mistral.ai/news/mixtral-8x22b" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-8x22b</a></span>
55. <span id="cite_note-DBRX-55">↑ <sup>[55.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-DBRX_55-0)</sup> <sup>[55.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-DBRX_55-1)</sup> Databricks (2024). *Introducing DBRX*. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a></span>
56. <span id="cite_note-NVIDIA_MoE-56">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-NVIDIA_MoE_56-0) NVIDIA (2024). *Applying Mixture of Experts in LLM Architectures*. <a href="https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/" class="external free" rel="nofollow">https://developer.nvidia.com/blog/applying-mixture-of-experts-in-llm-architectures/</a></span>
57. <span id="cite_note-ExpertChoice-57">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-ExpertChoice_57-0) Zhou, Y. et al. (2022). *Mixture‑of‑Experts with Expert Choice Routing*. <a href="https://arxiv.org/abs/2202.09368" class="external free" rel="nofollow">https://arxiv.org/abs/2202.09368</a></span>
58. <span id="cite_note-SparseUpcycling-58">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-SparseUpcycling_58-0) Komatsuzaki, A. et al. (2022). *Sparse Upcycling: Training Mixture‑of‑Experts from Dense Checkpoints*. <a href="https://arxiv.org/abs/2212.05055" class="external free" rel="nofollow">https://arxiv.org/abs/2212.05055</a></span>
59. <span id="cite_note-RAG-59">↑ <sup>[59.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-RAG_59-0)</sup> <sup>[59.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-RAG_59-1)</sup> <sup>[59.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-RAG_59-2)</sup> <sup>[59.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-RAG_59-3)</sup> Lewis, P. et al. (2020). *Retrieval‑Augmented Generation for Knowledge‑Intensive NLP Tasks*. <a href="https://arxiv.org/abs/2005.11401" class="external free" rel="nofollow">https://arxiv.org/abs/2005.11401</a></span>
60. <span id="cite_note-60">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-60) NVIDIA Blog (2025). *What is Retrieval‑Augmented Generation (RAG)*. <a href="https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/" class="external free" rel="nofollow">https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/</a></span>
61. <span id="cite_note-61">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-61) Beltagy, I. et al. (2020). *Longformer*. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
62. <span id="cite_note-62">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-62) Zaheer, M. et al. (2020). *Big Bird*. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
63. <span id="cite_note-63">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-63) Dao, T. et al. (2022). *FlashAttention*. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a></span>
64. <span id="cite_note-64">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-64) Dao, T. et al. (2023). *FlashAttention‑2*. <a href="https://arxiv.org/abs/2307.08691" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08691</a></span>
65. <span id="cite_note-65">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-65) Shah, M. et al. (2024). *FlashAttention‑3*. <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>
66. <span id="cite_note-66">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-66) Shazeer, N. (2019). *Fast Transformer Decoding: One Write‑Head is All You Need*. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
67. <span id="cite_note-67">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-67) Ainslie, J. et al. (2023). *GQA*. <a href="https://arxiv.org/abs/2305.13245" class="external free" rel="nofollow">https://arxiv.org/abs/2305.13245</a></span>
68. <span id="cite_note-68">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-68) Press, O. et al. (2022). ALiBi. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a></span>
69. <span id="cite_note-69">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-69) Su, J. et al. (2021). RoFormer: Rotary Position Embedding. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a></span>
70. <span id="cite_note-70">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-70) Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
71. <span id="cite_note-71">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-71) Chen, S. et al. (2023). Extending Context Window via Positional Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
72. <span id="cite_note-72">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-72) Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of LLMs. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
73. <span id="cite_note-73">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-73) Dai, Z. et al. (2019). *Transformer‑XL: Attentive Language Models Beyond a Fixed‑Length Context*. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a></span>
74. <span id="cite_note-74">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-74) Kitaev, N.; Kaiser, L.; Levskaya, A. (2020). *Reformer: The Efficient Transformer*. <a href="https://arxiv.org/abs/2001.04451" class="external free" rel="nofollow">https://arxiv.org/abs/2001.04451</a></span>
75. <span id="cite_note-75">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-75) Choromanski, K. et al. (2021). *Rethinking Attention with Performers*. <a href="https://arxiv.org/abs/2009.14794" class="external free" rel="nofollow">https://arxiv.org/abs/2009.14794</a></span>
76. <span id="cite_note-76">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-76) Wang, S. et al. (2020). *Linformer: Self‑Attention with Linear Complexity*. <a href="https://arxiv.org/abs/2006.04768" class="external free" rel="nofollow">https://arxiv.org/abs/2006.04768</a></span>
77. <span id="cite_note-77">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-77) Dettmers, T. et al. (2023). *QLoRA: Efficient Finetuning of Quantized LLMs*. <a href="https://arxiv.org/abs/2305.14314" class="external free" rel="nofollow">https://arxiv.org/abs/2305.14314</a></span>
78. <span id="cite_note-78">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-78) Hinton, G. et al. (2015). *Distilling the Knowledge in a Neural Network*. <a href="https://arxiv.org/abs/1503.02531" class="external free" rel="nofollow">https://arxiv.org/abs/1503.02531</a></span>
79. <span id="cite_note-79">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-79) Sanh, V. et al. (2019). *DistilBERT*. <a href="https://arxiv.org/abs/1910.01108" class="external free" rel="nofollow">https://arxiv.org/abs/1910.01108</a></span>
80. <span id="cite_note-80">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-80) Rajbhandari, S. et al. (2020). *ZeRO: Memory Optimizations Toward Training Trillion‑Parameter Models*. <a href="https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/" class="external free" rel="nofollow">https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/</a></span>
81. <span id="cite_note-81">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-81) Shoeybi, M. et al. (2019). *Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism*. <a href="https://arxiv.org/abs/1909.08053" class="external free" rel="nofollow">https://arxiv.org/abs/1909.08053</a></span>
82. <span id="cite_note-82">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-82) Hugging Face. *Transformers Documentation*. <a href="https://huggingface.co/docs/transformers" class="external free" rel="nofollow">https://huggingface.co/docs/transformers</a></span>
83. <span id="cite_note-83">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-83) Hugging Face. *Accelerate Documentation*. <a href="https://huggingface.co/docs/accelerate" class="external free" rel="nofollow">https://huggingface.co/docs/accelerate</a></span>
84. <span id="cite_note-84">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-84) Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. <a href="https://arxiv.org/abs/2001.08361" class="external free" rel="nofollow">https://arxiv.org/abs/2001.08361</a></span>
85. <span id="cite_note-85">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-85) Hoffmann, J. et al. (2022). Training Compute‑Optimal Large Language Models. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
86. <span id="cite_note-S4-86">↑ <sup>[86.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-S4_86-0)</sup> <sup>[86.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-S4_86-1)</sup> <sup>[86.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-S4_86-2)</sup> <sup>[86.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-S4_86-3)</sup> <sup>[86.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-S4_86-4)</sup> Gu, A.; Goel, K.; Ré, C. (2021). *Efficiently Modeling Long Sequences with Structured State Spaces (S4)*. <a href="https://arxiv.org/abs/2111.00396" class="external free" rel="nofollow">https://arxiv.org/abs/2111.00396</a></span>
87. <span id="cite_note-Mamba-87">↑ <sup>[87.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Mamba_87-0)</sup> <sup>[87.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Mamba_87-1)</sup> <sup>[87.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Mamba_87-2)</sup> <sup>[87.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Mamba_87-3)</sup> <sup>[87.4](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Mamba_87-4)</sup> Gu, A.; Dao, T. (2023/2024). *Mamba: Linear‑Time Sequence Modeling with Selective State Spaces*. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a></span>
88. <span id="cite_note-RetNet-88">↑ <sup>[88.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-RetNet_88-0)</sup> <sup>[88.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-RetNet_88-1)</sup> <sup>[88.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-RetNet_88-2)</sup> Sun, Y. et al. (2023). *Retentive Network: A Successor to Transformer for Large Language Models*. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a></span>
89. <span id="cite_note-Jamba-89">↑ <sup>[89.0](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Jamba_89-0)</sup> <sup>[89.1](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Jamba_89-1)</sup> <sup>[89.2](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Jamba_89-2)</sup> <sup>[89.3](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-Jamba_89-3)</sup> Lieber, O. et al. (2024). *Jamba: A Hybrid Transformer‑Mamba Language Model*. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a></span>
90. <span id="cite_note-90">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-90) Radford, A. et al. (2018). Improving Language Understanding by Generative Pre‑Training. <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" class="external free" rel="nofollow">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a></span>
91. <span id="cite_note-91">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-91) Devlin, J. et al. (2019). BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. <a href="https://arxiv.org/abs/1810.04805" class="external free" rel="nofollow">https://arxiv.org/abs/1810.04805</a></span>
92. <span id="cite_note-92">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-92) Dai, Z. et al. (2019). Transformer‑XL. <a href="https://arxiv.org/abs/1901.02860" class="external free" rel="nofollow">https://arxiv.org/abs/1901.02860</a></span>
93. <span id="cite_note-93">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-93) Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners. <a href="https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" class="external free" rel="nofollow">https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf</a></span>
94. <span id="cite_note-94">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-94) Lewis, M. et al. (2019). BART. <a href="https://arxiv.org/abs/1910.13461" class="external free" rel="nofollow">https://arxiv.org/abs/1910.13461</a></span>
95. <span id="cite_note-95">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-95) Raffel, C. et al. (2020). T5. <a href="https://jmlr.org/papers/volume21/20-074/20-074.pdf" class="external free" rel="nofollow">https://jmlr.org/papers/volume21/20-074/20-074.pdf</a></span>
96. <span id="cite_note-96">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-96) Beltagy, I. et al. (2020). Longformer. <a href="https://arxiv.org/abs/2004.05150" class="external free" rel="nofollow">https://arxiv.org/abs/2004.05150</a></span>
97. <span id="cite_note-97">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-97) Zaheer, M. et al. (2020). BigBird. <a href="https://arxiv.org/abs/2007.14062" class="external free" rel="nofollow">https://arxiv.org/abs/2007.14062</a></span>
98. <span id="cite_note-98">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-98) Brown, T. et al. (2020). Language Models are Few‑Shot Learners. <a href="https://arxiv.org/abs/2005.14165" class="external free" rel="nofollow">https://arxiv.org/abs/2005.14165</a></span>
99. <span id="cite_note-99">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-99) Su, J. et al. (2021). RoPE. <a href="https://arxiv.org/abs/2104.09864" class="external free" rel="nofollow">https://arxiv.org/abs/2104.09864</a></span>
100. <span id="cite_note-100">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-100) Press, O. et al. (2021/2022). ALiBi. <a href="https://arxiv.org/abs/2108.12409" class="external free" rel="nofollow">https://arxiv.org/abs/2108.12409</a></span>
101. <span id="cite_note-101">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-101) Fedus, W.; Zoph, B.; Shazeer, N. (2021/2022). Switch Transformers. <a href="https://arxiv.org/abs/2101.03961" class="external free" rel="nofollow">https://arxiv.org/abs/2101.03961</a></span>
102. <span id="cite_note-102">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-102) Du, N. et al. (2021). GLaM. <a href="https://arxiv.org/pdf/2112.06905.pdf" class="external free" rel="nofollow">https://arxiv.org/pdf/2112.06905.pdf</a></span>
103. <span id="cite_note-103">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-103) Hoffmann, J. et al. (2022). Chinchilla. <a href="https://arxiv.org/abs/2203.15556" class="external free" rel="nofollow">https://arxiv.org/abs/2203.15556</a></span>
104. <span id="cite_note-104">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-104) Chowdhery, A. et al. (2022). PaLM. <a href="https://arxiv.org/abs/2204.02311" class="external free" rel="nofollow">https://arxiv.org/abs/2204.02311</a></span>
105. <span id="cite_note-105">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-105) Shazeer, N. (2019). Fast Transformer Decoding. <a href="https://arxiv.org/abs/1911.02150" class="external free" rel="nofollow">https://arxiv.org/abs/1911.02150</a></span>
106. <span id="cite_note-106">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-106) Dao, T. et al. (2022). FlashAttention. <a href="https://arxiv.org/abs/2205.14135" class="external free" rel="nofollow">https://arxiv.org/abs/2205.14135</a></span>
107. <span id="cite_note-107">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-107) Touvron, H. et al. (2023). LLaMA. <a href="https://arxiv.org/abs/2302.13971" class="external free" rel="nofollow">https://arxiv.org/abs/2302.13971</a></span>
108. <span id="cite_note-108">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-108) Zhang, B.; Sennrich, R. (2019). RMSNorm. <a href="https://arxiv.org/abs/1910.07467" class="external free" rel="nofollow">https://arxiv.org/abs/1910.07467</a></span>
109. <span id="cite_note-109">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-109) Shazeer, N. (2020). GLU Variants. <a href="https://arxiv.org/abs/2002.05202" class="external free" rel="nofollow">https://arxiv.org/abs/2002.05202</a></span>
110. <span id="cite_note-110">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-110) Chen, S. et al. (2023). Position Interpolation. <a href="https://arxiv.org/abs/2306.15595" class="external free" rel="nofollow">https://arxiv.org/abs/2306.15595</a></span>
111. <span id="cite_note-111">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-111) Peng, B. et al. (2023). YaRN. <a href="https://arxiv.org/abs/2309.00071" class="external free" rel="nofollow">https://arxiv.org/abs/2309.00071</a></span>
112. <span id="cite_note-112">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-112) Kwon, W. et al. (2023). vLLM/PagedAttention. <a href="https://arxiv.org/abs/2309.06180" class="external free" rel="nofollow">https://arxiv.org/abs/2309.06180</a></span>
113. <span id="cite_note-113">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-113) OpenAI (2023). GPT‑4 Technical Report. <a href="https://arxiv.org/abs/2303.08774" class="external free" rel="nofollow">https://arxiv.org/abs/2303.08774</a></span>
114. <span id="cite_note-114">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-114) Gemini Team (2023). Gemini. <a href="https://arxiv.org/abs/2312.11805" class="external free" rel="nofollow">https://arxiv.org/abs/2312.11805</a></span>
115. <span id="cite_note-115">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-115) Gu, A.; Dao, T. (2023). Mamba. <a href="https://arxiv.org/abs/2312.00752" class="external free" rel="nofollow">https://arxiv.org/abs/2312.00752</a></span>
116. <span id="cite_note-116">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-116) Sun, Y. et al. (2023). RetNet. <a href="https://arxiv.org/abs/2307.08621" class="external free" rel="nofollow">https://arxiv.org/abs/2307.08621</a></span>
117. <span id="cite_note-117">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-117) Jiang, A.Q. et al. (2024). Mixtral of Experts. <a href="https://arxiv.org/abs/2401.04088" class="external free" rel="nofollow">https://arxiv.org/abs/2401.04088</a></span>
118. <span id="cite_note-118">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-118) Mistral AI (2024). Mixtral 8x22B. <a href="https://mistral.ai/news/mixtral-8x22b" class="external free" rel="nofollow">https://mistral.ai/news/mixtral-8x22b</a></span>
119. <span id="cite_note-119">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-119) Databricks (2024). Introducing DBRX. <a href="https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm" class="external free" rel="nofollow">https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm</a></span>
120. <span id="cite_note-120">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-120) Lieber, O. et al. (2024). Jamba. <a href="https://arxiv.org/abs/2403.19887" class="external free" rel="nofollow">https://arxiv.org/abs/2403.19887</a></span>
121. <span id="cite_note-121">[↑](https://systems-analysis.info/int/Large_language_model_architectures_%E2%80%94_%EB%8C%80%ED%98%95_%EC%96%B8%EC%96%B4_%EB%AA%A8%EB%8D%B8_%EC%95%84%ED%82%A4%ED%85%8D%EC%B2%98#cite_ref-121) Shah, M. et al. (2024). FlashAttention‑3. <a href="https://arxiv.org/abs/2407.08608" class="external free" rel="nofollow">https://arxiv.org/abs/2407.08608</a></span>