---
title: "FlashAttention-2"
source: "https://systems-analysis.info/eng/FlashAttention-2"
wiki: "systems-analysis.info/eng"
article: "FlashAttention-2"
language: "en"
categories:
  - "Category:English"
  - "Category:Large language models"
  - "Category:Machine learning"
  - "Category:Technology"
revision_id: 149
wiki_created_at: 2026-09-06T22:18:01Z
wiki_modified_at: 2026-09-06T22:18:01Z
downloaded_at: 2026-09-07T22:21:21Z
---

# FlashAttention-2

**FlashAttention-2** is an advanced algorithm designed to compute the attention mechanism in [large language models (LLMs)](https://systems-analysis.info/eng/Large_language_model "Large language model"). The algorithm was developed by **Tri Dao** and researchers from Stanford University and was introduced in July 2023<sup>[\[1\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_main-1)</sup>. Its key objective is to significantly accelerate the training and inference of transformer models by more efficiently utilizing GPU hardware resources, while maintaining full computational identity with the standard attention mechanism, i.e., **with no loss of accuracy**.

FlashAttention-2 is a logical successor to the **FlashAttention** algorithm, introduced by the same team in 2022. The new version addresses the issue of incomplete GPU utilization observed in its predecessor and achieves a nearly twofold speed increase compared to the first version.

## Background: The Attention Bottleneck in Transformers

The standard self-attention mechanism is a bottleneck when processing long text sequences in transformers. Its computational complexity and memory consumption grow **quadratically** (O(N²)) with the sequence length (N), imposing severe limitations on the maximum context length and scalability of LLMs<sup>[\[1\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_main-1)</sup>.

To address this problem, the **FlashAttention** algorithm was introduced in 2022<sup>[\[2\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-huggingface_optim-2)</sup>. Its key ideas were:

- **IO-awareness**: The algorithm minimizes expensive read/write operations between the slow GPU High Bandwidth Memory (HBM) and the fast on-chip Static Random-Access Memory (SRAM).
- **Tiling**: Computations are broken down into small blocks (tiles) that are processed in the fast SRAM, which avoids materializing the full attention matrix in memory.

This allowed for **linear** memory consumption growth (O(N)) and a 2–4x speedup compared to standard implementations<sup>[\[2\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-huggingface_optim-2)</sup>. FlashAttention became widely adopted and facilitated the emergence of models with significantly larger context windows, for example, from 2–4k tokens (GPT-3) to 128k (GPT-4) and beyond<sup>[\[3\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-tridao_blog_fa3-3)</sup>. For instance, in the **Falcon-40B** model, using FlashAttention sped up inference by 3x and overall generation throughput by 5x compared to GPT-3<sup>[\[4\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-e2e_analysis-4)</sup>.

## Development and Goals of FlashAttention-2

Despite its success, the first version of FlashAttention did not fully utilize the GPU's computational resources. On **NVIDIA A100** GPUs, its performance only reached **25–40%** of the theoretical maximum (FLOPs/s)<sup>[\[1\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_main-1)</sup>. The main reason was suboptimal utilization of Streaming Multiprocessors and redundant operations with shared memory<sup>[\[5\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-openreview_fa2-5)</sup>.

The goal of **FlashAttention-2** was to further accelerate computations through more effective work parallelization and minimization of auxiliary operations. The algorithm was completely rewritten using low-level primitives from the **NVIDIA CUTLASS 3.x** library to achieve maximum performance<sup>[\[6\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-hazyresearch_blog-6)</sup>.

## Technical Architecture and Principles of Operation

FlashAttention-2 introduces three key improvements to enhance parallelism and efficiency<sup>[\[1\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_main-1)</sup>:

### 1. Minimizing Non-Matrix Operations

The algorithm reduces the number of auxiliary floating-point operations that are not matrix multiplications (non-matmul FLOPs). Since GPU Tensor Cores are specifically optimized for matrix operations (GEMM) and perform them up to 16 times faster, this change allows the most powerful GPU units to be utilized for a larger portion of the time.

### 2. Improved Parallelism

In the original FlashAttention, the work on a single attention head was not parallelized, leading to idle time with long sequences and small batch sizes. FlashAttention-2 introduces **inter-block parallelism**: computations for a single attention head are now distributed across different GPU Streaming Multiprocessors, significantly increasing their utilization.

### 3. Optimized Work Partitioning within a Block

At the level of a single compute block, work was re-partitioned among thread groups (warps) to reduce data exchange via shared memory. This reduces the number of redundant read/write operations required for the Softmax normalization.

## Performance and Efficiency

Thanks to these architectural improvements, FlashAttention-2 demonstrates a significant increase in performance:

- **Twofold Speedup**: The algorithm runs approximately **2 times faster** than the first version of FlashAttention<sup>[\[1\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_main-1)</sup>.
- **High GPU Utilization**: On **NVIDIA A100** GPUs, it achieves **50–73%** of the theoretical maximum throughput (TFLOPs), which is close to the efficiency of optimized matrix multiplication (GEMM) operations<sup>[\[1\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_main-1)</sup>.
- **Record-breaking Computation Speed**:
  - On an **A100** GPU, it reaches speeds of up to **225 TFLOP/s** in an end-to-end training loop for a GPT-style model, corresponding to 72% utilization of the compute units. For comparison, standard attention under the same conditions loaded the GPU at less than 100 TFLOP/s<sup>[\[7\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_pdf-7)</sup>.
  - On an **H100** GPU, performance reaches **335 TFLOP/s**<sup>[\[7\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_pdf-7)</sup>.

This performance boost allows, for example, training a model with a **16k** token context window in the same amount of time previously required for an **8k** token window<sup>[\[5\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-openreview_fa2-5)</sup>. Importantly, the algorithm remains **exact** and deterministic, so its application does not affect the model's prediction quality<sup>[\[8\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-raschka_blog-8)</sup>.

## Application and Ecosystem Integration

FlashAttention-2 quickly became a standard tool in the LLM ecosystem. It is integrated into many popular frameworks and libraries:

- **PyTorch**: Native support.
- **Hugging Face Transformers**: Support is enabled with the parameter \`attn_implementation="flash_attention_2"\` when loading a model<sup>[\[9\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-linkedin_younes-9)</sup>. It is compatible with dozens of architectures (GPT, Llama, Falcon, BERT, etc.)<sup>[\[10\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-huggingface_gpu_infer-10)</sup>.
- **TensorRT-LLM**, **xFormers**, and **Triton**: The algorithm is implemented for these platforms, ensuring its widespread adoption<sup>[\[7\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_pdf-7)</sup>.

This integration allows FlashAttention-2 to be easily combined with other optimization techniques, such as quantization (GPTQ, QLoRA) and parameter-efficient fine-tuning (PEFT)<sup>[\[9\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-linkedin_younes-9)</sup>.

## Comparison with Subsequent Versions

### FlashAttention-3

Research in attention optimization is ongoing. In July 2024, Tri Dao introduced **FlashAttention-3**, which is aimed at leveraging the capabilities of the **NVIDIA Hopper** GPU architecture (H100/H200). Key innovations include<sup>[\[3\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-tridao_blog_fa3-3)</sup>:

- **FP8 Support**: Uses 8-bit floating-point computations for further acceleration.
- **Asynchronous Operations**: More effectively utilizes the asynchronous capabilities of the GPU.

FlashAttention-3 provides a **1.5–2x** speedup compared to FlashAttention-2 on H100 GPUs, reaching performance of up to **740 TFLOP/s** (75% of the theoretical maximum)<sup>[\[11\]](https://systems-analysis.info/eng/FlashAttention-2#cite_note-arxiv_fa3-11)</sup>.

## External links

- <a href="https://en.wikipedia.org/wiki/Transformer_(deep_learning)#FlashAttention" class="external text" rel="nofollow">FlashAttention — Wikipedia</a>

## Literature

- Dao, T. (2023). *FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning*. <a href="https://arxiv.org/abs/2307.08691" class="external text" rel="nofollow">arXiv:2307.08691</a>.
- Dao, T. et al. (2022). *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*. <a href="https://arxiv.org/abs/2205.14135" class="external text" rel="nofollow">arXiv:2205.14135</a>.
- Shah, J. et al. (2024). *FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision*. <a href="https://arxiv.org/abs/2407.08608" class="external text" rel="nofollow">arXiv:2407.08608</a>.
- Kwon, W. et al. (2023). *Efficient Memory Management for Large Language Model Serving with PagedAttention*. <a href="https://arxiv.org/abs/2309.06180" class="external text" rel="nofollow">arXiv:2309.06180</a>.
- Ye, Z. et al. (2025). *FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving*. <a href="https://arxiv.org/abs/2501.01005" class="external text" rel="nofollow">arXiv:2501.01005</a>.
- Chen, Y. et al. (2023). *FlashDecoding++: Faster Large Language Model Inference on GPUs*. <a href="https://arxiv.org/abs/2311.01282" class="external text" rel="nofollow">arXiv:2311.01282</a>.
- Liu, Y. et al. (2024). *FastAttention: Extending FlashAttention-2 to NPUs and Low-Resource GPUs*. <a href="https://openreview.net/forum?id=76NYyOrnfk" class="external text" rel="nofollow">OpenReview: 76NYyOrnfk</a>.
- Dege, P. et al. (2025). *FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs*. <a href="https://arxiv.org/abs/2506.01969" class="external text" rel="nofollow">arXiv:2506.01969</a>.
- Wang, G. et al. (2024). *FlashMask: Efficient and Rich Mask Extension of FlashAttention*. <a href="https://openreview.net/forum?id=rog0J435OO" class="external text" rel="nofollow">OpenReview: rog0J435OO</a>.
- Abbott, V.; Zardini, G. (2025). *FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness*. <a href="https://arxiv.org/abs/2412.03317" class="external text" rel="nofollow">arXiv:2412.03317</a>.

## References

1.  <span id="cite_note-arxiv_main-1">↑ <sup>[1.0](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_main_1-0)</sup> <sup>[1.1](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_main_1-1)</sup> <sup>[1.2](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_main_1-2)</sup> <sup>[1.3](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_main_1-3)</sup> <sup>[1.4](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_main_1-4)</sup> <sup>[1.5](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_main_1-5)</sup> Dao, Tri. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." *arXiv:2307.08691* \[cs.LG\], July 17, 2023. <a href="https://arxiv.org/abs/2307.08691" class="external autonumber" rel="nofollow">[1]</a></span>
2.  <span id="cite_note-huggingface_optim-2">↑ <sup>[2.0](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-huggingface_optim_2-0)</sup> <sup>[2.1](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-huggingface_optim_2-1)</sup> "Optimizing LLMs for Speed and Memory". *Hugging Face Documentation*. <a href="https://huggingface.co/docs/transformers/v4.42.0/en/llm_tutorial_optimization" class="external autonumber" rel="nofollow">[2]</a></span>
3.  <span id="cite_note-tridao_blog_fa3-3">↑ <sup>[3.0](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-tridao_blog_fa3_3-0)</sup> <sup>[3.1](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-tridao_blog_fa3_3-1)</sup> Dao, Tri. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision". *Tri Dao's Blog*. <a href="https://tridao.me/blog/2024/flash3/" class="external autonumber" rel="nofollow">[3]</a></span>
4.  <span id="cite_note-e2e_analysis-4">[↑](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-e2e_analysis_4-0) "FlashAttention vs FlashAttention-2 - an Analysis". *E2E Networks Blog*. <a href="https://www.e2enetworks.com/blog/shades-of-attention-flashattention-vs-flashattention-2-a-comprehensive-study" class="external autonumber" rel="nofollow">[4]</a></span>
5.  <span id="cite_note-openreview_fa2-5">↑ <sup>[5.0](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-openreview_fa2_5-0)</sup> <sup>[5.1](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-openreview_fa2_5-1)</sup> "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning". *OpenReview*. <a href="https://openreview.net/forum?id=mZn2Xyh9Ec" class="external autonumber" rel="nofollow">[5]</a></span>
6.  <span id="cite_note-hazyresearch_blog-6">[↑](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-hazyresearch_blog_6-0) "FlashAttention-2". *Hazy Research, Stanford University*. <a href="https://hazyresearch.stanford.edu/blog/2023-07-17-flash2" class="external autonumber" rel="nofollow">[6]</a></span>
7.  <span id="cite_note-arxiv_pdf-7">↑ <sup>[7.0](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_pdf_7-0)</sup> <sup>[7.1](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_pdf_7-1)</sup> <sup>[7.2](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_pdf_7-2)</sup> Dao, Tri. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning" (PDF). *arXiv:2307.08691*. <a href="https://arxiv.org/pdf/2307.08691.pdf" class="external autonumber" rel="nofollow">[7]</a></span>
8.  <span id="cite_note-raschka_blog-8">[↑](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-raschka_blog_8-0) Raschka, Sebastian. "Llama 2 and FlashAttention 2". *Ahead of AI Magazine*. <a href="https://magazine.sebastianraschka.com/p/research-highlights-in-three-sentences" class="external autonumber" rel="nofollow">[8]</a></span>
9.  <span id="cite_note-linkedin_younes-9">↑ <sup>[9.0](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-linkedin_younes_9-0)</sup> <sup>[9.1](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-linkedin_younes_9-1)</sup> Belkada, Younes. "Faster and more memory efficient models with Flash Attention 2!". *LinkedIn*. <a href="https://www.linkedin.com/posts/younes-belkada-b1a903145_flashattention-llms-activity-7112061650106474496-1dzz" class="external autonumber" rel="nofollow">[9]</a></span>
10. <span id="cite_note-huggingface_gpu_infer-10">[↑](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-huggingface_gpu_infer_10-0) "GPU inference". *Hugging Face Documentation*. <a href="https://huggingface.co/docs/transformers/v4.39.0/perf_infer_gpu_one" class="external autonumber" rel="nofollow">[10]</a></span>
11. <span id="cite_note-arxiv_fa3-11">[↑](https://systems-analysis.info/eng/FlashAttention-2#cite_ref-arxiv_fa3_11-0) Dao, Tri, et al. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision". *arXiv:2407.08608* \[cs.LG\], July 11, 2024. <a href="https://arxiv.org/abs/2407.08608" class="external autonumber" rel="nofollow">[11]</a></span>