--- title: "PEFT (Parameter-Efficient Fine-Tuning)" source: "https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)" wiki: "systems-analysis.info/eng" article: "PEFT_(Parameter-Efficient_Fine-Tuning)" language: "en" categories: - "Category:English" - "Category:Large language models" - "Category:Machine learning" - "Category:Technology" revision_id: 292 wiki_created_at: 2026-09-06T22:20:08Z wiki_modified_at: 2026-09-06T22:20:08Z downloaded_at: 2026-09-07T22:22:30Z --- # PEFT (Parameter-Efficient Fine-Tuning) **Parameter-Efficient [Fine-Tuning](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)")** (**PEFT**) is a collection of methods for adapting large [pre-trained](https://systems-analysis.info/eng/Pre-training_of_large_language_models "Pre-training of large language models") models, such as [large language models](https://systems-analysis.info/eng/Large_language_model "Large language model") (LLMs), to specific tasks with minimal computational and resource costs. Unlike traditional **full fine-tuning**, which requires updating all of the model's parameters, PEFT methods focus on modifying only a small fraction of the weights (less than 1-5% of the total), leaving the majority of the model unchanged ("frozen")^{[\[1\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-survey_arxiv_2025-1)}. This approach significantly reduces memory, storage, and training time requirements, making the adaptation of powerful foundation models more accessible and robust against the problem of catastrophic forgetting^{[\[2\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-mlm_problems_2025-2)}. ## Challenges of Full Fine-Tuning Traditional full fine-tuning, where all model parameters are updated, faces several critical challenges that have catalyzed the development of PEFT: - **High computational costs:** Updating hundreds of billions of parameters requires vast computational resources (high-performance GPUs/TPUs) and large amounts of video memory (VRAM), making the process expensive and inaccessible for many researchers. - **Storage inefficiency:** For each new task, a full, multi-gigabyte copy of the model must be stored, leading to an exponential increase in disk space requirements. - **Catastrophic Forgetting:** As a model adapts to new data, it "forgets" the general knowledge acquired during pre-training, which degrades its performance on other tasks. - **Risk of Overfitting:** On small datasets, models with billions of parameters are prone to "memorizing" training examples instead of learning generalizable patterns^{[\[2\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-mlm_problems_2025-2)}. ## Taxonomy of PEFT Methods PEFT methods can be classified by how they modify the model's parameters. There are three main categories: additive, selective, and reparameterization-based^{[\[3\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-huggingface_peft_methods-3)}. ### Additive Methods These methods freeze all original model weights and add new, small trainable modules. - **Adapters:** The earliest additive method. Small neural network modules with a "bottleneck" architecture are inserted between [transformer](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture") layers. Only the weights of these adapters are trained^{[\[4\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-houlsby2019-4)}. - **Soft Prompt-based Methods:** Instead of changing the model's weights, these methods add trainable vectors ("virtual [tokens](https://systems-analysis.info/eng/Token_(LLM) "Token (LLM)")") to the input data, which guide the model's behavior. Key variants include: - **Prompt Tuning:** Adds trainable vectors only to the input [embeddings](https://systems-analysis.info/eng/Embedding_(NLP) "Embedding (NLP)"). - **Prefix-Tuning:** Adds trainable vector prefixes to the hidden states at each layer of the attention mechanism, allowing for finer control^{[\[5\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-li2021prefix-5)}. - **P-Tuning v2:** A generalization of Prefix-Tuning that applies trainable prompts to all layers of the model, achieving performance comparable to full fine-tuning^{[\[6\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-liu2021ptuningv2-6)}. ### Selective Methods These methods do not add new parameters but select and fine-tune a small subset of existing ones. - **BitFit:** An extremely economical method that fine-tunes only the bias terms and the parameters of the normalization layers, updating less than 0.1% of the total parameters. - **Differential Pruning (Diff Pruning):** Uses a trainable mask to dynamically determine which weights should be updated during the training process^{[\[3\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-huggingface_peft_methods-3)}. ### Reparameterization Methods This category is based on the hypothesis that the weight updates for model adaptation have a low "intrinsic rank." Instead of updating full-sized weight matrices, these methods update their low-rank representation. - **LoRA ([Low-Rank Adaptation](https://systems-analysis.info/eng/Low-Rank_Adaptation_(LoRA) "Low-Rank Adaptation (LoRA)")):** The most popular PEFT method to date. It assumes that the weight update matrix \`ΔW\` can be approximated by the product of two low-rank matrices: \`ΔW = BA\`. During fine-tuning, the original matrix \`W\` is frozen, and only \`A\` and \`B\` are trained^{[\[7\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-hu2021lora-7)}. - **QLoRA:** Combines LoRA with quantization techniques to further reduce memory requirements, allowing models with 65 billion parameters to be fine-tuned on a single consumer GPU^{[\[8\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-dettmers2023qlora-8)}. ## Performance and Resource Comparison PEFT methods can achieve performance comparable to full fine-tuning while radically reducing costs. | Model | Method | Trainable Parameters (%) | Benchmark Result (Avg. Accuracy) | Source | |------------|-----------------------|--------------------------|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------| | BERT-Large | Full Fine-Tuning | 100% | 80.4 ([GLUE](https://systems-analysis.info/eng/GLUE_Benchmark "GLUE Benchmark")) | ^{[\[4\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-houlsby2019-4)} | | BERT-Large | Adapters | 3.6% | 80.0 (GLUE) | ^{[\[4\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-houlsby2019-4)} | | LLaMA-7B | LoRA | 0.83% | 74.7% | ^{[\[9\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-dora_paper-9)} | | LLaMA-7B | DoRA (a LoRA variant) | 0.84% | 78.1% | ^{[\[9\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-dora_paper-9)} | Comparative performance and efficiency of PEFT methods Key advantages of PEFT in resource savings: - **GPU Memory (VRAM):** For the LLaMA 65B model, full fine-tuning theoretically requires \>780 GB of VRAM, whereas **QLoRA** allows it to be fine-tuned on a GPU with \<48 GB of VRAM^{[\[8\]](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_note-dettmers2023qlora-8)}. - **Storage Space:** Checkpoints saved after PEFT take up megabytes instead of gigabytes. This allows for storing hundreds of "adapters" for different tasks in the space that a single fully fine-tuned model would occupy. ## Application Areas Initially developed for NLP, PEFT methods have been successfully adapted for a wide range of tasks: - **Computer Vision (CV) and Vision-Language Models (VLM):** Adapting models like Vision Transformer (ViT) and Segment Anything Model (SAM) for image segmentation tasks, including in biomedicine. - **Code Generation and Analysis:** Customizing LLMs for the specifics of particular software projects, internal APIs, or codebases. - **Generative Models:** "Styling" image generation models, such as Stable Diffusion, using LoRA adapters (the Dreambooth technique). - **Speech Synthesis:** Adapting models to generate speech with a specific voice, intonation, or emotional tone. ## External links - PEFT library repository on GitHub - Official documentation for the PEFT library by Hugging Face - Fine-tuning (deep learning) — Wikipedia ## See also - [LLM cost optimization](https://systems-analysis.info/eng/LLM_cost_optimization "LLM cost optimization") ## Literature - Hu, E.J. et al. (2021). *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv:2106.09685. - Dettmers, T. et al. (2023). *QLoRA: Efficient Finetuning of Quantized LLMs*. arXiv:2305.14314. - Houlsby, N. et al. (2019). *Parameter-Efficient Transfer Learning for NLP*. ICML 2019. - Li, X.L.; Liang, P. (2021). *Prefix-Tuning: Optimizing Continuous Prompts for Generation*. arXiv:2101.00190. - Liu, X. et al. (2022). *P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks*. arXiv:2110.07602. - Ben Zaken, E.; Ravfogel, S.; Goldberg, Y. (2021). *BitFit: Simple Parameter-Efficient Fine-Tuning for Transformer-Based Masked Language Models*. arXiv:2106.10199. - Guo, D.; Rush, A.M.; Kim, Y. (2020). *Parameter-Efficient Transfer Learning with Diff Pruning*. arXiv:2012.07463. - Jiang, Z. et al. (2024). *MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning*. arXiv:2405.12130. - Mao, K. et al. (2024). *A Survey on LoRA of Large Language Models*. arXiv:2407.11046. - Chen, S. et al. (2024). *Parameter-Efficient Fine Tuning: A Comprehensive Analysis Across Applications*. arXiv:2404.13506. - Zhang, J. et al. (2025). *Parameter-Efficient Fine-Tuning for Foundation Models*. arXiv:2501.13787. ## References 1. [↑](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-survey_arxiv_2025_1-0) "Parameter-Efficient Fine-Tuning for Foundation Models". *arXiv:2501.13787*. [1] 2. ↑ ^{[2.0](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-mlm_problems_2025_2-0)} ^{[2.1](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-mlm_problems_2025_2-1)} "5 Problems Encountered Fine-Tuning LLMs with Solutions". *Machine Learning Mastery*. [2] 3. ↑ ^{[3.0](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-huggingface_peft_methods_3-0)} ^{[3.1](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-huggingface_peft_methods_3-1)} "PEFT: Parameter-Efficient Fine-Tuning Methods for LLMs". *Hugging Face Blog*. [3] 4. ↑ ^{[4.0](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-houlsby2019_4-0)} ^{[4.1](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-houlsby2019_4-1)} ^{[4.2](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-houlsby2019_4-2)} Houlsby, N., et al. "Parameter-Efficient Transfer Learning for NLP". *Proceedings of the 36th International Conference on Machine Learning*. [4] 5. [↑](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-li2021prefix_5-0) Li, X.L., Liang, P. "Prefix-Tuning: Optimizing Continuous Prompts for Generation". *arXiv:2101.00190*. [5] 6. [↑](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-liu2021ptuningv2_6-0) Liu, X., et al. "P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks". *arXiv:2110.07602*. [6] 7. [↑](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-hu2021lora_7-0) Hu, E.J., et al. "LoRA: Low-Rank Adaptation of Large Language Models". *arXiv:2106.09685*. [7] 8. ↑ ^{[8.0](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-dettmers2023qlora_8-0)} ^{[8.1](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-dettmers2023qlora_8-1)} Dettmers, T., et al. "QLoRA: Efficient Finetuning of Quantized LLMs". *arXiv:2305.14314*. [8] 9. ↑ ^{[9.0](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-dora_paper_9-0)} ^{[9.1](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning)#cite_ref-dora_paper_9-1)} "Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications". *arXiv:2404.13506*. [9]