---
title: "LLM error mitigation"
source: "https://systems-analysis.info/eng/LLM_error_mitigation"
wiki: "systems-analysis.info/eng"
article: "LLM_error_mitigation"
language: "en"
categories:
  - "Category:English"
  - "Category:Large language models"
  - "Category:Machine learning"
  - "Category:Technology"
revision_id: 201
wiki_created_at: 2026-09-06T22:18:49Z
wiki_modified_at: 2026-09-06T22:18:49Z
downloaded_at: 2026-09-07T22:21:49Z
---

# LLM error mitigation

**Mitigating errors in [large language models](https://systems-analysis.info/eng/Large_language_model "Large language model")** (LLMs) is a set of methods and technologies aimed at improving the accuracy, reliability, and safety of artificial intelligence systems based on the [transformer architecture](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture"). The problem of errors, particularly [hallucinations](https://systems-analysis.info/eng/LLM_hallucinations "LLM hallucinations"), is one of the key barriers to the widespread adoption of LLMs in mission-critical domains. According to studies from 2024–2025, the frequency of hallucinations in publicly available LLMs ranges from 3% to 16%<sup>[\[1\]](https://systems-analysis.info/eng/LLM_error_mitigation#cite_note-vectara2024-1)</sup>.

## Typology of Errors

The modern classification of LLM errors includes several main categories, each requiring specific approaches to mitigation.

### Hallucinations

**Hallucinations** are the generation of plausible but factually incorrect content. According to a study by Huang et al. (2023), two main types are distinguished<sup>[\[2\]](https://systems-analysis.info/eng/LLM_error_mitigation#cite_note-huang2023survey-2)</sup>:

- **Factual hallucinations** — discrepancies with verifiable facts, including the creation of non-existent facts (fabrication). In a 2024 study, Stanford University found that LLMs invented over 120 non-existent legal cases<sup>[\[3\]](https://systems-analysis.info/eng/LLM_error_mitigation#cite_note-ai_index2024-3)</sup>.
- **Logical hallucinations** — violations of logical consistency in reasoning.

Statistics from 2024 show that chatbots hallucinate in 27% of cases, with 46% of generated texts containing factual errors<sup>[\[3\]](https://systems-analysis.info/eng/LLM_error_mitigation#cite_note-ai_index2024-3)</sup>.

### Systematic Biases (Bias)

Biases in LLMs manifest as social prejudices (e.g., associating professions with a specific gender) and demographic performance gaps. Studies in 2024 showed that among 10 tested models, the difference in scores for different demographic groups could reach 4 out of 10 points.

### Toxicity

Toxicity is defined as the generation of offensive, harmful, or discriminatory content. The toxicity metric varies widely depending on the model and the context of use.

## Methods for Error Mitigation

Strategies for combating errors can be divided into two broad groups: methods that modify the model and the training process, and methods applied at the inference stage.

### Model and Training Process Modification

#### Fine-tuning and Instruction Tuning

**Supervised [Fine-Tuning](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)") (SFT)** allows [pre-trained](https://systems-analysis.info/eng/Pre-training_of_large_language_models "Pre-training of large language models") models to be adapted to specific tasks. To reduce computational costs, **Parameter-Efficient Fine-Tuning ([PEFT](https://systems-analysis.info/eng/PEFT_(Parameter-Efficient_Fine-Tuning) "PEFT (Parameter-Efficient Fine-Tuning)"))** methods are used, such as [LoRA](https://systems-analysis.info/eng/Low-Rank_Adaptation_(LoRA) "Low-Rank Adaptation (LoRA)") and QLoRA, which can reduce fine-tuning costs by up to 99% while maintaining effectiveness.

#### Reinforcement Learning from Human Feedback (RLHF)

[RLHF](https://systems-analysis.info/eng/Reinforcement_learning_from_human_feedback_(RLHF) "Reinforcement learning from human feedback (RLHF)") is a two-stage process in which a reward model is first trained based on human preferences, and then the main LLM is optimized to generate responses that maximize this reward. The method has proven effective in models like InstructGPT and [GPT](https://systems-analysis.info/eng/GPT_(OpenAI) "GPT (OpenAI)")-4, significantly improving their alignment with user expectations<sup>[\[4\]](https://systems-analysis.info/eng/LLM_error_mitigation#cite_note-openai2024-4)</sup>.

#### Constitutional AI

Developed by Anthropic, the **[Constitutional AI](https://systems-analysis.info/eng/Constitutional_AI "Constitutional AI")** method is an alternative to RLHF. Instead of direct human feedback, the model is trained to follow a set of principles (a "constitution"). This reduces the need for human oversight by 80-90% and effectively prevents the generation of harmful content<sup>[\[5\]](https://systems-analysis.info/eng/LLM_error_mitigation#cite_note-anthropic_cai-5)</sup>.

#### Architectural Solutions

- **[Mixture of Experts](https://systems-analysis.info/eng/Mixture-of-Experts_(MoE) "Mixture-of-Experts (MoE)") (MoE)**: A sparsely activated architecture that allows for a significant increase in model capacity without a proportional increase in computational costs. It is speculated that GPT-4 uses 8 experts with 220 billion parameters each.
- **Attention Mechanism Modifications**: Techniques such as **Grouped Query Attention (GQA)** (in [Llama](https://systems-analysis.info/eng/LLaMA_(Meta_AI) "LLaMA (Meta AI)") 3 models) and **Sparse Attention** reduce computational complexity and memory requirements, enabling the processing of longer contexts.

### Methods at the Inference Stage

#### Retrieval-Augmented Generation (RAG)

[RAG](https://systems-analysis.info/eng/Retrieval-augmented_generation_(RAG) "Retrieval-augmented generation (RAG)") is one of the most effective methods for reducing factual errors. Before generating a response, the system queries an external knowledge base (e.g., Wikipedia, corporate documentation, scientific articles), retrieves relevant information, and provides it to the model along with the original prompt. This "grounds" the response in verified facts. RAG systems achieve a 56.8% *exact match* on the **TriviaQA** benchmark and outperform traditional models by 60–80% in reducing factual errors.

#### Advanced Prompting Techniques

- **[Chain-of-Thought](https://systems-analysis.info/eng/Chain-of-Thought_Prompting "Chain-of-Thought Prompting") (CoT)**: A prompting technique that encourages the model to generate a step-by-step chain of reasoning before giving the final answer. This significantly improves results in tasks requiring logical and mathematical calculations.
- **Chain of Draft (CoD)**: An evolution of CoT where the model iteratively edits drafts of its response, which allows it to achieve accuracy comparable to CoT while using significantly fewer [tokens](https://systems-analysis.info/eng/Token_(LLM) "Token (LLM)").

#### Intrinsic Self-Correction

Research from TACL 2024 showed that an LLM's ability for self-correction without external information is limited. Effective self-correction typically requires the use of external tools, such as code interpreters for verifying calculations or search engines for validating facts<sup>[\[6\]](https://systems-analysis.info/eng/LLM_error_mitigation#cite_note-tacl2024-6)</sup>.

## Error Evaluation Methods

To measure progress in error mitigation, specialized metrics and benchmarks are used.

- **Traditional metrics**: [Perplexity](https://systems-analysis.info/eng/Perplexity_(metric) "Perplexity (metric)"), [BLEU](https://systems-analysis.info/eng/BLEU_(Bilingual_Evaluation_Understudy) "BLEU (Bilingual Evaluation Understudy)"), and [ROUGE](https://systems-analysis.info/eng/ROUGE_(metric) "ROUGE (metric)"). They are useful for evaluating fluency and n-gram overlap but perform poorly in assessing factual accuracy.
- **Modern approaches**:
  - **FactScore** decomposes long texts into atomic facts and evaluates the percentage of facts supported by a knowledge base.
  - **SAFE (Search-Augmented Factuality Evaluator)** — a method from Google that uses search to verify facts and achieves 72% agreement with human evaluators while being 20 times cheaper.
  - **[TruthfulQA](https://systems-analysis.info/eng/TruthfulQA_Benchmark "TruthfulQA Benchmark")** — a benchmark focused on the ability of models to avoid generating popular misconceptions.

## See also

- [Contextual forgetting](https://systems-analysis.info/eng/Contextual_forgetting "Contextual forgetting")

## Literature

- Huang, L. et al. (2023). *A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions*. <a href="https://arxiv.org/abs/2311.05232" class="external text" rel="nofollow">arXiv:2311.05232</a>.
- Min, S. et al. (2023). *FActScore: Fine-Grained Atomic Evaluation of Factual Precision in Long-Form Text Generation*. <a href="https://arxiv.org/abs/2305.14251" class="external text" rel="nofollow">arXiv:2305.14251</a>.
- Wei, J. et al. (2024). *Long-Form Factuality in Large Language Models (SAFE)*. <a href="https://arxiv.org/abs/2403.18802" class="external text" rel="nofollow">arXiv:2403.18802</a>.
- Lewis, P. et al. (2020). *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks*. <a href="https://arxiv.org/abs/2005.11401" class="external text" rel="nofollow">arXiv:2005.11401</a>.
- Hu, E. et al. (2021). *LoRA: Low-Rank Adaptation of Large Language Models*. <a href="https://arxiv.org/abs/2106.09685" class="external text" rel="nofollow">arXiv:2106.09685</a>.
- Ouyang, L. et al. (2022). *Training Language Models to Follow Instructions with Human Feedback*. <a href="https://arxiv.org/abs/2203.02155" class="external text" rel="nofollow">arXiv:2203.02155</a>.
- Madaan, A. et al. (2023). *Self-Refine: Iterative Refinement with Self-Feedback*. <a href="https://arxiv.org/abs/2303.17651" class="external text" rel="nofollow">arXiv:2303.17651</a>.
- Wang, X. et al. (2022). *Self-Consistency Improves Chain-of-Thought Reasoning in Language Models*. <a href="https://arxiv.org/abs/2203.11171" class="external text" rel="nofollow">arXiv:2203.11171</a>.
- Anthropic (2024). *Constitutional AI: Harmlessness from AI Feedback*. <a href="https://arxiv.org/abs/2212.08073" class="external text" rel="nofollow">arXiv:2212.08073</a>.
- Maslej, N. et al. (2024). *Artificial Intelligence Index Report 2024*. <a href="https://arxiv.org/abs/2405.19522" class="external text" rel="nofollow">arXiv:2405.19522</a>.

## References

1.  <span id="cite_note-vectara2024-1">[↑](https://systems-analysis.info/eng/LLM_error_mitigation#cite_ref-vectara2024_1-0) "Hallucination Leaderboard". *Vectara*. (2024-2025). Retrieved July 4, 2025.</span>
2.  <span id="cite_note-huang2023survey-2">[↑](https://systems-analysis.info/eng/LLM_error_mitigation#cite_ref-huang2023survey_2-0) Huang, L., et al. (2023). "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions". *arXiv:2311.05232*.</span>
3.  <span id="cite_note-ai_index2024-3">↑ <sup>[3.0](https://systems-analysis.info/eng/LLM_error_mitigation#cite_ref-ai_index2024_3-0)</sup> <sup>[3.1](https://systems-analysis.info/eng/LLM_error_mitigation#cite_ref-ai_index2024_3-1)</sup> Stanford Human-Centered AI (2024). "AI Index Report 2024".</span>
4.  <span id="cite_note-openai2024-4">[↑](https://systems-analysis.info/eng/LLM_error_mitigation#cite_ref-openai2024_4-0) OpenAI (2024). "Learning to Reason with LLMs". *Technical Blog*.</span>
5.  <span id="cite_note-anthropic_cai-5">[↑](https://systems-analysis.info/eng/LLM_error_mitigation#cite_ref-anthropic_cai_5-0) Anthropic (2024). "Constitutional AI: Harmlessness from AI Feedback". *Research Paper*.</span>
6.  <span id="cite_note-tacl2024-6">[↑](https://systems-analysis.info/eng/LLM_error_mitigation#cite_ref-tacl2024_6-0) "When Can LLMs Actually Correct Their Own Mistakes?". *Transactions of the Association for Computational Linguistics*. (2024).</span>