LLM error mitigation

Mitigating errors in large language models (LLMs) is a set of methods and technologies aimed at improving the accuracy, reliability, and safety of artificial intelligence systems based on the transformer architecture. The problem of errors, particularly hallucinations, is one of the key barriers to the widespread adoption of LLMs in mission-critical domains. According to studies from 2024–2025, the frequency of hallucinations in publicly available LLMs ranges from 3% to 16%.

Typology of Errors

The modern classification of LLM errors includes several main categories, each requiring specific approaches to mitigation.

Hallucinations

Hallucinations are the generation of plausible but factually incorrect content. According to a study by Huang et al. (2023), two main types are distinguished:

• Factual hallucinations — discrepancies with verifiable facts, including the creation of non-existent facts (fabrication). In a 2024 study, Stanford University found that LLMs invented over 120 non-existent legal cases.
• Logical hallucinations — violations of logical consistency in reasoning.

Statistics from 2024 show that chatbots hallucinate in 27% of cases, with 46% of generated texts containing factual errors.

Systematic Biases (Bias)

Biases in LLMs manifest as social prejudices (e.g., associating professions with a specific gender) and demographic performance gaps. Studies in 2024 showed that among 10 tested models, the difference in scores for different demographic groups could reach 4 out of 10 points.

Toxicity

Toxicity is defined as the generation of offensive, harmful, or discriminatory content. The toxicity metric varies widely depending on the model and the context of use.

Methods for Error Mitigation

Strategies for combating errors can be divided into two broad groups: methods that modify the model and the training process, and methods applied at the inference stage.

Model and Training Process Modification

Fine-tuning and Instruction Tuning

Supervised Fine-Tuning (SFT) allows pre-trained models to be adapted to specific tasks. To reduce computational costs, Parameter-Efficient Fine-Tuning (PEFT) methods are used, such as LoRA and QLoRA, which can reduce fine-tuning costs by up to 99% while maintaining effectiveness.

Reinforcement Learning from Human Feedback (RLHF)

RLHF is a two-stage process in which a reward model is first trained based on human preferences, and then the main LLM is optimized to generate responses that maximize this reward. The method has proven effective in models like InstructGPT and GPT-4, significantly improving their alignment with user expectations.

Constitutional AI

Developed by Anthropic, the Constitutional AI method is an alternative to RLHF. Instead of direct human feedback, the model is trained to follow a set of principles (a "constitution"). This reduces the need for human oversight by 80-90% and effectively prevents the generation of harmful content.

Architectural Solutions

• Mixture of Experts (MoE): A sparsely activated architecture that allows for a significant increase in model capacity without a proportional increase in computational costs. It is speculated that GPT-4 uses 8 experts with 220 billion parameters each.
• Attention Mechanism Modifications: Techniques such as Grouped Query Attention (GQA) (in Llama 3 models) and Sparse Attention reduce computational complexity and memory requirements, enabling the processing of longer contexts.

Methods at the Inference Stage

Retrieval-Augmented Generation (RAG)

RAG is one of the most effective methods for reducing factual errors. Before generating a response, the system queries an external knowledge base (e.g., Wikipedia, corporate documentation, scientific articles), retrieves relevant information, and provides it to the model along with the original prompt. This "grounds" the response in verified facts. RAG systems achieve a 56.8% exact match on the TriviaQA benchmark and outperform traditional models by 60–80% in reducing factual errors.

Advanced Prompting Techniques

• Chain-of-Thought (CoT): A prompting technique that encourages the model to generate a step-by-step chain of reasoning before giving the final answer. This significantly improves results in tasks requiring logical and mathematical calculations.
• Chain of Draft (CoD): An evolution of CoT where the model iteratively edits drafts of its response, which allows it to achieve accuracy comparable to CoT while using significantly fewer tokens.

Intrinsic Self-Correction

Research from TACL 2024 showed that an LLM's ability for self-correction without external information is limited. Effective self-correction typically requires the use of external tools, such as code interpreters for verifying calculations or search engines for validating facts.

Error Evaluation Methods

To measure progress in error mitigation, specialized metrics and benchmarks are used.

• Traditional metrics: Perplexity, BLEU, and ROUGE. They are useful for evaluating fluency and n-gram overlap but perform poorly in assessing factual accuracy.
• Modern approaches:
  • FactScore decomposes long texts into atomic facts and evaluates the percentage of facts supported by a knowledge base.
  • SAFE (Search-Augmented Factuality Evaluator) — a method from Google that uses search to verify facts and achieves 72% agreement with human evaluators while being 20 times cheaper.
  • TruthfulQA — a benchmark focused on the ability of models to avoid generating popular misconceptions.

See also

• Contextual forgetting

Literature

• Huang, L. et al. (2023). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv:2311.05232.
• Min, S. et al. (2023). FActScore: Fine-Grained Atomic Evaluation of Factual Precision in Long-Form Text Generation. arXiv:2305.14251.
• Wei, J. et al. (2024). Long-Form Factuality in Large Language Models (SAFE). arXiv:2403.18802.
• Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401.
• Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
• Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv:2203.02155.
• Madaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651.
• Wang, X. et al. (2022). Self-Consistency Improves Chain-of-Thought Reasoning in Language Models. arXiv:2203.11171.
• Anthropic (2024). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
• Maslej, N. et al. (2024). Artificial Intelligence Index Report 2024. arXiv:2405.19522.