--- title: "Contextual forgetting" source: "https://systems-analysis.info/eng/Contextual_forgetting" wiki: "systems-analysis.info/eng" article: "Contextual_forgetting" language: "en" categories: - "Category:English" - "Category:Large language models" - "Category:Machine learning" - "Category:Technology" revision_id: 103 wiki_created_at: 2026-09-06T22:17:26Z wiki_modified_at: 2026-09-06T22:17:26Z downloaded_at: 2026-09-07T22:21:02Z --- # Contextual forgetting **Contextual forgetting in [large language models](https://systems-analysis.info/eng/Large_language_model "Large language model")** is a multifaceted phenomenon in which a large language model (LLM) loses, ignores, or inefficiently uses previously provided information within a single interaction^{[\[1\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-howard-cds-1)}. Unlike human memory, LLMs do not have long-term state storage and rely solely on the **[context window](https://systems-analysis.info/eng/Context_window "Context window")**—a limited volume of text (in [tokens](https://systems-analysis.info/eng/Token_(LLM) "Token (LLM)")) that the model can process at one time. This window acts as the model's short-term or working memory^{[\[2\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-liu2023-lost-in-middle-2)}. The most well-known manifestation of this limitation is the **"Lost in the Middle"** problem—the tendency of models to process information located at the beginning and end of a long context more effectively than information in the middle^{[\[2\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-liu2023-lost-in-middle-2)}. This phenomenon is not a failure but a fundamental property stemming from the [transformer architecture](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture") and its training principles. ## Two Types of Forgetting: Contextual and Catastrophic It is important to distinguish between two fundamentally different types of "forgetting" in LLMs: in-context and catastrophic. ### In-Context Forgetting (Lost in the Middle) This type of forgetting occurs **during a single interaction session** (inference) with an already trained model. It is related to the limitations of the **context window**. When the volume of a dialogue or document exceeds the window size, the model "forgets" the oldest fragments to make room for new ones. Even within the window, information from the middle of the context may be used less effectively. This is a limitation of the model's working memory^{[\[3\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-aclanthology-lost-in-middle-3)}. In popular writing, this phenomenon is also called **"Context Degradation Syndrome" (CDS)**^{[\[1\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-howard-cds-1)}. ### Catastrophic Forgetting (Model Drift) This type of forgetting, also known as "model drift," occurs **during the [fine-tuning](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)") process** of a model on new data. When a model, [pre-trained](https://systems-analysis.info/eng/Pre-training_of_large_language_models "Pre-training of large language models") on a vast corpus of general knowledge, is fine-tuned on a highly specialized dataset (e.g., medical texts), its weights are altered. This can lead to the degradation or "erasure" of previously learned knowledge and skills unrelated to the new task^{[\[4\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-greyling-catastrophic-4)}. ## Causes and Mechanisms Contextual forgetting is a direct consequence of the transformer architecture and the geometry of vector spaces. ### The "Lost in the Middle" Effect A 2023 Stanford University study titled "Lost in the Middle" clearly demonstrated that LLM performance in retrieving information from a long context follows a **U-shaped curve**^{[\[2\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-liu2023-lost-in-middle-2)}. The accuracy of responses is highest when the relevant information is at the very beginning (**primacy effect**) or at the very end (**recency effect**) of the context, and it drops significantly if the information is "hidden" in the middle. The reasons for this phenomenon include: - **Attention mechanism**: The transformer architecture inherently pays disproportionate attention to the initial tokens (so-called "attention sinks") to maintain global coherence, as well as to the local context, which weakens the "focus" on the middle^{[\[5\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-neurips-poster-vectors-5)}. - **Pre-training data**: Models are most often trained on relatively short texts, where important information is rarely located tens of thousands of tokens from the beginning, which hinders their ability to effectively use very long contexts^{[\[6\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-an2024-context-fall-short-6)}. ## Manifestations and Consequences - **Context Degradation Syndrome**: During long dialogues, the model begins to "lose the thread of the conversation," repeat answers, contradict previously established facts, and provide increasingly general and vague responses^{[\[1\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-howard-cds-1)}. - **Failures in multi-step tasks**: In tasks where conditions are refined over several turns, the model might "latch onto" an incorrect initial assumption and ignore subsequent clarifications, leading to a complete inability to solve the task^{[\[7\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-arxiv-multi-turn-lost-7)}. - **Unreliable document analysis**: When analyzing long reports or legal documents, an LLM may miss key facts located in the central sections, making it an unreliable tool for such tasks. ## Mitigation and Prevention Strategies Researchers and developers employ several approaches to address the problem of contextual forgetting. ### Increasing the Context Window The most straightforward approach is to increase the size of the context window. Modern models like **[Claude](https://systems-analysis.info/eng/Claude_(Anthropic) "Claude (Anthropic)") 3** (200k tokens) and **[Gemini](https://systems-analysis.info/eng/Gemini_(Google) "Gemini (Google)") 1.5 Pro** (up to 2 million tokens) have significantly expanded this limit^{[\[8\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-anthropic-claude3-family-8)[\[9\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-google-gemini1-5-pro-9)}. However, research shows that simply increasing the window size does not guarantee its effective use, and the "lost in the middle" problem persists^{[\[2\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-liu2023-lost-in-middle-2)}. ### Advanced Prompt Engineering Properly structuring prompts can significantly improve performance. Anthropic suggests the following practices^{[\[10\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-anthropic-long-context-tips-10)}: - **Placing documents at the beginning**: Place long texts at the very beginning of the prompt, before the instructions and the question. - **Using XML tags**: Wrap documents in \`\\` tags for clear separation. - **Justifying answers with quotes**: Instruct the model to first extract relevant quotes and then formulate its answer based on them. ### Memory Externalization: Retrieval-Augmented Generation (RAG) A fundamentally different approach is not to place all information into the context window but to offload it to an external system (a [vector database](https://systems-analysis.info/eng/Vector_database "Vector database")) and provide it on demand. 1. **Retrieve**: When a query is received, the system searches for relevant information in the external database. 2. **Augment**: The found snippets are added to the original query. 3. **Generate**: The LLM generates a response based on the provided context. **[RAG](https://systems-analysis.info/eng/Retrieval-augmented_generation_(RAG) "Retrieval-augmented generation (RAG)")** allows working with virtually unlimited amounts of data and provides access to fresh and verified information, which reduces the risk of [hallucinations](https://systems-analysis.info/eng/LLM_hallucinations "LLM hallucinations") and is the most reliable solution to date^{[\[11\]](https://systems-analysis.info/eng/Contextual_forgetting#cite_note-google-cloud-rag-11)}. ## External links - Lost in the Middle: How Language Models Use Long Contexts — the original Stanford University study. - Announcement of Claude with a 100K token context window from Anthropic. ## Literature - Liu, N. F. et al. (2023). *Lost in the Middle: How Language Models Use Long Contexts*. arXiv:2307.03172. - An, C. et al. (2024). *Why Does the Effective Context Length of LLMs Fall Short?*. arXiv:2410.18745. - Ding, J. et al. (2023). *LongNet: Scaling Transformers to 1,000,000,000 Tokens*. arXiv:2307.02486. - Yang, A. et al. (2024). *Context Parallelism for Scalable Million-Token Inference*. arXiv:2411.01783. - Chen, S. et al. (2023). *Extending Context Window of Large Language Models via Positional Interpolation*. arXiv:2306.15595. - Ding, Y. et al. (2024). *LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens*. arXiv:2402.13753. - Li, S. et al. (2023). *Functional Interpolation for Relative Positions Improves Long Context Transformers*. arXiv:2310.04418. - Dong, Z. et al. (2024). *Exploring Context Window of Large Language Models via Decomposed Positional Vectors*. arXiv:2405.18009. - Laban, P. et al. (2025). *LLMs Get Lost in Multi-Turn Conversation*. arXiv:2505.06120. - Li, R. et al. (2024). *Extending Context Window in Large Language Models with Segmented Base Adjustment for Rotary Position Embeddings*. *Applied Sciences*, 14(7), 3076. DOI:10.3390/app14073076. - Yang, A. & Reizenstein, J. (2024). *Exploring Context Window of LLMs via Decomposed Positional Vectors* (NeurIPS Poster). NeurIPS 2024. ## References 1. ↑ ^{[1.0](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-howard-cds_1-0)} ^{[1.1](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-howard-cds_1-1)} ^{[1.2](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-howard-cds_1-2)} Howard, James. "Context Degradation Syndrome: When Large Language Models Lose the Plot". *jameshoward.us*. [1] 2. ↑ ^{[2.0](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-liu2023-lost-in-middle_2-0)} ^{[2.1](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-liu2023-lost-in-middle_2-1)} ^{[2.2](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-liu2023-lost-in-middle_2-2)} ^{[2.3](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-liu2023-lost-in-middle_2-3)} Liu, Nelson F.; et al. "Lost in the Middle: How Language Models Use Long Contexts". *arXiv*. [2] 3. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-aclanthology-lost-in-middle_3-0) Liu, Nelson F.; et al. "Lost in the Middle: How Language Models Use Long Contexts". *ACL Anthology*. [3] 4. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-greyling-catastrophic_4-0) Greyling, Cobus. "Catastrophic Forgetting In LLMs". *Medium*. [4] 5. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-neurips-poster-vectors_5-0) "Exploring Context Window of Large Language Models via Decomposed Positional Vectors". *NeurIPS Proceedings*. [5] 6. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-an2024-context-fall-short_6-0) An, Chenxin; et al. "Why Does the Effective Context Length of LLMs Fall Short?". *arXiv*. [6] 7. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-arxiv-multi-turn-lost_7-0) "LLMs Get Lost In Multi-Turn Conversation". *arXiv*. [7] 8. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-anthropic-claude3-family_8-0) "Introducing the next generation of Claude". *Anthropic*. [8] 9. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-google-gemini1-5-pro_9-0) "Google's Gemini 1.5 Pro - Revolutionizing AI with a 1M Token Context Window". *Medium*. [9] 10. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-anthropic-long-context-tips_10-0) "Long context prompting tips". *Anthropic Documentation*. [10] 11. [↑](https://systems-analysis.info/eng/Contextual_forgetting#cite_ref-google-cloud-rag_11-0) "What is Retrieval-Augmented Generation (RAG)?". *Google Cloud*. [11]