--- title: "In-Context Learning" source: "https://systems-analysis.info/eng/In-Context_Learning" wiki: "systems-analysis.info/eng" article: "In-Context_Learning" language: "en" categories: - "Category:English" - "Category:Large language models" - "Category:Machine learning" - "Category:Technology" revision_id: 191 wiki_created_at: 2026-09-06T22:18:41Z wiki_modified_at: 2026-09-06T22:18:41Z downloaded_at: 2026-09-07T22:21:44Z --- # In-Context Learning **In-Context Learning** (**ICL**) is a fundamental capability of [large language models](https://systems-analysis.info/eng/Large_language_model "Large language model") (LLMs) to learn new tasks "on the fly" using only examples (demonstrations) provided within the context (prompt) of a query. A key feature is that this adaptation process occurs without updating the model's weights (parameters), meaning without traditional [fine-tuning](https://systems-analysis.info/eng/Fine-tuning_(deep_learning) "Fine-tuning (deep learning)")[\[1\]](https://systems-analysis.info/eng/In-Context_Learning#cite_note-lakera-1)[\[2\]](https://systems-analysis.info/eng/In-Context_Learning#cite_note-hopsworks-2). This mechanism allows models to exhibit remarkable flexibility, solving tasks for which they were not specifically trained. ICL has become one of the key breakthroughs that has made large language models so powerful and versatile[\[3\]](https://systems-analysis.info/eng/In-Context_Learning#cite_note-gradient_pub-3). ## How It Works A precise understanding of how ICL works remains an active area of research; however, there are several leading theories that explain this phenomenon. ### Transformer as a Meta-Optimizer One popular theory suggests that the [Transformer architecture](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture") learns to implement learning algorithms within its forward passes during [pre-training](https://systems-analysis.info/eng/Pre-training_of_large_language_models "Pre-training of large language models"). When the model receives a prompt with examples, it implicitly performs a form of optimization to solve the presented task by adjusting its internal states (activations) rather than its weights[\[4\]](https://systems-analysis.info/eng/In-Context_Learning#cite_note-arxiv_grad-4). ### Bayesian Inference Another theory views ICL as a form of Bayesian inference. A model pre-trained on vast amounts of data has a prior understanding of numerous concepts. The examples in the context serve as evidence that allows the model to update its posterior probability distribution over the latent concept. In other words, the examples help the model "understand" which specific task, out of the thousands it knows, needs to be solved at that moment[\[5\]](https://systems-analysis.info/eng/In-Context_Learning#cite_note-stanford-5). ## Types of In-Context Learning Depending on the number of examples provided, ICL is divided into three main types. - **[Few-shot Learning](https://systems-analysis.info/eng/Few-Shot_and_Zero-Shot_Learning "Few-Shot and Zero-Shot Learning")**: This is the most common and balanced approach. The model is provided with a few (typically 2 to 10) demonstration examples. *Example (sentiment classification):* Text: "What a beautiful day!" Sentiment: Positive Text: "I hate being stuck in traffic." Sentiment: Negative Text: "This movie was rather average." Sentiment: **Expected output:** Neutral - **One-shot Learning**: The model is given only one example. This is often sufficient to set the output format and significantly improve performance compared to the zero-shot approach. - **Zero-shot Learning**: The model is not provided with any examples, only an instruction or a description of the task. In this case, the model relies entirely on the knowledge acquired during its pre-training. ## Practical Applications The proper application of ICL allows for solving a wide range of tasks without costly development and fine-tuning. - **For creative and stylistic tasks** (e.g., generating code in a specific style, writing text in the manner of a particular author): - **[Few-shot Learning](https://systems-analysis.info/eng/Few-Shot_and_Zero-Shot_Learning "Few-Shot and Zero-Shot Learning")** is recommended. - Examples help the model grasp the required style, format, and output structure. - **For simple tasks with clear instructions** (e.g., translation, summarization, answering simple questions): - **[Zero-shot Learning](https://systems-analysis.info/eng/Few-Shot_and_Zero-Shot_Learning "Few-Shot and Zero-Shot Learning")** is often sufficient. - Modern models handle such tasks quite well if they were part of their pre-training. - **For tasks where the output format is crucial** (e.g., generating JSON, entity extraction): - **One-shot** or **Few-shot Learning** is recommended. - Even a single example can clearly define the required response structure, preventing formatting errors. ## Advantages and Disadvantages
Comparison of Advantages and Disadvantages of ICL
Advantages Disadvantages
  • Flexibility and Speed: Instant adaptation to new tasks without the need for retraining.
  • Resource Efficiency: Does not require data collection, labeling, or the computational resources needed for fine-tuning.
  • Accessibility: Allows users without ML expertise to configure models using simple text examples.
  • Context Window Limitations: The number of examples is limited by the model's maximum context length.
  • Sensitivity to Examples: The result is highly dependent on the quality, order, and format of the provided demonstrations.
  • High Inference Costs: Long prompts with many examples increase the cost and time of generation.
  • Security Risks: Providing confidential information as examples can be insecure.
Comparison of Advantages and Disadvantages of ICL ## Comparison with Other Paradigms ### ICL vs. Fine-tuning Fine-tuning modifies the model's weights, "imprinting" new knowledge into it. This makes the model an expert in a narrow domain but reduces its overall flexibility. ICL, in contrast, does not change the weights and is more flexible, but it may underperform on highly specialized tasks that require deep domain knowledge. ### ICL vs. RAG (Retrieval-Augmented Generation) Both methods extend the model's context, but for different purposes: - **ICL** uses examples to **teach** the model *how* to perform a task (demonstrating a skill). - **[RAG](https://systems-analysis.info/eng/Retrieval-augmented_generation_(RAG) "Retrieval-augmented generation (RAG)")** uses retrieved information to **provide** the model with facts needed for the response (providing knowledge). In practice, ICL and RAG are often combined to achieve the best results. ## External links - In-context learning (natural language processing) — Wikipedia ## Literature - Brown, T. B. et al. (2020). *Language Models are Few-Shot Learners*. arXiv:2005.14165. - Dai, D. et al. (2022). *Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers*. arXiv:2212.10559. - Panwar, M.; Ahuja, K.; Goyal, N. (2024). *In-Context Learning through the Bayesian Prism*. arXiv:2306.04891. - Müller, S. et al. (2021). *Transformers Can Do Bayesian Inference*. arXiv:2112.10510. - Garg, S. et al. (2022). *What Can Transformers Learn In-Context? A Case Study of Simple Function Classes*. arXiv:2208.01066. - Min, S. et al. (2022). *Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?*. arXiv:2202.12837. - Wang, X. et al. (2023). *Explaining and Finding Good Demonstrations for In-Context Learning*. arXiv:2302.13971. - Xie, S. et al. (2024). *A Survey on In-Context Learning*. arXiv:2301.00234. - Yu, Z.; Ananiadou, S. (2024). *How Do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads Are Two Towers for Metric Learning*. arXiv:2402.02872. - Wibisono, K. C.; Wang, Y. (2024). *From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When*. arXiv:2406.00131. - Chan, J. K. et al. (2022). *Data Distributional Properties Drive Emergent In-Context Learning in Transformers*. arXiv:2205.05055. - Hahn, M.; Goyal, N. (2023). *A Theory of Emergent In-Context Learning as Implicit Structure Induction*. arXiv:2303.07971. ## References 1. [↑](https://systems-analysis.info/eng/In-Context_Learning#cite_ref-lakera_1-0) What is In-Context Learning (ICL)? // Lakera.ai 2. [↑](https://systems-analysis.info/eng/In-Context_Learning#cite_ref-hopsworks_2-0) In-Context Learning (ICL) // Hopsworks.ai 3. [↑](https://systems-analysis.info/eng/In-Context_Learning#cite_ref-gradient_pub_3-0) In-Context Learning, In Context // The Gradient 4. [↑](https://systems-analysis.info/eng/In-Context_Learning#cite_ref-arxiv_grad_4-0) Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers // arXiv, 2022. 5. [↑](https://systems-analysis.info/eng/In-Context_Learning#cite_ref-stanford_5-0) Understanding In-Context Learning // Stanford Human-Centered AI, 2023.