--- title: "Multimodal large language models" source: "https://systems-analysis.info/eng/Multimodal_large_language_models" wiki: "systems-analysis.info/eng" article: "Multimodal_large_language_models" language: "en" categories: - "Category:English" - "Category:Large language models" - "Category:Machine learning" - "Category:Technology" revision_id: 265 wiki_created_at: 2026-09-06T22:19:45Z wiki_modified_at: 2026-09-06T22:19:45Z downloaded_at: 2026-09-07T22:22:18Z --- # Multimodal large language models **Multimodal Large Language Models (MLLMs)** are a class of artificial intelligence models capable of processing and generating information across various modalities, including text, images, audio, and video^{[\[1\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-encord_intro-1)}. Unlike unimodal language models, which work exclusively with text, MLLMs integrate information from different sources to solve complex content understanding and generation tasks. The core concept behind MLLMs is the creation of a unified vector representation (embedding) for different modalities. This allows the model to establish semantic connections between, for example, an image and its textual description^{[\[2\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-acm_survey-2)}. A key breakthrough that laid the foundation for modern MLLMs was the use of contrastive learning to align visual and text representations in a shared feature space, as implemented in the **CLIP** model^{[\[3\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-radford2021-3)}. ## History of Development ### Early Period (2013–2020) The conceptual foundations of multimodal AI were laid in 2013, when researchers from Stanford demonstrated the feasibility of zero-shot learning using word vector representations^{[\[4\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-deoldify2013-4)}. In 2016, the **FAIR** (Meta AI) team showed the effectiveness of using natural language descriptions to train computer vision models, achieving 11.5% accuracy on ImageNet without direct training^{[\[5\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-openai_fair_2016-5)}. ### The CLIP Era (2021) A revolutionary moment was the release of the **CLIP** (*Contrastive Language-Image Pre-training*) model by OpenAI in January 2021. The model, trained on 400 million image-text pairs, demonstrated the ability to classify images without specialized training on specific tasks. CLIP became the foundation for many subsequent MLLMs^{[\[6\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-stanford_cs_clip-6)}. ### Scaling and Innovation (2022–2024) Following the success of CLIP, numerous key models emerged: - **Flamingo** (DeepMind, 2022) — an 80-billion-parameter model that showed outstanding few-shot learning capabilities. - **BLIP** (Salesforce, 2022) — a unified architecture for understanding and generation. - **GPT-4V** (OpenAI, 2023) — the first commercial multimodal model of its scale. - **LLaVA** (Microsoft, 2023) — a popular open-source alternative to GPT-4V. - **Gemini** (Google, 2023) — a natively multimodal architecture, designed from the ground up to work with different data types. - **GPT-4o** (OpenAI, 2024) — a model capable of processing text, audio, and video in real time with low latency^{[\[1\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-encord_intro-1)}. - **Claude 3.5 Sonnet** (Anthropic, 2024) — a model with improved capabilities for visual information analysis. ## Architectural Approaches ### Dual-Encoder Architecture This uses separate encoders for each modality, which project data into a shared representation space. A prime example is **CLIP**, where a vision transformer processes images and a text transformer processes language data. Its advantages include modularity and computational efficiency, while its disadvantage is limited cross-modal interaction^{[\[7\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-viso_ai_mllm-7)}. ### Encoder-Decoder Architecture A single encoder processes the multimodal input, and a decoder generates the text output. The **Flamingo** model uses a *Perceiver Resampler* mechanism to handle variable-length visual inputs and cross-attention layers. This approach enables rich inter-modal interaction but requires significant computational resources^{[\[8\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-determined_ai_arch-8)}. ### Alignment Architecture This approach uses frozen, pre-trained encoders connected via a small, trainable alignment module. For example, **BLIP-2** uses a **Q-Former** (*Querying Transformer*) as a lightweight bridge between a frozen vision encoder and a language model, requiring significantly fewer trainable parameters^{[\[9\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-clarifai_blip2-9)}. ## Key Models ### GPT-4V / GPT-4o (OpenAI) The GPT-4 family of models is estimated to have up to **1.8 trillion** parameters (in a mixture-of-experts architecture). The **GPT-4o** model, released in May 2024, supports real-time processing of text, images, audio, and video. On the **MMMU** benchmark, it achieves **69.1%** accuracy^{[\[10\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-encord_mmmu_perf-10)}. ### Gemini (Google) A natively multimodal architecture, trained from scratch on text, images, audio, and video. **[Gemini 1.5 Pro](https://systems-analysis.info/eng/Gemini_(Google) "Gemini (Google)")** supports a context window of up to **10 million tokens** and outperforms GPT-4 on 30 out of 32 popular benchmarks^{[\[11\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-daveai_gemini-11)}. ### Claude 3 (Anthropic) A family of models (Haiku, Sonnet, Opus) with a context window of up to 200,000 tokens. **[Claude 3 Opus](https://systems-analysis.info/eng/Claude_(Anthropic) "Claude (Anthropic)")** scores **58.5%** on the MMMU benchmark. To enhance model safety, the [Constitutional AI](https://systems-analysis.info/eng/Constitutional_AI "Constitutional AI") approach is used^{[\[12\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-anthropic_claude3-12)}. ### LLaVA (Open-Source Model) Combines the CLIP vision encoder with the Vicuna language model. Variants with 7, 13, and 34 billion parameters are available. The model achieves 85.1% of GPT-4's relative performance on synthetic tasks^{[\[13\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-llava_paper-13)}. ## Application Areas - **Visual Question Answering (VQA)**: Allows users to ask questions about visual content. - **Document Analysis**: Modern MLLMs can process up to 2,000 pages per minute. - **Medical Imaging**: Models like **Med-PaLM M** (Google) analyze medical images and clinical data. - **Robotics**: Models like **RT-2** (Google DeepMind) enable robots to understand visual environments and execute commands given in natural language. ## Current Limitations - **Hallucinations**: The hallucination rate in generated content is estimated to be between 27–46%. Models may describe non-existent objects or misinterpret visual information^{[\[14\]](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_note-arxiv_hallucinations-14)}. - **High Computational Requirements**: Training and using MLLMs require significant computational infrastructure. - **Data Bias**: Underrepresentation of demographic groups, languages, and cultures in training data leads to systematic errors. ## External links - A Comprehensive Guide to Multimodal LLMs (Encord Blog) - Multimodal LLMs: The Complete Guide (Viso.ai) - Multimodal learning — Wikipedia ## See also - [Multimodal reasoning](https://systems-analysis.info/eng/Multimodal_reasoning "Multimodal reasoning") - [PanGu (Huawei)](https://systems-analysis.info/eng/PanGu_(Huawei) "PanGu (Huawei)") - [Phi (Microsoft)](https://systems-analysis.info/eng/Phi_(Microsoft) "Phi (Microsoft)") ## Literature - Radford, A. et al. (2021). *Learning Transferable Visual Models From Natural Language Supervision*. arXiv:2103.00020. - Alayrac, J.-B. et al. (2022). *Flamingo: a Visual Language Model for Few-Shot Learning*. arXiv:2204.14198. - Li, J. et al. (2022). *BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation*. arXiv:2201.12086. - Li, J. et al. (2023). *BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models*. arXiv:2301.12597. - Liu, H. et al. (2023). *Visual Instruction Tuning*. arXiv:2304.08485. - Driess, K. et al. (2023). *PaLM-E: An Embodied Multimodal Language Model*. arXiv:2303.03378. - Brohan, A. et al. (2023). *RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control*. arXiv:2307.15818. - Yue, X. et al. (2023). *MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI*. arXiv:2311.16502. - Tsimpoukelli, M. et al. (2021). *Multimodal Few-Shot Learning with Frozen Language Models*. arXiv:2106.13884. - Singhal, K. et al. (2023). *Med-PaLM 2: Towards Expert-Level Medical Question Answering with Large Language Models*. arXiv:2305.09617. - Yin, S. et al. (2023). *A Survey on Multimodal Large Language Models*. arXiv:2306.13549. ## References 1. ↑ ^{[1.0](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-encord_intro_1-0)} ^{[1.1](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-encord_intro_1-1)} “A Comprehensive Guide to Multimodal LLMs”. *Encord Blog*. [1] 2. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-acm_survey_2-0) “A Survey on Multimodal Large Language Models”. *ACM Computing Surveys*. [2] 3. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-radford2021_3-0) Radford, A., et al. “Learning Transferable Visual Models From Natural Language Supervision”. *arXiv:2103.00020*. [3] 4. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-deoldify2013_4-0) DeOldify, J. “Zero-Shot Learning by Predicting Attributes”. *arXiv:1312.5650*. [4] 5. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-openai_fair_2016_5-0) “Learning from captions: A milestone in visual language understanding”. *OpenAI Blog*. [5] 6. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-stanford_cs_clip_6-0) “Understanding CLIP”. *Stanford CS231n*. [6] 7. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-viso_ai_mllm_7-0) “Multimodal LLMs: The Complete Guide”. *Viso.ai*. [7] 8. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-determined_ai_arch_8-0) “The Architectures of Multimodal Language Models”. *Determined AI*. [8] 9. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-clarifai_blip2_9-0) “Understanding BLIP-2: The New Vision-Language Model”. *Clarifai Blog*. [9] 10. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-encord_mmmu_perf_10-0) “MMMU: A New Benchmark for Multimodal LLMs”. *Encord Blog*. [10] 11. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-daveai_gemini_11-0) “Google Gemini: A Deep Dive”. *DaveAI Blog*. [11] 12. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-anthropic_claude3_12-0) “Introducing the Claude 3 Family”. *Anthropic*. [12] 13. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-llava_paper_13-0) Liu, H., et al. “Visual Instruction Tuning”. *arXiv:2304.08485*. [13] 14. [↑](https://systems-analysis.info/eng/Multimodal_large_language_models#cite_ref-arxiv_hallucinations_14-0) “Hallucinations in Multimodal Large Language Models”. *arXiv:2308.08726*. [14]