Jamba (language model)

From Systems analysis Wiki
Jump to navigation Jump to search

Jamba is a family of large language models (LLMs) developed by the Israeli research company AI21 Labs. Jamba introduced one of the first production-grade hybrid architectures that combines key elements from two dominant approaches in AI development: Transformers and State Space Models (SSMs), specifically the Mamba architecture[1].

Jamba's primary goal is to address a fundamental trade-off in modern LLMs: the high quality and performance (characteristic of Transformers) versus the efficiency and ability to process ultra-long contexts (characteristic of SSMs). By combining these approaches and adding sparsity through Mixture-of-Experts (MoE), Jamba offers a model that is simultaneously powerful, efficient, and capable of handling vast amounts of text in a single query. Since the first release in March 2024, AI21 Labs has extended the family with several generations, from the large enterprise models (up to Jamba 1.7) to compact models such as Jamba Reasoning 3B and the Jamba2 family released in January 2026[2][3].

Jamba's Architecture in Detail

Jamba does not simply alternate between Transformer and Mamba layers. It employs a meticulously designed block structure, where each block consists of eight layers. Every layer combines a sequence-mixing component — either an attention (Transformer) block or a Mamba block — with a feed-forward (MLP) component[4].

Two independent design axes define the released model:

  • Attention-to-Mamba ratio (1:7): Of the eight layers in a block, one uses an attention (Transformer) mixer, responsible for "deep" understanding and precise recall, while the other seven use Mamba mixers, responsible for efficient sequence processing across a long context. This asymmetric split lets the model perform the heavy but powerful attention operations less frequently and the lightweight, fast Mamba operations more often[4].
  • MoE frequency (every other layer): Independently of the ratio above, the feed-forward component is replaced by a Mixture-of-Experts (MoE) module in every second layer[4][5].

Mixture-of-Experts (MoE) Integration

Jamba uses the MoE architecture to increase total model capacity without a matching increase in compute.

  • MoE replaces the single feed-forward (MLP) block in every other layer, regardless of whether that layer's mixer is attention or Mamba[4][6]. Because attention layers are rare (one per block), most MoE layers actually sit atop Mamba layers.
  • The first Jamba model had 16 experts.
  • For each token, a router network selects the top 2 experts (Top-2 gating).

This means that although the model's total parameter count is large (52 billion), only 2 of the 16 experts are active in any given MoE layer during each token's processing, keeping the number of active parameters — and therefore the computation — much smaller.

Evolution of Jamba Models

Jamba-v0.1 (March 2024)

The first model introduced in this family has the following specifications:

Jamba-v0.1 Technical Specifications
Specification Value
Total Parameters 52 billion
Active Parameters ~12 billion
Number of Experts (MoE) 16 (2 active)
Context Window 256,000 tokens
License Apache 2.0[7]

Thanks to its hybrid architecture, Jamba-v0.1 can process a context length of 256,000 tokens — equivalent to a novel of several hundred pages — and can be deployed on a single 80 GB GPU (with 8-bit weights)[4]. It was released as a pretrained base model (without instruction tuning) under the Apache 2.0 license, so that the community could study and build on the new architecture[4].

Jamba-Instruct (May 2024)

In May 2024, AI21 released Jamba-Instruct, an instruction-tuned version of the base model aligned for commercial use, with the same 256K context window. Unlike the rest of the family, it was offered only as a hosted model through AI21 Studio and cloud partners, without open weights, and was later superseded by the Jamba 1.5 family[8].

Jamba 1.5 (August 2024)

In August 2024, AI21 Labs introduced the instruction-tuned Jamba 1.5 family, comprising two versions: Jamba 1.5 Mini (12B active parameters out of 52B total) and Jamba 1.5 Large (94B active parameters out of 398B total)[9][10]. These models demonstrated significant improvements:

  • Up to 2.5 times faster inference on long contexts compared to models of comparable size.
  • Support for nine languages: English, Spanish, French, Portuguese, Italian, Dutch, German, Arabic, and Hebrew[11].

Beginning with this release, the flagship Mini and Large models (versions 1.5 through 1.7) are distributed under the Jamba Open Model License (a permissive license allowing research and commercial use under its terms) rather than Apache 2.0[11].

Jamba 1.6 (March 2025)

On March 6, 2025, AI21 released Jamba 1.6 in the same two sizes — Jamba Mini 1.6 (12B active / 52B total) and Jamba Large 1.6 (94B active / 398B total) — retaining the 256K context window and the Jamba Open Model License. AI21 positioned it as the leading open model for private enterprise deployment: it reported that Jamba Large 1.6 outperforms Mistral Large 2, Llama 3.3 70B, and Command R+ on quality, while Jamba Mini 1.6 outperforms Ministral 8B, Llama 3.1 8B, and Command R7B[12][13].

Jamba 1.7 (July 2025)

On July 3, 2025, AI21 released Jamba 1.7 (Jamba Large 1.7 and Jamba Mini 1.7), building on the same hybrid architecture and 256K context window with improvements in grounding (more complete, context-faithful answers) and instruction following, and expanded support for self-hosted deployment[14][15].

Jamba Reasoning 3B (October 2025)

On October 8, 2025, AI21 released Jamba Reasoning 3B, a compact open-source reasoning model built on the same SSM-Transformer architecture and presented as the first in a new series of small Jamba models. Despite its 3-billion-parameter size, it supports a 256K context window (and can handle up to 1 million tokens) and runs on consumer devices such as laptops and phones; according to AI21, it delivers 2–5x efficiency gains and outperforms comparable on-device models from DeepSeek, Google, Meta, and Microsoft, particularly on instruction following (IFBench) and general knowledge (MMLU-Pro, Humanity's Last Exam). Unlike the Jamba 1.5–1.7 models, Jamba Reasoning 3B is released under the Apache 2.0 license[2][16].

Jamba2 (January 2026)

On January 8, 2026, AI21 introduced the Jamba2 family — Jamba2 Mini (a MoE model with 12B active parameters out of 52B total) and the dense Jamba2 3B — compact open models optimized for reliability and steerability in enterprise workloads such as grounded question answering and agentic systems, without the "thinking-token" overhead of reasoning models[3]. Both models keep the 256K context window and are released under the Apache 2.0 license. They were built by post-training on top of the Jamba 1.5 pre-training base — mid-training on 500 billion curated tokens, a state-passing phase for the Mamba layers, supervised fine-tuning, DPO, and several on-policy reinforcement-learning phases — and, according to AI21, lead their categories on instruction-following (IFBench, IFEval, Collie) and grounding (FACTS) benchmarks[3][17]. Jamba2 3B, like Jamba Reasoning 3B, runs on consumer devices, including iPhones, Androids, Macs, and PCs[18]. In the AI21 Studio API, Jamba Mini 2 (snapshot 2026-01) replaced Jamba Mini 1.7, while the flagship Jamba Large remains at version 1.7[19].

Key Advantages and Performance

  • Massive Context Window: 256,000 tokens — at the time of its release, the longest context window among production-grade open models and one of the largest among all available models (including proprietary ones). According to the Jamba-1.5 technical report, the Jamba 1.5 models were also the only ones with a confirmed effective context length of 256K on the RULER benchmark[10]. This makes Jamba well suited to tasks requiring the analysis of large documents, such as legal contracts, scientific papers, entire codebases, or long dialogues[12].
  • High Performance and Efficiency: In benchmarks, Jamba demonstrates performance comparable to or exceeding that of leading open models of a similar size, such as Llama and Mixtral, while achieving roughly 3 times higher throughput on long contexts. Its efficiency comes largely from the hybrid design: at a 256K context, the base model needs only about 4 GB for its attention cache, against roughly 32 GB for Mixtral and 128 GB for Llama-2 7B[1][4].
  • Openness and Accessibility: Jamba is distributed under permissive licenses — Apache 2.0 for Jamba-v0.1, Jamba Reasoning 3B, and the Jamba2 models, and the Jamba Open Model License for the Jamba 1.5, 1.6, and 1.7 models — allowing free use in commercial and research applications. The model weights are available on the Hugging Face platform[13][3].

Benchmark Results

Jamba models show competitive results on various benchmarks:

  • Jamba 1.5 Mini scored 46.1 on Arena Hard, the strongest open model in its size class at release[9].
  • Jamba 1.5 Large scored 65.4 on Arena Hard, outperforming Llama 3.1 70B and 405B[9].
  • The later Jamba 1.6 improved these figures, with Jamba Large 1.6 scoring 76.5 and Jamba Mini 1.6 scoring 51.2 on Arena Hard, and leading on long-context benchmarks such as CRAG and FinanceBench[12].

Applications and Availability

Jamba is optimized for business applications and supports capabilities such as function calling, structured JSON output, grounded generation (RAG), and document processing. The models are available on multiple platforms, including:

  • Hugging Face
  • Google Cloud Vertex AI
  • Microsoft Azure
  • NVIDIA API catalog
  • Amazon Bedrock[20]
  • AI21 Studio

The compact Jamba Reasoning 3B is additionally distributed for local inference via Kaggle, LM Studio, and llama.cpp[2].

From Jamba 1.6 onward, AI21 has emphasized private, secure deployment, allowing enterprises to run the models entirely on-premises or in a virtual private cloud (VPC) so that sensitive data is never exposed to the model vendor[12].

To support cost-effective inference, AI21 Labs introduced ExpertsInt8, a quantization technique for MoE models that quantizes only the expert (MLP) weights to INT8 and dequantizes them at runtime inside the GPU kernel. It allows Jamba 1.5 Large (and later versions) to be served on a single node with eight 80 GB GPUs, retaining the full 256K-token context without quality loss[10][21].

Literature

  • Lieber, O.; et al. (2024). Jamba: A Hybrid Transformer‑Mamba Language Model. arXiv:2403.19887.
  • Jamba Team (2024). Jamba‑1.5: Hybrid Transformer‑Mamba Models at Scale. arXiv:2408.12570.
  • Gu, A.; Dao, T. (2023). Mamba: Linear‑Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.
  • Gu, A.; Goel, K.; Ré, C. (2021). Efficiently Modeling Long Sequences with Structured State Spaces (S4). arXiv:2111.00396.
  • Fedus, W.; et al. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961.
  • Yun, L.; et al. (2024). Toward Inference‑Optimal Mixture‑of‑Expert Large Language Models. arXiv:2404.02852.
  • Cai, W.; et al. (2024). A Survey on Mixture of Experts in Large Language Models. arXiv:2407.06204.
  • Gupta, V.; et al. (2024). Lynx: Enabling Efficient MoE Inference through Dynamic Batch‑Aware Expert Selection. arXiv:2411.08982.
  • Liu, J.; et al. (2024). A Survey on Inference Optimization Techniques for Mixture of Experts Models. arXiv:2412.14219.
  • Hsieh, C.‑P.; et al. (2024). RULER: What's the Real Context Size of Your Long‑Context Language Models?. arXiv:2404.06654.

References

  1. ↑ 1.0 1.1 "Introducing Jamba: AI21's Groundbreaking SSM-Transformer Model". AI21 Labs Blog. [1]
  2. ↑ 2.0 2.1 2.2 "Introducing Jamba Reasoning 3B: Tiny Model, Huge Possibilities". AI21 Labs Blog. [2]
  3. ↑ 3.0 3.1 3.2 3.3 "Introducing Jamba2: The open source model family for enterprise reliability and efficiency". AI21 Labs Blog. [3]
  4. ↑ 4.0 4.1 4.2 4.3 4.4 4.5 4.6 Lieber, O., et al. (2024). Jamba: A Hybrid Transformer-Mamba Language Model. arXiv:2403.19887.
  5. ↑ "Jamba 1.5 LLMs Leverage Hybrid Architecture to Deliver Superior Reasoning and Long Context Handling". NVIDIA Technical Blog. [4]
  6. ↑ "Jamba Documentation". Hugging Face Transformers. [5]
  7. ↑ "ai21labs/Jamba-v0.1". Hugging Face. [6]
  8. ↑ "Built for the Enterprise: Introducing AI21's Jamba-Instruct Model". AI21 Labs Blog. [7]
  9. ↑ 9.0 9.1 9.2 "Announcing the Jamba-1.5 model family". AI21 Labs Blog. [8]
  10. ↑ 10.0 10.1 10.2 Jamba Team (2024). Jamba-1.5: Hybrid Transformer-Mamba Models at Scale. arXiv:2408.12570.
  11. ↑ 11.0 11.1 "ai21labs/AI21-Jamba-Large-1.5". Hugging Face. [9]
  12. ↑ 12.0 12.1 12.2 12.3 "AI21's Jamba 1.6: The Best Open Model for Private Enterprise Deployment". AI21 Labs Blog. [10]
  13. ↑ 13.0 13.1 "ai21labs/AI21-Jamba-Large-1.6". Hugging Face. [11]
  14. ↑ "ai21labs/AI21-Jamba-Large-1.7". Hugging Face. [12]
  15. ↑ "Changelog". AI21 Labs Documentation. [13]
  16. ↑ "ai21labs/AI21-Jamba-Reasoning-3B". Hugging Face. [14]
  17. ↑ "ai21labs/AI21-Jamba2-Mini". Hugging Face. [15]
  18. ↑ "ai21labs/AI21-Jamba2-3B". Hugging Face. [16]
  19. ↑ "Jamba Foundation Models". AI21 Labs Documentation. [17]
  20. ↑ "Jamba-1.5 family of models by AI21 Labs is now available in Amazon Bedrock". AWS What's New. [18]
  21. ↑ Jamba Team (2025). "Jamba: Hybrid Transformer-Mamba Language Models". ICLR 2025 (OpenReview). [19]
Retrieved from "https://systems-analysis.info/eng/index.php?title=Jamba_(language_model)&oldid=196"