---
title: "Nemotron (NVIDIA)"
source: "https://systems-analysis.info/eng/Nemotron_(NVIDIA)"
wiki: "systems-analysis.info/eng"
article: "Nemotron_(NVIDIA)"
language: "en"
categories:
- "Category:English"
- "Category:Large language models"
- "Category:LLM families"
- "Category:Machine learning"
- "Category:Technology"
revision_id: 268
wiki_created_at: 2026-09-06T22:19:48Z
wiki_modified_at: 2026-09-06T22:19:48Z
downloaded_at: 2026-09-07T22:22:20Z
---
# Nemotron (NVIDIA)
**Nemotron** is a family of open-weight [large language models](https://systems-analysis.info/eng/Large_language_model "Large language model"), multimodal models, reward models, retrieval models, and agent-oriented systems developed by NVIDIA. Major contributions have come from NVIDIA's Applied Deep Learning Research organization and the teams responsible for NeMo, Megatron-LM, TensorRT-LLM, ModelOpt, and NVIDIA Inference Microservices.
The name has been used for several technically distinct generations. These include dense decoder-only Transformers such as Nemotron-4, models adapted from Meta's Llama family, dense hybrid Mamba–Transformer models such as Nemotron-H and Nemotron Nano 2, and sparse hybrid Mamba–Attention [Mixture-of-Experts](https://systems-analysis.info/eng/Mixture-of-Experts_(MoE) "Mixture-of-Experts (MoE)") (MoE) models in the Nemotron 3 generation. The wider ecosystem also includes systems for visual and document understanding, speech, retrieval, safety classification, synthetic-data generation, reward modelling, model compression, and experimental diffusion-language modelling.
As of 10 July 2026, NVIDIA's principal Nemotron 3 reasoning tiers are:[\[1\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronModels-1)
- **Nemotron 3 Nano** — approximately 31.6 billion total parameters and 3.2 billion non-embedding parameters activated per token;
- **Nemotron 3 Super** — approximately 120.6 billion total and 12.7 billion active parameters;
- **Nemotron 3 Ultra** — approximately 550 billion total and 55 billion active parameters.
All three use hybrid Mamba-2, attention, and MoE components and support contexts of up to approximately one million tokens under the documented long-context configurations.[\[2\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3Whitepaper-2)[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3)[\[4\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperReport-4)[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
**Nemotron 3 Nano Omni**, released in April 2026, extends the Nano backbone with native text, image, video, and audio inputs and a 256,000-token multimodal context.[\[6\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoOmniReport-6) **Nemotron-Labs-3-Puzzle-75B-A9B**, released on 6 July 2026, is a compressed Super derivative optimized for interactive and high-concurrency deployment.[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7) NVIDIA described Ultra, released on 4 June 2026, as the largest and most capable model in the three-tier Nemotron 3 family.[\[8\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-UltraCard-8)
Nemotron releases do not use one family-wide licence. Depending on the checkpoint, governing terms include the NVIDIA AI Foundation Models Community License, NVIDIA Open Model License Agreement, NVIDIA Nemotron Open Model License, NVIDIA Open Model Agreement, OpenMDW License Agreement 1.1, and model-specific terms associated with Llama-derived releases. Licence terms must therefore be checked for the exact checkpoint rather than inferred from the Nemotron name alone.
## History and background
### NVIDIA model-development stack
Nemotron was developed in the context of NVIDIA's broader generative-AI stack. Relevant components include:
- DGX and HGX accelerator systems;
- Megatron-LM and NeMo Framework for distributed model training;
- NeMo Aligner and NeMo RL for post-training;
- NeMo Curator for data processing;
- NeMo Gym and NeMo Evaluator for environments and evaluation;
- ModelOpt for pruning, distillation, and quantization;
- TensorRT-LLM, vLLM, and SGLang integrations for inference;
- NVIDIA NIM for containerized model serving;
- NeMo Guardrails and safety classifiers for runtime controls.
These products and libraries are separable from the model weights. A downloadable checkpoint can be used through a third-party inference framework without adopting the complete NVIDIA deployment stack, subject to the checkpoint's licence and software compatibility.[\[9\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NeMoDocs-9)
### Naming ambiguity
The name **Nemotron-3** was first used for an eight-billion-parameter model released in 2023. NVIDIA reused the similar name **Nemotron 3** in December 2025 for a new generation based on hybrid Mamba–Attention MoE architectures. The 2023 model is not the Nano tier of the 2025–2026 generation and does not share its architecture, scale, or context length.
The hyphenation in official model names also varies. For clarity, this article uses:
- **Nemotron-3 8B** for the 2023 model;
- **Nemotron-4** for the 2024 dense models;
- **Nemotron 3 Nano, Super, and Ultra** for the 2025–2026 generation;
- **Nemotron-Labs** for experimental research checkpoints that are not presented as the principal production tiers.
### Nemotron-3 8B (2023)
The first publicly distributed Nemotron-branded language model was Nemotron-3-8B-Base-4K, a dense decoder-only Transformer with an approximately 4,096-token context. NVIDIA reported that it was trained on approximately 3.8 trillion tokens covering 53 natural languages and 37 programming languages, using 1,024 NVIDIA A100 GPUs for approximately 19 days.[\[10\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3_8BCard-10)
The release included base and conversational variants trained through supervised fine-tuning and SteerLM. It was distributed under the NVIDIA AI Foundation Models Community License. NVIDIA did not publish a standalone arXiv technical report for this checkpoint.
### Nemotron-4 15B (2024)
Nemotron-4 15B was documented in February 2024. It is a dense decoder-only Transformer with approximately 15 billion parameters, a 4,096-token context, Grouped-Query Attention (GQA), Rotary Position Embeddings (RoPE), and a 256,000-token vocabulary.[\[11\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_15B-11)
The model was trained on approximately eight trillion tokens. NVIDIA reported a training configuration of as many as 384 DGX H100 systems, or 3,072 H100 GPUs, over approximately 13 calendar days. Peak Model FLOPs Utilization was reported as 34.3% in the stated configuration.
The report emphasized multilingual performance and compared the model with other open-weight systems available in early 2024. These comparisons are historical and should not be interpreted as a ranking against later model generations.
### Nemotron-4 340B (2024)
Nemotron-4 340B was released in June 2024 in Base, Instruct, and Reward variants.[\[12\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_340B-12)
The dense model contains approximately 340 billion parameters, including approximately 331.6 billion non-embedding parameters. It was trained on approximately nine trillion tokens:
- about eight trillion tokens in the main pre-training run;
- about one trillion tokens of continued pre-training with a greater weight on selected high-quality sources.
The report describes training on as many as 768 DGX H100 systems, or 6,144 H100 GPUs, between December 2023 and May 2024. Peak Model FLOPs Utilization was reported as 42.4%.
NVIDIA selected the model size so that an FP8 checkpoint could be deployed on one eight-GPU DGX H100 node. This statement applies to the documented weight precision and deployment configuration, not to arbitrary context lengths or serving batch sizes.
More than 98% of the data used in the model's alignment pipeline was synthetically generated. The Instruct and Reward models were intended in part to form a reproducible synthetic-data-generation system: one checkpoint generated candidate answers and another scored or filtered them.[\[12\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_340B-12)
### Llama-3.1-Nemotron-70B (2024)
In October 2024, NVIDIA released Llama-3.1-Nemotron-70B-Instruct and Llama-3.1-Nemotron-70B-Reward, derived from Meta's Llama 3.1 70B Instruct model.
The Instruct checkpoint was optimized with reinforcement learning using the Nemotron reward model. NVIDIA reported scores of 85.0 on Arena-Hard, 57.6% on length-controlled AlpacaEval 2, and 8.98 on MT-Bench under the evaluation configurations used in October 2024.[\[13\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-LN70BInstruct-13)
The Reward checkpoint combined Bradley–Terry pairwise preference modelling with multi-attribute regression based on HelpSteer2. NVIDIA reported 94.1% on RewardBench at release.[\[14\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-LN70BReward-14)
These leaderboard values are date- and harness-specific. They do not imply that the model remained first after later systems and revised evaluation protocols were introduced.
### Transition to reasoning and hybrid architectures (2025)
In 2025 NVIDIA expanded the family in three main directions.
**Llama-Nemotron** applied neural architecture search, compression, continued pre-training, supervised fine-tuning, and reinforcement learning to Llama 3.x checkpoints. The main variants were Nano 8B, Super 49B, and Ultra 253B.[\[15\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-LlamaNemotron-15)
**Nemotron-H** introduced dense hybrid Mamba-2–Transformer models at 8B and 56B scale, together with a 47B compressed derivative. Most attention layers were replaced by Mamba-2 sequence-model layers to reduce decode-time cache and computation.[\[16\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronH-16)
**Nemotron Nano 2** developed the hybrid approach into a nine-billion-parameter reasoning model. Its 12B parent checkpoint was trained on approximately 20 trillion tokens in FP8 and compressed through an extended Minitron procedure.[\[17\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Nano2-17)
### Nemotron 3 generation (2025–2026)
NVIDIA announced the Nemotron 3 family on 15 December 2025. The initial white paper described planned Nano, Super, and Ultra tiers, but only Nano was released at announcement. Super and Ultra were subsequently released in March and June 2026 and should no longer be described as forthcoming.[\[2\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3Whitepaper-2)
| Date | Release | Main characteristics |
|------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|
| 15 December 2025 | **Nemotron 3 Nano** | 31.6B total / 3.2B active; hybrid Mamba-2, attention, and conventional MoE; up to 1M context |
| 11 March 2026 | **Nemotron 3 Super** | 120.6B total / 12.7B active; LatentMoE, native MTP, mixed NVFP4 pre-training; up to 1M context |
| March 2026 | **Nemotron-Cascade 2** | Research post-training checkpoint at approximately 30B total / 3B active, using Cascade RL and multi-domain on-policy distillation |
| 28 April 2026 | **Nemotron 3 Nano Omni** | Native text, image, video, and audio inputs; 30B-A3B-class language backbone; 256K context |
| 4 June 2026 | **Nemotron 3 Ultra** | 550B total / 55B active; LatentMoE, MTP, NVFP4 pre-training, multi-environment RLVR, MOPD; up to 1M context |
| 25 June 2026 | **Nemotron-Labs-TwoTower** | Experimental diffusion-language model based on two copies of the Nemotron 3 Nano backbone |
| 6 July 2026 | **Nemotron-Labs-3-Puzzle-75B-A9B** | Deployment-oriented compression of Super; approximately 75.3B total / 9.3B active |
| 7 July 2026 | **Nemotron-Labs-Diffusion** | Experimental 3B, 8B, and 14B tri-mode models supporting autoregressive, diffusion, and self-speculative decoding |
[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3)[\[18\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperCard-18)[\[6\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoOmniReport-6)[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)[\[19\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Cascade2-19)[\[20\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-TwoTower-20)[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7)[\[21\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Diffusion-21)
## Theoretical foundations
### Dense decoder-only Transformer architecture
Nemotron-4 models use a standard autoregressive decoder-only Transformer. For a token sequence $x_{1},\ldots,x_{T}$, the language-model probability is factorized as:
$$
p(x_{1},\ldots,x_{T}) = \prod\limits_{t = 1}^{T}p_{\theta}\left( x_{t} \mid x_{1},\ldots,x_{t - 1} \right),
$$
where $\theta$ denotes the model parameters.
Scaled dot-product attention is:
$$
\operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \operatorname{softmax}\left( \frac{\mathbf{Q}\mathbf{K}^{\mathsf{T}}}{\sqrt{d_{k}}} \right)\mathbf{V},
$$
where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ are query, key, and value matrices.
Grouped-Query Attention uses more query heads than key–value heads. Several query heads share one key–value head, reducing the size and memory bandwidth of the KV cache relative to full multi-head attention.[\[22\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-GQA-22)
Nemotron-4 uses Rotary Position Embeddings, which rotate pairs of hidden dimensions according to token position. In simplified two-dimensional form:
$$
\operatorname{RoPE}(\mathbf{x}_{m},m) = \begin{pmatrix}
{x_{m}^{(1)}\cos(m\theta_{1}) - x_{m}^{(2)}\sin(m\theta_{1})} \\
{x_{m}^{(1)}\sin(m\theta_{1}) + x_{m}^{(2)}\cos(m\theta_{1})} \\
\vdots
\end{pmatrix}.
$$
Nemotron-4 feed-forward layers use squared ReLU:
$$
f(x) = \left( \max(0,x) \right)^{2}.
$$
The documented models omit dropout and most bias terms and use separate input and output embedding matrices.[\[11\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_15B-11)[\[12\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_340B-12)
| Parameter | Nemotron-4 15B | Nemotron-4 340B |
|------------------------------|----------------|-----------------|
| Approximate total parameters | 15B | 340B |
| Transformer layers | 32 | 96 |
| Hidden dimension | 6,144 | 18,432 |
| Query heads | 48 | 96 |
| Key–value heads | 8 | 8 |
| Vocabulary | 256,000 | 256,000 |
| Published context | 4,096 tokens | 4,096 tokens |
### Hybrid Mamba–Transformer architecture
Nemotron-H and later hybrid models replace most self-attention layers with Mamba-2 state-space layers. A simplified linear state-space recurrence is:
$$
\mathbf{h}_{t} = \mathbf{A}_{t}\mathbf{h}_{t - 1} + \mathbf{B}_{t}\mathbf{x}_{t},\qquad\mathbf{y}_{t} = \mathbf{C}_{t}\mathbf{h}_{t},
$$
where $\mathbf{x}_{t}$ is the input, $\mathbf{h}_{t}$ is a recurrent state, and $\mathbf{y}_{t}$ is the output. In Mamba and Mamba-2, important state-space parameters depend on the input; the model is therefore more expressive than a fixed linear time-invariant recurrence.[\[23\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Mamba2-23)
During autoregressive generation, a conventional attention layer must read a growing KV cache. A recurrent Mamba layer instead maintains a fixed-size state. This can substantially reduce decode-time memory traffic. The complete hybrid model does not have constant cost independent of sequence length because it still contains attention layers, embeddings, expert layers, and other operations.
Nemotron-H-56B uses 54 Mamba-2 layers, 54 feed-forward layers, and 10 attention layers. Nemotron-H-8B uses 24 Mamba-2 layers, 24 feed-forward layers, and four attention layers. NVIDIA reported training the 56B model on approximately 20 trillion tokens and the 8B model on approximately 15 trillion tokens.[\[16\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronH-16)
### Mixture-of-Experts routing
The Nemotron 3 generation adds sparse MoE capacity. For token representation $\mathbf{h}$, a simplified MoE layer is:
$$
\operatorname{MoE}(\mathbf{h}) = \sum\limits_{i \in \mathcal{S}(\mathbf{h})}g_{i}(\mathbf{h})E_{i}(\mathbf{h}),
$$
where:
- $E_{i}$ is an expert feed-forward network;
- $\mathcal{S}(\mathbf{h})$ is the selected subset of experts;
- $g_{i}(\mathbf{h})$ is the routing weight.
A simplified router may be written as:
$$
g(\mathbf{h}) = \operatorname{TopK}\left( \sigma\left( W_{g}\mathbf{h} \right) \right),
$$
where $\sigma$ is a sigmoid gate and $W_{g}$ is a learned projection.
Sparse activation lowers the number of expert parameters used for an individual token. It does not remove the need to store or distribute the full expert pool, and it introduces routing, communication, load-balancing, and memory-bandwidth costs.
### Nemotron 3 Nano architecture
Nemotron 3 Nano contains approximately 31.6 billion total parameters and 3.2 billion active non-embedding parameters per token; some model-card totals include embeddings and are therefore slightly higher.[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3)
The 52-layer architecture contains:
- 23 blocks combining Mamba-2 and MoE;
- 23 additional MoE blocks;
- six GQA attention blocks.
Each MoE layer has 128 routed experts and one shared expert. Six routed experts are selected for each token. The model uses sigmoid routing and auxiliary-loss-free load balancing.
Selected architectural values are:
| Parameter | Nemotron 3 Nano |
|------------------------------------|------------------|
| Total layers | 52 |
| Hidden dimension | 2,688 |
| Routed experts per MoE layer | 128 |
| Routed experts activated per token | 6 |
| Shared experts | 1 |
| Expert hidden dimension | 1,856 |
| Mamba state dimension | 128 |
| Maximum documented context | 1,048,576 tokens |
The model uses RMSNorm, untied embeddings, no dropout, and no bias in its main linear layers. Mamba blocks do not require an explicit absolute positional embedding because order is represented by the recurrent state. Attention blocks serve as periodic global interaction points.[\[24\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-RMSNorm-24)
### LatentMoE in Super and Ultra
Nemotron 3 Super and Ultra replace conventional expert feed-forward layers with **LatentMoE**. The design first projects the token representation to a lower-dimensional latent space and performs expert computation and routing around that representation. The goal is to increase useful expert capacity without scaling expert storage, bandwidth, and communication in direct proportion to the model's full hidden dimension.
LatentMoE is intended to optimize both:
- **accuracy per FLOP** — quality relative to arithmetic computation;
- **accuracy per parameter or byte** — quality relative to stored weights, memory bandwidth, and expert-sharding overhead.
Nemotron 3 Super has the following published architecture:[\[4\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperReport-4)
| Parameter | Nemotron 3 Super |
|--------------------------------------|--------------------------------------------------------|
| Total parameters | 120.6B |
| Active parameters per forward pass | 12.7B including embeddings; 12.1B excluding embeddings |
| Total layers | 88 |
| Hidden dimension | 4,096 |
| Query heads / key–value heads | 32 / 2 |
| Mamba state dimension | 128 |
| Routed experts per layer | 512 |
| Routed experts activated per token | 22 |
| Expert hidden dimension | 2,688 |
| Shared expert intermediate dimension | 5,376 |
| MoE latent dimension | 1,024 |
| Multi-Token Prediction layers | 2, with shared weights |
| Maximum documented context | 1,048,576 tokens |
The 88-block parent layout used by Super consists of 40 Mamba blocks, 40 MoE blocks, and eight attention blocks.[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7)
Nemotron 3 Ultra scales the same design:[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
| Parameter | Nemotron 3 Ultra |
|--------------------------------------|------------------------|
| Total parameters | Approximately 550B |
| Active parameters per token | Approximately 55B |
| Total layers | 108 |
| Hidden dimension | 8,192 |
| Query heads / key–value heads | 64 / 2 |
| Mamba state dimension | 128 |
| Mamba heads | 256 |
| Routed experts per layer | 512 |
| Routed experts activated per token | 22 |
| Expert hidden dimension | 5,120 |
| Shared expert intermediate dimension | 10,240 |
| MoE latent dimension | 2,048 |
| Multi-Token Prediction layers | 2, with shared weights |
| Maximum documented context | 1,048,576 tokens |
### Multi-Token Prediction
Multi-Token Prediction (MTP) adds auxiliary heads trained to predict multiple future tokens. During speculative decoding, the auxiliary component drafts several candidate tokens and the principal model verifies them. Accepted drafts reduce the number of full-model decoding iterations.
For a primary next-token loss $\mathcal{L}_{1}$ and auxiliary future-token losses $\mathcal{L}_{2},\ldots,\mathcal{L}_{K}$, a simplified objective is:
$$
\mathcal{L}_{MTP} = \mathcal{L}_{1} + \sum\limits_{k = 2}^{K}\lambda_{k}\mathcal{L}_{k},
$$
where $\lambda_{k}$ controls the contribution of each future prediction.
Real speedup depends on draft acceptance rate, batch size, sequence length, hardware, runtime implementation, and the cost of the verifier. An MTP head does not guarantee that every forward pass produces several accepted output tokens.
Super and Ultra use two shared-weight MTP layers. Puzzle reuses and further trains the Super MTP component, with its report describing a 25–30% improvement in average accepted draft length after additional training.[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7)
### NVFP4 training and quantization
NVFP4 is an NVIDIA four-bit floating-point format intended for Blackwell-class hardware. Nemotron 3 Super and Ultra were pretrained with a mixed-precision NVFP4 recipe rather than being trained entirely in four-bit arithmetic.
The Ultra report describes:
- E2M1 four-bit values for selected matrix multiplications;
- two-dimensional block quantization;
- stochastic rounding for gradients;
- random Hadamard transforms in selected gradient computations;
- higher precision for sensitive components, including embeddings, attention projections, latent projections, MTP layers, Mamba output projections, and the final portion of the network.
NVIDIA reported an average training-loss difference below approximately 0.4% in selected NVFP4-versus-BF16 continuation studies. This is a training-loss comparison under the developer's procedure, not a universal estimate of downstream accuracy loss.[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
Post-training checkpoints are also distributed in BF16, FP8, and NVFP4 variants. Quantized variants can differ in long-context quality, supported kernels, storage, throughput, and hardware requirements.
### Model compression: Minitron, Puzzle, and MiniPuzzle
**Minitron** combines structured pruning and knowledge distillation. Pruning can remove complete layers or reduce hidden, feed-forward, and attention dimensions. NVIDIA reported that deriving four- and eight-billion-parameter models from Nemotron-4 15B reduced the required training computation substantially relative to training the same compact architectures from scratch.[\[25\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Minitron-25)
**Puzzle** is a block-level neural architecture search method. Candidate replacements are evaluated through local distillation, after which an optimization procedure selects a combination satisfying a deployment constraint. Puzzle was used to derive the 49B Llama-Nemotron Super from Llama 3.3 70B and the 253B Llama-Nemotron Ultra from Llama 3.1 405B.[\[26\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Puzzle-26)
**MiniPuzzle** extends this approach to hybrid Mamba–Transformer models. Nemotron-H-47B was derived from Nemotron-H-56B with approximately 63 billion distillation tokens. NVIDIA reported approximately 20% higher output speed than the uncompressed 56B parent under the stated evaluation setup.[\[16\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronH-16)
**Iterative Puzzle**, introduced for Puzzle-75B-A9B, applies several moderate compression stages separated by short recovery stages. It jointly searches over:
- removal or reduction of MoE expert capacity;
- per-layer routed-expert activation budgets;
- Mamba state-space dimensions;
- heterogeneous layer choices.
The final Puzzle model keeps the parent's 88-block Mamba/MoE/attention pattern but reduces the Mamba state dimension from 128 to 96 and uses heterogeneous expert configurations. It is then recovered through knowledge distillation, software-engineering reinforcement learning, checkpoint averaging, quantization, and MTP adaptation.[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7)
## Training data and pre-training
### Nemotron-4 corpus
The Nemotron-4 15B and 340B reports describe a mixture broadly divided into:
- approximately 70% English-language text;
- approximately 15% multilingual text;
- approximately 15% source code.
The multilingual portion covered 53 natural languages, while the code collection covered dozens of programming languages. Processing included exact and approximate deduplication, language identification, heuristic filtering, and learned quality classifiers.[\[11\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_15B-11)[\[12\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_340B-12)
Nemotron-4 15B was trained on approximately eight trillion tokens. Nemotron-4 340B used the same broad first-stage scale followed by approximately one trillion tokens of continued pre-training with a modified data mixture.
The complete source inventory, document-level provenance, and all licensing metadata were not released, so the original training runs cannot be reproduced exactly from the reports alone.
### Nemotron-CC
**Nemotron-CC** is a Common Crawl processing pipeline and dataset containing approximately 6.3 trillion tokens:
- approximately 4.4 trillion tokens of filtered and deduplicated text;
- approximately 1.9 trillion tokens of synthetically reformulated text.
The pipeline uses quality classifiers and synthetic rewriting to recover information from documents that would otherwise be excluded by aggressive web-quality filtering. The work was accepted as a long paper at ACL 2025.[\[27\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronCC-27)
**Nemotron-CC-Math** is a mathematics-oriented subset containing approximately 133 billion tokens. Its extraction pipeline was designed to preserve mathematical notation, LaTeX, and code structures.[\[28\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronCCMath-28)
### Nemotron 3 Nano pre-training
Nemotron 3 Nano was trained on approximately 25 trillion tokens with a Warmup–Stable–Decay learning-rate schedule.[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3)
The report divides training into:
- approximately 23.5 trillion tokens of broad pre-training data;
- approximately 1.5 trillion tokens emphasizing selected higher-quality sources;
- approximately 121 billion additional tokens for long-context continued pre-training.
The pre-training data cutoff was approximately June 2025. Named public dataset components include Nemotron-CC variants, code, mathematics, and specialized STEM datasets. NVIDIA released substantial synthetic and processed datasets, although not every raw source in the original mixture can be redistributed.
### Nemotron 3 Super pre-training
Super was trained from scratch on approximately 25 trillion text tokens:[\[4\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperReport-4)
- **Phase 1 — 20 trillion tokens:** broad domain and language coverage;
- **Phase 2 — 5 trillion tokens:** higher-quality and capability-focused data.
The model then received long-context continued pre-training. The report describes approximately 34 billion tokens in a one-million-token setting followed by approximately 17 billion tokens using a mixture of long and short sequences.
Super was NVIDIA's first Nemotron 3 checkpoint to use LatentMoE, native MTP, and mixed NVFP4 pre-training. The pre-training data cutoff was June 2025, while post-training data extended into February 2026.[\[18\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperCard-18)
### Nemotron 3 Ultra pre-training
Ultra was trained on approximately 20 trillion text tokens and then extended to a one-million-token context.[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
Relative to Super, the released data additions include:
- approximately 173 billion tokens of refreshed GitHub code with a cutoff of 30 September 2025;
- synthetic legal-domain datasets;
- updated factual, moral-scenario, generative, and multiple-choice datasets;
- additional agentic and post-training collections.
The public base-model card identifies a pre-training data cutoff in September 2025. The Ultra GenRM card identifies post-training data extending into May 2026.[\[8\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-UltraCard-8)
A lower token count than Super does not imply less total training computation. Ultra has substantially more total and active parameters, and computation also depends on sequence length, architecture, precision, optimizer state, and distributed-system efficiency.
### Nemotron 3 Nano Omni training
Nano Omni was not pretrained from scratch as a completely independent model. It uses the pretrained Nemotron 3 Nano language backbone with:
- the C-RADIOv4-H vision encoder;
- the Parakeet-TDT-0.6B-v2 audio encoder;
- learned modality projectors.
Its multi-stage supervised training progressively introduced vision, audio, combined modalities, and longer contexts. The technical report lists approximately 434 million training examples and approximately 466.9 billion tokens across seven principal SFT stages, followed by text, vision, and omni-modal reinforcement learning.[\[6\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoOmniReport-6)
The context was increased from 128K in the previous Nemotron Nano V2 VL line to 256K. Images are processed at dynamic resolution, video tokens are temporally compressed with a three-dimensional convolution, and audio is represented at approximately 12.5 encoder tokens per second.
### Synthetic-data generation
The Nemotron-4 340B pipeline used a small human-annotated seed and a much larger volume of synthetic data. Its stages included:
- generating instructions from task templates;
- sampling multiple candidate answers;
- checking tasks with known answers through deterministic or executable verifiers;
- comparing answers with a language-model judge;
- assigning multi-attribute scores with a reward model;
- iteratively training stronger generators and evaluators.
The report states that roughly 20,000 human-annotated examples—approximately 10,000 instruction examples and 10,000 HelpSteer2 examples used for reward modelling—seeded a much larger alignment corpus.[\[12\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_340B-12)
NVIDIA's licences for several newer checkpoints explicitly state that NVIDIA does not claim ownership of model outputs. This does not resolve third-party copyright, privacy, confidentiality, or dataset-licensing questions associated with a particular generated corpus.
## Post-training and alignment
### HelpSteer and HelpSteer2
**HelpSteer** contains 37,120 examples annotated on five attributes:[\[29\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-HelpSteer-29)
- helpfulness;
- correctness;
- coherence;
- complexity;
- verbosity.
Each attribute uses a Likert-style scale from zero to four.
**HelpSteer2** contains 10,681 prompts with two responses per prompt, for 21,362 prompt–response examples. More than 95% of the prompts were drawn from ShareGPT. NVIDIA describes a substantial annotation-quality filtering process.[\[30\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-HelpSteer2-30)
**HelpSteer2-Preference** adds pairwise preferences to the same examples, allowing regression-based and pairwise reward models to be trained from overlapping data.[\[31\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-HelpSteer2Preference-31)
### SteerLM
SteerLM is attribute-conditioned supervised fine-tuning. A simplified pipeline is:
1. train an attribute-prediction model;
2. annotate a larger response corpus with predicted attributes;
3. train a language model conditioned on target attribute values;
4. generate and filter additional examples for further training.
At inference time, the conditioning values can influence properties such as helpfulness, complexity, or verbosity. This gives the user or application some control without requiring a separate reinforcement-learning policy for each output style.[\[32\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-SteerLM-32)
### DPO and Reward-aware Preference Optimization
Direct Preference Optimization optimizes a policy directly from preferred and rejected responses without first training a separate scalar reward model.[\[33\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-DPO-33)
**Reward-aware Preference Optimization** (RPO) is an NVIDIA framework connecting several preference-optimization objectives. In simplified pairwise form:
$$
\mathcal{L}_{RPO}(\theta) = {\mathbb{E}}\left\lbrack d\left( \Delta r_{\phi},\beta\log\frac{\pi_{\theta}(y_{w} \mid x)}{\pi_{ref}(y_{w} \mid x)} - \beta\log\frac{\pi_{\theta}(y_{l} \mid x)}{\pi_{ref}(y_{l} \mid x)} \right) \right\rbrack,
$$
where:
- $y_{w}$ and $y_{l}$ are preferred and rejected responses;
- $\Delta r_{\phi}$ is their difference under an explicit reward function;
- $\pi_{\theta}$ is the policy being optimized;
- $\pi_{ref}$ is a reference policy;
- $d$ is a selected distance function.
Different choices of target reward, distance, and normalization reproduce or approximate several existing preference-learning methods.[\[34\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-RPO-34)
### Multi-environment RLVR
Nemotron 3 post-training uses Reinforcement Learning from Verifiable Rewards across multiple environment types, including:
- competition mathematics;
- source-code generation;
- software-engineering sandboxes;
- question answering;
- tool use;
- instruction following;
- long-context tasks.
The Nano report describes a Group Relative Policy Optimization-style procedure with masked importance sampling.[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3) The broader Nemotron 3 programme uses NeMo Gym environments and asynchronous rollout infrastructure so that long trajectories do not force all workers to wait for the slowest sample.
Verifiable rewards reduce dependence on subjective human judgements for tasks with executable or exact outcomes. They do not solve reward hacking, test leakage, environment overfitting, or incorrect verifier design.
### Super post-training
Super was post-trained with supervised fine-tuning and several reinforcement-learning stages. NVIDIA increased the share of long-horizon tool-use and software-engineering data relative to Nano and expanded asynchronous training infrastructure for terminal, code, and multi-step agent environments.[\[4\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperReport-4)
The released resources include post-training datasets and reinforcement-learning environments, but the full production system and every non-redistributable source are not included.
### Multi-teacher On-Policy Distillation in Ultra
Ultra adds **Multi-teacher On-Policy Distillation** (MOPD). Several specialist teacher policies evaluate trajectories sampled by the current student. The student then learns from token-level preferences or scores assigned by the teachers while remaining close to its own on-policy state distribution.[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
The released report describes teachers specializing in areas such as:
- software engineering and terminal work;
- office and productivity workflows;
- web research;
- conversational tool use;
- instruction following;
- factuality and hallucination avoidance;
- general reasoning.
NVIDIA reported improvements from the RLVR checkpoint to the final MOPD checkpoint on Terminal-Bench, GDPVal, SWE-bench, TauBench, BrowseComp, instruction-following, and factuality evaluations. Improvements were smaller on some self-contained reasoning benchmarks, which the authors attributed partly to limitations of on-policy distillation when the student rarely samples the reasoning paths learned by the teacher.
MOPD results are developer-reported. They do not establish that a multi-teacher merge will outperform every specialist on every task or avoid negative transfer between domains.
### Reasoning-budget control
Recent Nemotron reasoning checkpoints support configurable reasoning modes through their chat templates. Depending on the checkpoint and serving framework, an application can:
- disable an extended reasoning trace;
- enable detailed reasoning;
- set or approximate a token budget for reasoning.
The returned reasoning text is model-generated output. It should not be described as guaranteed access to a complete hidden computational process, nor does a longer trace guarantee a more accurate answer.
## Main models
### Summary table
| Model | Release | Total / active parameters | Maximum published context | Architecture | Principal qualification |
|-----------------------------|---------------|--------------------------------------|---------------------------|-------------------------------------------|-----------------------------------------------------------------|
| **Nemotron-3 8B** | 2023 | 8B / 8B | 4K | Dense Transformer | Early Nemotron-branded model; separate from the 2025 generation |
| **Nemotron-4 15B** | February 2024 | Approximately 15B / dense | 4K | Dense Transformer with GQA | Eight-trillion-token pre-training run |
| **Nemotron-4 340B** | June 2024 | Approximately 340B / dense | 4K | Dense Transformer with GQA | Base, Instruct, and Reward checkpoints |
| **Llama-3.1-Nemotron-70B** | October 2024 | 70B / dense | 128K | Llama 3.1 derivative | Instruct and Reward variants |
| **Llama-Nemotron Nano** | 2025 | 4B or 8B / dense | 128K | Llama-derived Transformer | Configurable reasoning mode |
| **Llama-Nemotron Super** | 2025 | 49B / dense | 128K | Puzzle-compressed Llama 3.3 | Derived from Llama 3.3 70B |
| **Llama-Nemotron Ultra** | 2025 | 253B / dense | 128K | Puzzle-compressed Llama 3.1 | Derived from Llama 3.1 405B |
| **Nemotron-H** | April 2025 | 8B, 47B, or 56B / dense | Checkpoint-specific | Hybrid Mamba-2–Transformer | Efficient long-generation research family |
| **Nemotron Nano 2** | August 2025 | 9B / dense | 128K | Hybrid Mamba-2–Transformer | Compressed from a 12B parent |
| **Nemotron 3 Nano** | December 2025 | 31.6B / 3.2B | 1,048,576 | Hybrid Mamba-2–Attention MoE | 128 routed experts; six active |
| **Nemotron 3 Super** | March 2026 | 120.6B / 12.7B | 1,048,576 | Hybrid Mamba-2–Attention LatentMoE | Native MTP and NVFP4 pre-training |
| **Nemotron 3 Nano Omni** | April 2026 | 30B-A3B-class backbone plus encoders | 256K | Hybrid MoE with vision and audio encoders | Text, image, video, and audio input |
| **Nemotron 3 Ultra** | June 2026 | Approximately 550B / 55B | 1,048,576 | Hybrid Mamba-2–Attention LatentMoE | Largest and most capable main Nemotron 3 tier |
| **Nemotron-Labs-3-Puzzle** | July 2026 | 75.3B / 9.3B | 1,048,576 | Compressed hybrid LatentMoE | Deployment-optimized Super derivative |
| **Nemotron-Labs-Diffusion** | July 2026 | 3B, 8B, and 14B | Checkpoint-specific | Tri-mode AR/diffusion models | Experimental decoding research |
[\[10\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3_8BCard-10)[\[11\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_15B-11)[\[12\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_340B-12)[\[15\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-LlamaNemotron-15)[\[16\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronH-16)[\[17\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Nano2-17)[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3)[\[4\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperReport-4)[\[6\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoOmniReport-6)[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7)[\[21\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Diffusion-21)
### Nemotron-4 15B
Nemotron-4 15B has 32 layers, a hidden dimension of 6,144, 48 query heads, eight key–value heads, and approximately 15 billion parameters. Approximately 3.2 billion parameters belong to embeddings, while about 12.5 billion are non-embedding parameters.[\[11\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_15B-11)
Selected developer-reported base-model results include:
| Benchmark | Nemotron-4 15B | Setting |
|--------------------------------------------------------------------------------------------------------|----------------|-----------------------------------|
| [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark") | 64.2% | 5-shot |
| BBH | 58.7% | 3-shot |
| [GSM8K](https://systems-analysis.info/eng/GSM8K_(Grade_School_Math_8K) "GSM8K (Grade School Math 8K)") | 46.0% | 8-shot, majority-style evaluation |
| [HumanEval](https://systems-analysis.info/eng/HumanEval_Benchmark "HumanEval Benchmark") | 31.6% | 0-shot, Pass@1 |
The multilingual comparisons in the report were made against models available in February 2024. Tokenization, prompts, scoring code, and data-contamination risk affect comparisons with later systems.
### Nemotron-4 340B
Nemotron-4 340B has 96 layers, a hidden dimension of 18,432, 96 query heads, and eight key–value heads.[\[12\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_340B-12)
Selected reported Base results include:
| Benchmark | Score |
|------------------------------------------------------------------------------------------|-------|
| [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark") | 81.1% |
| BBH | 85.4% |
| [HumanEval](https://systems-analysis.info/eng/HumanEval_Benchmark "HumanEval Benchmark") | 57.3% |
| ARC-Challenge | 94.3% |
The Instruct model's alignment sequence included code-focused supervised fine-tuning, general supervised fine-tuning, Direct Preference Optimization, and several Reward-aware Preference Optimization rounds.
Selected Instruct results were:
| Benchmark | Score |
|--------------------------------------------------------------------------------------------------------|-------|
| [MT-Bench](https://systems-analysis.info/eng/MT-Bench_(benchmark) "MT-Bench (benchmark)") | 8.22 |
| [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark") | 78.7% |
| [GSM8K](https://systems-analysis.info/eng/GSM8K_(Grade_School_Math_8K) "GSM8K (Grade School Math 8K)") | 92.3% |
| [HumanEval](https://systems-analysis.info/eng/HumanEval_Benchmark "HumanEval Benchmark") | 73.2% |
| Arena-Hard | 54.2 |
| AlpacaEval 2 length-controlled | 41.5% |
The Reward model replaced the vocabulary projection with a five-attribute prediction head corresponding to HelpSteer2's annotation dimensions. NVIDIA reported 92.0% on RewardBench in the 2024 evaluation.[\[12\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N4_340B-12)
### Llama-Nemotron Nano, Super, and Ultra
The Llama-Nemotron family used Llama 3.x checkpoints as starting points:[\[15\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-LlamaNemotron-15)
| Model | Approximate parameters | Parent model |
|---------------------------|------------------------|--------------------------------------|
| Llama-Nemotron Nano 8B | 8B | Llama 3.1 8B Instruct |
| Llama-Nemotron Nano 4B | 4B | Minitron compression of Llama 3.1 8B |
| Llama-Nemotron Super 49B | 49B | Puzzle compression of Llama 3.3 70B |
| Llama-Nemotron Ultra 253B | 253B | Puzzle compression of Llama 3.1 405B |
The training pipeline included:
1. neural architecture search and feed-forward-layer fusion;
2. distillation and continued pre-training;
3. supervised fine-tuning, including synthetic reasoning traces;
4. reinforcement learning;
5. conversational alignment.
NVIDIA marketed the family as an early open-weight reasoning line with one checkpoint capable of switching between direct and extended-reasoning outputs. The implementation used system-prompt or chat-template controls rather than two independently pretrained models.
The Ultra 253B report gives 76.0% on GPQA Diamond and approximately 80.8% on AIME 2024 in its reasoning-enabled configuration. These values use the developer's prompts, sampling, and scoring settings.[\[15\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-LlamaNemotron-15)
### Nemotron-H
Nemotron-H models were trained from scratch to evaluate dense hybrid Mamba–Transformer architectures.[\[16\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronH-16)
Nemotron-H-56B was trained on approximately 20 trillion tokens with an FP8 pre-training recipe. Nemotron-H-8B used approximately 15 trillion tokens. The 47B model was compressed from the 56B model with MiniPuzzle.
Selected reported results were:
| Benchmark | Nemotron-H-56B | Nemotron-H-47B |
|---------------------------------------------------------------------------------------------------------------------------------|----------------|----------------|
| [MMLU-Pro](https://systems-analysis.info/eng/MMLU-Pro_Benchmark "MMLU-Pro Benchmark"), 5-shot chain of thought | 60.5 | 61.8 |
| [MMLU](https://systems-analysis.info/eng/MMLU_Benchmark "MMLU Benchmark"), 5-shot | 84.2 | 83.6 |
| [GSM8K](https://systems-analysis.info/eng/GSM8K_(Grade_School_Math_8K) "GSM8K (Grade School Math 8K)"), 8-shot chain of thought | 93.7 | 93.3 |
| [HumanEval](https://systems-analysis.info/eng/HumanEval_Benchmark "HumanEval Benchmark"), 0-shot | 60.4 | 61.0 |
NVIDIA reported that the 47B model was up to approximately three times faster than selected similarly sized Transformer baselines in long-generation settings and approximately 20% faster than the uncompressed 56B parent. Throughput figures depend on batch size, context, output length, precision, and runtime.
The original report does not establish that an unmodified one-million-token checkpoint can be served on a single consumer graphics card. Claims of this form should identify the exact quantization, context allocation, offloading method, and runtime.
### Nemotron Nano 2
Nemotron-Nano-12B-v2-Base is a 62-layer hybrid parent model trained on approximately 20 trillion tokens in FP8. The nine-billion-parameter release was derived through structured pruning and distillation.[\[17\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Nano2-17)
The post-training pipeline included:
- supervised fine-tuning on approximately 80 billion tokens;
- reasoning traces from external and internally generated sources;
- Group Relative Policy Optimization;
- Direct Preference Optimization;
- additional conversational reinforcement learning.
The model supports a 128,000-token context. NVIDIA demonstrated the BF16 checkpoint on one 22-GB A10G GPU under the report's specified serving settings. This does not mean every batch size or maximum-context workload fits within 22 GB.
The report describes output-throughput gains of approximately three to six times over selected similarly sized Transformer baselines for long reasoning outputs. Comparisons used specific hardware and implementation versions.
### Nemotron 3 Nano
Nemotron 3 Nano is the first released checkpoint of the 2025 Nemotron 3 generation. It combines Mamba-2, attention, and sparse MoE layers and was trained on approximately 25 trillion tokens.[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3)
The model supports:
- direct-response and extended-reasoning modes;
- configurable reasoning budgets;
- structured tool calls;
- agentic post-training across several environments;
- contexts up to approximately one million tokens;
- BF16, FP8, and NVFP4 distributions.
The Hugging Face configurations may default to a shorter context, such as 256K, because allocating a one-million-token context requires substantially more memory. The lower default is a deployment setting rather than the maximum documented model capability.
### Nemotron 3 Super
Nemotron 3 Super was released on 11 March 2026. It is a 120.6B-total, 12.7B-active hybrid LatentMoE model trained on 25 trillion text tokens.[\[18\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperCard-18)[\[4\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperReport-4)
It introduced three main technologies to the released Nemotron 3 line:
- LatentMoE;
- native Multi-Token Prediction;
- mixed NVFP4 pre-training.
NVIDIA reported throughput up to 2.2 times that of GPT-OSS-120B and up to 7.5 times that of Qwen3.5-122B in an 8K-input, 64K-output comparison. Measurements used B200 GPUs and selected the better result from vLLM and TensorRT-LLM for each model. Different precisions were used for some baselines, so the values should be read as deployment measurements rather than architecture-only comparisons.[\[4\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3SuperReport-4)
### Nemotron 3 Nano Omni
Nano Omni uses the Nemotron 3 Nano language backbone together with C-RADIOv4-H and Parakeet-TDT encoders.[\[6\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoOmniReport-6)
Supported inputs are:
- text;
- images;
- video;
- audio;
- combinations of temporally interleaved visual and audio streams.
Output is text. The model does not natively generate images, video, or speech.
The architecture introduces:
- dynamic-resolution image processing;
- pixel-shuffle visual-token reduction;
- Conv3D temporal compression for video;
- optional efficient video sampling;
- audio encoding at approximately 12.5 tokens per second;
- a 256,000-token context.
The training report states that inputs ranging from short audio clips to approximately 20 minutes were included in training. The context capacity can represent longer audio, but accepted duration does not imply uniform accuracy over every multi-hour recording.
NVIDIA reported higher throughput than Qwen3-Omni and Nemotron Nano V2 VL on a B200 using specific NVFP4, workload, and interactivity settings. These are hardware- and harness-dependent serving results.[\[6\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoOmniReport-6)
### Nemotron 3 Ultra
Nemotron 3 Ultra was released on 4 June 2026 and documented in an arXiv report submitted on 12 June. It has approximately 550 billion total and 55 billion active parameters.[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)[\[8\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-UltraCard-8)
Ultra was:
- pretrained on approximately 20 trillion text tokens;
- extended to a one-million-token context;
- supervised-fine-tuned;
- trained with multi-environment RLVR;
- distilled from several specialist teachers through MOPD;
- released in Base, post-trained BF16, and NVFP4 forms.
NVIDIA reports that Ultra can achieve substantially higher throughput than several other large open-weight models in long-generation workloads. The headline comparison used 8,000 input and 64,000 output tokens, NVFP4 weights on GB200 hardware, TensorRT-LLM for Ultra, and vLLM for the comparison systems. It therefore measures a model–precision–runtime–hardware combination rather than model architecture in isolation.[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
Ultra is the flagship reasoning and agentic model in the main Nemotron 3 hierarchy as of July 2026. It is not a native multimodal checkpoint; Nano Omni is the current native multimodal branch.
### Nemotron-Labs-3-Puzzle-75B-A9B
Puzzle-75B-A9B is a compressed Super derivative released on 6 July 2026.[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7)[\[35\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleCard-35)
It contains approximately:
- 75.3 billion total parameters;
- 9.3 billion active parameters.
The compression pipeline used Iterative Puzzle, knowledge distillation, software-engineering reinforcement learning, quantization, and MTP adaptation.
In the report's serving tests:
- Puzzle provided approximately twice the server throughput of Super at a matched user-throughput constraint on one eight-B200 node;
- MTP increased the reported relative token throughput further;
- one-H100 one-million-token concurrency increased from one Super request to eight Puzzle requests.
The concurrency result concerns a specific compressed checkpoint, cache representation, and serving setup. It does not imply that every one-million-token workload can be served on one H100 with the same batch, output, latency, or precision requirements.
### Nemotron-Labs-TwoTower
TwoTower is an experimental diffusion-language modelling system built from two copies of the Nemotron 3 Nano backbone. One tower performs denoising while the other performs generation or verification.[\[20\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-TwoTower-20)
The model was trained on approximately 2.1 trillion tokens. Its authors report retaining approximately 98.7% of the selected autoregressive baseline quality while obtaining 2.42 times higher wall-clock generation throughput in their evaluation.
TwoTower is a research checkpoint and should not be treated as a replacement for the main Nano, Super, or Ultra instruction models without considering its different training objective and runtime requirements.
### Nemotron-Labs-Diffusion
Nemotron-Labs-Diffusion, published on 7 July 2026, is an experimental family at 3B, 8B, and 14B scale.[\[21\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Diffusion-21)
A single jointly trained architecture supports:
- conventional left-to-right autoregressive decoding;
- diffusion-style block denoising;
- self-speculative decoding, in which a diffusion component drafts and an autoregressive component verifies.
The report argues that autoregressive and diffusion objectives are complementary: the autoregressive objective supplies strong left-to-right language priors, while diffusion training improves look-ahead planning. NVIDIA reported that the 8B model produced six times more candidate tokens per forward pass than Qwen3-8B and approximately four times higher SPEED-Bench throughput on a GB200 under the stated SGLang configuration.
The results concern new research models and an early evaluation. Diffusion-language serving has different scheduling, latency, sampling, and implementation trade-offs from standard autoregressive decoding.
## Specialized models and associated research
### Reward models
Reward-model releases include:
- Nemotron-4 340B Reward;
- Llama-3.1-Nemotron-70B Reward;
- Nemotron 3 Ultra GenRM;
- smaller HelpSteer-derived reward models.
Reward models are used for response ranking, data filtering, rejection sampling, and reinforcement learning. Their scores are not objective measures of truth or safety. They can inherit annotation preferences, judge-model biases, length preferences, and domain gaps.
### OpenReasoning-Nemotron
OpenReasoning-Nemotron is a collection of Qwen-derived models from approximately 1.5B to 32B, fine-tuned on reasoning data associated with DeepSeek-R1-0528 and NVIDIA-generated datasets.[\[36\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-OpenReasoning-36)
Some published results use selection over many independently sampled solutions, such as GenSelect@64. Such scores include substantial test-time computation and are not directly comparable with single-sample Pass@1 results.
### Nemotron-Cascade 2
Nemotron-Cascade 2 is a post-training research model with approximately 30 billion total and three billion active parameters. It applies expanded Cascade reinforcement learning and multi-domain on-policy distillation.[\[19\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Cascade2-19)
The authors report high-level performance on selected mathematics and programming competitions, including gold-medal-level thresholds in their IMO-, IOI-, and ICPC-style evaluations. These are benchmark claims under the paper's tasks and scoring procedures, not official participation in the corresponding human competitions.
### Nemotron Elastic
Nemotron Elastic trains nested subnetworks of different sizes within one parent model. The reported configurations include approximately 6B, 9B, and 12B variants. The objective is to reduce the cost of producing several deployment sizes compared with independently training every model.[\[37\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Elastic-37)
### Nemotron-CrossThink
Nemotron-CrossThink studies reinforcement learning outside competition mathematics, including scientific, factual, and general reasoning domains.[\[38\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-CrossThink-38)
Reported percentage improvements are relative to the paper's starting checkpoints and should not be confused with absolute benchmark scores.
### Nemotron-UltraLong
Nemotron-UltraLong investigates extending 128K models to contexts as large as four million tokens through efficient long-context training.[\[39\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-UltraLong-39)
The work is a research direction rather than evidence that every Nemotron checkpoint supports four million tokens. The principal released Nemotron 3 text models document one million tokens, while Nano Omni documents 256K.
### Jet-Nemotron
Jet-Nemotron applies post-training neural architecture search to compact hybrid models in approximately two- and four-billion-parameter classes.[\[40\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-JetNemotron-40)
### Vision and document models
The Nemotron ecosystem includes:
- **Nemotron Nano V2 VL** — a vision-language model for document, image, table, and video understanding;[\[41\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoV2VL-41)
- **Nemotron Parse 1.1** — document OCR, layout, and structured-content extraction;[\[42\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Parse-42)
- **Nemotron ColEmbed V2** — late-interaction visual-document retrieval embeddings.[\[43\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-ColEmbed-43)
These models have different architectures and output types from the main generative language models.
### Speech models
Nemotron-associated speech releases cover:
- automatic speech recognition;
- text-to-speech;
- speech translation;
- audio understanding.
Nano Omni uses the Parakeet-TDT audio encoder but generates text rather than speech. Other Nemotron Speech services or models may provide speech outputs through separate acoustic and vocoder components.
### Safety models
Llama 3.1 Nemotron Safety Guard 8B V3 is a content-safety classifier covering 23 safety categories across nine languages. NVIDIA reported approximately 84.2% aggregate accuracy in its published evaluation.[\[44\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-SafetyRecipe-44)
Safety-classifier accuracy depends on taxonomy, class balance, language, threshold, and adversarial prompt distribution. A classifier should not be treated as a complete security boundary.
## Reported benchmark results
Benchmark results in this section are primarily developer-reported. They can depend on:
- checkpoint precision;
- chat template;
- system prompt;
- reasoning budget;
- temperature and sampling count;
- tool availability;
- agent framework;
- context-management policy;
- timeout and retry settings;
- judge model;
- benchmark revision;
- repository and dependency state.
Numbers from different reports should not be interpreted as a controlled time series unless the same harness, checkpoint precision, and test version were used.
### Nemotron 3 Nano
Selected results reported for the post-trained Nemotron 3 Nano checkpoint include:[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3)
| Benchmark | Nemotron 3 Nano 30B-A3B |
|-----------------------------------------------------------------------------------------------------|-------------------------|
| [MMLU-Pro](https://systems-analysis.info/eng/MMLU-Pro_Benchmark "MMLU-Pro Benchmark") | 78.30 |
| AIME 2025, no tools | 89.06 |
| AIME 2025, with tools | 99.17 |
| [GPQA](https://systems-analysis.info/eng/GPQA_Diamond_Benchmark "GPQA Diamond Benchmark"), no tools | 73.04 |
| [LiveCodeBench](https://systems-analysis.info/eng/LiveCodeBench "LiveCodeBench") v6 | 68.25 |
| SWE-bench with OpenHands | 38.76 |
| TauBench V2 average | 49.04 |
| Arena-Hard-V2 average | 67.65 |
| RULER-100 at 1M | 86.34 |
The report used NeMo Evaluator and task-specific harnesses. Some comparison-system scores were reproduced by NVIDIA, while others were taken from model developers or secondary evaluations. Tool-enabled and no-tool results measure different systems.
NVIDIA reported that, for an 8K-input and 16K-output workload on one H200, Nano achieved 3.3 times the throughput of the tested Qwen3-30B-A3B-Thinking configuration and 2.2 times that of the tested GPT-OSS-20B configuration.[\[3\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3NanoReport-3)
### Super and Puzzle comparison
The Puzzle technical report evaluated BF16 and NVFP4 versions of Puzzle and its Super parent in the same table.[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7)
| Benchmark | Puzzle BF16 | Super BF16 | Puzzle NVFP4 | Super NVFP4 |
|-------------------------------------------------------------------------------------------------------------|-------------|------------|--------------|-------------|
| [MMLU-Pro](https://systems-analysis.info/eng/MMLU-Pro_Benchmark "MMLU-Pro Benchmark") | 82.4 | 83.8 | 82.2 | 83.5 |
| AIME 2025, no tools | 89.7 | 92.2 | 89.9 | 89.9 |
| HMMT February 2025, no tools | 93.4 | 94.2 | 92.9 | 93.7 |
| [GPQA](https://systems-analysis.info/eng/GPQA_Diamond_Benchmark "GPQA Diamond Benchmark"), no tools | 78.6 | 80.5 | 78.0 | 79.7 |
| LiveCodeBench selected period | 81.1 | 82.1 | 79.9 | 81.5 |
| [SWE-bench](https://systems-analysis.info/eng/SWE-bench_(benchmark) "SWE-bench (benchmark)") with OpenHands | 56.9 | 59.5 | 56.9 | 58.7 |
| TauBench V2 average | 60.2 | 60.8 | 59.9 | 61.3 |
| RULER at 1M | 92.2 | 93.9 | 93.2 | 93.8 |
| MMLU-ProX multilingual average | 77.5 | 79.5 | 76.5 | 79.0 |
The largest compression-related gaps in the report occurred on some instruction-following and agentic evaluations. Puzzle preserved closer performance on several reasoning and long-context tests.
At a fixed operating point of 100 output tokens per second per user on one eight-B200 node, the report lists:
| Variant | Relative raw server throughput versus single-step Super | Relative request completion after verbosity adjustment |
|---------------------|---------------------------------------------------------|--------------------------------------------------------|
| Super, single-step | 1.00× | 1.00× |
| Super, MTP | 3.04× | 3.04× |
| Puzzle, single-step | 2.03× | 2.16× |
| Puzzle, MTP | 4.63× | 4.91× |
| Nano, single-step | 5.84× | 3.87× |
These are serving-system measurements, not benchmark-independent model properties.
### Nemotron 3 Ultra
Selected post-trained Ultra results reported in its technical report include:[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
| Category | Benchmark | Nemotron 3 Ultra |
|-------------------------|--------------------------------------|------------------|
| Agentic | Terminal-Bench 2.1 | 56.4 |
| Agentic | GDPVal | 46.7 |
| Software engineering | SWE-bench Verified | 70.7 |
| Software engineering | SWE-bench Multilingual | 67.7 |
| Professional research | ProfBench with search | 56.0 |
| Coding agents | PinchBench | 90.0 |
| Conversational tool use | TauBench V3 average | 70.9 |
| Web research | BrowseComp | 44.4 |
| Reasoning | GPQA, no tools | 87.0 |
| General knowledge | MMLU-Pro | 86.8 |
| Instruction following | IFBench | 81.7 |
| Long context | RULER at 1M | 94.7 |
| Long context | LongBench v2 at approximately 1M | 61.9 |
| Multilingual | MMLU-ProX average over ten languages | 83.0 |
The report also gives an IOI 2025 score of 570 under its evaluation, which the authors compare with the scores of the second- and third-ranked human contestants. This is a model evaluation using the paper's prompting and judging pipeline, not official participation in the International Olympiad in Informatics.
Some Ultra benchmarks were run through open NeMo Evaluator containers. Others, including parts of BrowseComp, TauBench V3, ProfBench, PinchBench, Vals.ai Financial Agent, and LongBench v2, used official implementations or internal scaffolding that NVIDIA stated it intended to release later. This reduces immediate independent reproducibility for those results.[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
### Multimodal Nano Omni results
Nano Omni was evaluated on document understanding, OCR, chart understanding, graphical-interface use, long video, audio-visual reasoning, speech recognition, and audio understanding.[\[6\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoOmniReport-6)
The report describes leading or competitive results on:
- OCRBench-V2;
- MMLongBench-Doc;
- ChartQA;
- CharXiv;
- ScreenSpot and ScreenSpot-Pro;
- OSWorld;
- WorldSense;
- DailyOmni;
- VoiceBench.
These benchmarks use different metrics and cannot be combined into one meaningful “multimodal accuracy” score. The report's speed comparisons also use different precision formats for some systems and a B200-specific serving configuration.
## Deployment and applications
### NVIDIA NIM
NVIDIA NIM packages model servers as containerized microservices exposing APIs compatible with common generation interfaces. Nemotron NIM deployments may include:
- TensorRT-LLM optimization;
- model-specific quantization;
- OpenAI-compatible endpoints;
- multi-GPU and multi-node serving;
- Kubernetes deployment through Helm;
- monitoring and enterprise-support integration.
The NIM container has separate software terms from the model checkpoint. Using an open-weight model through NIM does not make the container itself subject only to the model's licence.
### Independent serving frameworks
Depending on checkpoint and version, Nemotron models can be used with:
- TensorRT-LLM;
- vLLM;
- SGLang;
- Hugging Face Transformers;
- NeMo Framework;
- llama.cpp and GGUF-derived community conversions;
- Ollama and LM Studio for compatible compressed formats.
New architectures may initially require development or nightly releases. Mamba state caching, LatentMoE routing, NVFP4 kernels, MTP speculative decoding, and million-token contexts are not supported equally by every runtime.
### Hardware requirements
A model's active parameter count is not its storage requirement. All experts must be stored, distributed, or offloaded even when only a subset is active for each token.
Approximate raw weight storage before runtime overhead is:
$$
\text{weight bytes} \approx \text{parameter count} \times \frac{\text{bits per parameter}}{8}.
$$
For example, 550 billion weights require approximately:
- 1.1 TB at 16 bits per weight;
- 550 GB at eight bits per weight;
- 275 GB at four bits per weight.
Actual deployments require additional memory for:
- quantization scales and metadata;
- embeddings or higher-precision layers;
- recurrent Mamba states;
- KV caches for attention layers;
- CUDA graphs and workspaces;
- routing buffers;
- batch scheduling;
- speculative-decoding state;
- framework overhead.
Consequently, a 550B-A55B model does not have the memory footprint of a dense 55B model.
### Long-context workloads
One-million-token models can be applied to:
- large source-code repositories;
- collections of legal or technical documents;
- long-running agent histories;
- extensive event logs;
- multi-document research;
- long conversations;
- structured retrieval and synthesis.
Nominal context capacity does not ensure perfect recall or reasoning across every token. Retrieval, hierarchical summaries, external memory, and selective context loading may remain more efficient and reliable than placing all available material in one request.
### Synthetic-data generation
Nemotron models are used to:
- generate instructions and answers;
- produce reasoning traces;
- translate or transform datasets;
- create code and tests;
- generate tool-use trajectories;
- score and filter candidate examples;
- train specialized student models.
Reward-model filtering can improve average quality but can also amplify the reward model's preferences and blind spots.
### Software engineering and agents
Nemotron 3 models target agentic workloads including:
- repository analysis;
- code generation and refactoring;
- terminal execution;
- bug correction;
- test generation;
- incident and cybersecurity triage;
- IT-ticket automation;
- retrieval and web search;
- tool calling;
- multi-agent decomposition.
Performance depends on the complete agent system, including permissions, tool definitions, execution environments, context management, retries, timeouts, and verification.
### Multimodal and document applications
Nano Omni and specialist models support:
- OCR and document conversion;
- table, chart, and diagram interpretation;
- video and audio-video question answering;
- graphical-interface interaction;
- screen-based computer use;
- speech transcription;
- long-recording analysis;
- visual-document retrieval.
For production document processing, page rendering, OCR preprocessing, file parsing, and layout reconstruction may be performed outside the neural checkpoint.
### Retrieval and safety
Embedding and reranking models in the Nemotron ecosystem can be used for retrieval-augmented generation, while Safety Guard and NeMo Guardrails can inspect prompts, retrieved content, tool calls, and responses.
A retrieval model does not guarantee that the generator will use retrieved evidence correctly. A safety classifier does not guarantee resistance to adaptive attacks.
## Licensing
Nemotron licensing changed across generations.
| Release group | Principal governing terms or qualification |
|--------------------------------|-----------------------------------------------------------------------------------------------|
| Nemotron-3 8B (2023) | NVIDIA AI Foundation Models Community License |
| Nemotron-4 340B | NVIDIA Open Model License Agreement |
| Llama-derived Nemotron models | Checkpoint-specific NVIDIA terms together with applicable Llama-derived notices or conditions |
| Nemotron 3 Nano | NVIDIA Nemotron Open Model License |
| Nemotron 3 Super | NVIDIA Nemotron Open Model License |
| Nemotron 3 Nano Omni | NVIDIA Open Model Agreement |
| Nemotron 3 Ultra | OpenMDW License Agreement 1.1 |
| Nemotron-Labs-3-Puzzle-75B-A9B | OpenMDW License Agreement 1.1 |
[\[45\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NemotronLicense-45)[\[46\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NVIDIAOpenAgreement-46)[\[47\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-OpenMDW-47)[\[48\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-NanoOmniCard-48)[\[8\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-UltraCard-8)[\[35\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleCard-35)
The NVIDIA Nemotron Open Model License and NVIDIA Open Model Agreement permit commercial use and creation and distribution of derivative works subject to their terms. NVIDIA states that it does not claim ownership of outputs generated using covered works. OpenMDW 1.1 is a permissive model-specific licence covering model weights and associated distributed artefacts.
These summaries are not a substitute for the governing text. Software repositories, data files, NIM containers, and third-party base-model components can carry separate terms.
The term **open-weight** is generally more precise than suggesting that every Nemotron release includes the complete raw training corpus and a fully reproducible production environment. NVIDIA releases substantial datasets, code, recipes, checkpoints, and environments, but only data for which it claims redistribution rights can be made public.[\[2\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3Whitepaper-2)[\[5\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-N3UltraReport-5)
## Limitations and open problems
### Benchmark comparability
Most Nemotron benchmark tables are produced by NVIDIA. Even when benchmark data are public, results can vary because of:
- different system prompts;
- different reasoning budgets;
- different tool permissions;
- differing maximum outputs;
- repeated sampling;
- proprietary judge models;
- benchmark fixes and revisions;
- modified agent harnesses;
- different hardware and precision.
A claim that one model “outperforms” another is meaningful only for the specified checkpoint and protocol.
### Generation length and verbosity
Reasoning models can consume many tokens before returning an answer. High token throughput does not necessarily imply a high number of completed user requests if the model generates substantially longer answers.
The Puzzle report explicitly adjusts raw throughput for model verbosity and shows a material difference between token throughput and estimated request-completion rate.[\[7\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-PuzzleReport-7) Applications should measure task success, latency, output length, and cost together.
### Hallucination and factual error
Nemotron models can produce:
- unsupported factual claims;
- fabricated citations;
- incorrect calculations;
- insecure code;
- misread documents or images;
- incorrect tool arguments;
- false statements about whether an operation completed.
Reward models, web search, and long reasoning traces can reduce some errors but do not provide proof of correctness. Important claims require source verification; code requires testing; irreversible actions require controlled permissions and, where appropriate, human approval.
### Agentic error accumulation
A long-running agent can compound an early error across many actions. Failure modes include:
- selecting the wrong tool;
- misreading tool output;
- modifying the wrong file;
- losing track of the original objective;
- repeating an operation;
- failing to notice a partial result;
- accepting a misleading web page or prompt injection;
- leaking secrets through an external tool;
- executing untrusted code.
More tool calls provide more opportunities for correction but also more opportunities for failure.
### Long-context reliability
A one-million-token window does not guarantee:
- uniform recall at every position;
- correct ordering of widely separated events;
- resolution of contradictory instructions;
- immunity to prompt injection in documents;
- preservation of exact numerical state;
- stable reasoning over the complete context.
Hybrid models reduce KV-cache pressure but still need global attention anchors or other mechanisms to propagate information. Relevant details can be lost through routing, state compression, context truncation, or attention selection.
### Mamba-state limitations
Mamba layers provide fixed-size recurrent states during decoding. Compressing an arbitrarily long history into a fixed state can lose information. Periodic attention layers mitigate this limitation but reintroduce KV-cache and quadratic-prefill costs.
Mamba-state quantization is also less mature than conventional Transformer weight quantization. Selective higher-precision storage can preserve quality but complicates serving systems.
### MoE routing and communication
MoE inference requires expert placement and token routing across devices. Potential problems include:
- uneven expert utilization;
- all-to-all communication overhead;
- limited network bandwidth;
- hot experts;
- routing instability;
- reduced efficiency at small batch sizes;
- poor utilization when requests have different lengths.
A low active-parameter count does not automatically yield low latency.
### Quantization
FP8 and NVFP4 reduce storage and bandwidth but can affect:
- long-context retrieval;
- routing decisions;
- numerical stability;
- recurrent-state quality;
- speculative-decoding acceptance;
- calibration-sensitive tasks.
Reported “accuracy retention” values are averages over selected benchmark sets. Individual tasks may degrade more substantially.
### Training-data transparency
NVIDIA has released unusually large processed and synthetic datasets, training recipes, checkpoints, and reinforcement-learning environments. Nevertheless, public materials do not include:
- every raw licensed source;
- the complete document-level provenance map;
- every filtering decision;
- all production prompts and teacher models;
- every human-annotation process;
- the complete distributed-training infrastructure.
Exact reproduction of the largest runs therefore remains impractical for most independent researchers.
### Synthetic-data dependence
Synthetic data can improve scale and target specific capabilities, but it may also:
- reproduce teacher errors;
- narrow linguistic diversity;
- amplify judge-model preferences;
- reward verbosity or stylistic conformity;
- introduce benchmark-like templates;
- cause model collapse through repeated self-training;
- transfer incorrect tool-use conventions.
Executable verification helps for mathematics and code but is unavailable for many factual, social, and open-ended tasks.
### Environmental and infrastructure cost
Frontier Nemotron models require large accelerator clusters, high-bandwidth networks, storage, cooling, and electricity. Sparse activation reduces some per-token arithmetic but does not eliminate weight storage, communication, or training cost.
Public reports do not provide a complete independently audited lifecycle carbon assessment for the largest checkpoints.
## Safety, ethics, and governance
### Development-time safety
NVIDIA model cards and safety materials describe practices including:
- training-data filtering;
- AEGIS safety taxonomies;
- automated testing with garak;
- red-team exercises;
- safety-focused supervised examples;
- policy and preference optimization;
- multilingual safety classifiers;
- model-card documentation.
These measures reduce selected risks but do not establish universal safety.
### NeMo Guardrails
NeMo Guardrails is an Apache-2.0-licensed framework for adding programmable controls around language-model applications.[\[49\]](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_note-Guardrails-49)
Possible controls include:
- topic restrictions;
- personal-information detection;
- jailbreak and prompt-injection detection;
- output moderation;
- retrieval-grounding checks;
- tool-call validation;
- custom policy flows.
Runtime guardrails are separate software components and can be disabled or incorrectly configured. They should be combined with operating-system isolation, least privilege, network restrictions, and auditing.
### Agent security
Tool-using Nemotron deployments may need:
- sandboxed code execution;
- filesystem and network allowlists;
- short-lived credentials;
- approval gates for destructive actions;
- transaction limits;
- provenance records;
- rollback mechanisms;
- monitoring for loops and anomalous tool use.
Model alignment alone is not an adequate security boundary for an autonomous agent.
### Privacy and confidential information
Inputs can contain source code, documents, images, audio, credentials, or personal data. Deployers should define:
- whether inputs and outputs are logged;
- retention periods;
- access permissions;
- encryption and residency requirements;
- whether data can be used for additional training;
- procedures for deletion and incident response.
Self-hosting gives an organization more control but also transfers operational and compliance responsibility to the deployer.
### Bias and representation
Nemotron models can inherit biases from web corpora, synthetic teachers, annotators, and reward models. Performance and safety can differ by language, dialect, culture, and domain.
The presence of multilingual benchmark scores does not establish equivalent quality or fairness across all supported languages. Domain-specific evaluations are needed for high-impact deployment.
### Open-weight misuse
Open-weight availability supports independent research, local deployment, adaptation, and auditing. It also permits removal of safeguards and lowers barriers to some harmful applications.
The relevant balance depends on model capability, deployment context, available safeguards, and jurisdiction. Licence permission should not be confused with legal or ethical acceptability of every use.
## Research directions
Published Nemotron work indicates several continuing directions:
- **Hybrid sequence modelling.** Improving the allocation of Mamba, attention, and expert layers for long-context quality and decode efficiency.
- **Latent expert architectures.** Increasing expert diversity while reducing memory bandwidth and communication.
- **Multi-Token Prediction.** Raising draft acceptance and improving speculative decoding across different batch sizes.
- **Diffusion language modelling.** Combining autoregressive and denoising objectives, as in TwoTower and Nemotron-Labs-Diffusion.
- **Deployment-aware compression.** Extending Minitron, Puzzle, MiniPuzzle, and Iterative Puzzle to heterogeneous MoE and multimodal systems.
- **Ultra-long context.** Improving memory retention, retrieval, and prompt-injection resistance beyond nominal context-window expansion.
- **Multi-environment reinforcement learning.** Training agents across code, terminals, web search, office workflows, and other interactive environments.
- **Multi-teacher distillation.** Combining specialist capabilities without destructive interference or dependence on inaccessible teacher distributions.
- **Native multimodality.** Joint reasoning over text, images, video, audio, interfaces, documents, and tools.
- **Low-precision training.** Improving NVFP4 stability and portability while limiting long-context and recurrent-state degradation.
- **Open evaluation.** Releasing complete prompts, containers, trajectories, judge configurations, and reproducible serving measurements.
- **Secure agent deployment.** Developing capability-aware permissions, provenance systems, sandboxing, and runtime verification.
NVIDIA has not published a binding specification or release schedule for a generation after Nemotron 3. Statements about an unreleased Nemotron 4 generation would therefore be speculative as of 10 July 2026.
## External links
- NVIDIA Nemotron model portal
- NVIDIA Nemotron 3 research page
- NVIDIA organization on Hugging Face
- Nemotron training recipes and developer repository
- NVIDIA NeMo documentation
- NVIDIA API and NIM catalogue
- OpenMDW License Agreement 1.1
## See also
- [Large language model](https://systems-analysis.info/eng/Large_language_model "Large language model")
- [Transformer (deep learning architecture)](https://systems-analysis.info/eng/Transformer_architecture "Transformer architecture")
- [Reinforcement learning from human feedback](https://systems-analysis.info/eng/Reinforcement_learning_from_human_feedback_(RLHF) "Reinforcement learning from human feedback (RLHF)")
- [AI agent](https://systems-analysis.info/eng/AI_Agent "AI Agent")
## Literature
- Parmar, J. et al. *Nemotron-4 15B Technical Report*. arXiv:2402.16819, 2024.
- Adler, B. et al. *Nemotron-4 340B Technical Report*. arXiv:2406.11704, 2024.
- Muralidharan, S. et al. *Compact Language Models via Pruning and Knowledge Distillation*. arXiv:2407.14679, 2024.
- Bercovich, A. et al. *Puzzle: Distillation-Based NAS for Inference-Optimized LLMs*. arXiv:2411.19146, 2024.
- Su, D. et al. *Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset*. ACL, 2025.
- Blakeman, A. et al. *Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models*. arXiv:2504.03624, 2025.
- Bercovich, A. et al. *Llama-Nemotron: Efficient Reasoning Models*. arXiv:2505.00949, 2025.
- Basant, A. et al. *NVIDIA Nemotron Nano 2*. arXiv:2508.14444, 2025.
- NVIDIA. *NVIDIA Nemotron 3: Efficient and Open Intelligence*. arXiv:2512.20856, 2025.
- Blakeman, A. et al. *Nemotron 3 Nano*. arXiv:2512.20848, 2025.
- NVIDIA. *Nemotron 3 Super*. arXiv:2604.12374, 2026.
- NVIDIA. *Nemotron 3 Nano Omni*. arXiv:2604.24954, 2026.
- NVIDIA. *Nemotron 3 Ultra*. arXiv:2606.15007, 2026.
- Reda, F. et al. *Nemotron-TwoTower*. arXiv:2606.26493, 2026.
- Bercovich, A. et al. *Nemotron-Labs-3-Puzzle-75B-A9B*. arXiv:2607.04371, 2026.
- Fu, Y. et al. *Nemotron-Labs-Diffusion*. arXiv:2607.05722, 2026.
- Dao, T.; Gu, A. *Transformers are SSMs*. ICML, 2024.
- Ainslie, J. et al. *GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints*. arXiv:2305.13245, 2023.
## References
1. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronModels_1-0) NVIDIA Developer. *NVIDIA Nemotron AI Models*. Accessed 10 July 2026. https://developer.nvidia.com/nemotron
2. ↑ [2.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3Whitepaper_2-0) [2.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3Whitepaper_2-1) [2.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3Whitepaper_2-2) NVIDIA (2025). *NVIDIA Nemotron 3: Efficient and Open Intelligence*. arXiv:2512.20856. https://arxiv.org/abs/2512.20856
3. ↑ [3.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-0) [3.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-1) [3.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-2) [3.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-3) [3.4](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-4) [3.5](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-5) [3.6](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-6) [3.7](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-7) [3.8](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3NanoReport_3-8) Blakeman, A. et al. (2025). *Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning*. arXiv:2512.20848. https://arxiv.org/abs/2512.20848
4. ↑ [4.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperReport_4-0) [4.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperReport_4-1) [4.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperReport_4-2) [4.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperReport_4-3) [4.4](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperReport_4-4) [4.5](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperReport_4-5) [4.6](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperReport_4-6) NVIDIA (2026). *Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning*. arXiv:2604.12374. https://arxiv.org/abs/2604.12374
5. ↑ [5.00](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-0) [5.01](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-1) [5.02](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-2) [5.03](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-3) [5.04](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-4) [5.05](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-5) [5.06](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-6) [5.07](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-7) [5.08](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-8) [5.09](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-9) [5.10](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-10) [5.11](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3UltraReport_5-11) NVIDIA (2026). *Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning*. arXiv:2606.15007. https://arxiv.org/abs/2606.15007
6. ↑ [6.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoOmniReport_6-0) [6.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoOmniReport_6-1) [6.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoOmniReport_6-2) [6.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoOmniReport_6-3) [6.4](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoOmniReport_6-4) [6.5](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoOmniReport_6-5) [6.6](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoOmniReport_6-6) NVIDIA (2026). *Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence*. arXiv:2604.24954. https://arxiv.org/abs/2604.24954
7. ↑ [7.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-0) [7.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-1) [7.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-2) [7.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-3) [7.4](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-4) [7.5](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-5) [7.6](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-6) [7.7](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-7) [7.8](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleReport_7-8) Bercovich, A. et al. (2026). *Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs*. arXiv:2607.04371. https://arxiv.org/abs/2607.04371
8. ↑ [8.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-UltraCard_8-0) [8.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-UltraCard_8-1) [8.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-UltraCard_8-2) [8.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-UltraCard_8-3) NVIDIA. *NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16*. Hugging Face model card, released 4 June 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
9. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NeMoDocs_9-0) NVIDIA. *NeMo Framework Documentation*. https://docs.nvidia.com/nemo/
10. ↑ [10.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3_8BCard_10-0) [10.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3_8BCard_10-1) NVIDIA. *Nemotron-3-8B-Base-4K*. Hugging Face model card, 2023. https://huggingface.co/nvidia/nemotron-3-8b-base-4k
11. ↑ [11.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_15B_11-0) [11.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_15B_11-1) [11.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_15B_11-2) [11.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_15B_11-3) [11.4](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_15B_11-4) Parmar, J. et al. (2024). *Nemotron-4 15B Technical Report*. arXiv:2402.16819. https://arxiv.org/abs/2402.16819
12. ↑ [12.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_340B_12-0) [12.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_340B_12-1) [12.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_340B_12-2) [12.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_340B_12-3) [12.4](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_340B_12-4) [12.5](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_340B_12-5) [12.6](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_340B_12-6) [12.7](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N4_340B_12-7) Adler, B. et al. (2024). *Nemotron-4 340B Technical Report*. arXiv:2406.11704. https://arxiv.org/abs/2406.11704
13. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-LN70BInstruct_13-0) NVIDIA. *Llama-3.1-Nemotron-70B-Instruct-HF*. Hugging Face model card, 2024. https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
14. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-LN70BReward_14-0) NVIDIA. *Llama-3.1-Nemotron-70B-Reward*. Hugging Face model card, 2024. https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward
15. ↑ [15.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-LlamaNemotron_15-0) [15.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-LlamaNemotron_15-1) [15.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-LlamaNemotron_15-2) [15.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-LlamaNemotron_15-3) Bercovich, A. et al. (2025). *Llama-Nemotron: Efficient Reasoning Models*. arXiv:2505.00949. https://arxiv.org/abs/2505.00949
16. ↑ [16.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronH_16-0) [16.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronH_16-1) [16.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronH_16-2) [16.3](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronH_16-3) [16.4](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronH_16-4) Blakeman, A. et al. (2025). *Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models*. arXiv:2504.03624. https://arxiv.org/abs/2504.03624
17. ↑ [17.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Nano2_17-0) [17.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Nano2_17-1) [17.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Nano2_17-2) Basant, A. et al. (2025). *NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model*. arXiv:2508.14444. https://arxiv.org/abs/2508.14444
18. ↑ [18.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperCard_18-0) [18.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperCard_18-1) [18.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-N3SuperCard_18-2) NVIDIA. *NVIDIA-Nemotron-3-Super-120B-A12B-BF16*. Hugging Face model card, released 11 March 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
19. ↑ [19.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Cascade2_19-0) [19.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Cascade2_19-1) NVIDIA (2026). *Nemotron-Cascade 2: Post-Training Open Models with Cascade Reinforcement Learning*. arXiv:2603.19220. https://arxiv.org/abs/2603.19220
20. ↑ [20.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-TwoTower_20-0) [20.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-TwoTower_20-1) Reda, F. et al. (2026). *Nemotron-TwoTower: Diffusion Language Modeling with Decoupled Denoising and Generation*. arXiv:2606.26493. https://arxiv.org/abs/2606.26493
21. ↑ [21.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Diffusion_21-0) [21.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Diffusion_21-1) [21.2](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Diffusion_21-2) Fu, Y. et al. (2026). *Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding*. arXiv:2607.05722. https://arxiv.org/abs/2607.05722
22. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-GQA_22-0) Ainslie, J. et al. (2023). *GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints*. arXiv:2305.13245. https://arxiv.org/abs/2305.13245
23. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Mamba2_23-0) Dao, T.; Gu, A. (2024). *Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality*. ICML 2024; arXiv:2405.21060. https://arxiv.org/abs/2405.21060
24. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-RMSNorm_24-0) Zhang, B.; Sennrich, R. (2019). *Root Mean Square Layer Normalization*. arXiv:1910.07467. https://arxiv.org/abs/1910.07467
25. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Minitron_25-0) Muralidharan, S. et al. (2024). *Compact Language Models via Pruning and Knowledge Distillation*. arXiv:2407.14679. https://arxiv.org/abs/2407.14679
26. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Puzzle_26-0) Bercovich, A. et al. (2024). *Puzzle: Distillation-Based NAS for Inference-Optimized LLMs*. arXiv:2411.19146. https://arxiv.org/abs/2411.19146
27. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronCC_27-0) Su, D. et al. (2025). *Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset*. ACL 2025; arXiv:2412.02595. https://arxiv.org/abs/2412.02595
28. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronCCMath_28-0) Karimi Mahabadi, R. et al. (2025). *Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset*. arXiv:2508.15096. https://arxiv.org/abs/2508.15096
29. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-HelpSteer_29-0) Wang, Z. et al. (2023). *HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM*. arXiv:2311.09528. https://arxiv.org/abs/2311.09528
30. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-HelpSteer2_30-0) Wang, Z. et al. (2024). *HelpSteer2: Open-source Dataset for Training Top-Performing Reward Models*. arXiv:2406.08673. https://arxiv.org/abs/2406.08673
31. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-HelpSteer2Preference_31-0) Wang, Z. et al. (2024). *HelpSteer2-Preference: Complementing Ratings with Preferences*. arXiv:2410.01257. https://arxiv.org/abs/2410.01257
32. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-SteerLM_32-0) Dong, Y. et al. (2023). *SteerLM: Attribute Conditioned SFT as an User-Steerable Alternative to RLHF*. arXiv:2310.05344. https://arxiv.org/abs/2310.05344
33. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-DPO_33-0) Rafailov, R. et al. (2023). *Direct Preference Optimization: Your Language Model Is Secretly a Reward Model*. NeurIPS 2023; arXiv:2305.18290. https://arxiv.org/abs/2305.18290
34. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-RPO_34-0) Sun, S. et al. (2025). *Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment*. arXiv:2502.00203. https://arxiv.org/abs/2502.00203
35. ↑ [35.0](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleCard_35-0) [35.1](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-PuzzleCard_35-1) NVIDIA. *NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4*. Hugging Face model card, released 6 July 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
36. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-OpenReasoning_36-0) NVIDIA. *OpenReasoning-Nemotron*. Hugging Face collection, 2025. https://huggingface.co/collections/nvidia/openreasoning-nemotron-685824d42db3e24d8b8f5e39
37. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Elastic_37-0) Taghibakhshi, A. et al. (2025). *Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs*. arXiv:2511.16664. https://arxiv.org/abs/2511.16664
38. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-CrossThink_38-0) Akter, S. N. et al. (2025). *Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning*. arXiv:2504.13941. https://arxiv.org/abs/2504.13941
39. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-UltraLong_39-0) Xu, C. et al. (2025). *From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models*. arXiv:2504.06214. https://arxiv.org/abs/2504.06214
40. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-JetNemotron_40-0) Gu, Y. et al. (2025). *Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search*. arXiv:2508.15884. https://arxiv.org/abs/2508.15884
41. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoV2VL_41-0) Deshmukh, A. S. et al. (2025). *NVIDIA Nemotron Nano V2 VL*. arXiv:2511.03929. https://arxiv.org/abs/2511.03929
42. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Parse_42-0) Chumachenko, K. et al. (2025). *NVIDIA Nemotron Parse 1.1*. arXiv:2511.20478. https://arxiv.org/abs/2511.20478
43. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-ColEmbed_43-0) de Souza P. Moreira, G. et al. (2026). *Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrieval*. arXiv:2602.03992. https://arxiv.org/abs/2602.03992
44. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-SafetyRecipe_44-0) NVIDIA Developer Blog (2025). *Safeguard Agentic AI Systems with the NVIDIA Safety Recipe*. https://developer.nvidia.com/blog/safeguard-agentic-ai-systems-with-the-nvidia-safety-recipe/
45. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NemotronLicense_45-0) NVIDIA. *NVIDIA Nemotron Open Model License*. Last modified 15 December 2025. https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
46. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NVIDIAOpenAgreement_46-0) NVIDIA. *NVIDIA Open Model Agreement*. Released 2 April 2026. https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/
47. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-OpenMDW_47-0) Linux Foundation. *OpenMDW License Agreement 1.1*. https://openmdw.ai/license/1-1/
48. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-NanoOmniCard_48-0) NVIDIA. *Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16*. Hugging Face model card. https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
49. [↑](https://systems-analysis.info/eng/Nemotron_(NVIDIA)#cite_ref-Guardrails_49-0) NVIDIA. *NeMo Guardrails*. GitHub. https://github.com/NVIDIA-NeMo/Guardrails