The constant evolution of large language models (LLMs) presents both immense opportunities and significant deployment challenges for developers. Building sophisticated AI agents, robust chatbots, or efficient RAG systems requires models that are not only powerful but also highly optimized for various hardware configurations and use cases. NVIDIA addresses this with the release of Nemotron 3.5 Lightning, a new addition to their family of open models designed to deliver leading efficiency and accuracy for specialized AI agent development.

Nemotron 3.5 Lightning stands out by offering open weights, training data, and recipes, empowering developers with transparency and control over their AI solutions. Its focus on agentic workflows makes it a compelling choice for applications demanding long-running autonomous agents, sub-agent workhorse deployments, and efficient local inference, even on personal hardware. This model, released under the OpenMDW-1.1 License Agreement, aims to democratize access to high-performance LLMs, slated for general availability on August 11, 2026.

Architecture and Training: A Deep Dive

At its core, Nemotron 3.5 Lightning is a 30 billion parameter model with 3 billion actively utilized parameters, leveraging a sophisticated hybrid Mixture-of-Experts (MoE) architecture. This design intelligently interweaves Mamba-2 and MoE layers with select Attention layers, creating a powerful synergy between the efficiency of state-space models and the robustness of Transformers. This hybrid approach is key to its performance and ability to handle extensive contexts.

The model boasts an impressive context length of up to 1 million tokens, a capability crucial for complex agentic tasks and deep document understanding. This extensive context window is supported by a rigorous, multi-stage training methodology:

1. Pre-Training: The initial phase involved training on over 20 trillion tokens using an NVFP4 recipe. This vast corpus comprised crawled and synthetically generated data across code, math, science, and general knowledge domains, managed with Megatron-LM. The pre-training data had a cutoff of September 2025.
2. Continued Pre-Training for Multi-Token Prediction (MTP): A distinctive feature, MTP layers were specifically trained to predict multiple future tokens, not just the next one. This enriches the training signals, enhancing the base model's distributional alignment and overall predictive power.
3. Supervised Fine-Tuning (SFT): The model underwent fine-tuning on synthetic data tailored for code, math, science, tool calling, instruction following, structured outputs, and general knowledge. This stage specifically incorporated data to bolster long-range retrieval and multi-document aggregation capabilities.
4. Reinforcement Learning (RL): Employing Group Relative Policy Optimization (GRPO) in a multi-environment setting, the model was refined across diverse tasks including math, code, science, instruction following, multi-step tool use, multi-turn conversations, and structured output. An asynchronous RL architecture, decoupled training from inference and leveraged MTP to accelerate rollout generation, using NeMo RL and NeMo Gym.
5. Post-Training Quantization (PTQ): To optimize for deployment efficiency, PTQ was applied using Nvidia Model Optimizer. This involved a "Four Over Six NVFP4" recipe, with W4A16 applied to routed and shared experts, and FP8 per-tensor dynamic scales for Mamba in_proj/out_proj and KV cache. Calibration was performed using a subset of Nemotron Ultra validation data. The post-training data cutoff was May 2026.

Optimized Deployment and Speculative Decoding

Nemotron 3.5 Lightning is engineered for high-performance inference, particularly on NVIDIA GPU-accelerated systems. It supports the latest NVIDIA Blackwell GPUs (DGX Spark / GB10, GB200, GeForce RTX 5090), NVIDIA Hopper (H100, H200), and even NVIDIA Ampere GPUs via W4A16 quantization.

A key to its rapid text generation is the integration of multiple speculative decoding methods:

• DSpark: Recommended for DGX Spark and low-concurrency data center deployments, DSpark is a semi-autoregressive drafter that proposes a block of candidate tokens in a single forward pass using a parallel backbone.
• DFlash: This method utilizes a lightweight block-diffusion model to generate an entire draft block in one forward pass.
• MTP (Multi-Token Prediction): Beyond its training benefits, MTP is also leveraged as a speculative decoding strategy, where the network predicts several future tokens simultaneously.

The model is readily deployable using popular serving frameworks like vLLM, TensorRT-LLM, and SGLang. Below are examples demonstrating deployment with vLLM, showcasing its flexibility across different hardware:

python export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark

1x DGX Spark (GB10) with DSpark speculative decoding

vllm serve --model $MODEL_CKPT
--moe-backend marlin
--kv-cache-dtype fp8
--max-model-len 1048576
--enable-prefix-caching
--speculative_config.num_speculative_tokens 3
--mamba-backend flashinfer
--mamba-cache-mode align
--reasoning-parser nemotron_v3
--speculative_config.method dspark
--tool-call-parser qwen3_coder
--enable-auto-tool-choice

1x H100 for max throughput (no speculative decoding for max batching)

vllm serve --model $MODEL_CKPT
--max-num-seqs 256
--max-num-batched-tokens 16384
--enable-prefix-caching
--async-scheduling
--mamba-backend flashinfer
--moe-backend humming
--linear-backend humming
--mamba-ssu-algorithm horizontal
--mamba-cache-mode align
--mamba-ssm-cache-dtype float16
--enable-mamba-cache-stochastic-rounding
--mamba-cache-philox-rounds 5
--reasoning-parser nemotron_v3
--tool-call-parser qwen3_coder
--enable-auto-tool-choice

W4A16 on Ampere-class GPUs

vllm serve --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
--moe-backend humming
--linear-backend humming
--max-num-seqs 256
--max-num-batched-tokens 32768
--enable-prefix-caching
--async-scheduling
--quantization modelopt_fp4
--mamba-backend flashinfer
--mamba-cache-mode align
--mamba-ssu-algorithm simple
--reasoning-parser nemotron_v3
--tool-call-parser qwen3_coder
--enable-auto-tool_choice

Developers targeting high throughput on H100s might opt for configurations without speculative decoding to maximize batching, though DSpark can still be used for interactive scenarios with lower concurrency. For multi-GPU serving, such as an 8x H100 setup, expert parallelism is enabled to handle long contexts efficiently. It's important to note that the full 1M-token context window is served by default on H100 and GB200, but --max-model-len can be adjusted to conserve KV-cache memory, especially at high concurrency.

API Client and Advanced Features

Interacting with Nemotron 3.5 Lightning is straightforward using an OpenAI-compatible client. This allows seamless integration into existing workflows. Recommended sampling settings are a temperature of 1.0 and top_p of 0.95.

The model offers granular control over its behavior, including explicit reasoning control:

python from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") MODEL = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4"

Reasoning ON (default behavior)

response_on = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "Write a haiku about GPUs"}], max_tokens=16000, temperature=1.0, top_p=0.95, extra_body={"chat_template_kwargs": {"enable_thinking": True}} ) print("Reasoning ON:", response_on.choices[0].message.content)

Reasoning OFF (direct answer)

response_off = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "What is the capital of Japan?"}], max_tokens=16000, temperature=1.0, top_p=0.95, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) print("Reasoning OFF:", response_off.choices[0].message.content)

Streaming response example

stream = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "Explain speculative decoding in two sentences"}], max_tokens=16000, temperature=1.0, top_p=0.95, stream=True, ) print("Streaming response:") for chunk in stream: print(chunk.choices[0].delta.content or "", end="", flush=True) print(" ")

For AI agent systems, Nemotron 3.5 Lightning provides robust tool-calling capabilities. Ensure your serving backend is launched with appropriate parsers (--reasoning-parser nemotron_v3 --tool-call-parser qwen3_coder). For coding agents, it's advised to add extra_body={"chat_template_kwargs": {"force_nonempty_content": True}} to API calls to ensure non-empty content responses.

python tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }]

tool_response = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "What's the weather in Santa Clara?"}], tools=tools, max_tokens=16000, temperature=1.0, top_p=0.95, extra_body={"chat_template_kwargs": {"force_nonempty_content": True}}, ) print("Tool Call:", tool_response.choices[0].message.tool_calls)

Data and Ethical Considerations

Nemotron 3.5 Lightning was trained on an extensive corpus of over 20 trillion tokens, encompassing English, 19 other spoken languages, and 43 programming languages. The data sources are diverse, including webpages, dialogue, articles, and specialized content in legal, math, science, and finance domains. Post-training further expanded language coverage to include Chinese, among others.

NVIDIA acknowledges and actively addresses potential biases in the training data. Datasets like FinePDFs and EssentialWeb showed representational skews (e.g., more mentions of "male" than "female," "White" as the most frequent ethnic identifier). To mitigate these, NVIDIA recommends strategies such as bias audits, fine-tuning with demographically balanced datasets, and counterfactual data augmentation. The data generation process for synthetic data involves distillation from strong teacher models, aggressive quality filtering, and automated verification (compilers, numerical checks, language identification). Crucially, the pipeline includes targeted keyword and regex filters to remove reasoning traces that might implicitly align with specific political entities or promote nationalistic narratives. NVIDIA commits to releasing its final pre-training and post-training data, with a sample set openly available and gated access for remaining code, math, and multilingual data for training purposes.

Practical Takeaways for Developers

NVIDIA Nemotron 3.5 Lightning presents a robust, performant, and flexible foundation for developing next-generation AI agents and applications. Its hybrid MoE architecture, combined with Mamba-2, delivers efficiency and accuracy, especially for long context windows up to 1M tokens. The open-source nature (weights, data, recipes) provides an unprecedented level of control and transparency. Developers can leverage sophisticated speculative decoding techniques (DSpark, DFlash, MTP) and optimized deployments across a range of NVIDIA hardware, from personal RTX GPUs to enterprise-grade Blackwell and Hopper systems, through vLLM, TensorRT-LLM, and SGLang. The model's strong agentic capabilities, reasoning control, and tool-calling features position it as a powerful workhorse for complex, multi-step AI workflows, while NVIDIA's commitment to addressing data biases supports responsible AI development.

FAQ

Q: What specific hardware is recommended for deploying Nemotron 3.5 Lightning for optimal performance?

A: For optimal performance, NVIDIA Blackwell GPUs such as DGX Spark (GB10) or GB200 are recommended. NVIDIA Hopper series (H100, H200) also provide excellent performance. For deployments on Ampere-class GPUs, W4A16 quantization is supported, extending its reach to a broader range of NVIDIA hardware.

Q: How does Nemotron 3.5 Lightning handle long context lengths, and what are the implications for KV-cache management?

A: Nemotron 3.5 Lightning supports an impressive context length of up to 1 million tokens, a feature critical for complex agentic tasks. During deployment, frameworks like vLLM can serve the full context window by default. However, managing the KV-cache efficiently is crucial. If you are memory-constrained or require more KV-cache headroom at high concurrency, it is recommended to adjust the --max-model-len parameter to a smaller value that matches your specific workload requirements.

Q: What is the significance of Multi-Token Prediction (MTP) in Nemotron 3.5 Lightning?

A: MTP plays a dual role in Nemotron 3.5 Lightning. During the continued pre-training phase, MTP layers are trained to predict several future tokens simultaneously, providing richer training signals and improving the base model's overall predictive accuracy and distributional alignment. In deployment, MTP also functions as a speculative decoding strategy, enabling the model to generate multiple candidate tokens at once, thereby accelerating text generation and enhancing inference efficiency.