NVIDIA's Nemotron 3.5 Bolt Squeezes a Local AI Agent Into 4GB

retained text

NVIDIA's Nemotron 3.5 Bolt Squeezes a Local AI Agent Into 4GB
NVIDIA's new 4GB open model targets local voice assistants, NPCs and robots, outperforming peers on conversation benchmarks while using 80% fewer tokens.
- NVIDIA released Nemotron 3.5 Bolt E4B 4GB, an open small model built on Gemma 4 E4B via RL post-training
- Targets local voice assistants, game NPCs and robots on Jetson, RTX and DGX Spark hardware
- Runs in ~4GB VRAM with 548ms TTFT on RTX 5070; outputs up to 80% fewer tokens than peers
- Wins on role-play, instruction hierarchy and tool grounding; loses to Qwen 3.5 4B on Tau2-Bench and long context
- Ships with DFlash parallel drafter delivering 21% higher throughput over Gemma's MTP drafter
- Weights, data, and training recipes available on NVIDIA Research, Ollama, LM Studio and Unsloth
NVIDIA has released Nemotron 3.5 Bolt, a compact language model derived from Google’s Gemma 4 E4B. NVIDIA post-trained the model for conversational agents, tool use, and local deployment on Jetson boards, RTX GPUs, and DGX Spark workstations.
The roughly 4GB deployment target leaves memory for speech, vision, retrieval, and application code. That headroom matters on an 8GB Jetson or 12GB consumer GPU, where the language model must share resources with the rest of the agent stack.
Four gigabytes leaves room for the stack
NVIDIA tuned Bolt around the behavior and resource constraints of interactive agents:
- Multi-turn tool use: Supports asynchronous tool calls across a conversation.
- Conversational mode: Produces concise responses suited to speech interfaces.
- Configurable reasoning: Allows reasoning to be disabled or capped to meet latency budgets.
- 64K-token context: Accommodates conversation history, tool definitions, retrieved data, and application state.
- 35 supported languages: Builds on a base model pretrained across more than 140 languages.
In NVIDIA’s reported evaluations, Bolt averages about 40% higher quality than Nemotron 3 Nano 4B across selected conversational-agent benchmarks. It also generates up to 80% fewer tokens than peer models in the company’s comparisons. Shorter outputs reduce decoding work, response latency, and gaps in voice interactions.
Persona leads, long-context losses
NVIDIA evaluates the model across conversation and persona, multi-turn tool use, instruction control, context robustness, and state and action grounding. The company compares Bolt with Qwen 3.5 4B and Gemma 4 E4B, with Bolt leading most of the reported tests.
RoleBench, IHEval, and RelyToolBench indicate strong persona control, instruction handling, and tool grounding. NVIDIA also reports gains on counterintuitive instructions and complex agent workflows.
Qwen leads Bolt on the Tau2-Bench results, BFCL-v3, and LongBench v2 at 48K context. Those results favor Qwen for structured customer-service flows, function-heavy applications, and long-document reasoning. Bolt’s 64K context window defines its capacity; the LongBench score shows that context length alone does not guarantee stronger reasoning across the full window.
Edge latency sets the limit
Quantized deployments provide the approximately 4GB footprint used in NVIDIA’s hardware tests. The company ran Q4_K_M builds through llama.cpp at batch size 1, using prompts with 3,000 input tokens and 50 generated tokens. BF16 checkpoints require more memory.
At nearly 5.8 seconds, the Orin Nano’s time to first token can disrupt a voice interface. Its 4.2GB model footprint still leaves roughly half of the device’s memory for perception and control software. NVIDIA demonstrates that constraint with the Reachy Mini demo, which runs Bolt alongside the Cosmos 3 Edge vision model on one Orin Nano 8GB.
Post-training trims the chatter
Bolt starts from Gemma 4 E4B and adds post-training for instruction following, multi-turn tool calls, and conversation. NVIDIA used reinforcement learning with verifiable rewards, or RLVR, to score outputs against checkable outcomes. Reinforcement learning from human feedback, or RLHF, supplied preference signals. Both pipelines ran on NeMo RL and NeMo-Gym.
Each training sample appeared in reasoning and non-reasoning modes, with two non-reasoning examples for every reasoning example in a batch. That mix favors brief responses for routine agent requests while retaining configurable reasoning for harder tasks.
As post-training progressed, reasoning traces and final answers grew longer. NVIDIA countered that verbosity growth by increasing group-relative penalties for reasoning and answer length in its generative reward-model environment. The resulting model produces substantially fewer tokens in the company’s peer comparisons.
For Q4_K_M quantization, NVIDIA calibrated llama.cpp’s importance matrix with about one million tokens of reinforcement-learning rollouts from the same training checkpoint. The calibration identifies weights that are more sensitive to compression, helping the quantized build remain close to the reported BF16 benchmark scores.
DFlash drafts tokens in parallel
DFlash, the release’s speculative-decoding drafter, proposes multiple draft tokens in one forward pass. Bolt then verifies those candidates in parallel, preserving the target model’s output distribution. NVIDIA reports up to 21% higher throughput than Gemma 4 E4B’s original multi-token prediction drafter, although the realized gain depends on hardware, prompt length, and runtime configuration.
Downloads and deployment paths
NVIDIA has published the technical report, model checkpoints, and reinforcement-learning dataset list. The available assets include:
- Model weights: BF16 and GGUF Q4_K_M checkpoints.
- Drafting support: An NVFP4 DFlash drafter variant.
- Training resources: The report, dataset list, NeMo RL, and NeMo-Gym references.
- Deployment options: Ollama, LM Studio, Unsloth, Jetson AI Lab, and NVIDIA’s In-Game Inferencing SDK.
Developers should review the model card’s license and use terms before distribution, especially when bundling weights into commercial software or devices.
Match the model to the workload
Production evaluations need to reproduce tool-call success, time to first token, decode throughput, answer length, peak memory, and behavior near the context limit on the target hardware. Those measurements will determine whether Bolt’s compact footprint and concise output outweigh Qwen’s gains on structured workflows and long-context reasoning.

document fetched 2026-10-24 15:22