NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning

extract

Always-on agents are taking on longer, more complex enterprise workflows. A single request can involve reading a document, checking information across sources, following company policy, and calling several tools. NVIDIA Nemotron 3.5 Super VL is an open, customizable multimodal model for these specialized workflows. Its 120B-parameter mixture-of-experts (MoE) architecture activates 12B parameters per token, balancing model capacity with inference efficiency. On a single GPU, this model delivers up to 5x higher throughput for agentic tasks compared to other open models in its size category. The model combines reasoning, instruction following, and tool use with text, image, document, and video understanding. By combining these capabilities into one model, teams can handle text and visual tasks without deploying separate text and vision-language models or passing task context between them. Developers can post-train Nemotron 3.5 Super VL with NVIDIA NeMo using their own data, tools, policies, and workflows. This post explains how to use that foundation to build specialized enterprise agents and describes the training and inference techniques that power it. Why Nemotron 3.5 Super VL for specialized agents? Agents need to understand both their tasks and operating environments. A customer-service agent must follow company return policies. Research agents connect evidence across sources, while document agents interpret tables, charts, and page layouts. Nemotron 3.5 Super VL combines reasoning and visual understanding across a range of agent workflows, including: - Coding: Test and validate code, analyze errors, and recommend fixes. - Web research: Search across websites, connect information, and verify answers. - Document intelligence: Extract information from tables, charts, and page layouts to produce summaries and reports. - Cybersecurity: Identify security vulnerabilities and triage threats to prioritize remediation. - Video understanding: Extract information for search, content review, and incident analysis. Within a system of models, Nemotron 3.5 Super VL handles specialized, multistep work that needs reasoning, tool use, and text and visual understanding together. Smaller models, such as NVIDIA Nemotron 3.5 Lightning, handle high-volume text tasks, while frontier models such as NVIDIA Nemotron 3 Ultra tackle complex planning and orchestration. Customize Nemotron 3.5 Super VL for domain data, tools, and policies Specialized agents must learn how an organization works. Domain terminology, tool interfaces, operating procedures, and policy requirements shape how agents complete tasks successfully. Organizations in industries ranging from cybersecurity and finance to healthcare are post-training Nemotron 3.5 Super VL to outperform frontier models in accuracy and efficiency across their production workflows. For example: - Scale AI, an AI data and technology provider, post-trained Nemotron 3.5 Super VL for automated vendor credit processing at Skydance Corporation, a media and entertainment company. Inference cost for this workload will decrease by 40x and latency by 5x. - Ramp, a financial operations platform provider, is converting receipt images into structured JSON containing merchant information, dates, and payment data. Ramp expects up to 50% higher throughput and 10x lower latency. - AI21 and Alice post-trained Nemotron 3.5 Super VL for threat intelligence and vulnerability analysis, improving accuracy by 41%. Teams can use NVIDIA NeMo open libraries to post-train models to meet these requirements. Developers can use NVIDIA NeMo AutoModel for supervised fine-tuning (SFT) and low-rank adaptation (LoRA) to teach domain-specific behavior. Reinforcement learning (RL) improves performance in agentic environments that reflect organizational workflows. Alongside Nemotron 3.5 Super VL, NVIDIA is releasing open datasets on Hugging Face. These include agentic tool-use and conversational data, finance reasoning and tool-calling datasets, instruction-following and chat data, long-context transduction and translation data, fairness data, and 40 vision-language datasets. These datasets can supplement proprietary data for SFT and RL. For multimodal data, start with NeMo Data Designer resources for chart question answering and long-document understanding. Explore NVIDIA NeMo RL and NVIDIA NeMo Gym for RL and environment-based evaluation. Organizations can adapt and deploy the model’s open weights in supported local, data center, or cloud environments to meet their infrastructure and data requirements. For customization resources, see the Nemotron 3.5 Super VL usage cookbook on GitHub. Combine text and visual understanding in one workflow Nemotron 3.5 Super VL combines text, image, document, and video understanding with reasoning and instruction following to answer questions across these inputs. Its visual training covers charts, tables, optical character recognition (OCR), multipage documents, and video understanding. A two-stage SFT pipeline trains the language and vision components, combining visual evidence with reasoning, chat, safety, long-context data, and agent trajectories. For document intelligence, the same model can interpret a chart, connect it with a table elsewhere in a document, and summarize both. Developers can tailor the workflow to their own documents and evaluation criteria. Diverse training data generated with NeMo Data Designer helped the model better understand complex charts. On CharXiv, a chart-reasoning benchmark, the model scored 80.8%, compared with 63% for Nemotron 3 Nano Omni. With Python code interpretation, it reached 84%. For the approach and lessons learned, read the chart-understanding deep dive and the open-source chart question-answering pipeline. This model was built with support and expertise from Nemotron Coalition organizations. For instance, high-quality datasets from Perplexity helped improve and streamline the model’s mid-training. Environments from Prime Intellect enabled building better teacher models. Nemotron 3.5 Super VL is also one of the top open models on the OCRBench v2 leaderboard. Accuracy and efficiency across agent tasks Evaluations of Nemotron 3.5 Super VL span Hermes, LangChain, OpenClaw, OpenCode, Pi, Codex, and Claude. In addition to coding tasks, agent harness providers use their own evaluations to test a wide range of agent tasks. On Hermes Bench and LangChain behavioral evaluations, Nemotron 3.5 Super VL scores higher than Nemotron 3 Ultra. For always-on agents, efficiency determines how much work a deployment can complete. Throughput, token usage, tool calls, and the steps needed to reach a correct result all affect task completion time. This model was also optimized to run locally on NVIDIA DGX Spark. An independent evaluation by EXO Labs placed Nemotron 3.5 Super VL on the Pareto frontier for accuracy and task completion time. Nemotron 3.5 Super VL under the hood The model builds on a strong foundation and uses a hybrid architecture, speculative decoding, context reuse, and mixed precision to reduce inference work across repeated agent calls. Building a strong base model Nemotron 3.5 Super VL was distilled from Nemotron 3 Ultra. Ultra’s coding capabilities, combined with the SkewReverse loss used during distillation, helped improve coding accuracy. During post-training, multi-teacher on-policy distillation (MOPD) combines guidance from specialized teachers across text and vision. The model generates responses, and the teachers score them for video reasoning and tracking, vision tool use, and alignment. Additional training on examples of agent actions, code repositories, and software engineering tasks improved the model’s performance on agent tasks. Nemotron 3.5 Super VL outperforms Nemotron 3 Super on SWE-bench and Terminal-Bench across multiple evaluation setups. Continued pretraining with high-quality natural and synthetic data improved Indic-language capabilities and contributed to higher overall scores after post-training. This stage added 10B Indic tokens, drawn from a 12-language subset of Nemotron-pretraining-Indic-v1 and natural Indic text. NVIDIA is releasing the full dataset, which contains about 2.8T tokens across 22 Indian languages and was developed with IIT Madras and its AI4Bharat lab. The following results are based on the post-trained Nemotron 3.5 Super VL. Developers can use NeMo to build on this foundation and customize Nemotron 3.5 Super VL for their applications, adapting its reasoning and agentic capabilities to their data, tools, and workflows. LiLiCorr: An open speculative decoding technique LiLiCorr improves parallel speculative decoding by jointly evaluating token candidates across an entire draft, capturing dependencies that independent token predictions miss. A lightweight network computes compatibility scores in a single pass to produce more coherent drafts, leaving only a low-cost, sequential final selection step. On DGX Spark, LiLiCorr is the recommended drafter for low-latency inference with Nemotron 3.5 Super VL. It delivers up to 2.3x faster token generation than decoding without speculation. NVFP4 and mixed precision The model uses NVIDIA Model Optimizer for post-training quantization and quantization-aware distillation (QAD). The NVFP4 recipe uses 4-bit weights and activations on the NVIDIA Blackwell architecture. The same checkpoint runs on non-Blackwell platforms with 4-bit weights and 16-bit activations. Precision is selected separately for sensitive components. NVFP4 makes the model smaller, and QAD helps it retain its accuracy at lower precision. During QAD, the quantized model learns to match the outputs of the original BF16 model, which closes the accuracy gap that quantization introduces. QAD also supports storing the key-value (KV) cache in NVFP4 instead of FP8 while still meeting accuracy targets. Because QAD happens during training, it adds no inference overhead. Multi-token prediction Multi-token prediction (MTP) gives Nemotron 3.5 Super VL an internal drafter that proposes several tokens for the main model to verify in one forward pass. Accepting multiple tokens at once reduces the number of compute-intensive verification steps needed to generate a response. A dedicated MTP-boosting training phase further aligned the drafter with the final model. Context reuse across agent turns Agents repeatedly pass system instructions, tool definitions, and conversation history back to the model. Prefix caching reuses computation from those shared inputs. The model’s serving optimizations preserve both attention caches and Mamba recurrent states. Additional Mamba checkpoints reduce the replay needed to recover inputs for speculative decoding, while internal checkpointing avoids extra model passes during prompt processing. Simplify production deployment with NVIDIA NIM and NVIDIA Dynamo NVIDIA NIM packages models with optimized inference engines, validated configurations, and standard APIs for cloud and data center deployment. NIM receives ongoing security, feature, and performance updates. Enterprise support is available through NVIDIA AI Enterprise. Get started with NIM. NVIDIA Dynamo supports distributed inference optimizations to increase throughput and reduce latency, including KV-cache-aware routing, KV cache management, and disaggregated serving. On 16 NVIDIA B200 GPUs at 50 tokens per second per user, KV-aware routing with session affinity and KV cache offload serves up to 8x as many users while reducing first-token latency by 85%. Get started with the Dynamo deployment recipes. Get started with Nemotron 3.5 Super VL Build specialized agents with multimodal reasoning and tool use. Customize the model for specialized workflows, then deploy it in a supported environment. The release includes base, post-trained, and NVFP4-quantized checkpoints with training recipes, data, and reinforcement-learning environments. See the Nemotron 3.5 technical report for model details and Hugging Face or ModelScope for deployment instructions. For hosted model access, see OpenRouter. Partner ecosystem Partners support customization and deployment of Nemotron 3.5 Super VL for domain-specific workflows. View partners: - Actual Computer - Canonical - Exo - LM Studio - Ollama - SGLang - Unsloth - vLLM - Amazon SageMaker JumpStart - Google Cloud Gemini Enterprise Agent Platform - Microsoft Foundry - OCI Enterprise AI - HCLTech - Infosys - Tata Consultancy Services - Thoughtworks - Dell - HPE Services - Red Hat