NVIDIA's Nemotron 3.5 Transcribe Tops Open Speech Leaderboard With 30B Model

retained text

NVIDIA's Nemotron 3.5 Transcribe Tops Open Speech Leaderboard With 30B Model
NVIDIA's open-weight 30B mixture-of-experts speech model handles transcription, diarization, translation and audio Q&A in 40 languages from one checkpoint.
- NVIDIA released Nemotron 3.5 Transcribe, a 30B MoE model with ~4B active parameters, open-weight.
- Ranks #1 among open-weight models on the Open ASR English leaderboard at 5.33% WER.
- Single checkpoint handles transcription in 40 languages, translation in 31, diarization in 13, and audio Q&A.;
- Matches 24B Voxtral Small accuracy on Artificial Analysis with roughly one-sixth the active parameters.
- Fine-tuning on medical speech cut held-out WER by 45.8% in about 14 GPU-hours on one GB300.
- Companion Granary v2 dataset ships 463K hours across 48 languages with an open curation pipeline.
NVIDIA has released Nemotron 3.5 Transcribe, an open-weight speech model that follows natural-language instructions. The mixture-of-experts checkpoint contains 30 billion parameters and activates about 4 billion for each generated token. According to NVIDIA’s model card, it led the open-weight English automatic speech recognition leaderboard at release and recorded the best results on two speaker-diarization benchmarks.
Audio and a text prompt determine the task and output format. The same checkpoint can transcribe speech, assign words to speakers, translate audio, answer questions about a recording, and produce a transcript with personally identifiable information removed. The release overview also provides the Granary v2 data pipeline and NeMo fine-tuning recipes.
A prompt controls the transcript
Prompt examples
- Transcribe this recording and label each speaker.
- Translate this audio into English.
- What decisions were made in this meeting?
- Return a transcript with PII redacted.
Domain hints can improve recognition before fine-tuning. A healthcare prompt might supply candidate terms such as HbA1c and metformin, instruct the model to use them only when supported by the audio, and specify medical spelling and number formats.
Thirty billion parameters, four billion active
Word error rate, or WER, counts substitutions, deletions, and insertions relative to a reference transcript; lower values indicate fewer errors. NVIDIA reports the following results from the model card and associated leaderboards:
Nemotron tied a model with 24 billion active parameters on the Artificial Analysis index while activating about 4 billion. AA-AgentTalk, a held-out voice-agent dataset, contributes half of that index. Results across benchmark rows are not directly comparable because each dataset, language mix, and scoring procedure differs.
Diarization error rate measures missed speech, false detections, and time assigned to the wrong speaker. Nemotron recorded the lowest rate among 14 systems on David AI’s 21-language benchmark, slightly ahead of NVIDIA’s dedicated Nemotron 3 Diarization model.
On the VoiceArena methodology benchmark, it achieved the lowest concatenated minimum-permutation WER among 14 systems and ranked first for both in-person and online sessions. That metric maps predicted speaker labels to reference speakers across a full session before calculating WER. Deletions accounted for 8.32% of reference words, compared with a benchmark median of about 31%.
Speaker cues enter the decoder
The language model combines Mamba state-space layers, Transformer attention layers, and sparsely routed expert modules. About 4 billion parameters process each token, reducing computation relative to activating all 30 billion. The active-parameter figure describes per-token work rather than total checkpoint size, which remains relevant to memory planning.
Sortformer speaker cues reach the language model alongside acoustic features. The decoder can therefore assign speaker labels while generating the transcript, preserving acoustic and speaker information within the same generation process.
Granary v2 teaches output choices
The public Granary v2 manifest covers 463,062 hours across 48 languages and 16 source corpora, up from 25 languages in Granary v1. Its pipeline creates three parallel transcript forms for each clip:
- tn_raw: text-normalized verbatim, such as “I paid, uh, twenty-five euros.”
- itn_raw: inverse-text-normalized with disfluencies, such as “I paid, uh, €25.”
- itn_clean: inverse-text-normalized without disfluencies, such as “I paid €25.”
Parallel forms make punctuation, disfluency handling, and number formatting explicit training targets. The pipeline also generates short captions, restores loanwords in code-switched speech, creates speech-grounded question-and-answer pairs, extracts contextual biasing terms, and assigns automatic quality scores to translations.
Medical tuning cuts WER
NVIDIA evaluated domain adaptation on Eka Care’s benchmark of recordings from Indian doctors and medical students. The reported progression shows gains from prompt design and further gains from fine-tuning:
On held-out speakers, fine-tuning reduced WER from 22.44% with the strongest prompt to 12.17%, a 45.8% relative reduction. Using the same speakers and scorer, the adapted model outperformed an open medical ASR specialist at 13.79% and a widely used general-purpose model at 14.92%.
The recipe trains the speech encoder and projection layer while applying low-rank adapters, or LoRA, to the language model. NVIDIA averaged checkpoints from two runs with different data mixtures. The workflow, including text generation and evaluation, took about 14 GPU-hours on one NVIDIA GB300.
Domain gains introduced measurable trade-offs. LibriSpeech test-clean WER rose from 1.54% to 1.69% after medical fine-tuning, and the specialist medical model retained the lead on brand-name drug recognition.
Batch deployment at high throughput
NVIDIA says the model runs on a single NVIDIA GPU, with targets ranging from Jetson AGX Thor and DGX Spark to Blackwell data-center hardware. Developers can serve the checkpoint through vLLM or run it locally with NeMo-Speech.cpp. The model accepts recorded audio and does not support live streaming.
One RTX PRO 6000 Blackwell Server Edition GPU processed 100 hours of audio in under three minutes using NVFP4 precision, 30-second clips, and high concurrency. That measurement represents aggregate batch throughput under the tested configuration; it does not establish single-file latency.
Baseten, DeepInfra, and Simplismart provide hosted endpoints for teams that do not want to operate the model. NVIDIA also offers a live demo for testing prompts on uploaded clips.
Where the speech stack gets shorter
Production speech systems commonly chain separate components for recognition, speaker attribution, translation, summarization, and redaction. Each handoff adds an interface, another deployment dependency, and an opportunity to lose timing or speaker context. Nemotron can generate several of those outputs from one recording and prompt.
Its clearest applications involve recorded, multi-speaker audio: meeting analysis, call-center review, media captioning, and medical dictation. Teams still need domain-specific evaluation for accents, overlapping speech, terminology, privacy requirements, and output consistency.
The release provides a complete adaptation path from data preparation through fine-tuning and serving. NVIDIA’s medical case study demonstrates that substantial domain gains are possible on one GPU, while the general-English regression shows why adapted checkpoints require evaluation beyond their target dataset.

document fetched 2026-10-29 03:02