From Three Models to One: How Deepdub Extended NVIDIA Nemotron 3.5 ASR for Real-Time Voice AI

Fine-tuning a single 600M-parameter streaming model to transcribe Hebrew, identify speaker gender, and detect the end of a turn - headed for production on NVIDIA Triton Inference Server.

Quick Summary

Deepdub builds premium voice AI for media localization and real-time conversational agents - products where a live audio stream has to be understood in milliseconds, and where understanding means more than words. Who is speaking, and when they have finished, matter as much as what was said. Until now, answering those three questions meant running three separate models on every stream. NVIDIA Nemotron 3.5 ASR a 600M-parameter open-weights streaming speech recognition modelon Deepdub's aligned Hebrew data - and by extending its output vocabulary with dedicated gender and end-of-turn tokens - we finetunes  against real-world noisefor large-scale serving on NVIDIA Triton Inference Server.

Voice AI That Has to Work in the Real World

Deepdub's technology powers two demanding families of products. Our localization platform brings film, television, and live broadcasts to global audiences in more than 100 languages and dialects, while our real-time voice agents hold natural spoken conversations with customers at enterprise scale, on infrastructure accelerated by NVIDIA GPUs.

Both product lines begin at the same point: audio arriving live, from callers on compressed telephone lines, commentators over roaring crowds, or speakers in untreated rooms. Neither can wait for a batch pipeline, and neither is served by a bare transcript.

Hebrew makes the problem concrete. It is a morphologically rich, grammatically gendered language, historically underserved by speech technology, and central to a market where Deepdub has deep roots and demanding enterprise customers. It is exactly the kind of language where an off-the-shelf ASR API falls short - and where owning and shaping the model pays off.

Why a Transcript Alone Isn't Enough

Every live stream flowing through our systems has to answer three questions, continuously:

What was said. Accurate, punctuated Hebrew transcription that survives accents, disfluencies, and noise.

Who is speaking. Hebrew is a gendered language: verbs, pronouns, and adjectives change form depending on whether you are addressing a man or a woman. A voice agent that cannot infer a caller's gender from the audio cannot even reply grammatically. In dubbing workflows, the same signal drives voice casting and speaker matching.

When they finished. In conversation, silence is not the end of a turn. People hesitate mid-sentence, think out loud, and trail off. Endpointing on silence alone makes agents interrupt too early or answer too late - and either one breaks the feeling of natural conversation.

The conventional architecture assigns each question its own model: a streaming ASR model, a speaker-gender classifier, and an end-of-turn detector layered over voice activity detection. It works, but it triples what we deploy, monitor, version, and pay GPU memory for - and every extra hop adds latency to a pipeline where each millisecond is felt by a listener.

Why NVIDIA Nemotron 3.5 ASR

NVIDIA Nemotron 3.5 ASR (nemotron-3.5-asr-streaming-0.6b) is a 600M-parameter multilingual streaming ASR model built on a Cache-Aware FastConformer-RNNT architecture. Instead of re-processing overlapping audio windows the way buffered streaming systems do, it caches the encoder's internal state and processes each frame exactly once - delivering low latency without paying for it in accuracy or compute. A single checkpoint covers 40 language-locales, Hebrew among them, generates punctuation and capitalization natively, and exposes its latency/accuracy operating point at inference time, from 80 ms chunks for voice agents up to 1.12 s for maximum accuracy.

Just as important, the model ships as open weights. We can inspect it, fine-tune it on proprietary data, and run it entirely inside our own infrastructure - with no per-call API billing and no customer audio leaving our environment.

But the property that mattered most to us is almost mundane: the model's output is a token stream. An RNNT decoder emits tokens as audio arrives, and nothing says those tokens must be words. Extend the vocabulary, and the model can be taught to emit signals. That single observation is what turns a speech recognizer into a real-time speech-understanding engine.

Teaching One Model to Do Three Jobs

We continued the training of Nemotron 3.5 ASR in four directions at once.

1. Production-grade Hebrew

Hebrew is one of the base model's 40 supported locales, but like other lower-resource languages it had the most headroom - NVIDIA's own experiments showed 31–32% relative word-error-rate reductions on Greek and Bulgarian from targeted continued training. We fine-tuned the model on Deepdub's aligned corpus, built and refined over years of professional localization work: real conversational speech, in real acoustic conditions, with transcripts matched to the punctuated, properly-cased text style the model produces.

The corpus behind that fine-tune is 3,000 hours of real human dialogs – natural rather than scripted or synthetic speech. Every hour was curated through Deepdub's ETL and alignment framework, the same production pipeline that powers our localization work: it ingests raw audio, segments it into utterances, aligns each segment with its transcript, and enforces consistent punctuation, casing, and text conventions across the set. The result is training data that already looks exactly like the model's target output - and because real dialogs carry natural turn-taking, hesitations, and speakers of both genders, the same corpus directly supports the gender and end-of-turn objectives described below.

2. A gender token

We enriched the training data with speaker-gender labels and extended the model's output vocabulary with dedicated male and female tokens. The fine-tuned model now predicts the speaker's gender inline with the transcript, in the same streaming pass - no second classifier, no extra hop, no added latency. Downstream, that one token is what lets our agents conjugate Hebrew correctly and lets our dubbing pipeline select gender-appropriate voices automatically.

3. Resilience to real-world noise

Production audio is rarely clean. We applied aggressive augmentation to the training data - injecting high levels of background noise into the training mix - so that transcription, gender prediction, and turn detection all hold up in the acoustic conditions where our customers actually operate: contact centers, live broadcast feeds, and crowded environments. The goal is a model whose accuracy degrades gracefully when conditions do, rather than collapsing.

4. An end-of-turn token

Finally, we added a dedicated end-of-turn (EOT) token to the vocabulary and generated synthetic training data emulating the turn structure of real calls: hesitations, mid-sentence pauses, back-and-forth exchanges, and genuine turn completions. Because the model sees both the acoustics and the words, it learns to distinguish “the speaker paused” from “the speaker is done” - a distinction that silence-based endpointing, by definition, cannot make.

From Three Models to One

The result is a single 600M-parameter streaming checkpoint that produces, in one forward pass per audio chunk, the Hebrew transcript, the speaker's gender, and end-of-turn events.

Figure 1. Deepdub's real-time voice-agent loop. Microphone audio streams into the fine-tuned Nemotron 3.5 ASR, which emits the Hebrew transcript together with gender and end-of-turn tokens. When the EOT token fires, the transcript and gender condition the LLM's response, which Deepdub's TTS synthesizes and plays back to the user - with the ASR stage served on NVIDIA Triton Inference Server.

The consolidation compounds:

• One engine to deploy, monitor, and version instead of three.
• A fraction of the GPU footprint per concurrent stream, which translates directly into serving cost.
• Lower, more predictable latency: no fan-out to secondary models and no cross-model synchronization on the hot path.
• One training loop: every improvement to our data now improves transcription, gender detection, and turn detection together.

And because the cache-aware encoder processes each frame exactly once, all three signals arrive at the cost of streaming ASR alone - not streaming ASR plus two side models.

Serving at Scale with NVIDIA Triton Inference Server

We plan to serve the unified model with NVIDIA Triton Inference Server, which already fits the GPU-accelerated stack behind Deepdub's production ASR and TTS. Triton's dynamic batching and concurrent model execution let us pack many low-latency streams onto each GPU; its sequence handling suits the stateful nature of cache-aware streaming inference; and its standard gRPC/HTTP interfaces and built-in metrics drop cleanly into our existing observability. Consolidation simplifies the deployment itself, too: one model repository entry, one scaling policy, one set of health metrics - instead of three of each.

Looking Ahead

Nothing about this recipe is specific to Hebrew, or to gender and turn-taking. The same token-space extension can carry other signals our products need in real time, and the same continued-training approach applies to every other locale our customers speak. We started with Hebrew because it is hard: gendered, morphologically rich, and unforgiving of shortcuts. A model that earns its place there earns its place anywhere.

Open weights made this possible. Because Nemotron 3.5 ASR can be inspected, retrained, and extended, we were able to reshape a general-purpose multilingual recognizer into a purpose-built engine for our pipelines - something no closed API allows.

Deepdub is a member of the NVIDIA Inception program, a global program that nurtures AI startups with developer training and resources, offers from NVIDIA and its partners, and preferred access to NVIDIA's AI ecosystem.

Resources

• Nemotron 3.5 ASR model card and open weights: huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
• NVIDIA's fine-tuning guide for Nemotron 3.5 ASR: huggingface.co/blog/nvidia/fine-tuning-nemotron-35-asr
• NVIDIA Triton Inference Server: developer.nvidia.com/triton-inference-server
• Deepdub: deepdub.ai

About Deepdub

Deepdub is a premium voice AI company pioneering expressive voice technologies and voice harnesses for voice agents and dubbing. Its platform powers real-time, emotionally expressive voice agents for enterprises as well as, AI dubbing and localization for film, television, and live broadcasts - combining proprietary voice models with GPU-accelerated infrastructure to deliver studio-grade quality at production scale.

By the Deepdub Research - Nir Krakowski, Moshe Michaelshvili, Chen Zarfati, Ohad Zohar, Tamir Cohen · August 2026