← sources

Voice dictation with one model.

source: reddit (reddit:LocalLLaMA)published 2026-11-13 20:08ingested 2026-11-14 02:42

retained text

Hey, I'm 14 and was looking into how AI voice dictation apps like Wispr Flow work.

As you might know they use a transcription model + LLM. The transcriber outputs one to one text as it hears it and then the LLM gets rid of fillers like "um" and adds punctuation, flow, corrected restarts and formatting.

I ran that on my laptop (RTX 3050 Ti, 4 GB VRAM), then trained a single model to do both steps at once. Here are some results:

**Speed.**

Whisper (base, large-v3-turbo or large-v3) plus Qwen3.5 (4B or 9B) took between 2.5 and 36 seconds per dictation, depending on the pair. The single model takes about 0.5 seconds for a short dictation and about 1 second for a long one.

Why a traditional dictation approach is slow locally:

\- Two models don't fit in 4 GB together. One ends up on the CPU, or they get swapped in and out.

\- The text is generated twice. Whisper writes the transcript, then the LLM reads it and writes all of it again.

\- The LLM also reads a cleanup prompt on every dictation.

**Compute.**

One model is a 2.2 GB file (Q8) and uses about 2.5 GB of VRAM. It loads once and stays loaded. An average dictation set-up running locally uses more than 4GB of VRAM

**Quality.**

The cleanup LLM only sees text. It can't hear that you stopped and started over, so it sometimes keeps the part you took back, and it can't fix a word Whisper got wrong. A single model can catch vocal cues and keep what is really worth keeping.

**Training**

The two-model pipeline still exists, but only at training time. I took real, unscripted speech from openly licensed datasets, had a large LLM write the finished text for each clip from its transcript, threw out the clips where that dropped real content, and trained Qwen3-ASR 1.7B (Apache-2.0) to go from audio straight to the finished text. It runs in llama.cpp.

Currently it, from my own experiences is one of, if not the best dictation tool that uses this little compute.

I put this into a free, open source dictation app called Typd (Windows with an NVIDIA GPU, Apple silicon Macs): [https://github.com/SameRainbows/typd-app](https://github.com/SameRainbows/typd-app) and [https://typd.mehmetdedeler.com](https://typd.mehmetdedeler.com).

Have you tried anything similiar?

document fetched 2026-11-14 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xvia6x/voice_dictation_with_one_model