I fine-tuned an LLM to clean up transcriptions for my voice-to-text app
retained text
I built a voice-to-text app, [WhisperNow](https://github.com/stephan271c/WhisperNow). But there are a thousand and one WisprFlow clones out there, so I went further and fine-tuned my own LLMs for the cleanup step. They remove fillers, false starts, self-corrections, and formats lists, all locally.
To finetune an LLM for this task, I first had to get data.
## The dataset
The dataset has 66,557 rows. I started by combining public datasets (aawaaz, handy and Disfl-QA) and adding synthetic dictations, such as emails, meeting notes, recipes and dictated commands. gpt-6.1-sol wrote the cleaned version of every transcript, using the same prompt the app sends to cloud models.
While training the models, I found some weak spots: long dictations, words the speech recognizer misheard, cut-off words ("interf interface") and correction words used literally ("scratch behind the dog's ears"). I generated more examples of each and retrained.
## The models
I did full fine-tunes of three base models, each on a single H100 or H200 rented on Modal:
| model | memory in use | 100-word dictation, RTX 5070 GPU | 100-word dictation, Ryzen 9 8945HS CPU |
|---|---|---|---|
| LFM2.5 1.2B (the app's default) | 1.6 GB | 0.18 s | 2.3 s |
| LFM2.5 2.6B | 3.3 GB | 0.34 s | 5.4 s |
| Qwen3.5 4B | 5.3 GB | 0.62 s | 8.9 s |
Through their APIs, Claude Haiku 5.5 and gpt-6-luna take about 1.4 s on the same dictation, glm-5.3-flash 2.6 s, and glm-5.3 at max reasoning 5.6 s.
To score the models, a judge model compares each output with the reference and counts factual errors: facts dropped, changed, kept from the abandoned side of a correction, or invented. The test rows are a held-out test split plus two benchmarks, one of long dictations and one of short dictations full of disfluencies (4,170 rows in all).
The models do best on short dictations:
- **Up to 25 words**, all three make fewer factual errors than glm-5.3-flash at low reasoning. The 2.6B and 4B also make fewer than glm-5.3 at max reasoning, and the 1.2B ties it.
- **The 4B** makes fewer errors than glm-5.3-flash at any length, and keeps up with glm-5.3 at max reasoning up to 100 words.
- **The 1.2B and 2.6B** make 1.0–1.5 times glm-5.3-flash's errors from 26 to 200 words, and 1.5–1.9 times above 200 words.
[Chart: factual errors by transcript length, for the three fine-tuned models and the two cloud glm-5.3 models](https://huggingface.co/category271/whispernow-qwen3.5-4b/resolve/main/charts/errors-by-length.png)
## Cloud models
I also scored seven more cloud models, including the off-the-shelf LFM2.5 2.6B, on a smaller set of 1,670 rows. The off-the-shelf 2.6B makes nearly four times the factual errors of the fine-tuned one.
If you'd rather use a cloud model, Claude Haiku 5.5 and gpt-6-luna (both at low reasoning) have the best price-to-performance ratio. glm-5.3-flash is about as accurate as Haiku and glm-5.3 at max reasoning is more accurate, but they're about two and four times as slow.
[Chart: factual errors for every model I tested](https://huggingface.co/category271/whispernow-qwen3.5-4b/resolve/main/charts/all-models.png)
## Links
- App (MIT, Windows, macOS and Linux): https://github.com/stephan271c/WhisperNow
- Models and dataset: https://huggingface.co/collections/category271/whispernow-6ae00accee969c7447cec326
- How the dataset was made: https://github.com/stephan271c/WhisperNow/blob/master/training/making-the-dataset.md
Happy to answer questions about the data or the training.document fetched 2026-10-29 00:44 · canonical https://reddit.com/r/LocalLLaMA/comments/1xhr7xz/i_finetuned_an_llm_to_clean_up_transcriptions_for