← sources

Real-time talking avatar generated locally with SoulX-FlashHead + LFM3-Audio

source: reddit (reddit:StableDiffusion)published 2026-11-15 18:42ingested 2026-11-16 02:37

retained text

I've been experimenting with real-time talking-head generation using SoulX-FlashHead, combined with local speech models to build an interactive avatar that can listen and respond.

The whole pipeline runs locally, and the demo is available in the audio.cpp v0.9.5 WebUI.

**The workflow combines three models:**

* **SoulX-FlashHead:** Animates a portrait in real time from generated speech.
* **OWSM v4:** Transcribes voice input.
* **LFM3-Audio:** Handles conversational responses and speech synthesis.

One interesting detail: LFM3-Audio is called twice. Its speech-to-speech path only offers one voice, so we use its TTS capability separately to synthesize responses in a different voice.

**Performance and limitations**

On an RTX 5090, the latency from submitting a question to the avatar starting to speak is around 850 ms. ASR runs in parallel.

The video generation stage is still GPU-intensive. I'd recommend an RTX 4090 or better for SoulX-FlashHead.

cuDNN is essential for real-time performance in this setup. Without it, inference is about 2× slower and VRAM usage doubles.

This is still a prototype. The extra LFM3-Audio call and WebUI overhead leave room for further latency improvements, particularly in a more optimized native application.

document fetched 2026-11-16 00:29 · canonical https://reddit.com/r/StableDiffusion/comments/1xx376j/realtime_talking_avatar_generated_locally_with