Real-time talking avatar generated locally with SoulX-FlashHead + LFM3-Audio
retained text
I've been experimenting with real-time talking-head generation using SoulX-FlashHead, combined with local speech models to build an interactive avatar that can listen and respond. The whole pipeline runs locally, and the demo is available in the audio.cpp v0.9.5 WebUI. **The workflow combines three models:** * **SoulX-FlashHead:** Animates a portrait in real time from generated speech. * **OWSM v4:** Transcribes voice input. * **LFM3-Audio:** Handles conversational responses and speech synthesis. One interesting detail: LFM3-Audio is called twice. Its speech-to-speech path only offers one voice, so we use its TTS capability separately to synthesize responses in a different voice. **Performance and limitations** On an RTX 5090, the latency from submitting a question to the avatar starting to speak is around 850 ms. ASR runs in parallel. The video generation stage is still GPU-intensive. I'd recommend an RTX 4090 or better for SoulX-FlashHead. cuDNN is essential for real-time performance in this setup. Without it, inference is about 2× slower and VRAM usage doubles. This is still a prototype. The extra LFM3-Audio call and WebUI overhead leave room for further latency improvements, particularly in a more optimized native application.
document fetched 2026-11-16 00:29 · canonical https://reddit.com/r/StableDiffusion/comments/1xx376j/realtime_talking_avatar_generated_locally_with