Building a Fully Local AI Assistant That Doesn't Need a Wake Word
retained text
Hi, Thanks for checking this out On Friday, one of my roommates was studying and kept asking questions out loud whenever he got stuck. I suggested using ChatGPT's Live Speech mode. It worked well, but we quickly ran into the usage limit. That got me thinking. This weekend, I decided to build something similar but **completely local**. The models run on my laptop, the data stays on my laptop, and it can be used anytime without worrying about limits or sending conversations to the cloud and i want to keep it goal/productivity oriented . The first image that came to my mind was **JARVIS** and **Miss Minutes**... okay, I know it bit over😄. KPI's are simple: * listen to me and response when i need any help, advice, suggestion before me saying hay ai * monitor my usage, and help keep me aligned with my goals and productivity. I've started building that idea not from scratch, but by looking at existing tools from a different perspective and trying to combine them into an always-available AI assistant. **Current pipeline:** `Silero VAD → Whisper ASR → Fine-tuned BERT → Qwen 3B → Kokoro TTS` The interesting part is the **fine-tuned BERT classifier**. Since the assistant is always listening, I didn't want to invoke a 3B LLM on every speech chunk just to ask "Should I respond?". Instead, a \~150 MB BERT model stays in memory and decides whether the input actually needs an LLM response. Only then is Qwen 3B loaded/invoked. This week's progress: * End-to-end integration testing. * Streaming LLM responses. * Local TTS output. * Everything runs **100% locally** on my laptop (models + data stay on-device). I honestly questioned whether fine-tuning BERT was worth the effort, but it turned into one of the best learning experiences in the project. **Next week:** Implementing my design(producer consumer model - attached img), building a multi-agent architecture, productivity/goal alignment. I'm also exploring ways to **avoid loading the full LLM into RAM**, and instead load only the parts needed for a query like llama.cpp.... I'd love to hear your feedback or suggestions
document fetched 2026-09-07 00:23 · canonical https://reddit.com/r/developersIndia/comments/1w93zrm/building_a_fully_local_ai_assistant_that_doesnt