Are smaller quants/models worth running as sub-agents or is it a waste?
retained text
Haven't had a very good experience with small quants, they tended to loop, fail tool-calls, or get stuck. Q8 QFN has been pretty astounding though when I set it up. I've seen people running a "main" model with sub-models. I've got a second machine that can run like 2-3bit quants of Q3.8-27B / QFN, I was considering trying to set it up with my main machine so the smaller model can be used as a sub-agent for coding but I'm wondering if it's even worth it when smaller quants just seem volatile and prone to error. I *have* tried setups that used subagents before, but it was through API which used full-sized models on real enterprise hardware... I haven't fully explored this sphere though, I'm sure there's usecases for specialist small models in coding that I am unaware of
document fetched 2026-11-19 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1xzb7s8/are_smaller_quantsmodels_worth_running_as