Thoughts about a single framework that dynamically adapts to run any MoE larger than RAM, across everything from phones and laptops to high-end, expensive GPUs
retained text
I am thinking and working on creating a framework that can run any type of MoE (Mixture of Experts) model supported by \`llama.cpp\` on any hardware or operating system. It would dynamically adapt to the specific characteristics of the hardware, aiming for optimal performance—measured in tokens per second (tok/s)—during both the prefill and decode phases. The concept is to build a home server setup using various consumer-grade devices to run local AI—though not in a mesh configuration (at least for now, due to latency concerns). What are your thoughts on this? What kind of use cases could this enable?
document fetched 2026-10-29 00:44 · canonical https://reddit.com/r/LocalLLaMA/comments/1xhsr2w/thoughts_about_a_single_framework_that