← sources

Thoughts about a single framework that dynamically adapts to run any MoE larger than RAM, across everything from phones and laptops to high-end, expensive GPUs

source: reddit (reddit:LocalLLaMA)published 2026-10-28 16:23ingested 2026-10-29 02:42

retained text

I am thinking and working on creating a framework that can run any type of MoE (Mixture of Experts) model supported by \`llama.cpp\` on any hardware or operating system.

It would dynamically adapt to the specific characteristics of the hardware, aiming for optimal performance—measured in tokens per second (tok/s)—during both the prefill and decode phases. 

The concept is to build a home server setup using various consumer-grade devices to run local AI—though not in a mesh configuration (at least for now, due to latency concerns).

What are your thoughts on this? What kind of use cases could this enable?

document fetched 2026-10-29 00:44 · canonical https://reddit.com/r/LocalLLaMA/comments/1xhsr2w/thoughts_about_a_single_framework_that