An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk
retained text
Follow-up to last week's post about streaming MoE experts from disk in a browser tab. That ran on hand-written engines for two models. This week the same disk tier sits under llama.cpp's own WebGPU backend, so it works for the MoE models llama.cpp already runs, with no per-model engine. The video is one real recording in Chrome. The sped-up parts say so on screen, with a real-time clock. Measured in that recording (MacBook Pro M4 Pro, 24 GB): |Model (GGUF)|File|On the GPU|Load from disk|Decode| |:-|:-|:-|:-|:-| |Qwen3-Next 80B-A3B Q4\_K\_M|48.4 GB|4.89 GB|48 s|2.9 tok/s (proof of concept)| |OLMoE 1B-7B Q4\_K\_M|4.2 GB|3.92 GB|5.3 s|22.4 tok/s| |LFM2 8B-A1B Q4\_K\_M|5.0 GB|3.89 GB|6.2 s|17.1 tok/s| Qwen3-30B-A3B runs too (14.7 GB file). **How it works.** The GGUF downloads once into the browser's private storage (OPFS). The GPU holds the dense layers, the KV cache and a fixed number of expert slots per layer, sized from a GPU memory budget you pick. At each MoE layer the router's picks are read back, and any expert not in a slot is read from the OPFS file (sync access handles in workers) into the least recently used slot. The paging is a small change to ggml-webgpu: a new buffer type, slot-mapped MUL\_MAT\_ID shaders, and the residency logic. **Correctness.** With paging on and off on the same backend, replies are identical (9/9 conversations on OLMoE and LFM2; on the two big ones, which do not fit unpaged, 8 vs 32 or 16 vs 64 slots give identical replies). Against llama.cpp on Metal, 4 to 8 of 9 replies match. The misses are WebGPU-vs-Metal rounding at near-ties: against llama.cpp's CPU backend, the WebGPU path is as close as Metal is. **Limits.** The 80B is slow: it reads about 359 MB from disk per token. Prompt processing is still slow (last week's fair complaint). Chrome or Edge 137+ only; tested on Apple silicon only. The first load copies the whole file into browser storage. The 80B model's own answer, asked how it could run in a browser tab: "It can't." It was running in a browser tab. Try it: https://diskformer.naklitechie.com · Code: https://github.com/NakliTechie/diskformer.js ·
document fetched 2026-10-14 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x4xjuh/an_80b_moe_qwen3next_48_gb_gguf_answering_in_a