SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.
retained text
I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs. **Numbers** (Gemma 4 E4B-it Q3\_K\_M on a laptop RTX 3050 8 GB, both runtimes using the same GGUF on the same machine, llama.cpp at commit `43fe9c64281`): ||SonicBoom|llama.cpp| |:-|:-|:-| |Decode (tg64)|**51.65 tok/s**|45.20 tok/s| |Decode @ depth 2048|47.13 tok/s|45.21 tok/s| |Prefill 4100 tokens|1051 tok/s|\~1944 tok/s| |TTFT, 4000-token prefix + 100 new|**503 ms** (prefix reuse)|| **Why it matters:** the quantized matmul kernels (MMVQ/MMQ) are ported from ggml, so the kernels are the same ones llama.cpp uses. The 14% decode lead comes entirely from the layer above them. A graph planner rewrites the model before it runs: activations are quantized once and reused, norm and activation are fused with quantization, dead computation is removed, and each decode step is captured and replayed as a single CUDA Graph. Code derived from ggml is about 8% of the core. The rest is the runtime. Prefix reuse is exact: a continuation is token-for-token identical to a full prefill, which makes multi-turn chat and coding-assistant workloads cheap. **What it can’t do yet:** * Prefill is about half of llama.cpp. It is a real batched path (int8 tensor-core plus cuBLAS), but there’s more to close. * One verified model: Gemma 4 E4B. The quant formats are Q3\_K, Q4\_K and Q5\_K only, so most Q4\_K\_M files from Hugging Face won’t load yet because they mix in Q6\_K and Q8\_0. Those formats are next, along with Qwen. * It has only been tested on sm\_86 (RTX 30-series), and sampling is greedy only. It’s not a drop-in llama.cpp replacement yet. The point of v0.0.1 is to show that the architecture works: once a graph compiler sits on top of the same kernels, the remaining gains come from the compiler instead of more hand-written kernels. Next on the roadmap are MLIR GPU codegen, speculative decoding, and more formats. GPL-3.0. Benchmark conditions, reproduce commands and the model sha256 are in the README: [https://github.com/hardenedlinux/SonicBoom](https://github.com/hardenedlinux/SonicBoom)
document fetched 2026-10-14 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x4scub/sonicboom_52_toks_decode_on_a_laptop_rtx_3050