← sources

SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.

source: reddit (reddit:LocalLLaMA)published 2026-10-13 09:46ingested 2026-10-14 02:43

retained text

I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs.

**Numbers** (Gemma 4 E4B-it Q3\_K\_M on a laptop RTX 3050 8 GB, both runtimes using the same GGUF on the same machine, llama.cpp at commit `43fe9c64281`):

||SonicBoom|llama.cpp|
|:-|:-|:-|
|Decode (tg64)|**51.65 tok/s**|45.20 tok/s|
|Decode @ depth 2048|47.13 tok/s|45.21 tok/s|
|Prefill 4100 tokens|1051 tok/s|\~1944 tok/s|
|TTFT, 4000-token prefix + 100 new|**503 ms** (prefix reuse)||

**Why it matters:** the quantized matmul kernels (MMVQ/MMQ) are ported from ggml, so the kernels are the same ones llama.cpp uses. The 14% decode lead comes entirely from the layer above them. A graph planner rewrites the model before it runs: activations are quantized once and reused, norm and activation are fused with quantization, dead computation is removed, and each decode step is captured and replayed as a single CUDA Graph. Code derived from ggml is about 8% of the core. The rest is the runtime.

Prefix reuse is exact: a continuation is token-for-token identical to a full prefill, which makes multi-turn chat and coding-assistant workloads cheap.

**What it can’t do yet:**

* Prefill is about half of llama.cpp. It is a real batched path (int8 tensor-core plus cuBLAS), but there’s more to close.
* One verified model: Gemma 4 E4B. The quant formats are Q3\_K, Q4\_K and Q5\_K only, so most Q4\_K\_M files from Hugging Face won’t load yet because they mix in Q6\_K and Q8\_0. Those formats are next, along with Qwen.
* It has only been tested on sm\_86 (RTX 30-series), and sampling is greedy only.

It’s not a drop-in llama.cpp replacement yet. The point of v0.0.1 is to show that the architecture works: once a graph compiler sits on top of the same kernels, the remaining gains come from the compiler instead of more hand-written kernels. Next on the roadmap are MLIR GPU codegen, speculative decoding, and more formats.

GPL-3.0. Benchmark conditions, reproduce commands and the model sha256 are in the README:  
[https://github.com/hardenedlinux/SonicBoom](https://github.com/hardenedlinux/SonicBoom)

document fetched 2026-10-14 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x4scub/sonicboom_52_toks_decode_on_a_laptop_rtx_3050