← sources

Calling all GPUs: help us benchmark SonicBoom v0.0.3 — a compiler-first AI runtime that beats llama.cpp on decode

source: reddit (reddit:LocalLLaMA)published 2026-10-19 03:40ingested 2026-10-20 02:40

retained text

Hi folks!

We are HardenedLinux, an old-school free software community. SonicBoom is our open-source (GPLv3)
inference runtime: a compiler-first alternative to the usual Python stack. No Python at build
or run time — one shared library, a C ABI, and Guile Scheme as the first-class language.

What v0.0.3 does on a desktop RTX 3050 8 GB (same session, same GGUF, vs llama.cpp -ngl 99):

| model | decode (tg64) | prefill (pp512) |
|---|---|---|
| Gemma 4 E4B Q3_K_M | 1.18× | 1.00× |
| Qwen2.5-0.5B Q4_K_M | 1.12× | 1.09× |

On real multi-turn agent workloads, time-to-first-token dropped several-fold since v0.0.2
(prefix reuse + plan caching). Output tokens are bit-identical to our golden runs, and prefix
reuse is bitwise identical to a full prefill.

How: all graph optimization and kernel generation happen on Boom IR, an MLIR dialect; hot
quantized matmuls run as attributed ggml-derived kernels. Every optimization has a switch and
an ablation number, so you can see what each one buys.

Where we are honest about gaps: Qwen decode at depth 512 is 0.98×, and we have only one GPU.
That is exactly why we need you.

The invitation: every release gets one benchmark issue. If you have an NVIDIA GPU (any
generation — Ada, Hopper, Blackwell especially welcome), one command produces a report you can
paste. Everyone whose results we publish is credited in the README.

→ v0.0.3 benchmark issue: [https://github.com/hardenedlinux/SonicBoom/issues/2](https://github.com/hardenedlinux/SonicBoom/issues/2)

Please post results and problems in the issue rather than here, so everything stays in one place.

document fetched 2026-10-20 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x9mitw/calling_all_gpus_help_us_benchmark_sonicboom_v003