Building a zero-dependency C inference engine for BitNet (1.58-bit) - lessons from hitting 36 tok/s on a Xeon CPU
retained text
<!-- SC_OFF --><div class="md"><p>Over the past few months I have been building a CPU-first inference engine from scratch in pure C99 (no Python, no CUDA, no BLAS, just GCC and make). The focus has been running 1.58-bit ternary models natively without heavy runtime overhead.</p> <p>Currently it hits 36.25 tok/s on BitNet b1.58-2B-4T on an Intel Xeon using 4 threads.</p> <p>A few technical details from the build:</p> <ol> <li>Native ternary SIMD: BitNet weights are packed 4 per byte (values -1, 0, +1). Instead of unpacking to float32 before math, we use custom AVX2 and AVX-512 routines to accumulate directly into integer registers using VNNI instructions (vpdpbusds).</li> <li>Minimal runtime overhead: The thread pool uses C11 atomics with spin-then-yield backoff rather than heavy mutex contention, so thread sync overhead is basically zero during token generation.</li> <li>Zero dependencies: Compiles into a single standalone binary that serves an OpenAI-compatible API endpoint directly.</li> </ol> <p>The biggest takeaway so far has been the DRAM ceiling. We spent weeks optimizing low-level matrix multiplication kernels, but decode speed at batch size 1 is kind of stuck at memory bandwidth. On our Xeon test box, we are running at roughly 95% of theoretical memory bandwidth, so faster compute kernels do not actually move end-to-end token latency until you batch across multiple sequences.</p> <p>Repo: <a href="https://github.com/shifulegend/project-zero">https://github.com/shifulegend/project-zero</a></p> <p>Curious what token rates others are seeing on different CPU architectures (especially AMD Zen or ARM NEON), or how folks are handling the memory bandwidth ceiling for local ternary inference.</p> </div><!-- SC_ON -->   submitted by   <a href="https://www.reddit.com/user/shifu_legend"> /u/shifu_legend </a> <br /> <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1vj1cin/building_a_zerodependency_c_inference_engine_for/">[link]</a></span>   <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1vj1cin/building_a_zerodependency_c_inference_engine_for/">[comments]</a></span>
document fetched 2026-08-09 07:00 · canonical https://reddit.com/r/LocalLLaMA/comments/1vj1cin/building_a_zerodependency_c_inference_engine_for