I built BitRun: A standalone runner for 1-bit LLMs (100–127 t/s on Apple Silicon & 40–45 t/s on PC CPUs with zero compile)
retained text
I built BitRun because I wanted to run 1-bit ternary LLMs on ordinary laptops without melting my battery, spinning up a dedicated GPU, or wrestling with complex build environments. Right now, BitRun gives you 98 to 127 tokens/sec on an Apple M4 Pro CPU and 40 to 45 tokens/sec on an Intel Core i5 PC CPU, pulling minimal power with 0% GPU usage. To make this actually work on modern hardware, I had to fix several low-level C++ kernel math bugs in Microsoft's research codebase (bitnet.cpp): **Memory stride clobbering:** In ggml-bitnet-mad.cpp, batch stride was ignored in the write-back stage, causing evaluated batch rows to silently overwrite each other in memory. **The infinite @ @ @ loop on ARM NEON:** Block size was hardcoded to 64 instead of the GGUF standard 128 elements in ggml-cpu-i2s.c. That 50% pointer arithmetic shift fed scrambled bytes into the NEON DotProd adders. **Activation mismatch:** The model graph initialized the FFN gate with SiLU, but the 2.4B model was trained with squared ReLU. Using SiLU distorted the activation range and produced gibberish. **Multi-turn BLAS crashes:** BLAS was claiming matrix multiplications past 32 context tokens and jumping into unmapped memory with unaligned byte offsets. **Windows MSVC lockout:** Removed artificial CMake checks that blocked native Visual Studio compilation. I submitted the fixes upstream to Microsoft: * **PR #642:** [https://github.com/microsoft/BitNet/pull/642](https://github.com/microsoft/BitNet/pull/642) * **PR #644:** [https://github.com/microsoft/BitNet/pull/644](https://github.com/microsoft/BitNet/pull/644) Then I packaged the patched engine into BitRun so anyone can test ternary models in 60 seconds with an Ollama-style workflow. No CMake, no Clang toolchains, no Python virtualenvs, and no heavy Docker images. **HOW TO RUN IT (ZERO COMPILE)** `Mac / Linux: curl -fsSL` [`https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh`](https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh) `| sh` `Windows (PowerShell): irm` [`https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1`](https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1) `| iex` **Pull and chat on pure CPU:** `bitrun pull falcon3-1b` `bitrun run falcon3-1b` You can also run "bitrun serve falcon3-1b" to get an OpenAI-compatible REST server on port 11435 (point Cursor, [Continue.dev](http://Continue.dev), or Open WebUI to it). Custom personas can be created using standard Modelfiles (bitrun create engineer -f Modelfile). **VERIFIED CPU SPEEDS (PURE CPU SIMD, 0% GPU)** Apple M4 Pro CPU (12 Cores): * Falcon3 1B Instruct (\~1.30 GB): 98 to 127 tokens/sec (\~1.9 GB RAM) * BitNet b1.58 2.4B (\~1.13 GB): \~40 tokens/sec (\~1.8 GB RAM) * BitNet 70M (\~79 MB): 1,350+ tokens/sec (< 150 MB RAM) Windows x86\_64 (Intel Core i5-12600K, AVX2): * Falcon3 1B Instruct: 39 to 45 tokens/sec (209.6 t/s prompt processing) * BitNet 70M: 492 tokens/sec Linux (WSL2 / 4 cores, AVX2): * Falcon3 1B Instruct: 40 to 45 tokens/sec **REALISTIC SCOPE** Ternary weights require Quantization-Aware Training (QAT) from scratch. You cannot just take an existing 70B FP16 model and post-quantize it down to 1.58-bit without destroying it. Because of that, the public model catalog right now is limited to models under 3B parameters (falcon3-1b, bitnet-2b, falcon3-3b, bitnet-70m). They will not replace Claude or GPT-4 for heavy coding. However, for low-power edge devices, battery-conscious laptop use, and background automations where you want 100+ tokens/sec on pure CPU, it is ready to run today. **GitHub Repo:** [https://github.com/IshekKhal/bitrun](https://github.com/IshekKhal/bitrun) **Upstream PRs:** [https://github.com/microsoft/BitNet/pull/642](https://github.com/microsoft/BitNet/pull/642) and [https://github.com/microsoft/BitNet/pull/644](https://github.com/microsoft/BitNet/pull/644) **Pre-compiled Releases:** [https://github.com/IshekKhal/bitrun/releases/tag/v0.1.0-alpha](https://github.com/IshekKhal/bitrun/releases/tag/v0.1.0-alpha) Drop your CPU benchmark numbers if you try it!
document fetched 2026-10-15 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1x5pao2/i_built_bitrun_a_standalone_runner_for_1bit_llms