← sources

[Guide] Ternary Bonsai 2 is finally 100% working on ik_llama.cpp (Native CUDA offload + Fast CPU -rtr)

source: reddit (reddit:LocalLLaMA)published 2026-10-31 15:01ingested 2026-11-01 02:40

retained text

Hey folks,

Quick heads up for anyone playing around with PrismML's Ternary Bonsai 2!

Until very recently, running it outside of Prism's own fork was a bit of a pain. Upstream llama.cpp uses a block size of QK=256 (TQ2\_0), while Prism uses QK=128 (PQ2\_0), and trying to offload layers to CUDA on other forks would bottleneck or break due to Hadamard sign tensor handling.

With the latest commits landed in ik\_llama.cpp, we finally have complete, native support across both CUDA and CPU. Here is how to set it up cleanly.

1. Convert upstream weights losslessly (PR #2694)

If you already have a native PQ2\_0 GGUF downloaded from Prism, you can skip this.

If you downloaded upstream TQ2\_0 or TQ1\_0 files, ik\_llama.cpp now ships with a script that losslessly repacks them to PQ2\_0 in just a few seconds (exact trits preserved, no requantization needed):

python3 scripts/repack\_ternary\_to\_prism.py Bonsai-2-27B-TQ2\_0.gguf Bonsai-2-27B-PQ2\_0.gguf

1. Native CUDA Full Offload (PR #2691 & PR #2747)

Native CUDA kernels for PQ2\_0 are now merged. You can offload everything to your Nvidia GPU without token corruption or falling back to heavy k-quants:

./build/bin/llama-cli -m Bonsai-2-27B-PQ2\_0.gguf -ngl 99 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20

1. CPU Inference with fast prompt ingestion (-rtr 1)

If you're testing on CPU, ik\_llama.cpp uses an 8-row interleaved format (PQ2\_0\_R8) backed by iqk\_mul\_mat kernels. Remember to pass -rtr 1 (runtime repack) to drastically accelerate prompt processing:

./build/bin/llama-cli -m Bonsai-2-27B-PQ2\_0.gguf -rtr 1 -t 8 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20

TL;DR: No more sign tensor bottlenecks on GPU. PR #2747 brings native CUDA kernels, PR #2694 gives us instant lossless repack from upstream weights, and CPU prompt ingestion with -rtr 1 is flying.

Big shoutout to Iwan Kawrakow and the contributors for getting ternary models working natively!

document fetched 2026-11-01 00:38 · canonical https://reddit.com/r/LocalLLaMA/comments/1xkcg2y/guide_ternary_bonsai_2_is_finally_100_working_on