Now you can grow Bonsai on your potato
retained text
No more excuses for GPU-poor folks not to start LLMing! # Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) >**\~9.3x** smaller than FP16 (ideal) | **98.2%** of FP16 intelligence retained | **\~47 tok/s** on an Apple M5 Max laptop # Highlights * **\~5.9 GB** language model (down from \~54 GB FP16) — full 27B-class reasoning on a standard laptop or a single GPU * **98.2% of FP16 intelligence retained**: 84.78 average across 14 thinking-mode benchmarks — far above the conventional IQ2\_XXS build (72.59) at about 82% of its footprint, and within 0.4 points of UD-Q4\_K\_XL at three times the footprint * **Retains thinking, reasoning, and agentic behavior** deep in the sub-4-bit regime, where conventional low-bit representations collapse: math within half a point of full precision (96.57), coding level with the baseline (89.42), agentic tool calling at 74.92 * **End-to-end ternary language weights** across embeddings, attention projections, MLP projections, and LM head, at a *true* 1.72 bits per weight — no high-precision escape hatches behind a low-bit label; the vision tower ships as a separate Q8\_0 mmproj pack * **262K-token context** on-device, kept practical by the Qwen3.8-27B hybrid-attention backbone (\~75% linear attention) * **Two GGUF packings** with custom ternary hybrid-attention kernels for llama.cpp (CUDA, Metal) — **PTQ1\_0** packs trits densely (1.75 bits/weight, 5.95 GB), **PQ2\_0** stores each trit in a 2-bit slot (2.13 bits/weight, 7.21 GB); packed weights are consumed directly, never expanded back to FP16
document fetched 2026-10-12 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1x37kxn/now_you_can_grow_bonsai_on_your_potato