[Boole-32B-A2.5B-preview] 9GB 1.8-bit MoE, 565 token/s on 1 RTX 5090
retained text
https://preview.redd.it/prwlaz7tce1i1.png?width=768&format=png&auto=webp&s=f33077813ca59ef6de82b8592d1d6047be7fa24e https://preview.redd.it/drmh7g6uce1i1.png?width=768&format=png&auto=webp&s=580e0ae3491be3350b52e4d25da3401a03b4c0f2 https://preview.redd.it/36j4ttmuce1i1.png?width=768&format=png&auto=webp&s=1bffcf3d7f2a3e8c76f4ed3616521e621d790a3d Hi r/LocalLLaMA, we're releasing Boole-32B-A2.5B-Preview, an early 1.8-bit version of Recursive's [flint](https://huggingface.co/RecursiveSI/flint). With mixed-precision quantization, expert pruning, and custom kernels (kernel changes in attached figure), we made the model 9x smaller than BF16, and 2x faster than INT4. After releasing our 1-bit LLM (Boole-1.7B), we heard a lot of requests for both faster and larger MoE models - so this is our first step in that direction. Try it out, and let us know what you'd like to see next! * Weights: [https://huggingface.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF](https://huggingface.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF) * Getting started: [https://github.com/PrecisionNumerics/boole-demo](https://github.com/PrecisionNumerics/boole-demo) * Our llama.cpp fork: [https://github.com/PrecisionNumerics/llama.cpp](https://github.com/PrecisionNumerics/llama.cpp)
document fetched 2026-11-14 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xv3h93/boole32ba25bpreview_9gb_18bit_moe_565_tokens_on_1