LFM2.5-2.6B model+KV cache quantization report
retained text
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1vi0d4i/lfm2526b_modelkv_cache_quantization_report/"> <img alt="LFM2.5-2.6B model+KV cache quantization report" src="https://preview.redd.it/3lr47ialdyhh1.png?width=140&height=92&auto=webp&s=e5244ac0f6576c041fa23dd663bb60e6609ec4e9" title="LFM2.5-2.6B model+KV cache quantization report" /> </a> </td><td> <!-- SC_OFF --><div class="md"><p><a href="https://www.liquid.ai/blog/lfm2-5-2-6b">LFM2.5-2.6B</a> is a new tiny model by LiquidAI, with benchmarks that put it head to head with much larger models.</p> <p>I've run llama-perplexity on many model GGUF quants, crossed with many KV cache quants, to understand the model's best overall quantization for any given amount of memory.</p> <p>I also show how different quantization metrics show (or hide) model degradation.</p> <p><a href="https://github.com/crusaderky/pixi-llm-recipes/blob/main/perplexity/LFM2.5-2.6B/README.md">Full report and commentary</a></p> <p><a href="https://htmlpreview.github.io/?https://raw.githubusercontent.com/crusaderky/pixi-llm-recipes/main/perplexity/LFM2.5-2.6B/report.html">Interactive HTML plots</a></p> <h1>If you don't have time to read</h1> <p><a href="https://github.com/crusaderky/pixi-llm-recipes/blob/main/perplexity/LFM2.5-2.6B/README.md#if-you-dont-have-time-to-read"></a></p> <ul> <li>The model fits on an 8GB Raspberry Pi with no material degradation and on a 4GB Raspberry Pi with contained degradation.</li> <li>DO NOT use Q4_K_M.</li> <li><em>On this model,</em> model quant quality degrades faster than KV cache quant.</li> <li>Abliteration comes with a flat cost of ~0.075 KLD.</li> <li>Logarithmic KLD and Top-1% plots lie to you by telling you that quality degradation is smooth, while it's actually a cliff.</li> </ul> </div><!-- SC_ON -->   submitted by   <a href="https://www.reddit.com/user/crusaderky"> /u/crusaderky </a> <br /> <span><a href="https://www.reddit.com/gallery/1vi0d4i">[link]</a></span>   <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1vi0d4i/lfm2526b_modelkv_cache_quantization_report/">[comments]</a></span> </td></tr></table>
document fetched 2026-08-08 06:58 · canonical https://reddit.com/r/LocalLLaMA/comments/1vi0d4i/lfm2526b_modelkv_cache_quantization_report