Kimi-K3.1-Pro-Coder (2.8T) | 1.77 TB -> 775 GB (-56%) | 36.8 TPS (Batch) / 10.8 TPS on 1x Blackwell | 96.7% on both AIME 2025 and LiveCodeBench
retained text
Up front: **I know a 775 GB quant is not for everyone**, and this is definitely a more experimental setup and quant than I usually do, but I think it was a great stress-test of what's possible and I believe there is more improvement possible in the future. That said, the base **Kimi-K3.1-Pro** is a **2.8T parameter** monster (\~1.77 TB even in native MXFP4/BF16), which normally means you're looking at roughly **22x 80GB GPUs** just to hold it in VRAM. My goal with this experiment was to see if I could get the full 93-layer model down to something that runs on a **single GPU + system RAM + fast NVMe** without gutting its coding and math reasoning. # What I built to make this work (Strata + dr-triton) Because all 384 routed experts per layer don't get hit equally, I quantized the routed MoE experts across 3 tiers (**2.81 bpw / 2.56 bpw / 1.44 bpw**, averaging \~1.92 bpw across MoE down from about 4.0bpw on base) while keeping all shared experts, attention, and embedding in native BF16/F32. To actually run it without hitting a PCIe/NVMe wall, I built two things alongside the weights: * [**Strata Fork**](https://github.com/grawnchtoader/strata) **:** A 3-tier MoE loader and offloading engine that splits experts across **Tier 1 (Hot GPU VRAM)**, **Tier 2 (Warm Pinned RAM)**, and **Tier 3 (Cold NVMe** `mmap`\*\*)\*\*, using router lookahead prefetching (`97.9%` effective hit rate in testing) so cold experts are already streaming in before the layer executes. * [**dr-triton**](https://github.com/grawnchtoader/dr-triton)**:** Fused Triton dequantization kernels so the compressed expert weights stay packed over PCIe and only expand in GPU SRAM later. **Can you use your own offloading / custom Strata setup?** * **Customizing the tier split:** Yes, you don't have to use my exact VRAM/RAM/NVMe split. In `Strata`, you can re-Configure how many tiers sit in VRAM vs. System RAM vs. NVMe depending on your rig (e.g., if you have 512 GB of DDR5, you can push almost all of Tier 3 into RAM; if you're on a 24–32 GB card, you shrink Tier 1 and lean heavier on RAM/NVMe). * **Using your own engine / loader:** You **cannot** drop these shards straight into vanilla `llama.cpp` or `vLLM` out of the box because the routed experts use a custom packed tensor layout. However, if you have your own offloading pipeline or custom PyTorch/Triton runner, you don't have to use the `Strata` scheduler—you can just import the standalone `dr-triton` / `strata_coder` dequant module and handle placement yourself. # Full 93-Layer Benchmarks (Base 1.77 TB vs. Coder 775 GB) Tested across all 93 layers (`Layer 0..92` \+ `lm_head`) on a single **NVIDIA RTX PRO 6000 Blackwell (96/102 GB VRAM) + 176 GB RAM** in Colab: |Benchmark / Metric|Base `Kimi-K3.1-Pro` (1.77 TB)|`Kimi-K3.1-Pro-Coder` (775 GB)|Difference| |:-|:-|:-|:-| |**AIME 2025 (All 30 Problems, Pass@1)**|29 / 30 (96.67%)|**29 / 30 (96.67%)**|Exact match on all 30 problems| |**LiveCodeBench Rolling Window (30 Probs)**|29 / 30 (96.67%)|**29 / 30 (96.67%)**|Easy 9/10, Med 10/10, Hard 10/10| |**Top-1 / Top-5 Token Agreement**|100% / 100%|**95.20% / 99.92%**|Mean KLD: `0.0191`| |**AIME 2025 Math Perplexity (PPL)**|2.2135|**2.2358**|`+0.0223` (`+1.01%`)| |**LiveCodeBench Hard Perplexity (PPL)**|2.1136|**2.1562**|`+0.0426` (`+2.02%`)| |**LiveCodeBench Overall Perplexity (PPL)**|2.6911|**2.7629**|`+0.0718` (`+2.67%`)| # Throughput (TPS) — Tested Colab Setup vs. Common Hardware Specs Since throughput on a 3-tier offloaded MoE depends heavily on how much fits in VRAM/RAM vs. how often you touch NVMe, here is what we measured on the Colab Blackwell instance alongside realistic numbers a bit below and above that spec: |Hardware Spec|Batch / Prefill Eval TPS|Single-Stream Decode TPS|Notes| |:-|:-|:-|:-| |**1x RTX 4090 / 5090 (24–32 GB)** \+ 128 GB DDR5 + Gen4 NVMe|\~11 – 14 tok/s|\~4 – 5.5 tok/s|Heavy RAM + NVMe lookahead streaming| |**2x RTX 3090 / 4090 (48 GB)** \+ 192 GB DDR5 + Gen5 NVMe|\~19 – 23 tok/s|\~7.5 – 9 tok/s|Full attention/shared + partial Tier 1 in VRAM| |**1x A100 / H100 (80 GB)** \+ 160 GB RAM + Gen4 NVMe|\~27 – 31 tok/s|\~9 – 10 tok/s|Tier 1 Hot experts resident in VRAM| |**TESTED: 1x RTX PRO 6000 Blackwell (96 GB)** \+ 176 GB RAM|**36.8 tok/s**|**10.8 tok/s**|Measured across full 93-layer AIME & LCB runs| |**2x PRO 6000 / 2x H100 (192 GB)** \+ 512 GB DDR5|\~55 – 62 tok/s|\~18 – 22 tok/s|Tiers 1+2 + most of Tier 3 stay in VRAM/RAM| While I admit even the the "lightest" hardware specs above are still above what the average setup might be, If you have hardware in the ranges above (and 775 GB download space) it would be great to see how this works out for you! The model card has all of the setup instructions and repos required out of the box, and you can also use the above instructions if you want to try with your own configurations. And the best thing about the experiment is this same process can also be used on models more attainable for the average user (and, if Kimi had not been trained in 4-bit, a lot more space could be shaved off). # Links * **Weights (Hugging Face, 96 shards / 775 GB):** [https://huggingface.co/Grawnch/Kimi-K3.1-Pro-Coder](https://huggingface.co/Grawnch/Kimi-K3.1-Pro-Coder) * **Strata (3-Tier Offloading & Lookahead):** [https://github.com/grawnchtoader/strata](https://github.com/grawnchtoader/strata) * **dr-triton (Fused Dequant Kernels):** [https://github.com/grawnchtoader/dr-triton](https://github.com/grawnchtoader/dr-triton)
document fetched 2026-11-09 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1xr777l/kimik31procoder_28t_177_tb_775_gb_56_368_tps