← sources

[Release] ISTA GSQ-RCO GGUFs for DeepSeek-V4.1-Flash-1017

source: reddit (reddit:LocalLLaMA)published 2026-10-24 08:05ingested 2026-10-24 11:01

retained text

We have released DeepSeek-V4.1-Flash-1017 quantized with GSQ and RCO, at 1.75 to 2.25 bits per weight.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model: 384 routed experts per layer, of which 6 are active per token, 763.2B parameters and a 1M-token context window. It is released with FP8 dense weights and FP4 routed experts.

**What's inside**

* Three quantized GGUFs: IQ1\_M (1.75 bpw), IQ2\_XXS (2.00 bpw) and IQ2\_XS (2.25 bpw), with backbones of 120.7 to 155.2 GB. The bit-width is the average over the 551.9B backbone parameters, which is what the search controls.
* One shared Engram file (110.6 GB): the model's two Engram lookup tables (196.6B parameters) at IQ4\_NL. It is identical for every build and needs to be downloaded only once. The tables are read one row per token, so they can stay memory-mapped on disk.
* Every build runs on a single 12 GB GPU with Strata (see model card, [https://github.com/Niko1221/Strata](https://github.com/Niko1221/Strata)).
* GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and group scales, closing most of the gap between scalar and vector quantization at low bit-widths while remaining deployable in standard GGUF types.
* RCO (Riemannian Constrained Optimization): enforces exact budgets by gradient descent on the task loss, without per-constraint tuning. Here it assigns a quantization type to every backbone tensor under the target bit-width, with separate ladders for the dense path (Q5\_K to Q8\_0) and the routed experts (IQ1\_S to IQ3\_S).
* An embedded chat template that reproduces DeepSeek's prompt encoder exactly, including the reasoning effort (1 to 100, or low / high / max).

**Results**

At 2.00 bpw the model matches the base model on every benchmark evaluated. All numbers were measured at reasoning effort 50.

* IQ1\_M (1.75 bpw, 120.7 GB backbone): AIME25 100.00, GPQA-Diamond 88.38 against 90.91, LiveCodeBench v6 88.57 against 89.71. Task average 92.32, retaining 98.7%.
* IQ2\_XXS (2.00 bpw, 138.2 GB backbone): AIME25 100.00, GPQA-Diamond 90.91, LiveCodeBench v6 89.71, identical to the base model. Task average 93.54, 100.0% of the base.
* IQ2\_XS (2.25 bpw, 155.2 GB backbone): results to follow.

**Running it**

llama.cpp does not support DeepSeek-V4.1 yet, so the GGUFs do not run there for now. They run in Strata, which serves every build on a single NVIDIA GPU with 12 GB of VRAM, with a one-click installer and an OpenAI-compatible endpoint on localhost.

**Links**

* Quantized models: [https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF)
* GSQ: paper [https://arxiv.org/abs/2604.18556](https://arxiv.org/abs/2604.18556) | code [https://github.com/IST-DASLab/GSQ](https://github.com/IST-DASLab/GSQ)
* RCO: paper [https://arxiv.org/abs/2605.00649](https://arxiv.org/abs/2605.00649) | code [https://github.com/IST-DASLab/RCO](https://github.com/IST-DASLab/RCO)

The repository ships the complete per-tensor RCO allocation for every build (1008 tensors each).

Feedback is welcome. Requests for models to quantize are also welcome.

From the ISTA Deep Algorithms and Systems Lab.

document fetched 2026-10-25 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1xe2wg8/release_ista_gsqrco_ggufs_for_deepseekv41flash1017