DeepSeek-V4.1-Flash-1017 on one RTX 3090: Big updates to the Delta Inference Engine v0.93. Same 41 tok/s at 192GB/IQ2 at better quality, usable from 64 GB RAM, first token ~40% sooner, new Unsloth quants at full speed, EXL3, basic multi-GPU, public CI and more
retained text
Big update to the Delta Inference Engine, you guys asked for more features, more benchmarks and less clickbait. I'm here to deliver **The short version:** I've added almost everything requested and more. Delta Inference Engine now supports 64/96/128 GB of RAM and fully supports the new Unsloth quants. Unsloth IQ2 runs at the same 41 tok/s (192 GB RAM) using a new native reader for the Engram tables and some clever CPU kernel work. Compared to the old default model on my 99-question benchmark it went from 52 to 57 correct, against 59 for the full model. It's no full deepseek model quant, but they're a lot more usable than you might expect. **Speed on one RTX 3090 24 GB** (i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe, Windows, no WSL). Writing speed, tok/s, greedy, one chat at a time: |RAM|UD-IQ2\_XXS, 2-bit, 301 GB|UD-IQ1\_S, 1-bit, 233 GB|UD-Q2\_K\_XL, 2-bit, 335 GB|UD-IQ3\_XXS, 3-bit, 395 GB + the default| |:-|:-|:-|:-|:-| |192 GB|41|41.0|31.3|28.2| |128 GB|34.0|40.6|26.6|22.8| |96 GB|29.3|36.5|22.9|18.3| |64 GB|21.0|30.6|15.2|11.7| The 128, 96 and 64 GB rows are the same PC with the rest of the RAM locked away. **Prompt reading:** long prompts read at up to 740 tok/s at 192 GB and 600 at 128 GB (at \~64K tokens). A short prompt starts answering in about 2 s. **Benchmark:** 99 hard questions, greedy, 43K token budget: |Set|UD-IQ2\_XXS (this release)|ds4 Q2 (last release)|Full model, API|Qwen 3.8 Flash-Next, API| |:-|:-|:-|:-|:-| |Knowledge (40)|12|9|13|7| |Maths (14)|10|10|10|8| |Reasoning (30)|25|23|25|21| |Coding (10)|10|10|10|8| |Hard coding (5)|0|0|1|0| |**Total (99)**|**57**|**52**|**59**|**44**| UD-IQ2\_XXS is within 2 points of the full API, never scores below ds4 Q2 on any set while running at the same speed, and used 11% fewer tokens across the run. **Agreement with DeepSeek's API on the next token, at the settings that ship:** |Model|General text|Code and maths|Agentic| |:-|:-|:-|:-| |UD-IQ2\_XXS|91.1%|94.5%|95.9%| |ds4 Q2|89.9%|93.1%|95.1%| |UD-Q2\_K\_XL|92.5%|95.3%|97.0%| |UD-IQ3\_XXS|92.7%|95.9%|97.4%| |UD-IQ1\_S|89.1%|91.5%|95.3%| For reference, two hosts of the full model agreed 98.3% and 98.9% in my test. By that measure Unsloth IQ2 is about 20% closer to the full model than the old ds4 Q2. If you want better quality: UD-Q2\_K\_XL (335 GB) runs 31 tok/s at 192 GB and 26.6 at 128 GB, at 92.5% / 95.3%. UD-IQ3\_XXS (395 GB plus the default) runs 28 and 22.8 at 92.7% / 95.9%. **What changed since last time:** * **First token \~40% sooner:** a \~180-token prompt went from about 3.8 s to 2.3 s. * **Agent turns read \~12% faster**, and tool calls that low-bit models write slightly wrong are now read as tool calls. * **Unsloth quants at full speed:** the engine reads their Engram tables directly. * **64 and 96 GB are real options:** SSD read-ahead (+22% at 64 GB), bigger expert memory, and prompt reads in file order (18-20% faster on low RAM). * **128 GB:** ds4 Q2 went from 33 to 36 tok/s. * **Default sampling:** the drafter now samples its drafts too, about +3.5% at default settings. * **First launch** shows which model suits your RAM, and you can pick any other or use your own. * **A Setup tab** in the chat page changes model, speed vs quality and the second GPU. * **Public CI:** the release zips are now built by GitHub Actions, so you can see how each one was made. * **EXL3:** 2.5 bpw is on the menu and the GPU kernel is about 2x faster. In my tests UD-Q2\_K\_XL was both faster and closer to the API, so I still point people there. * **Basic multi-GPU:** 7% uplift using a 3060 + 3090. It doesn't scale well to 3-4+ GPUs, so use Project Maya for those builds. * **Fixes:** contexts past 65,535 tokens, `</think>` in prose no longer ends the reasoning, no out-of-memory on the biggest files, a failed picture encoder no longer stops the model, faster prompts after a pause and many more. * **Support for custom model selection:** just add them to the custom model folder and pick them in the web-ui **Requirements:** * **OS:** Windows 10 or 11 * **GPU:** NVIDIA RTX with 12 GB of VRAM or more, driver 580 or newer. 24 GB is the fast path, and AMD GPUs aren't supported yet * **RAM:** 64 GB or more, 128 GB+ recommended * **Disk:** a fast NVMe SSD, with 233-395 GB for the model (UD-IQ3\_XXS also needs the default model) * **CPU:** any x86-64 CPU, Intel or AMD. AVX2 is the fast path All of this is one PC. It needs testing on other hardware. Repo: [https://github.com/musicurgy/delta-inference-engine](https://github.com/musicurgy/delta-inference-engine) MIT licensed. It's a Windows port of Xolotl, built on Strata. Thanks to DeepSeek for the model, Unsloth, nazeshinjite and ISTA-DASLab for the quants, and to the Xolotl and Project Maya communities, whose work and public numbers helped a lot. Claude wrote most of the code and I ran the project. If you try it, post your GPU, RAM, model and tok/s, good or bad.
document fetched 2026-11-06 00:45 · canonical https://reddit.com/r/LocalLLaMA/comments/1xoq4vq/deepseekv41flash1017_on_one_rtx_3090_big_updates