← sources

Tencent Hy4-flash (85 GB GGUF) on an RTX 3060 12 GB + 32 GB RAM, with a 20,000-token prompt

source: reddit (reddit:LocalLLaMA)published 2026-11-19 14:50ingested 2026-11-20 02:39

retained text

I found Hy4-flash on OpenRouter, it has like 64T weekly tokens if I am correct, this is insane amount, and in same day I saw how someone said "I wish it was in Strata", so I implemented it in RustingStrata, bc this model looks pretty smart for its size.

Hy4-flash IQ2\_S/IQ2\_XXS is 85 GB in two files (AtomicChat/Hy4-flash-GGUF). My box has an RTX 3060 12 GB, a Ryzen 5 7600X, 30 GiB of usable RAM and an NVMe drive. The model is 7 times bigger than the card and almost 3 times bigger than the RAM. I added it to RustingStrata, my Rust GGUF engine, to see what this machine does with it.

The prompt is 20,000 tokens of real text that never repeats: the project's docs followed by Rust source. Then 64 tokens of output.

||CUDA|Vulkan, same card|
|:-|:-|:-|
|Reading the 20,000-token prompt|11.6 tok/s (29 min)|10.6 tok/s (31 min)|
|Writing after it|3.3 tok/s|1.9 tok/s|

After a short prompt it writes 4.3 to 5.1 tok/s on CUDA.

So: half an hour to read 20k tokens, then about 3 tokens a second. Not something for a coding agent. But it is the full model on a 12 GB card, and it answers.

Why it is this slow: one token needs about 2.6 GB of expert weights, and most of them are on the SSD, not in RAM. The engine keeps as many experts as fit in VRAM, and for the others it splits the work between the CPU and the card, depending on which side is faster at that moment. The prompt is read 1024 tokens at a time, so each expert is loaded once per 1024 tokens and not once per token.

Things I could not test, and where I would like your numbers:

* An AMD card. The Vulkan path is the same code, but I only ran it on the 3060.
* A machine whose RAM holds the whole 85 GB. The engine then reads the file into RAM at start, so the SSD should stop mattering. I cannot check that here.
* Two cards (`--device 0,1`).

I wanted to compare with llama.cpp, but Hy4-flash support there is still a draft pull request (#32517), so these are RustingStrata numbers only.

To try it:

    rustingstrata chat --model Hy4-flash-AD-IQ2_S-IQ2_XXS-00001-of-00002.gguf

No flags needed. It uses CUDA on NVIDIA and Vulkan on everything else. `serve` gives an OpenAI-style endpoint.

Setup: RustingStrata, `bench --model Hy4-flash-AD-IQ2_S-IQ2_XXS-00001-of-00002.gguf --prompt-file prompt.txt --pp 20000 --tg 64`, Vulkan row with `--backend vulkan`.

Repo: [RustingStrata](https://github.com/RustingStudio/RustingStrata)

document fetched 2026-11-20 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1y0b8io/tencent_hy4flash_85_gb_gguf_on_an_rtx_3060_12_gb