Rotate K/V read order for better L2 cache hit rates by gaugarg-nv · Pull Request #32407 · ggml-org/llama.cpp
retained text
Significant prompt processing speedup with long contexts |gpu\_info|model\_type|n\_prompt|n\_gen|n\_depth|Master (t/s)|PR (t/s)|Speed-up| |:-|:-|:-|:-|:-|:-|:-|:-| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|131072|1290.10833|1822.36027|1.41| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|65536|2030.31282|2527.97409|1.25| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|32768|2931.96118|3142.20539|1.07| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|16384|3562.18311|3573.59336|1.00| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|8192|3854.83184|3850.97547|1.00| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|4096|3974.20194|4015.85501|1.01| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|2048|4079.23891|4077.58068|1.00| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|1024|4103.79141|4123.60737|1.00| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|512|4114.42862|4136.20591|1.01| |NVIDIA GeForce RTX 5090|qwen35 27B Q4\_K - Medium|512|0|0|4300.67808|4323.0908|1.01|
document fetched 2026-11-18 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xyi6hj/rotate_kv_read_order_for_better_l2_cache_hit