Is the memory a max-context KV reservation holds actually costing anything besides concurrency?
retained text
chose c. not confident my first instinct was that it slows decode down, like you're dragging the dead region around every step. i don't think that's right though. nothing reads it, it's just unavailable, so the cost is that concurrency gets capped by reservations instead of by live tokens and you hit the wall way below what the tokens would fit in. which i assume is the whole argument for paged kv. what i can't work out is whether it matters if your workload is uniform, if requests really do run near max context, is paging buying you anything except bookkeeping?
document fetched 2026-09-18 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1wj10mk/is_the_memory_a_maxcontext_kv_reservation_holds