8 agents × 250K context, concurrently, on two 3090s (~600+ tok/s total)
retained text
I am open to the idea that I am misguided, but my findings feel worth sharing: Short video of my UI with 8 agents each explaining a codebase in parallel long prose: https://imgur.com/JSqlXIi A few days ago I read the [Stanford/Berkeley TokenGate paper](https://old.reddit.com/r/LocalLLaMA/comments/1xfovxr/researchers_from_stanford_and_uc_berkeley/). The idea is to make long context cheap by pruning tokens up front. I tried it. It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard. So I went the other way - don't throw anything away, just don't look at all of it at once. Keep the whole context in system RAM and at each step pull in only the ~1% that matters right now. A few days later, I have two 3090s run 8 coding agents at once, **each holding their own 250K-token context**: - Every decode step, each attention layer reads only ~1% of the tokens, picked with a tiny `rank-64` "key sketch". - The full KV cache lives in system RAM (`int4` in my 80GB RAM VM, but `int8` should fit in 128gb). The GPU keeps the sketch and a small hot set. - Only the picked tokens cross PCIe. What it took: - **Triton kernels** that build the sketch, score every token, pick the top ~2K and pack them for the stock int8 attention kernel. CUDA-graph safe. - **vLLM patches** that switch dense/sparse per step, so short chats stay full speed. - **A RAM tier for the KV cache** with a prefix cache and recurrent-state checkpoints, so follow-ups don't recompute. - **Follow-ups fetch only the picked rows.** - **A scheduler fix** so one agent reading a huge prompt doesn't freeze the rest. - A frozen sketch basis (one file) plus the script to refit it. It's all in the repo as patches, not a fork. Apply them on top of [HyperQwen](https://github.com/syv-ai/HyperQwen) @ `ce9615c` (a public vLLM fork) and run the install script. Tests and every bench script and result are included. Model: [`dbirks/Qwen3.8-27B-W4A16-AutoRound`](https://huggingface.co/dbirks/Qwen3.8-27B-W4A16-AutoRound) + HyperQwen's `prepare/fetch_fast_variant.py`. All numbers were measured with [gpu-unlock](https://github.com/Zagitta/gpu-unlock) loaded (a kernel module that roughly doubles 3090 BF16 tensor throughput, mostly faster prompt reading). Measured (`Qwen3.8-27B W4A16`, `MTP-7`, 3090 Ti + 3090 at 250 W): - 8 × 250K: `~545 tok/s` total on explain-style output, `~790` on copy-heavy. - 8 agents at 176–205K generating nonstop for 6 min: `657 tok/s` box average, 69–90 per agent. - Recall at 250K: 31/32. Follow-ups start in ~5 s. - **Single long chats are faster too**: +15–41% at ~113K vs dense. Caveat: attention is approximate, so near-duplicate code lines occasionally get mixed up. Code: - Serving (patches + kernels): https://github.com/bradj00/sparsekv-hyperqwen Rig: * i9-12900K * Z790 motherboard * 96 GB DDR5 RAM * 3090 Ti + 3090, power capped at 250 W ------------- *edit:* benchmarks underway... - KLD testing - RULER, LongBench v2, cross-file code retrieval - GPQA, LiveCodeBench, IFBench, MMLU-Pro - SCBench + MTP speed - dense vs sparse vs RAM-tier
document fetched 2026-11-03 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1xlk43x/8_agents_250k_context_concurrently_on_two_3090s