← sources

8 agents × 250K context, concurrently, on two 3090s (~600+ tok/s total)

source: reddit (reddit:LocalLLaMA)published 2026-11-02 01:19ingested 2026-11-03 02:42

retained text

I am open to the idea that I am misguided, but my findings feel worth sharing:

Short video of my UI with 8 agents each explaining a codebase in parallel long prose: https://imgur.com/JSqlXIi

A few days ago I read the [Stanford/Berkeley TokenGate paper](https://old.reddit.com/r/LocalLLaMA/comments/1xfovxr/researchers_from_stanford_and_uc_berkeley/). The idea is to make long context cheap by pruning tokens up front.

I tried it. It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard.

So I went the other way - don't throw anything away, just don't look at all of it at once. Keep the whole context in system RAM and at each step pull in only the ~1% that matters right now.

A few days later, I have two 3090s run 8 coding agents at once, **each holding their own 250K-token context**:

- Every decode step, each attention layer reads only ~1% of the tokens, picked with a tiny `rank-64` "key sketch".
- The full KV cache lives in system RAM (`int4` in my 80GB RAM VM, but `int8` should fit in 128gb). The GPU keeps the sketch and a small hot set.
- Only the picked tokens cross PCIe.

What it took:

- **Triton kernels** that build the sketch, score every token, pick the top ~2K and pack them for the stock int8 attention kernel. CUDA-graph safe.

- **vLLM patches** that switch dense/sparse per step, so short chats stay full speed.

- **A RAM tier for the KV cache** with a prefix cache and recurrent-state checkpoints, so follow-ups don't recompute.

- **Follow-ups fetch only the picked rows.**

- **A scheduler fix** so one agent reading a huge prompt doesn't freeze the rest.

- A frozen sketch basis (one file) plus the script to refit it.


It's all in the repo as patches, not a fork. Apply them on top of [HyperQwen](https://github.com/syv-ai/HyperQwen) @ `ce9615c` (a public vLLM fork) and run the install script. Tests and every bench script and result are included.

Model: [`dbirks/Qwen3.8-27B-W4A16-AutoRound`](https://huggingface.co/dbirks/Qwen3.8-27B-W4A16-AutoRound) + HyperQwen's `prepare/fetch_fast_variant.py`. All numbers were measured with [gpu-unlock](https://github.com/Zagitta/gpu-unlock) loaded (a kernel module that roughly doubles 3090 BF16 tensor throughput, mostly faster prompt reading).

Measured (`Qwen3.8-27B W4A16`, `MTP-7`, 3090 Ti + 3090 at 250 W):

- 8 × 250K: `~545 tok/s` total on explain-style output, `~790` on copy-heavy.

- 8 agents at 176–205K generating nonstop for 6 min: `657 tok/s` box average, 69–90 per agent.

- Recall at 250K: 31/32. Follow-ups start in ~5 s.

- **Single long chats are faster too**: +15–41% at ~113K vs dense.


Caveat: attention is approximate, so near-duplicate code lines occasionally get mixed up.


Code:

- Serving (patches + kernels): https://github.com/bradj00/sparsekv-hyperqwen


Rig:

* i9-12900K

* Z790 motherboard

* 96 GB DDR5 RAM

* 3090 Ti + 3090, power capped at 250 W


-------------
*edit:* benchmarks underway...

- KLD testing

- RULER, LongBench v2, cross-file code retrieval

- GPQA, LiveCodeBench, IFBench, MMLU-Pro

- SCBench + MTP speed

- dense vs sparse vs RAM-tier

document fetched 2026-11-03 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1xlk43x/8_agents_250k_context_concurrently_on_two_3090s