← sources

Just "rediscovered" batching and swarming while running models locally.

source: reddit (reddit:LocalLLaMA)published 2026-10-31 01:16ingested 2026-11-01 02:40

retained text

Running one harness/agent gives overall less tokens-per-second (tps) than running multiple agents at the same time. Who'd have thought?

Though each agent has slower tps and a smaller cache, on average they are much more efficient overall in the token generation.

I'm Running a 3090 with 24gb of VRAM.
Qwen3.8 27B Q4_K_M    
Context: 197k

Of course the more agents you have, the less context each gets. Thus one needs to auto-compact / loop efficiently. This way one can run multiple agents on one piece of hardware and maximise the burn-rate without increasing the energy used.

From there swarms can be allowed, were each agent has the ability to spawn as many sub-agents with small tasks upto the optimum amount of agents per machine (12 on mine).

Agents generating | Total tok/s | Per agent
---|---|----
1 | 30 | 30
2 | 46 | 23
4 | 64 | 16
6 | 74 | 12
8 | 78 | 10
12 | 110 | 9
16 | 111 | 7

document fetched 2026-11-01 00:38 · canonical https://reddit.com/r/LocalLLaMA/comments/1xjx63n/just_rediscovered_batching_and_swarming_while