Just "rediscovered" batching and swarming while running models locally.
retained text
Running one harness/agent gives overall less tokens-per-second (tps) than running multiple agents at the same time. Who'd have thought? Though each agent has slower tps and a smaller cache, on average they are much more efficient overall in the token generation. I'm Running a 3090 with 24gb of VRAM. Qwen3.8 27B Q4_K_M Context: 197k Of course the more agents you have, the less context each gets. Thus one needs to auto-compact / loop efficiently. This way one can run multiple agents on one piece of hardware and maximise the burn-rate without increasing the energy used. From there swarms can be allowed, were each agent has the ability to spawn as many sub-agents with small tasks upto the optimum amount of agents per machine (12 on mine). Agents generating | Total tok/s | Per agent ---|---|---- 1 | 30 | 30 2 | 46 | 23 4 | 64 | 16 6 | 74 | 12 8 | 78 | 10 12 | 110 | 9 16 | 111 | 7
document fetched 2026-11-01 00:38 · canonical https://reddit.com/r/LocalLLaMA/comments/1xjx63n/just_rediscovered_batching_and_swarming_while