How can I use LFM 3 1.7B at high concurrency locally?
retained text
I'm trying to get a swarm of these agents to be able to do larger tasks when orchestrated so that I can try to use cloud models less, but I find that even with `--parallel 16`, after like 6 sequences running concurrently it just doesn't do any more. In addition, when I start another request, every other instance goes to about 1 tok/s at the same time until that request finishes prefill. Hardware: 16 GB RAM + 8 GB VRAM (RTX 4060 Laptop) OS: Windows 11 Command: `llama serve --hf-repo LiquidAI/LFM3-1.7B-GGUF --cache-type-k q8_0 --cache-type-v q8_0 --parallel 16 --kv-unified`
document fetched 2026-10-29 00:44 · canonical https://reddit.com/r/LocalLLaMA/comments/1xhvwzu/how_can_i_use_lfm_3_17b_at_high_concurrency