← sources

Does your agent hang up when it retries or cancels? In our vLLM test, unclosed retries wasted ~148 tokens per task

source: reddit (reddit:AI_Agents)published 2026-10-23 01:56ingested 2026-10-24 02:35

retained text

Agents retry a lot. Tool call times out, step is slow, a supervisor kills a subtask, the user hits stop. We wanted to know what happens on the model server when the client gives up, so we measured it on self-hosted vLLM 0.31.0 and counted every token generated after the cancel.



It came down to how the client walks away:



\- Closes the connection: generation stopped. 0 extra tokens.

\- Stops reading but leaves the socket open for 5 seconds: about 44 extra tokens per cancelled request.

\- Reads to the end and throws it away: about 53 extra, and about 175 on longer, natural answers.



The part that matters for agents is retries. With a 1.5 second deadline and one retry:



\- Old attempts that were never closed produced about 148 wasted tokens per task, against 0 when they were closed.

\- On-time completions fell from 38% to 21%, and to 11% when the old connection was left open. The abandoned attempt keeps competing with the retry for the same GPU.



So a timeout wrapper that just stops awaiting the stream, without closing it, can make the retry slower and burn tokens on the answer you already gave up on. In most clients the fix is closing or aborting the stream (AbortController, closing the response, cancelling the task) when you stop waiting.



Limits: tokens, not dollars, so no cost figure. vLLM and SGLang on L4 and A10G, two models, synthetic prompts. Not tested: llama.cpp, Ollama, TGI, hosted APIs, any agent framework's own timeout handling.



Report and data in the comments. Disclosure: I work at Sorami, the small security firm that ran this.



How do your agent frameworks handle a timed-out step? Do they actually abort the request, or just stop waiting for it?

document fetched 2026-10-24 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xd1awk/does_your_agent_hang_up_when_it_retries_or