You can't retry an LLM stream after the first token. What are you doing instead?
retained text
Been wiring a router in front of a few OpenAI-compatible providers, and failover is easy right up until the response starts streaming. Once the first SSE chunk leaves, a retry isn't invisible anymore. The client already has half an answer and there's no clean way to take it back. The options I've gone through, none of them good: Buffer the whole completion and flush at the end. Retries become trivial, streaming disappears. Only retry before the first token. Covers connection refused, 429, 5xx on connect. Does nothing for the provider that dies at token 400. Send an error event and let the client restart. Honest, but every client library surfaces it differently and most UIs just show a broken message. Seed a continuation request with the partial output and stitch. Works, but you get a visible seam and the model has to be told what it already said. Caching makes it worse. Retry against a different provider and you lose the prompt cache, so the retry can cost more than the original call. Weighted routing then sends the next turn somewhere else and loses it again. So: is anyone doing sticky routing keyed on the conversation rather than the API key? And has anyone found a client-side pattern that survives a mid-stream drop without the user noticing?
document fetched 2026-11-10 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xrnwif/you_cant_retry_an_llm_stream_after_the_first