Go worker handling bursty large-file transfers. Unlimited concurrency vs bounded semaphore and retries
retained text
I'm working on a Go-based worker that processes multiple large-file transfers concurrently over HTTP. Work arrives in bursts, and each task involves downloading data from a source and uploading it to a destination. I'm trying to optimize aggregate throughput and task completion reliability without causing excessive queueing, deadline failures, or unnecessary retries. The current implementation allows concurrent workloads without an explicit transfer semaphore limit. I'm considering testing a bounded limit of 16 active transfer workloads, along with retries for transient HTTP and network errors. I'd appreciate feedback from engineers who have built high-throughput Go services or distributed workers. A few specific questions: 1. **Concurrency:** Under what conditions would you prefer unrestricted concurrency over a semaphore limiting active workloads to 16? What metrics would tell you that concurrency is too high? 2. **Throughput vs. latency:** If aggregate throughput is already near the available bandwidth ceiling, can a concurrency limit improve task completion times, or would it mostly introduce queueing? 3. **Retries:** For transient errors such as HTTP 429, 500, 502, 503, 504, and temporary network failures, what retry policy would you recommend? How do you avoid retries amplifying an existing overload? 4. **Semaphore lifecycle:** In Go, would you hold a semaphore slot throughout the entire logical task, including retry backoff, or acquire a slot separately for each network attempt? What trade-offs matter? 5. **Deadlines and cancellation:** How would you ensure that tasks waiting for a semaphore don't consume their entire deadline before transferring any data? 6. **Benchmark methodology:** How many independent runs or observation windows would you use before trusting a performance improvement? Which metrics would you prioritize? My initial comparison would track aggregate bytes transferred, completed tasks, failure rate, P50/P95/P99 task duration, queue wait, and actual concurrent transfers. I'm especially interested in practical engineering experience, benchmark design, and failure modes that aren't obvious from simple throughput tests. Thanks!
document fetched 2026-10-13 00:26 · canonical https://reddit.com/r/devops/comments/1x3wr26/go_worker_handling_bursty_largefile_transfers