loading
loading
Accelerating agentic RL and evaluation research velocity with 45x faster GKE Agent Sandbox Tinsley Shi Product Manager Tomer Glottmann Staff Software Engineer, Technical Lead When scaling up agentic reinforcement learning (RL) and evaluation across massive parallel rollouts, frontier AI labs inevitably hit a bottleneck: Expensive GPU clusters sit idle, waiting minutes for CPU sandbox cold-starts, plus thousands of multi-gigabyte SWE-bench-style image pulls and scheduling backlogs. It’s a sandbox infrastructure problem that silently slows down your research and burns your training budget. To solve this fundamental infrastructure bottleneck, today we are introducing GKE Agent Sandbox optimized for RL along with the Agent Sandbox RL orchestration SDK, plus native integrations for popular RL gyms and harnesses, now generally available. As the operating system for modern AI, Kubernetes has evolved to power massive GPU/TPU training clusters and distributed inference. Now Kubernetes is expanding to drive the next AI compute frontier: agents. But unlike static workloads, agentic workloads evolve rapidly, so infrastructure must evolve just as fast. Rather than guessing at what RL researchers needed, we placed Kubernetes itself on an auto-research and verification loop driven by performance benchmarks and evaluations. We used heavy agentic benchmarks like SWE-bench to intentionally stress-test and break our own clusters. Every bottleneck that surfaced — from etcd timeouts to GPU idle spikes — was fed back into our development cycle to refine GKE’s core primitives. This resulted in a purpose-built sandbox layer for agentic RL and eval workloads that features: - 10x - 45x faster time-to-first-command: GKE can spin up a sandbox environment in 1–9 seconds instead of 45–85 seconds, keeping your expensive GPUs fully in use. - Reduced tail latency: We reduced the worst-case sandbox wait times from 7.5 minutes down to under 10 seconds. - 3x less control-plane churn: Each RL training step triggers a rollout burst, where thousands of sandboxes are requested at once. The SDK has an in-place recycling strategy that reuses pods across rollouts instead of deleting and re-creating them. That means 3x fewer pods being created, which keeps the Kubernetes API server stable under the churn. With this new primitive, AI labs and agent-native startups can now reliably run large scale agentic RL trajectories and evals simultaneously, minimizing accelerator idle time and drastically accelerating their research velocity. The real-world challenges of agentic RL infrastructure Before we talk about the solution, let’s be precise about what makes agentic RL so demanding for infrastructure in the first place. In a standard agentic RL loop, an LLM policy generates actions like code snippets on GPUs and executes them inside isolated CPU sandboxes to observe a reward signal. However, when scaling up this loop to support tens of thousands of parallel rollouts, three critical infrastructure bottlenecks emerge: - Accelerator idle costs, i.e., time-to-first-command (TTFC) and the tail latency trap: Agentic RL is a batch workload, and in a synchronous RL step, training cannot proceed until the slowest sandbox in the batch is ready. If standing up CPU sandboxes takes minutes to provision, pull images, and execute initial startup scripts, expensive accelerator capacity sits wasted. - Massive image cardinality: Standard container caching assumes a handful of static base images. However, in agentic RL, every single task (like thousands of GitHub repositories in SWE-bench or R2E-Gym) often requires a completely distinct OCI container image. Managing thousands of unique, large images per run can lead to severe image-pulling bottlenecks and storage friction. - Control-plane saturation: As a bursty batch workload, agentic RL typically executes batch-size image tasks and multiple rollouts per task (e.g. 4, 8, 16 rollouts per task), pushing an already large number of images to the extreme. For instance, one public dataset we tested against has 4,578 R2E images. Assuming four rollouts per image, that means 18,312 tasks, which require 18,312 sandboxes simultaneously. Standard Kubernetes control planes degrade significantly under these burst loads. Churning tens of thousands of ephemeral pods per minute creates API-server queue bottlenecks, pod state errors and triggers false node-health evictions. These are not hypothetical problems. They are the daily reality for frontier AI labs training state-of-the-art agents. GKE Agent Sandbox optimized for RL To fan-out the code execution sandboxes, we set up a relatively modest cluster — a 10-node gVisor sandbox pool with GKE image streaming enabled, the Agent Sandbox controller, and an in-cluster SDK driver to claim warm pods. We tested various strategies and setups including a large number of images and high cardinality. The infrastructure layer GKE Agent Sandbox is the open Kubernetes primitive for secure agent execution, featuring built-in SandboxWarmPool capabilities that eliminate cold-start overhead by maintaining pre-initialized, healthy environments. By integrating SandboxWarmPool with GKE Image Streaming, we effectively support RL and eval workloads that demand high image cardinality — even with thousands of large images (>1.2GB), while delivering the exceptionally low TTFC required for responsive agentic training. Its snapshot, suspend and resume capabilities support checkpointing for error recovery, and sandbox forking for parallel agent branching logic to explore multiple trails. This GKE primitive is already powering the agentic RL training infrastructure at Mistral AI, a frontier AI lab: “To push the boundaries of reinforcement learning, you need infrastructure that can instantly scale to handle unpredictable demand. By leveraging GKE's high-performance Agent Sandbox for RL, we can seamlessly orchestrate hundreds of thousands of secure environments across clusters and handle spikes of over 30,000 sandboxes on a single cluster. It provides the reliable foundation we need to accelerate our model training and iteration cycles.” - Jean-Malo Delignon, Research Engineer, Mistral AI Orchestration SDK and RL tool integrations To help researchers harness this power without writing Kubernetes YAML or embedding custom daemons into container images, we built the Agent Sandbox RL orchestration SDK. It exposes a clean, async Python API with pluggable warm-pooling strategies tailored to your specific evaluation or RL training pattern. We also built native integrations for RL tools like Gymnasium, NVIDIA NeMo Gym, and OpenHands, with more to come. Benchmarks: Reducing idle accelerator waste We ran this on a 10-node gVisor sandbox pool against two workloads: a 500-image SWE-bench environment, and a harsher 4,578-image R2E corpus that does not fit in local disk. 1.TTFC and tail latency: 10x–45x faster Concurrency ranged from 500 simultaneous tasks and sandboxes up to 18,312 (4,578 R2E images × 4 rollouts). The gain held across every setup. How we got here. We instrumented the controller, the SDK, and the RL fleet, then ran hundreds of comparison runs against that fixed 10-node budget. Two changes produced nearly all of the gain: - Image hydration moved off the critical path. A high-cardinality corpus far exceeds local disk, so pulling cold images mid-training creates I/O contention and multi-minute tails. The SDK plans image placement across nodes, and GKE Image Streaming plus upfront warm-pooling handle the rest — hydration finishes before the rollout ever asks for it. - The control plane is paced. High-concurrency rollouts trigger API-server thundering herds. Agent Sandbox Controller v1.0.0 adds rate controls that bound how fast warm pools refill, keeping etcd and the API server stable through the burst. The tuning knobs, the benchmark harness, and the load tests behind these numbers all ship in the agent-sandbox-rl example, which emits human-readable and JSON reports so you can reproduce the comparison on your own cluster. 2. 3x less control-plane and pod lifecycle churn during multi-trajectory rollouts Raw Kubernetes recreates a pod for every trajectory step. At 18,312 tasks that inflates scheduling overhead, wastes disk I/O, and — in our large-scale tests — triggered unbounded garbage-collection loops. The SDK's in-place recycle strategy keeps the pod alive instead, running an in-pod git reset and repository checkout between rollout episodes. Pod creations drop 3.1x and the control plane stays flat through the burst. The trade-off Warming environments up front spends inexpensive CPU and background cluster time so that image streaming and readiness checks never land on the critical path. For an RL fleet that’s a straightforward trade: Accelerator idle time is the expensive resource, and this approach eliminates it. One detail matters at training scale: The SDK counts and surfaces every failed sandbox as retriable rather than silently dropping it. An untracked drop is not just a lost rollout — it is reward bias. Get started - Build today: Explore the Agent Sandbox repository or the Agent Sandbox RL repository for the Python SDK and native integrations for RL tools including Gymnasium, NVIDIA NeMo Gym, and OpenHands. - Push accelerator utilization further: Explore the llm-d co-operative time-slicing repository and its Snapshot Agent guide to dynamically interleave independent RL jobs onto shared physical hardware. - Learn more: Read the official documentation on deploying secure execution environments with GKE Agent Sandbox.
document fetched 2026-09-30 02:59 · canonical https://cloud.google.com/blog/products/containers-kubernetes/accelerate-agentic-rl-with-gke-agent-sandbox