MBZUAI Opens RL360, the Agent Training Engine Behind K2 Horizon

retained text

MBZUAI Opens RL360, the Agent Training Engine Behind K2 Horizon
IFM open-sources the agentic reinforcement learning stack behind K2 Horizon, running 8,192 concurrent sandboxes and 1.3M rollouts across seven days.
- IFM open-sourced RL360, the RL infrastructure behind K2 Horizon's six domain experts, under Apache 2.0.
- Reference run hit 8,192 concurrent sandboxes, 150 steps in 7 days, and ~1.3M rollouts.
- Infrastructure failures are tagged and masked from loss, so sandbox crashes do not penalize the model.
- Generation and training overlap: 80s training step hides inside 40min of rollouts, with ~4s NCCL weight broadcast.
- Modular stack: Miles (rollouts), Harbor (sandboxes), SMG+SGLang (inference), Megatron-LM (training).
- Public W&B workspace shows K2 Horizon 7B reward climbing 0.373 to 0.901 on 32 coding tasks.
The Institute of Foundation Models at MBZUAI has released RL360 under the Apache 2.0 license. The system powered reinforcement-learning post-training for the K2 Horizon model family, handling tasks in which models write code, call tools, and interact with containerized sandboxes.
The release includes public W&B logs from a seven-day reference run. A rollout represents one task attempt, potentially spanning many model responses, tool calls, and environment interactions.
When infrastructure corrupts the reward
Agentic reinforcement learning assigns rewards after a model interacts with an environment. A single rollout can take minutes while the model executes code, invokes tools, and waits for a sandbox. At high concurrency, those delays leave training workers and GPU-serving capacity waiting for results.
Sandbox failures also contaminate the learning signal. If a crashed container receives the same low reward as an incorrect solution, the optimizer treats both outcomes as model errors. RL360 labels infrastructure failures separately and masks affected trajectories from the training loss, allowing valid rollouts to continue contributing updates.
Five components keep the loop moving
RL360 divides orchestration, training, inference, and sandbox execution across explicit service boundaries. Operators can add capacity to a saturated component without expanding every layer of the deployment.
Generation overlaps policy updates
RL360 generates the next rollout batch while the trainer updates the policy on the current batch. In the public run, an approximately 80-second training step overlapped with about 40 minutes of agent activity. Synchronizing the new weights across 96 SGLang engines took roughly four seconds using NCCL, NVIDIA’s library for high-speed collective communication.
Ray actors assemble trajectories locally
RL360 distributes trajectory assembly across Ray session actors, avoiding a central manager whose CPU and memory demands would rise with concurrency. Each actor compacts trajectories near their source and discards bulky provider metadata, including per-turn mixture-of-experts routing IDs whose volume can grow rapidly during long conversations.
8,192 sandboxes expose the bottlenecks
The reference run exposed three operational limits involving image distribution, resource contention, and container placement:
- Start sandboxes in waves. Launching all 8,192 containers simultaneously caused competing image pulls and cascading pod failures. Timed admission waves kept demand within registry capacity.
- Reserve CPU and memory per pod. Sandboxes on the same node competed for resources, allowing a heavy build to slow or terminate a neighboring workload. RL360 tags pods that still fail and excludes their trajectories from the loss.
- Keep images close to execution. Random placement triggered repeated cold image pulls. Pinning images to specific nodes increased the share of starts served from warm local caches.
A reproducible recipe with a large footprint
The repository includes a coding recipe that deliberately trains K2 Horizon 7B on a fixed set of 32 tasks. It samples eight attempts per task and raises mean reward from 0.373 to 0.901 across 99 rollout iterations.
GRPO, or Group Relative Policy Optimization, derives a learning signal by comparing multiple attempts for the same task. Dynamic filtering removes groups that lack useful reward variation. The reported overfitting result serves as a pipeline check on a fixed training set; generalization requires separate evaluation.
The documented deployment assumes a Linux x86_64 cluster managed by Slurm, with NVIDIA H100 or H200 GPUs, Docker, the NVIDIA Container Toolkit, and shared storage.
The published baseline targets a multi-node GPU cluster, so smaller deployments require changes to resource allocations and concurrency. The repository also provides a Qwen3-8B swap recipe for teams that lack access to K2 checkpoints.
The clearest fit is cluster-scale agent training
RL360 joins agentic reinforcement-learning projects such as verl, OpenRLHF, and TRL. Its scope centers on the operational demands of real execution environments: long rollouts, thousands of concurrent containers, distributed trajectory processing, weight synchronization, and failures that must stay out of the reward signal.
Teams building coding agents or tool-using assistants can adopt individual patterns such as failure masking, staged sandbox admission, node-local image caching, and overlapped generation. IFM says a planned successor named Wrench will target more than 10,000 samples per training step.

document fetched 2026-10-23 16:26