loading
loading
Reflection AI's Beam Ran 1.3 Billion Sandboxes on 10,500 GPUs Reflection AI reveals the raw infrastructure behind Beam's reinforcement learning run and commits to OpenEnv as its future environment standard. - Reflection AI ran ~1.3 billion sandboxes to train Beam, its 501B open-weight MoE model. - Fleet peaked at 883,006 vCPUs across 23 clusters on GCP and AWS, 83% of allocatable capacity. - RL used 10,500 GB300 GPUs for four weeks, 100M+ rollouts, 1M curated coding and agentic environments. - Asynchronous RL stayed stable even with rollouts over a day stale from the current policy. - 71 inference incidents were handled mid-run without terminating the training job. - Reflection will adopt OpenEnv, the Agentic AI Foundation's new RL environment standard, for future runs. Reflection AI has published infrastructure details for Beam, its 501-billion-parameter open-weight mixture-of-experts model. This architecture routes each token through selected specialist subnetworks, reducing the compute used per token. The disclosure covers Beam’s largest reinforcement-learning campaign and Reflection’s decision to use OpenEnv for future runs. A month of sandboxes Reflection’s figures describe a four-week campaign distributed across Google Cloud and AWS: A sandbox is an isolated execution environment where the model can write code, call tools, manipulate files, or solve a technical problem without affecting other tasks. Each rollout records a sequence of model actions, observations, and rewards that the trainer can use to update the model. The final run also hosted evaluations and ablation experiments alongside training. Ablations remove or alter individual components to measure their contribution, so these jobs added load beyond the primary rollout pipeline. Stale rollouts kept the fleet moving Synchronous reinforcement-learning systems pause while workers finish rollouts and receive updated model weights. Slow tasks can leave expensive accelerators waiting, especially when rollout lengths vary widely. Beam used asynchronous reinforcement learning, allowing rollout generation and weight updates to proceed in parallel. A long rollout could therefore contain tokens generated by several versions of the model. That creates off-policy data, meaning the trainer learns from interactions produced by older weights. Reflection says its algorithms reduced the mismatch between the training model and inference workers sufficiently to learn from interactions generated more than a day earlier. That tolerance allowed as many as 110,000 rollouts to remain active while updates continued. Updated weights reached the inference fleet in a median of roughly 12 seconds through a hierarchy built on RoCE networking and NVIDIA’s NVLink interconnect. Reflection also handled 71 inference incidents without restarting the month-long training job, indicating that failures could be isolated while the remaining fleet continued working. One interface for training and agents Reflection plans to standardize future environment work on OpenEnv, a community-governed framework hosted by the Agentic AI Foundation. The foundation added OpenEnv among four new projects, bringing its portfolio to 10 projects during its first year. OpenEnv packages isolated environments behind an interface modeled on Gymnasium, a widely used API for reinforcement-learning environments. It covers environment creation, containerized deployment, client access, and interaction through three core operations: env.reset() # start an episode obs, reward, done, meta = env.step(action) env.state() # inspect current state A client calls reset() to begin an episode, submits actions through step(), and can inspect the current state with state(). Each step can return an observation, numeric reward, completion status, and metadata. Some environments also expose tools through the Model Context Protocol, or MCP. Trainers can use the Gym-style interface, while external agents can access the same tools through MCP. Environment authors can therefore reuse a packaged task during training, evaluation, and inference. Infrastructure operators join the governance OpenEnv’s coordinating committee includes Meta-PyTorch, Reflection, Unsloth, Modal, Prime Intellect, NVIDIA, Mercor, Fleet AI, Microsoft, Hugging Face, and RadixArk. The group spans model developers, cloud infrastructure providers, training platforms, and developer-tool companies. Reflection says it will contribute lessons from Beam’s post-training campaign to the project. A shared contract gives environment authors a stable integration target and lets training systems replace individual sandbox providers without redesigning every task. Plans developers can act on Beam’s campaign suggests several engineering priorities for teams building reinforcement-learning systems: - Budget for CPU-heavy execution. Using 10,500 GPUs as the denominator, the reported peak corresponds to roughly 84 active vCPUs and 101 allocatable vCPUs per GPU. The appropriate ratio will vary with task duration, tool use, and sandbox overhead. - Separate rollout generation from training. Asynchronous pipelines can improve accelerator utilization, but trainers must record policy versions and account for stale trajectories. - Design for partial failure. Long-running jobs need retry policies, failure isolation, checkpointing, and observability across inference workers, sandboxes, networking, and weight distribution. - Package environments for reuse. Containerized, Gym-shaped environments with optional MCP exposure can serve training systems and deployed agents through separate interfaces. - Measure task-level efficiency. Aggregate fleet size does not reveal the cost or learning value of an individual environment. Cost, energy use, average utilization, and per-task efficiency remain undisclosed. Smaller teams can apply the same architecture to narrower workloads without reproducing Beam’s 1.3-billion-sandbox campaign. Reflection says it is publishing fine-tuning and reinforcement-learning tooling derived from the run, while OpenEnv provides a common format for sharing the environments that supply tasks and rewards.
document fetched 2026-10-28 02:52