Tired of waiting 3 hours to see the results. How do you scale up the number of RL robots training simultaneously using PPO?
retained text
Hello scientists! Is there a formula for adjusting hyperparameters for PPO based on the number of agents you're training simultaneously?
I'm using ML-Agents and Unity engine to train robots.
After extensive tuning, I finally got my quadruped robot to walk really well. However, training is extremely slow because I'm only training 10 robots simultaneously. If I increase the number to 100 or 1,000, the training performance deteriorates and the training graphs look terrible.
How can I overcome this? My hardware is capable of handling up to 10,000 robots training simultaneously, but I don't understand which parameters I need to adjust to make this work.
Is there some kind of formula for scaling hyperparameters based on the number of parallel agents?
Most of my initial hyperparameters were taken from the quadruped robot examples provided with ML-Agents.
Hyperparameters for 10 agents:
max steps per agent life: 5000
Observation space: 123
Action space: 12
trainer_type: ppo
hyperparameters:
batch_size: 2048
buffer_size: 20480
learning_rate: 0.0003
beta: 0.005
epsilon: 0.2
lambd: 0.95
num_epoch: 3
shared_critic: False
learning_rate_schedule: linear
beta_schedule: linear
epsilon_schedule: linear
checkpoint_interval: 500000
network_settings:
normalize: True
hidden_units: 512
num_layers: 3
vis_encode_type: simple
memory: None
goal_conditioning_type: hyper
deterministic: False
reward_signals:
extrinsic:
gamma: 0.995
strength: 1.0
network_settings:
normalize: False
hidden_units: 128
num_layers: 2
vis_encode_type: simple
memory: None
goal_conditioning_type: hyper
deterministic: False
init_path: None
keep_checkpoints: 10
even_checkpoints: False
max_steps: 30000000
time_horizon: 1000
summary_freq: 30000
threaded: False
self_play: None
behavioral_cloning: Nonedocument fetched 2026-10-14 00:39 · canonical https://reddit.com/r/reinforcementlearning/comments/1x4ssbd/tired_of_waiting_3_hours_to_see_the_results_how