← sources

Tired of waiting 3 hours to see the results. How do you scale up the number of RL robots training simultaneously using PPO?

source: reddit (reddit:reinforcementlearning)published 2026-10-13 10:13ingested 2026-10-14 02:42

retained text

Hello scientists! Is there a formula for adjusting hyperparameters for PPO based on the number of agents you're training simultaneously?

I'm using ML-Agents and Unity engine to train robots.

After extensive tuning, I finally got my quadruped robot to walk really well. However, training is extremely slow because I'm only training 10 robots simultaneously. If I increase the number to 100 or 1,000, the training performance deteriorates and the training graphs look terrible.

How can I overcome this? My hardware is capable of handling up to 10,000 robots training simultaneously, but I don't understand which parameters I need to adjust to make this work.

Is there some kind of formula for scaling hyperparameters based on the number of parallel agents?

Most of my initial hyperparameters were taken from the quadruped robot examples provided with ML-Agents.

Hyperparameters for 10 agents:

max steps per agent life: 5000

Observation space: 123

Action space: 12

            trainer_type:   ppo
            hyperparameters:
              batch_size:   2048
              buffer_size:  20480
              learning_rate:        0.0003
              beta: 0.005
              epsilon:      0.2
              lambd:        0.95
              num_epoch:    3
              shared_critic:        False
              learning_rate_schedule:       linear
              beta_schedule:        linear
              epsilon_schedule:     linear
            checkpoint_interval:    500000
            network_settings:
              normalize:    True
              hidden_units: 512
              num_layers:   3
              vis_encode_type:      simple
              memory:       None
              goal_conditioning_type:       hyper
              deterministic:        False
            reward_signals:
              extrinsic:
                gamma:      0.995
                strength:   1.0
                network_settings:
                  normalize:        False
                  hidden_units:     128
                  num_layers:       2
                  vis_encode_type:  simple
                  memory:   None
                  goal_conditioning_type:   hyper
                  deterministic:    False
            init_path:      None
            keep_checkpoints:       10
            even_checkpoints:       False
            max_steps:      30000000
            time_horizon:   1000
            summary_freq:   30000
            threaded:       False
            self_play:      None
            behavioral_cloning:     None

document fetched 2026-10-14 00:39 · canonical https://reddit.com/r/reinforcementlearning/comments/1x4ssbd/tired_of_waiting_3_hours_to_see_the_results_how