Distill-then-finetune took my get-up controller from 71% to 94% (and why training one network from scratch failed)
retained text
I've been working for a few months on a 2D MuJoCo ragdoll (SB3 PPO) that has to get up from lying on its back or face down and still be standing 20 s later. **What happened:** \- A single policy never got off the floor (0/160 genuine get-ups; the learned policy raised the head *less* often than random bang-bang actions). \- So I split the task into four policies with their own rewards/curricula (supine floor, prone floor, a hand-over policy, stand) and joined them with a rule-based controller. That slowly reached \~71%, then plateaued for a long time: reward tweaks, re-training the stand policy, hand-over timing gates etc. were all within noise. \- Then I **distilled the whole controller (switching rules included) into one 512x512 network with DAgger** (same level as the rules, 74%) and **fine-tuned it end to end with PPO on a 20 s "standing" reward**. It jumped to **94.2%** (supine 97.7%, prone 90.7%; 600 fresh starts per side, rules on the same seeds: 70.8%). \- Even a 64x64 network gets 85% with this recipe; 256x256 matches 512x512. Because it looked too good, I checked for shortcuts: the unphysical root force actuators are never applied, moving the opponent body away changes nothing, 150-500 N pushes after the get-up change nothing, and standing is calm (torso height std \~1 cm). **Caveats:** simulation only, 2D, one training run per config, starts come from the env's own lying-start generator (other poses untested). And this is *not* evidence that one network trains from scratch: big networks trained from scratch on the floor task failed repeatedly in my runs. Quick start (downloads the model from the Hub, \~1.5 min): `python scripts/demo_hf.py` Code + docs with every experiment and failure: [https://github.com/SungyupKim/standai](https://github.com/SungyupKim/standai) Checkpoints: [https://huggingface.co/SungyupKim/standai-getup](https://huggingface.co/SungyupKim/standai-getup) Happy to answer questions, and I'd love to hear whether anyone has seen the same "pieces first, then distill and finish end to end" pattern work (or fail) on other multi-phase tasks.
document fetched 2026-11-01 00:35 · canonical https://reddit.com/r/reinforcementlearning/comments/1xkaydw/distillthenfinetune_took_my_getup_controller_from