The algorithm that trains reasoning LLMs can't teach a robot to walk
retained text
Follow-up to an [earlier write-up](https://sreejithb.com/rl-policy-gradient/single) on the REINFORCE → PPO → GRPO lineage. That one ended with GRPO as "PPO minus the critic, plus a group-mean baseline", which raises an obvious question: if it's cheaper and good enough for DeepSeek's reasoning models, why does nobody use it for control? Two reasons, and they're different kinds of problem - presence of bias and a common feedback applied uniformally to each step. The latter is REINFORCE's original credit-assignment problem, back because the critic is gone. An LLM gets away with it (one terminal reward, quiet environment, replayable start state, pretrained prior). A robot doesn't. De Oliveira et al. (2511.03527) ran critic-free methods vs PPO on five classic control tasks. PPO won everywhere except CartPole, and GRPO's group weighting did no better than a plain batch mean. Write-up with interactive figures: [https://sreejithb.com/grpo\_dissect/single](https://sreejithb.com/grpo_dissect/single) Disclosure: the argument and the script are mine; Claude helped with polish and the visuals.
document fetched 2026-10-15 00:43 · canonical https://reddit.com/r/reinforcementlearning/comments/1x5kyut/the_algorithm_that_trains_reasoning_llms_cant