# The algorithm that trains reasoning LLMs can't teach a robot to walk — retained record

> Retained reddit record from reddit:reinforcementlearning.

Canonical: https://highsignal.app/data/records/6413addbe3a453c0

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:reinforcementlearning)  published 2026-10-14 07:31 ingested 2026-10-15 02:41

[original source ↗](https://www.reddit.com/r/reinforcementlearning/comments/1x5kyut/the_algorithm_that_trains_reasoning_llms_cant/)

## retained text



```
Follow-up to an [earlier write-up](https://sreejithb.com/rl-policy-gradient/single) on the REINFORCE → PPO → GRPO lineage. That one ended with GRPO as "PPO minus the critic, plus a group-mean baseline", which raises an obvious question: if it's cheaper and good enough for DeepSeek's reasoning models, why does nobody use it for control?

Two reasons, and they're different kinds of problem - presence of bias and a common feedback applied uniformally to each step. The latter is REINFORCE's original credit-assignment problem, back because the critic is gone. An LLM gets away with it (one terminal reward, quiet environment, replayable start state, pretrained prior). A robot doesn't.

  
De Oliveira et al. (2511.03527) ran critic-free methods vs PPO on five classic control tasks. PPO won everywhere except CartPole, and GRPO's group weighting did no better than a plain batch mean.

Write-up with interactive figures: [https://sreejithb.com/grpo\_dissect/single](https://sreejithb.com/grpo_dissect/single) Disclosure: the argument and the script are mine; Claude helped with polish and the visuals.


```



document fetched 2026-10-15 00:43 · canonical https://reddit.com/r/reinforcementlearning/comments/1x5kyut/the_algorithm_that_trains_reasoning_llms_cant
