← sources

Multi-Token Prediction Done Right [R]

source: reddit (reddit:MachineLearning)published 2026-10-30 18:53ingested 2026-10-31 02:37

retained text

Sharing one of our latest papers on Multi-Token Prediction (MTP). MTP is now a standard recipe for frontier pre-training but turns out, it can be done much better. 

We show the standard MTP gradient hides a KL self-distillation signal (which we want for improving speculative decoding) behind a noisy residual (that harms next-token prediction). 

I think the coolest part was the KL gradient–recovery trick to extract the KL gradient. You just need a one-line code change:  


h\_ntp += h\_mtp.detach() - h\_mtp

  
**Soft Multi-Token Prediction (Soft-MTP)** is essentially a free lunch that improves MTP across a range of tasks.

I'm super excited about this work. Please do check it out below:

💻 Code: [https://github.com/JaydenTeoh/Soft-MTP](https://github.com/JaydenTeoh/Soft-MTP)  
📝 Paper: [https://arxiv.org/abs/2610.38331](https://arxiv.org/abs/2610.38331)  
🤗 HuggingFace: [https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction](https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction)

document fetched 2026-10-31 00:27 · canonical https://reddit.com/r/MachineLearning/comments/1xjnz40/multitoken_prediction_done_right_r