Multi-Token Prediction Done Right [R]
retained text
Sharing one of our latest papers on Multi-Token Prediction (MTP). MTP is now a standard recipe for frontier pre-training but turns out, it can be done much better. We show the standard MTP gradient hides a KL self-distillation signal (which we want for improving speculative decoding) behind a noisy residual (that harms next-token prediction). I think the coolest part was the KL gradient–recovery trick to extract the KL gradient. You just need a one-line code change: h\_ntp += h\_mtp.detach() - h\_mtp **Soft Multi-Token Prediction (Soft-MTP)** is essentially a free lunch that improves MTP across a range of tasks. I'm super excited about this work. Please do check it out below: 💻 Code: [https://github.com/JaydenTeoh/Soft-MTP](https://github.com/JaydenTeoh/Soft-MTP) 📝 Paper: [https://arxiv.org/abs/2610.38331](https://arxiv.org/abs/2610.38331) 🤗 HuggingFace: [https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction](https://huggingface.co/collections/JaydenTeoh/soft-multi-token-prediction)
document fetched 2026-10-31 00:27 · canonical https://reddit.com/r/MachineLearning/comments/1xjnz40/multitoken_prediction_done_right_r