Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements (Dwarkesh Podcast)
retained text
Techmeme permalink: https://www.techmeme.com/260908/p49#a260908p49 <a href="https://www.dwarkesh.com/p/pretraining-progress-is-mostly-data"><img align="RIGHT" border="0" hspace="4" src="http://www.techmeme.com/260908/i49.jpg" vspace="4" /></a> <p><a href="https://www.techmeme.com/260908/p49#a260908p49" title="Techmeme permalink"><img height="12" src="http://www.techmeme.com/img/pml.png" style="border: none; padding: 0; margin: 0;" width="11" /></a> <a href="https://www.dwarkesh.com/">Dwarkesh Podcast</a>:<br /> <span style="font-size: 1.3em;"><b><a href="https://www.dwarkesh.com/p/pretraining-progress-is-mostly-data">Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements</a></b></span> — Breaking down 6 years of pretraining progress into data vs model improvements — How much of the rapid progress in AI that we've seen over the last few years 1 … </p>
document fetched 2026-09-09 11:52 · canonical https://dwarkesh.com/p/pretraining-progress-is-mostly-data