Meta trained small AI models to read raw bytes instead of tokens. They start out worse, but the paper predicts they end up better
retained text
[https://arxiv.org/pdf/2609.12303](https://arxiv.org/pdf/2609.12303) this paper suggests a model that skips models reading tokens that and has the model read raw bytes, basically one letter at a time. what they did: * trained small models (\~1B params), some on tokens and some on bytes * trained them partly by copying a bigger model (llama 3 8b), aka distillation * the tricky part is that the teacher thinks in tokens and the student reads bytes, so they had to work out a clean way to translate between the two what they found: * with a little training the token models win, not even close * with more training the token models kind of flatline while the byte ones keep improving * if you extend the trend lines, the byte model ends up beating llama 3.2 1b and gemma 3 1b by a few points * the byte models also needed about 6x less text to hit the same level the catch (and there’s a few): * the “beats llama and gemma” part is a prediction. they extended the curves past what they actually trained. in their real runs the token model was still ahead * reading letter by letter is slower. a token is \~4-5 letters, so it’s roughly 5x more steps for the same text. less data but more compute * tokens aren’t really dead, since the teacher model still uses them * only tested on small models, no idea yet if it holds at scale still pretty cool imo. for small models on phones/laptops, squeezing more quality out of the same size matters a lot. and getting rid of the giant word list might fix the dumb stuff like miscounting letters in a word anyone know if there’s work on making byte models run as fast as token ones?
document fetched 2026-10-25 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1xe88ja/meta_trained_small_ai_models_to_read_raw_bytes