# Meta trained small AI models to read raw bytes instead of tokens. They start out worse, but the paper predicts they end up better — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/ce6ab909dace6040

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-24 13:08 ingested 2026-10-25 02:39

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xe88ja/meta_trained_small_ai_models_to_read_raw_bytes/)

## retained text



```
[https://arxiv.org/pdf/2609.12303](https://arxiv.org/pdf/2609.12303)

this paper suggests a model that skips models reading tokens that and has the model read raw bytes, basically one letter at a time.

what they did:

* trained small models (\~1B params), some on tokens and some on bytes
* trained them partly by copying a bigger model (llama 3 8b), aka distillation
* the tricky part is that the teacher thinks in tokens and the student reads bytes, so they had to work out a clean way to translate between the two

what they found:

* with a little training the token models win, not even close
* with more training the token models kind of flatline while the byte ones keep improving
* if you extend the trend lines, the byte model ends up beating llama 3.2 1b and gemma 3 1b by a few points
* the byte models also needed about 6x less text to hit the same level

the catch (and there’s a few):

* the “beats llama and gemma” part is a prediction. they extended the curves past what they actually trained. in their real runs the token model was still ahead
* reading letter by letter is slower. a token is \~4-5 letters, so it’s roughly 5x more steps for the same text. less data but more compute
* tokens aren’t really dead, since the teacher model still uses them
* only tested on small models, no idea yet if it holds at scale

still pretty cool imo. for small models on phones/laptops, squeezing more quality out of the same size matters a lot. and getting rid of the giant word list might fix the dumb stuff like miscounting letters in a word

anyone know if there’s work on making byte models run as fast as token ones?
```



document fetched 2026-10-25 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1xe88ja/meta_trained_small_ai_models_to_read_raw_bytes
