← sources

Tokenization: A Survey for Modern NLP [R]

source: reddit (reddit:MachineLearning)published 2026-09-30 18:13ingested 2026-10-01 02:40

retained text

Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past \~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field.

  
We cover every aspect of tokenization: algorithms, evaluations, multilinguality, encodings, theory, etc. We even cover what you might want to replace tokenizers with (e.g., latent or visual tokenization). We also cover some topics that are closely adjacent to tokenization, such as constrained generation, token healing, and tokenizer security concerns.

  
Check it out!



[https://www.alphaxiv.org/abs/2609.tokenization-survey-modern-nlp](https://www.alphaxiv.org/abs/2609.tokenization-survey-modern-nlp)

document fetched 2026-10-01 00:30 · canonical https://reddit.com/r/MachineLearning/comments/1wuccjf/tokenization_a_survey_for_modern_nlp_r