tokenizers v1 (rust)
retained text
Hey all! I am Aritra from Hugging Face. I wanted to share an update on the \`tokenizers\` library that we have at Hugging Face. It has gone under major changes and we have finally released version 1 of it. Here are what we are most excited about: \> multiple language support \> multi-thread scaling \> minimal package size Read: [https://huggingface.co/blog/tokenizers-v1](https://huggingface.co/blog/tokenizers-v1)
document fetched 2026-09-22 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1wmfj04/tokenizers_v1_rust