# tokenizers v1 (rust) — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/fd6bdd1ab8ee7ec2

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-09-21 15:10 ingested 2026-09-22 02:40

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1wmfj04/tokenizers_v1_rust/)

## retained text



```
Hey all!  
I am Aritra from Hugging Face. I wanted to share an update on the \`tokenizers\` library that we have at Hugging Face. It has gone under major changes and we have finally released version 1 of it.

Here are what we are most excited about:

\> multiple language support  
\> multi-thread scaling  
\> minimal package size  
  
Read: [https://huggingface.co/blog/tokenizers-v1](https://huggingface.co/blog/tokenizers-v1)
```



document fetched 2026-09-22 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1wmfj04/tokenizers_v1_rust
