loading
loading
People who use less widely spoken languages could end up paying more to use artificial intelligence than those who speak English or other dominant languages. A new study published last month by the Coalition for Language Equity in AI Research found AI models require far more tokens for some languages than others. Tokens are the units of text or data that the models process. The researchers have called this a hidden “tokenizer tax,” which can make AI more expensive in languages with smaller online presence. A tokenizer is a tool that breaks down raw text, images, or audio into tokens. Even within the 20 most popular languages on the internet, there is a gap in token usage, the research found. The most expensive of the languages used between 1.5 and 5 times more tokens than the cheapest. This gap widened when researchers expanded the set to 200 languages, with the token cost gap in one AI model growing by 17 times. “A startup in [Bengaluru] may pay a surcharge on every prompt and completion while its competitor in Austin does not,” the study said. “The tokenizer tax also has implications beyond cost. Higher token counts translate into higher latency and reduced effective context capacity. These penalties hit hardest where compute and connectivity are already limited. Tokenizer inequity is a constraint on who can participate in the AI economy.” Non-English languages leave less room for documents or conversation history, and require more work to generate the answer, according to Pratyush Choudhury, co-founder of Activate AI, India’s first venture capital fund dedicated exclusively to AI. Activate backs multilingual voice AI startups like ElevenLabs, Sarvam, and Wispr Flow. “Think about a tutor: Retaining the student’s earlier mistakes and explanations is part of delivering a good lesson. If that history consumes the context window much faster in one language, the disadvantage reaches into the quality of the service,” he said. One way to improve language processing is to improve the underlying data. “Open data efforts matter here, like RESPIN, a collaborative initiative with [the Indian Institute of Science] funded by the Gates Foundation, where we collected text from native speakers of 38 dialects along with read speech,” Raoul Nanavati, co-founder and CEO of Mumbai-based AI voice infrastructure startup Navana.ai, told Rest of World. However, the study by the Coalition for Language Equity in AI Research found the rarity of a language isn’t the only determinant of how many tokens it uses. The script plays a big role. Asturian, which has relatively few speakers and no Common Crawl data, “benefits from the Latin-script advantage, as well as the representation from other related Romance languages like Italian,” while languages written in scripts the tokenizer rarely saw — Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki — were at the greatest disadvantage consistently across nearly all models in the analysis. Many tokenizers today remain disproportionately optimized for English and other high-resource languages, and the underlying issue is the lack of high-quality digital data, said Dheemanth Reddy, CEO of Maya Research, which is developing conversational AI models designed to speak and respond like native speakers for the next 5 billion internet users. But better training data alone doesn’t fix the tokenizer, he warned. “Token prices are falling very quickly, so the direct cost impact should become less severe over time, and better multilingual and synthetic data should improve model quality. But actually fixing the tokenization gap will still require labs to deliberately design future tokenizers to handle these languages and scripts better,” Reddy said. Choudhury said language efficiency should be a product requirement, alongside accuracy, speed, and reliability. “Perfect equality may be unrealistic. Large, persistent disparities should still require an explanation and an improvement plan,” he said. “I would ask every major lab to publish a language-by-language scorecard showing what it costs to complete comparable tasks successfully, and commit to measurable improvements for its worst-served languages.”
document fetched 2026-10-27 02:56 · canonical https://restofworld.org/2026/ai-tokenizer-tax-non-english-languages-cost-inequity