The CLEAR (Coalition for Language Equity in AI Research) consortium published the study “The Tokenizer Tax” on September 29, 2026: LLM tokenizers trained primarily on English and a dozen high-resource languages break down text in other languages into noticeably more tokens, and API tariffs charge by the token, so the same meaning in another language costs more. In measurements on the well-known FLORES-200 dataset of 203 language-script pairs, input in Amharic for GPT-5 costs approximately six times more than English, and the gap between tokenizers ranges from 2.7x for NLLB-200 to 17x for Falcon 3. CLEAR translated this surcharge from scattered observations into a public benchmark with a clear A–F rating scale and open code that anyone can run on their own languages.
What happened
CLEAR prepared a measurement-based, rather than review-based, work: they ran the tokenizers of major model families on identical text and calculated for each model how many times the most expensive language is more expensive than the cheapest in terms of token count. The base was the well-known FLORES-200 dataset with 203 language-script pairs, the metric was the max/min ratio of token counts plus the Gini coefficient, and ratings were assigned based on the ratio, from A for a gap up to 3x to F for a gap above 12x. A clear measurement: the phrase “People should benefit from AI, regardless of the language they speak” takes 13 tokens in English, 26 in Malayalam, and 78 in Amharic for GPT-5, because each character of the Ge'ez script breaks down into two single-byte tokens. According to the final scale, Meta's NLLB-200 received the only A rating with a ratio of 2.7x, the Gemini 3.8 Flash, Gemini 3.1 Pro, Gemma 3, and Gemma 4 families received a B rating with 5.1x, GPT-5 and gpt-oss received an F rating with 13.8x, and the worst result of about 17x was shown by Falcon 3. The reproduction code is posted in the clear-coalition-ai/token-cost-equity repository on GitHub, and to tie it to money, the study cites Anthropic Fable 5 tariffs: $10 per million input tokens and $50 per million output tokens.
Context
The gap is laid down at the tokenizer vocabulary assembly stage, not at the moment of the request. BPE tokenizers are trained on an English corpus and a dozen high-resource languages, and for characters for which there are no pre-prepared entries in the vocabulary, a byte-fallback remains: the character is replaced by its byte representation, causing rare scripts to pay for every character. The weight of the tax depends on how many languages are reflected in the vocabulary, which is separately examined in the publication with graphs of the tax's dependence on the number of languages. The sole holder of the A rating is also indicative: NLLB-200, with a vocabulary of about 256,000 tokens, was designed as a translation system, not a universal chat LLM. From this follows a careful conclusion: the problem is solved by the composition and size of the vocabulary, but there is currently no evidence that modern large models can expand the vocabulary without compromises on embedding dimensionality and computational cost.
Why this matters for the industry
For the industry, language stops being an invisible detail and becomes a measurable variable of unit economics: with a tariff per token, the language coefficient directly multiplies the bill for any multilingual product. CLEAR formulates specific requirements — to publish the max/min ratio on FLORES-200 and the share of languages with a ratio of less than 2x in model cards, to allocate vocabulary quotas for underused scripts, and to move to pricing normalized by characters or languages; the first provider to do this will set the standard and gain an argument in procurement. Practice is already changing: teams can run CLEAR's open code on their own multilingual traffic, calculate their own coefficients, and reassemble feature budgets, while redistributing “expensive” languages to B-class models by token ratio, such as Gemini 3.8 Flash or Gemma 4, instead of F-class models. The next step, indicated by the recommendations, is to include token efficiency by language in eval pipelines and in standard model comparison points during procurement.
Why this matters for users
If you call an LLM API in a language other than English, the language coefficient from the benchmark measurements directly affects your bill: the same prompt takes up more tokens, and the declared long context, for example 128,000 tokens, actually holds less text because expensive scripts consume the quota faster. For those building RAG and agent scenarios on non-English documents, this means that effective context length and pipeline cost depend on the language of the prompts, and budgets calculated on English data will be underestimated. Model choice also changes: in addition to quality and price per “average” token, a third criterion appears — token fairness — and the difference in grades between models turns into a direct difference in the bill, especially in budget tariffs where the price per token is critical. Your own language and traffic can be checked right now with the benchmark's open code, without waiting for vendor reports.
What is currently unknown / limitations
The figures of 13, 26, and 78 tokens are a measurement of one specific phrase, not statistics for the language: for budgets and pipelines, distributions by language and text type are needed, and the actual multiplier may differ significantly. The level of the effect for a specific language, including Russian, requires its own measurement on its own traffic using the open code: the published ratios were measured for the listed model versions and do not automatically transfer to all versions and tariffs. The only model with an A rating is a translation system, which shows the viability of the approach but does not prove that chat LLMs can expand the vocabulary without costs in embedding dimensionality and computation. Finally, the expected publication of language metrics in model cards and the shift to pricing by characters or languages is an interpretation of CLEAR's recommendations and a forecast, not announced vendor decisions.
Sources
- The Tokenizer Tax — CLEAR (Coalition for Language Equity in AI Research)
- The Tokenizer Tax: Why non-english AI users pay more for tokens — Rest of World
- clear-coalition-ai/token-cost-equity — benchmark reproduction code (GitHub)
Author
Look at AI, editorial team