The Token Tax: Why AI Speaks Czech Worse and Pricier

The article analyzes the structural disadvantage Czech faces when processed by large language models. Because tokenization is optimized for English, processing Czech text costs twice as much, the context window effectively shrinks by half, and answer quality drops. It covers Czech benchmarks (CzechBench, BenCzechMark), possible solutions (RAG, fine-tuning, better tokenizers), and the strategic dimension of the problem for the Czech market.

Analysis — February 2026

The same sentence. The same model. Twice the price. Why Czech pays a hidden surcharge in the age of artificial intelligence — and what to do about it.

The Preamble to the Universal Declaration of Human Rights is 320 words long in English. In Czech, 248. A shorter text, fewer words — and yet, when you run it through the tokenizer of the GPT-4 model (cl100k_base), Czech consumes 724 tokens versus the English 360. Twice as many. For a shorter text.

A team from the Czech company NEWTON Technologies (which runs the Beey.ai platform) carried out this experiment systematically across languages. The results revealed a structural inequality built into the very architecture of language models: Spanish costs 1.5× more than English, Czech 2×, Ukrainian 3×, and Greek almost 6×. And that is just for tokenization — for converting text into the format the model understands.

Czech is among the languages that large language models process more expensively, more slowly, and with worse results. It is not the fault of a single vendor or a single model. It is a consequence of the architecture of the entire ecosystem — from tokenization through training data to benchmarks. For ten million Czechs this has concrete consequences that reach far beyond the cost of an API call.

When you type the English word beautiful into ChatGPT, the model processes it as a single token. The Turkish güzel (the same word) needs three tokens. The Czech term překrásný requires at least two — the exact count depends on the given model's tokenizer. Why?

Large language models do not see words. They see tokens — substrings created by an algorithm called Byte-Pair Encoding (BPE). This algorithm learns from training data that has historically been more than ninety percent English (over 92% for GPT-3, almost 90% for LLaMA 2; newer models reduce this share but do not disclose exact figures). English words are therefore encoded efficiently — often entirely as a single token. Words in other languages have to be broken into smaller pieces, sometimes down to the level of individual characters.

Czech morphology makes this problem worse. Czech has seven cases, three genders, and rich conjugation and derivation. The word nepřekračovat is an ordinary form for a native speaker. For a BPE tokenizer trained on English it is a sequence of substrings, each of which consumes model capacity. The 2023 study by Petrov et al. systematically mapped this "tokenization premium" across dozens of languages and tokenizers — for Anglocentric models it found a premium on the order of tens of percent for Western European languages, while for languages with non-Latin scripts the difference can be more than tenfold.

The consequences are not only financial. Every model has a limited context window — the maximum number of tokens it can process at once. If Czech consumes twice as many tokens for the same content, the effective capacity of the context window shrinks by half. A Czech user can insert less text into the prompt, gets a shorter answer, and the model has less room to reason. You pay more for less.

Tokenization is only the first layer of the problem. The real impact shows up in the quality of the answers.

In 2024 and 2025, two independent benchmarks were created in the Czech Republic specifically designed to measure language-model performance in Czech. CzechBench, developed at CIIRC CTU and VŠB Ostrava, contains fifteen tasks — a combination of translated international benchmarks (ARC, GSM8K, MMLU, TruthfulQA) and original Czech datasets. It allows direct comparison of the same model's performance in English and Czech on parallel tasks.

BenCzechMark, created by teams from BUT Brno, Masaryk University, CTU, the University of Ostrava, and Hugging Face, is more ambitious — fifty tasks in eight categories, with ninety percent of the content being natively Czech rather than translated. The tests cover logical reasoning, grammar, factual knowledge about Czech culture, and the ability to estimate the probability of Czech texts.

The results of both benchmarks confirm what everyone who uses the models in Czech daily intuitively senses: performance drops. The 2025 MMLU-ProX study, which tested 36 models in 29 languages on identical questions, found declines of up to 24.3 percentage points between high-resource and low-resource languages.

And it is not only about accuracy. Research presented at the ICAIR 2025 conference examined, on a sample of 48 paired prompts from the field of elementary geometry, whether models generate shorter and less detailed answers in Czech than in English. The result: in the tested domain, English answers were systematically longer and more detailed, with a consistently medium-to-large effect (Cohen's d). Interestingly, an earlier study by the same team (ECEL 2025) on a task of recognizing geometric properties found the opposite trend — the Czech answers were more detailed. The overall picture thus suggests that a language bias in the length and detail of outputs exists, but its direction and magnitude vary by domain and task type. Broader benchmarks do not yet measure this effect systematically.

The team from AAAI 2025 (Language Ranker) explains it through a direct correlation: a model's performance in a given language correlates strongly with that language's share in the training data. The less data, the worse the performance — and Czech, with ten million speakers and limited digital content, falls into the middle category. Not as bad as Swahili, but far behind English, Chinese, or Spanish.

There are roughly seven thousand languages in the world. The largest multilingual models cover fewer than a hundred. Czech is indeed among those supported, but its representation in the training data is a fraction of a percent — vendors do not disclose exact figures, but reverse analyses of tokenizers (which reflect the composition of the training corpora) suggest something on the order of low single-digit percentages.

The Stanford HAI report from April 2025 names this problem the "digital divide" of language-model development. There are two main causes: a lack of quality data and a lack of data altogether. Even languages with large speaker communities can be "low-resource" if their digital footprint is small — Vietnamese, with almost a hundred million speakers, is an example.

For Czech, several specific factors complicate the situation. Morphological richness means that the same information requires more distinct word forms, and thus a larger vocabulary. A relatively small market limits the commercial incentive to invest in Czech language technologies. And most NLP research focused on small languages takes place in institutions in countries where those languages are not spoken — which leads to the cultural context and local specifics being overlooked.

Cross-lingual transfer — a model's ability to carry knowledge from one language to another — works, but imperfectly. A model trained predominantly on English data can answer in Czech because the shared latent representation captures semantic relationships across languages. But nuances are lost. Word order, idioms, cultural references, pragmatics — all of these suffer. The model knows what to say. It does not always know how to say it naturally in Czech.

Solutions exist, but each has its limitations.

Massively multilingual models such as Cohere's Aya (101 languages) or BLOOM try to cover as many languages as possible in a single model. The advantage is the sharing of knowledge between related languages — Czech benefits from Slovak, Polish, and partly German. The disadvantage is the trade-off between breadth and depth: a model that knows a hundred languages knows none of them as well as a specialized model.

Monolingual fine-tuning — further training of an existing model on Czech data — brings measurable improvement, but hits a data ceiling. A Vietnamese team, in collaboration with Stanford, fine-tuned five LLMs specialized in Vietnamese and achieved competitive results. For languages with limited data, however, the authors caution that multilingual models sometimes outperform monolingual fine-tuning — simply because fine-tuning has nothing to draw on.

Retrieval-Augmented Generation (RAG) offers a pragmatic compensation. Instead of relying on what the model "knows" from training, the system retrieves relevant Czech documents and presents them to the model as context. The quality of the answers then depends primarily on the quality of retrieval and indexing, not on Czech's representation in the training data. Multilingual embedding models such as BGE-M3, which can encode texts in more than a hundred languages into a shared vector space, make this path easier.

National benchmarks — CzechBench and BenCzechMark — are infrastructure without which progress cannot be measured. Without them, Czech AI communities would not know whether a new model is better or worse for Czech. With them, developers can optimize in a targeted way.

Tokenization itself is also gradually improving. Newer tokenizers (such as o200k_base in GPT-4o) offer significantly better multilingual compression. Research on byte-level architectures (e.g. Meta's Byte Latent Transformer) promises to remove the tokenization inequality on principle, even though these approaches are not yet common in production models. In the Czech Republic, Seznam.cz chose the path of continued pre-training — further training of open-source models (Mistral, LLaMA) on a large volume of clean Czech data, thereby compensating for their low representation of Czech in the original pre-training.

In the Czech context, the language divide paradoxically also has a strategic dimension. In the search market, two approaches meet: global models optimized for English (Google), and local players who understand the Czech context (Seznam). Whoever can bridge the language divide better — whether through a specialized RAG system, better tokenization, or more native embeddings — gains a competitive advantage that a global player can hardly replicate.

A similar logic applies to Czech companies deploying AI into customer service, document analysis, or automation. A model that hallucinates more often in Czech, generates shorter answers, and costs twice as much is not a neutral tool — it is a tool with an English accent. Companies that recognize and compensate for this asymmetry (with quality RAG, fine-tuning, evaluation on Czech benchmarks) will have a head start over the rest.

NLP research in the Czech Republic — at BUT Brno, CTU, Masaryk University, the University of Ostrava, Charles University — produces high-quality output with limited resources. BenCzechMark, with its fifty tasks and predominantly native content, is at a level comparable to benchmarks of much larger language communities. What is missing is not capability — it is investment in infrastructure, compute, and data that would make it possible to move from measuring the problem to solving it.

A large language model does not see language. It sees tokens — numerical codes derived from training data in which English forms the majority. The quality of the answers, the length of the outputs, the accuracy of the facts, the cost of processing — all of this changes according to the language you address the model in. Not because Czech is more complex (it is, but that need not be a disadvantage). Rather, because the architecture of the entire ecosystem is optimized for one language and the rest are along for the ride.

The Preamble to the Universal Declaration of Human Rights declares that all human beings are born free and equal. For processing the Czech version of this declaration you will pay twice as much as for the English one. So far, equality applies only to people — not to their languages.

Key sources: Beey.ai / NEWTON Technologies (CZ vs. EN tokenization analysis, 2025); Stanford HAI "Mind the Language Gap" (2025); BenCzechMark (Fajčík et al. 2024, BUT Brno + MU + CTU + UO + Hugging Face, 50 tasks); CzechBench (CIIRC CTU + VŠB Ostrava, 15 tasks); MMLU-ProX (Xuan et al. 2025, 29 languages, 36 models); ICAIR 2025 "English-Czech Output Bias in LLMs" (Tichá, Přibyl, Krátká, UJEP); Language Ranker (Li et al., AAAI 2025); Petrov et al. "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023); Cohere "The AI Language Gap" (policy primer 2024).

Methodological note

The concept, structure, and editorial line of the article are the work of the author, who prepared the content outline, established the key theses, and directed the entire creative process. Generative AI (Claude, Anthropic) was used as a technical tool for research, fact-checking, and expanding the author's draft.

The author edited the outputs continuously, verified the key findings, and approved the final wording. No part of the text was published without human review. All factual data were verified against the publicly available sources cited in the text.

This procedure complies with the requirements of Article 50 of EU Regulation 2024/1689 (the AI Act) on transparency of AI-generated content. #poweredByAI