Why Czech Without Diacritics Costs More Tokens, Not Fewer

The article refutes the intuitive assumption that Czech written without háčky and čárky (diacritical marks) saves tokens when processed by language models. On the contrary — text without diacritics consumes significantly more tokens (up to 67% more), because tokenizers (BPE) were trained predominantly on high-quality Czech texts with diacritics and have no efficient merges for the diacritic-free variants. Correct Czech with diacritics is thus paradoxically cheaper to process and more accurate for the model, and the authors recommend prepending a diacritics-restoration module where appropriate.

The phrase "Během výběrů" takes up three tokens in a tokenizer. The same phrase without diacritics — "Behem vyberu" — consumes more tokens. Not three, not two. Five.

A 67% increase. For text that was supposed to be "simpler."

In the previous article on the token tax, we showed that Czech pays double for processing in language models compared to English. The logical line of thinking for many readers was: if the háčky and čárky are the problem, why not write without them? After all, the SMS generation did it for years, surely the models can handle it, and we'll save tokens. The answer is unexpected — and it reveals something essential about how language models actually work.

The key evidence comes from the work of Kirill Semenov and Martin Popel of Charles University and the University of Zurich, presented at the TokShop workshop at the ICML 2025 conference in Vancouver. The authors were developing the InDia (Inline Diacritization) system for Czech–Ukrainian machine translation and in the process ran into a paradox that surprised them enough that they documented it as a separate research finding.

The experiment was straightforward. They took Czech phrases and compared how many tokens they require in different forms:

| Text form | Example | Token count | Increase |

|---|---|---|---|

| With diacritics | Během výběrů | 3 | — |

| Without diacritics | Behem vyberu | 5 | +67% |

| In capital letters | BĚHEM VÝBĚRŮ | 10 | +233% |

The authors explicitly note that in their literature review they found no prior research focused on comparing tokenization with and without diacritics. Their finding is a by-product of other research — not the goal of a systematic study.

The explanation is surprisingly mechanical. A tokenizer based on Byte-Pair Encoding (BPE) — the algorithm that converts text into numbers for a neural network — learns from training data. It goes through enormous corpora of text and looks for recurring byte sequences. The most frequent ones it merges into a single token. It repeats the process until it reaches the target vocabulary size.

The key detail: most high-quality Czech text on the internet — news articles, Wikipedia, scholarly literature, legislative texts — is written with diacritics. The word "výběrů" occurs a thousand times in the training data. The tokenizer knows it. It created an efficient merge for it — perhaps the whole word as a single token, perhaps two parts.

But "vyberu" without the háčky and čárky? That is a different byte sequence for the tokenizer. It occurs orders of magnitude less often in the training data — mostly in informal communication, chats, and text messages. The tokenizer never created an efficient merge for this form. When it encounters it, it breaks it into smaller pieces — sometimes all the way down to individual bytes.

It is the same mechanism we described for the token tax in general: words the tokenizer has seen often, it encodes efficiently. Words it has seen rarely, it breaks into fragments. Czech without diacritics is essentially a foreign dialect to the model — similar to proper Czech, but different enough that the merges are not shared.

A technical counterargument presents itself: Czech diacritical characters (č, ř, š, ž, ě, ů) are two bytes in UTF-8 encoding, whereas their plain counterparts (c, r, s, z, e, u) are one byte. At the level of raw bytes, text without diacritics is shorter. Shouldn't it therefore need fewer tokens?

No. And the reason illustrates a fundamental difference between compression at the byte level and compression at the tokenizer level.

A BPE tokenizer does not work directly with bytes — it works with learned merges. The two-byte character "č" (0xC4 0x8D in UTF-8) can be part of a token "čes" or "výběrů" that stands in for a whole word or a large part of it. The single-byte "c" does take up less space in the raw text, but the tokenizer has no efficient merge for the combination "ces" or "vyberu" — and so it breaks it into more pieces.

An analogy: imagine you have a dictionary of abbreviations for corporate communication. "KPI" is a single abbreviation, everyone knows it. If instead of "KPI" I wrote "key performance indicator," it takes up more space — even though each individual letter is "simpler" than the abbreviation. The tokenizer works similarly: it knows "výběrů" as an abbreviation, but it has to spell out "vyberu."

It is important to distinguish Czech diacritical characters from the problems tokenizers face with other languages. In Czech, characters such as č, ř, š, ž are represented in Unicode as precomposed characters — single code points (č = U+010D, ř = U+0159). This means that in the normalized NFC form they behave like ordinary letters.

A different situation arises for languages where diacritics are written as combining marks — separate characters added after the base letter. The Vietnamese "ệ" can be composed of three code points (e + circumflex + dot below the letter). Older-generation tokenizers (cl100k_base in GPT-4) used the pattern \p{L} for letters in the pre-tokenization regular expression, which did not include combining marks — and so they could separate them from the base character.

The newer tokenizer o200k_base (GPT-4o) addresses this problem by extending the regular expression with the \p{M} category for combining marks. For Czech, this fix is less significant — Czech characters do not decompose in NFC. But the principle is important: newer tokenizers are generally more accommodating toward diacritics.

The previous article cited data from the cl100k_base tokenizer — the one used by GPT-4 and GPT-3.5. Since then GPT-4o has arrived with the o200k_base tokenizer, which has twice the vocabulary (200,000 versus 100,000 tokens). How does this play out?

A study of tokenization efficiency for Ukrainian (Maksymenko and Turuta, Frontiers in AI, 2025) documents a marked drop in fertility — the average number of tokens per word — when moving to newer models. Ukrainian, another Slavic language suffering from tokenization inefficiency (in its case due to Cyrillic), saw a marked improvement with larger-vocabulary models compared to older generations.

The Occiglot project measured fertility for Czech across tokenizers on Wikipedia data:

| Tokenizer (vocabulary size) | Czech | English | Ratio |

|---|---|---|---|

| Llama-2 (32,000) | 2.315 | 1.429 | 1.62× |

| Mistral (32,000) | 2.487 | 1.424 | 1.75× |

| Phi-2 (51,000) | 3.046 | 1.502 | 2.03× |

| Gemma (256,000) | 2.063 | 1.283 | 1.61× |

The pattern is clear: a larger vocabulary generally means lower fertility for Czech. Gemma, with 256,000 tokens in its vocabulary, achieves a fertility of 2.063 for Czech — meaning that the average Czech word takes up roughly two tokens. Compared to Phi-2 with a smaller vocabulary (fertility 3.046), that is a one-third saving.

So the token tax is shrinking — but it is not disappearing. Even the most efficient large vocabularies show a surcharge of around 60% for Czech compared to English.

Tokenization efficiency is half the problem. The other half: if I write without diacritics, does the model understand me just as well?

The answer is surprisingly nuanced. Náplava, Straka, and Straková of Charles University (2021) achieved 99.22% accuracy in automatically restoring diacritics from text without háčky using a BERT model. This proves that the contextual information in Czech text without diacritics is almost always sufficient for unambiguous reconstruction — the model "knows" where the háčky belong. And what is more: after manual annotation of all the supposed errors, it turned out that 44% of them were in fact not errors — they were either acceptable variants (19%) or cases where the model corrected an error in the original data (25%).

Today's large language models will do even better, because they have seen an enormous amount of Czech informal communication written without diacritics in their training data. For ordinary conversation — questions, instructions, discussions — the loss of comprehension is minimal.

But there are traps. Czech homonym pairs, where diacritics distinguish meaning, are a genuine risk:

In ordinary conversation, context usually resolves the correct meaning. But in specialized tasks — legal texts, named-entity recognition, precise instructions — missing diacritics can lead to a silent error that the user is not aware of.

A study of the robustness of language models to noisy inputs (Chrabąszcz et al., arXiv, 2025), testing Polish data, shows that models handle removed diacritics better than transposed letters or missing words. Diacritic perturbations are among the less destructive types of noise. But "less destructive" does not mean "without impact" — on noisy inputs, models measurably lose accuracy.

For the ordinary user of ChatGPT, Claude, or another model, the conclusions are clear.

Write with diacritics. Not only because it is correct Czech. But because the model will process your text more efficiently — in fewer tokens, with better comprehension, with more accurate answers. Paradoxically: correct Czech is cheaper Czech.

If you cannot write with diacritics (a mobile phone without a Czech keyboard, a foreign device), the model will probably understand you. But you will pay more tokens for the same content and you risk silent errors on specialized tasks. For a quick question, it doesn't matter. For generating a legal document or analyzing data, it can matter a great deal.

For developers deploying language models in a Czech-language environment, the consequence is systemic. If your processing pipeline accepts user inputs without diacritics (from chatbots, text messages, forms), consider placing a diacritics-restoration model in front of it — adding the háčky and čárky by pre-sending the text through a language model. Research from Charles University shows that 99%+ accuracy is achievable. The investment in this step pays for itself through lower token costs and higher-quality answers.

It is remarkable that a systematic study comparing the tokenization of Czech with and without diacritics does not yet exist. The authors of the work on the InDia system state this explicitly. There is partial evidence — examples such as "Během výběrů" versus "Behem vyberu," general studies of tokenization inequality, comparative fertility tests for European languages. But no one has yet taken a Czech corpus of tens of thousands of sentences, stripped the diacritics, and compared the token count across six major tokenizers.

Such an experiment would be feasible with a single Python script in an afternoon. Use the Czech sub-corpus of FLORES-200, create a diacritics-free variant of each sentence, tokenize both ways via tiktoken (cl100k_base, o200k_base), the HuggingFace tokenizers (LLaMA, Mistral, Gemma), and evaluate statistically. The results would have practical impact — not only for academic research, but for anyone running a language-model-based service for Czech users.

If someone carries out this study, the results will probably confirm what the partial evidence suggests: Czech without diacritics is, paradoxically, more expensive to process. The háčky and čárky are not a luxury. They are a compressor.

Key sources: Semenov & Popel, "InCa and InDia: Inline Casing and Diacritization Preprocessing" (TokShop @ ICML 2025, Charles University & University of Zurich); the Occiglot project, "Tokenizer Evaluation on European Languages" (2024); Beey.ai / NEWTON Technologies, tokenization analysis (2025); Náplava, Straka & Straková, "Diacritics Restoration using BERT with Analysis on Czech Language" (PBML, Charles University, 2021); Petrov et al., "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023); Maksymenko & Turuta, "Tokenization efficiency of current foundational LLMs for the Ukrainian language" (Frontiers in AI, 2025); Chrabąszcz et al., "Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models" (arXiv:2506.07645, 2025).

Transparency of creation:

The concept, structure, and editorial line of the article are the work of the author, who prepared the content outline, set out the key theses, and directed the entire creation process. Generative AI (Claude, Anthropic) was used as a technical tool for research, fact-checking, and fleshing out the author's draft.

The author edited the outputs throughout, verified the key findings, and approved the final wording. No part of the text was published without human review. All factual data were verified against the publicly available sources cited in the text.

This procedure complies with the requirements of Article 50 of EU Regulation 2024/1689 (AI Act) on the transparency of AI-generated content. #poweredByAI