The Language Tax: How Much a Non-English Query to a Large Language Model Costs

The latest generation of large language models has dramatically narrowed the gap between English and the major European languages, but for hundreds of smaller languages the divide is instead widening. The article analyzes tokenization inefficiency, price discrimination against non-English users, and the surprising finding that even the most advanced models of 2025 internally think in English regardless of the input language.

When a Czech developer poses the same difficult question to the Qwen3-235B-Think model in English and in Czech, they get an answer of almost identical quality — the drop is 0.2 percentage points. The same question in Yoruba means a loss of over 31 points. The latest generation of large language models has dramatically narrowed the gap between English and the major European languages, but for hundreds of smaller languages the gap is, on the contrary, widening. And a surprising process is happening in the background: even the most advanced models of 2025 internally "think" in English, even when you ask them in Czech.

This analysis summarizes the current state of research on the multilingual performance of large language models, based on benchmarks and studies published predominantly in 2024–2025. It focuses on two key questions: how the language of the query affects the quality of the answer, and how the language of the reasoning chain affects the quality of the result. Both dimensions are critical for anyone deploying language models in a non-English environment — from a Czech search engine to an African education platform.

The most comprehensive current tool for measuring multilingual performance is the MMLU-ProX evaluation suite (Xuan et al., 2025), published in spring 2025 and accepted at the EMNLP conference. In its expanded May 2025 version it covers 29 typologically diverse languages, contains 11,829 questions in each of them, and unlike standard MMLU it offers 10 possible answers instead of four, which requires deeper reasoning and minimizes the effect of random guessing. Thirty-six top models were tested.

The results paint a clear hierarchy. At the top stands Qwen3-235B-Think (Alibaba, May 2025) with 80.7% on English. For Czech it reaches 80.5% — a practically indistinguishable result. Similarly minimal drops are shown by French (80.6%), Spanish (80.7%) and Italian (80.9%, paradoxically above English). At the opposite end of the spectrum stands Wolof with 36.9% — an absolute drop of 43.8 points.

GPT-4.1 (OpenAI, April 2025) reaches 79.8% on English, 77.5% on Czech (a 2.3-point drop), 75.5% on Chinese and 53.4% on Yoruba. DeepSeek-R1 (January 2025) shows 79.5% on English, 76.8% on Czech (a 2.7-point drop) and 58.6% on Wolof. On a different, simpler evaluation suite, MMMLU (multilingual MMLU with four answers), Gemini 3.1 Pro (Google, February 2026) leads with a score of 92.6% — the figures from the two suites cannot be compared directly because of their differing difficulty.

Crucially, model size significantly affects the extent of language degradation. While large models of the Qwen3-235B type show drops below 3 points for Czech, the smaller DeepSeek-R1-7B model reaches 46.0% on English but a mere 10.8% on Swahili — a 35.2-point drop. Small models simply cannot afford to "waste" capacity on languages that make up a fraction of the training data.

Other evaluation suites confirm this pattern. MGSM (mathematical word problems in 10 languages) consistently shows the lowest performance for Bengali. Belebele from Meta (122 language variants) demonstrated that performance correlates primarily with the volume of data in the pretraining phase. BenchMAX (2025, an extension of GPQA to 16 languages) showed that DeepSeek-V3 reaches over 50% on English and French scientific reasoning, but below 40% for Telugu.

The most significant finding of the research is that task complexity exponentially amplifies language differences. On standard MMLU, where it is enough to choose from 4 answers, GPT-4 already in 2023 surpassed the English performance of GPT-3.5 in 24 of the 26 tested languages — including Latvian, Welsh and Swahili. That looked promising. But MMLU-Pro, a harder variant with 10 answers, revealed markedly larger drops. And in combination with a non-English language on MMLU-ProX, the drops grew to 30 and more percentage points for low-resource languages.

The MLRBench evaluation suite (2025), focused on multilingual reasoning over long contexts, confirmed that degradation is especially pronounced on tasks requiring multi-step inference. Short factual queries work comparably across languages; difficult reasoning problems reveal gaping differences.

Direct research on the effect of query length in a multilingual context has not yet been published. Cross-lingual retrieval research nonetheless suggests that short, ambiguous queries are more problematic in a non-English environment, because the model lacks sufficient context to determine the correct meaning.

Three independent studies from 2024–2025 converge on the same conclusion: models with a predominance of English training data process information through English representations even with non-English input and output.

The study "Do Llamas Work in English?" (Wendler et al., ACL 2024, EPFL) identified three processing phases in the Llama-2 model: the input space captures surface features of the input language, the middle layers (the conceptual space) assign higher probability to English tokens, and the output layers translate the result into the target language.

"Do Multilingual LLMs Think In English?" (Schut, Gal and Farquhar, ICLR 2025, Oxford/DeepMind) extended the analysis to current models — Llama-3.1-70B, Mixtral-8×22B, Gemma-2-27B and Aya-23-35B — and tested them on French, German, Dutch and Mandarin. It found that the models first generate representations close to English for semantically loaded words (nouns, verbs), and only then translate them into the target language. Grammatical words (articles, prepositions) do not pass through this English routing. A practically important finding: activation steering is more effective when the vectors are computed in English — even with purely non-English input and output.

The degree of English routing depends on the composition of the training data. Aya-23-35B (trained on 23 languages) was the least Anglocentric, Gemma-2-27B (predominantly English corpus) the most. A separate study on Japanese models (Zhong et al., 2024) confirmed that LLM-jp (a balanced Japanese-English corpus) primarily uses Japanese as its internal language, whereas Llama-2 relies exclusively on English when processing Japanese.

In practice this means: even when a model answers in Czech, its "reasoning" takes place in the English conceptual space. The transition to the target language in the output layers is a potential source of degradation, especially for languages with specific morphology or syntax.

The key study Shi et al. (ICLR 2023) on the MGSM evaluation suite demonstrated that an English chain of thought (CoT) — reasoning in English even on a non-English query — consistently achieves comparable or better results than a native chain of thought. Both approaches massively outperformed direct answering without reasoning steps by 20 to 40 and more percentage points. This was on the PaLM-540B model.

Newer research from 2025, however, brings nuances specific to current models. For French, an English intermediary language brings no improvement. For Japanese and Latvian, the models process the input in the native language, but for optimal performance they need English as the reasoning language. For Swahili, the English intermediary language fails completely — the model has trouble both with comprehension and with reasoning.

The most interesting phenomenon is the spontaneous mixing of languages in reasoning models. DeepSeek-R1 explicitly admits that with a non-English query it may switch to English or Chinese during the reasoning chain. A 2025 study demonstrated that stronger reasoning correlates with increased language mixing. An attempt to suppress this behavior by introducing a reward for language consistency during training paradoxically worsened performance. On the contrary: a trained probe for strategic language mixing brought improvement on mathematical tasks.

The counterintuitive study "Could Thinking Multilingually Empower LLM Reasoning?" (2025) found that multilingual reasoning provides roughly 10 points higher performance upper bound than purely English reasoning. But standard selection methods (majority voting) cannot exploit this potential. Language mixing is thus a double-edged sword: it improves the diversity of reasoning paths, but complicates the selection of the best answer.

On the basis of MMLU-ProX, Czech is among the best-served non-English languages. Top models show drops below 3 percentage points. This places it on the level of the major Romance and Germanic languages. A more detailed view through Czech evaluation suites, however, reveals subtler problems.

BenCzechMark (published in TACL, MIT Press, 2025) is the most comprehensive Czech evaluation set, with 50 tasks in 8 categories. Crucially, 90% of the content is natively Czech, not translated. Dozens of models appear in the rankings. In the overall evaluation, Llama-450B (Meta) leads. In individual categories, Qwen-72B dominates (mathematics, information retrieval), Aya-23-35B (sentiment, language modeling), and surprisingly Gemma-2-9B outperforms much larger models in comprehension of Czech reading text.

MiniCzechBenchmark (2025), which achieves a correlation higher than 0.996 with the full benchmarks, identified 10–30% performance drops between English and Czech. The largest differences are shown by mathematics (where contamination by English data is suspected). DeepSeek-R1 reaches 78% on mathematics but degrades on Czech grammatical tasks. The most important finding: models can correctly answer Czech multiple-choice questions, but produce markedly worse free generation of Czech text.

A test on the Czech bar exam (1,840 questions, HAVEL & PARTNERS, 2024) showed that no standalone language model exceeded the 85% threshold needed to pass the written test. Claude-3-Opus performed best, closely followed by GPT-4-Turbo. A system with retrieval augmentation (RAG — GPT-4 + legal sources), however, passed all simulated attempts.

Currently the strongest multilingual model on MMLU-ProX. Qwen3-235B-Think shows almost zero drop for European languages compared to English. It declares support for 119 languages and dialects. Strong in Chinese and English simultaneously, which shows that balanced bilingual pretraining pays off.

A 2.3-point drop for Czech on MMLU-ProX. GPT-4o declares support for 50 and more languages covering 97% of the world's speakers. On multilingual MMLU, however, a 25% gap persists between English and Yoruba. Compared to GPT-4 (2023) the shift is noticeable: GPT-4 at the time surpassed the English performance of GPT-3.5 on standard MMLU in 24 of 26 languages, but on harder tasks the drops remained dramatic. GPT-4.1 markedly mitigated these drops for high-resource languages.

A 2.7-point drop for Czech. Primarily optimized for English and Chinese. A specific phenomenon: spontaneous language mixing during reasoning as an emergent behavior of reinforcement learning. On Wolof 58.6% — markedly above smaller models, but still 20.9 points below English.

The newest model in the analysis. Leads on the multilingual MMMLU evaluation suite with 92.6% (caution: a different and simpler test than MMLU-ProX, the results cannot be compared directly). Compared to Gemini 1.5, which brought a marked improvement in multilingual performance over version 1.0, it represents another generational leap.

Internal evaluation shows accuracy in expanded languages at the level of 95–98% of English performance. Claude 3 Opus reached over 90% on MGSM in more than 8 languages. For Czech it shows high grammatical quality of generated text among commercial models.

Officially supports only 8 languages. The training corpus of Llama-2 contained 89.7% English. Nevertheless, the latest Llama-450B leads overall on BenCzechMark. The shift from Llama-2 through Llama-3 to the latest variant shows that expanding multilingual data in training brings marked gains even while preserving English dominance.

Mistral Large 2 outperforms Llama 3.1 70B by an average of 6.3% across nine languages on multilingual MMLU. As the only significant European player, it invests in multilingual performance as a strategic advantage over American and Chinese competitors.

Beyond the quality of answers there is an economic dimension to the language gap. Analyses of tokenization systems have quantified the surcharge for non-English languages: a Spanish technical paragraph requires tens of percent more tokens than its English version. Portuguese and German show comparable surcharges. For languages with non-Latin scripts the difference can be even more dramatic — for some Southeast Asian languages the tokenizer consumes an order of magnitude more tokens for the same content.

This effect translates directly into API costs (you pay per token) and into processing speed. A query formulated in Czech is therefore not only potentially answered less accurately, but is also more expensive to process.

The overall picture of language degradation deserves three counter-arguments.

First, drops in benchmarks need not correspond to the real user experience. MMLU-ProX tests academic knowledge questions. Ordinary conversation, writing emails or simple programming — that is, the typical tasks of most users — may show markedly smaller drops. Studies focused on free text generation in non-English languages are still rare.

Second, the trend is toward rapid improvement. Between GPT-3.5 (2022) and Qwen3-235B (2025) the drop for Czech shrank from tens of percentage points to fractions. Extrapolating this trend suggests that for high-resource languages the language tax may fall to a statistically insignificant level within the next two years.

Third, the finding about internal "English thinking" need not be a problem, but a feature. If a model effectively uses the English conceptual space for reasoning and translates well into the target language, the result can be just as good as native processing. The MMLU-ProX data for Czech support this interpretation.

Nevertheless, it remains valid that difficult reasoning tasks in low-resource languages are demonstrably worse. Free text generation lags behind multiple-choice questions. And the tokenization surcharge is a real economic burden that no algorithmic progress eliminates — it requires retraining the tokenizer.

For developers deploying language models in a Czech environment the message is mostly positive: on standard evaluation suites Czech is among the best-served non-English languages, with minimal drops. The problem arises in two areas — free text generation (the 10–30% drops identified in MiniCzechBenchmark) and domain-specific reasoning, where even the best model did not pass the Czech bar exam without retrieval augmentation.

The optimal strategy depends on the type of task. For simple queries and conversation, Czech input is fully adequate. For difficult reasoning tasks — mathematics, logic, complex analyses — English input with subsequent translation probably remains more reliable. For reasoning models of the DeepSeek-R1 type it is paradoxically best to let the model decide the reasoning language itself, because suppressing language mixing worsens performance.

Research on the multilingual performance of large language models is an active field. Studies focused specifically on short, ambiguous queries in a multilingual environment, and on systematic comparison of free-text-generation quality across languages, are still missing. The mechanism by which internal English processing degrades output in morphologically rich languages such as Czech also remains an open question.

Methodological note: This analysis draws on academic studies and evaluation suites published in 2023–2026, with an emphasis on data from 2025. Main sources: MMLU-ProX (Xuan et al., EMNLP 2025), BenCzechMark (TACL/MIT Press, 2025), MiniCzechBenchmark (2025), "Do Multilingual LLMs Think In English?" (Schut et al., ICLR 2025), "Do Llamas Work in English?" (Wendler et al., ACL 2024), MGSM (Shi et al., ICLR 2023). Scores from the evaluation suites refer to models available as of February 2026. Limitations: the evaluation suites test predominantly multiple-choice questions, not free generation; the figures for the newest models (Gemini 3.1 Pro) come from a different suite (MMMLU) than the other results discussed here; for specifically Czech sub-results the current BenCzechMark ranking should be consulted.

Creation transparency:

The concept, structure and editorial line of the article are the work of the author, who prepared the content outline, established the key theses and directed the entire creation process. Generative AI (Claude, Anthropic) was used as a technical tool for research, fact-checking and fleshing out the author's draft.

The author edited the outputs continuously, verified the key findings and approved the final wording. No part of the text was published without human review. All factual data were verified against the publicly available sources cited in the text.

The procedure complies with the requirements of Art. 50 of EU Regulation 2024/1689 (the AI Act) on the transparency of AI-generated content. #poweredByAI