Tell the Model to Think in English: A Technique That Can Improve Results by Up to 10 Points

Research from Oxford and EPFL has shown that large language models (Llama, Gemma, Mixtral) internally think in English even when processing non-English inputs — key concepts first appear in English in the network's middle layers and are only translated into the target language in the final layers. For Czech users this means worse reasoning performance, cultural bias, and more terse answers; techniques such as cross-lingual prompting (XLT) or dictionary insertion can improve results by up to 10 points, but the real solution lies in models trained on sufficient Czech data, such as the upcoming OpenEuroLLM.

Give a language model the sentence "Le bateau naviguait en douceur sur l'" and watch what happens inside. In February 2025, the research team of Lisa Schut at Oxford applied to the Llama-3.1-70B model a technique called the logit lens — a method that makes it possible to peer into the intermediate layers of a neural network and decode which word the model is "considering" at a given moment. The result: when the model was generating French text about a boat sailing on water, in the middle layers the key nouns — "eau" (water), "lac" (lake), "soleil" (sun) — first appeared in English. Only in the final layers did the model translate them into French.

The model received its input in French. It answered in French. But in between, it thought in English.

This article explains why that is so, what it means for ten million Czech speakers, and what techniques exist for working with it — and also why none of them is a real solution.

This mechanism was first systematically described by Chris Wendler and colleagues from Switzerland's EPFL in 2024 in the paper "Do Llamas Work in English?" (arXiv:2402.10588). By analyzing models of the Llama-2 family using the logit lens, they identified three phases in the processing of non-English input.

In the first phase — the "input space" — the model processes linguistic units in the language of the input. In this phase the representations are ambiguous, far from any specific output word. In the second phase — the "concept space" — the semantically correct answer emerges in the middle layers, but with a higher probability for its English version than for the version in the input language. In the third phase — the "output space" — the representations finally shift into the region specific to the target language, and the model generates the answer in the requested language.

The key conclusion: the abstract "concept space" in which the model performs the core of its processing lies closer to English than to any other language. Wendler and colleagues phrased it cautiously — they spoke of proximity to English, not necessarily of an English intermediate step.

A follow-up study by Schut and colleagues (February 2025, arXiv:2502.15603) went further. The team from Oxford and Google DeepMind analyzed four open models — Llama-3.1-70B, Gemma-2-27B, Aya-23-35B, and Mixtral-8×22B — generating open-ended text in French, German, Dutch, and Mandarin. The conclusion was more unequivocal: the models make their key decisions in the representational space closest to English, regardless of the language of input or output. For semantically "loaded" words (nouns, key concepts), English representations appear first and are only then converted into the target language. Function words (prepositions, conjunctions) do not pass through this English intermediate stage.

The authors also demonstrated that so-called steering vectors — activation vectors used to influence the model's behavior — work better when constructed in English than when constructed in the input language. This confirms that the English representational space is not merely a by-product of the measurement, but a functional core of the processing.

The cause is simple: data. The training corpora of the dominant models are 60–90% English. Common Crawl, the largest publicly available internet corpus and the foundation of most training datasets, contains roughly 46% English text. Czech accounts for less than 1%. The less-represented half of the EU's official languages together make up 2.4% of the data.

When a neural network with billions of parameters learns on such an uneven distribution, it builds internal representations optimized for the dominant language. English becomes the model's "native language" — the language in which its conceptual representations are richest, most subtle, and most reliable.

A study of Japanese models (Zhong et al. 2024, ACL Findings 2025), however, showed that this need not be inevitable. The LLM-jp model, pretrained on a balanced mixture of Japanese and English, exhibited Japanese-centered internal representations. The Swallow model, fine-tuned on Japanese, used a mixed regime — both Japanese and English in the middle layers. Only Llama-2, with its predominantly English training, consistently routed through English.

The conclusion is important: English thinking is not a property of the transformer architecture. It is a consequence of the training data. A model trained on a sufficient amount of high-quality Czech would "think" in Czech.

For a Czech user, the models' English thinking has three concrete consequences.

Loss of performance in reasoning. When a model solves a mathematical problem posed in Czech, it must handle two tasks at once: understand the Czech problem statement and perform the computation. If the reasoning takes place in English, the model first internally "translates" the problem, then reasons, then translates the result back. Each translation introduces noise. A study by Barua et al. (August 2025, arXiv:2508.14828) systematically examined chain-of-thought reasoning in nine non-English languages and found that the gap between English and non-English reasoning widens on tasks requiring long, multi-step chains of thought. Mathematical problems are the most affected.

Cultural and domain filter. A model that "thinks in English" inevitably applies an Anglo-American conceptual framework. Yong et al. (May 2025, arXiv:2505.05408) demonstrated that models trained on English reasoning generalize poorly outside the natural-science and technical fields — they fail especially on questions from the humanities and on cultural knowledge. Czech history, the legal system, and cultural references are processed by the model through an English filter, which introduces interpretive distortion.

Asymmetry in the level of detail of answers. Empirical observations show that English answers are systematically longer and more detailed than Czech ones — even though Czech is a morphologically richer language. In English the model "thinks aloud" in more detail, because its English concept space is richer. In Czech it generates more concise, less elaborated outputs.

The research community has responded by developing prompting techniques that do not fix the models' English thinking but instead harness it to the user's advantage.

Cross-lingual prompting (XLT). Huang et al. (2023, EMNLP Findings) proposed a template that explicitly tells the model: think step by step in English, then formulate the result in the target language. Evaluation on seven benchmark suites covering reasoning, comprehension, and text generation demonstrated an improvement of over 10 percentage points on arithmetic reasoning and open-ended questions. XLT also reduces the variance in performance across languages — it narrows the gap between the best and the worst language.

The principle is elegant: instead of suppressing the English intermediate step, it makes use of it. The model is instructed to work in its "native" language and only to phrase the output in Czech. The problem: the user has to know that this technique exists and actively use it. Most Czech users do not.

Dictionary insertion prompting (DIP). Lu et al. (November 2024, arXiv:2411.01141) chose a different path: into a non-English query they insert English equivalents of key words drawn from a dictionary. The model thus receives a bilingual input that makes its internal conversion to English easier. The technique was tested on 200 languages from the FLORES-200 language set, and on the GSM8K mathematical benchmark it yielded a substantial improvement over the standard way of posing the query.

Dictionary insertion is more effective than XLT for languages with little training data, because it addresses the problem already at the input — it helps the model understand the task more precisely. For Czech, where the problem is the quality of reasoning rather than comprehension of the input, the benefit is smaller, but measurable.

Forcing the reasoning language. Yong et al. (May 2025, arXiv:2505.05408) examined what happens when you explicitly forbid the model to think in English and force it to generate the reasoning chain in the target language. The finding: for well-represented languages (French, German, Spanish), reasoning in the target language is comparable to English. For under-represented languages, performance drops sharply. Moreover, models forced to think in an under-represented language generate longer and less effective reasoning chains — you pay more for a worse result.

For Czech — a moderately represented language — the situation is mixed. Barua et al. (2025) found that for well-represented languages, reasoning in the given language works at the level of English, but for other languages the English intermediate step still wins.

One of the most remarkable findings of the past year is the identification of a pattern that Yong et al. named "quote-and-think." When an English-centric reasoning model is given a task in a non-English language, its reasoning chain typically contains direct quotes from the non-English problem statement in quotation marks, surrounded by English reasoning text.

The model literally "quotes" the Czech phrase, then "thinks" about it in English, then quotes the next phrase. It is a sophisticated mechanism of cross-lingual transfer — the model uses its multilingual ability to understand the input and its English dominance to reason. At the same time, however, it confirms a fundamental asymmetry: the model cannot think about Czech text in Czech.

All three workarounds share a common trait: they accept English thinking as a given and try to exploit it. They are ways around the problem, not solutions to it. And they have three fundamental limitations.

First: they work mainly for formalizable tasks. Mathematics, logic, structured questions — these are areas where the English intermediate step does not cause large losses, because the core of the problem is language-independent. For tasks requiring a deep understanding of the Czech context — law, history, cultural nuance, irony, humor — the English concept space is insufficient, and no prompting technique will remedy that.

Second: they require an experienced user. The ordinary Czech user of ChatGPT, Claude, or Gemini does not know about cross-lingual prompting. They do not know they should tell the model to "think in English." They do not know that the model generates more concise and less detailed answers in Czech. They receive a degraded product without knowing it.

Third: they confirm that the problem is in the data, not in the prompts. The study of Japanese models (Zhong et al. 2024) showed it clearly: a model trained on a sufficient amount of the target language "thinks" in that language. Seznam's SeLLMa model, fine-tuned on Czech texts using the continued-pretraining method, improves the quality of Czech — but to what extent it shifts the concept space closer to Czech, the publicly published data do not say.

The real solution is not a better prompt. The real solution is more Czech data in training, a better Czech tokenizer, and models whose concept space is not one-sidedly tilted toward English. That is precisely the goal of projects such as OpenEuroLLM (coordinated by ÚFAL MFF UK, budget €37.4 million, first models in mid-2026) and the Czech csmpt7b model from BUT-FIT (VUT Brno), which was the first to use a Czech tokenizer. But until such models are commonly available, the following holds: when you speak Czech with a language model, you are answered by someone who thinks in English and only then translates.

As long as reality is what it is, there are several measurably effective practices.

For tasks requiring reasoning (mathematics, analysis, planning), add an instruction at the end of your query: "Think step by step in English, answer in Czech." According to the data from the XLT study, this yields an average improvement of over 10 percentage points on reasoning-focused benchmark suites.

For complex questions requiring Czech context (law, history, official matters), the English intermediate step does not help — the model lacks sufficient English knowledge of Czech realities. Here it is more effective to provide the context directly in the query: relevant passages of the law, historical data, definitions of terms.

For creative writing in Czech, expect the model to generate more formal, less idiomatic text with an anglicizing sentence structure. For more natural Czech, you need to instruct it explicitly: "Write in natural colloquial Czech" or "Use Czech idiomatic expressions."

And for all tasks the following holds: if you want a more detailed answer, ask for it explicitly. By default the model generates shorter outputs in Czech than in English — not because it has less to say, but because its Czech "output space" is poorer.

Key sources

Wendler et al. (2024): "Do Llamas Work in English? On the Latent Language of Multilingual Transformers." EPFL. arXiv:2402.10588. Analysis of Llama-2 using the logit lens, three phases of language processing, English concept space. arXiv

Schut et al. (2025): "Do Multilingual LLMs Think In English?" Oxford, Google DeepMind. arXiv:2502.15603. Extension to 4 models and open-ended generation; steering vectors confirm English dominance. arXiv

Huang et al. (2023): "Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting." Microsoft Research. EMNLP Findings 2023. Cross-lingual prompting template, improvement of 10+ points. ACL Anthology

Lu et al. (2024): "Dictionary Insertion Prompting for Multilingual Reasoning." arXiv:2411.01141. Dictionary insertion technique, tested on 200 languages. arXiv

Yong et al. (2025): "Crosslingual Reasoning through Test-Time Scaling." arXiv:2505.05408. The "quote-and-think" pattern, language forcing, limitations for under-represented languages. arXiv

Barua et al. (2025): "Long Chain-of-Thought Reasoning Across Languages." arXiv:2508.14828. Systematic study of 9 languages; the gap widens for multi-step reasoning. arXiv

Zhong et al. (2024): "Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?" ACL Findings 2025. Japanese models as proof that the English intermediate step is not inevitable. arXiv

OpenEuroLLM: Coordinated by ÚFAL MFF UK, budget €37.4 million, 24 official EU languages. openeurollm.eu

Fajčík et al. (2024): "BenCzechMark: A Czech-centric Multitask and Multimetric Benchmark for Large Language Models." BUT-FIT, VUT Brno. arXiv:2412.17933. 50 tasks, the first comprehensive Czech benchmark for language models. arXiv

Transparency of creation:

The concept, structure, and editorial line of the article are the work of the author, who prepared the content sketch, established the key theses, and directed the entire creative process. Generative AI (Claude, Anthropic) was used as a technical tool for research, fact-checking, and fleshing out the author's outline.

The author edited the outputs on an ongoing basis, verified the key findings, and approved the final wording. No part of the text was published without human oversight. All factual data were verified against the publicly available sources cited in the text.

The procedure complies with the requirements of Art. 50 of EU Regulation 2024/1689 (the AI Act) on the transparency of AI-generated content. #poweredByAI