From Expensive Experiment to Standard: Nine Years of Mixture-of-Experts Architecture

The article charts nine years of development of the Mixture-of-Experts (MoE) architecture - from the pioneering paper "Outrageously Large Neural Networks" (Google, 2017) to today's models such as DeepSeek-V3, Kimi K2, Llama 4, and Qwen3. It explains the four engineering layers that made MoE work (routing, load balancing, fine-grained/shared experts, training stabilization), and the empirical finding that experts do not specialize by topic, but by syntactic and positional patterns. The conclusion outlines related principles of conditional computation - Mixture-of-Depths, State Space Models (Mamba), and the hybrid Jamba architecture.

K terminologii: anglických termínů je tu hodně...

A note on terminology: there are a lot of English terms here. Where it made sense I used a Czech equivalent, but the field's nomenclature is English and a literal translation often confuses more than it helps. For an unfamiliar concept I recommend asking an AI assistant ("what is X in the context of LLMs") — it's more illustrative than a dictionary. Just the basics up front: sparse = řídký, dense = hustý.

On 26 December 2024, the Chinese company DeepSeek-AI published a technical report on its V3 model. The number that appeared on the first page looked like a typo: 671 B parameters in total, 37 B active per token, trained on 14.8 trillion tokens. The company stated that the main pre-training phase took roughly 2.664 million H800 GPU-hours — an order of magnitude less than what Meta had published for training Llama 3 405B. Within a few weeks it had become a name on every benchmark leaderboard. That closed an arc that began in 2017 at Google with one seemingly crazy paper titled "Outrageously Large Neural Networks."

DeepSeek-V3 is not the only model that would suffice to explain the shift. The open Kimi K2 from Moonshot AI (July 2025) goes to a trillion parameters with 32 B active: 384 routed experts plus one shared, with eight activated per token. Meta's Llama 4 (April 2025) is its first MoE family ever — the Scout variant has 109 B / 17 B active and a context of up to 10 million tokens; the Maverick variant has 400 B / 17 B active. Alibaba's Qwen3-235B-A22B (May 2025) has 128 experts and top-8 routing. MiniMax-Text-01 (January 2025) combines 456 B / 45.9 B active with Lightning Attention. Mixtral 8×7B, which in January 2024 became the first mass-deployed open MoE model, now looks like a prototype.

Going back to Shazeer's 2017 paper (arXiv:1701.06538), one finds that the main thesis was settled then: a model with a sparsely-gated MoE layer has capacity well above a dense baseline model, yet activates only a small subset of its parameters per sample. From this follows a simple but non-trivial implication: parameters and the number of operations (FLOPs) can be decoupled. What unfolded between 2017 and 2026 was essentially the hard engineering question of how to carry that idea from the LSTM into the world of the Transformer architecture, and how to stabilize training at a scale at which the original MoE simply collapsed.

In short: today's MoE models work thanks to four problems solved one after another — the formulation of routing, load balancing, the fine-grained / shared expert architectural pattern, and stabilizing training at large scale. No magic. Four engineering layers stacked on top of one another.

The original concept of a mixture of experts — a gating network that splits the input space among several separate networks — comes from Jacobs, Jordan, Nowlan and Hinton in 1991 (Neural Computation 3(1), "Adaptive Mixtures of Local Experts"). For most of the next three decades the idea remained largely theoretical.

Shazeer and colleagues (alongside them Mirhoseini, Maziarz, Davis, Le, Hinton and Dean) revived it in 2017 in the text "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (arXiv:1701.06538). They inserted an MoE layer between the LSTM layers of a machine-translation model and, in aggregate, trained a model with up to 137 billion parameters. The key technical innovation was noisy top-k gating: for each input a score is computed across all experts, Gaussian noise proportional to a learned scale is added to the logits, and the top-k experts are selected; the output is a weighted sum of their outputs. The noise improves exploration and prevents the router from degenerating into a constant choice. To this the authors added two auxiliary loss functions — one to balance importance, the other to balance expert load. Without them the system collapses: the router picks a few favorite experts, they train faster, become more attractive, and a positive feedback loop buries the rest of the model.

The leap into the Transformer world was made by the GShard paper from Lepikhin and colleagues (arXiv:2006.16668, 2020), which scaled multilingual translation to ~600 B parameters with 2,048 experts and top-2 routing. GShard is also the source of vocabulary still used today: the capacity factor (how many tokens an expert may accept before it starts dropping tokens), expert parallelism (experts distributed across devices), and all-to-all communication (dispatching tokens to experts and combining them back). All-to-all is the main communication bottleneck of MoE training; its cost grows with the number of devices and is proportional to the volume of data divided by the bisection bandwidth.

In January 2021 the Switch Transformer (Fedus, Zoph, Shazeer, arXiv:2101.03961) simplified routing to top-1. Each token goes to exactly one expert, and a capacity buffer decides when tokens are simply skipped over the residual path. Switch was the first model to convincingly show that sparse models with hundreds of billions of parameters can be trained in bfloat16. It also published one of the first sparse → dense distillations: it showed that a large sparse model can, under compression of 10 to 100× — that is, removing up to 99 % of the parameters — be distilled into a small dense student that retains roughly 30 % of the teacher's quality lead over a dense baseline. From this comes a practical lesson that is sometimes forgotten: an MoE model can, in some cases, be handed off to a dense student, yielding a reasonable compromise between training efficiency and inference cost.

The stability of training large MoE models was addressed by the ST-MoE paper from Zoph and colleagues (arXiv:2202.08906, 2022). Its main contribution is called the router z-loss: it penalizes large router logits in the form (log Σ exp(x_j))² and thereby keeps numerical ranges within manageable bounds. With a coefficient of 10⁻³ it improves stability with minimal loss of quality. It became the de facto mandatory seasoning of every sparse model. ST-MoE is also the first paper to carefully analyze what the experts actually learn.

In December 2023, or rather January 2024, Mistral AI came out with Mixtral 8×7B (technical report arXiv:2401.04088). The model is based on Mistral 7B; each of the 32 layers has 8 experts, routing is top-2, and the experts are single-layer MLPs with SwiGLU activation. Numerically: 47 B parameters in total, but approximately 13 B are activated per token. A 32 k token context, open Apache 2.0 license. Somewhat later (April 2024) came Mixtral 8×22B with 141 B / 39 B active.

Mixtral was not an architectural revolution — top-2 routing of eight experts was more or less a direct heir of GShard. Its significance was practical: for the first time there existed an open, accessible, well-documented MoE model on which the community could get hands-on with everything from a vLLM server to LoRA fine-tuning. Beyond that, Mixtral brought the first thoroughly published analysis of expert specialization in a modern LLM.

The real theoretical advance was the DeepSeekMoE paper from January 2024 (Dai et al., arXiv:2401.06066, ACL 2024). DeepSeekMoE named two problems of the classic MoE design: knowledge hybridity (a single expert has to hold so much varied knowledge that it cannot specialize) and knowledge redundancy (different experts have to learn the same thing). For both it proposed two mutually supporting techniques.

The first is fine-grained expert segmentation: instead of N experts of standard FFN size, one introduces mN finer experts (each with m-times smaller inner dimension), and activates mK of them instead of K. The computational cost stays constant, but the number of possible combinations of active experts rises sharply. With 64 fine experts and top-8 routing, the model has 4.4·10⁹ combinations versus 120 combinations for 16 experts with top-2. Specialization has room to develop.

The second is shared expert isolation: one or a few experts are always active and take over the role of "general knowledge." The routed experts therefore do not have to keep learning the shared things over and over, and can truly specialize. DeepSeekMoE 16B, with two shared and 64 routed experts (activating six per token), matches the performance of the dense DeepSeek 7B and LLaMA-2 7B at roughly 40 % of the compute.

This pattern — fine-grained + shared — became standard. You will find it in DeepSeek-V2 (236 B / 21 B active, 2 shared and 160 routed experts), in DeepSeek-V3 (671 B / 37 B, 1 shared and 256 routed), in Kimi K2 (1 T / 32 B, 1 shared and 384 routed). Llama 4 partially adopts it, alternating MoE and dense layers in some variants.

DeepSeek-V3 added another theoretical advance that also appears set to take hold: auxiliary-loss-free load balancing (Wang et al., arXiv:2408.15664, 2024). The classic auxiliary loss has one unpleasant property — its gradient interferes with the model's main loss function. The strength of the regularization has to be set so that it balances but does not destroy quality, and this equilibrium is fragile. DeepSeek-V3 sidesteps it elegantly: for each expert a bias term b_i is introduced, which is added to the affinity score when selecting the top-k, but the gating value (the combination weight) is computed from the original affinity score. After each step the bias is dynamically adjusted — lowered for overloaded experts, raised for underused ones, at a rate γ. The network's main gradient is thus untouched by balancing; balancing runs in parallel as a control loop.

In terms of design cleanliness this is elegant: separating two functions (prediction quality / load balance) that until then shared a single gradient into two independent signals. In practice it means less interference and measurably better results for the same number of parameters.

One of the empirical findings that recurs in the MoE literature across models, and that intuitively surprises, is this: experts do not specialize by high-level topic. No "math expert," no "code expert," no "biology expert." Instead they specialize by syntactic and positional patterns — punctuation, conjunctions, certain morphemes, indentation.

The Mixtral study sums it up unambiguously: in the distribution of tokens across experts it observes no clear patterns by topic — the distribution is very similar for arXiv articles, for biology, and for philosophy. At the same time the authors observed strong positional locality — consecutive tokens often go to the same expert, and this effect strengthens in the higher layers of the model (typically layers 15 and 31 of 32). Tokens such as the keyword self in Python, or a specific whitespace pattern in indentation, route consistently, but not according to "what the text is about."

ST-MoE went further in its analysis and arrived at a distinction: encoder experts specialize, decoder experts do not. In the encoder you find experts that learn punctuation, conjunctions, articles, verbs; in the decoder the distribution is far less legible. And in multilingual models experts do not specialize by language, as naive intuition might expect — on the contrary, similar syntactic roles across languages often fall to the same expert.

From the standpoint of interpretability this is a mild defeat for anthropomorphic metaphors. An "expert" in MoE is not a person who knows one thing. It is more of a learned filter that reacts to a low-level pattern in a token's representation, and semantic coherence arises only from the interaction of many such filters across layers. The paper "A Closer Look into Mixture-of-Experts in Large Language Models" (Lo et al., arXiv:2406.18219, Findings NAACL 2025) extends the analysis to Mixtral, DeepSeekMoE and Grok-1 and broadly confirms the above observation.

Practical implication: if you want to use an MoE model for interpretive purposes (for example, "switch off the expert section that decides the ethical aspects"), you will hit a wall. Specialization exists, but not in the terms in which a human naturally thinks of it.

The question "why does MoE work" was for a long time answered more or less intuitively: parameters give the model more capacity, sparsity keeps the computational cost low. A mathematically cleaner answer came in the paper by Krajewski and Ludziejewski et al., "Scaling Laws for Fine-Grained Mixture of Experts" (arXiv:2402.07871, ICML 2024).

The authors added to the traditional scaling law (laws of scaling) — loss as a function of size N and tokens D — a third variable: granularity G, defined as the ratio of FFN size to the size of a single expert. G=1 means a standard expert the size of the whole FFN; G=2 means each expert is half that. From a fit over a large range of experiments comes a twofold conclusion: the efficiency gap between dense and MoE models widens as the model size and training budget grow; and the common practice of setting expert size to mirror the size of the feed-forward layer is not optimal at almost any compute budget.

In other words: the advantage of MoE grows with the budget (unlike some earlier scaling-law studies that claimed the opposite), and the optimal granularity G is generally greater than 1. This directly justifies why DeepSeek-V3 goes to 256 experts with top-8 and why Kimi K2 goes to 384 — history has shown that a coarse-grained design à la Mixtral 8×7B leaves room on the table.

This is the theoretical layer I would today recommend that anyone designing their own MoE read twice. It defines how to solve the joint optimization of three hyperparameters (N, D, G) for a given compute budget. It is not a definitive certainty — the predictions have a reasonable confidence interval — but it is the only available systematic answer to the question "how big should an expert be."

Empirically it turns out that the trade-off between sparsity and quality is meaningful only in a certain regime. With a very small model and a very small batch size, MoE usually does not pay off — expert utilization is low, and the all-to-all communication overhead swallows the savings. With a large model and a large batch size, sparsity dominates. Between them is a wide gray zone in which the specific deployment details decide.

MoE is not the only branch of conditional computation that is production-relevant in 2026. In parallel, other architectural principles have taken hold or are taking hold that share the same philosophy — perform on a given position only as much computation as is necessary.

Mixture-of-Depths (Raposo et al., arXiv:2404.02258, DeepMind 2024) applies the same idea as MoE, but to the depth axis instead of the width axis. In each layer a router selects the top-k tokens that actually pass through the layer; the other tokens skip it via the residual path. Capacity is defined up front, so the computation graph remains static. An iso-FLOP MoD model is, with post-training sampling, up to 50 % faster than a dense baseline. From a theoretical standpoint, MoD is in fact a degenerate MoE with two "experts" per layer — the layer itself and the identity.

In a separate development branch, State Space Models advanced. Mamba (Gu and Dao, arXiv:2312.00752, 2023) introduced a selective SSM with input-dependent parameters and a hardware-aware parallel scan. The key conclusion: linear complexity in sequence length (unlike the quadratic complexity of attention), 5× higher inference throughput, and, in its 3B version, it outperforms Transformers of the same size. Mamba-2 (Dao and Gu, arXiv:2405.21060, 2024) additionally formally connected SSMs and attention under the heading of structured state space duality.

The practical convergence showed up in Jamba from the Israeli AI21 Labs (Lieber et al., arXiv:2403.19887, 2024). Jamba combines Transformer attention, the Mamba SSM, and MoE in a single model: the attention:Mamba ratio is 1:7, MoE is in every other layer, with 12 B active parameters out of 52 B total. The main practical gain is in memory for long-context inference: thanks to the hybrid Attention-Mamba architecture, Jamba's KV cache takes up only 4 GB even at a 256k context, versus roughly 32 GB for Mixtral at the same context. Long-context throughput is 3× higher than Mixtral 8×7B. Jamba-1.5-Large (arXiv:2408.12570, 2024) scaled the same principle to 94 B active.

MiniMax-Text-01 from January 2025 (arXiv:2501.08313) takes a similar route, only instead of Mamba it uses Lightning Attention — an I/O-aware linear attention variant. The layers alternate: one softmax block to seven lightning blocks, plus MoE with 32 experts. Native Sparse Attention from DeepSeek (arXiv:2502.11089, ACL 2025) adds natively trainable sparse attention with three branches (compression, block selection, sliding window) and reports a 9× speedup of the forward pass, 6× of the backward pass, and 11.6× of the decode pass relative to full attention.

What follows from this: the "dense vs. sparse" dichotomy is no longer sharp. The frontier models of 2025–2026 mix sparsity at the expert level (MoE), at the layer level (MoD), at the attention level (sparse / linear attention), and at the architecture level (hybrid SSM-Transformer-MoE). Sparsity has become a design principle applied to different components independently.

The argument against thoughtlessly deploying MoE rests on a whole series of concrete situations.

The first and most serious objection is the memory footprint. MoE reduces FLOPs per token, but memory is proportional to the total number of parameters. Mixtral 8×7B needs all 47 B parameters in memory, even though it activates only 13 B per token. For inference at the memory edge (single GPU, edge deployment, mobile) this is usually a fundamental obstacle. Applying aggressive quantization (Llama 4 in Int4, Jamba's ExpertsInt8 with INT8 weights converted to BF16 before computation) mitigates this problem but does not remove it.

The second objection is batch size sensitivity. MoE pays off only when each expert receives enough tokens. At batch size 1 (the typical use case in a chat with a single user) expert utilization is extremely low and the hardware runs under capacity. In production this means that MoE models scale substantially better on a shared inference server with many concurrent users than in single-stream mode. The practical threshold sits at around eight tokens per expert; below it, effective utilization drops steeply.

The third objection is all-to-all communication overhead. When experts are distributed across multiple GPUs, each layer must perform two all-to-all operations (dispatch + combine). This is the main scaling limit. DeepSeek-V3 mitigated it with node-limited routing (each token is sent to at most four nodes — a principle inherited from DeepSeek V2's device-limited routing) and computation-communication overlap. But in a homogeneous cloud infrastructure with a slow interconnect (typically cross-rack in ordinary clouds), the overhead remains painful.

The fourth is fine-tuning fragility. MoE models are prone to overfitting during fine-tuning, especially on small data. The distribution of tokens across experts can degenerate on a small dataset, and uneven expert use makes convergence harder. In some cases the solution is to freeze the router and fine-tune only the expert weights; in others nothing helps, and a dense student model distilled from an MoE teacher is the cleaner choice.

The fifth is debugging and interpretability. Routing adds a non-trivial layer of nondeterminism — a small change in the input can reroute a token to a different expert and qualitatively change the output. For production systems with reproducibility requirements this is a risk.

If one were to sum up the sharpest form of the criticism: MoE is an optimization for the training budget and for a large parallel inference server. In single-stream low-latency inference on limited hardware, dense models often remain more competitive, especially if speculative decoding (EAGLE-3, Medusa) is added. Anyone optimizing single-stream latency today should not jump on MoE just because it is fashionable.

And yet: for models at the frontier, trained once with a budget of tens of millions of dollars and operated for millions of users in parallel, there is no competition today. DeepSeek-V3 came in at an estimated 5.576 million dollars (total training, i.e. 2.788 million H800 GPU-hours at 2 dollars per hour, per the technical report). A dense model of comparable quality would need multiples of that. This economic consideration is the quiet reason MoE took over the open-weight scene.

Sparsity did not become a miracle pill. It became an engineering standard whose application has clear rules. If you are training a frontier model and have a large compute budget available, MoE with fine-grained experts, a shared expert, and auxiliary-loss-free balancing is probably the optimal choice today. Derive the hyperparameters — granularity, number of routed experts, top-k — from scaling laws, not from intuition. If you are serving a model to hundreds of thousands of users in parallel, MoE makes sense. If you are serving a single-stream low-latency application, it does not.

And if you want to understand all of this theoretically, reading a few key papers will pay you back handsomely: Shazeer 2017 for the concept, the Switch Transformer 2021 for routing and the auxiliary loss, ST-MoE 2022 for stability and expert specialization, DeepSeekMoE 2024 for the fine-grained / shared design, Krajewski 2024 for scaling laws, Wang 2024 for auxiliary-loss-free balancing. Whoever reads this half-dozen with care is set for the next two years of development.

Plenty of open questions remain. Why exactly bias balancing works better than the auxiliary loss, no one has yet settled theoretically — in December 2025 the paper "A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models" (arXiv:2512.03915) appeared, which is the first serious attempt. The optimal sparsity ratio at very large scale is not settled. The convergence of MoE, SSMs and linear attention into a single hybrid recipe is still under way. Multimodal MoE — balancing between modality and domain specialization — is an open topic.

Meanwhile the community moves on. GPT-5, Claude Opus 4.7, Gemini 3 — the properties of all the newest models, whether or not they are openly published, move within a paradigm that Shazeer described nine years ago in a single preprint. An expensive experiment has become the new default state. The expensive experiments are now somewhere else.

Main sources (chronological):

Transparency of creation:

The conception, structure, and editorial line of the article are the work of the author, who prepared the content sketch, set out the key theses, and directed the entire creation process. Generative AI (Claude, Anthropic) was used as a tool for research, locating primary sources, and the formulational development of the author's content sketch.

The author edited the outputs continuously, verified the key findings, and approved the final wording. No part of the text was published without human review. All factual data were verified against the publicly available sources listed in the text.

The procedure complies with the transparency requirements for AI-generated content under Art. 50 of EU Regulation 2024/1689 (the AI Act). #poweredByAI