Chapter 6 Why MoE Is Now Standard in Frontier Systems

6.1 Dense vs. Sparse Models for Fixed FLOPs

6.1.1 Parameters for Free: The Core MoE Argument

When an interviewer asks why you would choose a mixture-of-experts architecture over a dense model, the answer is not simply “more parameters.” The precise answer is more parameters at the same FLOPs budget. This single idea is what has convinced every major frontier lab to adopt MoE, and it is tested at every seniority level once you are interviewing for roles that touch training infrastructure or model architecture. At the senior level you need to be able to quantify it: explain why perplexity curves diverge at identical training compute and what that implies for how you spend a fixed GPU budget.

A standard dense transformer sends every token through the same feedforward network at every layer. A sparse MoE transformer replaces each FFN block with $N$ expert FFNs but activates only $K$ of them per token. If each expert is the same size as the original FFN and $K = 1$, the forward-pass arithmetic is identical to the dense model-the same number of multiply-accumulate operations-but the model now stores $N$ times as many FFN parameters in memory.

The MoE layer output for token $t$ with residual-stream input $\mathbf{u}$ is:

where $g_{i} ( \mathbf{u} ) = 0$ for all but the top-$K$ experts by router score. The nonzero gates are derived from a softmax over the selected indices and combine the chosen experts’ outputs as a weighted average. When $K = 1$ this collapses to a single FFN evaluation with the residual added back: identical FLOPs to the dense case, $N \times$ the stored parameters.

The empirical case is now settled. Switch Transformer showed that at a fixed training FLOP budget, increasing the expert count from 1 to 256 monotonically reduced language-modeling loss. OlMoE replicated the result on a modern architecture: the MoE training curve decays faster and reaches a better floor than a compute-matched dense baseline. DeepSeek’s early MoE ablations compared dense, naive-MoE, and improved-MoE variants across multiple benchmarks and found consistent improvement as routing quality rose. The same pattern reappears in the DeepSeek-V2 serving results: remarkably low error rates at activated-parameter counts far below what a comparable dense model would require.

At a fixed compute budget, sparse MoE models consistently outperform dense models because they pack more parameters into the same FLOPs, and language models use those extra parameters productively.

Language modeling has two separable demands: computing good representations of context (which requires FLOPs) and memorizing factual associations, vocabulary patterns, and domain idioms (which requires parameter capacity). Dense architectures must serve both demands with the same parameter budget. MoE decouples them: the activated experts handle computation while the dormant ones hold capacity the model can draw on when the right inputs arrive.

This is not an abstract argument. It is the reason that hashing-based routing-a fixed deterministic function with no semantic content-still provides performance gains over a single FFN at the same activated FLOPs. The capacity benefit is real and does not require perfectly learned routing to appear.

The costs are also real. More total parameters means more memory: even if your activated FLOPs are modest, you must store the full weight tensor for every expert somewhere in your serving cluster. Routing introduces systems complexity-token dispatching, all-to-all communication in expert-parallel configurations, and load-balancing machinery to prevent expert collapse. Dense models are simpler to checkpoint, replicate, and fine-tune. But for any setting where training compute is the binding constraint and memory plus engineering overhead are tractable, the FLOPs-per-loss advantage of MoE is now essentially settled empirical fact.

A Worked Example

Consider a 7B-parameter dense baseline with $d_{\text{model}} = 4096$, FFN hidden dimension 11008 (SwiGLU, $\approx 2 . 7 \times d_{\text{model}}$), and 32 layers. Each FFN block has roughly $3 \times 4096 \times 11008 \approx 135$M parameters (three projections in the gated network), giving about 4.3B FFN parameters across the full model.

Now design a FLOP-matched MoE: 8 experts per layer, each expert at one-quarter the hidden dimension (2752), top-2 routing:

• Parameters per expert: $3 \times 4096 \times 2752 \approx 34$M

• Total expert parameters per layer: $8 \times 34$M $= 272$M

• Total expert parameters across 32 layers: $32 \times 272$M $\approx 8 . 7$B (2$\times$ the dense FFN budget)

• Activated per token per layer: $2 \times 34$M $= 68$M, roughly half the dense FFN per layer

For an exact FLOPs match, use top-1 routing with full-size experts: 8 experts each at 135M parameters, activating 1. The activated compute is identical to the dense baseline; the stored FFN parameters jump from 135M to $8 \times 135$M $= 1 . 08$B per layer.

Training both variants for 1T tokens at identical compute budget: empirical results typically show a 0.1-0.2 nats/token improvement in perplexity for the MoE, translating to meaningfully better downstream benchmark scores. The memory cost for the 8-expert MoE is approximately $8 \times$ the dense FFN footprint-that is the price paid for those gains.

What You Actually Decide in Practice

• Use MoE when training compute is the bottleneck, not memory. If you are already memory-limited at inference and cannot shard experts across devices, dormant parameters become dead weight that yields no performance return.

• Fine-grained experts beat fewer large ones. Ablations in both DeepSeek MoE and OlMoE consistently show that 64 quarter-size experts outperform 16 full-size experts at the same activated-FLOPs budget. Default to fine-grained experts when you have the routing infrastructure to support them.

• $K = 2$ is the canonical starting point. Top-1 routing is maximally compute-efficient but gives the router no gradient signal to compare alternatives. Top-2 provides that comparison at the cost of doubling pertoken FFN FLOPs relative to top-1. Most production systems (Mixtral, DeepSeek, Grok) use $K = 2$ or a fine-grained equivalent.

• Expert specialization is less semantic than the name implies. Hashing-based routing provides real gains without any learned affinity between tokens and experts. If a reviewer tells you MoE works because experts specialize by domain, that framing is not wrong but it is incomplete: the parameter-capacity benefit alone explains the FLOPs-matched quality gap.

• Every-layer MoE is now standard at frontier scale. Earlier designs alternated dense and MoE layers to reduce fine-tuning instability. Modern production systems that absorb the engineering overhead apply MoE at every FFN layer.

How This Shows Up in Interviews

• “Why does a sparse MoE outperform a dense model trained on the same compute?” Strong answer: the MoE decouples parameter capacity from compute. At equal FLOPs, the activated experts do the same arithmetic as the dense FFN, but the model holds additional capacity in dormant experts that it exploits for memorization. Empirically, that capacity is productive across diverse tasks.

• “What engineering costs do you accept when switching from dense to MoE?” Strong answer: three categories. Memory: total parameter count grows by up to $N \times$, requiring expert-parallel sharding. Routing: non-differentiable top-$K$ selection requires auxiliary load-balancing losses to prevent collapse. Stability: MoE models are more prone to loss spikes and fine-tuning divergence; z-loss on router logits and float32 router computation are standard mitigations.

• Senior follow-up: “How does the MoE FLOPs advantage interact with compute-optimal scaling?” Compute-optimal reasoning applies to activated parameters and training FLOPs. Dormant MoE parameters are essentially free from a training-compute standpoint-you are not paying for them in multiply-accumulate operations. This means a MoE can simultaneously be compute-optimal in its activated parameter count and parameter-rich in its total count. The catch is that serving cost scales with total parameters, not activated ones, so the efficiency argument flips at inference time.

Key Takeaways

• At identical training FLOPs, MoE models consistently achieve lower perplexity than dense models-this has been replicated across architectures, scales, and labs.

• The mechanism is decoupling: activated parameters set compute cost; total parameters set capacity. MoE scales capacity without scaling compute.

• Fine-grained experts (many small ones over few large ones) and $K = 2$ routing are the dominant production defaults.

• Primary costs are aggregate memory ($N \times$ the dense FFN footprint), routing infrastructure, and loadbalancing complexity to prevent expert collapse.

• Expert specialization is real but secondary; the parameter-capacity argument alone explains the FLOPsmatched quality gain.

6.1.2 Activated vs. Total Parameters

When a candidate describes a mixture-of-experts model as a “70B model” without qualifying which 70B they mean, they have already lost the thread of the economics. Understanding the difference between activated and total parameters is the single most important framing skill for reasoning about MoE systems: it determines training compute, serving memory, hardware topology, and how to interpret the benchmark plots that labs publish. Interviewers test this at every seniority level; senior engineers must derive concrete infrastructure numbers from both quantities on the spot.

In a dense transformer every parameter participates in every forward pass. The model has $P_{\text{total}}$ parameters and uses all of them to process each token, so $P_{\text{activated}} = P_{\text{total}}$.

A sparse MoE transformer breaks this coupling. For each token, the router activates only $K$ of the $N$ available experts per MoE layer. The rest are stored in memory but contribute zero FLOPs during that token’s forward pass. The two quantities are:

• Total parameters: all weights in the model, including every expert that will never fire for a given input. This determines the memory required to host the model.

• Activated parameters: the subset of weights that execute per token per forward pass. This determines per-token FLOPs and, by extension, training compute and inference latency.

For a model with $L$ MoE layers, $N$ experts per layer, each expert containing $P_{e}$ parameters, and $K$ experts activated per token, the two quantities are:

The ratio $N / K$ is the sparsity factor: it describes how many times larger the total FFN parameter count is relative to the activated FFN parameter count. A model with 64 experts and top-4 routing has a sparsity factor of 16. Training compute scales with $P_{\text{activated}}$; hosting cost scales with $P_{\text{total}}$.

Activated parameters set the compute price; total parameters set the memory price-and for MoE these two prices are decoupled by the sparsity factor.

For training, the activated-parameter count is all that matters for FLOPs. Compute-optimal reasoning following Chinchilla-style analysis should tune token count and activated model capacity together-dormant expert weights cost nothing in training compute. A 70B-total MoE model with 7B activated parameters trains at the FLOPs cost of a 7B dense model. If the resulting quality matches a 13B or 20B dense model-which empirical results from Switch Transformer, OlMoE, and the DeepSeek MoE line suggest it can-the MoE is simply a better use of a fixed training budget.

For serving, total-parameter count dominates. Every weight tensor, including experts that fire rarely or never for a particular query, must live somewhere in memory at inference time. The router’s decision is data-dependent; you cannot know in advance which expert will be needed for each token, so all experts must be resident. A 70B-total MoE costs the same inference-time memory as a 70B dense model, even if only 7B parameters contribute arithmetic per token. Latency per token is low (7B worth of FLOPs), but the weight memory bill is high.

This asymmetry explains why labs publish benchmark results with “activated parameters” on the x-axis. On that axis MoE looks spectacular: DeepSeek-V2’s MMLU results plotted against activated parameters sit far above comparably activated dense models at the same compute. That framing is not misleading-it accurately captures compute efficiency-but it does not tell the hosting story. An interviewer who asks “what is the catch?” is probing exactly this gap.

A Worked Example

Take a 7B dense baseline:

• $d_{\text{model}} = 4096$, SwiGLU FFN with $d_{\text{ff}} = 11008$

• FFN parameters per layer: $3 \times 4096 \times 11008 \approx 135$M

• 32 transformer layers; FFN total: $32 \times 135$M $\approx 4 . 3$B

• Non-FFN (attention, embeddings, norms): $\approx 2 . 7$B

• Total $\approx 7$B, activated $\approx 7$B

Now design a FLOP-matched MoE that targets the same per-token compute:

• 64 experts per layer, each at one-quarter the FFN hidden dimension ($d_{\text{ff}}^{e} = 2752$)

• Parameters per expert: $3 \times 4096 \times 2752 \approx 34$M

• Top-4 routing; activated FFN per layer: $4 \times 34$M $= 136$M (matches dense)

• Total FFN per layer: $64 \times 34$M $= 2 . 18$B

• Total FFN across 32 layers: $32 \times 2 . 18$B $\approx 70$B

• Total model: $70$B $+ 2 . 7$B $\approx 73$B; activated: $\approx 7$B

Training compute for a 1T-token run (using the $\approx 6 N D$ approximation with $N = P_{\text{activated}}$):

This is identical for the dense and MoE variants by construction.

Serving memory in BF16:

• Dense: $7$B $\times 2$ bytes $= 14$ GB-fits on a single A100-40GB.

• MoE: $73$B $\times 2$ bytes $\approx 146$ GB-requires at least two H100-80GB GPUs for weights alone, with expert-parallel sharding across devices.

The sparsity factor is $N / K = 64 / 4 = 16$. The MoE stores 16$\times$ more FFN parameters than the FLOPmatched dense model. You pay for all 73B weights in memory but compute with only 7B of them per token.

What You Actually Decide in Practice

• Use activated parameters for training budget planning. When computing your compute-optimal token-to-parameter ratio, plug in activated parameters. Sizing by total parameters overstates your training FLOPs and produces a model that is undertrained relative to its capacity.

• Use total parameters for serving infrastructure planning. When provisioning GPUs for inference, total parameters set the memory floor. Expert parallelism is mandatory once total parameters exceed a single node’s HBM capacity (typically 640 GB for an 8-way H100 NVLink node at BF16 weights before activations and KV cache).

• High sparsity factors amplify both the benefit and the hosting cost. A sparsity factor of 32-as in DeepSeek-V3’s 256 routed experts with top-8 routing-gives 32$\times$ more FFN parameters than a FLOPmatched dense baseline. The quality gain is real; the memory bill is proportionally real.

• Know which inference regime you are in. In memory-bound inference (small batches, long context), latency is dominated by weight loading, not arithmetic. A dense 7B model can outperform a 73B-total / 7B-activated MoE on latency in this regime because it loads far fewer bytes from HBM per token. In compute-bound inference (large batches), activated parameters dominate and the MoE and dense models have comparable throughput-the MoE wins on quality at that compute level.

• Benchmark comparisons require the right denominator. Before accepting a “quality vs. activated parameters” claim, verify what the total-parameter hosting cost is. A fair head-to-head for practitioners is either equal activated parameters (same training cost) or equal total parameters (same serving cost). Equal total parameters is the test the MoE must also pass at inference time.

How This Shows Up in Interviews

• “A lab claims their MoE model has 70B parameters and matches a 13B dense model in quality. What would you ask before accepting that comparison?” Strong answer: first, clarify whether 70B is total or activated. If it is 70B total with 7B activated, the training compute comparison is coherent-it trained at 7B FLOPs cost. But 70B total means roughly 10$\times$ the memory of a 7B dense model, which changes the serving calculus entirely. Second, ask whether the quality comparison is FLOP-controlled (same training compute) or size-controlled (same total parameters): beating a 13B dense at equal training FLOPs is expected; beating a 70B dense at equal total parameters would be a much stronger result.

• “How do you provision inference hardware for a 70B-total / 7B-activated MoE?” Strong answer: start with total parameters to set the memory floor. At BF16 that is $\approx$140 GB, requiring at minimum two H100- 80GB GPUs with expert-parallel sharding. Latency per token is set by activated-parameter FLOPs (7B worth), not total. Batch throughput is limited by the all-to-all communication dispatch added by expert routing, so benchmark throughput degradation against a purely dense serving stack before committing.

• Senior follow-up: “When does a dense model win at inference even if the MoE wins at training?” The senior answer requires distinguishing inference regimes. In memory-bandwidth-bound serving (single-user or small-batch), each token forces the hardware to stream weight bytes from HBM regardless of whether those bytes are used for arithmetic. A 73B-total MoE streams 10$\times$ more bytes per token than a 7B dense model even though the arithmetic is identical. Time-to-first-token and per-token latency both suffer in this regime. Only at large batch sizes, where the hardware is arithmetic-bound, do the two models converge on latency-at that point the MoE wins because its quality is higher at the same activatedparameter count.

Key Takeaways

• Activated parameters set per-token FLOPs and training compute; total parameters set memory footprint and hosting cost-MoE decouples these two quantities by the sparsity factor $N / K$.

• Training budget reasoning must use activated parameters; infrastructure provisioning must use total parameters. Mixing these up leads to hardware surprises in either direction.

• The sparsity factor quantifies the memory-to-compute leverage of a MoE design: a factor of 16 means 16$\times$ more FFN parameters in storage than in any single token’s forward pass.

• Benchmark plots that place “activated parameters” on the x-axis are accurate for compute efficiency comparisons but obscure the full memory cost of hosting all dormant expert weights.

• In memory-bandwidth-bound inference regimes, total parameter count governs latency; in compute-bound regimes, activated parameter count governs throughput-know which regime applies before choosing MoE over dense.

6.2 Routing: The Heart of Every MoE

6.2.1 Token-Choice Top-K Routing

Every MoE interview question eventually lands on the routing function. An interviewer at a frontier lab will ask you to describe the forward pass of a MoE layer, and the answer lives in two lines of math: a learned inner product and a top-$K$ gate. This question appears at the ML engineer and research engineer level at companies running production MoE systems, and a strong answer requires knowing the mechanics cold, the design rationale, and where the gradient actually flows. Getting any of these wrong signals that you have read about MoE without having thought through the details.

The router is a lightweight learned function that maps each token’s hidden state to exactly $K$ experts. The forward pass for a single MoE layer proceeds in four steps.

First, compute affinity scores between the incoming token hidden state $u_{t} \in \mathbb{R}^{d}$ and $N$ learned router embeddings $\{ e_{i} \}_{i = 1}^{N}$, where $e_{i} \in \mathbb{R}^{d}$:

The softmax normalizes the $N$ raw dot-products into a probability-like vector. Its role here is not to produce a probability distribution in the probabilistic sense; it is purely a normalization so that the downstream weighted sum behaves well.

Second, select the top-$K$ experts by score:

Third, form the sparse gate: $g_{i , t} = s_{i , t}$ if $i \in \mathcal{T}_{t}$, else $g_{i , t} = 0$.

Fourth, compute the layer output with a residual connection:

The router itself contains $N \times d$ parameters - one embedding vector per expert. For $N = 64$ experts and $d = 4096$, that is roughly 256K parameters, negligible compared to the expert FFN weights being routed into. DeepSeek-V3 replaces the pre-top-$K$ softmax with per-expert sigmoid scores and renormalizes after top- $K$ selection, which avoids the winner-takes-all compression that softmax applies before the discrete cut; the downstream layer norms absorb any resulting scale shift.

The entire routing decision for one token - an $N \times d$ matrix-vector product followed by a sort - costs fewer FLOPs than a single attention query.

Token-choice top-$K$ routing directly answers the question the model cares about: given this token’s representation, which experts process it best? The affinity $\text{score} u_{t}^{\top} e_{i}$ measures alignment between the token and the expert’s learned specialization direction. High-affinity experts get the token; all others are skipped. This lets different experts develop different processing competencies over training without any engineered inductive bias - the specialization emerges from the gradient signal.

The top-$K$ gate is also essential for training efficiency. If you removed it and instead used a soft gate weighted by the full softmax (all $N$ experts, all the time), you would pay $N$ expert forward passes per token. For $N = 64$ that is 64 times the compute of a dense model. Top-$K$ ensures only $K$ expert forward passes run per token: for $K = 6$ out of 64 experts, the FFN component costs roughly $6 / 64 \approx 9 \%$ of the all-expert baseline. This is what makes MoE parameter-efficient at fixed FLOP budget.

The alternative - expert-choice routing, where each expert picks its favorite tokens instead of tokens picking their favorite experts - achieves perfect expert utilization but breaks down at inference time when you process a single token. Token-choice has no such constraint: each token independently selects $K$ experts regardless of batch size, which is exactly what you need for autoregressive generation.

A Worked Example

Consider a DeepSeek-MoE-style layer with $N = 64$ routed experts, $K = 6$, and $d_{\text{model}} = 4096$. Each expert has intermediate dimension $d_{\text{ffn}}^{\text{expert}} = 1024$ (one-quarter the size of a standard dense FFN with $d_{\text{ffn}} = 4096$).

Token $t$ arrives with hidden state $u_{t} \in \mathbb{R}^{4096}$.

Step 1 - Affinity scores. Compute $z_{i} = u_{t}^{\top} e_{i}$ for all 64 experts. This is a $( 64 \times 4096 )$ matrix times a $( 4096 , )$ vector: $64 \times 4096 \times 2 \approx 524 , 000$ multiply-adds. Apply softmax across the 64 values.

Step 2 - Top-6 selection. Partial-sort the 64 scores; retain indices $\mathcal{T}_{t} = \{ i_{1} , . . . , i_{6} \}$.

Step 3 - Expert forward passes. Each expert is a two-layer MLP with weight matrices $W_{\text{up}} \in \mathbb{R}^{1024 \times 4096}$

and $W_{\text{down}} \in \mathbb{R}^{4096 \times 1024}$. One expert costs $2 \times 4096 \times 1024 \times 2 \approx 16 . 8$M FLOPs. Six experts cost $\approx 100$M FLOPs.

Step 4 - Weighted aggregation.

Compare to a dense equivalent: one FFN with $d_{\text{ffn}} = 4096$ costs $2 \times 4096 \times 4096 \times 2 \approx 67$M FLOPs. Our six fine-grained experts cost $\sim 100$M FLOPs - about 1.5$\times$ the dense cost per token. In exchange, the full model carries $64 \times 2 \times 4096 \times 1024 \approx 536$M FFN parameters, versus the dense model’s $2 \times 4096 \times 4096 \approx 33 . 6$M. That is a 16$\times$ ratio of parameters to FLOP overhead in the FFN component alone.

What You Actually Decide in Practice

• Co-design $K$ with expert granularity. Use $K = 2$ when each expert is full-sized (dense FFN equivalent), as in Mixtral and DBRX. Increase $K$ proportionally when you halve or quarter expert size. If each expert is $1 / m$ the dense FFN size, activating $m / 2$ experts keeps the per-token FLOP cost near parity with the dense model while maximizing the parameter-to-FLOP ratio.

• Compute router logits in float32. Even in bfloat16 training runs, compute the affinity scores and the softmax in float32. The softmax exponential can underflow rare-expert logits to zero in bfloat16, causing those experts to never receive tokens and never recover - effectively zeroing out parameters. The float32 cast is one line of code and costs nothing at scale.

• Do not invest in a heavier router. There is a tempting idea that a two-layer MLP router would route more intelligently than a linear inner product. In practice the gradient signal to the router is weak - you only observe gradients through the $K$ selected experts - so a more complex router does not learn a meaningfully better routing policy. The systems cost of additional router FLOPs is real; the quality benefit is not.

• Decide on renormalization after top-K. Renormalizing the selected gate weights to sum to 1 makes the output a proper convex combination and is slightly more interpretable. Not renormalizing is also fine; the next layer norm absorbs the scale. Some architectures (DeepSeek-V3 with sigmoid gates) renormalize; others (early DeepSeek MoE with softmax gates) do not. Pick one and be consistent.

• Sigmoid at the gate vs. softmax before top-K. Softmax before top-K creates competitive suppression: a single dominant expert can concentrate most of the probability mass before the discrete cut even happens, starving the other selected experts of meaningful gate weight. Sigmoid scores are independent per expert, so the top-$K$ selection is less distorted by outlier logits. DeepSeek-V3’s switch to sigmoid with post-selection renormalization reflects this reasoning.

How This Shows Up in Interviews

• “Walk me through the forward pass of a MoE layer.” A strong answer describes the four steps (affinity scoring, top-$K$ selection, expert forward passes, weighted aggregation with residual) and gives the key equations. Bonus: mention the FLOP count breakdown and why routing cost is negligible.

• “Why not use a soft gate over all experts instead of top-K?” Activating all $N$ experts on every token multiplies the FFN FLOP cost by $N$. Top-$K$ is the mechanism that preserves the FLOP budget of the dense baseline while expanding the parameter count. Without it, you have a dense model with $N$ FFNs and no efficiency advantage.

• Senior follow-up: “How does the router learn, given that top-K is not differentiable?” The short answer: gradients flow only through the $K$ selected experts. The gate weight $s_{i , t}$ is differentiable with respect to the router embeddings $e_{i}$, so the model does learn to rank experts correctly on routed tokens. But experts that are never selected receive no gradient signal and never improve. This is precisely why auxiliary load-balancing losses are not optional - without them, one or two high-scoring experts absorb all traffic, the remaining experts atrophy, and you effectively have a one-expert model with a lot of wasted parameters.

Key Takeaways

• The router is a $( N \times d )$ learned embedding matrix plus an inner product, softmax (or sigmoid), and top-$K$ gate. Total routing cost per token is negligible relative to the expert FFN forward passes.

• Token-choice top-$K$ is the industry-standard routing mechanism in virtually every major production MoE: Mixtral, all DeepSeek variants, Grok, Llama 4, Qwen MoE, OlMoE.

• $K$ and expert size are co-designed: fine-grained (quarter-sized) experts require higher $K$ to maintain FLOP parity with a dense baseline; coarse-grained (full-sized) experts use $K = 2$.

• Always compute router logits in float32, even in bfloat16 training runs, to prevent rare-expert underflow.

• Top-$K$ is non-differentiable; the auxiliary load-balancing loss is what prevents router collapse and keeps all experts active throughout training.

6.2.2 Expert-Choice and Global-Assignment Alternatives

Before token-choice top-$K$ became the consensus, the field seriously explored letting experts pick their own tokens and solving global assignment problems to route optimally. Understanding why those approaches were abandoned matters in interviews: the trade-offs reveal deep constraints about MoE training, and an experienced interviewer will sometimes propose one of these alternatives to test whether you know why the field walked away from them. A mid-level candidate says “token-choice is standard.” A senior candidate explains exactly why the alternatives fail.

Expert-choice routing inverts the selection direction. Instead of each token picking its $K$ favorite experts, each expert picks its $K_{e}$ favorite tokens. Concretely, for a batch of $T$ tokens and $N$ experts, compute the full $N \times T$ affinity matrix $S$ where $S_{i , t} = u_{t}^{\top} e_{i}$. Expert $i$ then selects:

the top-$K_{e}$ tokens by affinity in row $i$ of $S$. Setting $K_{e} = T / N$ yields perfect load balance by construction: every expert processes exactly the same number of tokens in every batch.

Global-assignment routing casts routing as a constrained optimization problem. Given the affinity matrix $S$, find an assignment $A \in \{ 0 , 1 \}^{N \times T}$ that maximizes total affinity:

where $C$ is an expert capacity constraint. Approximate solvers include the Sinkhorn algorithm (iterated row-and-column normalization, $O ( N \cdot T \cdot$iter$)$ per layer per step) and linear-sum assignment via the Hungarian algorithm ($O ( ( N + T )^{3} )$ in the worst case). The method S-BASE (Clark et al., 2020) demonstrated this approach with measurably better token-expert affinity coverage than greedy top-$K$.

Noisy top-$K$ (stochastic routing) is a third alternative: add learned Gaussian noise to router logits before the top-$K$ selection. Shazeer (2017) parameterized this as:

where $W_{\text{noise}}$ controls per-expert noise magnitude and is itself learned. The noise forces exploration: even non-top-$K$ experts receive tokens occasionally, enabling gradient signal to flow to all experts.

All three alternatives sacrifice the token’s agency - the guarantee that each token reaches its highest-affinity experts - to achieve load balance or exploration, and that trade-off costs measurably in model quality or inference compatibility.

Expert-choice routing has one compelling advantage - perfect expert utilization - and two serious drawbacks that together make it impractical.

The first drawback is inference incompatibility. At inference time you process one or a few requests, not a large batch. The “each expert picks top-$K_{e}$ tokens from a batch” formulation breaks down for $T = 1$: every expert would try to select $K_{e} = 1 / N$ of a single token, which is undefined. You are forced either to revert to token-choice at inference (creating a train-test mismatch the model has never seen during training) or to construct an artificial batch at inference and accept the latency cost. Token-choice has no such issue; each token independently selects $K$ experts regardless of batch size.

The second drawback is variable per-token coverage. In expert-choice, a token is processed by however many experts happen to select it, which can be zero. A token with a diffuse affinity vector (high entropy, low peak) may be skipped entirely, with its hidden state propagating through the layer via the residual connection only, receiving no MoE computation at all. Token-choice guarantees every token is processed by exactly $K$ experts, no exceptions.

OlMoE ran the definitive comparison under controlled conditions: the same 7B-class MoE architecture, same training data, same compute budget, only the routing direction differing. Token-choice achieved substantially better validation loss at every measured checkpoint. The gap was consistent and not noise - expert-choice’s load-balance advantage did not compensate for its coverage disadvantage.

Global assignment achieves better affinity coverage than greedy token-choice (the globally optimal assignment, not a per-token greedy one) and better balance than naive token-choice without an auxiliary loss. But solving even an approximate linear program on every forward pass is prohibitively slow. Sinkhorn iterations on a $64 \times 4096$ affinity matrix (64 experts, 4096 tokens per batch) at 10 iterations add roughly 5.2M FLOPs per layer per step. Across 64 layers and millions of training steps, this cost is non-trivial, and the quality gains over token-choice with a simple auxiliary balancing loss are marginal. No production MoE system uses global assignment.

Noisy top-$K$ was ultimately abandoned for a simpler reason: auxiliary-loss balancing achieves comparable exploration with no train-test mismatch. The noise added during training is removed at inference (to be deterministic), but the model has learned routing distributions shaped by noisy training. Auxiliary-loss balancing applies the same uniform-load pressure without introducing any inference divergence.

A Worked Example

Consider a batch of $T = 4096$ tokens processed by $N = 8$ experts, with token-choice $K = 2$ versus expert-choice $K_{e} = 1024$ (each expert selects 1024 of 4096 tokens). Hidden dimension $d = 2048$.

Token-choice: For each of 4096 tokens, compute 8 dot products with router embeddings, take top-2. Routing cost: $4096 \times 8 \times 2048 \times 2 \approx 134$M FLOPs for the affinity matrix (shared by both methods). Result: every token processed by exactly 2 experts; 8192 total expert invocations across the batch.

Expert-choice: Same affinity matrix. Each of 8 experts sorts 4096 scores and selects top-1024. In the ideally balanced case, each token is selected by $8 \times 1024 / 4096 = 2$ experts on average - identical throughput to token-choice. But consider a skewed batch where 512 tokens are from a single domain that all 8 experts prefer. Each expert fills most of its 1024 slots with those tokens. The remaining 3584 tokens compete for 8 $\times 512 \approx 4096$ remaining slots; some get 2 expert invocations, others get 0. A token that gets 0 expert invocations passes through the MoE layer unchanged. In a 32-layer model, that token may miss many expert computations, degrading output quality on an input that contained no unusual or out-of-distribution content.

At inference time with $T = 1$: token-choice routes to exactly $K = 2$ experts, always. Expert-choice requires defining $K_{e}$ for a batch of size 1; $K_{e} = 1 / 8$ is not an integer. The method is undefined without patching.

What You Actually Decide in Practice

• Never use expert-choice if you need single-query inference. This is not a performance trade-off - it is an architectural incompatibility. Any system that serves user requests one at a time cannot use expert-choice routing without a structural workaround that reintroduces token-choice at inference.

• Cite global assignment to show depth, but never ship it. S-BASE and Sinkhorn-based routing are useful interview references demonstrating that you know the design space. Follow immediately with the reason they are not in production: the approximate solver cost is not justified by quality gains over token-choice plus auxiliary loss.

• Hashing is a useful null hypothesis for ablations. Routing tokens to experts by a fixed hash function (no learned affinity, no gradient) still outperforms a dense model at the same FLOP count. This benchmark reveals that the gains from MoE come primarily from having more total parameters and from the implicit data parallelism over the token stream, not from intelligent routing. A weak router that merely prevents collapse is sufficient; heroic routing engineering has diminishing returns.

• Noisy top-$K$ was abandoned in favor of auxiliary-loss balancing. If your concern is expert collapse (all tokens routing to one expert), the auxiliary loss is simpler, more stable, and has no inference-time side effects. Noisy top-$K$ was a reasonable early attempt but is not used in modern releases.

• RL-based routing is principled but impractical. Discrete routing decisions are a natural RL target (the routing choice is an action; the loss improvement is the reward). Clark et al. (2020) benchmarked RL-based routing and found it roughly on par with hashing - not better than top-$K$ with balancing. The gradient variance and compute overhead rule it out for large-scale training.

How This Shows Up in Interviews

• “What is the difference between token-choice and expert-choice routing?” Token-choice: each token selects $K$ best experts (variable per-expert load). Expert-choice: each expert selects $K_{e}$ best tokens (perfect expert utilization, variable per-token coverage). Token-choice wins empirically and is inferencecompatible. Expert-choice is inference-incompatible for single-query serving.

• “Why not solve for the globally optimal routing assignment?” Global assignment (Sinkhorn, linear program) adds a solver step that costs $O ( N \cdot T )$ per layer per forward pass. At training scale, this overhead is non-trivial, and the quality gain over token-choice with auxiliary-loss balancing is marginal. No production system uses it.

• Senior follow-up: “If you had to use expert-choice for some reason, how would you handle inference?” The cleanest approach is to train with expert-choice and switch to token-choice at inference, accepting the train-test distribution shift. To reduce that shift, constrain the expert-choice training objective so that every token is guaranteed at least $K$ expert invocations (solve a minimum-coverage assignment rather than a top-$K$ per expert). In practice this requires a more expensive constrained solver and still produces inference behavior the model has not seen during training. The honest answer is: if you need single-query inference, do not use expert-choice - this is exactly why the field did not adopt it.

Key Takeaways

• Expert-choice routing guarantees load balance but cannot be used for single-query inference, making it incompatible with the vast majority of LLM serving systems.

• Global assignment (Sinkhorn, linear program) is theoretically optimal routing but too slow for production; its quality gains over token-choice with auxiliary balancing are marginal.

• OlMoE ablations confirm token-choice produces cleaner loss curves and better final perplexity than expert-choice under the same compute budget.

• Hashing outperforms dense models even with no learned routing, which tells you the benefit of MoE comes from parameter count and token-stream parallelism, not routing intelligence.

• RL-based and noisy top-$K$ routing were early principled explorations; both were abandoned in favor of the much simpler auxiliary-loss balancing approach that is now universal.