Chapter 7 MoE Training and Systems Considerations

7.1 Fine-Grained and Shared Experts

7.1.1 The DeepSeek MoE Blueprint

When an interviewer asks you to design the FFN structure for a large MoE, the right starting point is not an empty whiteboard - it is the configuration that DeepSeek published and that every subsequent open MoE has copied. The fine-grained expert blueprint from DeepSeek MoE has become the de facto standard because it solves a specific parameter-efficiency problem with a clean arithmetic argument. Knowing the configuration, the numbers, and the FLOP trade-offs cold is the difference between sketching an architecture and actually defending one at the senior engineer level.

The vanilla MoE is straightforward: copy a dense $\text{FFN} N$ times and activate $K$ copies per token. If each copy is the same size as the dense FFN and $K = 1$, the forward FLOP count is unchanged while total parameters multiply by $N$. The catch is that with a small $N$ (say, 8), the parameter multiplier is modest. If you want more parameters, you need more experts - but more full-size experts means more memory, more communication, and diminishing returns once you have more experts than training tokens can specialize them.

Fine-grained experts cut through this directly. The insight: shrink each expert to $1 / m$ of the standard FFN intermediate dimension and have $m$ times as many experts. Total parameter count stays the same, but now you have $m$ times more distinct expert FFNs. You then activate $K$ of them per token, setting $K$ large enough so that the activated FFN capacity stays near the dense baseline.

Formally, let $d_{\text{model}}$ be the hidden dimension and $d_{\text{ffn}} = 4 d_{\text{model}}$ be the standard intermediate dimension. With granularity factor $m$, each fine-grained expert has intermediate dimension $d_{\text{expert}} = d_{\text{ffn}} / m$. Total experts: $N_{\text{routed}} = m \cdot N_{\text{dense}}$, where $N_{\text{dense}}$ is however many full-size experts you would otherwise have. Activated routed experts: $K$. To maintain roughly the same activated FLOP count as the dense model:

In practice $K$ is set slightly below $m$ (because shared experts, covered in the next subsection, absorb part of the capacity), but this relation is the governing design equation.

Fine-grained experts trade expert size for expert count at fixed activated FLOP cost, giving a multiplicative parameter expansion that scales as the granularity factor $m$ while leaving the per-token computation budget nearly unchanged.

More distinct FFN parameter matrices mean more distinct processing pathways. The scaling argument is empirical but robust: at fixed training FLOPs, adding more expert parameters consistently lowers validation loss, and fine-grained experts are the mechanism that lets you add those parameters without inflating activated compute.

There is also a combinatorial argument. With $N = 8$ full-size experts and $K = 2$, the model can form

$=^{\binom{8}{2}} 28$ distinct expert combinations per token. With $N = 64$ quarter-size experts and $K = 6$, the same token can be processed by$\approx^{\binom{64}{6}} 74 , 000 , 000$ distinct combinations - an enormous expansion of the expressible function space at the same FLOP budget. The router learns to navigate this larger space, and empirically it does so successfully.

The architecture also scales cleanly. DeepSeek MoE v1 (16B total, 2.8B active), DeepSeek-V2 (236B total, 21B active), and DeepSeek-V3 (671B total, 37B active) all use the same structural blueprint: fine-grained experts of the same relative size, the same granularity factor, the same routing mechanism. What changed across versions was only the absolute parameter count and number of experts, not the architecture. This stability is itself strong evidence that the blueprint is correct.

A Worked Example

Take a single MoE layer with $d_{\text{model}} = 2048$, standard dense FFN intermediate dimension $d_{\text{ffn}} = 8192$ ($4 \times$ hidden), and granularity factor $m = 4$. Each fine-grained expert has $d_{\text{expert}} = 2048$.

Total parameter count per MoE layer. Each expert is a two-layer MLP with $W_{\text{up}} \in \mathbb{R}^{2048 \times 2048}$ and $W_{\text{down}} \in \mathbb{R}^{2048 \times 2048}$:

With 64 routed + 2 shared = 66 experts: $66 \times 8 . 4$M $\approx 554$M FFN parameters per layer.

Dense equivalent: $2 \times 2048 \times 8192 = 33 . 6$M per layer. The fine-grained MoE has $554 / 33 . 6 \approx 16 . 5 \times$ more FFN parameters per layer.

Activated FLOP count per token. Active experts: 6 routed + 2 shared = 8 total. One expert forward pass:

Eight active experts: $8 \times 16 . 8$M $\approx 134$M FLOPs.

Dense equivalent FFN: $2 \times 2048 \times 8192 \times 2 \approx 67$M FLOPs. The fine-grained MoE costs $\approx 2 \times$ the dense FFN FLOPs per token while carrying $16 . 5 \times$ the FFN parameters - a 8$\times$ improvement in the parameter-to-FLOP ratio relative to the dense baseline.

Configuration summary at a glance: 64 routed + 2 shared experts, $K = 6$ active routed, each expert is $1 / 4$ the size of a standard dense FFN. This is the DeepSeek MoE v1 prototype configuration, replicated (with different absolute sizes) in every subsequent release.

What You Actually Decide in Practice

• Start at $m = 4$ (quarter-size experts) as your baseline. Finer granularity ($m = 8$, $m = 16$) gives more total parameters but shrinks the per-expert matrix below the efficient tile size for H100 tensor cores (roughly $128 \times 128$ in bfloat16). Below an intermediate dimension of $\sim$512 for $d_{\text{model}} = 4096$, matrix multiply efficiency degrades measurably. OlMoE, Llama 4, and DeepSeek all operate at $m = 4$ or close to it.

• Set $K$ so that activated FFN capacity is 1.5-2$\times$ the dense baseline. At $K = m$ you match the dense FLOP budget exactly (all activated experts combined equal one standard FFN). Including 1-2 shared experts pushes you to 1.5-2$\times$. Staying below $2 \times$ keeps the MoE competitive with a dense model of the same total training compute.

• Keep all fine-grained experts the same size. Mixed expert sizes force variable-length matrix multiply kernels that kill GPU utilization. Every expert in the routed pool and every shared expert should have the same $d_{\text{expert}}$. Uniform expert size is a hard engineering requirement, not an aesthetic preference.

• Scale total experts with model size, not just parameter count. DeepSeek MoE v1 uses 64 routed experts at 16B total parameters; DeepSeek-V3 uses 256 routed experts at 671B. The ratio is sub-linear, not proportional. Expert count is bounded by memory (each expert’s weights must fit on some device) and communication overhead (more experts $\Rightarrow$ more all-to-all traffic). A practical ceiling around 256 experts appears in current large-scale deployments.

• Do not increase $K$ without a FLOP budget analysis. Every additional active expert adds $2 \times d_{\text{model}} \times d_{\text{expert}} \times 2$ FLOPs per token per layer. At 1T training tokens and 32 layers, each added active expert costs roughly FLOPs $= 2 \times d_{\text{model}} \times d_{\text{expert}} \times 2 \times 10^{12} \times 32$ total compute. For $d_{\text{model}} = 4096$, $d_{\text{expert}} = 1024$: that is $\sim$5 $\times 10^{20}$ FLOPs - half a zettaFLOP per extra active expert. Treat $K$ as a compute budget line item.

How This Shows Up in Interviews

• “How would you design the FFN structure for a 7B active parameter MoE?” Anchor to fine-grained experts: pick $m = 4$, 64 total routed experts at $1 / 4$-size, $K = 6$ active routed plus 1-2 shared. Walk through the FLOP arithmetic: $8 \times 2 \times d_{\text{model}} \times d_{\text{expert}} \times 2 \approx 2 \times$ the dense FFN FLOPs at $\sim$16$\times$ the FFN parameters. Name the DeepSeek MoE configuration as the template.

• “What is the parameter efficiency gain from fine-grained experts versus vanilla MoE?” Vanilla MoE with $N = 8$ full-size experts and $K = 2$ gives $8 \times$ FFN parameters at $2 \times$ FFN FLOPs. Fine-grained $m = 4$, $N = 64$, $K = 6$ gives $\approx 16 \times$ FFN parameters at the same $\sim 2 \times$ FFN FLOPs. The parameter-to-activated-FLOP ratio doubles.

• Senior follow-up: “What limits expert granularity, and where does DeepSeek’s $m = 4$ come from?” Two limits: (1) Tensor core efficiency. Matrix multiplies are efficient when the inner dimension is a multiple of the GEMM tile size; below $\sim$512 for H100 in bfloat16, MFU drops. At $d_{\text{model}} = 4096$ and $m = 4$, $d_{\text{expert}} = 1024$, which is comfortably above this floor. At $m = 8$ you get $d_{\text{expert}} = 512$ - still acceptable but tighter. At $m = 16$ you get 256, which starts to hurt. (2) Expert parallelism communication. More experts means more all-to-all calls with smaller payloads per call, increasing the ratio of communication to compute. At $m = 4$ the compute-to-communication ratio is favorable on InfiniBand or NVLink clusters; at $m = 16$ it often is not.

Key Takeaways

• Fine-grained experts shrink each FFN to $1 / m$ of the standard intermediate dimension and multiply expert count by $m$, giving $m$ times more total FFN parameters at roughly the same activated FLOP cost per token.

• The canonical configuration from DeepSeek MoE: $m = 4$ (quarter-size experts), 64 routed + 2 shared, $K = 6$ active routed. This blueprint is used verbatim or with minor variation in OlMoE, Llama 4, MiniMax, and DeepSeek-V2/V3.

• Activated FLOP overhead: roughly $2 \times$ the dense FFN cost with the canonical configuration - a small premium for a large parameter expansion.

• Expert granularity is hardware-constrained: below $d_{\text{expert}} \approx 512$ for typical hidden dimensions, tensor core utilization degrades and expert parallelism communication overhead grows.

• The MoE architecture was fixed at DeepSeek MoE v1; scaling to 671B total parameters in V3 required only changing expert count and model width, not restructuring the blueprint.

7.1.2 When Shared Experts Help and When They Don’t

An interviewer will ask you to defend adding a shared expert to your MoE design, and the honest answer is nuanced: the DeepSeek MoE paper showed gains; the OlMoE independent replication found none. If you answer with a confident “shared experts always help” or “shared experts don’t matter,” you reveal that you read one paper without understanding what the disagreement tells you about the mechanism. The right answer explains when the gain appears and gives a principled decision rule for a new architecture.

A shared expert is an MLP that processes every token, unconditionally, in addition to whatever routed experts the router selects. In a standard fine-grained MoE layer, the output is:

With $n_{s}$ shared experts added, this becomes:

There is no gate on the shared experts. Every token, regardless of content, passes through all $n_{s}$ shared MLPs. The shared experts are the same size as the fine-grained routed experts, so all active computation uses uniform matrices - important for kernel efficiency.

The gradient structure differs fundamentally from routed experts. A routed expert receives gradient signal only from the tokens the router sends to it, which under balanced routing is $K / N_{\text{total}}$ of all tokens. A shared expert receives gradient from every token in every batch - a dense, high-signal gradient at every step. This means shared experts converge faster and to a more stable minimum than any individual routed expert.

A shared expert is a fully trained dense FFN embedded inside a sparse MoE layer; it handles whatever processing is needed for every token, leaving the routed pool to specialize on the residual token-specific transformation.

The hypothesis is that some MLP transformations are genuinely universal across all token types - common syntactic structures, frequency-heavy vocabulary items, low-level pattern normalization - that every routed expert would otherwise need to partially replicate. By factoring this shared component out, routed experts can specialize more cleanly on the remaining token-specific work, improving effective capacity per routing decision.

This argument is compelling in theory. The empirical record is split. The original DeepSeek MoE paper reports consistent benchmark improvements from adding one shared expert (orange bars) versus no shared expert (blue bars), with further gains on some tasks from a second shared expert. But these are not controlled ablations over a single variable; the shared expert configuration also affects how the authors set $K$ and $N$.

OlMoE ran a cleaner ablation: same architecture, same training run, only varying the presence of shared experts. Their result (comparing purple versus teal validation loss curves across training) shows no significant difference at their scale - roughly a 7B total parameter model. This is a careful third-party test with no confounds, and it found zero gain.

The reconciliation is likely scale and data. At DeepSeek’s pre-training scale (hundreds of billions of tokens, models with 16B-671B total parameters), routed experts are heavily underfit early in training. An expert receiving $1 / 64$ of all tokens takes far longer to converge than a shared expert seeing 100% of tokens. The shared expert acts as a well-converged baseline that stabilizes learning for the underfitted routed experts. At OlMoE’s smaller scale, routed experts converge faster relative to training time, and the stabilizing benefit disappears.

A Worked Example

Consider a 7B active-parameter MoE with $d_{\text{model}} = 4096$, $d_{\text{expert}} = 1024$ (quarter-size experts), 64 routed experts, and $K = 6$ active routed. Compare three configurations:

Config A: 0 shared experts, $K = 6$ routed. Activated FLOPs per token (FFN component): $6 \times 2 \times$

Config C: 2 shared experts, $K = 6$ routed (8 active total), matching DeepSeek MoE v1. Activated FLOPs: $8 \times 100$M$/ 6 \approx 134$M ($+ 34 \%$). Total FFN parameters: $66 \times 8 . 4$M $= 554$M ($+ 3 \%$).

The FLOP cost of shared experts is unconditional and permanent. Across 1T training tokens and 32 layers, moving from Config A to Config C adds:

That is 1 exaFLOP of training compute - roughly 10% of the total compute for a mid-size training run. If that exaFLOP delivers measurably better validation loss, it is well spent. If it does not (OlMoE’s finding), it is a 10% tax with no return.

What You Actually Decide in Practice

• Default to 0 shared experts; add 1 only if ablations show favorable gain-per-FLOP. OlMoE’s result is a strong null hypothesis. Do not start with shared experts because DeepSeek used them. Run a 1B-scale ablation for 100B tokens comparing 0 shared vs. 1 shared vs. “1 more routed expert” at the same total FLOP cost. If the shared expert wins the gain-per-FLOP comparison, add it. If adding a routed expert wins, add that instead.

• Size shared experts identically to routed experts. If your routed experts have $d_{\text{expert}} = 1024$, the shared experts should also have $d_{\text{expert}} = 1024$. Matching sizes means the same matrix multiply kernel handles all active experts uniformly. A differently-sized shared expert creates a two-kernel path that complicates the implementation with no clear upside.

• The 2-shared configuration in DeepSeek MoE is partly motivated by accounting symmetry. With 2 shared + 6 routed, all 8 active experts are equal-size, and the total activated capacity is $8 \times d_{\text{expert}} = 2 \times d_{\text{ffn}}$ - exactly twice the dense FFN capacity. This is convenient bookkeeping but not a principled reason to prefer 2 over 1. Llama 4 uses 1 shared expert; some Chinese releases tried more and retreated to 0 or 1.

• At $\geq 100$B total parameters, shared experts become more worth testing. The mechanism - stabilizing slow-to-converge routed experts - is proportionally stronger when experts see fewer tokens relative to their parameter count. A model with 256 routed experts and 1T training tokens has each expert seeing roughly $10^{12} / 256 \approx 4 \times 10^{9}$ tokens, which is already tight for full convergence of an MLP. A shared expert seeing all $10^{12}$ tokens provides a well-converged anchor.

• Do not freeze or treat shared experts differently from routed experts. They are standard parameters trained with the same optimizer, learning rate, and weight decay as every other weight in the model. The only difference is the absence of a routing gate. Freezing shared experts after warm-up, or using a different learning rate, introduces complexity with no theoretical justification.

How This Shows Up in Interviews

• “Why would you add a shared expert to a MoE layer?” The hypothesis: some MLP-level transformations are universal across token types; factoring them into a dedicated always-on expert allows routed experts to specialize more purely. Shared experts also receive a dense gradient (every token, every step), so they converge faster and act as a stable baseline during early training when routed experts are underfit.

• “OlMoE found no gain from shared experts but DeepSeek did. How do you reconcile that?” Scale and training budget. DeepSeek’s models are 16B-671B total parameters trained on hundreds of billions of tokens; OlMoE is $\sim$7B total. At larger scale, each routed expert sees fewer tokens relative to its capacity, leaving more useful gradient for a shared expert to absorb. At OlMoE’s scale, routed experts converge fast enough that a shared component adds nothing net. This is an empirically testable hypothesis: the crossover point is somewhere between 7B and 16B total parameters.

• Senior follow-up: “How would you run the ablation to decide whether to include a shared expert in a new MoE?” Train three 1B-parameter models for 100B tokens each, holding all hyperparameters fixed and varying only: (a) 0 shared, 64 routed, $K = 6$; (b) 1 shared, 64 routed, $K = 6$ ($+ 17 \%$ FFN FLOPs); (c) 0 shared, 64 routed, $K = 7$ (same $+ 17 \%$ FLOPs via one extra active routed expert). Compare held-out perplexity against the FLOP budget. If (b) beats (c), the shared expert is worth its cost. If (c) beats (b), adding a routed expert is the better use of that compute, and you skip the shared expert. This design is the right frame because it controls for the FLOP budget change, which the original DeepSeek ablation did not cleanly do.

Key Takeaways

• A shared expert is an MLP that always fires for every token, with no routing gate; it receives dense gradient signal at every training step, unlike routed experts which are each gated by the top-$K$ selection.

• Empirical evidence is split: DeepSeek MoE reports gains; OlMoE’s controlled ablation finds none. The likely explanation is scale - shared experts help when routed experts are underfit relative to training token budget.

• Each shared expert unconditionally adds $2 \times d_{\text{model}} \times d_{\text{expert}} \times 2$ FLOPs per token; at 1T training tokens and 32 layers, that is $\sim$1 exaFLOP per shared expert for typical model sizes. Budget this explicitly.

• Default recommendation: start with 0 shared experts; ablate against the equivalent FLOP spent on an additional active routed expert before committing to a shared expert.

• Size shared experts identically to routed experts for kernel uniformity; train them end-to-end with no special treatment.

7.2 Load Balancing and Stability

7.2.1 Auxiliary Balancing Losses

Every MoE training run I have been involved with hit the same failure mode within the first few thousand steps when load balancing was omitted: one or two experts absorbed almost all the tokens, the rest went dark, and the model ended up as a very expensive dense model wearing a MoE costume. The auxiliary balance loss is the mechanism that prevents this collapse, and understanding its gradient mechanics - why it is differentiable when the routing decision is not, and why it must be tuned carefully - is the core MoE training question at the research engineer level.

Without any intervention, token-choice top-$K$ routing falls into a degenerate local minimum. A randomly initialized router has slightly different affinities for each expert. One expert wins marginally, gets routed more tokens, receives more gradient signal, improves faster, attracts even more tokens, and the cycle compounds. The OlMoE ablation is definitive: with 8 experts and no balancing, 2 experts absorbed roughly $50 \%$ of all tokens within early training; the remaining 6 were effectively dormant. The model had silently reduced itself to a 2-expert MoE while paying for 8.

The auxiliary balance loss breaks this cycle. For $N$ experts and a batch of $T$ tokens, define two vectors:

where $\mathcal{T}_{t}$ is the set of top-$K$ experts selected for token $t$ and $s_{i , t}$ is the pre-top-$K$ softmax score. $f_{i}$ is the fraction of tokens hard-routed to expert $i$ (discrete, non-differentiable); $P_{i}$ is the average router probability assigned to expert $i$ (soft, differentiable). The auxiliary loss is:

with small coefficient $\alpha$ (typically $10^{- 4}$ to $10^{- 2}$). The gradient with respect to $P_{i}$ is:

The overloaded expert (large $f_{i}$) receives the strongest downward gradient on its router probability. The underloaded expert (small $f_{i}$) receives a weaker penalty and naturally attracts more traffic over time. Crucially, $f_{i}$ acts as a fixed multiplier - the gradient flows only through the differentiable $P_{i}$, yet the penalty is proportional to actual load.

Beyond per-expert balancing, you need a parallel device-level loss when using expert parallelism. Replace $f_{i}$ with $f_{d}$, the fraction of tokens routed to GPU $d$:

where $P_{d} = \sum_{i \in \text{device} d} P_{i}$ is the cumulative router probability for device $d$. Expert-level balance ensures model quality; device-level balance ensures training throughput.

A third stabilizer is the z-loss on router logits. The softmax normalizer $Z_{t} = \sum_{i} \exp ( z_{i , t} )$ can grow unchecked if router logits drift large, leading to numerical instability and loss spikes, especially in bfloat16 where the precision is limited. The z-loss penalizes large normalizers:

Removing the z-loss from a trained MoE produces visible validation loss spikes and measurably worse final perplexity in controlled ablations.

The auxiliary balance loss is not optional: without it, an $N$-expert MoE silently degrades into a 1-2 expert model, wasting the majority of its parameter budget and producing a worse model than a smaller, correctly trained dense baseline.

The $f_{i} \cdot P_{i}$ formulation exploits a key asymmetry. $f_{i}$ is the hard routing fraction - non-differentiable because it depends on the discrete top-$K$ decision. But it acts as a fixed multiplier on the differentiable $P_{i}$, so the gradient flows through $P_{i}$ all the way back to the router embeddings $e_{i}$. An overloaded expert’s affinity scores are suppressed; an underloaded expert’s affinity scores are preserved, and its relative attractiveness grows as competitors are pushed down. Over thousands of training steps, the routing distribution approaches uniformity without requiring any explicit constraint solver.

The alternative approaches fail in practice. RL-based routing uses the correct abstraction (discrete decision, policy gradient), but gradient variance is too high and compute overhead too large. Global assignment (optimal transport, Sinkhorn) finds the best matching but adds latency per forward pass that overwhelms the quality gain. Noisy top-$K$ (adding learned Gaussian noise to logits) helps with exploration but hurts specialization and creates train-test mismatch. The auxiliary loss wins by being cheap (negligible FLOPs), differentiable (through $P_{i}$), and tuneable (via $\alpha$).

The z-loss complements the balance loss by addressing a different failure mode. The balance loss controls which expert receives each token; the z-loss controls the scale of the affinity scores themselves. Together they address the two ways MoE training can go wrong: routing collapse and numerical instability.

A Worked Example

Consider $N = 8$ experts and a batch of $T = 1024$ tokens. After 2000 training steps with no balancing loss, the router has collapsed: expert 0 receives 512 tokens, expert 1 receives 384 tokens, and experts 2-7 receive a combined 128 tokens.

Computed fractions: $f_{0} = 0 . 500$, $f_{1} = 0 . 375$, $f_{2} = \cdot \cdot \cdot = f_{7} \approx 0 . 018$.

Introducing the auxiliary loss with $\alpha = 0 . 01$, $N = 8$:

Gradient on expert 0: $\partial \mathcal{L} / \partial P_{0} = 0 . 01 \times 8 \times 0 . 500 = 0 . 040$.

Gradient on expert 7: $\partial \mathcal{L} / \partial P_{7} = 0 . 01 \times 8 \times 0 . 018 = 0 . 00144$.

Expert 0’s affinity is pushed down with a gradient $27 \times$ stronger than expert 7’s. Over subsequent steps, expert 0 loses tokens to experts 2-7, which now receive some gradient signal and begin to specialize. The target balanced state has $f_{i} = 1 / N = 0 . 125$ for all $i$, giving:

which is small relative to a typical next-token language modeling loss of $\sim 2$-$3$ nats. At balance, the auxiliary loss contributes less than $0 . 5 \%$ of total gradient magnitude and stops fighting the routing distribution.

For the z-loss: with $\beta = 10^{- 3}$ and router logits that have drifted to a maximum of $z_{\max} = 5$, the normalizer is $Z_{t} \leq N e^{5} = 8 \times 148 \approx 1200$, giving $\mathcal{L}_{z} \approx 10^{- 3} \times ( \log 1200 )^{2} \approx 0 . 053$. After the z-loss pulls logits back to $z_{\max} \approx 2$, the normalizer drops to $Z_{t} \approx 8 \times 7 . 4 \approx 59$, and $\mathcal{L}_{z} \approx 10^{- 3} \times ( \log 59 )^{2} \approx 0 . 017$ - a 3$\times$ reduction in the stability penalty, with logits now safely away from the bfloat16 precision limit.

What You Actually Decide in Practice

• Set $\alpha$ in $[ 10^{- 3} , 10^{- 2} ]$ for pre-training; decay toward $10^{- 4}$ once utilization is healthy. At $\alpha = 10^{- 2}$, the balance loss gradient is comparable to the language modeling gradient for a severely collapsed router. At $\alpha = 10^{- 4}$, it is a light touch that maintains balance without distorting routing. Monitor the expert utilization histogram (fraction of steps where each expert is the bottom or top decile); if variance is low, decay $\alpha$.

• Always add the device-level loss when using expert parallelism. Expert-level balance does not imply device-level balance. With 64 experts on 8 GPUs, even uniform per-expert load can still result in one GPU handling a disproportionate share of a given batch if the popular experts happen to co-locate. The device-level loss costs almost nothing to compute (aggregate $f_{i}$ into device buckets) and prevents GPU utilization hot spots that bottleneck the entire cluster.

• Include the z-loss from step 1, coefficient $\beta \approx 10^{- 3}$. Loss spikes from unconstrained router logits are hard to recover from and can contaminate checkpoints mid-run. The z-loss is cheap and its effect is measurable: removing it produces $\sim$0.1-0.2 nats worse final validation perplexity in published ablations. Do not wait to see instability before adding it.

• Compute router logits in float32, always. In bfloat16, two logits that differ by less than $2^{- 7}$ (the machine epsilon at large magnitude) round to the same value, causing artificial ties in the affinity ranking. These ties resolve arbitrarily, adding stochasticity to what should be a deterministic top-$K$ decision. Cast to float32 before the inner product and softmax; cast back to bfloat16 after the gate weights are computed.

• Watch for token dropping and treat it as a signal of imbalance. When an expert’s load factor (tokens received / capacity limit) exceeds 1.0, tokens are dropped: they pass through the layer via the residual connection with no expert computation. Token dropping at inference creates batch-level stochasticity (whether a token is dropped depends on who else is in the batch). Sustained dropping during training indicates the auxiliary loss is under-tuned; increase $\alpha$ temporarily.

How This Shows Up in Interviews

• “What happens if you train a MoE without any load balancing?” Router collapse to 1-2 experts occurs within thousands of steps. The remaining experts receive no gradient signal and remain near random initialization. You end up with a dense model of size $K \times d_{\text{expert}}$ carrying the dead weight of $N - K$ additional expert parameter sets, worse than a correctly-sized dense model at the same FLOP budget.

• “Explain the Switch Transformer auxiliary loss and why it is differentiable when the routing decision is not.” Define $f_{i}$ as the hard token fraction (non-differentiable) and $P_{i}$ as the soft router probability (differentiable). The loss is $\alpha N \sum_{i} f_{i} P_{i}$. The gradient $\partial \mathcal{L} / \partial P_{i} = \alpha N f_{i}$ flows through $P_{i}$ to the router embeddings, with $f_{i}$ serving as a load-proportional multiplier. No gradient passes through the discrete top-$K$ decision itself.

• Senior follow-up: “Why do you need both a per-expert and a per-device balancing loss, and can one subsume the other?” They address different failure modes that are independent. Expert-level balance ensures uniform training exposure across all parameter sets - a model quality concern. Device-level balance ensures uniform GPU utilization - a throughput concern. With $k$ experts per GPU, even perfect per-expert balance allows a GPU to be overloaded if all $k$ of its experts are popular on a given batch. The device-level loss catches this and prevents one GPU from becoming the critical path in the all-to-all dispatch. Neither loss can be derived from the other; both are required when using expert parallelism.

Key Takeaways

• Without a balance loss, routing collapse to 1-2 experts is universal and fast; you silently lose most of the parameter budget you paid for.

• The auxiliary balance loss $\mathcal{L} = \alpha N \sum_{i} f_{i} P_{i}$ is differentiable through $P_{i}$ even though $f_{i}$ is not; the hard routing fraction acts as a fixed, load-proportional multiplier on the router gradient.

• Include a device-level version of the same loss when using expert parallelism; expert-level balance does not imply device-level balance.

• Z-loss ($\mathcal{L}_{z} = \beta ( \log Z_{t} )^{2}$) prevents router logits from drifting large; removing it causes loss spikes and measurably worse final perplexity.

• Start with $\alpha \approx 10^{- 2}$, compute router logits in float32, add z-loss from step 1, and monitor expert utilization histograms throughout pre-training.

7.2.2 Auxiliary-Loss-Free Balancing (DeepSeek-V3)

After running long MoE pre-training jobs, a pattern becomes clear: the auxiliary balance loss is doing something uncomfortable. Every gradient step, it actively penalizes the routing the language model would naturally prefer, pushing away from the best expert and toward the least-loaded one. DeepSeek-V3’s bias update scheme decouples load control from loss minimization entirely - the optimizer sees only the language modeling gradient, while a separate online controller handles balance. This design shows up in senior-level interviews as the question that distinguishes engineers who understand auxiliary losses from those who have thought about what they actually cost.

The standard auxiliary loss creates a constant tension between two objectives: route to the best expert (language modeling) and route uniformly (balance). These objectives are not aligned, and every training step the balance loss gradient steers the router away from whatever distribution the language model would prefer. At large scale with many experts, this interference becomes harder to tune and can destabilize training.

DeepSeek-V3 replaces batch-level auxiliary balancing with a per-expert bias $b_{i} \in \mathbb{R}$ for each expert $i = 1 , . . . , N$. The routing decision uses the biased score:

where $s_{i , t} =$ Sigmoid$( u_{t}^{\top} e_{i} )$ is the base affinity. Top-$K$ selection is done on $\tilde{s}_{i , t}$. Critically, the gate weight in the output sum uses $s_{i , t}$ only, not $\tilde{s}_{i , t}$:

where $\mathcal{T}_{t} =$ TopK$( \{ \tilde{s}_{i , t} \} , K )$. The bias adjusts who gets selected; the gate weight is determined by the unbiased learned affinity. Output quality is therefore governed by the model’s actual competence estimates, not by load corrections.

The $b_{i}$ values are updated by a simple online rule at the end of each batch, outside of backpropagation:

where $\gamma > 0$ is a fixed step size. This is not gradient descent. It is a feedback controller: track load, increment the bias of underloaded experts, decrement the bias of overloaded experts. No loss term, no interaction with the optimizer, no gradient interference.

Despite claiming to be “auxiliary loss-free,” DeepSeek-V3 also includes a complementary sequence-wise auxiliary loss with a small coefficient $\beta_{\text{seq}}$. The batch-level controller handles average balance across the training corpus; the sequence-wise loss ensures that within a single sequence, no expert is overwhelmed by an unusual token distribution. At inference time, a single long sequence with atypical content can still saturate an expert even when batch-level balance is perfect. The sequence-wise loss provides this per-inference guarantee.

The key innovation is treating load balancing as an operations problem solved by a feedback controller, not as a learning problem penalized through the loss, leaving the gradient optimizer free to find the best routing distribution without a conflicting balance objective.

The standard auxiliary loss conflates two objectives with fundamentally different structures. Language modeling is a differentiable learning problem with a clear signal: minimize prediction error. Load balancing is a logistics problem: equalize throughput across $N$ parallel compute units. Conflating them means the optimizer receives a gradient that is a mixture of these two objectives, and the mixture coefficient $\alpha$ must be tuned across every model size, expert count, and data distribution.

The bias controller handles the logistics problem with a logistics tool - a feedback control loop that observes the current state (token counts), measures the error (deviation from $T / N$), and applies a corrective signal ($\pm \gamma$). This is more appropriate than gradient descent for a problem with no loss landscape to optimize, just a constraint to satisfy.

The practical benefit is cleaner gradient signal for the router. The router embeddings $e_{i}$ are trained exclusively by language modeling gradients; every gradient step is an improvement in routing quality, not a compromise between quality and balance. The controller handles balance externally, so the optimizer is free to find the best routing distribution the data supports.

There is a cost: the online controller introduces a step size hyperparameter $\gamma$ and requires global visibility of per-expert token counts (an all-reduce over data-parallel replicas). These are manageable engineering concerns. The bigger limitation is that the controller responds to average load over time; a single batch with extreme imbalance is not corrected within that batch - only subsequent batches benefit. This is why the sequence-wise auxiliary loss is retained: it provides within-sequence balance that the controller cannot supply.

A Worked Example

Consider $N = 64$ experts, a batch of $T = 2048$ tokens, target load $T / N = 32$ tokens per expert, and $\gamma = 0 . 001$.

After a single batch, expert 5 received 90 tokens (overloaded by 58) and expert 30 received 8 tokens (underloaded by 24).

Bias updates:

• Expert 5: $b_{5} \leftarrow b_{5} - 0 . 001$

• Expert 30: $b_{30} \leftarrow b_{30} + 0 . 001$

Suppose the imbalance persists for 500 consecutive batches at the same severity. After 500 steps:

Expert 5’s effective routing score is reduced by 0.5 across all tokens; expert 30’s is increased by 0.5. For context, the base affinity scores $s_{i , t} =$ Sigmoid$( u_{t}^{\top} e_{i} )$ are in $[ 0 , 1 ]$. A bias shift of $\pm 0 . 5$ is a large correction - sufficient to move expert 30 from below the top-$K$ threshold into the selected set for many tokens, even without any change to the router embeddings $e_{i}$.

Crucially, the gate weight for token $t$ routed to expert 30 is still $s_{30 , t} \in [ 0 , 1 ]$, not $s_{30 , t} + 0 . 5$. If expert 30 has low affinity (e.g., $s_{30 , t} = 0 . 1$), its contribution to the output is $0 . 1 \times \text{FFN}_{30} ( u_{t} )$ - a small but nonzero contribution. The model quality is determined by whether $\text{FFN}_{30}$ has learned anything useful for this token, not by how much the controller has inflated expert 30’s routing score.

Now compare to the auxiliary loss approach: with $\alpha = 0 . 01$ and the same imbalance, the gradient pushes $P_{5}$ down with magnitude $0 . 01 \times 64 \times ( 90 / 2048 ) = 0 . 028$ per step. This gradient modifies the router embeddings $e_{5}$, changing how expert 5 appears to all tokens, not just those in the overloaded batch. The interference with the language modeling gradient is direct and persistent.

What You Actually Decide in Practice

• Set $\gamma$ so that severe imbalance corrects over $\sim$1000 steps, not 10. A step size that corrects imbalance in 10 steps will oscillate: expert 5 gets $b_{5} = - 0 . 5$ in 500 steps, the router avoids it, now expert 5 is underloaded, $b_{5}$ climbs back, and the cycle repeats with period $\sim$1000 steps. Target a correction timescale of several thousand steps; values around $\gamma = 10^{- 3}$ to $10^{- 4}$ are typical for normalizing $b_{i}$ relative to affinity scores in $[ 0 , 1 ]$.

• Do not include $b_{i}$ in the gate weight. This is the most critical implementation detail. If $\tilde{s}_{i , t} = s_{i , t} + b_{i}$ is used both for selection and as the gate weight, then a heavily biased expert receives artificially inflated gate weights and dominates the output sum even if its learned affinity is low. The output quality becomes dependent on load history, which is nonsensical. Use $s_{i , t}$ for the gate, $\tilde{s}_{i , t}$ only for the top-$K$ selection.

• The “auxiliary loss-free” claim is accurate for batch-level balance, not total. DeepSeek-V3 retains a sequence-wise auxiliary loss with coefficient $\beta_{\text{seq}} \approx 3 \times 10^{- 4}$ (roughly $3$-$10 \times$ smaller than a typical batch-level coefficient). The sequence-wise loss handles within-inference imbalance that the controller cannot address. This is a meaningful engineering improvement over the prior art - batch-level gradient interference is eliminated - but read the fine print before claiming the model is fully loss-free.

• Aggregate $f_{i}$ counts across data-parallel replicas before each controller update. If each data-parallel replica updates $b_{i}$ independently based on its local shard of the batch, replicas will apply different corrections to the same $b_{i}$, causing divergence. Use an all-reduce (sum and divide by the number of replicas) over the per-expert token counts before computing the update. This is a single collective call per step with a payload of $N$ integers - negligible overhead.

• Use this approach when $N \geq 64$; for smaller $N$, the standard auxiliary loss is simpler. The interaction between auxiliary loss gradients and language modeling gradients becomes harder to tune as $N$ grows, because the balance loss gradient is spread across more experts with weaker per-expert signal. For $N \leq 16$ (Mixtral-style MoEs), the standard auxiliary loss is well understood and easily tuned. The bias controller is most valuable at $N \geq 64$.

How This Shows Up in Interviews

• “How does DeepSeek-V3 achieve load balance without an auxiliary loss?” Per-expert bias scalars $b_{i}$ are added to routing scores for top-$K$ selection but excluded from gate weights. An online controller updates $b_{i}$ by $\pm \gamma$ each step based on whether expert $i$ was over or under-loaded in the previous batch. No gradient flows through this update; the optimizer sees only language modeling gradients.

• “Why is it important to exclude the bias from the gate weight computation?” The bias is a loadcontrol signal, not a quality signal. If it were included in the gate weight, an expert with inflated $b_{i}$ (promoted to compensate for low affinity) would also receive a high gate weight, amplifying its output regardless of whether its FFN has learned anything useful for those tokens. Keeping the gate weight $= s_{i , t}$ ensures that output quality is governed by learned competence, not load management history.

• Senior follow-up: “DeepSeek-V3 claims to be auxiliary loss-free but still includes a sequence-wise auxiliary loss. Is this a meaningful distinction or marketing?” It is a meaningful engineering distinction, though the terminology overstates the case. The batch-level auxiliary loss is what causes gradient interference: it has a large coefficient, acts every batch, and its gradient direction conflicts with the language modeling gradient in proportion to $f_{i}$. Removing it cleanly separates the two objectives. The sequence-wise loss has a coefficient $3$-$10 \times$ smaller, targets a different failure mode (within-sequence imbalance at inference time), and creates far less interference because routing distributions within a single sequence are already partially balanced by the learned router. The claim is accurate in engineering terms even if incomplete in literal terms.

Key Takeaways

• DeepSeek-V3 replaces the batch-level auxiliary loss with per-expert bias scalars $b_{i}$ updated by an online feedback controller ($b_{i} \leftarrow b_{i} \pm \gamma$ based on token load), with no gradient flowing through the update.

• The bias adjusts top-$K$ selection but is excluded from gate weights; output quality is determined by learned affinity $s_{i , t}$, not by load corrections.

• The method retains a sequence-wise auxiliary loss with a small coefficient for within-inference balance robustness; the improvement is specifically the elimination of batch-level gradient interference, not the removal of all balance objectives.

• Bias-based control is most valuable at $N \geq 64$ experts; for smaller MoEs the standard auxiliary loss is simpler and well-understood.

• Implementation requires a global all-reduce over per-expert token counts before each controller update to prevent divergence across data-parallel replicas.

7.3 Expert Parallelism Gotchas for Interviews

7.3.1 Token Dropping and Capacity Factors

The question “why does our MoE API return different outputs for the same prompt at temperature zero?” is one of those interview traps that separates engineers who have run MoE in production from those who have only read about it. The answer is almost always token dropping - the mechanism that prevents any single expert from being overwhelmed. Understanding capacity factors and what happens to dropped tokens is a staple of senior MoE system design questions, sitting at the intersection of routing correctness, memory management, and serving reliability.

Expert parallelism works by dispatching tokens to different devices via an all-to-all collective. Each expert lives on one or a few GPUs and can process only a bounded number of tokens before its on-device buffer fills. The capacity factor $C$ defines this bound:

where $T$ is the total number of tokens in the current dispatch batch and $N$ is the number of experts. Each expert is pre-allocated a slot buffer of exactly this size. If more tokens route to an expert than the buffer allows, the excess tokens are dropped.

A dropped token is not erased from the sequence. Its hidden-state vector passes unchanged through the residual connection, completely bypassing the MLP. The MoE layer contributes nothing to that token’s representation at this layer; the output is numerically identical to the input for that position. The sequence length and causal mask are unaffected - the token is still there, just unprocessed.

Capacity factors in practice range from $C = 1 . 0$ (zero tolerance for imbalance) to $C = 2 . 0$ (buffer twice the fair-share load). Switch Transformer and most production systems use $C \approx 1 . 25$ to $1 . 5$ as the default range.

The capacity factor is the MoE’s circuit breaker: it trades a small number of under-processed tokens for a guarantee that every expert finishes in bounded, predictable time.

The alternative to capacity-capped dispatch is allowing arbitrarily deep per-expert queues. Under token-choice top-$K$ routing, popular experts can attract several times their fair share of tokens in a single batch - especially early in training when the router has collapsed onto a small set of preferred experts. Without a cap, that GPU stalls draining the queue, creating a latency tail that ruins batch efficiency for every other expert on the device. With a fixed capacity, every expert’s forward pass takes the same wall-clock time regardless of routing decisions, which is exactly the property you need to keep GPUs synchronized, saturate tensor cores, and hide all-to-all latency.

The cost is representation quality for the dropped tokens. A token dropped from its top-ranked expert falls back to the residual only. If the routing has learned to rely on a specific expert for specific token types - which it does, given sufficient training - those tokens receive a degraded representation at that layer. Empirically, per-batch drop rates above roughly 5% consistently correlate with measurable perplexity increases.

The stochasticity angle is the subtler point. At inference time, whether a particular token gets dropped depends on who else is in the same dispatch batch. A prompt running in isolation may have zero drops; the same prompt batched with other high-traffic sequences may lose tokens to overloaded experts. The token’s position in the batch, and therefore its output representation, changes based on which other requests arrived concurrently - producing different completions even at temperature zero, with no randomness introduced by sampling.

A Worked Example

Consider a 7B-parameter MoE with 64 fine-grained experts (each one-quarter the size of a standard FFN) spread across 8 H100s, 8 experts per GPU. An inference batch contains 1024 tokens. With $C = 1 . 25$:

Total slot buffers: $64 \times 20 = 1280$ slots. The system can absorb up to 25% routing imbalance before any drops occur.

Now suppose the batch is heavy with English function words - “the,” “a,” “of” - whose hidden states cluster together, routing them all toward expert 3. Expert 3 receives 35 tokens but its buffer holds only 20; 15 tokens are dropped. Those 15 positions pass through the MoE layer unchanged.

Memory cost of the buffers: $64 \times 20 \times 4096 \times 2$ bytes $\approx 10 . 5$ MB, negligible. Raising $C$ to $2 . 0$ doubles the buffer to 21 MB but reduces the drop rate to near zero for typical routing distributions, a clear win unless the system is severely memory-constrained.

What You Actually Decide in Practice

• Set $C = 1 . 0$ only when memory is critically constrained and your balance loss is mature. A capacity of 1.0 guarantees no extra buffer memory but drops tokens the moment any imbalance exists. Accept it only if per-expert drop rates stay consistently below 1%.

• Default to $C = 1 . 25$-$1 . 5$ in most production serving deployments. This absorbs typical hot-expert bursts from mildly skewed batches without materially increasing buffer memory.

• Monitor drop rate as a first-class training metric. A per-expert drop rate above 5% averaged over a training run is a load-balancing failure, not a capacity-factor failure. Raising $C$ papers over the symptom; the correct fix is re-tuning the balance loss coefficient.

• Consider no-drop inference routing for latency-sensitive APIs. Route overflow tokens to their secondranked expert rather than dropping them. This eliminates batch-dependent non-determinism at temperature zero. Accept the mild training-inference distribution mismatch - in practice, the effect is small when overflow rates are low.

• Fine-grained experts reduce drop pressure naturally. Smaller experts mean each expert handles fewer tokens on average, so the same capacity factor tolerates a larger absolute routing imbalance before the buffer fills. This is one of the underappreciated systems benefits of the DeepSeek fine-grained expert design.

How This Shows Up in Interviews

• “A user reports that your MoE API returns different outputs for the same prompt at temperature zero. You have confirmed there is no sampling involved. What could cause this?” Strong answer: token dropping is batch-dependent. The same prompt, batched with other requests, may share expert buffers with high-traffic sequences. If a hot expert is saturated by other sequences in the batch, tokens from this prompt are dropped at that layer, changing the representation and thus the output. The fix is either no-drop inference routing or request isolation.

• “What is the capacity factor in a mixture-of-experts model, and what trade-off does it control?” Strong answer: capacity $= \lfloor C \cdot T / N \rfloor$ sets the maximum number of tokens any expert processes in one dispatch round. $C$ trades representation quality (fewer drops at higher $C$) against memory and buffer latency (larger pre-allocated buffers at higher $C$). Setting $C$ too low causes frequent drops on imbalanced batches; setting it too high wastes memory that could hold activations or KV cache entries.

• Senior follow-up: “If you eliminate token dropping at inference time by routing overflow tokens to their next-ranked expert, what new problem might arise and when does it matter?” Senior answer: the model trained under a regime where overflow tokens were dropped, not re-routed. Re-routing changes the effective per-expert load distribution seen at inference, shifting the activation statistics that downstream layers were calibrated on. For mildly imbalanced batches this discrepancy is small enough to ignore. For heavily skewed batches - such as a batch of all-code prompts on a model trained on mixed data - the effect can be meaningful. One mitigation is to include some no-drop re-routing during fine-tuning so the model sees both modes.

Key Takeaways

• Each expert has a fixed slot buffer of capacity $= \lfloor C \cdot T / N \rfloor$ tokens; tokens exceeding the cap are dropped and receive only the residual-stream pass-through with no MLP contribution.

• Token dropping is batch-dependent: whether a given token is dropped depends on which other requests occupy the same dispatch batch, making it the primary source of non-determinism in MoE serving at temperature zero.

• Production capacity factors of $1 . 25$-$1 . 5$ absorb typical routing imbalance; persistent drop rates above 5% signal a load-balancing failure that requires fixing the balance loss, not just raising $C$.

• No-drop inference routing eliminates serving non-determinism at the cost of a tolerable training-inference distribution mismatch.

• Fine-grained experts (smaller size, more count) reduce the per-expert token variance and therefore reduce drop rates at any given capacity factor.

7.3.2 Device-Level vs. Expert-Level Balancing

Once an interviewer is satisfied that you understand expert collapse, the pivot is almost immediate: “but what if your experts are perfectly balanced and your GPUs still aren’t?” That question targets device-level balancing - the second tier of the two-layer hierarchy that every serious MoE training system implements. Conflating expert-level and device-level balancing is one of the most common mistakes I see in system design conversations about MoE at scale, and at senior level it is a reliable differentiator.

MoE training systems maintain two distinct load-balancing objectives that operate at different granularities of the parallelism stack.

Expert-level balancing targets the routing distribution itself. The Switch Transformer auxiliary loss is:

where $N$ is the number of experts, $f_{i}$ is the fraction of tokens dispatched to expert $i$ in the batch (a hard routing count), and $P_{i}$ is the mean router probability assigned to expert $i$ across the same batch (a soft, differentiable quantity). The product $f_{i} \cdot P_{i}$ couples the discrete routing decision to the continuous router output, giving useful gradients while penalizing load concentration. Minimizing $\mathcal{L}_{\text{expert}}$ drives the router toward a uniform distribution over experts.

Device-level balancing has the same algebraic structure but redefines the summation unit from individual experts to device groups:

where $D$ is the number of devices, $F_{d}$ is the fraction of tokens dispatched to device $d$, and $P_{d}$ is the total router probability directed toward all experts residing on device $d$. Minimizing this term discourages the router from routing a disproportionate share of tokens to a single GPU, which would saturate that device’s inbound all-to-all bandwidth regardless of how evenly the load is distributed among the experts on that device.

Expert-level balancing prevents expert collapse; device-level balancing prevents network hotspots - they are complementary constraints at different granularities of the parallelism hierarchy.

The two objectives can be satisfied simultaneously or independently, and that is exactly the problem. Consider 64 experts on 8 GPUs (8 experts per GPU). Expert-level balancing can be perfectly satisfied with every expert receiving exactly $T / 64$ tokens. But if the routing is correlated - say, high-frequency English function words all cluster toward experts on GPUs 0 and 1 - those two GPUs still receive double the inbound all-to-all traffic of GPUs 2-7. The expert-level loss sees balance at the per-expert granularity; the device-level loss fires a separate penalty for the bandwidth hotspot.

Before device-level losses were standard, practitioners would observe mysteriously variable step times and diagnose the culprit as imbalanced communication rather than imbalanced compute. The fix of adding $\mathcal{L}_{\text{device}}$ with a smaller coefficient (typically 10-50$\times$ smaller than the expert-level coefficient) is cheap and dramatically stabilizes per-step time variance on multi-node runs. The DeepSeek training reports note exactly this kind of per-device communication balancing as a necessary addition when scaling beyond 16 billion active parameters.

DeepSeek-V2 also introduced a complementary structural mechanism: top-$M$ device selection. Before picking the top-$K$ experts for a token, the router first restricts its candidate pool to the top-$M$ devices by aggregated affinity score. This hard constraint bounds the number of distinct all-to-all hops any single token can generate, making worst-case communication cost predictable even with hundreds of experts spread across many devices. The loss-based approach and the structural approach are complementary: the loss gently guides the router during training; the device cap provides a hard budget guarantee at both training and inference time.

A Worked Example

A 16B-parameter MoE has 64 routed experts spread across 16 H100s (4 experts per GPU). A training batch dispatches 4096 tokens.

Ideal per-expert load: $4096 / 64 = 64$ tokens. Ideal per-device load: $4096 / 16 = 256$ tokens.

Suppose the router concentrates traffic toward experts 0-7 (GPUs 0 and 1) because these experts handle high-frequency subword tokens. Each of the 8 overloaded experts receives 128 tokens (2$\times$ fair share), while the remaining 56 experts average 48 tokens.

Expert-level loss response: $f_{i} \cdot P_{i}$ for experts 0-7 is elevated. The gradient pushes router scores for these experts downward. Improvement is gradual because the loss signal is shared across all 64 experts.

Device-level loss response: GPUs 0 and 1 each receive $128 \times 4 = 512$ tokens, giving $F_{0} = F_{1} = 0 . 125$ against the ideal $0 . 0625$. The device-level penalty fires immediately and more directly against the GPU-level concentration, pushing the router to diversify across devices even when no single expert appears badly overloaded.

Without the device-level loss, the expert-level loss alone would require many gradient steps to redistribute load sufficiently, during which all-to-all bandwidth on GPUs 0 and 1 remains the per-step bottleneck and step time shows high variance.

What You Actually Decide in Practice

• Start with $\alpha_{\text{expert}} \approx 10^{- 2}$ and $\alpha_{\text{device}} \approx 10^{- 3}$. The device-level loss is a secondary constraint; it should be strong enough to prevent hotspots but weak enough not to override the expert-level routing signal.

• Apply top-$M$ device selection when scaling beyond 32 experts. Restricting each token to the top- $M = 3$-$4$ devices bounds worst-case communication cost without meaningfully degrading routing quality, since routing to a seventh or eighth device almost never helps representation quality.

• Switch to sequence-wise balancing at inference time. Batch-level losses work well during training when batches are large and representative. At inference, request batches are small and user-controlled - a single unusual query can dominate a batch. A sequence-wise loss that balances routing within each sequence individually is more robust to out-of-distribution input patterns.

• Instrument per-GPU inbound token count, not just per-expert token count. A per-expert utilization dashboard can show perfect balance while two GPUs are starving for bandwidth. Add a per-device token histogram to your training dashboard from day one.

• Understand what DeepSeek-V3’s online bias update $b_{i}$ actually controls. The bias terms increment or decrement based on running per-expert load, replacing the batch-level expert-load auxiliary loss. They do not directly control how load is distributed across devices. The sequence-wise auxiliary loss DeepSeek-V3 retains is specifically there to handle device-level concentration on unusual inference sequences - the system is not “auxiliary-loss-free” at the device level.

How This Shows Up in Interviews

• “You have 64 experts on 16 GPUs. Training throughput is fine on average, but two GPUs consistently show 3$\times$ higher network utilization than the rest. What is likely happening and how do you fix it?” Strong answer: the router has learned to concentrate dispatches on a subset of devices hosting popular experts. Per-expert utilization metrics may look balanced, but the device-level aggregation is skewed. The fix is to add $\mathcal{L}_{\text{device}}$ with a coefficient in the $10^{- 3}$ range and, if using fine-grained experts with many hundreds of experts, also apply top-$M$ device selection to structurally bound the worst-case inbound traffic per GPU.

• “Explain the difference between expert-level and device-level load balancing in a MoE training system.” Strong answer: expert-level balancing penalizes routing concentration at individual-expert granularity via the $f_{i} \cdot P_{i}$ structure, preventing expert collapse and ensuring parameter utilization. Device-level balancing applies the same $F_{d} \cdot P_{d}$ structure at per-GPU granularity, preventing all-to-all communication hotspots. The two objectives can conflict: expert-level balance does not imply device-level balance if multiple high-traffic experts co-locate on the same GPU.

• Senior follow-up: “DeepSeek-V3 claims ‘auxiliary-loss-free balancing’ but still uses a sequence-level auxiliary loss. Is this a contradiction, and what does each component actually control?” Senior answer: not a contradiction. The online bias $b_{i}$ replaces the batch-level expert-load auxiliary loss - it adjusts individual expert attractiveness reactively without adding a differentiable loss term to the training objective. However, $b_{i}$ operates on batch-level running averages, which work well during training but cannot prevent a single unusual inference sequence from collapsing onto a few experts and overloading the corresponding devices. The sequence-wise auxiliary loss is a separate, lighter regularizer that operates within individual sequences to ensure routing robustness at inference time. The two mechanisms address different timescales: $b_{i}$ handles slow training-time drift; the sequence-wise loss handles fast inference-time spikes.

Key Takeaways

• Expert-level and device-level balancing are separate objectives that can be satisfied independently; satisfying one does not guarantee the other.

• Both use the $f \cdot P$ inner-product structure from Switch Transformer, but differ in their summation unit: individual experts versus device groups.

• Device-level hotspots appear as high all-to-all bandwidth variance across GPUs, not as per-expert utilization imbalance - monitor both to catch both failure modes.

• Top-$M$ device selection (DeepSeek-V2) provides a structural hard cap on per-token communication cost that is complementary to loss-based balancing.

• At inference time, sequence-wise balancing is more robust than batch-wise balancing because user request distributions are uncontrolled and can create sharp within-batch skew that batch-level mechanisms are slow to correct.