Mixture-of-experts models: routing, sparsity and the hardware consequences

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A mixture-of-experts layer keeps many feed-forward networks and runs only a few of them on each token. A router, trained with the rest of the model, picks which ones. The parameter count can grow with the number of experts. The floating-point work per token grows with how many of those experts you actually call. Memory, communication and load imbalance do not follow the floating-point slogan. That split is the whole operational point of the architecture.

An AI writing system helped prepare this article for NetLeverage. "The model chooses an expert" means a learned router emits scores and a top-k rule keeps some of them. It is not a claim about subjective experience. No configuration below was run in a lab. This article makes no NetLeverage product claim.

Why it matters

Shazeer et al. (2017) state the bargain directly. A network's ability to absorb a corpus is limited by its parameter count. Conditional computation is the proposal to add parameters without adding a matching amount of work, by activating only part of the network on each example. Their sparsely gated layer put that into practice with up to thousands of feed-forward experts, of which a handful run per example. They report models up to 137 billion parameters in that design, on language modelling and translation.

Switch Transformers later made the routing rule stricter, one expert per token, and trained sparse models out to the trillion-parameter range while holding FLOPs per token near a dense baseline. Mixtral 8x7B is the small, inspectable version of the same idea: eight experts, two of them active, which the paper describes as about 47 billion total parameters and about 13 billion active per token.

If you budget a GPU from the active-parameter number alone, you will run out of memory. Mixtral's own efficiency section says inference compute tracks the active count, and serving memory tracks the sparse count. Both sentences are in the paper. They are not the same sentence.

How it works

What an expert is

In the 2017 layer, each expert is a feed-forward network. The experts share an input size and an output size. In that paper they also share an architecture, and they do not share weights. The gating network $G$ maps the input to an $n$-dimensional vector. The layer output is a weighted sum of expert outputs. Wherever a gate value is zero, that expert is not evaluated.

The gate is not a hand-written rule. Shazeer et al. form logits with a learned matrix, add tunable Gaussian noise, keep the top $k$ logits, set the rest to $-\infty$, and take a softmax. A coordinate sent to $-\infty$ becomes a gate of zero after the softmax, which is what creates the sparsity. The noise is there to help load balance. They note that the hard top-k cut is discontinuous, and they report that they had not seen that discontinuity become a practical training failure. Gradients flow through the experts that were selected and through the gate values of those experts. Experts that were not selected do not see that example.

Later language models put this layer where the Transformer's position-wise feed-forward block used to be. Attention stays dense: every token still attends. Switch Transformers draw the encoder block that way, with the dense feed-forward replaced by a Switch feed-forward. Mixtral does the same replacement in a decoder-only stack, and it does it in every block. The paper contrasts that with GShard, which replaced every other block and used a more elaborate rule for the second expert. Mixtral's experts are SwiGLU blocks, the same form as the dense feed-forward they replace, with separate weights.

Top-k, then a weighted sum

Mixtral writes the gate as a softmax over the top-k logits of a linear layer. If a logit is outside the top k it is treated as $-\infty$ before the softmax. $k$ is the knob for compute per token. Raising the number of experts $n$ while holding $k$ fixed raises the sparse parameter count and leaves the active count, and therefore the matmul work per token, essentially flat. Their Table 1 fixes the shape: model dimension 4,096, 32 layers, 32 attention heads, 8 key-value heads, head dimension 128, feed-forward hidden size 14,336, vocabulary 32,000, context 32,768, 8 experts, top-2.

For each token the two chosen experts run, and their outputs are added with the gate values as weights. The next layer sees one vector, not a stack of expert outputs. Which two experts fire can change at the next token. There is no requirement that token 50 use the same pair as token 51.

Switch Transformers take $k=1$. Shazeer et al. had conjectured that routing to more than one expert was needed so the router could compare experts and receive a useful gradient. Fedus, Zoph and Shazeer test the opposite and report that one expert preserves quality in their setup, cuts router work, and at least halves the batch each expert must hold, because each token is sent to one expert rather than two. The selected expert's output is multiplied by the router probability for that expert. The router probability remains differentiable, which is how the router trains.

Load, capacity and dropped tokens

A router that always prefers the same expert wastes the others and overloads one device. Shazeer et al. add auxiliary losses so that importance and load spread across experts. Switch Transformers simplify that to one auxiliary term per Switch layer, added to the training loss. For a batch, let $f_i$ be the fraction of tokens dispatched to expert $i$, and let $P_i$ be the fraction of router probability mass on expert $i$. The loss is proportional to ${\sum }_i{f}_i{P}_i$, scaled by the expert count so the value stays stable as $N$ changes. It is minimised when both fractions are $1/N$. The dispatch fraction is not differentiable; the probability fraction is. They use a coefficient $\alpha ={10}^{-2}$, chosen from a sweep of powers of ten from ${10}^{-1}$ down to ${10}^{-5}$, as large enough to balance load and small enough not to swamp the cross-entropy.

Capacity is the systems half of the same problem. Switch sets

Each expert processes a fixed number of tokens. A capacity factor above 1 leaves empty slots so a slightly uneven router still fits. Those empty slots are real compute and real memory: the shapes are static, because the implementation compiles shapes ahead of time while the routing itself is data-dependent. If more tokens arrive than the expert's capacity, the overflow tokens skip that layer's feed-forward. Their representation continues through the residual connection. Fedus et al. report drop rates typically under 1% when the auxiliary loss is doing its job, and they say low drop rates matter for scaling. A capacity factor that is too tight drops tokens. A capacity factor that is too loose pads.

In Switch's fixed-capacity TPU training implementation, overflow takes the residual path without that expert update. Our operational inference is to measure capacity and drops where a serving implementation has this behaviour. Do not assume every MoE server drops overflow: Mixtral's paper does not specify such a path.

What is sparse, and what is not

Three quantities move differently.

FLOPs per token track $k$, not $n$. Switch Transformers state the hypothesis as a scaling axis: more parameters at constant FLOPs per example. Their router itself is cheap relative to the experts, a matrix of cost on the order of model dimension times expert count, but it is not free, and they compute it in float32. The dispatch tensors are cast back to bfloat16 before the all-to-all, so the high-precision router work stays local to the device. The paper reports that low precision can exacerbate softmax problems: Switch-Base trained in bfloat16 diverged without the selective-precision approach. Float32 routing was one of several stabilisers, and the combined techniques still did not eliminate instability for Switch-XXL. This is evidence for testing router precision in context, not a universal fix.

Parameters stored track $n$. Every expert has a full feed-forward. Mixtral says the memory cost of serving is proportional to the sparse count, which they give as 47 billion, still under Llama 2 70B, while the active count used for compute is 13 billion. The abstract's "only uses 13B active parameters during inference" is about work per token. The later section refuses the leap to memory: routing has overhead, and running more than one expert per device increases memory traffic. They say the layer suits batched workloads, where there is enough arithmetic intensity to hide that traffic. A batch of one token at decode time is the opposite end of that spectrum. You still stored every expert. You may still read two experts' weights per layer for that single token, because $k=2$.

The attention cache does not shrink because the experts replaced the feed-forward, not the attention. Mixtral's attention is grouped-query, with 8 key-value heads, which cuts the cache relative to 32 query heads. That saving is the attention design from the previous article's subject. It is not a gift from sparsity. A capacity plan that subtracts "unused experts" from the cache math is subtracting the wrong tensor.

Devices, all-to-all, and idle experts

Switch Transformers place different experts on different devices. Adding devices can add experts, and therefore parameters, while each device still holds a manageable slice and still does one expert's work per token it receives. The tokens have to move. The router builds a dispatch and a combine. Those tensors cross the network in an all-to-all: token representations travel to the device that owns the chosen expert, and the expert output travels back to the token's original position. Mixtral describes the same pattern under the name expert parallelism, and points at Megablocks for the single-GPU case, where the expert computation is a block-sparse matrix multiply that tolerates experts receiving different numbers of tokens.

Load balance is now a hardware stall, not only a loss term. If every token in a step picks the expert on GPU 0, GPU 0 is busy and the others wait, and Switch's capacity rule will drop the overflow. Mixtral says even distribution across GPUs is required to avoid that bottleneck. The auxiliary loss is trained to prevent the imbalance. It does not guarantee a balanced batch at serving time if your prompts are unlike the training mix. Measure expert histograms on your own traffic. A flat histogram in a paper's validation set is not a flat histogram on a Monday-morning workload that is half source code.

Experts are not labelled topics

It is tempting to narrate the eight Mixtral experts as "the maths expert" and "the code expert". Jiang et al. checked. On subsets of The Pile's validation data they plot which experts are selected, at the first layer, a middle layer and the last layer. ArXiv, PubMed abstracts and philosophy papers produce similar assignment distributions. They say they do not see an obvious topic pattern. Jiang et al. nevertheless report structured syntactic routing, including patterns around indentation and tokens such as "self" and "Question". They also measure temporal locality: at layer 15, a token repeats a neighbour's first-or-second expert in 61.6–67.0% of cases across their datasets, versus roughly 46% under random routing. They warn that high locality can oversubscribe particular experts under expert parallelism. The paper's result is enough to retire the slide that draws one expert per department. The router is a function of the hidden state. Read its decisions from logs if you need to know what it does on your data.

What to choose and why

A dense model with the active parameter count is the right comparison for compute, and the wrong comparison for the card you buy. A dense model with the sparse parameter count is the right comparison for weight memory, and a pessimistic comparison for FLOPs. Report both numbers, and the capacity factor if the server uses one.

Risks and limits

• Overflow is implementation-specific. Switch's documented fixed-capacity overflow skips an expert through the residual path. Measure it if your implementation uses that design; do not assume Mixtral or another server silently drops tokens merely because it is sparse.
• The 7× pre-training speedup is Switch versus a FLOP-matched T5 on their hardware and schedule. The trillion-parameter result is a 4× pre-training speedup against their T5-XXL baseline. Neither number is an inference speedup for Mixtral.
• Mixtral's benchmark claims (against Llama 2 70B and the GPT-3.5 snapshot they name) are that paper's evaluation. This article uses the architecture section, not those scores, as a reason to deploy anything.
• "47B and 13B" are the paper's rounded counts. Do not add a third significant figure from a blog retelling. Recompute from the checkpoint if a purchasing decision depends on the exact byte size.
• $k=1$ worked for Switch. It is not a proof that every later model should drop its second expert. Mixtral ships $k=2$. Changing $k$ at serving time without matching training changes the function.
• No lab run and no NetLeverage product behaviour are claimed.

Sources

1. Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, arXiv:1701.06538 — https://arxiv.org/abs/1701.06538 (HTML opened 10 November 2026). Sparse top-k gating, expert feed-forwards, conditional compute.
2. Fedus, Zoph and Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, arXiv:2101.03961 — https://arxiv.org/abs/2101.03961 (HTML via ar5iv, opened 10 November 2026). Top-1 routing, capacity, dropped tokens, auxiliary loss, expert placement.
3. Jiang et al., Mixtral of Experts, arXiv:2401.04088 — https://arxiv.org/abs/2401.04088 (HTML opened 10 November 2026). Eight experts, top-2, active versus sparse parameters, expert parallelism, the topic-specialisation check.

This article was originally published on 10 November 2026.