Module 7. Mixture of Experts

Solution. Summing every passport row—both vocabulary matrices, biased attention, the full expert bank, routers, norms, and learned sinks—gives 116,829,156,672. Therefore

$116{,}829{,}156{,}672-116{,}789{,}050{,}944=40{,}105{,}728.$

This baseline keeps normalization scales and attention sinks; the difference isolates linear-layer biases. The declared active count is 5,132,849,472: it charges the shared token path and four selected experts per MoE layer, includes one full output-vocabulary projection, and does not charge an indexed input embedding lookup as though every row were multiplied. This convention is necessary to reproduce the number.

Solution. The config gives hidden width 2048, 40 layers, 256 routed plus one shared expert, top-8, expert width 512, and one MTP layer. Multiplying all 35B parameters by $8/256$ incorrectly treats the whole model as the routed bank. Only routed experts receive that factor. The accounting structure is

$P_{active}=P_{shared\ path}+P_{shared\ expert}+\frac{k}{E}P_{routed\ bank},$

Attention/DeltaNet, the output head, router, norms, and shared expert do not receive the $k/E$ factor. Add MTP separately only when the selected execution path runs it, and state the input-embedding convention.

Solution. After $t$ steps the biased scores are $s_0(t)=0.768525-0.01875t$ and $s_6(t)=0.710950+0.01875t$. One step gives 0.749775 versus 0.729700, so expert 0 remains ahead. Equality occurs at

$t=\frac{0.768525-0.710950}{2\cdot0.01875}=1.5353,$

so the first integer step beyond the crossing is step 2: 0.731025 versus 0.748450. This constructed example shows a boundary crossing under two fixed opposing increments. Both a sign controller and a proportional controller can produce those increments for suitable loads; the calculation does not reproduce the full balancing simulation.

Solution. A particular expert is missed by one token with probability $1-k/E$. Under independent routing it is missed by all $N$ tokens with probability $(1-k/E)^N$, so the expected touched fraction is

$1-(1-k/E)^N.$

Substitution gives $1-(1-16/896)^{16}=0.25046$ and $1-(1-16/896)^{64}=0.68437$. Trained routing may differ because expert choices are skewed and correlated.

Solution. Idealized one-way dispatch sends $Tkd b$ bytes. With $T=2048$, $k=16$, and bf16,

$2048\cdot16\cdot7168\cdot2=448\ \text{MiB}$

in hidden space and 224 MiB in latent space. The reduction is therefore $7168/3584=\mathbf{2\times}$. Counting a return/combine path doubles both values to 896 and 448 MiB without changing the ratio. Metadata, route indices, projections, padding, and network protocol are additional costs.

Demo for this project exercise: English project-demo notebook for Module 7, sections “Demo 1” and “Demo 2” as starting points for this module’s projects. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.