Modern LLMs7 / 20

Module 7. Mixture of Experts

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 21 · Correct: 0

Question 7.4.1

7.4. A typed contract for routing and parameter accounting

Why is the total/active parameter ratio not a guaranteed MoE speedup?

One correct optionAIt does not model actual compute time, data movement, or device utilizationBTotal is always counted in bf16 whereas active is counted only after 4-bit quantizationCRouted experts are excluded from total but automatically included in active

A) It does not model actual compute time, data movement, or device utilization

B) Total is always counted in bf16 whereas active is counted only after 4-bit quantization

C) Routed experts are excluded from total but automatically included in active

Answer and explanation

Correct answer: A. The ratio measures parameter sparsity under a stated convention. Attention and output-projection parameters are included in this module's gpt-oss active count. That count does not determine attention runtime, communication cost, or kernel utilization.

Question 7.5.1

7.5. Rebuilding both gpt-oss models from open code

What total does the open gpt-oss-120b implementation yield after its bias parameters are counted?

One correct optionA116,789,050,944 — the count without linear biasesB116,829,156,672 — matrices, norms, sinks, and visible biasesC117,000,000,000 — the rounded model-size headline

A) 116,789,050,944 — the count without linear biases

B) 116,829,156,672 — matrices, norms, sinks, and visible biases

C) 117,000,000,000 — the rounded model-size headline

Answer and explanation

Correct answer: B. Including attention, router, and expert biases gives 116,829,156,672. The 116,789,050,944 count retains the same matrices, norms, and attention sinks but excludes linear-layer biases, whose contribution is 40,105,728.

Question 7.5.2

7.5. Rebuilding both gpt-oss models from open code

What should be checked before comparing active-parameter headlines from two MoE models?

One correct optionACompare only routed-expert counts and top-k in one MoE layerBConvert both checkpoints to the same numerical storage formatCAlign the treatment of embeddings, output heads, shared parts, and auxiliary modules

A) Compare only routed-expert counts and top-k in one MoE layer

B) Convert both checkpoints to the same numerical storage format

C) Align the treatment of embeddings, output heads, shared parts, and auxiliary modules

Answer and explanation

Correct answer: C. Active parameters have no single universal convention. A comparison must align the shared path, expert path, and auxiliary components.

Question 7.6.1

7.6. A numerical router trace

Selection bias changes top-k membership. What defines mixture weights in the studied sigmoid router?

One correct optionAThe clean sigmoid scores of the experts that were selectedBScores plus bias after a softmax over every expertCRaw logits after replacing them by integer top-k ranks

A) The clean sigmoid scores of the experts that were selected

B) Scores plus bias after a softmax over every expert

C) Raw logits after replacing them by integer top-k ranks

Answer and explanation

Correct answer: A. Bias affects selection, while mixture weights are normalized from the clean sigmoid scores of selected experts. Selection score and mixture weight are distinct.

Question 7.6.2

7.6. A numerical router trace

A top-1 router normalizes its sole gate to 1. Away from route changes, what is that gate’s logit gradient?

One correct optionAThe sigmoid derivative of the selected logit.BZero: this normalized gate is constant.CThe selected expert index.

A) The sigmoid derivative of the selected logit.

B) Zero: this normalized gate is constant.

C) The selected expert index.

Answer and explanation

Correct answer: B. The discrete index is not differentiated. Router gradients depend on the complete gating contract.

Question 7.7.1

7.7. Load balancing without conflating three mechanisms

What does Auxiliary-Loss-Free Balancing remove?

One correct optionAEvery effect of changed routing on expert trainingBThe direct interference gradient from a separate balancing objectiveCThe need to measure token-assignment frequencies for experts

A) Every effect of changed routing on expert training

B) The direct interference gradient from a separate balancing objective

C) The need to measure token-assignment frequencies for experts

Answer and explanation

Correct answer: B. The method adds no separate auxiliary objective to language-model loss. Bias still changes routes and therefore which experts receive LM gradients.

Question 7.7.2

7.7. Load balancing without conflating three mechanisms

Which bias-update rule is the main one in the Loss-Free Balancing paper?

One correct optionAA proportional step equal to the full load deviationBA gradient step on router z-loss with an adaptive rateCA fixed sign step up or down relative to mean load

A) A proportional step equal to the full load deviation

B) A gradient step on router z-loss with an adaptive rate

C) A fixed sign step up or down relative to mean load

Answer and explanation

Correct answer: C. The paper’s main algorithm uses the sign of the load deviation. The notebook’s proportional controller is separate and explicitly pedagogical.

Question 7.7.3

7.7. Load balancing without conflating three mechanisms

How is router z-loss computed across tokens?

One correct optionASquare each token’s logsumexp, then average.BAverage logsumexp first, then square that mean.CCompute the variance of expert selection frequencies.

A) Square each token’s logsumexp, then average.

B) Average logsumexp first, then square that mean.

C) Compute the variance of expert selection frequencies.

Answer and explanation

Correct answer: A. ST-MoE uses the mean squared log partition. Squaring the mean generally gives a different result.

Question 7.7.4

7.7. Load balancing without conflating three mechanisms

What balancing scheme does the DeepSeek-V3 report describe?

One correct optionARouter z-loss alone, without measuring load.BNo auxiliary losses at any level.CBias-based global balancing plus a small complementary sequence-level loss.

A) Router z-loss alone, without measuring load.

B) No auxiliary losses at any level.

C) Bias-based global balancing plus a small complementary sequence-level loss.

Answer and explanation

Correct answer: C. An auxiliary-loss-free main controller does not imply that the entire model uses no auxiliary losses.

Question 7.8.1

7.8. Fine-grained experts and always-on shared capacity

How should C(128,4)=10,668,000 be interpreted?

One correct optionA10,668,000 unordered top-4 subsets before any extra routing constraintsB128 independently learned skills, each of which activates 4 expertsC4 experts that every batch is guaranteed to choose from all 128

A) 10,668,000 unordered top-4 subsets before any extra routing constraints

B) 128 independently learned skills, each of which activates 4 experts

C) 4 experts that every batch is guaranteed to choose from all 128

Answer and explanation

Correct answer: A. This is a combinatorial upper bound on subsets, not a count of learned skills. Training, correlated routes, and router constraints reduce the realized patterns.

Question 7.8.2

7.8. Fine-grained experts and always-on shared capacity

What is the most defensible interpretation of a shared expert?

One correct optionAIt is guaranteed to store grammar, arithmetic, and common function wordsBIt provides always-on capacity for common work and may reduce bank redundancyCIt replaces attention for tokens that occurred frequently during training

A) It is guaranteed to store grammar, arithmetic, and common function words

B) It provides always-on capacity for common work and may reduce bank redundancy

C) It replaces attention for tokens that occurred frequently during training

Answer and explanation

Correct answer: B. A shared expert always participates in the MoE branch and is motivated by common computation and reduced duplication. Its semantic content requires intervention-based evidence.

Question 7.8.3

7.8. Fine-grained experts and always-on shared capacity

Which change preserves expert weights and active FFN work when each of eight width-i experts is split into four?

One correct optionAChoose 2 of 32 width-i/4 experts.BChoose 8 of 128 width-i experts.CChoose 8 of 32 width-i/4 experts instead of 2 of 8 width-i experts.

A) Choose 2 of 32 width-i/4 experts.

B) Choose 8 of 128 width-i experts.

C) Choose 8 of 32 width-i/4 experts instead of 2 of 8 width-i experts.

Answer and explanation

Correct answer: C. Both E·i and k·i stay constant. Routing and communication costs can still change.

Question 7.9.1

7.9. What expert specialization evidence can actually support

How should routing frequency be turned into a claim about expert specialization?

One correct optionAName the expert after its ten most frequently routed tokensBChoose the most frequent domain and treat it as the expert’s functionCRun an intervention or ablation and measure the functional change

A) Name the expert after its ten most frequently routed tokens

B) Choose the most frequent domain and treat it as the expert’s function

C) Run an intervention or ablation and measure the functional change

Answer and explanation

Correct answer: C. Routing frequency is correlational. A causal intervention or ablation tests whether the capability changes when the expert is modified.

Question 7.10.1

7.10. Inference: the expert bank, all-to-all, and batch union

For E=128, k=4, and N=16, what is the expected touched-expert fraction under independent uniform routing?

One correct optionAAbout 39.83% from 1-(1-4/128)^16BExactly 3.125%, because one token selects 4 of 128CAbout 86.89%, which is the same setup with N=64

A) About 39.83% from 1-(1-4/128)^16

B) Exactly 3.125%, because one token selects 4 of 128

C) About 86.89%, which is the same setup with N=64

Answer and explanation

Correct answer: A. The formula gives 0.3982897, or about 39.83%. It assumes independent uniform routing and is not a prediction of a trained router.

Question 7.10.2

7.10. Inference: the expert bank, all-to-all, and batch union

What does 2Thkb measure in the Expert Parallelism estimate?

One correct optionALogical dispatch-and-return payload for all assignments.BExact per-GPU traffic under any placement.CA mandatory lower bound on network traffic.

A) Logical dispatch-and-return payload for all assignments.

B) Exact per-GPU traffic under any placement.

C) A mandatory lower bound on network traffic.

Answer and explanation

Correct answer: A. Local assignments need no network transfer, aggregation can reduce messages, and padding or protocols add bytes.

Question 7.11.1

7.11. LatentMoE and alternatives to fixed discrete top-k

What is the central idea of LatentMoE?

One correct optionAReplace the discrete expert bank by a continuous basis-matrix mixtureBKeep discrete top-k while doing dispatch and expert compute in a narrower spaceCRoute attention heads only and remove feed-forward experts entirely

A) Replace the discrete expert bank by a continuous basis-matrix mixture

B) Keep discrete top-k while doing dispatch and expert compute in a narrower space

C) Route attention heads only and remove feed-forward experts entirely

Answer and explanation

Correct answer: B. LatentMoE keeps discrete experts and top-k. A narrower routed representation reduces communication and expert compute before projection back to model width.

Question 7.12.1

7.12. Level-one code: policies should be visible

One assignment is removed by capacity overflow. What happens to the remaining mixture weight?

One correct optionAEvery MoE runtime must leave it numerically unchangedBEvery implementation must reroute the token to another expertCRuntime policy decides; the notebook renormalizes surviving weights

A) Every MoE runtime must leave it numerically unchanged

B) Every implementation must reroute the token to another expert

C) Runtime policy decides; the notebook renormalizes surviving weights

Answer and explanation

Correct answer: C. Dropping and rerouting policies vary. The course notebook renormalizes surviving weights, but that is not a universal MoE contract.

Question 7.14.1

7.14. Reading the gpt-oss implementation

How does the gpt-oss router differ from the DeepSeek-V3-style contract discussed here?

One correct optionAgpt-oss selects top-k logits then softmaxes them; DeepSeek uses sigmoid plus selection biasBgpt-oss uses sigmoid plus bias whereas DeepSeek softmaxes selected logitsCgpt-oss has no trained router whereas DeepSeek has no discrete top-k

A) gpt-oss selects top-k logits then softmaxes them; DeepSeek uses sigmoid plus selection bias

B) gpt-oss uses sigmoid plus bias whereas DeepSeek softmaxes selected logits

C) gpt-oss has no trained router whereas DeepSeek has no discrete top-k

Answer and explanation

Correct answer: A. The open gpt-oss router selects top-k logits and softmaxes them. In the studied DeepSeek-style contract, selection uses sigmoid score plus bias, while mixture weights use clean scores.

Question 7.15.1

7.15. A source-aware snapshot of current MoE architectures

How should the “Kimi K3: 2.8T / 104B” line be read?

One correct optionAAs proof of exactly a 26.9× speedup on any hardwareBAs a total/active passport requiring the expert contract and counting conventionCAs the sizes of the vision encoder and language decoder

A) As proof of exactly a 26.9× speedup on any hardware

B) As a total/active passport requiring the expert contract and counting convention

C) As the sizes of the vision encoder and language decoder

Answer and explanation

Correct answer: B. A total/active pair is only the beginning of a passport. Interpretation also needs top-k, shared/routed structure, attention, auxiliary modules, and the counting convention.

Question 7.15.2

7.15. A source-aware snapshot of current MoE architectures

Qwen3.5 includes MTP. Must its weights count as active on every ordinary decoding step?

One correct optionAYes: the field guarantees execution.BOnly if the specified execution path uses MTP.CNo: MTP is never used for inference.

A) Yes: the field guarantees execution.

B) Only if the specified execution path uses MTP.

C) No: MTP is never used for inference.

Answer and explanation

Correct answer: B. A configuration lists available components; an active count describes a particular execution path.

Question 7.16.1

7.16. Multimodal MoE is not one architecture

What may change when visual tokens enter a shared MoE router while top-k stays fixed?

One correct optionAThe expert compute of every individual token must doubleBEvery expert automatically becomes modality-specificCThe batch expert union and communication profile may widen

A) The expert compute of every individual token must double

B) Every expert automatically becomes modality-specific

C) The batch expert union and communication profile may widen

Answer and explanation

Correct answer: C. With fixed top-k, per-token expert compute need not increase. A more diverse batch may touch more distinct experts and change all-to-all traffic.