Module 20. The complete LLM: from tokens to training and generation — quizzes and answer checking · Modern LLMs

Question 20.1.1

20.1. Three computations that should not be conflated

What distinguishes a forward pass from generation?

One correct optionAA forward pass computes a distribution; generation also selects and appends a tokenBA forward pass updates weights, while generation only computes lossCA forward pass always selects the most likely token

A) A forward pass computes a distribution; generation also selects and appends a token

B) A forward pass updates weights, while generation only computes loss

C) A forward pass always selects the most likely token

Answer and explanation

Correct answer: A. Selection, appending, and stopping are generation rules. Computing logits alone does not choose a continuation.

Question 20.3.1

20.3. Embedding lookup starts the computation

For sequence [BOS, A, B, EOS], the inputs are [BOS, A, B]. What are the targets?

One correct optionA[BOS, A, B]B[A, B, EOS]C[EOS, EOS, EOS]

A) [BOS, A, B]

B) [A, B, EOS]

C) [EOS, EOS, EOS]

Answer and explanation

Correct answer: B. Targets are shifted by one position so that every row predicts the next token from its known past.

Question 20.4.1

20.4. RMSNorm controls scale before a sublayer

Why does RMSNorm in this model preserve causal visibility?

One correct optionAIt subtracts a mean over all sequence positionsBIt zeroes keys after the current positionCIt computes scale separately within each representation row

A) It subtracts a mean over all sequence positions

B) It zeroes keys after the current position

C) It computes scale separately within each representation row

Answer and explanation

Correct answer: C. RMSNorm operates across features within one position. The attention mask separately controls visibility between positions.

Question 20.5.1

20.5. Projections and RoPE attach a position to a comparison

What does RoPE transform in the base M20 configuration?

One correct optionAQueries and keys; values remain unrotatedBOnly values, because they are combinedCAll linear weights whenever a new token arrives

A) Queries and keys; values remain unrotated

B) Only values, because they are combined

C) All linear weights whenever a new token arrives

Answer and explanation

Correct answer: A. Positional rotations act on Q/K activations. Fixed frequencies are not learned projection matrices.

Question 20.6.1

20.6. Causal attention and shared key/value heads

With h_q=4 and h_kv=2, which zero-based query heads share the first K/V head?

One correct optionA0 and 2B0 and 1COnly 0

A) 0 and 2

B) 0 and 1

C) Only 0

Answer and explanation

Correct answer: B. The map g(j)=floor(j/2) assigns consecutive pairs of query heads to each K/V head. Their query projections remain distinct.

Question 20.7.1

20.7. SwiGLU completes the block

What travels along the first residual branch of a pre-norm block?

One correct optionAOnly the normalized attention input N(H)BThe attention probability matrix ACThe representation H before attention normalization

A) Only the normalized attention input N(H)

B) The attention probability matrix A

C) The representation H before attention normalization

Answer and explanation

Correct answer: C. The block computes Z=H+Attention(N(H)). Normalization belongs to the transformation branch; the skip connection carries H.

Question 20.8.1

20.8. The stack, output projection, and parameter inventory

The example has V=4, d=m=2, and one block. How many parameters remain after tying E and U?

One correct optionA42B50C34

A) 42

B) 50

C) 34

Answer and explanation

Correct answer: A. The untied model has 50 parameters. Tying removes one 4×2 table of 8 parameters; the final normalization is still present.

Question 20.10.1

20.10. From a next-token distribution to sequence likelihood and loss

The sole selected target, EOS, has probability 0.183502. What is the masked mean loss?

One correct optionAAbout 0.565177 natsBAbout 1.695530 natsCAbout 0.183502 nats

A) About 0.565177 nats

B) About 1.695530 nats

C) About 0.183502 nats

Answer and explanation

Correct answer: B. Only one target is selected, so the denominator is one. The loss is −ln(0.183502), with no extra division by the number of input positions.

Question 20.11.1

20.11. Backpropagation reaches every learned part

How are gradients accumulated for tied embedding table E?

Second pass — after Section 11.

One correct optionAOnly the contribution from the final layer is usedBTwo independent tables receive separate updatesCContributions from input lookup and output projection are added

A) Only the contribution from the final layer is used

B) Two independent tables receive separate updates

C) Contributions from input lookup and output projection are added

Answer and explanation

Correct answer: C. One parameter participates in two graph locations. The chain rule adds both contributions before the parameter update.

Question 20.12.1

20.12. AdamW turns derivatives into a new model

What follows from the loss falling from 1.109374 to 1.084832 after one AdamW step?

One correct optionALoss decreased on the same sequence under the specified settingsBThe model necessarily improved on unseen textsCEvery later update must also decrease loss

A) Loss decreased on the same sequence under the specified settings

B) The model necessarily improved on unseen texts

C) Every later update must also decrease loss

Answer and explanation

Correct answer: A. This verifies one update on the training example, not monotonic convergence or generalization.

Question 20.13.1

20.13. Generation adds a selection rule

Top-2 removes EOS from the selection distribution. What does that imply about the base model?

One correct optionAIts EOS weights have been set to zeroBIts EOS probability may still be positiveCIt can no longer terminate any sequence

A) Its EOS weights have been set to zero

B) Its EOS probability may still be positive

C) It can no longer terminate any sequence

Answer and explanation

Correct answer: B. Top-k changes the decoder distribution q without updating weights. The original p can still assign positive probability to EOS.

Question 20.14.1

20.14. KV-cache preserves a computation already performed

Which cache can be used after training updates the weights?

One correct optionAThe old cache is always valid if token IDs are unchangedBChanging only the next position index is sufficientCRecompute prefix K/V using the new weights

A) The old cache is always valid if token IDs are unchanged

B) Changing only the next position index is sufficient

C) Recompute prefix K/V using the new weights

Answer and explanation

Correct answer: C. K/V depend on weights and intermediate activations. Identical text does not make a cache from earlier weights valid.

Question 20.15.1

20.15. MoE replaces a specific operator

Where does MiniLLM's sigmoid router use the balancing offset?

Second pass — after Section 15.

One correct optionAIn expert selection; mixture weights normalize the original sigmoid scoresBOnly in normalizing the selected mixture weightsCIt replaces both expert weights and expert outputs

A) In expert selection; mixture weights normalize the original sigmoid scores

B) Only in normalizing the selected mixture weights

C) It replaces both expert weights and expert outputs

Answer and explanation

Correct answer: A. Top-k selection on score+bias is separate from normalization of the original scores. This is the routing contract of the example.

Question 20.19.1

20.19. Where the construction changes: SSMs and diffusion

What can remain unchanged when attention is replaced by a causal SSM?

One correct optionAMandatory separate K/V records for every past positionBNext-token prediction and the autoregressive probability productCThe exact internal structure of every Transformer block

A) Mandatory separate K/V records for every past position

B) Next-token prediction and the autoregressive probability product

C) The exact internal structure of every Transformer block

Answer and explanation

Correct answer: B. A causal SSM changes the context operator and carried state. It can retain the autoregressive objective without retaining the old block structure.

Question 20.19.2

20.19. Where the construction changes: SSMs and diffusion

Why does the diffusion probability sum over hidden sequences?

One correct optionATo replace token probabilities with their arithmetic meanBBecause every diffusion step appends exactly one token on the rightCDifferent refinement trajectories can reach the same clean result

A) To replace token probabilities with their arithmetic mean

B) Because every diffusion step appends exactly one token on the right

C) Different refinement trajectories can reach the same clean result

Answer and explanation

Correct answer: C. The product of reverse transitions describes one trajectory. The sum marginalizes intermediate states, not token positions.