Modern LLMs6 / 20

Module 6. Normalization, activations, and the anatomy of a Transformer block

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 22 · Correct: 0

Question 6.4.1

6.4. A block makes two successive residual updates

What does the identity path in a Pre-LN residual block guarantee?

One correct optionAThe Jacobian contains an identity term, but the total gradient need not be nonzeroBAny trained sublayer can be removed with almost no quality loss because the residual path completely bypasses its computationCThe residual stream automatically keeps unit RMS at every depth regardless of the sublayer weights and outputs

A) The Jacobian contains an identity term, but the total gradient need not be nonzero

B) Any trained sublayer can be removed with almost no quality loss because the residual path completely bypasses its computation

C) The residual stream automatically keeps unit RMS at every depth regardless of the sublayer weights and outputs

Answer and explanation

Correct answer: A. For h+G(h), the Jacobian is I+J_G. With G(h)=−h, the terms cancel. A bypass also does not establish that removing a trained branch preserves quality.

Question 6.5.1

6.5. What LayerNorm and RMSNorm compute

How does RMSNorm differ from LayerNorm in its basic formula?

One correct optionARMSNorm uses whole-batch statistics instead of per-token statisticsBRMSNorm subtracts the mean but does not rescaleCRMSNorm does not center the vector and normalizes by its root mean square

A) RMSNorm uses whole-batch statistics instead of per-token statistics

B) RMSNorm subtracts the mean but does not rescale

C) RMSNorm does not center the vector and normalizes by its root mean square

Answer and explanation

Correct answer: C. LayerNorm subtracts the mean and divides by the standard deviation. RMSNorm does not subtract the mean: before the learned per-coordinate gain, it divides all coordinates by one shared root-mean-square denominator.

Question 6.5.2

6.5. What LayerNorm and RMSNorm compute

At rms = 10⁻³, how much larger is the RMSNorm multiplier with eps=10⁻⁶ than with eps=10⁻⁵?

One correct optionAAbout 1.41×BAbout 2.35 timesCExactly 10×

A) About 1.41×

B) About 2.35 times

C) Exactly 10×

Answer and explanation

Correct answer: B. The multipliers are about 707 and 302, so the ratio is roughly 2.35. Epsilon also depends on activation scale and reduction precision, not only on the working dtype’s machine epsilon.

Question 6.5.3

6.5. What LayerNorm and RMSNorm compute

When is RMSNorm approximately invariant to positive input scaling with fixed ε>0?

One correct optionAWhen mean square greatly exceeds ε both before and after scalingBFor every nonzero vector, regardless of εCOnly when the input has zero mean

A) When mean square greatly exceeds ε both before and after scaling

B) For every nonzero vector, regardless of ε

C) Only when the input has zero mean

Answer and explanation

Correct answer: A. Require both mean(x²)≫ε and c²mean(x²)≫ε. Factoring c out gives an effective ε/c²; a very small c can make the stabilizer matter even for a large original input.

Question 6.5.4

6.5. What LayerNorm and RMSNorm compute

How should RMSNorm statistics be computed for large FP16 values?

One correct optionAPromote the input to FP32 before squaringBSquare in FP16, then promote the meanCRemove ε to avoid overflow

A) Promote the input to FP32 before squaring

B) Square in FP16, then promote the mean

C) Remove ε to avoid overflow

Answer and explanation

Correct answer: A. The square of 1000 exceeds FP16 range. Promotion after overflow cannot recover it.

Question 6.6.1

6.6. Placement changes the computation graph

What is the structural difference between Post-LN and Pre-LN?

One correct optionAPost-LN normalizes the residual sum; Pre-LN normalizes the sublayer input before additionBPost-LN is restricted to encoder blocks, whereas Pre-LN is valid only inside decoder blocksCPre-LN removes residual addition and concatenates the input with each sublayer output instead

A) Post-LN normalizes the residual sum; Pre-LN normalizes the sublayer input before addition

B) Post-LN is restricted to encoder blocks, whereas Pre-LN is valid only inside decoder blocks

C) Pre-LN removes residual addition and concatenates the input with each sublayer output instead

Answer and explanation

Correct answer: A. Post-LN places normalization after h+F(h), on the direct path. Pre-LN computes h+F(Norm(h)) and preserves a clean identity route. This affects initialization-time gradients without determining every training outcome.

Question 6.6.2

6.6. Placement changes the computation graph

What does Peri-LN add to ordinary Pre-LN?

One correct optionAIt removes input normalization and keeps only final-logit normalizationBIt replaces residual addition with a learned stream-mixing matrixCIt also normalizes the sublayer output before residual addition

A) It removes input normalization and keeps only final-logit normalization

B) It replaces residual addition with a learned stream-mixing matrix

C) It also normalizes the sublayer output before residual addition

Answer and explanation

Correct answer: C. Peri-LN surrounds the sublayer: N_out(F(N_in(h))). This controls the scale of the written update. With a learned gain, that update need not have exactly unit RMS.

Question 6.6.3

6.6. Placement changes the computation graph

What does the 36-layer none/pre/post toy experiment establish?

One correct optionAPost-LN always trains better than Pre-LN in real LLMs because every layer forces the RMS back to oneBWith the same linear weights, norm placement strongly changes forward scale, but does not prove training convergenceCPeri-LN and OLMo2 implement the same computation graph, normalizing both the input and output of every sublayer

A) Post-LN always trains better than Pre-LN in real LLMs because every layer forces the RMS back to one

B) With the same linear weights, norm placement strongly changes forward scale, but does not prove training convergence

C) Peri-LN and OLMo2 implement the same computation graph, normalizing both the input and output of every sublayer

Answer and explanation

Correct answer: B. With one fixed input and weight stack, the final RMS values are roughly 4·10⁵, 6.42, and 1.0. The last value follows from explicit normalization at every step; the experiment isolates forward-scale dynamics only.

Question 6.7.1

6.7. Q/K normalization: check the equation and the axis

How should original QKNorm be distinguished from the notebook’s Q/K RMSNorm?

One correct optionAOriginal QKNorm uses L2 normalization and a learned scale; the notebook uses RMS normalization with a different norm contractBAfter choosing epsilon they are identical: L2 norm and RMSNorm differ only by a constant absorbed into the learned scaleCQKNorm normalizes value vectors only, whereas Q/K RMSNorm is applied to the attention mask before softmax

A) Original QKNorm uses L2 normalization and a learned scale; the notebook uses RMS normalization with a different norm contract

B) After choosing epsilon they are identical: L2 norm and RMSNorm differ only by a constant absorbed into the learned scale

C) QKNorm normalizes value vectors only, whereas Q/K RMSNorm is applied to the attention mask before softmax

Answer and explanation

Correct answer: A. Original QKNorm uses unit L2 vectors and a learned scale. The RMSNorm variant has a different denominator and retains division by √d_h. Positive epsilon makes scale invariance approximate.

Question 6.7.2

6.7. Q/K normalization: check the equation and the axis

How do Q/K RMSNorm axes differ in the inspected OLMo2 and Qwen3 implementations?

One correct optionAOLMo2 normalizes whole Q/K projections before head splitting; Qwen3 normalizes within each headBBoth normalize only over the time axisCQwen3 joins all heads; OLMo2 normalizes each independently

A) OLMo2 normalizes whole Q/K projections before head splitting; Qwen3 normalizes within each head

B) Both normalize only over the time axis

C) Qwen3 joins all heads; OLMo2 normalizes each independently

Answer and explanation

Correct answer: A. A shared denominator couples heads: changing one affects others. Independent head reductions avoid that coupling.

Question 6.8.1

6.8. Why a gate adds a matrix

How does a gated FFN differ fundamentally from a plain two-matrix FFN?

One correct optionAIt mixes sequence positions through an additional attention matrix instead of transforming each position independentlyBIt removes the nonlinearity between matrix projections and reduces the FFN to a single linear mappingCIt creates input-dependent multiplicative interaction between two branches before down projection

A) It mixes sequence positions through an additional attention matrix instead of transforming each position independently

B) It removes the nonlinearity between matrix projections and reduces the FFN to a single linear mapping

C) It creates input-dependent multiplicative interaction between two branches before down projection

Answer and explanation

Correct answer: C. One branch computes SiLU(W_gx), another W_ux; their product passes through W_d. The input-dependent gate need not lie in [0,1].

Question 6.8.2

6.8. Why a gate adds a matrix

How accurate is GELU(z) ≈ z·sigmoid(1.702z) on [-6,6] in the module’s calculation?

One correct optionAThe maximum error is about 0.35BThe maximum error is about 0.0203CIt is accurate only for z>0

A) The maximum error is about 0.35

B) The maximum error is about 0.0203

C) It is accurate only for z>0

Answer and explanation

Correct answer: B. The module’s dense grid gives a maximum of about 0.02033. The public gpt-oss code also uses 1.702, but its activation additionally includes clipping and the up+1 shift.

Question 6.8.3

6.8. Why a gate adds a matrix

Can SiLU(g) in SwiGLU be interpreted as a probability?

One correct optionANo: g·sigmoid(g) can be negative or greater than oneBYes: sigmoid lies in [0,1], so the entire gate does tooCYes, if its output multiplies the up branch

A) No: g·sigmoid(g) can be negative or greater than one

B) Yes: sigmoid lies in [0,1], so the entire gate does too

C) Yes, if its output multiplies the up branch

Answer and explanation

Correct answer: A. Multiplication by g changes the range. “Gate” describes its role in the product, not probability normalization.

Question 6.9.1

6.9. Budgeting SwiGLU and reading its variants

Why can a model-specific clamp not be changed freely only at inference time?

One correct optionAIt is part of the function under which the weights were trainedBIt is used only while packaging the checkpoint and does not participate in the model forward passCIt affects only the weight-file size and does not change the activations computed by the model

A) It is part of the function under which the weights were trained

B) It is used only while packaging the checkpoint and does not participate in the model forward pass

C) It affects only the weight-file size and does not change the activations computed by the model

Answer and explanation

Correct answer: A. The gate cap, two-sided up clipping, 1.702 coefficient, and up+1 shift define the gpt-oss computation graph. Changing them changes the model function; low-precision motivation should not be asserted without a source.

Question 6.9.2

6.9. Budgeting SwiGLU and reading its variants

In the gpt-oss variant with g=u=z, what are the outputs at z=7 and z=9?

One correct optionAAbout 7 and 9BAbout 56 and 72CAbout 56 and 56

A) About 7 and 9

B) About 56 and 72

C) About 56 and 56

Answer and explanation

Correct answer: C. On the positive diagonal both branches hit their clamp limits: gate=7 and up=7, hence up+1≈8 and the output is about 56. Zero derivative past a threshold applies to the clipped branch; negative up clipping still affects negative inputs.

Question 6.10.1

6.10. From tensor shapes to 8,030,261,248 parameters

What fraction of a complete Llama-3.1-8B dense block is in the FFN matrices?

One correct optionAAbout 19.2%BAbout 80.8%CExactly 50%

A) About 19.2%

B) About 80.8%

C) Exactly 50%

Answer and explanation

Correct answer: B. The FFN has 176.16M parameters, attention 41.94M, and norms 8,192, for a 218.112M full block. Thus the FFN share is about 80.8% for this configuration.

Question 6.10.2

6.10. From tensor shapes to 8,030,261,248 parameters

How do the estimates 13d² and 13d²+2d relate for Llama-3.1-8B?

One correct optionA13d² counts attention+FFN matrices only; two RMSNorm gains make the full block 13d²+2dB13d² already includes both norms, while the extra +2d counts the input and output embedding tablesC13d² describes an MHA block, while +2d appears only after replacing MHA with GQA

A) 13d² counts attention+FFN matrices only; two RMSNorm gains make the full block 13d²+2d

B) 13d² already includes both norms, while the extra +2d counts the input and output embedding tables

C) 13d² describes an MHA block, while +2d appears only after replacing MHA with GQA

Answer and explanation

Correct answer: A. At d=4096, the matrix part is 218,103,808 parameters; norms add 8,192, giving 218,112,000. Small is not zero when the goal is an exact count.

Question 6.11.1

6.11. Element-wise alternatives: DyT and DyISRU

Why is DyT not identical to ordinary RMSNorm?

One correct optionADyT computes whole-batch statistics and depends on other examples, whereas RMSNorm processes each token independentlyBRMSNorm is a linear function with a fixed coefficient, while DyT merely adds a tanh nonlinearityCDyT acts element-wise, while RMSNorm couples coordinates through a shared denominator

A) DyT computes whole-batch statistics and depends on other examples, whereas RMSNorm processes each token independently

B) RMSNorm is a linear function with a fixed coefficient, while DyT merely adds a tanh nonlinearity

C) DyT acts element-wise, while RMSNorm couples coordinates through a shared denominator

Answer and explanation

Correct answer: C. DyT maps the same coordinate identically regardless of its neighbors; RMSNorm’s denominator changes with the whole vector. The notebook’s alpha≈1.43 is a course-owned synthetic fit, not a paper result.

Question 6.11.2

6.11. Element-wise alternatives: DyT and DyISRU

With γ=1 and β=0, which b in x/√(b+x²) gives slope 0.5 at zero?

One correct optionAb=4Bb=0.5Cb=1.43

A) b=4

B) b=0.5

C) b=1.43

Answer and explanation

Correct answer: A. The derivative at zero is 1/√b, and 1/√4=0.5. The coefficient 1.43 belongs to a separate scalar DyT fit.

Question 6.12.1

6.12. Several residual streams: what mHC constrains

What does Manifold-Constrained Hyper-Connections (mHC) constrain?

One correct optionAHidden states from every residual stream are forced onto one low-dimensional feature manifoldBMixing matrices across multiple residual streams are projected into a constrained spaceCOnly the late-stage learning-rate schedule is constrained, while the residual-link structure remains unchanged

A) Hidden states from every residual stream are forced onto one low-dimensional feature manifold

B) Mixing matrices across multiple residual streams are projected into a constrained space

C) Only the late-stage learning-rate schedule is constrained, while the residual-link structure remains unchanged

Answer and explanation

Correct answer: B. mHC constrains the residual mixing matrix across streams. Finite Sinkhorn iterations give an approximately doubly stochastic matrix, so both row and column sums must be checked. The constraint is on connections, not a hidden-state manifold.

Question 6.12.2

6.12. Several residual streams: what mHC constrains

What should be checked after 20 Sinkhorn iterations?

One correct optionANonnegativity and residuals of both row and column sumsBOnly one column sum: 20 iterations guarantee everything elseCEquality of the result to the identity matrix

A) Nonnegativity and residuals of both row and column sums

B) Only one column sum: 20 iterations guarantee everything else

C) Equality of the result to the identity matrix

Answer and explanation

Correct answer: A. After finitely many iterations, one set of sums can still be inaccurate. Even an exact doubly stochastic matrix need not be the identity.

Question 6.15.1

6.15. Reading gpt-oss from code rather than from a guessed config field

How does original gpt-oss code split the packed gate/up projection?

One correct optionAEven positions are gate, odd positions are upBThe first half is gate, the second half is upCThe same projected values serve both branches

A) Even positions are gate, odd positions are up

B) The first half is gate, the second half is up

C) The same projected values serve both branches

Answer and explanation

Correct answer: A. The code uses x[..., ::2] and x[..., 1::2]. Splitting into halves preserves shapes but changes the function.