Module 6. Normalization, activations, and the anatomy of a Transformer block

Solution. In GQA, $Q$ and $O$ are $d\times d$, while K/V width is one quarter of $d$. The two K/V matrices therefore contribute $2d(d/4)=0.5d^2$, giving

$P_{attn}=2d^2+0.5d^2=2.5d^2.$

SwiGLU has gate, up, and down projections. With $d_{ff}=3.5d$, their count is $3dd_{ff}=10.5d^2$. Thus

$P_{matrix}=13d^2=13\cdot4096^2=218{,}103{,}808.$

Two RMSNorm scales add $2d=8192$, yielding 218,112,000. Biases are excluded by this architecture contract.

Solution. RMSNorm divides by $\sqrt{\operatorname{RMS}(x)^2+\epsilon}$. Here the signal-square term is $10^{-6}$, so

$g_{10^{-6}}=1/\sqrt{2\cdot10^{-6}}\approx707.1,$ $g_{10^{-5}}=1/\sqrt{11\cdot10^{-6}}\approx301.5.$

The larger epsilon more than halves the gain because it dominates this small signal. Epsilon must therefore reflect reduction precision and expected activation scale, not dtype machine epsilon alone.

Solution. Use $g=\min(z,7)$, $u=\operatorname{clip}(z,-7,7)$, and $y=g\sigma(1.702g)(u+1)$:

At $z=-9$, the gate is not lower-clipped and its sigmoid nearly suppresses it, but the up branch is still changed to $-7$ and enters as $u+1=-6$. The output is small because of the gate, not because the clip is irrelevant. At 7 and 9, both used branches are identical, so the outputs match.

Demo for this project exercise: English project-demo notebook for Module 6, section “Demo 1”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

Demo for this project exercise: English project-demo notebook for Module 6, section “Demo 2”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

Solution. For a positive scale $c$, factor it out of the denominator:

Keeping $\epsilon>0$ fixed therefore changes the normalization. With a single value $x=10^{-3}$, $\epsilon=10^{-6}$, and unit learned scale, the output is $1/\sqrt2\approx0.7071$. Doubling the input gives $2/\sqrt5\approx0.8944$. The stabilizer is comparable to the squared signal, so the difference is substantial.

When RMS is reduced over all heads together, scaling one head changes the shared denominator and hence the untouched heads too. Separate head-wise reductions keep their denominators independent. Predict which heads will change before comparing the two notebook calculations.

Solution. Each row and column of the matrix in §12 sums to $0.75+0.25=1$. Its first output is $0.75\cdot2+0.25\cdot6=3$; its second is $0.25\cdot2+0.75\cdot6=5$. Thus $AX=(3,5)^\top\ne X$.

Nonnegative entries and unit row sums make each output a convex combination, bounded between 2 and 6. The output range can shrink: here it is 3 to 5. Unit column sums additionally preserve the sum across streams: $3+5=2+6$. Neither property preserves each stream separately. After a finite number of Sinkhorn–Knopp iterations, check both sum residuals before treating these properties as exact.