Modern LLMs3 / 20

Module 3. The Basic Transformer

Try each exercise first. Expand worked answers or project guidance when needed; their open state is shared between RU/EN.

Download course files (.zip) 2588.1 KB

The archive preserves the source folder structure, both languages, and supporting files.

Exercise 1. The correct softmax axis

scores has shape [B, h, T_q, T_k] = [2, 4, 3, 5].

What is the shape of weights?
Which sum must equal one?
What does softmax(dim=-2) do?

Show answer and solution

Solution. The shape remains [2,4,3,5]. For each fixed (batch, head, query), the sum across the final key axis is one. dim=-2 normalizes the three query positions for each key. The code runs but no longer implements attention.

Exercise 2. Dot-product scaling

Let $d_k=64$ and assume independent, zero-mean, unit-variance coordinates. Estimate the variance of $q^\top k$ before and after scaling.

Show answer and solution

Solution. Under the stated assumptions, the variance is 64 before scaling. Dividing by $\sqrt{64}=8$ divides variance by $8^2$ , leaving 1. The notebook samples 50,000 pairs with seed 18, prints the empirical variances, and checks an 8% tolerance around these theoretical values.

Exercise 3. Attention by hand

Using the matrices in §4.5, reconstruct the second causal output row.

Show answer and solution

Solution. The available logits are [0,0.707], giving softmax weights [0.330238,0.669762]. Therefore

Z_2\approx0.330238[1,0]+0.669762[0,2] \approx[0.330238,1.339523].

Exercise 4. LayerNorm

Normalize $x=[1,2,3,4]$ with $\gamma=1$ , $\beta=0$ , and $\varepsilon=0$ .

Show answer and solution

Solution. $\mu=2.5$ , $\sigma^2=1.25$ , and standard deviation $\sqrt{1.25}\approx1.1180$ . Thus

[-1.3416,-0.4472,0.4472,1.3416].

Exercise 5. Layer parameters

For $d_{model}=512$ and $d_{ff}=2048$ , count MHA and FFN weights without biases.

Show answer and solution

Solution. MHA: $4\cdot512^2=1\,048\,576$ . FFN: $2\cdot512\cdot2048=2\,097\,152$ . The total is $3\,145\,728$ , and the FFN owns $2/3$ .

Exercise 6 [project]. Catch future leakage

Implement a decoder block once with a causal mask and once without it. Create two sequences that share their first three positions but differ at the fourth. Compare logits at the first three positions.

Project demos and guidance

Expected result. With masking, the logits agree up to floating-point tolerance. Without masking, at least some earlier logits differ. The module notebook includes a reproducible test.

Exercise 7 [project]. Draw the block from memory

On a blank page, reconstruct:

one complete encoder layer;
one complete original decoder layer;
a modern pre-norm decoder-only block.

Label every arrow with a tensor shape, then compare your result with VIZ 02 and VIZ 08.

Project demos and guidance

Readiness criterion. You included a separate Add & Norm after every original sublayer, sent decoder queries and encoder keys/values into cross-attention, and preserved $d_{model}$ along the residual stream.

Exercise 8 [project]. A tiny language model

Wrap one DecoderBlock with token and positional embeddings, a final norm, and a vocabulary head. Overfit the model on a short periodic sequence. Check that:

loss decreases;
generated continuation follows the pattern;
the causal-invariance test passes;
removing the causal mask exposes the next target token, even if the loss on this simple period does not decrease further.

Project demos and guidance

Boundary of the result. A single preceding token determines the next item of this period. Success does not show that attention is necessary or that the model uses long contexts. This experiment checks training and generation on a simple task, without a separate generalization test.