Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.
Enable JavaScript for automatic checking. Questions and expandable answers are available below.
Checked: 0 / 46 · Correct: 0
Reset progress Question 3.1.1 3.1. Why the Transformer was needed
What problem did the Transformer solve compared with recurrent processing?
A) Positions are processed in parallel and the link is direct
B) The parameter count grows more slowly as the stack gets deeper
C) The model no longer needs any positional signal at its input
Answer and explanation Correct answer: A. In a recurrent network the path between distant positions equals their distance, and training is sequential in time. Attention gives a path of length one and a parallel pass over the whole sequence.
Question 3.1.2 3.1. Why the Transformer was needed
When is a reader ready to move from M3 to M4?
A) Recognizing the names in the diagram is sufficient
B) They can reconstruct the computation and answer the self-check
C) They must first read every external source
Answer and explanation Correct answer: B. Reconstructing the computation tests understanding beyond recognizing names. The reference sections remain available in later chapters.
Question 3.2.1 3.2. Notation and tensor shapes
What does d_model denote in the base Transformer?
A) The number of positions the model handles in one forward pass
B) The inner width of the FFN between its two linear projections
C) The width of the residual stream, shared across every layer
Answer and explanation Correct answer: C. It is the one width seen by the input and output of every sublayer. The inner dimensions — d_k, d_v, d_ff — live inside sublayers and never reach the bus.
Question 3.2.2 3.2. Notation and tensor shapes
B = 2, T = 5, h = 4 heads. What is the shape of the attention weight matrix?
A) 2 × 4 × 5 × 5 — one square matrix for each of the 4 heads
B) 2 × 5 × 32 — the weights live in the same width as the residual stream
C) 2 × 4 × 5 × 8 — each position gets a weight vector of length 8
Answer and explanation Correct answer: A. Explicitly stored weights require B·h·T² values. An optimized algorithm can compute the result in tiles without storing the full matrix.
Question 3.3.1 3.3. Step 1: tokens, embeddings, and positions
What does an embedding lookup do?
A) It compresses the vocabulary to a multiple of the head count
B) It replaces a token identifier with a row of a learned matrix
C) It computes the token's similarity with all others in the vocabulary
Answer and explanation Correct answer: B. It is plain indexing: an integer becomes a vector of width d_model. The matrix trains with the model, and modern LLMs often tie it to the output projection.
Question 3.3.2 3.3. Step 1: tokens, embeddings, and positions
What happens when rows of unmasked self-attention input are permuted without positional information?
A) Every output row stays in its original location
B) The output becomes zero because the order changed
C) Output rows undergo the same permutation
Answer and explanation Correct answer: C. This is permutation equivariance. A fixed causal mask breaks the assumptions of this statement.
Question 3.4.1 3.4. Step 2: scaled dot-product attention
How should the roles of Q, K and V be read?
A) The query asks, the key answers a match, the value is what travels
B) All three are copies of the input differing only in projection scale
C) Q and K encode position while V encodes the content of the token
Answer and explanation Correct answer: A. The match between query and key sets the weight, and it is the value's content that travels. The three projections train independently, so "what to look for" and "what to hand over" are different things.
Question 3.4.2 3.4. Step 2: scaled dot-product attention
Q, K, and V have shape 2 × 4 × 5 × 8. What is the shape of Z = AV before merging heads?
A) 2 × 4 × 5 × 5 — the head output inherits the weight matrix shape
B) 2 × 4 × 5 × 8 — key weights are applied to the values
C) 2 × 5 × 32 — the head returns straight into the residual stream
Answer and explanation Correct answer: B. The weight matrix B × h × T × T multiplies values of B × h × T × d_v and yields a positional output at head width. It returns to the bus only after concatenating heads and applying W_O.
Question 3.5.1 3.5. Masks: who is allowed to see whom
How does a padding mask differ from a causal mask?
A) Padding is applied before the softmax while causal is applied after
B) Padding is needed only in training and causal only at inference
C) One hides non-existent tokens, the other hides future positions
Answer and explanation Correct answer: C. Boolean allowed masks combine with AND. One excludes PAD keys; the other excludes keys later than the query position.
Question 3.5.2 3.5. Masks: who is allowed to see whom
Why can the softmax not be computed first and forbidden weights zeroed afterwards?
A) The remaining weights would stop summing to one along the row
B) Zeroing after the softmax would need an extra pass over the matrix
C) Zeroing weights after softmax changes the output vector's dimensionality
Answer and explanation Correct answer: A. Softmax normalizes across the whole row including forbidden keys, and part of the mass leaks to them. Logits must be masked with minus infinity before normalization so the distribution covers only what is allowed.
Question 3.5.3 3.5. Masks: who is allowed to see whom
What does True mean in this chapter’s allowed mask?
A) The connection is permitted
B) The connection is forbidden
C) It depends on the input tensor dtype
Answer and explanation Correct answer: A. The name allowed matches the meaning of True. Another API may use the inverse convention.
Question 3.6.1 3.6. Multi-head attention
With d_model = 32, h = 4, and the standard equal split, what is the head width?
A) 32: each head retains the entire projected width
B) 8: the projected result is split across four heads
C) 128: head count multiplies the resulting width
Answer and explanation Correct answer: B. 32 / 4 = 8. The projection-weight count stays 4·32²; explicitly stored attention weights also depend on head count.
Question 3.6.2 3.6. Multi-head attention
Why does multi-head attention need the output projection W_O?
A) It casts the output to the right precision before the residual add
B) It normalizes head outputs so that their scales become comparable
C) It learns to mix head channels and sets the output-vector width
Answer and explanation Correct answer: C. W_O mixes head outputs within the attention sublayer and returns the required residual width. Removing it removes this particular mixing operation; later FFNs or projections can still mix those channels.
Question 3.7.1 3.7. The FFN: nonlinear processing at each position
What does it mean that the FFN is position-wise?
A) The same weights are applied to every position independently
B) A separate set of layer weights is learned for each position
C) The FFN processes positions sequentially, from left to right
Answer and explanation Correct answer: A. Mixing across positions is attention's job; the FFN only transforms each vector in place. A division of labour: attention routes, the FFN computes.
Question 3.7.2 3.7. The FFN: nonlinear processing at each position
What happens if the nonlinearity between the two FFN linear layers is removed?
A) They become attention across positions
B) They can be combined into one affine transformation
C) They still represent an arbitrary nonlinear function
Answer and explanation Correct answer: B. A composition of affine maps remains affine. Width alone does not create nonlinearity; ReLU, GELU, or another activation supplies it.
Question 3.8.1 3.8. Residual connections and LayerNorm
Along which axis does LayerNorm normalize a Transformer representation?
A) Along the batch: statistics are taken over all examples at once
B) Along positions: each position is normalized against its neighbours
C) Along features: statistics are taken inside a single vector
Answer and explanation Correct answer: C. Here LayerNorm computes statistics over one token’s final feature axis. For a fixed input vector, the result does not depend on other batch items.
Question 3.8.2 3.8. Residual connections and LayerNorm
How is the residual update written in the pre-norm scheme?
A) x ← x + Sub(Norm(x)): the sublayer reads a normalized state
B) x ← Norm(x + Sub(x)): the norm follows the residual addition
C) x ← Norm(x) + Sub(x): norm and sublayer are applied in parallel
Answer and explanation Correct answer: A. Pre-norm leaves a direct residual path around the sublayer, with normalization inside the update branch. This does not guarantee perfect gradient stability throughout the network.
Question 3.9.1 3.9. The encoder layer and encoder stack
Which keys can a position read in ordinary encoder self-attention?
A) Only keys to its left
B) All content positions in the input, excluding PAD keys
C) Only keys from decoder outputs
Answer and explanation Correct answer: B. The encoder uses bidirectional context. Padding restrictions still apply: the full input does not mean PAD keys are included.
Question 3.9.2 3.9. The encoder layer and encoder stack
Why do encoder-only models remain useful?
A) They are cheaper to train and so serve as drafting models
B) They generate text faster thanks to having no causal mask
C) Classification and retrieval need a text vector, not a continuation
Answer and explanation Correct answer: C. Classification and retrieval can use representations of an already available text. An encoder processes it bidirectionally; generating a continuation is not required.
Question 3.10.1 3.10. The decoder layer and cross-attention
Where do Q, K and V come from in decoder cross-attention?
A) Q from the decoder, K and V from the encoder output
B) All three from the encoder: the decoder only applies the weights
C) Q and K from the encoder, V from the decoder's current state
Answer and explanation Correct answer: A. The decoder asks and the encoder answers: the query is formed by what has been generated so far while the content comes from the encoded input. That is the mechanism transferring information between sequences.
Question 3.10.2 3.10. The decoder layer and cross-attention
What targets correspond to the toy training input BOS, A, B?
A) BOS, A, B — the same tokens
B) A, B, EOS — the next token at each position
C) Only EOS at the final position
Answer and explanation Correct answer: B. The shift defines the next-token targets. Causal masking prevents a position from reading its target from the next input position.
Question 3.11.1 3.11. Training: teacher forcing and cross-entropy
What does teacher forcing mean?
A) The model trains on its own predictions from previous steps
B) A teacher of a larger model supplies the target distributions
C) Ground-truth tokens are fed as input rather than generated ones
Answer and explanation Correct answer: C. Known training prefixes allow positions to be processed in parallel under a causal mask. During generation, the model’s selected tokens may create prefixes that were rare in training.
Question 3.11.2 3.11. Training: teacher forcing and cross-entropy
A 100-token vocabulary. What does cross-entropy measure, and what is the uniform-prediction reference value?
A) The model's surprise at the true token; about 4.605 nats for uniform logits
B) The share of argmax matches with truth; by construction it cannot exceed 1.000
C) The distance between embeddings; the reference follows from the 100-row matrix norm
Answer and explanation Correct answer: A. Here cross-entropy is the negative log-probability of the true token, measured in nats. A uniform distribution gives NLL = ln(V) = ln(100) ≈ 4.605. This is a baseline, not an upper bound: a confidently wrong model can incur a much larger loss.
Question 3.11.3 3.11. Training: teacher forcing and cross-entropy
Can a token excluded by the loss mask remain available as context?
A) No, its key and value are automatically removed
B) Yes; a separate attention mask controls access
C) Only if its token ID is zero
Answer and explanation Correct answer: B. A loss mask selects scored predictions; an attention mask selects accessible connections.
Question 3.12.1 3.12. Generation: the same forward pass in a different loop
Why is generation sequential when training is parallel?
A) The causal mask at inference forbids computing positions at once
B) Token t+1 cannot be fed as input until it has been generated
C) Positional embeddings are computed one by one in order
Answer and explanation Correct answer: B. In training the whole true sequence is known and the mask merely forbids peeking. In generation the future simply does not exist yet, and that is a constraint of the task, not of the implementation.
Question 3.12.2 3.12. Generation: the same forward pass in a different loop
What does a causal model’s KV cache retain when weights and positions stay fixed?
A) Probabilities of all future tokens
B) Only queries from previous positions
C) Past tokens’ keys and values at each layer
Answer and explanation Correct answer: C. A new token creates a new query. Past keys and values can be reused because appending future tokens leaves earlier causal states unchanged.
Question 3.13.1 3.13. A minimal scaled dot-product attention implementation
What does the displayed scaled_dot_product_attention function return for float16 inputs?
A) float32; the MHA wrapper casts before its output projection
B) float16, because the function always preserves input dtype
C) float64 regardless of the input dtypes
Answer and explanation Correct answer: A. The teaching function computes in at least float32, or float64 when an input uses float64. Its dtype contract differs from the MHA wrapper’s.
Question 3.13.2 3.13. A minimal scaled dot-product attention implementation
How does the causal mask change during decode with a cache and a query offset?
A) No mask is needed at all: the cache holds only permitted positions
B) The query position is compared against the key's absolute cache index
C) The mask is built from the cache length and applied to all rows at once
Answer and explanation Correct answer: B. With T_q = 1 and a long cache the mask row must permit every past key and forbid future ones relative to the absolute position. An off-by-one here raises no exception — the model simply sees one token more or less.
Question 3.14.1 3.14. A minimal Transformer block
Both update branches in a pre-norm block return zero, including biases. What is the output?
A) The original x, through the residual additions
B) LayerNorm(x), because the norm acts on the main path
C) A zero tensor, because both branches are zero
Answer and explanation Correct answer: A. Each update becomes x + 0. This requires zero branch outputs and pre-norm; post-norm would still normalize the main path.
Question 3.14.2 3.14. A minimal Transformer block
How can a block's causality be checked?
A) Feed two sequences that differ only in a future token
B) Compare the block output with the same implementation run without a mask
C) Verify that the attention-weight matrix is strictly lower-triangular
Answer and explanation Correct answer: A. Compare outputs over the shared prefix with allclose and explicit rtol/atol. Bitwise equality is appropriate only for one deterministic CPU path; fused or GPU kernels can change rounding slightly. Shape or triangularity alone misses errors such as applying the mask after softmax.
Question 3.15.1 3.15. A complete shape trace for a toy model
B = 2, T = 5, d_model = 32, h = 4, d_ff = 128. What is the shape of the packed QKV?
A) 2 × 5 × 32 — packing does not change the residual stream width
B) 2 × 5 × 96 — three projections of 32 computed as one matrix
C) 2 × 4 × 5 × 8 — packing splits the tensor by heads immediately
Answer and explanation Correct answer: B. The head split comes next and gives 2 × 4 × 5 × 8. Scores are then 2 × 4 × 5 × 5, and after concatenation and W_O the stream returns to 2 × 5 × 32.
Question 3.15.2 3.15. A complete shape trace for a toy model
What is excluded from the main layer-weight count in §15.1?
A) All four attention projections
B) All weights of the wide FFN
C) Biases, norm parameters, embeddings, and the model’s other layers
Answer and explanation Correct answer: C. 4d_model² + 2d_model·d_ff counts MHA and FFN weights for one encoder layer. A complete model includes additional parameters.
Question 3.16.1 3.16. The 2017 original and a modern LLM: what survived
Which components of the base Transformer survive in a modern decoder-only LLM?
A) Tokens into vectors, attention, FFN, residual connection and norm
B) The encoder stack, cross-attention and sinusoidal positions
C) Post-norm ordering, ReLU in the FFN and training on translation
Answer and explanation Correct answer: A. The immobile core: discrete input becomes vectors, position and mask impose structure, attention routes, the FFN computes, residual and norm sustain depth. Everything else changed.
Question 3.16.2 3.16. The 2017 original and a modern LLM: what survived
Which substitution is characteristic of a modern LLM against the 2017 original?
A) Attention has been replaced by fully connected position mixing
B) Absolute sinusoids have been replaced by RoPE and its extensions
C) The residual connection has been replaced by learned hyper-connections in every model
Answer and explanation Correct answer: B. The list of substitutions is long: MHA to GQA or MLA, post-norm to pre-norm, LayerNorm to RMSNorm, ReLU MLP to SwiGLU. But each changes a detail of the core, not the core itself.
Question 3.17.1 3.17. Common confusions
Why does a large attention weight not prove causal influence?
A) Because weights are normalized and their absolute size is conventional
B) Because the weight is computed before the output projection W_O
C) The weight shows a route; the contribution also depends on the value
Answer and explanation Correct answer: C. The contribution also depends on the value vector, output projection, and other paths through the model. One mixing coefficient cannot establish a token’s complete causal contribution.
Question 3.17.2 3.17. Common confusions
Why does the causal mask not make training sequential?
A) All positions compute in one pass; the mask merely forbids looking ahead
B) Because the mask is applied only to the first layer of the stack
C) Because training uses a bidirectional mask rather than a causal one
Answer and explanation Correct answer: A. The true sequence is known in advance, so predictions for every position are computed in parallel. Only generation becomes sequential, where the future does not exist yet.
Question 3.18.1 3.18. Self-check
What should a reader do if several self-check questions remain difficult?
A) Skip the corresponding definitions until the end of the course
B) Revisit the relevant sections before Module 4
C) Immediately increase the size of the toy model
Answer and explanation Correct answer: B. The self-check identifies a part of the computation to revisit. Module 4 uses these concepts to study position representations.
Question 3.18.2 3.18. Self-check
Which axis does attention softmax normalize when a query has allowed keys?
A) Normalization runs over features inside a single attention head
B) Normalization runs over the batch so examples stay comparable
C) Normalization runs over keys: one query's weights sum to one
Answer and explanation Correct answer: C. Each query distributes unit mass over its allowed keys. Using a different axis can leave tensor shapes valid while changing the operation. Check sums along the key axis; successful execution alone does not establish correctness.
Question 3.19.1 3.19. Exercises with solutions
What does the exercise with two sequences differing only in the future check?
A) On the tested inputs, the shared prefix stays equal within the stated tolerance
B) That the model is robust to small changes in its input data
C) That positional embeddings are applied correctly to both rows
Answer and explanation Correct answer: A. Changing future tokens can expose leakage into earlier positions. The comparison uses explicit rtol/atol. Passing establishes the result for the tested inputs, not causality for every sequence; mask inspection and other probes complement it.
Question 3.19.2 3.19. Exercises with solutions
What does successfully learning the period 0,1,2,3,4,5 establish?
A) The model has learned to handle long contexts
B) Training and generation work on this simple task
C) Attention is necessary to predict this period
Answer and explanation Correct answer: B. One preceding token determines the next token here. Success checks the implementation without demonstrating long-range dependencies or generalization to new data.
Question 3.20.1 3.20. The module notebook
Why does the notebook verify its embedded numerical contract by SHA-256?
A) To confirm the notebook was downloaded from the official repository
B) To speed startup: checking a sum is cheaper than recomputing values
C) So that an edit to the reference values cannot pass silently
Answer and explanation Correct answer: C. The hash identifies the embedded bytes. It checks integrity, not mathematical correctness; separate recalculation and independent implementations check correctness.
Question 3.20.2 3.20. The module notebook
Why is the published notebook shipped without saved outputs?
A) So the reader gets the results of their own run, not someone else's
B) Because saved outputs inflate the file size several times over
C) Because outputs contain file paths from the author's specific machine
Answer and explanation Correct answer: A. Someone else's numbers in the cells create the illusion that the code was verified on your machine too. A clean notebook forces execution and shows your own numbers — including any discrepancies.
Question 3.21.1 3.21. Reading route and sources
Which source is the primary reference for the 2017 Transformer architecture?
A) The survey post The Illustrated Transformer with its diagram walkthrough
B) The paper Attention Is All You Need by Vaswani and co-authors
C) The documentation of the framework the architecture was implemented in
Answer and explanation Correct answer: B. Illustrated walkthroughs help on a second pass, but the primary source fixes the formulations and notation. Discrepancies between popular retellings are usually resolved by it.
Question 3.21.2 3.21. Reading route and sources
Why does the reading route suggest several sources instead of one?
A) Because no single source covers every section of the module
B) Because the sources contradict each other and must be cross-checked
C) Formula, picture and code explain the same thing from different sides
Answer and explanation Correct answer: C. The paper gives precision, the illustration intuition, the annotated implementation the mapping from formulas to lines of code. A gap left after one pass usually closes on another format.
Question 3.22.1 3.22. Final cheat sheet and the next step
What is the operation order in an encoder block of the original post-norm Transformer?
A) Attention → residual addition and norm → FFN → residual addition and norm
B) Attention → FFN → one residual addition and one norm after both sublayers
C) One input norm → attention → FFN with no residual connections
Answer and explanation Correct answer: A. Each sublayer in the original post-norm design has its own wrapper: add the sublayer input to its output, then normalize. Add & Norm therefore appears between attention and the FFN as well as after the FFN.
Question 3.22.2 3.22. Final cheat sheet and the next step
What does the conclusion "the computational skeleton remains recognizable" mean?
A) That modern models are trained in the same way as the original was
B) That the changes touch details of the core rather than its scheme
C) That differences between architecture generations do not matter in practice
Answer and explanation Correct answer: B. The Transformer branch studied here retains attention, FFNs, and residual paths. Replacing components can substantially affect quality and cost; other families introduced in M1 may use a different graph.