Modern LLMs5 / 20

Module 5. Modern Attention Mechanisms

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 23 · Correct: 0

Question 5.4.1

5.4. SDPA, masks, and the square-root scale

Why does SDPA divide dot products by $\sqrt{d_h}$ ?

One correct optionATo normalize dot-product variance to order oneBTo force softmax to select exactly one allowed key per rowCTo make output shape independent of query- and KV-head counts

A) To normalize dot-product variance to order one

B) To force softmax to select exactly one allowed key per row

C) To make output shape independent of query- and KV-head counts

Answer and explanation

Correct answer: A. If all query and key components are independent, zero-mean, and unit-variance, then Var(q·k)=d_h. Division by sqrt(d_h) restores unit variance. This motivates the initial scale; it does not assert permanent independence of learned features.

Question 5.4.2

5.4. SDPA, masks, and the square-root scale

Which keys may query position 5 attend to with W=3 including the current position?

One correct optionA0, 1, 2B3, 4, 5C4, 5, 6

A) 0, 1, 2

B) 3, 4, 5

C) 4, 5, 6

Answer and explanation

Correct answer: B. The condition 0 ≤ p_q − p_k < 3 retains positions 3–5 and excludes the future.

Question 5.4.3

5.4. SDPA, masks, and the square-root scale

Why can SDPA is_causal=True be wrong for one query after a prefix?

One correct optionAThe rectangular mask is upper-left alignedBSDPA cannot use float64CCaching automatically disables every mask

A) The rectangular mask is upper-left aligned

B) SDPA cannot use float64

C) Caching automatically disables every mask

Answer and explanation

Correct answer: A. Row 0 need not be token position 0. Supply an explicit position-aware mask with is_causal disabled.

Question 5.5.1

5.5. Worked storage arithmetic: four head layouts and one hybrid model

Under which storage policy does the gpt-oss-120b K/V estimate at 131,072 positions fall from 9 to 4.504 GiB?

One correct optionAAll hybrid KV values use int8 rather than bf16 storageBThe runtime caps 18 local caches at 128 positions; 18 other layers retain the full contextCHalf of each layer’s query heads are disabled at long context

A) All hybrid KV values use int8 rather than bf16 storage

B) The runtime caps 18 local caches at 128 positions; 18 other layers retain the full context

C) Half of each layer’s query heads are disabled at long context

Answer and explanation

Correct answer: B. 4.5 GiB for full layers plus 4.5 MiB for local layers gives 4.50439453125 GiB of K/V payload for one sequence. A mask alone does not guarantee eviction; weights and workspace are excluded.

Question 5.5.2

5.5. Worked storage arithmetic: four head layouts and one hybrid model

What does the 56.89× factor mean in the teaching MLA comparison?

One correct optionAEnd-to-end DeepSeek-V2 speedup over every possible MHA modelBReduction of the full checkpoint including MoE, embeddings, and optimizerCRaw-byte ratio between two stated KV-cache geometries

A) End-to-end DeepSeek-V2 speedup over every possible MHA model

B) Reduction of the full checkpoint including MoE, embeddings, and optimizer

C) Raw-byte ratio between two stated KV-cache geometries

Answer and explanation

Correct answer: C. It compares the stated MHA baseline with (512+64)·61·2 MLA storage. It is not the same baseline as the paper’s 93.3% cache reduction.

Question 5.6.1

5.6. GQA is a grouping invariant, not a quality guarantee

With $H_q=64$ and $H_{kv}=8$ , what is the GQA group size $g=H_q/H_{kv}$ ?

One correct optionA

g=8

: one KV head serves eight query headsB

g=1/8

: each query head owns one eighth of a KV headC

g=72

: the head counts are added after projection

A) $g=8$ : one KV head serves eight query heads

B) $g=1/8$ : each query head owns one eighth of a KV head

C) $g=72$ : the head counts are added after projection

Answer and explanation

Correct answer: A. 64/8=8. MHA has g=1, while MQA has H_kv=1.

Question 5.6.2

5.6. GQA is a grouping invariant, not a quality guarantee

Which quality conclusion is supported by the original GQA paper?

One correct optionAEvery GQA with group size eight is guaranteed to match MHA qualityBIn the authors’ tests, GQA was near MHA quality and MQA speedCMQA always outperforms GQA when both are trained from scratch

A) Every GQA with group size eight is guaranteed to match MHA quality

B) In the authors’ tests, GQA was near MHA quality and MQA speed

C) MQA always outperforms GQA when both are trained from scratch

Answer and explanation

Correct answer: B. The result is empirical and protocol-specific. It motivates the tradeoff but does not define a universal optimal group size.

Question 5.7.1

5.7. MLA: cache a shared latent and move the projections

Why does MLA decouple the RoPE branch when absorbing projections?

One correct optionARoPE is nonlinear, so no matrices can ever be composed around itBThe branch exists only for saved-format compatibility with GQACRotating reconstructed keys makes the effective query depend on the key position

A) RoPE is nonlinear, so no matrices can ever be composed around it

B) The branch exists only for saved-format compatibility with GQA

C) Rotating reconstructed keys makes the effective query depend on the key position

Answer and explanation

Correct answer: C. (R_m q)^T R_n A c_n produces effective query A^T R_(n-m)^T q, which depends on n. RoPE is linear; RA and AR may not even have compatible shapes.

Question 5.7.2

5.7. MLA: cache a shared latent and move the projections

How can q^T A c be evaluated without expanding key A c?

One correct optionA(A^T q)^T cB(A q)^T c for arbitrary shapesCq^T c with no A

A) (A^T q)^T c

B) (A q)^T c for arbitrary shapes

C) q^T c with no A

Answer and explanation

Correct answer: A. For A of shape d_k × d_c, A^T q has width d_c. Associativity moves the projection to the query side.

Question 5.8.1

5.8. FlashAttention: dense attention without the quadratic HBM object

Which three quantities does online softmax carry across tiles?

One correct optionARunning maximum, normalization sum, and weighted output numeratorBThe full logit row, probability row, and its causal maskCMaximum index, top-k keys, and an unweighted sum of values

A) Running maximum, normalization sum, and weighted output numerator

B) The full logit row, probability row, and its causal mask

C) Maximum index, top-k keys, and an unweighted sum of values

Answer and explanation

Correct answer: A. State (m, l, n) supports exact rescaling when a later tile raises the maximum, with final output n/l.

Question 5.8.2

5.8. FlashAttention: dense attention without the quadratic HBM object

What does FlashAttention make linear in sequence length?

One correct optionAThe number of dense dot products between all Q and K rowsBAdditional memory: the full

N\times N

matrix is not stored in HBMCThe number of keys visible to each query in dense attention

A) The number of dense dot products between all Q and K rows

B) Additional memory: the full $N\times N$ matrix is not stored in HBM

C) The number of keys visible to each query in dense attention

Answer and explanation

Correct answer: B. Dense arithmetic remains O(N²d). Additional storage becomes linear; IO is reduced but still depends on SRAM and tile geometry.

Question 5.8.3

5.8. FlashAttention: dense attention without the quadratic HBM object

What should online softmax do with a fully masked tile before any allowed key?

One correct optionAEvaluate exp(-inf - (-inf))BSkip the tile and keep the empty stateCReturn uniform weights over forbidden keys

A) Evaluate exp(-inf - (-inf))

B) Skip the tile and keep the empty state

C) Return uniform weights over forbidden keys

Answer and explanation

Correct answer: B. This avoids undefined subtraction. If every tile is empty, apply the chosen empty-row policy.

Question 5.8.4

5.8. FlashAttention: dense attention without the quadratic HBM object

Why does streaming computation not guarantee linear memory for the whole program?

One correct optionAA prebuilt dense N × N mask itself uses quadratic memoryBSoftmax retains every training epochCThe output always has shape N × N

A) A prebuilt dense N × N mask itself uses quadratic memory

B) Softmax retains every training epoch

C) The output always has shape N × N

Answer and explanation

Correct answer: A. Account for inputs, output, masks and temporaries. The teaching streaming path builds its mask per tile.

Question 5.9.1

5.9. Windowed attention and layer schedules

What does distance $L(W-1)$ describe for L consecutive local layers with equal windows W, including the current position, and no dilation?

One correct optionAGuaranteed preservation of every fact across the stated distanceBExact equivalence to one global layer with the same radiusCAn upper bound on a dependency path, not a memory guarantee

A) Guaranteed preservation of every fact across the stated distance

B) Exact equivalence to one global layer with the same radius

C) An upper bound on a dependency path, not a memory guarantee

Answer and explanation

Correct answer: C. A signal may travel through intermediate layers, but every hop transforms it. Path existence is not reliable direct retrieval.

Question 5.10.1

5.10. “Attention sink” refers to two different objects

How is the learned sink implemented in gpt-oss?

One correct optionAAn extra scalar score per head with no value contributionBThe first real token whose KV is forbidden from leaving the windowCOne fixed key vector shared by every query head

A) An extra scalar score per head with no value contribution

B) The first real token whose KV is forbidden from leaving the window

C) One fixed key vector shared by every query head

Answer and explanation

Correct answer: A. The sink is appended as a logit column, participates in normalization, and is removed before the V product.

Question 5.10.2

5.10. “Attention sink” refers to two different objects

Why can an ordinary fixed key not generally replace a constant learned sink?

One correct optionAAn ordinary key necessarily adds a nonzero value to the outputB

q^\top k

varies with the query, while a learned sink is constantCThe key cannot use the same data format as ordinary keys

A) An ordinary key necessarily adds a nonzero value to the output

B) $N\times N$ varies with the query, while a learned sink is constant

C) The key cannot use the same data format as ordinary keys

Answer and explanation

Correct answer: B. An ordinary dot product changes with q. Exact equivalence needs a special score slot or an augmented constant coordinate.

Question 5.10.3

5.10. “Attention sink” refers to two different objects

Does adding 2 only to token logits preserve the result when a learned sink is fixed?

One correct optionAYes, for all inputsBGenerally no: the sink probability changesCYes, because the sink value is zero

A) Yes, for all inputs

B) Generally no: the sink probability changes

C) Yes, because the sink value is zero

Answer and explanation

Correct answer: B. Softmax shift invariance requires shifting every logit, including the sink.

Question 5.11.1

5.11. Sparse attention: selecting a subset of positions

When was DSA officially introduced?

One correct optionAIn the DeepSeek-V2 technical report published in May 2024BIn the closed GLM-5.1 release announced in December 2025CIn the open DeepSeek-V3.2-Exp release on 29 September 2025

A) In the DeepSeek-V2 technical report published in May 2024

B) In the closed GLM-5.1 release announced in December 2025

C) In the open DeepSeek-V3.2-Exp release on 29 September 2025

Answer and explanation

Correct answer: C. DeepSeek-V3.2-Exp was built on V3.1-Terminus and debuted DeepSeek Sparse Attention with a report, weights, and kernels.

Question 5.11.2

5.11. Sparse attention: selecting a subset of positions

How does the original MoBA score a block?

One correct optionAA separate new trainable routerBThe query dot product with the mean block keyCOnly the block index

A) A separate new trainable router

B) The query dot product with the mean block key

C) Only the block index

Answer and explanation

Correct answer: B. There are no extra router parameters; Q/K representations learn with the model. Arbitrary attention replacement still needs quality evaluation.

Question 5.11.3

5.11. Sparse attention: selecting a subset of positions

With N=2048, d=32, r=8, k=64, what is (Nr+kd)/(Nd)?

One correct optionA0.03125B0.28125C1.28125

A) 0.03125

B) 0.28125

C) 1.28125

Answer and explanation

Correct answer: B. r/d + k/N = 0.25 + 0.03125. This counts score multiplications only, excluding top-k, projections and value aggregation; it is not measured speedup.

Question 5.12.1

5.12. A transparent NumPy reference

Why can a fully masked row produce NaN even before a division by zero?

One correct optionAStable softmax encounters the undefined

-\infty-(-\infty)

BA causal mask necessarily adds positive infinity to the rowCThe bf16 format cannot represent a probability of zero

A) Stable softmax encounters the undefined $g=8$

B) A causal mask necessarily adds positive infinity to the row

C) The bf16 format cannot represent a probability of zero

Answer and explanation

Correct answer: A. A row with no finite logit defines no distribution. The implementation must reject it, zero an explicit padding row, or provide a finite sink score.

Question 5.14.1

5.14. Reading gpt-oss configuration and code

Why does np.tile give the correct shape but wrong GQA grouping?

One correct optionAIt converts K/V from float to integer during repetitionBIt repeats the whole block; each KV head must repeat consecutivelyCIt automatically averages adjacent KV heads before expansion

A) It converts K/V from float to integer during repetition

B) It repeats the whole block; each KV head must repeat consecutively

C) It automatically averages adjacent KV heads before expansion

Answer and explanation

Correct answer: B. The required order is kv0,kv0,…,kv1,kv1,… . Tiling produces kv0,kv1,…,kv0,kv1,… and silently changes assignments.

Question 5.16.1

5.16. Multimodal attention: where image information is stored

What does 4096 × 72 KiB = 288 MiB count for early-fusion visual tokens?

One correct optionAAll memory of the multimodal modelBOnly added K/V for one sequence in the specified all-full layoutCOnly vision-encoder memory

A) All memory of the multimodal model

B) Only added K/V for one sequence in the specified all-full layout

C) Only vision-encoder memory

Answer and explanation

Correct answer: B. The hypothetical decoder uses 72 KiB per position. Weights, vision activations and temporary arrays are excluded. Local layers require window-aware accounting.