Modern LLMs4 / 20

Module 4. Positional representations: from sinusoids to multiple axes

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 23 · Correct: 0

Question 4.1.1

4.1. Motivation: content attention is not an ordering mechanism

What does permutation equivariance mean for bidirectional self-attention without position or a mask?

One correct optionAPermuting input positions produces the same permutation of the outputsBEvery permutation leaves the entire output tensor unchangedCSoftmax reconstructs order from attention values even without a mask

A) Permuting input positions produces the same permutation of the outputs

B) Every permutation leaves the entire output tensor unchanged

C) Softmax reconstructs order from attention values even without a mask

Answer and explanation

Correct answer: A. Equivariance means that the output transforms with the input; it is not permutation invariance. A causal mask adds access asymmetry but not a full distance geometry.

Question 4.3.1

4.3. Fourier intuition, complex rotations, and what is exact

What is exactly true for one sine-cosine pair in classic positional encoding?

One correct optionAIts dot product depends on the sum of absolute positionsBThe dot product equals cos((m−n)θ) for one fixed frequency θCRelative dependence appears only after projection weights are trained

A) Its dot product depends on the sum of absolute positions

B) The dot product equals cos((m−n)θ) for one fixed frequency θ

C) Relative dependence appears only after projection weights are trained

Answer and explanation

Correct answer: B. The identity sin(mθ)sin(nθ)+cos(mθ)cos(nθ)=cos((m−n)θ) is exact. RoPE differs by inserting relative phase directly into projected query-key geometry.

Question 4.4.1

4.4. Formalism: the frequency ladder and the rotary identity

Where is RoPE applied in the attention block studied here?

One correct optionARoPE rotates only scalar features after normalizationBAttention constructs Q and K without any linear projectionsCQ and K are projected first, then their coordinate pairs are rotated

A) RoPE rotates only scalar features after normalization

B) Attention constructs Q and K without any linear projections

C) Q and K are projected first, then their coordinate pairs are rotated

Answer and explanation

Correct answer: C. An arbitrary matrix W generally does not satisfy WR=RW. RoPE integrates cleanly because it is applied after the Q/K projections and preserves tensor shape.

Question 4.4.2

4.4. Formalism: the frequency ladder and the rotary identity

Partial RoPE rotates D coordinates of a head of width d_k. Which dimension defines the frequencies?

One correct optionAAlways the full residual width d_modelBThe rotated width DCSequence length T

A) Always the full residual width d_model

B) The rotated width D

C) Sequence length T

Answer and explanation

Correct answer: B. There are D/2 rotated pairs. Standard attention scaling still uses 1/√d_k.

Question 4.5.1

4.5. Worked example: three placements, one relative displacement

Why does a common position shift preserve a RoPE score when q, k, and frequencies are fixed?

One correct optionAThe common phase cancels, leaving only relative phaseBPositions are normalized by context length firstCThe √d scale removes absolute coordinates

A) The common phase cancels, leaving only relative phase

B) Positions are normalized by context length first

C) The √d scale removes absolute coordinates

Answer and explanation

Correct answer: A. R_{m+c}^T R_{n+c}=R_{n−m}. For fixed q and k, the logit depends on displacement rather than the coordinate origin.

Question 4.5.2

4.5. Worked example: three placements, one relative displacement

The rotated dot product is 1.144248 with d_k=4. What is the standard pre-mask attention logit?

One correct optionA1.144248B0.286062C0.572124

A) 1.144248

B) 0.286062

C) 0.572124

Answer and explanation

Correct answer: C. Divide the dot product by √4=2, not by four.

Question 4.6.1

4.6. Reading the RoPE frequency spectrum

What does increasing rope_theta primarily do?

One correct optionAIt removes the fastest frequencies and erases local orderBIt slows higher-index pairs more strongly, adding longer wavelengthsCIt automatically guarantees a larger usable context

A) It removes the fastest frequencies and erases local order

B) It slows higher-index pairs more strongly, adding longer wavelengths

C) It automatically guarantees a larger usable context

Answer and explanation

Correct answer: B. The fastest pair θ₀=1 is unchanged. A larger base reallocates the spectrum toward slower high-index coordinates; it does not guarantee long-context quality.

Question 4.6.2

4.6. Reading the RoPE frequency spectrum

Why is the largest wavelength of one RoPE pair not a hard context limit?

One correct optionAAll coordinate pairs share one common period for the full vectorBThe causal mask removes periodicity from every rotary componentCThe full code combines many frequencies; one repeated pair does not repeat the vector

A) All coordinate pairs share one common period for the full vector

B) The causal mask removes periodicity from every rotary component

C) The full code combines many frequencies; one repeated pair does not repeat the vector

Answer and explanation

Correct answer: C. Each pair is periodic, but the complete code is multi-frequency. Wavelengths and turn counts are diagnostics rather than a physical global limit.

Question 4.7.1

4.7. Context extension changes a spectrum, not just one number

Why is Position Interpolation normally followed by fine-tuning?

One correct optionAIt compresses every phase and changes the familiar local geometryBIt adds a new table of learned vectors for every positionCIt works only in bidirectional encoders with a full mask

A) It compresses every phase and changes the familiar local geometry

B) It adds a new table of learned vectors for every position

C) It works only in bidirectional encoders with a full mask

Answer and explanation

Correct answer: A. PI maps m→m/s and slows the whole spectrum. New positions stay inside the old angular range, but local geometry changes as well.

Question 4.7.2

4.7. Context extension changes a spectrum, not just one number

What is the most accurate description of static NTK-aware scaling?

One correct optionAA theorem guaranteeing safe extrapolation without fine-tuningBA heuristic that reshapes the spectrum through a new baseCA form of ALiBi with a learned slope for every attention head

A) A theorem guaranteeing safe extrapolation without fine-tuning

B) A heuristic that reshapes the spectrum through a new base

C) A form of ALiBi with a learned slope for every attention head

Answer and explanation

Correct answer: B. The base·s^(D/(D−2)) formula is motivated by spectral structure, but its effectiveness is model- and length-dependent and must be measured.

Question 4.7.3

4.7. Context extension changes a spectrum, not just one number

The entire query and key are each scaled by a≈1.3466. How does their dot product change?

One correct optionAEvery runtime must multiply final logits by exactly 1.3466BEvery inverse frequency is divided by an additional factor of 1.3466CIf the factor enters q and k, their dot-product score receives about 1.8133

A) Every runtime must multiply final logits by exactly 1.3466

B) Every inverse frequency is divided by an additional factor of 1.3466

C) If the factor enters q and k, their dot-product score receives about 1.8133

Answer and explanation

Correct answer: C. Bilinearity gives (aq)ᵀ(ak)=a²qᵀk, with a²≈1.8133. Partial RoPE requires an extra check because unrotated channels may not receive the factor.

Question 4.7.4

4.7. Context extension changes a spectrum, not just one number

What does the static NTK-aware formula do for s≥1 and D>2?

One correct optionAPreserve the first frequency and divide the last by sBLeave all frequencies unchangedCGuarantee model quality at any length

A) Preserve the first frequency and divide the last by s

B) Leave all frequencies unchanged

C) Guarantee model quality at any length

Answer and explanation

Correct answer: A. The frequency ratio is s^(−2i/(D−2)): one at i=0 and 1/s at the last pair.

Question 4.8.1

4.8. ALiBi: a distance prior in the logits

What does ALiBi do?

One correct optionAIt adds a head-specific linear penalty for relative distance to the logitBIt rotates queries and keys by a shared relative-position angleCIt learns a separate absolute-position table for every attention head

A) It adds a head-specific linear penalty for relative distance to the logit

B) It rotates queries and keys by a shared relative-position angle

C) It learns a separate absolute-position table for every attention head

Answer and explanation

Correct answer: A. ALiBi leaves Q/K unchanged and adds a positional bias. An exponential distance prior appears under fixed content scores, while full attention still depends on qᵀk.

Question 4.8.2

4.8. ALiBi: a distance prior in the logits

Why is a finite ALiBi penalty insufficient for causal attention?

One correct optionAALiBi makes all values V negativeBFuture keys must be masked, not merely assigned lower scoresCALiBi requires a learned absolute-position table

A) ALiBi makes all values V negative

B) Future keys must be masked, not merely assigned lower scores

C) ALiBi requires a learned absolute-position table

Answer and explanation

Correct answer: B. A finite score generally retains nonzero softmax weight. Future positions must be excluded by the mask.

Question 4.9.1

4.9. NoPE and hybrid position mechanisms

Which statement about NoPE is correct?

One correct optionANoPE always means strict alternation between RoPE and non-RoPE layersBPure NoPE models and hybrids with explicit position in only some layers both existCNoPE is used only in encoders that do not have a causal attention mask

A) NoPE always means strict alternation between RoPE and non-RoPE layers

B) Pure NoPE models and hybrids with explicit position in only some layers both exist

C) NoPE is used only in encoders that do not have a causal attention mask

Answer and explanation

Correct answer: B. NoPE means no explicit positional mechanism in the relevant layer or model. Interleaved RoPE/NoPE is one hybrid design, not the definition.

Question 4.10.1

4.10. Reading the gpt-oss positional configuration

What can be inferred reliably from initial_context_length=4096 in the gpt-oss config?

One correct optionAAll model pretraining used only sequences of exactly 4096 tokensBThe model necessarily underwent one particular YaRN fine-tuning phaseC4096 is the reference length of the published positional scaling contract

A) All model pretraining used only sequences of exactly 4096 tokens

B) The model necessarily underwent one particular YaRN fine-tuning phase

C) 4096 is the reference length of the published positional scaling contract

Answer and explanation

Correct answer: C. The config describes scaling geometry. A full training chronology requires separate documentation and cannot be reconstructed from one field.

Question 4.11.1

4.11. Level-one code: explicit shapes and explicit scaling placement

How should the teaching implementation handle NaN in the frequency vector?

One correct optionASilently replace NaN with zeroBContinue and hide NaN in softmaxCReject the invalid input explicitly

A) Silently replace NaN with zero

B) Continue and hide NaN in softmax

C) Reject the invalid input explicitly

Answer and explanation

Correct answer: C. Shape and positivity checks are insufficient: NaN≤0 is false. Finiteness must be checked separately.

Question 4.12.1

4.12. What the notebook validates

A cached prefix contains p tokens at positions 0…p−1. Which position belongs to the next query?

One correct optionApB0 because this call contains one tokenCAlways p−1

A) p

B) 0 because this call contains one token

C) Always p−1

Answer and explanation

Correct answer: A. The current block size does not reset absolute position. Cached keys have already been rotated at their positions.

Question 4.12.2

4.12. What the notebook validates

Why do integer position_ids not guarantee an exact phase?

One correct optionASine accepts only negative indicesBConversion to working floats and multiplication by frequencies introduce roundingCEvery integer ID requires a learned parameter

A) Sine accepts only negative indices

B) Conversion to working floats and multiplication by frequencies introduce rounding

C) Every integer ID requires a learned parameter

Answer and explanation

Correct answer: B. For example, FP16 merges 2048 and 2049. FP32 delays this limit but does not provide arbitrary precision.

Question 4.13.1

4.13. M-RoPE and time-aligned multimodal position

Which position IDs does M-RoPE use for ordinary text tokens?

One correct optionA(m,0,0): only the temporal rotary section receives a positionB(m,m,m): the text path is equivalent to ordinary one-dimensional RoPEC(0,h,w): text uses the same coordinates as a still image

A) (m,0,0): only the temporal rotary section receives a position

B) (m,m,m): the text path is equivalent to ordinary one-dimensional RoPE

C) (0,h,w): text uses the same coordinates as a still image

Answer and explanation

Correct answer: B. All three text axes receive the same index. Every rotary pair therefore sees position m regardless of its assigned section.

Question 4.13.2

4.13. M-RoPE and time-aligned multimodal position

Which statement about TMRoPE is supported by the published technical report?

One correct optionABy definition it forbids the use of spatial coordinatesBIt always assigns one temporal ID to exactly 40 ms in every modelCIt is documented in Qwen2.5-Omni; 40 ms belongs to that implementation

A) By definition it forbids the use of spatial coordinates

B) It always assigns one temporal ID to exactly 40 ms in every model

C) It is documented in Qwen2.5-Omni; 40 ms belongs to that implementation

Answer and explanation

Correct answer: C. TMRoPE was published in the March 2025 Qwen2.5-Omni report. Time alignment is the general idea; the duration represented by one ID is architecture-specific.

Question 4.13.3

4.13. M-RoPE and time-aligned multimodal position

What is interleaved by Qwen3-VL Interleaved-MRoPE?

One correct optionATraining and evaluation documentsBLayers with and without softmaxCTemporal and spatial axis assignments across frequency pairs

A) Training and evaluation documents

B) Layers with and without softmax

C) Temporal and spatial axis assignments across frequency pairs

Answer and explanation

Correct answer: C. This allocates axes across the spectrum. It is separate from adjacent or split-half pairing within each rotary pair.

Question 4.14.1

4.14. The 2026 landscape is a branching architecture map

What is the more accurate description of Llama 4 positional architecture?

One correct optionAA RoPE/NoPE hybrid with chunked attention and temperature tuningBOrdinary RoPE with mandatory YaRN scaling in every layerCPure ALiBi with a linear bias and no rotary component

A) A RoPE/NoPE hybrid with chunked attention and temperature tuning

B) Ordinary RoPE with mandatory YaRN scaling in every layer

C) Pure ALiBi with a linear bias and no rotary component

Answer and explanation

Correct answer: A. The published implementation includes NoPE layers, RoPE elsewhere, and chunked attention. Temperature tuning applies to the NoPE layers. One rope_theta field does not identify the entire design.