Modern LLMs4 / 20

Module 4. Positional representations: from sinusoids to multiple axes

Try each exercise first. Expand worked answers or project guidance when needed; their open state is shared between RU/EN.

Download course files (.zip) 2588.1 KB

The archive preserves the source folder structure, both languages, and supporting files.

E1. Common-shift invariance

Hold q, k, and the frequencies fixed. Prove that shifting both positions by $c$ leaves a RoPE dot product unchanged.

Show answer and solution

Solution.

(R_{m+c}q)^T(R_{n+c}k) =q^TR_{m+c}^TR_{n+c}k =q^TR_{n-m}k.

This uses $R_a^T=R_{-a}$ and the rotation group law $R_aR_b=R_{a+b}$ . Thus $R_{m+c}^TR_{n+c}=R_{-(m+c)}R_{n+c}=R_{n-m}$ . The shared offset cancels: with q and k fixed, position enters only through relative displacement $n-m$ .

E2. Spectral coverage

For $D=64$ , count the pairs completing at least one turn by position 4096 for bases 10,000, 150,000, and 500,000.

Show answer and solution

Solution. Rotary pair $i=0,\ldots,D/2-1$ has frequency $\theta_i=\text{base}^{-2i/D}$ . It completes a turn when $L\theta_i\ge2\pi$ , or

i\le \frac{D}{2}\frac{\ln(L/2\pi)}{\ln(\text{base})}.

For $D=64$ and $L=4096$ , the right-hand side is about 22.52, 17.40, and 15.80. Counting integer indices from zero gives 23, 18, and 16 pairs. This is phase coverage in the chosen window, not a hard count of distinguishable positions.

E3. YaRN scenario

With base 150,000, factor 32, $D=64$ , reference length 4096, $\beta_{fast}=32$ , and $\beta_{slow}=1$ , compute the ramp and scaling factors.

Show answer and solution

Solution. Substitution into the original gpt-oss ramp-boundary formulas gives low≈8.0928 and high≈17.3980. Hence pairs $i=0,\ldots,8$ retain their original frequencies (9 pairs), $i=9,\ldots,17$ are transitional (9), and $i=18,\ldots,31$ are fully interpolated (14). With truncate=True, the stored boundaries are 8/18, while the integer partition remains the same.

The attention factor is $a=1+0.1\ln32=1.346574$ . If both $q$ and $k$ receive it, bilinearity gives $(aq)^T(ak)=a^2q^Tk$ , so the score factor is $a^2=1.813260$ , not $a$ .

E4. Text M-RoPE

Show that coordinates $(m,m,m)$ produce the same output as standard 1D RoPE.

Show answer and solution

Solution. M-RoPE assigns rotary pairs to axes such as time, height, and width. A text token uses coordinates $(m,m,m)$ , so every pair with frequency $\theta_i$ receives angle $m\theta_i$ , regardless of its axis assignment. Concatenating all pairs therefore reproduces standard 1D RoPE exactly. A difference appears only when a multimodal token has unequal axis coordinates.

E5 [project]

Select an open model with a published configuration. Reconstruct its frequency spectrum and write three separate lists: what follows from the config, what follows from implementation code, and what requires a technical report.

Project demos and guidance

Demo for this project exercise: English project-demo notebook for Module 4, sections “Demo 1” and “Demo 2” as starting points for this module’s projects. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.