Modern LLMs8 / 20

Module 8. Beyond the Transformer: SSMs, Mamba, and Hybrid Sequence Models

Try each exercise first. Expand worked answers or project guidance when needed; their open state is shared between RU/EN.

Download course files (.zip) 2588.1 KB

The archive preserves the source folder structure, both languages, and supporting files.

E1. Approximation threshold

For $a=-1$ , solve for the step at which replacing the exact ZOH input coefficient with $\Delta$ creates ten-percent relative error.

Show answer and solution

Solution.

For $\dot h=ah+Bx$ , exact ZOH discretization gives input coefficient $(e^{a\Delta}-1)B/a$ . With $a=-1$ and $B=1$ , this is $1-e^{-\Delta}$ , whereas the small-step approximation uses $\Delta$ . Hence relative error against the exact coefficient is

\frac{\Delta}{1-e^{-\Delta}}-1=0.1 \quad\Rightarrow\quad \Delta\approx0.193747558.

Substitution gives $\Delta/(1-e^{-\Delta})\approx1.1$ . The small- $\Delta$ expansion starts with error $\Delta/2$ , so a root near 0.2 is also a useful magnitude check.

E2. Complete recurrent cache

For 32 layers, $d_{inner}=8192$ , $d_{state}=16$ , $d_{conv}=4$ , and bf16, compute the two state components and the crossover against 128 KiB/token.

Show answer and solution

Solution. One layer's SSM state has $d_{inner}d_{state}$ bf16 elements:

$8192\cdot16\cdot2=262144\ \text{bytes}=0.25\ \text{MiB}.$

Across 32 layers this is 8 MiB. Convolution state uses $d_{inner}d_{conv}$ elements: $8192\cdot4\cdot2=65536$ bytes per layer, or 2 MiB total. Complete recurrent cache is 10 MiB and does not grow with sequence length. At 128 KiB/token, KV reaches it at $10\text{ MiB}/128\text{ KiB}=\mathbf{80}$ tokens and exceeds it afterward.

E3. Mamba-2-like cache

Set $d_{state}=128$ and convolution channels to $8192+2\cdot128$ . Find the complete cache and the first integer token beyond crossover.

Show answer and solution

Solution. The SSM component is

$32\cdot8192\cdot128\cdot2=64\ \text{MiB}.$

Convolution has $8192+2\cdot128=8448$ channels, so its state uses $32\cdot8448\cdot4\cdot2=2.0625$ MiB. Total is 66.0625 MiB. Dividing by 0.125 MiB of KV per token gives 528.5: KV is still smaller at 528 tokens and first becomes strictly larger at 529.

E4 [project]

Compare sequential and chunked scans at a sequence length not divisible by the chunk size. The partial final chunk must agree with the reference recurrence.

Project demos and guidance

Demo for this project exercise: English project-demo notebook for Module 8, section “Demo 1”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

E5 [project]

Compare additive, delta, and gated-delta memory on a synthetic task. Use identical keys, values, and state dimensions to compare interference, rewriting, and forgetting. Report current-key error separately from error over all previously written keys.

Project demos and guidance

Demo for this project exercise: English project-demo notebook for Module 8, section “Demo 2”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.