Module 8. Beyond the Transformer: SSMs, Mamba, and Hybrid Sequence Models — quizzes and answer checking · Modern LLMs

Question 8.4.1

8.4. Discretization with Zero-Order Hold

What does Zero-Order Hold compute for a linear SSM?

One correct optionAThe exact discrete transition for an input held constant over the intervalBAn approximate distribution over coordinates of the previous stateCOnly the eigenvalues associated with the observation matrix

A) The exact discrete transition for an input held constant over the interval

B) An approximate distribution over coordinates of the previous state

C) Only the eigenvalues associated with the observation matrix

Answer and explanation

Correct answer: A. ZOH gives $\bar A=e^{\Delta A}$ and the exact held-input coefficient $\bar B$ . A neural implementation may still choose another effective input update.

Question 8.4.2

8.4. Discretization with Zero-Order Hold

Why is |a|<10⁻¹⁴ insufficient to replace the ZOH input factor by Δ?

One correct optionAThe product Δa matters: tiny a with large Δ need not be a small step.BBecause expm1 only accepts positive a.CBecause zero eigenvalues are forbidden.

A) The product Δa matters: tiny a with large Δ need not be a small step.

B) Because expm1 only accepts positive a.

C) Because zero eigenvalues are forbidden.

Answer and explanation

Correct answer: A. For a=−10⁻¹⁵ and Δ=10¹⁵ the product is −1. Use the exact zero-mode limit at a=0.

Question 8.5.1

8.5. Worked example: the cost of replacing exact ZOH by $\Delta B$

At which $\Delta$ does the $\Delta B$ approximation have ten-percent relative error for $a=-1$ ?

One correct optionAAbout 0.10BAbout 0.19375CAbout 0.50

A) About 0.10

B) About 0.19375

C) About 0.50

Answer and explanation

Correct answer: B. The root of $\bar A=e^{\Delta A}$ is approximately 0.193747558.

Question 8.6.1

8.6. What selectivity changes in Mamba

What does selectivity mean in Mamba?

One correct optionAThe model selects one of several full-attention mechanismsBThe recurrent-state size grows with token importanceCWriting, reading, and step parameters depend on the current input

A) The model selects one of several full-attention mechanisms

B) The recurrent-state size grows with token importance

C) Writing, reading, and step parameters depend on the current input

Answer and explanation

Correct answer: C. Input-dependent $\bar A=e^{\Delta A}$ , $C_t$ , and $\Delta_t$ let different tokens be written, exposed, and forgotten differently.

Question 8.7.1

8.7. Mamba-2 and the scope of State Space Duality

What is the correct scope of State Space Duality?

One correct optionAA structured SSM class corresponds to a semiseparable masked matrix formBEvery continuous SSM is bitwise identical to ordinary softmax attentionCEvery recurrent network is replaced by a single dense-attention layer

A) A structured SSM class corresponds to a semiseparable masked matrix form

B) Every continuous SSM is bitwise identical to ordinary softmax attention

C) Every recurrent network is replaced by a single dense-attention layer

Answer and explanation

Correct answer: A. SSD connects a defined structured SSM class to multiple execution forms. It is not an identity between all SSMs and softmax attention.

Question 8.7.2

8.7. Mamba-2 and the scope of State Space Duality

What does associativity of state-transform composition permit?

One correct optionAArbitrary token reordering.BRegrouping operations while preserving step order.CDiscarding the state at a chunk boundary.

A) Arbitrary token reordering.

B) Regrouping operations while preserving step order.

C) Discarding the state at a chunk boundary.

Answer and explanation

Correct answer: B. Associativity is not commutativity. The boundary state carries contributions from earlier chunks.

Question 8.8.1

8.8. Delta-rule memory and adaptive forgetting

What does the delta rule do for a normalized key at $\beta=1$ ?

One correct optionAIt sums the new and old values without removing the old associationBIt exactly replaces the readout along the selected key directionCIt clears the entire state matrix before inserting the new association

A) It sums the new and old values without removing the old association

B) It exactly replaces the readout along the selected key direction

C) It clears the entire state matrix before inserting the new association

Answer and explanation

Correct answer: B. The update uses the error $\bar A=e^{\Delta A}$ , so retrieval along the normalized key becomes exactly $v$ .

Question 8.9.1

8.9. What a fixed state can and cannot preserve

What does rising MSE in the toy additive linear memory show?

One correct optionAEvery Mamba necessarily loses all records after thirty-two tokensBEvery softmax-attention system always produces exactly zero errorCA fixed matrix memory can accumulate interference among associations

A) Every Mamba necessarily loses all records after thirty-two tokens

B) Every softmax-attention system always produces exactly zero error

C) A fixed matrix memory can accumulate interference among associations

Answer and explanation

Correct answer: C. This is a result for the declared synthetic associative memory, not a Mamba benchmark or a theorem about every SSM.

Question 8.9.2

8.9. What a fixed state can and cannot preserve

What does a logit multiplier of 20 do in the softmax control?

One correct optionASets temperature to 20 and flattens the weights.BGuarantees exact recovery of arbitrary values.CSets temperature to 0.05 and sharpens the weights.

A) Sets temperature to 20 and flattens the weights.

B) Guarantees exact recovery of arbitrary values.

C) Sets temperature to 0.05 and sharpens the weights.

Answer and explanation

Correct answer: C. Multiplying logits by 20 is dividing by temperature 0.05. Finite-temperature error is generally nonzero.

Question 8.10.1

8.10. Memory accounting and core-only arithmetic

Which states belong in a Mamba-style serving cache?

One correct optionAThe main SSM state and the causal-convolution tailBOnly the last hidden state of the final layerCThe complete list of input tokens and their logits

A) The main SSM state and the causal-convolution tail

B) Only the last hidden state of the final layer

C) The complete list of input tokens and their logits

Answer and explanation

Correct answer: A. Autoregressive causal convolution also needs its recent tail; counting only ssm_state is incomplete.

Question 8.10.2

8.10. Memory accounting and core-only arithmetic

A complete Mamba-1-like cache is 10 MiB and KV costs 128 KiB/token. Where is crossover?

One correct optionAAt 64 tokensBAt 80 tokensCAt 512 tokens

A) At 64 tokens

B) At 80 tokens

C) At 512 tokens

Answer and explanation

Correct answer: B. $\bar A=e^{\Delta A}$ . The 10 MiB includes 8 MiB of SSM state and 2 MiB of convolution state.

Question 8.10.3

8.10. Memory accounting and core-only arithmetic

A Mamba-2-like cache is 66.0625 MiB and KV costs 128 KiB/token. Which is the first integer token beyond crossover?

One correct optionAToken 512BToken 528CToken 529

A) Token 512

B) Token 528

C) Token 529

Answer and explanation

Correct answer: C. The exact ratio is 528.5, so token 529 is the first integer position at which the KV cache is larger.

Question 8.10.4

8.10. Memory accounting and core-only arithmetic

With 10 MiB of fixed state and 128 KiB of KV per token, when is KV strictly larger?

One correct optionAStarting at token 81.BStarting at token 80.CStarting at token 79.

A) Starting at token 81.

B) Starting at token 80.

C) Starting at token 79.

Answer and explanation

Correct answer: A. The sizes are equal at 80; strict inequality begins at 81.

Question 8.11.1

8.11. Three hybrid patterns

Why does a hybrid with occasional full-attention layers retaining all K/V use O(N) cache memory?

One correct optionAThose full-attention layers retain K/V for every tokenBThe recurrent state doubles at every attention layerCThe MoE router stores a separate log for every position

A) Those full-attention layers retain K/V for every token

B) The recurrent state doubles at every attention layer

C) The MoE router stores a separate log for every position

Answer and explanation

Correct answer: A. Fewer such layers reduce the coefficient while retaining linear growth with length. If every addressable branch uses a fixed local window, this unbounded-cache conclusion no longer follows.

Question 8.11.2

8.11. Three hybrid patterns

Which layer pattern is published for Qwen3.5?

One correct optionASeven Mamba layers followed by one full-attention layerBThree Gated DeltaNet layers followed by one full-attention layerCParallel Mamba-2 and attention branches inside every block

A) Seven Mamba layers followed by one full-attention layer

B) Three Gated DeltaNet layers followed by one full-attention layer

C) Parallel Mamba-2 and attention branches inside every block

Answer and explanation

Correct answer: B. The official contract uses a repeated 3:1 layout: three Gated DeltaNet → MoE layers and one Gated Attention → MoE layer.

Question 8.12.1

8.12. Level-1 code: exact discretization, delta update, and scan composition

Why can an affine scan be parallelized?

One correct optionAAll steps are computed independently of every preceding stateBEvery transition matrix is required to be strictly orthogonalCAffine-transition composition forms an associative operator

A) All steps are computed independently of every preceding state

B) Every transition matrix is required to be strictly orthogonal

C) Affine-transition composition forms an associative operator

Answer and explanation

Correct answer: C. Associativity permits tree-structured prefix composition and correct chunk stitching.

Question 8.12.2

8.12. Level-1 code: exact discretization, delta update, and scan composition

Which test is especially important for a chunked scan?

One correct optionAA length not divisible by the chunk sizeBOnly a length exactly equal to one chunkCOnly an all-zero sequence

A) A length not divisible by the chunk size

B) Only a length exactly equal to one chunk

C) Only an all-zero sequence

Answer and explanation

Correct answer: A. A partial final chunk exposes state-carry and off-by-one bugs hidden at perfectly divisible lengths.

Question 8.15.1

8.15. A current architecture passport

Why is the label linear attention insufficient for a serving passport?

One correct optionAIt omits license conditions and permitted modes of redistributionBIt omits layer ratios, state shapes, and the remaining KV cacheCIt applies only to encoders and cannot describe autoregressive decode

A) It omits license conditions and permitted modes of redistribution

B) It omits layer ratios, state shapes, and the remaining KV cache

C) It applies only to encoders and cannot describe autoregressive decode

Answer and explanation

Correct answer: B. Models with the same label may use a pure recurrent stack, a 3:1 hybrid, or a sparse/full branch with very different memory.

Question 8.16.1

8.16. Audio, video, and other long streams

Why is a recurrent carrier attractive for audio and video?

One correct optionAThose modalities never require exact access to the pastBTheir inputs are always statistically uniformCLong temporal streams can be updated with fixed-size state

A) Those modalities never require exact access to the past

B) Their inputs are always statistically uniform

C) Long temporal streams can be updated with fixed-size state

Answer and explanation

Correct answer: C. The advantage comes from the number of sequential time steps. Exact return to an event may still require attention or external memory.