Module 16. Multimodal LLMs: Vision, Audio, and Video — quizzes and answer checking · Modern LLMs

Question 16.2.1

16.2. Four families, four places to draw the boundary

Why is multimodal progress better described as coexisting families than as successive eras?

One correct optionADifferent families optimize different constraints and remain usefulBEvery new family fully preserves the architecture of the previous generationCHistorical labels are unrelated to training and inference differences

A) Different families optimize different constraints and remain useful

B) Every new family fully preserves the architecture of the previous generation

C) Historical labels are unrelated to training and inference differences

Answer and explanation

Correct answer: A. CLIP-like encoders serve retrieval, connectors retrofit existing LLMs, and unified/omni systems support generation or streaming output. A newer topology does not obsolete every earlier one.

Question 16.4.1

16.4. The first design: encoder, connector, language decoder

When do visual features create ordinary language-decoder KV state?

One correct optionAWith any ViT encoder, even when it connects through cross-attentionBWhen visual positions enter the decoder self-attention streamCOnly when the language decoder also generates images

A) With any ViT encoder, even when it connects through cross-attention

B) When visual positions enter the decoder self-attention stream

C) Only when the language decoder also generates images

Answer and explanation

Correct answer: B. The module’s KV accounting applies to visual positions inside ordinary decoder self-attention. Cross-attention or external visual memory needs different accounting.

Question 16.5.1

16.5. CLIP, SigLIP, and worked example C: reading InfoNCE numerically

What is the InfoNCE loss in the teaching example at temperature τ=0.1?

One correct optionA0.643738 — the value at τ=0.5B1.386294 — the uniform ln 4 reference valueC0.007409 — the result of sharper softmax competition

A) 0.643738 — the value at τ=0.5

B) 1.386294 — the uniform ln 4 reference value

C) 0.007409 — the result of sharper softmax competition

Answer and explanation

Correct answer: C. With the same similarities, lower temperature spreads the logits. Loss falls from 0.643738 at τ=0.5 to 0.007409 at τ=0.1; ln 4 is the uniform four-way value.

Question 16.6.1

16.6. Worked example A: three ways to spend visual tokens

Four local tiles and one thumbnail each contribute 576 tokens. How many visual tokens result?

One correct optionA2,880 = 5·576B2,304 = 4·576 without the thumbnailC5,184 = the raw grid at 1008×1008

A) 2,880 = 5·576

B) 2,304 = 4·576 without the thumbnail

C) 5,184 = the raw grid at 1008×1008

Answer and explanation

Correct answer: A. The teaching NeXT scenario uses five views: four local 336×336 tiles and one global thumbnail. Thus 2,880 describes a tiling policy, not one 672×672 grid.

Question 16.7.1

16.7. Worked example B: when images create ordinary decoder KV

How much KV memory do 1,296 visual positions require at 128 KiB per position?

One correct optionA1,296 MiB — one MiB per positionB162 MiB — the result of 1,296·128 KiBC72 MiB — the cost of 576 positions without merging

A) 1,296 MiB — one MiB per position

B) 162 MiB — the result of 1,296·128 KiB

C) 72 MiB — the cost of 576 positions without merging

Answer and explanation

Correct answer: B. 1,296·128 KiB = 162 MiB. A resampler does not change that value if position count and KV format stay fixed. Reducing positions changes total KV; cross-attention requires accounting for a different storage path.

Question 16.8.1

16.8. Connectors control where computation happens

Which resource does a Q-Former or resampler primarily reduce?

One correct optionAThe parameter count of every vision-encoder layerBThe bandwidth of a separate audio decoderCThe number of visual positions passed to the language model

A) The parameter count of every vision-encoder layer

B) The bandwidth of a separate audio decoder

C) The number of visual positions passed to the language model

Answer and explanation

Correct answer: C. Learned queries extract a bounded set of representations from a larger feature grid. This creates a bottleneck while reducing LLM context length.

Question 16.8.2

16.8. Connectors control where computation happens

Why does freezing an LLM not remove all backward work when training its projector?

One correct optionAThe projector still needs gradients through input activationsBEvery LLM weight is still updatedCFreezing doubles visual sequence length

A) The projector still needs gradients through input activations

B) Every LLM weight is still updated

C) Freezing doubles visual sequence length

Answer and explanation

Correct answer: A. Frozen weights need no weight gradients or optimizer state, but the loss must remain differentiable with respect to the projector output.

Question 16.9.1

16.9. Dynamic resolution makes compute depend on image geometry

How does M-RoPE relate to visual-token count?

One correct optionAIt encodes coordinates; resize, tiling, and merging change lengthBIt always merges four neighboring patches into one tokenCIt replaces dynamic resolution with one fixed visual grid

A) It encodes coordinates; resize, tiling, and merging change length

B) It always merges four neighboring patches into one token

C) It replaces dynamic resolution with one fixed visual grid

Answer and explanation

Correct answer: A. Position encoding preserves geometry. Resizing, tiling, spatial merging, temporal sampling, and pruning determine token count.

Question 16.9.2

16.9. Dynamic resolution makes compute depend on image geometry

What controls the geometric image budget in Qwen2-VL?

One correct optionAAlways the semantic difficulty of the imageBDimensions, aspect ratio and pixel limitsCOnly the user's task name

A) Always the semantic difficulty of the image

B) Dimensions, aspect ratio and pixel limits

C) Only the user's task name

Answer and explanation

Correct answer: B. Geometry-dependent dynamic resolution is not a learned semantic difficulty classifier.

Question 16.10.1

16.10. Unified output reveals a conflict between semantics and fidelity

How can a unified model remain internally modular?

One correct optionAOnly if the modalities are never trained jointlyBShared context can coexist with specialized internal modulesCOnly if the system returns text and no other output modality

A) Only if the modalities are never trained jointly

B) Shared context can coexist with specialized internal modules

C) Only if the system returns text and no other output modality

Answer and explanation

Correct answer: B. Janus separates visual pathways, InternVL-U uses an MMDiT head, and Thinker–Talker has a speech decoder. A unified product does not require one internal representation.

Question 16.11.1

16.11. Worked example E: video is an information-selection problem

Why does a large context window not solve long-video understanding by itself?

One correct optionAVideo frames cannot be represented as discrete visual tokensBA language-decoder KV cache cannot encode temporal coordinatesCFrames and events must still be selected within the budget

A) Video frames cannot be represented as discrete visual tokens

B) A language-decoder KV cache cannot encode temporal coordinates

C) Frames and events must still be selected within the budget

Answer and explanation

Correct answer: C. Two hours at 1 fps and 576 tokens per frame produce 4.15 million tokens. Sampling, temporal pooling, and event selection determine retained evidence.

Question 16.12.1

16.12. Thinker–Talker: coordinating two clocks

How does TMRoPE differ from ARIA in Thinker–Talker?

One correct optionATMRoPE aligns input time; ARIA coordinates output stream ratesBTMRoPE generates speech; ARIA recognizes individual imagesCTMRoPE exists only in Talker; ARIA only in the vision encoder

A) TMRoPE aligns input time; ARIA coordinates output stream rates

B) TMRoPE generates speech; ARIA recognizes individual images

C) TMRoPE exists only in Talker; ARIA only in the vision encoder

Answer and explanation

Correct answer: A. They operate at different levels: positional geometry of audio/video input events versus adaptive interleaving of output streams.

Question 16.13.1

16.13. Code level 1: geometry before frameworks

Why does patch_layout reject silent integer truncation?

One correct optionATo make every image automatically square before tokenizationBTo avoid losing a border without explicit resize or paddingCTo force spatial merging to use one universal factor

A) To make every image automatically square before tokenization

B) To avoid losing a border without explicit resize or padding

C) To force spatial merging to use one universal factor

Answer and explanation

Correct answer: B. Integer division could hide an incomplete patch. A processor may explicitly resize or pad; accounting for padded regions follows its actual interface and does not always require a separate pixel mask.

Question 16.13.2

16.13. Code level 1: geometry before frameworks

What does the unpatchify(patchify(image)) round trip verify?

One correct optionASimilarity between image and caption embeddingsBNorm preservation through the connector MLPCMutual consistency of the two operations on the tested input

A) Similarity between image and caption embeddings

B) Norm preservation through the connector MLP

C) Mutual consistency of the two operations on the tested input

Answer and explanation

Correct answer: C. Matching inverse permutations can pass even if their patch order violates the specification. A separate coordinate fixture is needed to check row-major order; the notebook includes one.

Question 16.15.1

16.15. Audio is a continuous evidence stream

Why does audio need its own systems passport beyond text-token count?

One correct optionACost depends on audio rate, compression, and streaming latencyBAudio is always processed without Transformer componentsCSpeech contains no semantics until explicit transcription

A) Cost depends on audio rate, compression, and streaming latency

B) Audio is always processed without Transformer components

C) Speech contains no semantics until explicit transcription

Answer and explanation

Correct answer: A. Audio has its own frame or codec-token rate. First-audio latency, real-time factor, interruptions, and semantic quality require separate measurement.

Question 16.16.1

16.16. Reading the public landscape by design axis

How should the internal architecture of a proprietary multimodal API be described?

One correct optionAInfer the hidden stack from one endpoint’s accepted formatsBRecord I/O and avoid guessing hidden topology without evidenceCTreat every voice API as one end-to-end Transformer

A) Infer the hidden stack from one endpoint’s accepted formats

B) Record I/O and avoid guessing hidden topology without evidence

C) Treat every voice API as one end-to-end Transformer

Answer and explanation

Correct answer: B. One interface may hide a cascade or several specialized models. Architecture claims require technical documentation, code, or configuration.

Question 16.16.2

16.16. Reading the public landscape by design axis

Which variant uses ViR in the InternVL3.5 report?

One correct optionAInternVL3.5-FlashBEvery VLM regardless of encoderCOnly Chameleon

A) InternVL3.5-Flash

B) Every VLM regardless of encoder

C) Only Chameleon

Answer and explanation

Correct answer: A. The Flash router selects 256 or 64 visual positions per tile. This property must not be assigned to every model in the family.

Question 16.17.1

16.17. Follow one request through a multimodal passport

What is the minimum multimodal passport needed to price a request?

One correct optionAModel name and its best published benchmark resultBOnly maximum resolution and maximum context lengthCProcessor, fusion, positions, state, latency, and memory

A) Model name and its best published benchmark result

B) Only maximum resolution and maximum context length

C) Processor, fusion, positions, state, latency, and memory

Answer and explanation

Correct answer: C. Cost appears at preprocessing, encoder, fusion, prefill, KV, and output-decoder stages. A capability label does not specify them.

Question 16.14.1

16.14. Level 2: what the notebook makes reproducible

What does the tiny VLM's held-out split test?

One correct optionATransfer to natural photographsBPreviously unseen height pairsCDifferent brightness for familiar height pairs

A) Transfer to natural photographs

B) Previously unseen height pairs

C) Different brightness for familiar height pairs

Answer and explanation

Correct answer: C. The 32 training and 16 held-out renders share the same 16 height pairs. Rendering changes; compositions do not.