Modern LLMs10 / 20

Module 10. KV Caches and Efficient Inference

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 20 · Correct: 0

Question 10.4.1

10.4. Formalism and worked example A: how many bytes does one token add?

For Llama-3-8B, use 32 layers, 8 KV heads, head dimension 128, and bf16. What is the KV payload per token?

One correct optionA128 KiB per token — that is 2·32·8·128·2 bytes by the cache formulaB512 KiB per token, because all 32 query heads are counted inC16 KiB per token, since K and V share one common table

A) 128 KiB per token — that is 2·32·8·128·2 bytes by the cache formula

B) 512 KiB per token, because all 32 query heads are counted in

C) 16 KiB per token, since K and V share one common table

Answer and explanation

Correct answer: A. The two in the formula stands for K and V, and query heads do not enter it. The same count gives 2,560 KiB for a 70B MHA equivalent and 68.625 KiB for DeepSeek-V3 with MLA.

Question 10.4.2

10.4. Formalism and worked example A: how many bytes does one token add?

Why does MLA need its own memory formula instead of the standard GQA one?

One correct optionABecause MLA stores the cache at reduced precision while GQA uses bf16BIt caches a latent plus a separate RoPE part, not per-head K and VCBecause MLA reconstructs the keys anew on every decode step

A) Because MLA stores the cache at reduced precision while GQA uses bf16

B) It caches a latent plus a separate RoPE part, not per-head K and V

C) Because MLA reconstructs the keys anew on every decode step

Answer and explanation

Correct answer: B. Ordinary KV stores 2LHkv·dh scalars; MLA stores L(dc+dr). At bf16, the V3 example uses 61·(512+64)·2 = 70272 bytes = 68.625 KiB per token.

Question 10.5.1

10.5. One request, two computational regimes

Which profile is common for a long prefill and a small-batch decode?

One correct optionAPrefill needs more memory, decode needs more memory bandwidthBPrefill works with the cache while decode builds it from scratch each stepCPrefill often approaches the compute limit; decode often approaches the weight-plus-KV bandwidth limit

A) Prefill needs more memory, decode needs more memory bandwidth

B) Prefill works with the cache while decode builds it from scratch each step

C) Prefill often approaches the compute limit; decode often approaches the weight-plus-KV bandwidth limit

Answer and explanation

Correct answer: C. Prefill reuses weights across many prompt positions. Decode adds one token per request and reads accumulated state. These are common regimes, not guarantees: batch size, context, kernels, and hardware matter.

Question 10.6.1

10.6. Worked example B: the bandwidth ceiling and the KV term

Use 16 GB of weights, 128 KiB/token, 3.35 TB/s, and batch one. What bandwidth ceiling does the 8B teaching model give at 128K?

One correct optionA101 tok/s: the decode step reads 16 GiB of cache on top of the weightsB209 tok/s: context length does not affect the speed of reading weightsC196 tok/s: the cache adds only a few percent to what must be read

A) 101 tok/s: the decode step reads 16 GiB of cache on top of the weights

B) 209 tok/s: context length does not affect the speed of reading weights

C) 196 tok/s: the cache adds only a few percent to what must be read

Answer and explanation

Correct answer: A. BW/(W+NbKV) gives 100.965 tok/s under the one-read traffic assumption. Other work and traffic are excluded. The 1M row gives 22.778 tok/s but cannot fit the hypothetical 80-GiB capacity.

Question 10.6.2

10.6. Worked example B: the bandwidth ceiling and the KV term

In GQA, which head counts enter KV storage and the teaching QK/AV FLOP estimate?

One correct optionAStorage uses Hkv; the QK/AV estimate uses HqBBoth calculations always use HkvCBoth calculations always use Hq

A) Storage uses Hkv; the QK/AV estimate uses Hq

B) Both calculations always use Hkv

C) Both calculations always use Hq

Answer and explanation

Correct answer: A. KV is retained per KV head; each query head still performs attention. Including QK/AV changes the 8B/128K intensity from 0.48222 to 2.55334 FLOP/B.

Question 10.7.1

10.7. Batch economics: where weight amortization stops being the whole story

What does the batch knee B* = W / KV(N) mean?

One correct optionAThe largest batch fitting into 80 GiB of the given device's memoryBThe batch size where total KV traffic equals one read of the weightsCThe batch at which accelerator utilization reaches 100 percent

A) The largest batch fitting into 80 GiB of the given device's memory

B) The batch size where total KV traffic equals one read of the weights

C) The batch at which accelerator utilization reaches 100 percent

Answer and explanation

Correct answer: B. B* marks a change in the dominant traffic term, not a capacity limit or an optimal batch. More requests still raise aggregate throughput beyond it, with diminishing weight-amortization gains.

Question 10.8.1

10.8. PagedAttention: decouple logical length from physical placement

What problem does PagedAttention solve?

One correct optionAIt speeds up attention itself by splitting the logit matrix into blocksBIt compresses the cache by keeping pages at reduced precision on diskCFragmentation: the cache is allocated in pages, not one contiguous buffer

A) It speeds up attention itself by splitting the logit matrix into blocks

B) It compresses the cache by keeping pages at reduced precision on disk

C) Fragmentation: the cache is allocated in pages, not one contiguous buffer

Answer and explanation

Correct answer: C. Preallocating for the maximum length wastes memory on requests that never reach it. Pages allow allocation as the sequence grows and let requests share common prefixes.

Question 10.8.2

10.8. PagedAttention: decouple logical length from physical placement

Why is copy-on-write needed when a partially filled KV block is shared?

One correct optionABranches diverge from a shared prefix and begin writing into one blockBTo keep pages from being evicted to disk while another request reads themCTo synchronize writes between different groups of serving devices

A) Branches diverge from a shared prefix and begin writing into one block

B) To keep pages from being evicted to disk while another request reads them

C) To synchronize writes between different groups of serving devices

Answer and explanation

Correct answer: A. While the block is only read, no copy is needed. The moment one branch appends its token into the shared partial block, the contents must diverge — otherwise the branches corrupt each other's cache.

Question 10.8.3

10.8. PagedAttention: decouple logical length from physical placement

How many unused slots does an exactly 2048-token sequence have with 16-token pages?

One correct optionA0B7.5 in every individual sequenceC128

A) 0

B) 7.5 in every individual sequence

C) 128

Answer and explanation

Correct answer: A. 2048 is divisible by 16. The 7.5-slot expectation applies to uniform remainders across varying lengths, not this fixed length.

Question 10.9.1

10.9. Prefix caching: do not rebuild an identical beginning

What does prefix caching give?

One correct optionAIt shrinks the cache by dropping positions with low attention weightBIt avoids recomputing an exactly matching prefix under matching model and execution settingsCIt speeds up decode by predicting the next tokens of the shared prefix

A) It shrinks the cache by dropping positions with low attention weight

B) It avoids recomputing an exactly matching prefix under matching model and execution settings

C) It speeds up decode by predicting the next tokens of the shared prefix

Answer and explanation

Correct answer: B. Reuse depends on an identical token history and compatible state, not merely similar text. The uncached suffix still attends to the prefix; billing depends on the service contract.

Question 10.9.2

10.9. Prefix caching: do not rebuild an identical beginning

A 7680-token prefix is cached and a 512-token suffix arrives. Which work remains?

One correct optionASuffix projections and suffix attention to the prefix and earlier suffix positionsBSuffix projections only; attention is unnecessaryCFull prefill of all 8192 positions

A) Suffix projections and suffix attention to the prefix and earlier suffix positions

B) Suffix projections only; attention is unnecessary

C) Full prefill of all 8192 positions

Answer and explanation

Correct answer: A. Caching avoids rebuilding prefix state, not reading it for new attention queries. The teaching QK/AV-inclusive ratio is 14.402, versus 16 for projections alone.

Question 10.9.3

10.9. Prefix caching: do not rebuild an identical beginning

Identical token blocks follow different earlier prefixes. Are their KV states interchangeable?

One correct optionANo: state depends on preceding context and model settingsBYes: a hash of the block alone is enoughCYes: equal block length guarantees it

A) No: state depends on preceding context and model settings

B) Yes: a hash of the block alone is enough

C) Yes: equal block length guarantees it

Answer and explanation

Correct answer: A. Matching local tokens does not imply matching contextual state. The demo key includes the full prefix and model revision.

Question 10.10.1

10.10. Tiered placement: when HBM is only the first level

What is the main trade-off of tiered KV offloading?

One correct optionAA cache on the CPU takes more room because of different page alignmentBOffloaded blocks cannot be shared between several concurrent requestsCSpace is cheaper, but bringing blocks back runs over a slow bus

A) A cache on the CPU takes more room because of different page alignment

B) Offloaded blocks cannot be shared between several concurrent requests

C) Space is cheaper, but bringing blocks back runs over a slow bus

Answer and explanation

Correct answer: C. Compare the expected benefit of reuse with writing, storage, and restore costs. If transfer is more expensive than recomputing the required state, offloading does not pay.

Question 10.11.1

10.11. Prefill/decode disaggregation: give each phase its own pool

Why separate prefill and decode onto different device groups?

One correct optionAThe phases have different profiles; separate pools can protect decode from long prefillsBSo that the cache fits entirely into the memory of one of the groupsCSo that training and inference can share one and the same cluster

A) The phases have different profiles; separate pools can protect decode from long prefills

B) So that the cache fits entirely into the memory of one of the groups

C) So that training and inference can share one and the same cluster

Answer and explanation

Correct answer: A. Independent pools allow different batching and parallelism policies. The trade-off is KV transfer and coordination; separation is not always better at low load.

Question 10.12.1

10.12. Code (level 1): geometry, bandwidth, and private pages

How should a KV pool behave when blocks run out during append?

One correct optionASilently overwrite the oldest blocks of that same sequenceBRaise an explicit error without partially changing the poolCHalve the block size repeatedly until free space is found

A) Silently overwrite the oldest blocks of that same sequence

B) Raise an explicit error without partially changing the pool

C) Halve the block size repeatedly until free space is found

Answer and explanation

Correct answer: B. Preflight the allocation. If it fails, preserve page tables and reference counts so that the scheduler can decide how to proceed.

Question 10.12.2

10.12. Code (level 1): geometry, bandwidth, and private pages

A cache retains absolute positions [2,5). What should truncate(1) do?

One correct optionAReject rollback outside the retained intervalBRestore the evicted position 1CReset all absolute positions to zero

A) Reject rollback outside the retained interval

B) Restore the evicted position 1

C) Reset all absolute positions to zero

Answer and explanation

Correct answer: A. The earlier prefix is gone. Reject without mutation; recovering it requires a separate snapshot or recomputation.

Question 10.14.1

10.14. Compression I: eviction changes which history exists

How does eviction based on accumulated attention mass differ from a simple streaming policy?

One correct optionAIt offloads blocks to the CPU while streaming deletes them for goodBIt works during prefill, whereas streaming applies only at decodeCIt looks at accumulated attention weights rather than token position

A) It offloads blocks to the CPU while streaming deletes them for good

B) It works during prefill, whereas streaming applies only at decode

C) It looks at accumulated attention weights rather than token position

Answer and explanation

Correct answer: C. Streaming decides in advance and identically: keep the head and the tail of the window. Attention-aware ranks tokens by contribution but must accumulate statistics and pays its own price.

Question 10.15.1

10.15. Compression II: quantization, low rank, and the order of operations

Why does bf16→3 bits per coordinate not guarantee a 16/3 reduction in total memory?

One correct optionAGroup scales, zero points and page metadata occupy space as wellBBecause only K is quantized while V always stays in the original dtypeCBecause the first layers of the stack are always excluded from quantization

A) Group scales, zero points and page metadata occupy space as well

B) Because only K is quantized while V always stays in the original dtype

C) Because the first layers of the stack are always excluded from quantization

Answer and explanation

Correct answer: A. 16/3 compares ideal value codes only. Scales, zero points when used, alignment, high-precision residuals, and page metadata add bytes; model weights remain too.

Question 10.15.2

10.15. Compression II: quantization, low rank, and the order of operations

Why does the suggested evaluation order leave eviction until last?

One correct optionAEviction irreversibly removes part of the context and may change the model outputBEviction works only after the KV cache has been moved to local SSDCEviction requires every KV block to use the same bit width

A) Eviction irreversibly removes part of the context and may change the model output

B) Eviction works only after the KV cache has been moved to local SSD

C) Eviction requires every KV block to use the same bit width

Answer and explanation

Correct answer: A. Evicted positions are unavailable to future attention unless recomputed, potentially breaking a long-range dependency. Quantization and architectural compression also need quality checks: this is an evaluation order, not a quality guarantee.

Question 10.17.1

10.17. Reading a model’s inference passport

What must an inference passport contain besides the weight size?

One correct optionAThe model's release date and the licence its weights are published underBKV per token, target length, batch, engine and cache quantization modeCThe parameter count together with the layer count and vocabulary size

A) The model's release date and the licence its weights are published under

B) KV per token, target length, batch, engine and cache quantization mode

C) The parameter count together with the layer count and vocabulary size

Answer and explanation

Correct answer: B. Weights are a constant, the cache is a function of load. Without length, batch and cache scheme the weight size does not answer "will this fit on the card".