Module 9. Pretraining: Data, Scaling Laws, and Optimizers — quizzes and answer checking · Modern LLMs

Question 9.4.1

9.4. Formalism: compute, a fitted loss surface, and its analytic optimum

What does a compute-optimal scaling law optimize under the constraint C = 6ND?

One correct optionAThe split of a fixed budget between model size and data volumeBThe number of training steps at a pre-selected model sizeCThe convergence rate of the optimizer under a fixed schedule

A) The split of a fixed budget between model size and data volume

B) The number of training steps at a pre-selected model size

C) The convergence rate of the optimizer under a fixed schedule

Answer and explanation

Correct answer: A. The law answers "how to divide the compute", not "how much is needed". The budget is given from outside, and the optimum splits it between N and D so as to minimize loss.

Question 9.4.2

9.4. Formalism: compute, a fitted loss surface, and its analytic optimum

For a budget of $7.23\cdot10^{23}$ FLOP, the analytic optimum of the fitted law gives $N=35.7$ B and $D=3.38$ T. What is $D/N$ ?

One correct optionAExactly 20 tokens per parameterB94.7 — close to one hundred tokens per parameterC1868 — the ratio in the Llama 3 8B/15T scenario

A) Exactly 20 tokens per parameter

B) 94.7 — close to one hundred tokens per parameter

C) 1868 — the ratio in the Llama 3 8B/15T scenario

Answer and explanation

Correct answer: B. The fitted exponents on $N$ and $D$ differ, so the optimal ratio changes with budget. The value 94.7 belongs to this calculation; it is not a universal Chinchilla constant.

Question 9.5.1

9.5. From raw web pages to a training distribution

What are effective epochs for a domain inside a data mixture?

One correct optionAThe number of steps after which the domain loss stops decreasingBThe domain's share of the mixture multiplied by the total epoch countCThe domain token presentations divided by its unique-token pool: average reuse

A) The number of steps after which the domain loss stops decreasing

B) The domain's share of the mixture multiplied by the total epoch count

C) The domain token presentations divided by its unique-token pool: average reuse

Answer and explanation

Correct answer: C. Giving a small domain a substantial mixture weight causes repeated use of its documents. Effective epochs quantify that repetition; memorization risk increases gradually rather than appearing at one universal epoch threshold.

Question 9.5.2

9.5. From raw web pages to a training distribution

DCLM starts from a standardized candidate pool of roughly 240T tokens, while DCLM-Baseline trains on 2.6T tokens. Why should those numbers be reported separately?

One correct optionAThe candidate pool and the selected training volume describe different stages of the pipelineBThe 2.6T figure is only the validation set and is unrelated to trainingCThe 240T figure counts bytes whereas 2.6T counts tokens after a unit conversion

A) The candidate pool and the selected training volume describe different stages of the pipeline

B) The 2.6T figure is only the validation set and is unrelated to training

C) The 240T figure counts bytes whereas 2.6T counts tokens after a unit conversion

Answer and explanation

Correct answer: A. The candidate pool defines the space in which selection strategies are compared. A run’s training volume records the selected token presentations that the model actually consumes.

Question 9.6.1

9.6. Worked example A: two budgets and the cost of a mnemonic

Why can the "20 tokens per parameter" rule give the wrong optimum?

One correct optionAIt was derived for encoder models and does not carry to decoder-onlyBThe D/N ratio itself depends on budget, while the rule treats it as constantCIt ignores vocabulary size, which is part of the model's parameter count

A) It was derived for encoder models and does not carry to decoder-only

B) The D/N ratio itself depends on budget, while the rule treats it as constant

C) It ignores vocabulary size, which is part of the model's parameter count

Answer and explanation

Correct answer: B. When α ≠ β the optimal N and D grow with budget at different exponents, so their ratio is not constant. At Llama-3's budget the rule of thumb gives 77.6B against an exact 35.7B.

Question 9.7.1

9.7. The isoFLOP valley and the fitted cost of long training

What does an isoFLOP curve show?

One correct optionAHow loss falls over time at a fixed model sizeBHow the required budget grows as the quality target risesCLoss as a function of N at constant budget, with a minimum at the optimum

A) How loss falls over time at a fixed model size

B) How the required budget grows as the quality target rises

C) Loss as a function of N at constant budget, with a minimum at the optimum

Answer and explanation

Correct answer: C. The fitted valley is relatively shallow on the data-heavy side: training a smaller model longer may increase fitted loss only modestly. That geometry makes token-heavier allocations worth considering.

Question 9.7.2

9.7. The isoFLOP valley and the fitted cost of long training

Llama-3 8B: 8.03B parameters and 15T tokens. How far from its budget's optimum is that?

One correct optionAD/N is about 1868 rather than 94.7, nearly 20 times larger in this comparisonBThe model sits exactly at the optimum: 15T is what the law prescribesCThe model is undertrained: the law would call for around 30T tokens

A) D/N is about 1868 rather than 94.7, nearly 20 times larger in this comparison

B) The model sits exactly at the optimum: 15T is what the law prescribes

C) The model is undertrained: the law would call for around 30T tokens

Answer and explanation

Correct answer: A. At $N$ , the analytic optimum of the fitted law is 35.7B parameters and 3.38T tokens. The 8.03B/15T scenario incurs $\Delta L=0.0258$ inside that law but uses a model about 4.4 times smaller, reducing dense inference cost.

Question 9.8.1

9.8. Inference-aware allocation: when does the smaller model repay its training?

Why can an inference-aware objective prefer a smaller model trained on more data?

One correct optionAA larger model is always cheaper to train and therefore prepays its serving costBTraining cost is paid once, while the cost of every generated token is paid repeatedlyCA smaller model always requires a more expensive accelerator class

A) A larger model is always cheaper to train and therefore prepays its serving cost

B) Training cost is paid once, while the cost of every generated token is paid repeatedly

C) A smaller model always requires a more expensive accelerator class

Answer and explanation

Correct answer: B. In the teaching model, dense decode costs roughly $N$ FLOP per token. Extra compute spent training the smaller model can be recovered over a large serving volume; the break-even in this scenario is about $2.12\cdot10^{12}$ generated tokens.

Question 9.9.1

9.9. The data wall: token counts stop being interchangeable

What does the "data wall" mean?

One correct optionAA limit on the context length available during model trainingBA ceiling on corpus read speed from disk on large clustersCThe pool of quality text grows slower than training runs consume it

A) A limit on the context length available during model training

B) A ceiling on corpus read speed from disk on large clusters

C) The pool of quality text grows slower than training runs consume it

Answer and explanation

Correct answer: C. Compute capacity can grow faster than the supply of high-quality, sufficiently independent data. Reuse, synthetic transformations, and better curation address different parts of that constraint.

Question 9.10.1

9.10. Muon: change the spectrum of a matrix update

What is Muon's core idea for matrix parameters?

One correct optionANarrow the update’s singular-value spread with a short matrix iterationBScale the step coordinate-wise by the accumulated second momentCZero out update components whose magnitude falls below a threshold

A) Narrow the update’s singular-value spread with a short matrix iteration

B) Scale the step coordinate-wise by the accumulated second moment

C) Zero out update components whose magnitude falls below a threshold

Answer and explanation

Correct answer: A. Coordinate-wise scaling is AdamW. Muon instead flattens the update's spectrum with a Newton–Schulz iteration, and it carries no second moment — hence the saving on optimizer state. A finite quintic trace does not set every nonzero singular value exactly to one.

Question 9.10.2

9.10. Muon: change the spectrum of a matrix update

Which Gram matrix is cheaper for a 64×32 update?

One correct optionA64×64 without transposingB32×32 after transposingC64×32 without multiplication

A) 64×64 without transposing

B) 32×32 after transposing

C) 64×32 without multiplication

Answer and explanation

Correct answer: B. Transposing makes the left Gram matrix square on the smaller dimension.

Question 9.11.1

9.11. MuonClip: correct the parameter source of unstable attention logits

What does QK-Clip do when it detects excessively large attention logits?

One correct optionAClips the logits from above while leaving the projection weights aloneBRescales the q and k projection weights, bringing logits back under thresholdCSkips the optimizer step for the layer where the outlier occurred

A) Clips the logits from above while leaving the projection weights alone

B) Rescales the q and k projection weights, bringing logits back under threshold

C) Skips the optimizer step for the layer where the outlier occurred

Answer and explanation

Correct answer: B. QK-Clip acts on the parameter source of excessive bilinear logits by rescaling $N$ and $W_K$ , rather than clipping one forward pass. The Kimi K2 report presents it as one component of the stabilization system for a 15.5T-token run; the reported absence of loss spikes belongs to the complete run, not to an isolated causal test.

Question 9.11.2

9.11. MuonClip: correct the parameter source of unstable attention logits

Where should post-correction logit regrowth start?

One correct optionAThe original, uncorrected maximaBThe corrected maximaCAlways zero

A) The original, uncorrected maxima

B) The corrected maxima

C) Always zero

Answer and explanation

Correct answer: B. Starting at the old maxima would simulate a branch where the correction never happened.

Question 9.12.1

9.12. Code (level 1): small functions that expose their assumptions

Why does this Newton–Schulz implementation transpose a tall matrix (more rows than columns)?

One correct optionAEvery tall matrix necessarily diverges without transpositionBTransposition replaces matrix multiplication by element-wise operationsCTo form the smaller left Gram matrix, then restore the original orientation

A) Every tall matrix necessarily diverges without transposition

B) Transposition replaces matrix multiplication by element-wise operations

C) To form the smaller left Gram matrix, then restore the original orientation

Answer and explanation

Correct answer: C. For an m×n input with m>n, transposition gives n×m X and an n×n left Gram matrix rather than m×m. The saving depends on shape; there is no universal factor of two. Transposing the result restores the input shape.

Question 9.14.1

9.14. µP and MetaP: make small-model tuning informative

What problem does µP solve?

One correct optionATransferring tuned hyperparameters across widths within a consistently parameterized familyBTraining stability when moving from bf16 to still lower precisionCAligning the learning-rate schedule with the training set length

A) Transferring tuned hyperparameters across widths within a consistently parameterized family

B) Training stability when moving from bf16 to still lower precision

C) Aligning the learning-rate schedule with the training set length

Answer and explanation

Correct answer: A. Without a consistent parameterization, useful update scales and learning rates can shift substantially with width. µP defines a model family in which tuning on a small proxy transfers much more reliably; it is not a guarantee across arbitrary architectures.

Question 9.15.1

9.15. Parallelism: fitting the training state before optimizing communication

What does ZeRO-3's ideal memory floor mean?

One correct optionAThe minimum below which training must fall back to CPU offloadBWeights, gradients and states are sharded across ranks; activations are notCThe volume occupied by one model layer on one accelerator of the cluster

A) The minimum below which training must fall back to CPU offload

B) Weights, gradients and states are sharded across ranks; activations are not

C) The volume occupied by one model layer on one accelerator of the cluster

Answer and explanation

Correct answer: B. ZeRO-3 shards the listed persistent state. Activation memory must be accounted for separately: at fixed local microbatch size it does not acquire the same inverse-rank scaling. Gather windows, communication buffers, and allocator overhead also affect peak memory.

Question 9.15.2

9.15. Parallelism: fitting the training state before optimizing communication

For eight stages, f=7/(m+7). What is the smallest m with f strictly below 5%?

One correct optionA133B134C128

A) 133

B) 134

C) 128

Answer and explanation

Correct answer: B. At m=133 the result is exactly 5%; m=134 gives 7/141 ≈ 4.965%.

Question 9.16.1

9.16. Four directions in contemporary pretraining

Why can reported wall-clock training time not be equated directly with the arithmetic compute estimate $C\approx6ND$ ?

One correct optionATime is always reported for one accelerator while the formula is for the full clusterBThe formula counts only the forward pass whereas time counts forward and backwardCAccelerator count, peak throughput, MFU, communication, and downtime stand between arithmetic FLOP and wall-clock time

A) Time is always reported for one accelerator while the formula is for the full cluster

B) The formula counts only the forward pass whereas time counts forward and backward

C) Accelerator count, peak throughput, MFU, communication, and downtime stand between arithmetic FLOP and wall-clock time

Answer and explanation

Correct answer: C. $N$ estimates useful arithmetic work. Converting it to seconds requires cluster size and effective throughput; without MFU and downtime, neither quantity determines the other.