VKVlad KulikovPublic / lectures
Modern LLMs9 / 20

Module 9. Pretraining: Data, Scaling Laws, and Optimizers

Try each exercise first. Expand worked answers or project guidance when needed; their open state is shared between RU/EN.

Download course files (.zip) 2588.1 KB

The archive preserves the source folder structure, both languages, and supporting files.

E1. Loss gap at a fixed training budget

Under the fitted law used in this module, evaluate the 8.03B-parameter, 15T-token scenario. Compare it with the analytic optimum at the same compute budget.

Show answer and solution

Solution. The budget is

C=6⋅8.03⋅109⋅15⋅1012=7.227⋅1023 FLOP.C=6\cdot8.03\cdot10^9\cdot15\cdot10^{12} =7.227\cdot10^{23}\ \text{FLOP}.

The scenario gives

L(8.03B,15T)=1.948310.L(8.03\text{B},15\text{T})=1.948310.

The fitted optimum is

N∗=35.663B,D∗=3.377T,N^*=35.663\text{B},\qquad D^*=3.377\text{T},

with

L∗=1.922505.L^*=1.922505.

Therefore

ΔL=L(8.03B,15T)−L(N∗,D∗)≈0.025804.\Delta L=L(8.03\text{B},15\text{T})-L(N^*,D^*)\approx\mathbf{0.025804}.

The difference uses unrounded loss values. This is arithmetic inside the stated approximation, not a claim about the realized quality of a particular 8B or 36B model. The notebook reproduces the calculation in “1. Scaling law and the exact compute-optimal split.”

E2. Inference-aware break-even

The 8.03B model from E1 used 7.227⋅10237.227\cdot10^{23} training FLOP. A 70B model reaches the same fitted loss with less training. If dense decoding costs approximately 2N2N FLOP per generated token, find the lifetime output volume at which the smaller model becomes cheaper in total compute.

Show answer and solution

Solution. Solving

L(70B,D70)=1.948310L(70\text{B},D_{70})=1.948310

gives

D70=1.094026⋅1012D_{70}=1.094026\cdot10^{12}

and

C70=6⋅70B⋅D70=4.59491⋅1023 FLOP.C_{70}=6\cdot70\text{B}\cdot D_{70} =4.59491\cdot10^{23}\ \text{FLOP}.

The large model saves

ΔC=2.63209⋅1023 FLOP\Delta C=2.63209\cdot10^{23}\ \text{FLOP}

in pretraining, while each generated token costs an extra

2(70−8.03)B=1.2394⋅1011 FLOP.2(70-8.03)\text{B}=1.2394\cdot10^{11}\ \text{FLOP}.

Thus

T∗=ΔC2(N70−N8)=2.12368⋅1012 tokens.T^*=\frac{\Delta C}{2(N_{70}-N_8)} =\mathbf{2.12368\cdot10^{12}}\ \text{tokens}.

Below that volume, the 70B alternative wins in this simplified total-compute objective; above it, the smaller model does. See “2. Equal-quality break-even and a training-time scenario.”

E3. Training-state memory and the 1F1B bubble

Assume 16 bytes per parameter for a 7B model, 64 ranks, and ideal ZeRO-3. Then compute the pipeline bubble for eight stages with 32 and 128 microbatches.

Show answer and solution

Solution. The fully replicated state is

7⋅109⋅16=112⋅109 bytes=112 GB=104.308 GiB.7\cdot10^9\cdot16=112\cdot10^9\ \text{bytes} =112\ \text{GB}=104.308\ \text{GiB}.

Ideal ZeRO-3 over 64 ranks gives

1.75 GB=1.630 GiB per rank.1.75\ \text{GB}=1.630\ \text{GiB per rank}.

For the pipeline,

f(8,32)=739=17.9%,f(8,32)=\frac{7}{39}=\mathbf{17.9\%},

and

f(8,128)=7135=5.2%.f(8,128)=\frac{7}{135}=\mathbf{5.2\%}.

At least 134 microbatches are required to fall strictly below 5% under this formula. The notebook contains the full tables and negative tests.

E4 [project]

Start with the eight synthetic documents in Demo 1. Explain every retention or rejection decision and report survival relative both to the initial pool and to each stage’s input. Change the filter order and similarity threshold, then distinguish changes in rejection attribution from changes in the retained set. Language and quality labels are supplied. An optional extension uses a small pinned, licensed external sample with actual classifiers and timings; a 100 GB download is unnecessary.

Project demos and guidance

Project demo: English Module 9 project-demo notebook, section “Demo 1.” It supplies the starting experiment; the exercise asks you to change its conditions and explain the outcome.

E5 [project]

Run the CPU mechanism in Demo 2: gather a matrix from two owned shards, average equal-sized local-batch gradients, and update each shard. Check parity with an unsharded update over several steps. Separately tabulate a hypothetical 1B model under ZeRO stages 0–3 at 1, 2, and 8 ranks using the 16-byte policy. Add a specified gather buffer and activation reserve. Explain why total peak memory no longer scales as one number divided by rank count. No distributed runtime is required.

Project demos and guidance

Project demo: English Module 9 project-demo notebook, section “Demo 2.” It supplies the matrix experiment and a separate hypothetical memory ledger.