VKVlad KulikovPublic / lectures
Modern LLMs13 / 20

Module 13. Supervised Fine-Tuning and Data Work

Try each exercise first. Expand worked answers or project guidance when needed; their open state is shared between RU/EN.

Download course files (.zip) 2588.1 KB

The archive preserves the source folder structure, both languages, and supporting files.

E1. Labels after serialization

A system, user, and assistant message contain 12, 8, and 20 content tokens. Each message adds five structural tokens, and the dialogue begins with BOS. Compute the supervised fraction under assistant-only loss when the assistant end-of-turn token is included.

Show answer and solution

Solution. BOS contributes 1 token, three message shells contribute 3⋅5=153\cdot5=15, and content contributes 12+8+20=4012+8+20=40, for 56 total. Under assistant-only loss, system/user content and structure plus the assistant header receive label -100. The target contains 20 assistant-content tokens and its EOT: 21 tokens, or 21/56=37.5%21/56=37.5\%. The target boundary must be derived after applying the real chat template, not guessed from raw strings.

E2. First-Fit Decreasing

Pack

[403,527,315,180,267,165,425,1347,259,230,626,556,443,174,392,754,120,267,72,126,76,326,128,514] into 2,048-token containers and compare utilization with padding each sequence separately.

Show answer and solution

Solution. FFD sorts lengths descending, then places each item in the first existing bin with enough space. It produces

text
[1347, 626, 72]                  = 2045
[754, 556, 527, 180]            = 2017
[514, 443, 425, 403, 259]       = 2044
[392, 326, 315, 267, 267, 230,
 174, 76]                        = 2047
[165, 128, 126, 120]             = 539

Useful length is 8,692. Independent padding gives 8692/(24⋅2048)=17.684%8692/(24\cdot2048)=17.684\% utilization; five FFD rows give 8692/(5⋅2048)=84.883%8692/(5\cdot2048)=84.883\%. This reduces fixed-shape forward rows by 4.8×, not necessarily wall time by 4.8×. A segment-aware attention mask must prevent cross-example attention.

E3. Compute, best-of-NN, and LoRA

Revisit the four scenarios in §§7, 10, 11 and 16. For SFT, use an 8.03B model, 200,000 examples averaging 1,200 tokens and three epochs. Calculate FLOP and runtime on one accelerator with a 990 TFLOP/s peak at 40% MFU, then compare the workload with pretraining the same model on 15 trillion tokens.

For independent candidates with ppass=0.3p_{\mathrm{pass}}=0.3, find best-of-NN coverage at N=1,4,8,16N=1,4,8,16. Next, estimate FLOP and runtime for one pass of a 32B model over 1,000 traces of 9,000 tokens, using eight accelerators with the same peak rate and MFU. For the LoRA calculation, adapt q/k/v/o in 32 layers with d=4096d=4096, dkv=1024d_{kv}=1024 and rank 16. Report adapter parameters, their fraction of the 8.03B base and training-state MiB at 16 bytes per trainable parameter.

Show answer and solution

Solution. One epoch contains 200000⋅1200=240200000\cdot1200=240M tokens; three contain D=720D=720M. With C≈6NDC\approx6ND,

CSFT=6⋅8.03B⋅720M=3.46896×1019 FLOP.C_{SFT}=6\cdot8.03\text{B}\cdot720\text{M}=3.46896\times10^{19}\ \text{FLOP}.

Dividing by 990⋅1012⋅0.4990\cdot10^{12}\cdot0.4 FLOP/s gives 24.33 hours; the stated 15T-token pretraining workload is about 20,833 times larger. Independent best-of-NN coverage is 1−(1−0.3)N1-(1-0.3)^N: 0.3000, 0.7599, 0.9424, and 0.9967 for N=1,4,8,16N=1,4,8,16.

The 32B scenario has 1000⋅9000=91000\cdot9000=9M tokens, hence 6⋅32B⋅9M=1.728×10186\cdot32\text{B}\cdot9\text{M}=1.728\times10^{18} FLOP and 9.09 minutes on eight stated devices.

A LoRA adapter for a dout×dind_{out}\times d_{in} matrix has r(din+dout)r(d_{in}+d_{out}) parameters. Per layer q/o contribute 16(4096+4096)16(4096+4096) each and k/v 16(4096+1024)16(4096+1024) each: 425,984 per layer, or 13,631,488 over 32 layers (0.1698% of the base). At the teaching 16-byte parameter-state accounting, that is 208 MiB.

E4 [project]

Extend the CPU dialogue task in Demo 1 with a second turn and a tool observation, supervising only the chosen assistant actions. Compare separate and packed execution on logits, summed loss, and gradients. Deliberately introduce a cross-example attention link and locate the resulting mismatch. Optionally pin a small model’s official tokenizer and chat template, then repeat the boundary inspection using its real token IDs.

Project demos and guidance

Demo for this project exercise: English project-demo notebook for Module 13, section “Demo 1”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

E5 [project]

In Demo 2, change the true update rank and training-set size. Compare full updates with LoRA by trainable parameters and held-out MSE across several seeds. Separate rank constraints from finite-optimization error. An optional extension uses small verified-solution and multi-turn tool datasets with a common pretrained base, adding runtime and memory measurements and preserving synthetic-data provenance and verifier decisions.

Project demos and guidance

Demo for this project exercise: English project-demo notebook for Module 13, section “Demo 2”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.