Module 14. Learning from Feedback: RLHF, DPO, RLOO, and GRPO — exercises and worked solutions · Modern LLMs

E1. Bradley–Terry and DPO by hand

Compute $\sigma(\Delta)$ for $\Delta\in\{0.5,1,2,4\}$ and recover the logit associated with a 75% win rate. Then evaluate two DPO pairs at $\beta=0.1$ . The policy sequence log probabilities are $(-12.0,-11.5)$ for both pairs. The reference values are $(-13.0,-10.8)$ for pair I and $(-12.8,-11.65)$ for pair II. The preferred completion is listed first.

Show answer and solution

Solution. The sigmoid values are 0.622459, 0.731059, 0.880797, and 0.982014. The inverse logit is $\log(0.75/0.25)=\log3=1.098612$ .

For pair I, policy chosen-minus-rejected margin is $-12-(-11.5)=-0.5$ and reference margin is $-13-(-10.8)=-2.2$ . Their difference is 1.7, so $z=0.1\cdot1.7=0.17$ and $L=-\log\sigma(z)=0.611755$ . For pair II, reference margin is $-12.8-(-11.65)=-1.15$ ; the margin gap is 0.65, giving $z=0.065$ and $L=0.661175$ .

The derivative sign matters: $dL/dz=\sigma(z)-1$ , so the values are −0.457602 and −0.483756. At $\beta=0.5$ , pair I has $z=0.85$ , loss 0.355865, and $dL/dz=-0.299433$ . The full parameter gradient also contains $\beta$ and log-probability derivatives.

E2. GRPO group and persistent-state memory

For rewards [1,0,0,1,0,0,1,0], compute population-standardized advantages and the probability of a degenerate group of size 8 when the independent success probability is 0.3. Then compare the teaching PPO and DPO memory models for 8.03B parameters using 16 bytes per trainable parameter and 2 bytes per frozen bf16 parameter.

Show answer and solution

Solution. Three of eight rewards equal one, so $\mu=3/8=0.375$ and population standard deviation is $\sqrt{\frac18\sum_i(r_i-\mu)^2}=0.484123$ . Standardization gives 1.290994 to successes and −0.774597 to failures; advantages sum to zero. A group is degenerate if all samples succeed or all fail: $0.3^8+0.7^8=0.057714$ .

For 8.03B parameters, trainable state is $(8.03\cdot10^9\cdot16)/2^{30}\approx119.656$ GiB and a frozen bf16 copy is $(8.03\cdot10^9\cdot2)/2^{30}\approx14.957$ GiB. The teaching PPO anatomy has two trainable networks (policy and value) plus two frozen ones (reference and reward), giving $36\cdot8.03\cdot10^9/2^{30}\approx269.227$ GiB. DPO has one trainable policy plus one frozen reference, giving 134.613 GiB. These are persistent-state floors without activations or rollout buffers.

E3. Rollout economics

Generate 8 completions of 4,000 tokens for each of 256 prompts. Estimate generation time on eight accelerators at 209.4 tokens/s each. Compare it with a $6ND$ training estimate for an 8.03B model at 990 TFLOP/s per accelerator and 40% MFU.

Show answer and solution

Solution. Rollout volume is $D=256\cdot8\cdot4000=8{,}192{,}000$ tokens. Eight devices provide aggregate rate $8\cdot209.4=1675.2$ tok/s, so generation takes $D/1675.2=4890.2$ s = 81.50 minutes.

Training arithmetic is

$C=6ND=6\cdot8.03\text{B}\cdot8.192\text{M}\approx3.946\cdot10^{17}\ \text{FLOP}.$

Effective throughput is $8\cdot990\text{ TFLOP/s}\cdot0.4=3.168\cdot10^{15}$ FLOP/s, hence 124.6 s = 2.076 minutes. The ratio is 39.25: autoregressive rollout, not one training pass, dominates this scenario. It is not a universal RLVR constant.

E4 [project]

Compare run_dpo at $\beta=0.1$ and $\beta=0.5$ using the same pairs and 140 updates. Begin with the default zero initialization, which makes both runs deterministic. Then set init_std=0.02 and repeat the comparison for seeds 41, 42 and 43. Within each seed, both $\beta$ values must start from identical weights and an identical reference copied from those weights.

Record initial and final loss, training-pair ordering accuracy, held-out correct-answer probability and reference KL. Explain why losses measured at different $\beta$ values are not a direct scale of answer quality, and why correct training-pair ordering does not establish generation quality. As an optional extension, use an open 0.5–3B model with pinned templates and masks, then inspect response lengths and 50 generated answers.

Project demos and guidance

Demo for this project exercise: English project-demo notebook for Module 14, section “Demo 1”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

E5 [project]

Implement a GRPO-like loop for a binary-verifier task. Compare skipping, zeroing, and resampling equal-reward groups. Measure rollout cost, zero-variance rate, and sensitivity to the number of completions $G$ .

Project demos and guidance

Demo for this project exercise: English project-demo notebook for Module 14, section “Demo 2”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.