RL for LLM8 / 13

Module 8. RLHF and direct preference optimization

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 0 · Correct: 0

Question P01 · Why beta does not determine realized KL

Two DPO runs use the same $\beta$ but different learning rates, epoch counts, and data. Must they end with the same KL to the reference?

Your answer or notes

Answer and explanation

No. In the variational objective $\beta$ prices KL; in DPO it scales the classification logit. Realized drift depends on data, log-probability normalization, optimizer, update count, and parameterization. Measure KL separately under a fixed protocol.

Question P02 · Zero reference support

For one completion $\pi_{\mathrm{ref}}(y\mid x)=0$ while the current policy assigns positive probability. What happens to forward KL and the DPO ratio?

Your answer or notes

Answer and explanation

Forward KL is infinite for policy mass outside reference support, and $\beta$ is not finite. Softmax models usually have positive mathematical probabilities, but hard masks, truncation, or numerical underflow can create effective zeros; support is part of the contract.

Question P03 · cDPO is not automatically Robust DPO

Why can label smoothing with $\varepsilon$ not be called an unbiased noise correction without qualification?

Your answer or notes

Answer and explanation

cDPO mixes positive and flipped binary losses as a conservative heuristic. Robust DPO derives a particular correction under a symmetric label-flip model. Neither repairs a systematically wrong rubric, leakage, or distribution shift.

Question P04 · Stale fixed pairs

After several epochs the policy nearly perfectly separates old pairs, but fresh current-policy responses show new errors. What should be done?

Your answer or notes

Answer and explanation

Stop drawing conclusions from old pair accuracy, collect fresh current-policy candidates, label or verify them, re-check base capabilities, and consider iterative DPO or online RL. A fixed set need not remain representative.

Question P05 · Margin up, chosen logp down

Mean DPO margin rises from 0.2 to 1.1 while mean logps/chosen falls. Is that a contradiction?

Your answer or notes

Answer and explanation

No. DPO optimizes a relative reference-corrected gap. The rejected completion may lose likelihood faster, while unlabelled completions gain mass. Track chosen and rejected logp separately and evaluate fresh generations independently.

Question P06 · Truncating one completion

The chosen completion is cut at 256 tokens while the rejected completion fits in 90. Can the pair be treated as an ordinary DPO example?

Your answer or notes

Answer and explanation

Only if the truncation rule was defined in advance and the truncated strings are the actual preference objects. Otherwise the label concerns a different sequence than the loss. Filter or re-label such pairs and report truncation rates.

Question P07 · Choose from the data contract

You have separate desirable and undesirable completions but no same-prompt pairs. What is the sound first choice: DPO, KTO, or artificial pairing?

Your answer or notes

Answer and explanation

Use a method matching the observed feedback, such as KTO or another binary objective. Pairing unrelated prompts changes the question and breaks the prompt-dependent normalizer cancellation behind DPO.

Question P08 · UltraFeedback provenance

May trl-lib/ultrafeedback_binarized be described as direct human preference data?

Your answer or notes

Answer and explanation

No. It is a derivative of UltraFeedback with synthetic model-generated feedback. It is useful for engineering practice, but the feedback provenance, candidate generators, and binarization must be stated.

Question P09 · A shared chat template is mathematical

Why must policy and reference log-probabilities use the same chat template and response mask?

Your answer or notes

Answer and explanation

Otherwise $\beta$ compares different token sequences or token sets. The margin no longer implements the DPO formula and may measure formatting rather than preference.