Module 14. Learning from Feedback: RLHF, DPO, RLOO, and GRPO — quizzes and answer checking · Modern LLMs

Question 14.4.1

14.4. Classical RLHF as an algorithmic contract

Which stages form the classical RLHF route?

One correct optionASFT, reward-model training, and policy updates through PPOBPretraining, teacher distillation, and output-probability calibrationCPair collection, critic-only training, and best-of-group selection

A) SFT, reward-model training, and policy updates through PPO

B) Pretraining, teacher distillation, and output-probability calibration

C) Pair collection, critic-only training, and best-of-group selection

Answer and explanation

Correct answer: A. This is a three-stage algorithmic route. Policy, reference, reward model, and critic define four logical roles, although an implementation may share backbones, offload, or shard them.

Question 14.5.1

14.5. Worked example A: invert a Bradley–Terry model

Annotators prefer one response in 75% of comparisons. What score difference does the Bradley–Terry model imply?

One correct optionA0.750 — the win rate is directly the score differenceB1.099 — ln(0.75/0.25), the inverse sigmoid transformC0.622 — the sigmoid evaluated at the observed win rate

A) 0.750 — the win rate is directly the score difference

B) 1.099 — ln(0.75/0.25), the inverse sigmoid transform

C) 0.622 — the sigmoid evaluated at the observed win rate

Answer and explanation

Correct answer: B. Bradley–Terry maps a score difference through a sigmoid. The inverse map is the logit, giving ln(3) ≈ 1.099.

Question 14.6.1

14.6. DPO: eliminate the explicit reward model

What does DPO remove from the classical RLHF loop?

One correct optionAOnly KL regularization, while retaining the reward model and PPOBOnly PPO clipping, replacing it with an ordinary policy gradientCIt removes the explicit reward model and PPO, training directly on pairs

A) Only KL regularization, while retaining the reward model and PPO

B) Only PPO clipping, replacing it with an ordinary policy gradient

C) It removes the explicit reward model and PPO, training directly on pairs

Answer and explanation

Correct answer: C. Under the chosen KL-regularized formulation, reward differences can be written through policy ratios. This yields a direct pairwise loss, not a universal standalone response evaluator.

Question 14.7.1

14.7. Worked example B: read the DPO margin carefully

For a DPO margin z = 0.170, what is the magnitude of the local derivative with respect to z?

One correct optionA0.4576 — it is σ(−z)B1.0000 — the DPO derivative is independent of the marginC0.6118 — the derivative equals the loss itself

A) 0.4576 — it is σ(−z)

B) 1.0000 — the DPO derivative is independent of the margin

C) 0.6118 — the derivative equals the loss itself

Answer and explanation

Correct answer: A. For L = softplus(−z), the derivative magnitude with respect to z is σ(−z). The full parameter gradient also contains β and log-probability derivatives.

Question 14.7.2

14.7. Worked example B: read the DPO margin carefully

For the same log-ratio gap, β increases from 0.1 to 0.5. What happens in the numerical example?

One correct optionAThe margin is unchanged because β cancels inside the differenceBThe margin grows to 0.85 and the local pair weight falls to 0.299CThe margin shrinks to 0.034 and the local weight rises above 0.5

A) The margin is unchanged because β cancels inside the difference

B) The margin grows to 0.85 and the local pair weight falls to 0.299

C) The margin shrinks to 0.034 and the local weight rises above 0.5

Answer and explanation

Correct answer: B. In this fixed example, β scales the margin passed to the sigmoid. This describes local pair geometry; its effect on full training also depends on data and policy dynamics.

Question 14.7.3

14.7. Worked example B: read the DPO margin carefully

Which positions normally enter the log-probability sum for a DPO pair?

One correct optionAEvery position, including the shared prompt and batch paddingBOnly the first token at which the two responses divergeCCompletion tokens; the shared prompt and padding are masked out

A) Every position, including the shared prompt and batch padding

B) Only the first token at which the two responses diverge

C) Completion tokens; the shared prompt and padding are masked out

Answer and explanation

Correct answer: C. The preferred and rejected responses share a prompt. The teaching contract sums log-probabilities over completion tokens only and masks padding explicitly.

Question 14.7.4

14.7. Worked example B: read the DPO margin carefully

How large is the DPO derivative magnitude with respect to z when z<0?

One correct optionAAbove 0.5, approaching 1 as z → −∞BAlways below 0.5CAlways equal to β

A) Above 0.5, approaching 1 as z → −∞

B) Always below 0.5

C) Always equal to β

Answer and explanation

Correct answer: A. 0.5 is the value at z=0. The parameter gradient also includes β and derivatives of log probabilities.

Question 14.8.1

14.8. Direct preference losses differ by contract, not branding

Why are DPO, IPO, KTO, SimPO, and ORPO not merely different names for one method?

One correct optionAThey change labels, the reference, normalization, or the objectiveBThey differ only by laboratory and publication yearCThey use one loss but different variants of the Adam optimizer

A) They change labels, the reference, normalization, or the objective

B) They differ only by laboratory and publication year

C) They use one loss but different variants of the Adam optimizer

Answer and explanation

Correct answer: A. For example, KTO accepts individual binary labels, SimPO uses no separate reference policy, and ORPO combines SFT and preference learning in one objective.

Question 14.9.1

14.9. RLOO and GRPO: construct a baseline from sibling rollouts

How does RLOO's baseline differ from GRPO's full-group mean?

One correct optionARLOO uses the previous group whereas GRPO uses the current groupBRLOO excludes the current response from its own baselineCRLOO averages log rewards whereas GRPO averages rewards

A) RLOO uses the previous group whereas GRPO uses the current group

B) RLOO excludes the current response from its own baseline

C) RLOO averages log rewards whereas GRPO averages rewards

Answer and explanation

Correct answer: B. A leave-one-out baseline uses the other responses. The full-group mean contains the current response reward, so the two estimators do not automatically share the same statistical properties.

Question 14.9.2

14.9. RLOO and GRPO: construct a baseline from sibling rollouts

How should a GRPO group with identical rewards be handled?

One correct optionAReuse the previous group mean and continue with the same objectiveBAdd ε to the denominator and treat the relative signal as recoveredCExplicitly filter the group or assign zero advantages: no relative signal exists

A) Reuse the previous group mean and continue with the same objective

B) Add ε to the denominator and treat the relative signal as recovered

C) Explicitly filter the group or assign zero advantages: no relative signal exists

Answer and explanation

Correct answer: C. With zero variance, z-normalization is undefined. A small ε can prevent NaNs but cannot create information, so handling must be part of the explicit training contract.

Question 14.9.3

14.9. RLOO and GRPO: construct a baseline from sibling rollouts

Which policy forms the denominator of the clipped objective ratio?

One correct optionAThe policy that sampled this rollout batchBNecessarily the initial SFT policyCThe policy after the next update

A) The policy that sampled this rollout batch

B) Necessarily the initial SFT policy

C) The policy after the next update

Answer and explanation

Correct answer: A. The old policy describes the sampler. The reference supplies a separate KL anchor.

Question 14.9.4

14.9. RLOO and GRPO: construct a baseline from sibling rollouts

All rewards are equal and advantages are zero. Can weights still update?

One correct optionAYes, through a separate KL termBNo, every gradient must vanishCYes, division by zero creates a signal

A) Yes, through a separate KL term

B) No, every gradient must vanish

C) Yes, division by zero creates a signal

Answer and explanation

Correct answer: A. Zero advantage removes the policy-gradient term, but the KL gradient need not vanish.

Question 14.10.1

14.10. Worked example D: a deliberately simplified memory model

In the 8.03B teaching memory model, how do PPO and DPO compare?

One correct optionA269.2 GiB and 134.6 GiB in the stated teaching configurationB134.6 GiB versus 119.7 GiB regardless of roles or implementationC119.7 GiB for both algorithms in every configuration

A) 269.2 GiB and 134.6 GiB in the stated teaching configuration

B) 134.6 GiB versus 119.7 GiB regardless of roles or implementation

C) 119.7 GiB for both algorithms in every configuration

Answer and explanation

Correct answer: A. This is a scenario for persistent parameter states: 16 bytes per trained parameter and 2 per frozen parameter. It excludes activations, KV state, sharding, and shared backbones.

Question 14.10.2

14.10. Worked example D: a deliberately simplified memory model

When can a fixed DPO dataset remove the reference model from memory during optimization?

One correct optionAWhen the reference is quantized to four bits at every stepBWhen its log-probabilities have been precomputed for every pairCWhen the trainable policy and reference always share the same weights

A) When the reference is quantized to four bits at every step

B) When its log-probabilities have been precomputed for every pair

C) When the trainable policy and reference always share the same weights

Answer and explanation

Correct answer: B. For a fixed pair dataset, reference log-probabilities can be computed once. Optimization then keeps one trained network; this is a workflow property, not a universal DPO implementation rule.

Question 14.11.1

14.11. Worked example E: rollout generation can dwarf the update

Why can rollout generation dominate the time budget in online RL?

One correct optionAIt necessarily runs on CPU while accelerators remain idleBEvery rollout requires training a new reward modelCAutoregressive decode produces tokens sequentially

A) It necessarily runs on CPU while accelerators remain idle

B) Every rollout requires training a new reward model

C) Autoregressive decode produces tokens sequentially

Answer and explanation

Correct answer: C. In the module scenario, long responses produce a 39.3× ratio between generation and one estimated training pass. The number depends on length, hardware, and throughput, but the mechanism is general.

Question 14.12.1

14.12. What later GRPO variants are actually fixing

Which problems do DAPO and Dr. GRPO address?

One correct optionAGroup selection, clipping, length normalization, and token efficiencyBOnly reward-model training speed on the original labelsCRestoring a mandatory critic to every group-relative RL method

A) Group selection, clipping, length normalization, and token efficiency

B) Only reward-model training speed on the original labels

C) Restoring a mandatory critic to every group-relative RL method

Answer and explanation

Correct answer: A. DAPO introduces dynamic sampling, asymmetric clipping, token-level normalization, and overlong-response shaping. Dr. GRPO separately revisits normalization that can induce undesirable bias.

Question 14.12.2

14.12. What later GRPO variants are actually fixing

Responses have lengths 1 and 3 with per-token values 2 and 0. What is the token mean?

One correct optionA0.5B1C2

A) 0.5

B) 1

C) 2

Answer and explanation

Correct answer: A. The sum is 2 across 4 tokens. A response mean would be 1.

Question 14.15.1

14.15. RLVR still obeys Goodhart's law

Why does a verifiable reward not eliminate reward hacking automatically?

One correct optionAThe verifier is always slower than the model and cannot score responsesBThe policy optimizes the check, not necessarily the intended taskCFormal tests cannot be used for mathematics or programming

A) The verifier is always slower than the model and cannot score responses

B) The policy optimizes the check, not necessarily the intended task

C) Formal tests cannot be used for mathematics or programming

Answer and explanation

Correct answer: B. Verifiability makes the signal reproducible, but tests, rules, and length limits can contain loopholes. Hidden tasks, rollout audits, and verifier evaluation are still required.

Question 14.16.1

14.16. Three open patterns: RLAIF, multi-stage reasoning, and environments

What is the published Constitutional AI route?

One correct optionAThe constitution only filters the raw dataset before SFTBThe principles are appended as a system prompt to every user requestCCritique and revision under principles, followed by RLAIF with a reward model and RL

A) The constitution only filters the raw dataset before SFT

B) The principles are appended as a system prompt to every user request

C) Critique and revision under principles, followed by RLAIF with a reward model and RL

Answer and explanation

Correct answer: C. The method includes supervised critique and revision followed by RLAIF: a model produces constitutional preferences, which train a reward model and then an RL policy. Human effort is reduced, not eliminated.