RL for LLM9 / 13

Module 9. RLVR and GRPO: verifiable rewards, group-relative estimates, and stability

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 6 · Correct: 0

Question P01 · The RLVR boundary

Choose the only case that is by itself RLVR training.

One correct option1The model samples 32 solutions, tests select the best, and weights stay fixed.2Deterministic test outcomes are used as rewards in a policy update.3The model is supervised-fine-tuned on solutions that happened to pass tests.4A reward model reranks responses only for display.

The model samples 32 solutions, tests select the best, and weights stay fixed.

Deterministic test outcomes are used as rewards in a policy update.

The model is supervised-fine-tuned on solutions that happened to pass tests.

A reward model reranks responses only for display.

Answer and explanation

The correct answer is B. RLVR uses a verifiable outcome inside the policy-learning update. A is test-time search, C is supervised learning on filtered data, and D is reranking without parameter updates.

Question P02 · Probability of an informative group

A binary reward has single-sample success probability $p=0.25$ and group size $G=4$ . Compute the probability that the group contains at least one success and at least one failure.

Informative-group probabilityAbsolute tolerance: ±0.000001

Answer and explanation

P_{\mathrm{info}}=1-p^G-(1-p)^G =1-0.25^4-0.75^4=0.6796875.

The calculation assumes conditionally independent samples. Correlated completions provide less effective diversity.

Question P03 · The denominator policy

Completions were generated by policy revision v17; optimization is now evaluating v19; KL is measured to a frozen SFT model. Which policy belongs in the probability-ratio denominator?

Your answer or notes

Answer and explanation

The denominator is the actual behavior policy v17, including the decoding transformations used during sampling. v19 is the current policy and the SFT model is the reference anchor; neither can replace the source of the stored actions.

Question P04 · Advantages in a symmetric group

A group has rewards $[1,1,0,0]$ . Compute the advantages after subtracting the mean and dividing by the population standard deviation.

A₁Absolute tolerance: ±0.000001A₂Absolute tolerance: ±0.000001A₃Absolute tolerance: ±0.000001A₄Absolute tolerance: ±0.000001

Answer and explanation

The mean is $0.5$ and the population standard deviation is also $0.5$ , hence

\widehat A=[1,1,-1,-1].

They sum to zero. These are response-level signals, not causal token-level credit assignments.

Question P05 · When clipping is active

For ratio $\rho=1.35$ , advantage $A=2$ , and symmetric clip interval $[0.8,1.2]$ , compute the unclipped and clipped terms. Which branch enters the minimum?

Unclipped termAbsolute tolerance: ±0.000001Clipped termAbsolute tolerance: ±0.000001

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

Answer and explanation

The unclipped term is $0.5$ . The clipped term is $1.2\cdot2=2.40$ , so the minimum is $2.40$ . With positive advantage, upper clipping limits further incentive to increase an already enlarged ratio.

Question P06 · Implicit prompt reweighting

Two groups have centered rewards $[-0.1,0.1]$ and $[-1,1]$ . What happens after each group is divided by its own standard deviation? Why is this not neutral?

Your answer or notes

Answer and explanation

Both become $0.5$ . Per-group standardization removes the difference in raw reward scale and therefore reweights prompts: a small, possibly noisy gap receives the same normalized magnitude as a large gap. This may stabilize numbers, but it changes the objective’s weighting.

Question P07 · Length and reduction unit

Two responses have equal-magnitude advantages and lengths 20 and 200. Compare per-response token averaging with one global average over all active tokens.

Your answer or notes

Answer and explanation

Per-response averaging gives equal total weight to the responses, so each short-response token has ten times the coefficient. A global active-token average gives equal token coefficients, so the long response receives about ten times the total contribution. The reduction unit is part of the objective, not a cosmetic implementation choice.

Question P08 · The four DAPO components

Name the four changes bundled in DAPO and explain why “we use DAPO” is insufficient for reproducibility.

Your answer or notes

Answer and explanation

DAPO highlights Clip-Higher, Dynamic Sampling, Token-Level Policy Gradient Loss with global token normalization, and Overlong Reward Shaping. A reproducible run still needs group and batch sizes, clip bounds, degenerate-group handling, exact reduction, KL choices, truncation and length rules, and policy revisions. Implementations may use only part of the recipe or combine it with other variants.

Question P09 · GSPO sequence ratio

The token ratios are $(1.21,0.81)$ . Compute the length-normalized GSPO sequence ratio.

GSPO ratioAbsolute tolerance: ±0.000001

Answer and explanation

P_{\mathrm{info}}=1-p^G-(1-p)^G =1-0.25^4-0.75^4=0.6796875.

This is the geometric mean of token ratios. It is GSPO’s normalized sequence construction, not the exact full-sequence likelihood-ratio product.

Question P10 · Entropy is not diversity

A model produces 64 nearly identical strings with punctuation variation. Mean token entropy remains high, but every answer uses the same wrong algorithm. What should be measured in addition to token entropy?

Your answer or notes

Answer and explanation

Measure outcome and response-level diversity: pass@k or success@k, unique normalized answers, semantic or strategy clusters, length distribution, and repeated-template rates. Local uncertainty can be spent on surface form without creating new solution strategies.

Question P11 · Terminal or truncation

A response hits max_new_tokens without EOS. Why should it not automatically be treated as an ordinary incorrect terminal? Give two defensible treatments and their risks.

Your answer or notes

Answer and explanation

The limit is an external truncation: the eventual outcome is unobserved. Excluding the response reduces ambiguous noise but may let the policy evade negative reward by continuing forever. A declared soft length penalty encourages completion but can shorten genuinely difficult reasoning prematurely. The stop reason must be logged separately in either case.

Question P12 · Reading the teaching run

In the teaching run, mean training-prompt accuracy rises from 0.31 to 0.41, entropy roughly halves, and held-out success disappears. Which conclusion is justified?

One correct option1GRPO universally harms generalization.2Low entropy is always harmful.3Optimization improved measured reward on part of the training distribution, while coverage and transfer required separate diagnosis.4The verifier must be wrong.

GRPO universally harms generalization.

Low entropy is always harmful.

Optimization improved measured reward on part of the training distribution, while coverage and transfer required separate diagnosis.

The verifier must be wrong.

Answer and explanation

The correct answer is C. The small reproducible run demonstrates a possible concentration and train–transfer gap. It does not establish a universal property of GRPO, a universal entropy threshold, or a verifier bug.