RL for LLM9 / 13

Module 9. RLVR and GRPO: verifiable rewards, group-relative estimates, and stability

Try each exercise first. Expand worked answers or project guidance when needed; their open state is shared between RU/EN.

Download course files (.zip) 382.1 KB

The archive preserves the source folder structure, both languages, and supporting files.

RLVR or not?

Classify each case as RLVR, not RLVR, or insufficient information.

A model produces 16 programs, tests select the best, and weights do not change.
A model produces programs and test outcomes enter a policy-gradient update.
A policy is trained with DPO on a fixed set of paired mathematical solutions.
An LLM judge scores responses and PPO follows.
An agent receives reward from a simulator state.

Explain every label briefly.

Show answer and solution

Not an RLVR training stage: tests are used for Best-of-N only; weights do not change.
RLVR: verifiable test outcomes enter a policy update.
Not RLVR in this stage: fixed-pair DPO has no fresh verifier reward inside the update.
Insufficient information / usually not RLVR: an LLM judge is a learned evaluator rather than an automatically verifiable rule; the regime is typically RLHF or RLAIF.
RLVR in the broad operational sense when simulator transition and reward are reproducible and enter learning.

Three policies

Responses were collected by checkpoint step_1200; optimization has reached step_1203; KL is measured against the original SFT model. Identify the old, current, and reference policies. Which one belongs in the importance-ratio denominator?

Show answer and solution

Old policy: step_1200; current policy: step_1203; reference policy: the original SFT model. The importance denominator is the actual behavior policy step_1200, including sampling transforms.

Two verifier error types

Give one false-positive and one false-negative reward example for:

a numerical task;
a programming task.

Explain how repeated optimization could amplify each error.

Show answer and solution

Numerical false positive: parse gold 42 from an incorrect sentence ending in 41. Numerical false negative: reject the correct fraction 1/2 because only decimals are accepted. Code false positive: incomplete tests accept a constant answer. Code false negative: a correct program times out in an unstable environment. Repeated optimization promotes systematic passes and suppresses systematic failures.

“Critic-free”

Evaluate this statement in three independent parts:

“GRPO has no critic, therefore it has no baseline and provides unbiased token-level credit assignment.”

Show answer and solution

GRPO can remove a learned value critic. It still has a baseline, usually the group mean. A single response-level advantage applied to every token does not identify causal token contributions and is not an unbiased token-level credit assignment oracle.

The cost of dynamic sampling

Why can removing all-equal-reward groups simultaneously:

increase the useful signal per update;
increase generation cost;
change the training-prompt distribution?

Name at least two audit metrics.

Show answer and solution

Filtering increases the nonzero-gradient fraction, but requires extra generation and shifts training toward intermediate-difficulty prompts. Log rejection fraction/reason, generated tokens per accepted group, original pass-rate distribution, and full-distribution held-out quality.

Match the variants

Match each method to its defining change.

Dr. GRPO
DAPO
GSPO
original GRPO

A. geometric-mean sequence ratio and response-level clipping
B. group standard-deviation scaling and per-response token averaging
C. remove standard-deviation scaling and response-specific length normalization
D. Clip-Higher, dynamic sampling, shared token normalization, and overlong handling

Why may a production run match none of the rows exactly?

Show answer and solution

1–C, 2–D, 3–A, 4–B. Implementations can combine Dr. GRPO reward centering, DAPO token normalization, asymmetric clipping, and token-level ratios, forming a hybrid.

Entropy and diversity

Explain why the following are not interchangeable:

mean token entropy;
number of distinct strings in a group;
number of distinct solution strategies;
pass@k.

Give one example where one is high while another is low.

Show answer and solution

Token entropy is local vocabulary uncertainty; string diversity can be punctuation variation; strategy diversity requires semantic grouping; pass@k measures at least one correct sample. Many paraphrases can yield high string diversity but one strategy, while two deterministic solution templates can create strategy diversity concentrated at one branching token.

Truncation

A response reaches the length limit without producing EOS. Compare:

reward it as incorrect;
mask it from the loss;
add a soft length penalty;
increase the limit.

Give one possible benefit and one risk for each.

Show answer and solution

Incorrect reward creates termination pressure but may punish sound incomplete work. Masking removes unknown-sign noise but can reward never stopping. A soft penalty gives a graded signal but may shorten difficult solutions prematurely. A larger limit reveals more endings but raises cost and repetition risk.

Informative-group probability

For binary reward, $p=0.1$ , and $G=8$ , compute

P_{\mathrm{info}}=1-p^G-(1-p)^G.

How many prompts are expected to obtain 32 informative groups? State one assumption that may make the calculation optimistic.

Show answer and solution

P_{\mathrm{info}}=1-0.1^8-0.9^8\approx0.56953278,

so $32/P_{\mathrm{info}}\approx56.19$ prompt draws. Conditional independence and stationary $p$ are the fragile assumptions.

One success in a group

A group of $G=8$ contains one reward 1 and seven rewards 0. Compute normalized advantages using population standard deviation and verify that they sum to zero.

Show answer and solution

Mean $1/8$ , population standard deviation $\sqrt7/8$ . The success has $\sqrt7\approx2.6458$ and each failure has $-1/\sqrt7\approx-0.3780$ . Their sum is zero.

Token weight and response length

Two responses have rewards $[1,0]$ and lengths $[50,500]$ . Original GRPO uses group standard-deviation scaling and token coefficient $\widehat A_i/(G|o_i|)$ .

Compute the advantages.
Compute one-token weights.
Compute total response contributions before importance ratios.
What is the token-weight ratio?

Show answer and solution

Advantages are $[1,-1]$ . Token coefficients are $1/(2\cdot50)=0.01$ and $-1/(2\cdot500)=-0.001$ . Response contributions are $0.5$ and $-0.5$ ; token magnitudes differ by 10.

DAPO soft overlong penalty

Let $L_{\max}=1000$ and $L_{\mathrm{cache}}=200$ . Compute $R_{\mathrm{length}}$ at lengths 750, 900, 1000, and 1200. Why is the value at 900 not a binary penalty?

Show answer and solution

The no-penalty boundary is 800. Penalties are 0, -0.5, -1, and -1. Length 900 lies inside the linear cache interval.

GSPO sequence ratio

Token ratios are

(1.01,0.99,1.02,1.00).

Compute

s=\exp\left(\frac14\sum_t\log\rho_t\right).

Compare it with the arithmetic mean and explain the geometric mean.

Show answer and solution

s=(1.01\cdot0.99\cdot1.02\cdot1.00)^{1/4}\approx1.0049.

The arithmetic mean is 1.005. The geometric mean follows from sequence probability products and can be computed stably through mean log-ratios.

Independent pass@k model

For one-attempt success probability $p=0.2$ , compute at least one success for $k=1$ , $4$ , and $16$ :

1-(1-p)^k.

Why must an LLM group be checked before using this independence model?

Show answer and solution

pass@1 = 0.2, pass@4 = 0.5904, pass@16 $\approx0.9719$ . LLM samples can be correlated and prompt-specific; finite-sample pass@k estimation also uses a separate combinatorial estimator.

Numerical verifier

Implement:

Python

def verify_numeric_answer(
    response: str,
    gold: str,
    rtol: float = 1e-6,
    atol: float = 1e-9,
) -> int:
    ...

Contract:

the last \\boxed{...} takes precedence;
otherwise use the last number or fraction;
support thousands separators, decimals, scientific notation, and fractions;
a zero denominator, NaN, infinity, or no answer returns 0;
compare with abs(a-g) <= atol + rtol*max(1, abs(g)).

Add at least eight tests, including two adversarial cases.

Open checks:

Python

assert verify_numeric_answer(r"\\boxed{3/7}", "0.428571") == 1
assert verify_numeric_answer(r"wrong 42, final \\boxed{41}", "42") == 0
assert verify_numeric_answer("1,000", "1000") == 1

Show answer and solution

Python

from fractions import Fraction
import math
import re

NUMBER_PATTERN = re.compile(
    r"-?\d+\s*/\s*-?\d+|-?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?"
)
BOXED_PATTERN = re.compile(r"\\boxed\{([^{}]*)\}")
THOUSANDS_SEPARATOR = re.compile(r"(?<=\d),(?=\d{3}(?:\D|$))")


def extract_numeric_answer(text: str) -> str | None:
    if not isinstance(text, str):
        raise TypeError("text must be a string")
    boxed = BOXED_PATTERN.findall(text)
    if boxed:
        candidate = boxed[-1].strip()
        return candidate or None
    normalized = THOUSANDS_SEPARATOR.sub("", text)
    matches = list(NUMBER_PATTERN.finditer(normalized))
    return matches[-1].group(0).strip() if matches else None


def parse_numeric(value: str) -> float:
    if not isinstance(value, str):
        raise TypeError("value must be a string")
    value = THOUSANDS_SEPARATOR.sub("", value).replace(" ", "").strip()
    if not value:
        raise ValueError("empty numeric value")
    if "/" in value:
        parts = value.split("/")
        if len(parts) != 2:
            raise ValueError("fraction must contain one slash")
        numerator, denominator = (int(part) for part in parts)
        if denominator == 0:
            raise ZeroDivisionError("fraction denominator must be non-zero")
        result = float(Fraction(numerator, denominator))
    else:
        result = float(value)
    if not math.isfinite(result):
        raise ValueError("numeric value must be finite")
    return result


def verify_numeric_answer(
    response: str,
    gold: str,
    rtol: float = 1e-6,
    atol: float = 1e-9,
) -> int:
    if rtol < 0 or atol < 0 or not math.isfinite(rtol) or not math.isfinite(atol):
        raise ValueError("rtol and atol must be finite and non-negative")
    answer = extract_numeric_answer(response)
    if answer is None:
        return 0
    try:
        observed = parse_numeric(answer)
        expected = parse_numeric(gold)
    except (TypeError, ValueError, ZeroDivisionError, OverflowError):
        return 0
    tolerance = atol + rtol * max(1.0, abs(expected))
    return int(abs(observed - expected) <= tolerance)

Checks:

Python

cases = [
    (r"\\boxed{42}", "42", 1),
    ("answer: 1/2", "0.5", 1),
    (r"wrong 42, final \\boxed{41}", "42", 0),
    ("1,000", "1000", 1),
    ("1e3", "1000", 1),
    (r"\\boxed{1/0}", "0", 0),
    ("no answer", "1", 0),
    ("42 items minus 1 = 41", "42", 0),
]
for response, gold, expected in cases:
    assert verify_numeric_answer(response, gold) == expected

The fraction alternative comes before the ordinary-number alternative in the regular expression; otherwise 1/2 could be split into two integers. This verifier deliberately does not attempt symbolic equivalence. Its transparent contract is appropriate only for the stated class of numeric answers.

Group-relative advantages

Implement:

Python

import numpy as np

def group_relative_advantages(
    rewards,
    scale: bool = True,
    eps: float = 1e-8,
) -> np.ndarray:
    ...

Requirements:

use population standard deviation;
scale=False returns centering only;
a degenerate group returns zeros when scale=True;
reject NaN and infinity.

Test the groups [1,0,0,0], [0.9,1.0,0.8,0.9] and [1,1,1,1].

Open checks:

Python

a = group_relative_advantages([1, 0, 0, 0])
expected = [np.sqrt(3), -1 / np.sqrt(3), -1 / np.sqrt(3), -1 / np.sqrt(3)]
assert np.allclose(a, expected, atol=1e-7)
assert np.array_equal(group_relative_advantages([1, 1, 1, 1]), np.zeros(4))

Show answer and solution

Python

import math
import numpy as np


def group_relative_advantages(
    rewards,
    scale: bool = True,
    eps: float = 1e-8,
) -> np.ndarray:
    r = np.asarray(rewards, dtype=float)
    if r.ndim != 1 or r.size == 0 or not np.all(np.isfinite(r)):
        raise ValueError("rewards must be a non-empty finite 1D array")
    if eps < 0 or not math.isfinite(eps):
        raise ValueError("eps must be finite and non-negative")
    centered = r - r.mean()
    if not scale:
        return centered
    std = float(r.std(ddof=0))
    if std == 0.0:
        return np.zeros_like(r)
    return centered / (std + eps)

Python

assert np.allclose(
    group_relative_advantages([1, 0, 0, 0]),
    [np.sqrt(3), -1/np.sqrt(3), -1/np.sqrt(3), -1/np.sqrt(3)],
    atol=1e-7,
)
assert np.allclose(
    group_relative_advantages([0.9, 1.0, 0.8, 0.9], scale=False),
    [0.0, 0.1, -0.1, 0.0],
)
assert np.array_equal(group_relative_advantages([1, 1, 1, 1]), np.zeros(4))

The zero result for a degenerate group comes from the zero centered numerator. We do not divide numerical noise by eps and manufacture a relative signal when every reward is equal.

Asymmetric clipping

Implement token-level surrogate terms and a mask for the active clipped branch:

Python

import numpy as np

def clipped_terms(
    ratios,
    advantages,
    eps_low: float = 0.2,
    eps_high: float | None = None,
) -> np.ndarray:
    ...


def clip_activity(
    ratios,
    advantages,
    eps_low: float = 0.2,
    eps_high: float | None = None,
) -> np.ndarray:
    ...

Compare symmetric clipping 0.2/0.2 and Clip-Higher 0.2/0.28 on

text

ratios     = [1.50, 1.25, 0.70, 1.10]
advantages = [1.00, 1.00,-1.00, 1.00]

Open checks:

Python

out = clipped_terms([1.5, 0.7, 1.1], [1.0, -1.0, -1.0], eps_low=0.2)
assert np.allclose(out, [1.2, -0.8, -1.1])
active = clip_activity([1.5, 0.7, 1.1], [1.0, -1.0, -1.0], eps_low=0.2)
assert np.array_equal(active, [True, True, False])

Show answer and solution

Python

import math
import numpy as np


def _paired_vectors(ratios, advantages):
    rho = np.asarray(ratios, dtype=float)
    adv = np.asarray(advantages, dtype=float)
    if rho.ndim != 1 or adv.shape != rho.shape or rho.size == 0:
        raise ValueError("ratios and advantages must be equal non-empty vectors")
    if not np.all(np.isfinite(rho)) or not np.all(np.isfinite(adv)):
        raise ValueError("ratios and advantages must be finite")
    if np.any(rho <= 0):
        raise ValueError("ratios must be strictly positive")
    return rho, adv


def clipped_terms(ratios, advantages, eps_low=0.2, eps_high=None):
    rho, adv = _paired_vectors(ratios, advantages)
    if eps_high is None:
        eps_high = eps_low
    if eps_low < 0 or eps_high < 0 or not math.isfinite(eps_low) or not math.isfinite(eps_high):
        raise ValueError("clip widths must be finite and non-negative")
    clipped = np.clip(rho, 1.0 - eps_low, 1.0 + eps_high)
    return np.minimum(rho * adv, clipped * adv)


def clip_activity(ratios, advantages, eps_low=0.2, eps_high=None):
    rho, adv = _paired_vectors(ratios, advantages)
    if eps_high is None:
        eps_high = eps_low
    if eps_low < 0 or eps_high < 0:
        raise ValueError("clip widths must be non-negative")
    return ((adv > 0) & (rho > 1.0 + eps_high)) | ((adv < 0) & (rho < 1.0 - eps_low))

Python

ratios = np.array([1.50, 1.25, 0.70, 1.10])
advs = np.array([1.00, 1.00, -1.00, 1.00])
sym = clipped_terms(ratios, advs, eps_low=0.2)
asym = clipped_terms(ratios, advs, eps_low=0.2, eps_high=0.28)
assert np.allclose(sym, [1.2, 1.2, -0.8, 1.1])
assert np.allclose(asym, [1.28, 1.25, -0.8, 1.1])
assert np.array_equal(
    clip_activity(ratios, advs, eps_low=0.2, eps_high=0.28),
    [True, False, True, False],
)

Clip-Higher releases the second positive term but does not change the lower bound for a negative advantage. The mask marks positions where the minimum selected the constant clipped branch, so the local derivative with respect to the ratio is zero.

Three normalizers

For one reward/length batch, return token coefficients and total response contributions for:

original GRPO;
Dr. GRPO with fixed length $L$ ;
DAPO with a shared active-token denominator.

Test rewards=[1,0], lengths=[10,1000], and L=1000. Explain which reduction equalizes response totals and which makes them proportional to length.

Signatures:

Python

import numpy as np

def original_grpo_token_weights(rewards, lengths, *, eps: float = 1e-8) -> list[np.ndarray]: ...
def dr_grpo_token_weights(rewards, lengths, *, fixed_length: int) -> list[np.ndarray]: ...
def dapo_token_weights(rewards, lengths, *, scale_rewards: bool = True) -> list[np.ndarray]: ...
def response_contributions(token_weights) -> np.ndarray: ...

Open checks:

Python

rewards, lengths = [1.0, 0.0], [10, 1000]
grpo = response_contributions(original_grpo_token_weights(rewards, lengths))
dr = response_contributions(dr_grpo_token_weights(rewards, lengths, fixed_length=1000))
dapo = response_contributions(dapo_token_weights(rewards, lengths))
assert np.allclose(grpo, [0.5, -0.5])
assert abs(dr[0] - 0.0025) < 1e-9
assert abs(dapo[0] - 0.009901) < 1e-6

Show answer and solution

Python

import math
import numpy as np


def _rewards_and_lengths(rewards, lengths):
    r = np.asarray(rewards, dtype=float)
    lens = np.asarray(lengths, dtype=int)
    if r.ndim != 1 or r.size == 0 or not np.all(np.isfinite(r)):
        raise ValueError("rewards must be a non-empty finite vector")
    if lens.ndim != 1 or lens.shape != r.shape or np.any(lens <= 0):
        raise ValueError("lengths must be positive and match rewards")
    return r, lens


def _advantages(rewards, *, scale, eps=1e-8):
    centered = rewards - rewards.mean()
    if not scale:
        return centered
    if eps < 0 or not math.isfinite(eps):
        raise ValueError("eps must be finite and non-negative")
    std = float(rewards.std(ddof=0))
    return np.zeros_like(rewards) if std == 0.0 else centered / (std + eps)


def original_grpo_token_weights(rewards, lengths, *, eps=1e-8):
    r, lens = _rewards_and_lengths(rewards, lengths)
    adv = _advantages(r, scale=True, eps=eps)
    group_size = r.size
    return [
        np.full(int(length), a / (group_size * length), dtype=float)
        for a, length in zip(adv, lens)
    ]


def dr_grpo_token_weights(rewards, lengths, *, fixed_length):
    r, lens = _rewards_and_lengths(rewards, lengths)
    if fixed_length <= 0:
        raise ValueError("fixed_length must be positive")
    adv = _advantages(r, scale=False)
    group_size = r.size
    return [
        np.full(int(length), a / (group_size * fixed_length), dtype=float)
        for a, length in zip(adv, lens)
    ]


def dapo_token_weights(rewards, lengths, *, scale_rewards=True):
    r, lens = _rewards_and_lengths(rewards, lengths)
    adv = _advantages(r, scale=scale_rewards)
    total_active_tokens = int(lens.sum())
    return [
        np.full(int(length), a / total_active_tokens, dtype=float)
        for a, length in zip(adv, lens)
    ]


def response_contributions(token_weights):
    if not token_weights:
        raise ValueError("token_weights must be non-empty")
    return np.asarray([np.asarray(w, dtype=float).sum() for w in token_weights])

Python

rewards, lengths = [1.0, 0.0], [10, 1000]
assert np.allclose(
    response_contributions(original_grpo_token_weights(rewards, lengths)),
    [0.5, -0.5],
)
assert np.allclose(
    response_contributions(dr_grpo_token_weights(rewards, lengths, fixed_length=1000)),
    [0.0025, -0.25],
)
assert np.allclose(
    response_contributions(dapo_token_weights(rewards, lengths)),
    [10/1010, -1000/1010],
    atol=1e-8,
)

Original GRPO divides each response by its own length and therefore produces equal-magnitude response totals. In the shown Dr. GRPO and DAPO reductions, a token coefficient is not divided by that response’s length, so the total contribution becomes proportional to the number of active tokens. Their absolute scales differ because the global denominators and reward normalization differ.

GSPO sequence ratio and clipping

Implement:

Python

import numpy as np

def sequence_importance_ratio(new_logp, old_logp, mask=None) -> float:
    ...


def gspo_surrogate_term(
    new_logp,
    old_logp,
    advantage: float,
    eps_low: float = 3e-4,
    eps_high: float = 4e-4,
    mask=None,
) -> float:
    ...

Compute the first function as the exponential of the mean active log-ratio. Then use that ratio in the clipped sequence-level term. Tests must include padding masks and a very long sequence so probabilities are never multiplied directly.

Open checks:

Python

new = np.array([-0.4, -0.6])
old = np.array([-0.5, -0.5])
mask = np.array([1, 1])
ratio = sequence_importance_ratio(new, old, mask)
assert abs(ratio - 1.0) < 1e-12
assert abs(sequence_importance_ratio(old, old, mask) - 1.0) < 1e-12
assert abs(gspo_surrogate_term(new, old, 1.0, mask=mask) - 1.0) < 1e-12

Show answer and solution

Python

import math
import numpy as np


def _active_log_ratio(new_logp, old_logp, mask=None):
    new = np.asarray(new_logp, dtype=float)
    old = np.asarray(old_logp, dtype=float)
    if new.ndim != 1 or old.shape != new.shape or new.size == 0:
        raise ValueError("new_logp and old_logp must be equal non-empty vectors")
    if not np.all(np.isfinite(new)) or not np.all(np.isfinite(old)):
        raise ValueError("log-probabilities must be finite")
    active = np.ones_like(new, dtype=bool)
    if mask is not None:
        m = np.asarray(mask)
        if m.shape != new.shape or np.any((m != 0) & (m != 1)):
            raise ValueError("mask must match and contain only 0/1")
        active = m.astype(bool)
    if not active.any():
        raise ValueError("at least one token must be active")
    return (new - old)[active]


def sequence_importance_ratio(new_logp, old_logp, mask=None):
    mean_log_ratio = float(_active_log_ratio(new_logp, old_logp, mask).mean())
    return float(np.exp(mean_log_ratio))


def gspo_surrogate_term(
    new_logp,
    old_logp,
    advantage,
    eps_low=3e-4,
    eps_high=4e-4,
    mask=None,
):
    if not math.isfinite(advantage):
        raise ValueError("advantage must be finite")
    if eps_low < 0 or eps_high < 0:
        raise ValueError("clip widths must be non-negative")
    ratio = sequence_importance_ratio(new_logp, old_logp, mask)
    clipped = float(np.clip(ratio, 1.0 - eps_low, 1.0 + eps_high))
    return float(min(ratio * advantage, clipped * advantage))

Python

old = np.log([0.20, 0.40, 0.50, 0.30])
new = old + np.log([1.01, 0.99, 1.02, 10.0])
ratio = sequence_importance_ratio(new, old, mask=[1, 1, 1, 0])
expected = (1.01 * 0.99 * 1.02) ** (1 / 3)
assert np.isclose(ratio, expected)

# A long sequence remains numerically stable because log-ratios are summed.
long_old = np.full(100_000, -20.0)
long_new = long_old + 1e-4
assert np.isclose(sequence_importance_ratio(long_new, long_old), np.exp(1e-4))

The calculation uses a mean log-ratio and one exponential, avoiding the numerical failure of directly multiplying thousands of small probabilities.

Minimum rollout record

Create a data structure for one response with:

prompt_id, group_id;
token_ids, action_mask, old_logprobs;
reward components;
stop reason;
generator version.

Write a validator for aligned lengths, finite numbers, at least one active action, and an allowed stop reason. Add tests for a valid record, an empty mask, mismatched lengths, and NaN log-probabilities.

Starter contract:

Python

from dataclasses import dataclass

@dataclass(frozen=True)
class RolloutRecord:
    prompt_id: str
    group_id: str
    token_ids: tuple[int, ...]
    action_mask: tuple[int, ...]
    old_logprobs: tuple[float, ...]
    reward_components: tuple[float, ...]
    stop_reason: str
    generator_version: str


def validate_rollout_record(record: RolloutRecord) -> None:
    ...

This extends earlier work rather than repeating it. Adapt the batch validator from Module 6 Task C6 (validate_rollout_batch) to one record instead of a padded batch. Only the environment and stop-reason fields are new; the alignment and finiteness checks carry over.

Open checks:

Python

record = RolloutRecord(
    prompt_id="q1", group_id="g1", token_ids=(1, 2, 3), action_mask=(0, 1, 1),
    old_logprobs=(0.0, -0.2, -0.3), reward_components=(1.0,),
    stop_reason="eos", generator_version="v1",
)
assert validate_rollout_record(record) is None
empty_mask = RolloutRecord(
    prompt_id="q1", group_id="g1", token_ids=(1, 2, 3), action_mask=(0, 0, 0),
    old_logprobs=(0.0, -0.2, -0.3), reward_components=(1.0,),
    stop_reason="eos", generator_version="v1",
)
try:
    validate_rollout_record(empty_mask)
    raise AssertionError("an empty mask must be rejected")
except ValueError:
    pass

Show answer and solution

Python

from dataclasses import dataclass
import math


@dataclass(frozen=True)
class RolloutRecord:
    prompt_id: str
    group_id: str
    token_ids: tuple[int, ...]
    action_mask: tuple[int, ...]
    old_logprobs: tuple[float, ...]
    reward_components: tuple[float, ...]
    stop_reason: str
    generator_version: str


def validate_rollout_record(record: RolloutRecord) -> None:
    if not record.prompt_id or not record.group_id or not record.generator_version:
        raise ValueError("prompt_id, group_id, and generator_version must be non-empty")
    n = len(record.token_ids)
    if n == 0 or len(record.action_mask) != n or len(record.old_logprobs) != n:
        raise ValueError("token_ids, action_mask, and old_logprobs must align")
    if any(m not in (0, 1) for m in record.action_mask) or not any(record.action_mask):
        raise ValueError("action_mask must contain 0/1 and at least one active action")
    if not all(math.isfinite(x) for x in record.old_logprobs):
        raise ValueError("old_logprobs must be finite")
    if not record.reward_components or not all(math.isfinite(x) for x in record.reward_components):
        raise ValueError("reward_components must be a non-empty finite sequence")
    if record.stop_reason not in {"eos", "stop_string", "truncated", "environment", "error"}:
        raise ValueError("unsupported stop_reason")

Python

valid = RolloutRecord(
    prompt_id="p-17",
    group_id="g-4",
    token_ids=(101, 102, 103),
    action_mask=(0, 1, 1),
    old_logprobs=(-2.0, -0.4, -0.8),
    reward_components=(1.0, 0.1),
    stop_reason="eos",
    generator_version="checkpoint-1200",
)
validate_rollout_record(valid)

for invalid in (
    RolloutRecord("p", "g", (1, 2), (0, 0), (-1.0, -1.0), (1.0,), "eos", "v1"),
    RolloutRecord("p", "g", (1, 2), (1,), (-1.0, -1.0), (1.0,), "eos", "v1"),
    RolloutRecord("p", "g", (1,), (1,), (float("nan"),), (1.0,), "eos", "v1"),
):
    try:
        validate_rollout_record(invalid)
        raise AssertionError("invalid record was accepted")
    except ValueError:
        pass

The validator checks structural integrity but cannot prove that old_logprobs came from the actual behavior policy. That requires a replayable generation protocol and policy-version verification.