RL for LLM1 / 13

Module 1. Introduction: Reinforcement Learning for LLMs

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 9 · Correct: 0

Question P01 · The limits of "an LLM is a policy"

Step 1.1.4 — An LLM as a policy: the exact correspondence and its boundary

single answer

In the token formulation $s_t=(x,y_{<t})$ , $a_t=y_t$ , and an LLM genuinely defines $\pi_\theta(a\mid s)$ . What caveat does the module attach to that statement?

One correct option1The correspondence is approximate: softmax merely resembles a policy2Calling an LLM a policy describes how actions are selected; it does not claim the weights came from RL — a pretrained model already defines a policy3The correspondence holds only after RLHF fine-tuning4The correspondence requires a fully observable environment

The correspondence is approximate: softmax merely resembles a policy

Calling an LLM a policy describes how actions are selected; it does not claim the weights came from RL — a pretrained model already defines a policy

The correspondence holds only after RLHF fine-tuning

The correspondence requires a fully observable environment

Answer and explanation

Correct answer: Calling an LLM a policy describes how actions are selected; it does not claim the weights came from RL — a pretrained model already defines a policy

The second limit: in a deployed system an LLM is usually the policy or part of it, not the whole environment — tools, files, the browser and the user carry their own state.

Question P02 · Four questions instead of a label

Step 1.1.5 — Reward, return, discounting, and the RL objective

single answer

The module rejects "supervised learning has a right answer, RL only has a reward". What does it offer instead?

One correct option1A precise definition via the presence of a value function2Four questions: what decision the policy makes, what feedback is available, whether actions have delayed consequences, and how the data was obtained3A split into online and offline regimes4A check of whether an expectation over current-policy trajectories is being optimized

A precise definition via the presence of a value function

Four questions: what decision the policy makes, what feedback is available, whether actions have delayed consequences, and how the data was obtained

A split into online and offline regimes

A check of whether an expectation over current-policy trajectories is being optimized

Answer and explanation

Correct answer: Four questions: what decision the policy makes, what feedback is available, whether actions have delayed consequences, and how the data was obtained

Imitation learning, offline RL, contextual bandits and preference learning are four counterexamples, each breaking the short label in its own way.

Question P03 · What each abstraction hides

Step 1.2.1 — Three useful abstractions: response, token, and macro-action

multiple answers

The three formulations of an LLM task each hide something deliberately. Select every correct pairing.

Select all correct options1The contextual bandit hides per-token credit and decoding cost2The finite token MDP hides external hidden state and coarse actions3A POMDP or semi-MDP demands an explicit model of time, tool failures and partial observability4The contextual bandit claims the response physically appeared in one step

The contextual bandit hides per-token credit and decoding cost

The finite token MDP hides external hidden state and coarse actions

A POMDP or semi-MDP demands an explicit model of time, tool failures and partial observability

The contextual bandit claims the response physically appeared in one step

Answer and explanation

Correct answer:

The contextual bandit hides per-token credit and decoding cost
The finite token MDP hides external hidden state and coarse actions
A POMDP or semi-MDP demands an explicit model of time, tool failures and partial observability

The last is a common misreading: the internal token sequence is simply left unmodelled, not denied.

Question P04 · What a denser signal does not do

Step 1.2.2 — Sparse reward, credit assignment, and rollout cost

single answer

A value function, a group baseline and process rewards all make a sparse signal more informative. What does none of them do?

One correct option1Reduce the variance of the gradient estimate2Turn a bad reward into the right objective: denser feedback only helps optimize more precisely what is already specified3Work with a terminal reward of

0

1

4Apply to responses longer than a thousand tokens

Reduce the variance of the gradient estimate

Turn a bad reward into the right objective: denser feedback only helps optimize more precisely what is already specified

Work with a terminal reward of $0$ or $1$

Apply to responses longer than a thousand tokens

Answer and explanation

Correct answer: Turn a bad reward into the right objective: denser feedback only helps optimize more precisely what is already specified

This is the course's throughline in its first form: outcome quality starts with the criterion, not with the choice of optimizer.

Question P05 · Offline versus off-policy

Step 1.3.3 — Classical axes and their limits

single answer

Why are "offline" and "off-policy" not synonyms?

One correct option1Offline RL is always off-policy but not conversely2Offline is a setting: no fresh interaction, fixed data. Off-policy is a property of an estimator or algorithm relative to the behavior policy3Off-policy refers to evaluation, offline to critic architecture4The distinction is terminological and does not affect algorithms

Offline RL is always off-policy but not conversely

Offline is a setting: no fresh interaction, fixed data. Off-policy is a property of an estimator or algorithm relative to the behavior policy

Off-policy refers to evaluation, offline to critic architecture

The distinction is terminological and does not affect algorithms

Answer and explanation

Correct answer: Offline is a setting: no fresh interaction, fixed data. Off-policy is a property of an estimator or algorithm relative to the behavior policy

The neighbouring trap on the same step: "on-policy data" does not mean "each sample used exactly once" — PPO runs several epochs over minibatches.

Question P06 · The method map

Step 1.3.5 — SFT, DPO, PPO-RLHF, GRPO/RLVR, and Best-of-N

multiple answers

From the table in step 1.3.5 — which statements are true? Select all that apply.

Select all correct options1SFT is behavior cloning from demonstrations, not RL2DPO is offline preference optimization derived from the KL-regularized RLHF setting; it is not classical offline RL3Best-of-N is search and selection at frozen parameters, outside training4RLVR describes a feedback type, GRPO an algorithm5The absence of a learned critic in canonical GRPO means there is no baseline

SFT is behavior cloning from demonstrations, not RL

DPO is offline preference optimization derived from the KL-regularized RLHF setting; it is not classical offline RL

Best-of-N is search and selection at frozen parameters, outside training

RLVR describes a feedback type, GRPO an algorithm

The absence of a learned critic in canonical GRPO means there is no baseline

Answer and explanation

Correct answer:

SFT is behavior cloning from demonstrations, not RL
DPO is offline preference optimization derived from the KL-regularized RLHF setting; it is not classical offline RL
Best-of-N is search and selection at frozen parameters, outside training
RLVR describes a feedback type, GRPO an algorithm

The last is the most common misreading of the table: the group mean is the baseline.

Question P07 · What the temperature formula describes

Step 1.4.2 — Temperature softmax: a first computable policy

single answer

The formula $\pi_i(T) \propto \exp(z_i/T)$ and the monotonicity of entropy in $T$ are proved for fixed finite logits. Why need the API's temperature parameter not behave the same way?

One correct option1The API uses different units for temperature2In an API it is applied alongside top-p, top-k, repetition penalties and provider filters; the formula describes pure softmax over given logits, not the whole decoding process3Providers round logits to float16, which breaks monotonicity4Monotonicity only holds for

T \le 1

The API uses different units for temperature

In an API it is applied alongside top-p, top-k, repetition penalties and provider filters; the formula describes pure softmax over given logits, not the whole decoding process

Providers round logits to float16, which breaks monotonicity

Monotonicity only holds for $T \le 1$

Answer and explanation

Correct answer: In an API it is applied alongside top-p, top-k, repetition penalties and provider filters; the formula describes pure softmax over given logits, not the whole decoding process

The same gap returns in Module 6 (truncation and renormalization) and Module 3 (generation temperature is not training exploration).

Question P08 · Entropy at a given temperature

Step 1.4.3 — Reproducible experiment: $T\mapsto H(T)$

numeric answer · tolerance ±0.01

Logits $z = (3, 1, 0)$ , temperature $T = 2$ .

What is the policy entropy in nats?

Entropy, natsAbsolute tolerance: ±0.01

Answer and explanation

Correct answer: 0.906

Verified with policy_entropy: $0.905959$ . The ceiling here is $\ln 3 \approx 1.0986$ , so the distribution is already noticeably flattened but not yet uniform.

Question P09 · The exact corridor value under a finite horizon

Step 1.4.4 — Return and Monte Carlo evaluation in a corridor

coding task

A symmetric walk over cells $0..4$ ; start at $2$ ; left and right are equally likely. The episode terminates at $0$ (reward $0$ ) or at $4$ (reward $+1$ ), and is also cut off after max_steps actions.

Specification: compute $V(2) = \Pr(T_4 \le \texttt{max\_steps})$ exactly, by dynamic programming over the remaining step count — no simulation.

Input format: one line — max_steps. Output format: one line — the value (6 decimals).

Starter code:

def exact_corridor_value(max_steps: int) -> float:
    """Probability of reaching state 4 from state 2 within max_steps actions."""
    # YOUR CODE HERE
    raise NotImplementedError

if __name__ == "__main__":
    print(f"{exact_corridor_value(int(input())):.6f}")

Open tests:

stdin	stdout
`2`	`0.250000`
`4`	`0.375000`
`6`	`0.437500`
`20`	`0.499512`

Your answer or notes

Answer and explanation

Without a length cap the answer would be exactly $0.5$ ; with a cap of 20 it is $1023/2048 = 0.49951171875$ . The gap is small but the principle is not: an experiment must be compared against the formulation implemented in the code, not a nearly identical uncapped one.

Reference dynamic-programming implementation:

import numpy as np


def exact_corridor_value(max_steps: int) -> float:
    if not isinstance(max_steps, int) or isinstance(max_steps, bool) or max_steps <= 0:
        raise ValueError("max_steps must be a positive integer")

    # values[s] is the probability of reaching the right terminal
    # within the currently allowed number of actions when starting from s.
    values = np.zeros(5, dtype=np.float64)
    values[4] = 1.0
    for _ in range(max_steps):
        updated = values.copy()
        for state in (1, 2, 3):
            updated[state] = 0.5 * values[state - 1] + 0.5 * values[state + 1]
        updated[0] = 0.0
        updated[4] = 1.0
        values = updated
    return float(values[2])

The running time is $O(\texttt{max\_steps})$ and the memory cost is $O(1)$ because the state space is fixed.

Question P10 · Standard error of the Monte Carlo estimate

Step 1.4.4 — Return and Monte Carlo evaluation in a corridor

numeric answer · tolerance ±0.002

The corridor return is a Bernoulli variable with $p = 1023/2048 \approx 0.4995$ . The estimate uses $N = 400$ independent episodes.

What is the approximate standard error $\sqrt{p(1-p)/N}$ ?

Standard errorAbsolute tolerance: ±0.002

Answer and explanation

Correct answer: 0.025

At $N = 1000$ it would be $\approx 0.0158$ . Estimates of 0.48 or 0.52 at this $N$ do not contradict the theory — that is ordinary finite-sample error. The same logic returns in RLVR: group size drives variance and cost at once.