RL for LLM5 / 13

Module 5. Learning from fixed data: imitation, deep Q-methods, and offline RL

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 5 · Correct: 0

Question P01 · Terminal transition versus external truncation

A transition has reward $r=0.2$ , $\gamma=0.95$ , and $\max_{a'}Q_{\theta^-}(s',a')=1.6$ .

Compute the DQN target after a true terminal trap.
Compute the target when collection stops at an external time limit, $s'$ is observed, and the task could continue.
Explain why a single done=True flag can corrupt learning.

TerminalAbsolute tolerance: ±0.000001External truncationAbsolute tolerance: ±0.000001

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

Answer and explanation

A true terminal disables bootstrapping:

y_{\mathrm{terminal}}=0.2.

An external truncation retains continuation value when $s'$ is available:

y_{\mathrm{truncated}}=0.2+0.95\cdot1.6=1.72.

Collapsing both cases into done=True assigns zero continuation value to a technical boundary. That creates an artificial terminal and systematically depresses values near the limit. If the horizon is part of the task itself, the remaining budget must instead be represented in the state.

Question P02 · Map the support of a fixed log

At state $s$ , a log contains 900 left transitions, 100 up transitions, and no right or down transitions. A Q-network outputs

Q(s,\cdot)=(0.4,\ 0.3,\ 2.8,\ 2.1)

in the order left, up, right, down.

Which actions are empirically supported? What does a naive greedy policy choose? What extra assumption would be required to trust right?

Your answer or notes

Answer and explanation

left and up are supported by the log; right and down are unobserved. A naive greedy policy chooses right because $2.8$ is largest, but that number is entirely an extrapolation of the function approximator.

Trust would require explicit structure: known smoothness of dynamics and features, a validated transition model, symmetry, or external data connecting the action to observed behavior. A large neural output is not such an assumption.

Question P03 · Replay memory is not a fixed offline dataset

Two systems train from one million transitions.

System A keeps interacting, evicts old replay entries, and adds new ones.
System B receives one immutable file and never runs the environment while training.

Both use the same DQN target. Which system is offline RL, and why does the common local loss not make the regimes equivalent?

Your answer or notes

Answer and explanation

System B is offline RL because its data are fixed: the policy cannot test a new action or repair missing coverage. System A is ordinary off-policy online RL. Its mini-batch may come from older policies, but the future data stream still responds to the current agent.

A local loss does not define the statistical regime. Data provenance, exploration, and feedback between the learned policy and future observations are decisive.

Question P04 · What feedback standard DAgger requires

For each signal, decide whether it is sufficient for a standard DAgger iteration:

Why does the phrase “feedback on current-policy behavior” not make all four algorithms identical?

One correct option1the expert gives the correct action in each learner-visited state;2a person picks the better of two complete episodes;3a verifier returns an episode-level score;4a reward model scores a response.

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

the expert gives the correct action in each learner-visited state;

a person picks the better of two complete episodes;

a verifier returns an episode-level score;

a reward model scores a response.

Answer and explanation

Only 1 supplies the action label required by standard DAgger, so the pair $(s,\pi_E(s))$ can be added to supervised training.

Signals 2–4 are preferences or rewards. They can support reward modeling, policy gradients, search, or other objectives, but not the standard DAgger reduction without an additional mechanism that recovers expert actions.

The outer feedback loop is similar; the observation format determines the statistical problem and admissible loss.

Question P05 · Find the sign error in a CQL penalty

The function below operates on one state. q_values contains all action values and logged_action is the dataset action.

import numpy as np

def cql_penalty_bad(q_values: np.ndarray, logged_action: int) -> float:
    q = np.asarray(q_values, dtype=float)
    return float(np.log(np.exp(q).sum()) + q[logged_action])

Find the sign error.
Make the computation numerically stable.
Explain why this term alone is not a full CQL algorithm.

Your answer or notes

Answer and explanation

The discrete penalty is

y_{\mathrm{terminal}}=0.2.

so the logged value must be subtracted. A stable implementation is:

import numpy as np

def cql_penalty(q_values: np.ndarray, logged_action: int) -> float:
    q = np.asarray(q_values, dtype=float)
    if q.ndim != 1 or not 0 <= logged_action < q.size:
        raise ValueError("invalid q_values or action")
    m = float(q.max())
    logsumexp = m + float(np.log(np.exp(q - m).sum()))
    return logsumexp - float(q[logged_action])

Full CQL also includes Bellman regression, the action distribution used in the conservative term, a coefficient, a target network, optimization, and assumptions. The function computes only one local component.

Question P06 · An expectile above the mean

For one state, dataset-action values are $Q=(0,1,4)$ . Let $V=1$ and $\tau=0.8$ .

Compute the weighted squared residuals.
Which way will $V$ move?
Why is this not the same as taking the maximum 4?

Your answer or notes

Answer and explanation

Residuals $u=Q-V$ are $(-1,0,3)$ . A negative residual has weight $1-\tau=0.2$ , a positive residual has weight $\tau=0.8$ , and the zero residual contributes nothing. The terms are $0.2$ , $0$ , and $0.8\cdot9=7.2$ .

The large positive residual dominates, so optimization raises $V$ . An expectile is a smooth functional of the entire dataset-action value distribution. For finite $\tau<1$ it remains distinct from the maximum and depends on observation mass, whereas a max uses only the largest value.

Question P07 · One FQE target

At the next state, the evaluation policy chooses two actions with probabilities $(0.25,0.75)$ . The target Q-model outputs $(2,4)$ . The transition has $r=-0.1$ , $\gamma=0.9$ , and is non-terminal.

Compute the FQE target.
Compare it with a Q-learning target.
What changes at a true terminal?

FQE targetAbsolute tolerance: ±0.000001Q-learning targetAbsolute tolerance: ±0.000001Terminal targetAbsolute tolerance: ±0.000001

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

Answer and explanation

The target-policy continuation value is

y_{\mathrm{terminal}}=0.2.

Thus

y^{\mathrm{FQE}}=-0.1+0.9\cdot3.5=3.05.

Q-learning would use the maximum 4 and produce $-0.1+0.9\cdot4=3.5$ . FQE evaluates a fixed action distribution rather than improving it. At a true terminal, continuation is removed and both targets equal $r=-0.1$ .

Question P08 · A Decision Transformer target outside the data

Training return-to-go values lie between $-1$ and $1$ . Deployment requests $G_1^{\mathrm{target}}=20$ .

Name three possible outcomes. Which one is guaranteed by the DT training loss, and how should the regime be tested?

Your answer or notes

Answer and explanation

The model may ignore the unfamiliar condition and reproduce ordinary behavior, extrapolate into a bad region, or happen to generate a strong trajectory. The supervised loss guarantees none of these outside the training support.

Compare the requested value with the training range; evaluate actual return across several conditions inside and near support; use multiple seeds/episodes; and inspect failure rates as well as means. A return condition is an input, not a feasibility certificate.

Question P09 · Causal leakage in Decision Transformer

A preprocessing pipeline feeds current return-to-go, state, the current correct action, and asks the model to predict that same action.

Why is training accuracy misleading? What causal shift is required?

Your answer or notes

Answer and explanation

The current action directly reveals the label. The model can copy it without learning a deployable policy, so high accuracy measures leakage.

To predict $a_t$ , the model may use return conditions and states through $s_t$ , but only previous actions $a_{<t}$ . Logits and action targets must be shifted causally, just as logits and next-token labels are shifted in language modeling.

Question P10 · Match data to method

Match each dataset to the most direct baseline objective: SFT/BC, classical offline RL, DPO, or ILQL.

a prompt and one demonstration response without reward;
preferred/rejected response pairs;
prefix–token sequences with an outcome reward;
environment transitions $(s,a,r,s')$ .

For each, name one object absent from the other settings.

Match each item to an answer1234

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

BC/SFT

Classical offline RL

ILQL

Answer and explanation

1 → SFT/BC: direct demonstrator action labels, with no separate reward.
2 → DPO: a pairwise preference relation and a reference-policy likelihood ratio.
3 → ILQL: token transitions with reward/value information for learning $Q$ and $V$ .
4 → classical offline RL: explicit next-state dynamics and transition reward.

Real datasets can combine signals, but “offline” alone is insufficient to choose an algorithm.

Question P11 · Hidden tests for a behavior-support mask

A student implements

def behavior_support_mask(probabilities, threshold):
    """Return a Boolean mask for actions whose behavior probability is supported."""

Propose at least six hidden tests that reject return probabilities > 0 and constant masks.

Your answer or notes

Answer and explanation

Useful tests include:

a mixed vector with values above and below the threshold;
equality at the threshold, with an explicit >= or > contract;
zero probabilities;
thresholds 0 and 1;
invalid shapes, NaN, negatives, or a sum far from one according to the declared input contract;
Boolean dtype and shape preservation;
permutation equivariance;
all-supported and none-supported examples;
no mutation of the input.

probabilities > 0 fails on small positive probabilities below threshold. Constant masks fail on paired examples with opposite expected outcomes.

Question P12 · Report an internal experiment without overclaiming

In one pinned run, naive offline DQN solved no episodes while CQL and IQL solved all episodes. Choose the defensible statement and reject the others.

One correct option1“CQL and IQL are proven better than offline DQN.”2“In this configuration, naive offline DQN exploited out-of-data error while two protected objectives learned successful policies; broader comparison requires other seeds, logs, and hyperparameters.”3“Offline DQN always diverges.”4“All pessimistic methods are equivalent.”

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

“CQL and IQL are proven better than offline DQN.”

“In this configuration, naive offline DQN exploited out-of-data error while two protected objectives learned successful policies; broader comparison requires other seeds, logs, and hyperparameters.”

“Offline DQN always diverges.”

“All pessimistic methods are equivalent.”

Answer and explanation

B is defensible because it reports the observation, a plausible mechanism, and the boundary of inference.

A turns one run into a theorem. C is false in general: offline DQN can work with adequate coverage and approximation. D erases differences among action restriction, conservative value learning, and expectile/advantage-weighted methods. A teaching plot should report seed, log size, training budget, and that it is not an external benchmark.