Module 5. Learning from fixed data: imitation, deep Q-methods, and offline RL

1. Yes, under the token-level abstraction. State $s_t=(x,y_{<t})$, action $a_t=y_t$, and the mask is one only on response tokens.

2. It remains sequence maximum likelihood, but clones a broader process. User tokens are also treated as predicted actions, so it is not simply BC of the assistant policy without a qualification.

3. Yes, with matched masks and reduction. At the whole-response level the state is $x$ and the action is $y$. The identity $-\log\pi_\theta(y\mid x)=-\sum_t\log\pi_\theta(y_t\mid x,y_{<t})$ makes summed response-token NLL exactly the BC objective for that action, including EOS under the same convention. Backpropagation through the factorization does not change the objective. Averaging each response by its own length changes example weights; a global normalization by a fixed token count only rescales the batch objective.

4. Yes. Multiple demonstrations define an empirical distribution of expert actions, so maximizing their likelihood remains BC even when the expert is stochastic.

The unqualified slogan hides the unit of action, state construction, and mask—the elements that specify which policy is cloned.

1. The 1% error is measured under $d_t^{\pi_E}$. The learned policy can enter states rare under the expert, where its error rate differs.

2. The distributions may stay close when errors are rare and recoverable, dynamics are stable, recovery demonstrations are present, or the model generalizes well nearby.

3. Recoverable: one unnecessary move in a corridor. Irreversible: falling from a cliff or committing an irreversible transaction.

4. Evaluate closed-loop behavior: expected return, success probability, failure cost, recovery quality, or another metric on trajectories generated by the policy itself.

1. DAgger: the expert supplies a target action for a visited state.

2. Preference learning: the human supplies a pairwise comparison.

3. RL with verifiable reward: a program supplies scalar outcome feedback.

All three can gather feedback on current-policy behavior rather than only static demonstrations. Action labels, pairwise preferences, and scalar rewards nevertheless live in different observation spaces and induce different losses and estimators.

1. The target policy can differ from the behavior policy that generated transitions.

2. Standard DQN continues interacting and adding new transitions; its dataset is not fixed for the whole run.

3. The new region enters replay and can correct previous value estimates with actual outcomes.

4. In offline RL, an unsupported-action hypothesis cannot be tested. A value error can be selected and bootstrapped without corrective experience.

The DQN target network supplies the next-state value in a Bellman target, directly participates in bootstrapping, and is periodically copied or slowly updated.

The RLHF reference policy defines a distributional anchor for a KL penalty or likelihood ratio. It is not an estimate of future return and is usually fixed within a run.

Both may be copies, but one stabilizes a moving value target while the other defines the cost of deviation from reference behavior.

1. World A: $r(a_0)=0$, $r(a_1)=+1$. World B: $r(a_0)=0$, $r(a_1)=-1$.

2. The datasets are identical, so an algorithm receives the same input and must return the same output. No single action is optimal in both worlds.

3. Examples include reward smoothness, a known dynamics model, structural relations among actions, a causal model, or new interaction.

4. More capacity represents more candidate functions; it does not make the observations distinguish between two compatible worlds.

• BCQ → 1. Excessive restriction may exclude a rare but valuable improvement.
• TD3+BC → 2. An overly strong BC term reduces the method to cloning.
• CQL → 3. Excessive conservatism may systematically undervalue useful actions.
• IQL → 4. Poor expectile, temperature, or weight clipping may either prevent improvement or concentrate the policy too narrowly.

• DT: learns a conditional action distribution from history and return-to-go; no explicit Bellman bootstrap.
• DPO: learns a policy from pairwise preferences relative to a reference policy; no Q-function or Bellman target.
• ILQL: learns token-level $Q$ and $V$ from rewarded trajectories and uses sequential value recursion before advantage-weighted policy extraction.

Fixed data specify access to experience, not a unique objective. Demonstrations, preferences, and rewarded transitions support different counterfactual questions.

The active entries are $-0.2$, $-1.0$, and $-0.4$:

Dividing by all six positions gives $0.266667$ and makes gradient scale depend on how many masked positions are present. The usual token mean divides by active targets.

With $q=0.98$,

The first-order approximation is

and normalized regret is

Here $\varepsilon T=0.4$ is not very small, so higher-order terms are visible.

The maxima are $(4,100,3)$, hence

The second transition is terminal, so continuation value is masked out.

Standard DQN uses target-network maxima $10$ and $7$:

The online network selects actions $(1,0)$. The target network values those actions as $(1,-1)$, so

The example deliberately makes the rankings disagree: one network selects; the other evaluates.

At $\tau=0.8$, negative residuals have weight $0.2$ and positive residuals weight $0.8$:

The asymmetric weighting moves $V$ toward an upper expectile of values represented in the data.

Starting from 5, the undiscounted target sequence is

For $\gamma=0.5$,

The target is an input condition. If the dataset does not support that return, the model extrapolates; the condition does not alter environment dynamics or guarantee the outcome.

import numpy as np

def dqn_targets(rewards, terminated, next_q_target, gamma=0.99):
    r = np.asarray(rewards, dtype=float)
    term = np.asarray(terminated, dtype=bool)
    q = np.asarray(next_q_target, dtype=float)
    return r + gamma * (~term) * q.max(axis=1)

def double_dqn_targets(rewards, terminated, next_q_online, next_q_target, gamma=0.99):
    r = np.asarray(rewards, dtype=float)
    term = np.asarray(terminated, dtype=bool)
    online = np.asarray(next_q_online, dtype=float)
    target = np.asarray(next_q_target, dtype=float)
    actions = online.argmax(axis=1)
    selected = target[np.arange(len(r)), actions]
    return r + gamma * (~term) * selected

Production code should additionally validate shapes and $\gamma$; the reference module does so.

import numpy as np

def pessimistic_values(
    empirical_means: np.ndarray,
    counts: np.ndarray,
    *,
    radius_scale: float = 1.0,
) -> np.ndarray:
    means = np.asarray(empirical_means, dtype=np.float64)
    n = np.asarray(counts, dtype=np.float64)
    if means.shape != n.shape or np.any(n < 0) or not np.isfinite(radius_scale) or radius_scale < 0:
        raise ValueError("invalid shapes, counts, or radius_scale")
    out = np.full_like(means, -np.inf)
    seen = n > 0
    out[seen] = means[seen] - radius_scale / np.sqrt(n[seen])
    return out

This ignores sequential dynamics, representation error, systematic collection bias, and statistically calibrated confidence bounds. It illustrates support, not a general algorithm.