Module 8. RLHF and direct preference optimization

The updated policy generates responses, the frozen reference anchors KL, the reward model scores responses, and the critic estimates future return. These are roles, not four mandatory independent full copies. Sharing a policy/value backbone, LoRA, quantizing frozen models, caching reference log-probabilities, CPU offload, process separation, and smaller reward/value models can reduce memory. Fresh generation, forward passes, rollout storage, value estimation, and policy backpropagation remain.

The penalty is a Lagrangian form of the constraint. In ideal convex optimization an active constraint corresponds to a multiplier. Neural optimization is finite, stochastic, and non-convex, so selected $\beta$ does not predetermine realized KL. KL prices movement and preserves support; it does not validate the reward model or preserve every capability.

The value $-0.7$ is one sampled log-ratio and may be negative. Its expectation under the current policy is forward KL and is non-negative. Exact categorical KL sums over the vocabulary; a sampled sequence sum follows one response and may also be negative. Masks define policy actions, and the actual decoding distribution defines the correct probability after temperature and truncation.

The derivation uses same-prompt pairs, Bradley–Terry likelihood, the exact KL-regularized optimum, a fixed reference, compatible support, and consistent preprocessing. Labels must represent the intended rubric. If Bradley–Terry is misspecified, DPO remains a valid pairwise loss, but exact implicit-reward recovery becomes only an approximation.

DPO consumes prompt–completion pairs and uses no explicit Q-function, Bellman target, or bootstrapping. Classical offline RL typically consumes transitions or trajectories. Both face coverage risk, but offline RL may amplify value errors through bootstrapping while DPO may exploit limited pairs, length, style, or unlabelled alternatives. DPO is related fixed-data policy optimization, not literally classical value-based offline RL.

No. Both paired probabilities can fall while the rejected one falls faster and a third completion absorbs mass. Monitor chosen/rejected log-probabilities, separate reference ratios, margin, KL, lengths, gradient weights, and fresh-generation quality.

The faults are a length shortcut, asymmetric truncation, and stale reference log-probabilities. Preserve raw text, token IDs, masks, pre/post-truncation lengths, stop reason, tokenizer/template/checkpoint versions, decoding settings, provenance, and preprocessing version. Dividing by length changes the objective and does not repair stale caches, truncation, or bad labels.

Use DPO for good pairs with a reference; KTO for unpaired binary labels; ORPO for a one-stage SFT-plus-pair objective; IPO/cDPO/early stopping for runaway margins or label noise; RLVR with an online update for verifiable fresh rewards; and SimPO when explicit length-normalized scoring is desired. Each still requires independent quality and method-specific diagnostics.

A fixed sum $R-\alpha C$ commits to one price of risk. If the cost scale or violation rate changes, the objective does not require the policy to return to a declared budget. A constrained objective states $\mathbb E[C]\le d$, while projected dual ascent adapts the price of violation:

The multiplier rises above budget and falls when the constraint has slack. Reward and cost signals may use different data, features, failure modes, and thresholds, so they should remain separately auditable.

An average constraint still permits rare severe violations. Critical actions need hard filters or safe action sets, isolated tools, refusal or abort paths, and scenario-level checks. Module 13's release gate remains an independent layer.

The weights are $(0.500000,0.815485,0.121306)$ and

Expected reward is $0.525359$, KL is $0.162947$, and the regularized value is $0.362412$. The reference value is $0.2$.

A common shift multiplies all unnormalized weights by $e^{c/\beta}$, which cancels in normalization. Zero reference mass gives zero tilted mass for finite reward. These express reward-offset non-identifiability and the forward-KL support restriction.

The two $\beta\log Z(x)$ terms cancel inside the Bradley–Terry reward difference, yielding the reference-corrected pair logit. The observed-choice negative log-likelihood is

The chosen log-ratio is $0.2$, the rejected log-ratio $-0.6$, so $h=0.8$ and $z=0.4$. The loss is $0.513015$ and $\beta\sigma(-z)=0.200656$. The weight approaches zero for a large positive margin.

At $z=2$, the cDPO loss is $0.326928$. With $\tau=0.25$, the IPO target is $2$, and losses at $1,2,4$ are $1,0,4$. IPO penalizes both under- and overshooting a finite target.

Softmax normalization cancels, giving $\log p_1-\log p_2=2t$.

The third logit captures mass while the chosen logit declines more slowly than the rejected. Alert on growing margin with falling chosen log-probability, then check semantic quality independently.

The mixture gives

Reward increases with $q$ and the constraint requires $q\le0.25$, so $q^*=0.25$ and expected reward is $0.7$.

Also

Thus $q=1$ for $\lambda<0.4$, $q=0$ for $\lambda>0.4$, and every $q\in[0,1]$ is optimal at $\lambda=0.4$. One dual step is

An average budget permits the risky action at a limited frequency; a hard prohibition requires zero probability in the relevant states.

import numpy as np


def shape_token_rewards(p, q, terminal_reward, beta, mask):
    p, q, mask = [np.asarray(x, dtype=np.float64) for x in (p, q, mask)]
    if p.ndim != 1 or q.shape != p.shape or mask.shape != p.shape:
        raise ValueError("aligned vectors required")
    if np.any((mask != 0) & (mask != 1)) or mask.sum() == 0:
        raise ValueError("non-empty binary mask required")
    out = -beta * (p - q) * mask
    out[np.flatnonzero(mask)[-1]] += terminal_reward
    return out

Check:

out = shape_token_rewards([-0.2, -1.0, -2.0], [-0.2, -1.3, -1.8], 1.5, 0.2, [0, 1, 1])
assert np.isclose(out.sum(), 1.5 - 0.2 * (0.3 - 0.2))

import numpy as np


def likelihood_displacement_curve(ts):
    ts = np.asarray(ts, dtype=np.float64)
    logits = np.stack([-ts, -3 * ts, 4 * ts], axis=1)
    logits -= logits.max(axis=1, keepdims=True)
    p = np.exp(logits); p /= p.sum(axis=1, keepdims=True)
    return {"t": ts, "chosen": p[:, 0], "rejected": p[:, 1], "other": p[:, 2], "log_margin": 2 * ts}


def displacement_alert(curve):
    return bool(np.all(np.diff(curve["log_margin"]) > 0) and np.mean(np.diff(np.log(curve["chosen"]))) < 0)

The alert detects probability geometry, not semantic quality, so independent generation evaluation is still required.

import numpy as np


def dual_ascent_step(multipliers, expected_costs, limits, step_size):
    lam = np.asarray(multipliers, dtype=np.float64)
    costs = np.asarray(expected_costs, dtype=np.float64)
    bounds = np.asarray(limits, dtype=np.float64)
    if lam.shape != costs.shape or lam.shape != bounds.shape:
        raise ValueError("all arrays must have the same shape")
    if np.any(lam < 0) or not np.all(np.isfinite(lam)):
        raise ValueError("multipliers must be finite and non-negative")
    if step_size <= 0 or not np.isfinite(step_size):
        raise ValueError("step_size must be finite and positive")
    return np.maximum(0.0, lam + step_size * (costs - bounds))


def two_action_constrained_optimum(
    safe_reward,
    risky_reward,
    safe_cost,
    risky_cost,
    cost_limit,
):
    values = np.asarray(
        [safe_reward, risky_reward, safe_cost, risky_cost, cost_limit],
        dtype=np.float64,
    )
    if not np.all(np.isfinite(values)):
        raise ValueError("all inputs must be finite")
    if risky_cost <= safe_cost:
        raise ValueError("the teaching contract requires risky_cost > safe_cost")
    if cost_limit < safe_cost:
        raise ValueError("the cost budget is infeasible even for the safe action")
    q_max = np.clip(
        (cost_limit - safe_cost) / (risky_cost - safe_cost),
        0.0,
        1.0,
    )
    return float(q_max if risky_reward > safe_reward else 0.0)

The teaching example returns $0.25$. The multiplier rises under violation and decreases, with projection at zero, when the constraint has slack. Correct code validates the update algebra, not the semantics or coverage of the cost model.