Module 8. Maximum entropy, exponential models, and KL regularization

Exercise 1. A MaxEnt die

On $\mathcal X=\{1,2,3,4,5,6\}$ , find the maximum-entropy distribution subject to

\mathbb E[X]=4.5.

Show that
$q_\lambda(x)=\frac{e^{\lambda x}}{\sum_{j=1}^6e^{\lambda j}}.$
Prove that the mean is strictly increasing in $\lambda$ .
Numerically find $\lambda$ , the probabilities, and the entropy.
Show that
$\log_2 6-H_2(q^*) = D_{\mathrm{KL},2}(q^*\|U_6).$

Show answer and solution

We seek a distribution on $\{1,\ldots,6\}$ maximizing

H(q)=-\sum_{x=1}^6q(x)\log q(x)

subject to

\sum_xq(x)=1, \qquad \sum_xxq(x)=4.5.

1. Form of the solution

The Lagrangian is

\mathcal L = -\sum_xq(x)\log q(x) +\lambda\left(\sum_xxq(x)-4.5\right) +\nu\left(\sum_xq(x)-1\right).

At an interior solution,

-\log q(x)-1+\lambda x+\nu=0,

so

q_\lambda(x) = \frac{e^{\lambda x}}{Z(\lambda)}, \qquad Z(\lambda)=\sum_{j=1}^6e^{\lambda j}.

2. Why the mean is strictly increasing in $\lambda$

Let

A(\lambda)=\log Z(\lambda).

Then

A'(\lambda)=\mathbb E_{q_\lambda}[X],

and

\frac{d}{d\lambda}\mathbb E_{q_\lambda}[X] =A''(\lambda) =\operatorname{Var}_{q_\lambda}(X)>0.

The variance is positive for every finite $\lambda$ , since all six probabilities are positive and $X$ is not constant. Hence the equation

\mathbb E_{q_\lambda}[X]=4.5

has at most one solution. The limiting means as $\lambda\to-\infty$ and $\lambda\to+\infty$ are 1 and 6, respectively, so the solution exists and is unique.

3. Numerical answer

The moment equation gives

\boxed{\lambda\approx0.371048938081}.

The probabilities are

\boxed{ q^*\approx (0.054353168, 0.078771546, 0.114159977, 0.165446803, 0.239774440, 0.347494066) }.

A check:

\sum_x xq^*(x)=4.5.

The entropy is

\boxed{H_2(q^*)\approx2.327905448\text{ bits}}.

4. Entropy deficit as KL to uniform

For $U_6(x)=1/6$ ,

\begin{aligned} D_{\mathrm{KL},2}(q^*\|U_6) &=\sum_xq^*(x)\log_2\frac{q^*(x)}{1/6}\\ &=\sum_xq^*(x)\log_2q^*(x)+\log_2 6\\ &=\log_2 6-H_2(q^*). \end{aligned}

Numerically,

\boxed{ \log_2 6-H_2(q^*) \approx0.257057052\text{ bits} }.

This exact equality uses a uniform reference on a finite space. For a general reference, the relevant quantity is $D_{\mathrm{KL}}(q\|m)$ , not merely an entropy difference.

Exercise 2. One mean, two state spaces

Let $n=10$ and $\mathbb E[K]=7$ .

Find the MaxEnt form on $K\in\{0, \ldots,10\}$ under a uniform reference.
Find the MaxEnt distribution on $\{0,1\}^{10}$ under $\mathbb E[\sum_iX_i]=7$ .
Derive the induced distribution of the number of ones.
Explain the role of $\binom{10}{k}$ and why the two answers do not contradict each other.

Show answer and solution

Let $n=10$ and $\mathbb E[K]=7$ .

1. Uniform reference on the count space

If the elementary states are the counts

K\in\{0,1,\ldots,10\},

then

q_A(k) = \frac{e^{\lambda k}} {\sum_{j=0}^{10}e^{\lambda j}}.

The mean constraint gives

\boxed{\lambda\approx0.218710977833}.

The probabilities are approximately

\begin{aligned} q_A\approx(&0.02423503,0.03015981,0.03753302,0.04670878,\\ &0.05812775,0.07233833,0.09002299,0.11203105,\\ &0.13941946,0.17350355,0.21592023). \end{aligned}

This is a truncated geometric distribution.

2. Uniform reference on binary microstates

Now an elementary state is

x\in\{0,1\}^{10}, \qquad K(x)=\sum_{i=1}^{10}x_i.

The MaxEnt solution is

q_B(x) \propto \exp\left(\lambda\sum_i x_i\right) = \prod_i e^{\lambda x_i}.

Thus the coordinates are independent and identically distributed:

X_i\sim\operatorname{Bernoulli}(p), \qquad 10p=7,

so

\boxed{p=0.7}.

3. Induced distribution of the number of ones

Therefore

\boxed{ K\sim\operatorname{Binomial}(10,0.7) }

and

q_B(k) = \binom{10}{k}0.7^k0.3^{10-k}.

4. Why the answers differ

In the first problem, each $k$ is one state. In the second, the same count $k$ represents

\binom{10}{k}

distinct microstates. The reference induced on counts is therefore not uniform; it is proportional to $\binom{10}{k}$ :

q_B(k) \propto \binom{10}{k}e^{\lambda k}.

There is no contradiction: these are different probabilistic models. A mean constraint does not determine a MaxEnt answer without a state space and a base measure.

Exercise 3. The softmax variational formula

For $\tau>0$ , prove

\tau\log\sum_ie^{z_i/\tau} = \max_{q\in\Delta} \{q^\top z+\tau H(q)\}.

Also show that the gap for arbitrary $q$ is

\tau D_{\mathrm{KL}} \left(q\|\operatorname{softmax}(z/\tau)\right).

Show answer and solution

Define

s_i = \frac{e^{z_i/\tau}} {\sum_j e^{z_j/\tau}}, \qquad \tau>0.

For any $q\in\Delta$ ,

\begin{aligned} q^\top z+\tau H(q) &=\sum_iq_iz_i-\tau\sum_iq_i\log q_i\\ &=\tau\sum_iq_i \log\frac{e^{z_i/\tau}}{q_i}\\ &=\tau\sum_iq_i \log\frac{Zs_i}{q_i}\\ &=\tau\log Z -\tau D_{\mathrm{KL}}(q\|s), \end{aligned}

where

Z=\sum_i e^{z_i/\tau}.

Therefore

q^\top z+\tau H(q) \le \tau\log Z,

with equality uniquely at $q=s$ . Hence

\boxed{ \tau\log\sum_i e^{z_i/\tau} = \max_{q\in\Delta} \{q^\top z+\tau H(q)\} }.

The exact gap is

\boxed{ \tau\log\sum_i e^{z_i/\tau} - \bigl(q^\top z+\tau H(q)\bigr) = \tau D_{\mathrm{KL}}(q\|s) }.

This is stronger than merely identifying the optimum: it measures exactly how much worse an arbitrary distribution is under the objective.

Exercise 4. Conditional MaxEnt and moment matching

For the conditional log-linear model

q_\theta(y\mid x) = \frac{e^{\theta^\top f(x,y)}} {\sum_{y'}e^{\theta^\top f(x,y')}}

derive the gradient of conditional log-likelihood and state the exact conditions under which zero gradient implies equality of empirical and model feature means.

Show answer and solution

For one example,

\log q_\theta(y_i\mid x_i) = \theta^\top f(x_i,y_i) - \log\sum_{y'}e^{\theta^\top f(x_i,y')}.

Differentiation gives

\begin{aligned} \nabla_\theta\log q_\theta(y_i\mid x_i) &=f(x_i,y_i)\\ &\quad- \sum_{y'}q_\theta(y'\mid x_i)f(x_i,y'). \end{aligned}

Summing over the data,

\boxed{ \nabla_\theta\ell(\theta) = \sum_i f(x_i,y_i) - \sum_i\mathbb E_{Y\sim q_\theta(\cdot\mid x_i)}[f(x_i,Y)] }.

If an unregularized finite interior maximum exists, the first-order condition yields

\sum_i f(x_i,y_i) = \sum_i\mathbb E_{q_\theta(\cdot\mid x_i)}[f(x_i,Y)].

The conditional log-likelihood of a log-linear model is concave in $\theta$ , so any finite stationary point is globally optimal. Parameter uniqueness additionally requires a minimal, identifiable feature representation.

Qualifications:

with $L_2$ regularization, the two feature sums differ by the penalty gradient;
separable data may have no finite MLE;
linearly dependent features permit different $\theta$ values representing the same distribution;
for a deep network, a corresponding equality for all internal features does not follow, because the full model is not a linear exponential family in all weights.

Exercise 5. A Gibbs-optimal policy

Let

\pi_{\mathrm{ref}}=(0.6,0.3,0.1), \qquad r=(0,1,3), \qquad \beta=1.

Find $\pi^*$ .
Compute $Z$ and the optimal value $J(\pi^*)$ .
For $\pi=(1/3,1/3,1/3)$ , verify
$J(\pi^*)-J(\pi) = \beta D_{\mathrm{KL}}(\pi\|\pi^*).$
Explain what happens to an action assigned zero probability by the reference policy.

Show answer and solution

We have

\pi_{\mathrm{ref}}=(0.6,0.3,0.1), \qquad r=(0,1,3), \qquad \beta=1.

1. Optimal policy

The unnormalized weights are

\pi_{\mathrm{ref},i}e^{r_i} = (0.6,0.3e,0.1e^3).

The partition function is

\boxed{ Z=0.6+0.3e+0.1e^3 \approx3.424038241 }.

Therefore

\boxed{ \pi^* \approx (0.17523169, 0.23816456, 0.58660376) }.

The large reward of the third action overcomes its small reference probability, although the reference still enters multiplicatively.

2. Optimal value

For $\beta=1$ ,

\boxed{ J(\pi^*)=\log Z\approx1.230820627 }.

3. Checking the objective gap

For

\pi=(1/3,1/3,1/3),

we obtain

J(\pi) = \mathbb E_\pi[r] -D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{ref}}) \approx1.092817782.

Thus

J(\pi^*)-J(\pi) \approx0.138002845.

Direct calculation gives

D_{\mathrm{KL}}(\pi\|\pi^*) \approx0.138002845,

confirming

J(\pi^*)-J(\pi) = \beta D_{\mathrm{KL}}(\pi\|\pi^*).

4. Zero probability under the reference policy

If

\pi_{\mathrm{ref}}(y)=0,

then the Gibbs formula gives

\pi^*(y)=0

for every finite reward. Any policy with $\pi(y)>0$ has

D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{ref}})=+\infty.

Thus KL-regularized optimization cannot create support absent from the reference policy. In a practical LLM, softmax normally gives every token positive probability, although masks, grammar constraints, and finite-precision underflow can restrict the effective support.

Exercise 6. Python: MaxEnt, softmax, and reward overoptimization

Run M8_maxent_softmax_policy.py. It performs four reproducible experiments:

solves the MaxEnt die problem with mean $4.5$ ;
compares truncated-geometric and binomial distributions under one mean constraint;
numerically verifies the softmax variational formula;
traces regularization strength for an intentionally misspecified proxy reward containing a reward-hacking action.

Explain why the last experiment demonstrates a possibility, not a universal inevitability of true-quality degradation.

Show answer and solution

The complete checked solution is the standalone file

text

M8_maxent_softmax_policy.py

It generates four PNG files under generated_experiment/ and prints reproducible numerical results.

Key values:

text

MaxEnt die:
lambda = 0.371048938081
entropy = 2.327905448379 bits
log2(6) - H = KL(p || uniform) = 0.257057052342 bits

Count state space, n=10, E[K]=7:
lambda = 0.218710977833
entropy = 3.158029428 bits

Binary microstate space:
K ~ Binomial(10, 0.7)
entropy of K = 2.566668139 bits

Softmax identity:
objective gap for random q
= tau * KL(q || softmax)
= 0.582011246754

Reward-overoptimization toy:
true reward peaks near beta = 0.397530
peak expected true reward = -0.272802779
at beta = 0.02 the hacking action receives probability ~1,
proxy reward tends to 0.8, while true reward tends to -3.

The last experiment is deliberately constructed with a rare action that has high proxy reward and poor true reward. As $\beta$ decreases, the policy eventually concentrates on it.

This establishes a possibility, not a universal inevitability. If proxy reward equals true reward, or remains positively correlated with it all along the path, expected true reward need not decline. The exact local condition is

\frac{d}{d\alpha} \mathbb E_{\pi_\alpha}[r_{\mathrm{true}}] = \operatorname{Cov}_{\pi_\alpha} (r_{\mathrm{true}},r_{\mathrm{proxy}}).

Code solution · the same source file for RU/EN

PythonM8_maxent_softmax_policy.pyDownload .py

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326

"""Reproducible experiments for Module 8.

The script demonstrates:
1. maximum entropy on a die with a prescribed mean;
2. dependence of MaxEnt on the chosen state space;
3. the softmax variational identity;
4. a finite-action reward-overoptimization path.

Run:
    python M8_maxent_softmax_policy.py

Dependencies:
    numpy, scipy, matplotlib
"""

from __future__ import annotations

from pathlib import Path
import math

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import brentq
from scipy.special import logsumexp


OUTPUT_DIR = Path(__file__).resolve().parent / "generated_experiment"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)


def normalize_log_weights(log_weights: np.ndarray) -> np.ndarray:
    """Return normalized probabilities from unnormalized log weights."""
    log_weights = np.asarray(log_weights, dtype=float)
    return np.exp(log_weights - logsumexp(log_weights))


def entropy_bits(probabilities: np.ndarray) -> float:
    """Shannon entropy in bits."""
    p = np.asarray(probabilities, dtype=float)
    positive = p > 0
    return float(-np.sum(p[positive] * np.log2(p[positive])))


def kl_nats(p: np.ndarray, q: np.ndarray) -> float:
    """KL(p || q) in nats for strictly positive q on the support of p."""
    p = np.asarray(p, dtype=float)
    q = np.asarray(q, dtype=float)
    positive = p > 0
    if np.any(q[positive] <= 0):
        return float("inf")
    return float(np.sum(p[positive] * (np.log(p[positive]) - np.log(q[positive]))))


def solve_exponential_tilt(states: np.ndarray, target_mean: float) -> tuple[float, np.ndarray]:
    """Solve p_lambda(x) proportional to exp(lambda*x) with a target mean."""
    states = np.asarray(states, dtype=float)
    if states.ndim != 1 or states.size == 0 or not np.all(np.isfinite(states)):
        raise ValueError("states must be a nonempty finite one-dimensional array")
    low, high = states.min(), states.max()
    if not (low <= target_mean <= high):
        raise ValueError("target_mean must lie in the convex hull of states")

    if low == high:
        return 0.0, np.full(states.size, 1.0 / states.size)
    if target_mean == low:
        p = (states == low).astype(float)
        p /= p.sum()
        return float("-inf"), p
    if target_mean == high:
        p = (states == high).astype(float)
        p /= p.sum()
        return float("inf"), p

    # Rescale the statistic; this leaves the distribution unchanged and makes
    # bracketing independent of the units in which states are measured.
    scaled = (states - low) / (high - low)
    target = (target_mean - low) / (high - low)

    def probabilities(eta: float) -> np.ndarray:
        return normalize_log_weights(eta * scaled)

    def residual(eta: float) -> float:
        return float(probabilities(eta) @ scaled - target)

    left, right = -1.0, 1.0
    while residual(left) > 0:
        left *= 2.0
    while residual(right) < 0:
        right *= 2.0
    root = brentq(residual, left, right, xtol=1e-12)
    return float(root / (high - low)), probabilities(root)


def experiment_die(target_mean: float = 4.5) -> dict[str, object]:
    """MaxEnt distribution on a six-sided die with a prescribed mean."""
    states = np.arange(1, 7, dtype=float)
    lam, p = solve_exponential_tilt(states, target_mean)
    entropy = entropy_bits(p)
    uniform = np.full_like(p, 1.0 / len(p))
    deficit = np.log2(len(p)) - entropy
    kl_to_uniform_bits = kl_nats(p, uniform) / np.log(2.0)

    assert np.isclose(p.sum(), 1.0, atol=1e-12)
    assert np.isclose(p @ states, target_mean, atol=1e-11)
    assert np.isclose(deficit, kl_to_uniform_bits, atol=1e-12)

    print("\n=== Experiment 1: MaxEnt die ===")
    print(f"lambda = {lam:.12f}")
    print("probabilities =", np.array2string(p, precision=9))
    print(f"mean = {p @ states:.12f}")
    print(f"entropy = {entropy:.12f} bits")
    print(f"log2(6) - H = {deficit:.12f} bits")
    print(f"KL(p || uniform) = {kl_to_uniform_bits:.12f} bits")

    plt.figure(figsize=(7.2, 4.6))
    plt.bar(states.astype(int), p)
    plt.xlabel("Die face")
    plt.ylabel("Probability")
    plt.title(f"Maximum-entropy die with E[X] = {target_mean:g}")
    plt.tight_layout()
    plt.savefig(OUTPUT_DIR / "M8_die_maxent.png", dpi=180)
    plt.close()

    return {
        "lambda": lam,
        "probabilities": p,
        "entropy_bits": entropy,
        "deficit_bits": deficit,
    }


def experiment_state_space() -> dict[str, object]:
    """Compare two MaxEnt problems with n=10 and E[K]=7."""
    n = 10
    target_mean = 7.0
    counts = np.arange(n + 1, dtype=float)

    # Model A: the count k itself is one state, with a uniform reference on counts.
    lam_count, p_count = solve_exponential_tilt(counts, target_mean)

    # Model B: uniform reference over 2^n binary microstates.
    bernoulli_p = target_mean / n
    p_binomial = np.array(
        [
            float(math.comb(n, int(k)))
            * bernoulli_p**k
            * (1.0 - bernoulli_p) ** (n - k)
            for k in counts
        ],
        dtype=float,
    )

    p_binomial /= p_binomial.sum()

    assert np.isclose(p_count @ counts, target_mean, atol=1e-10)
    assert np.isclose(p_binomial @ counts, target_mean, atol=1e-10)
    assert not np.allclose(p_count, p_binomial)

    print("\n=== Experiment 2: one mean, two state spaces ===")
    print(f"count-space lambda = {lam_count:.12f}")
    print("count-space probabilities =", np.array2string(p_count, precision=8))
    print("binomial probabilities   =", np.array2string(p_binomial, precision=8))
    print(f"count-space entropy = {entropy_bits(p_count):.9f} bits")
    print(f"binomial entropy    = {entropy_bits(p_binomial):.9f} bits")

    width = 0.38
    plt.figure(figsize=(8.2, 4.8))
    plt.bar(counts - width / 2, p_count, width=width, label="Uniform reference on counts")
    plt.bar(counts + width / 2, p_binomial, width=width, label="Uniform reference on microstates")
    plt.xlabel("Number of ones K")
    plt.ylabel("Probability")
    plt.title("Same mean E[K] = 7, different state spaces")
    plt.legend()
    plt.tight_layout()
    plt.savefig(OUTPUT_DIR / "M8_state_space.png", dpi=180)
    plt.close()

    return {
        "lambda_count": lam_count,
        "count_probabilities": p_count,
        "binomial_probabilities": p_binomial,
    }


def softmax(logits: np.ndarray, temperature: float = 1.0) -> np.ndarray:
    """Stable softmax."""
    if temperature <= 0:
        raise ValueError("temperature must be positive")
    z = np.asarray(logits, dtype=float) / temperature
    return normalize_log_weights(z)


def entropy_nats(probabilities: np.ndarray) -> float:
    """Shannon entropy in nats."""
    p = np.asarray(probabilities, dtype=float)
    positive = p > 0
    return float(-np.sum(p[positive] * np.log(p[positive])))


def experiment_softmax_variational() -> dict[str, float]:
    """Numerically verify the softmax variational formula."""
    logits = np.array([2.0, 0.5, -1.0, 1.2], dtype=float)
    temperature = 0.7
    optimum = softmax(logits, temperature)
    optimum_value = temperature * logsumexp(logits / temperature)
    direct_value = float(optimum @ logits + temperature * entropy_nats(optimum))

    rng = np.random.default_rng(8)
    candidate = rng.dirichlet(np.ones(len(logits)))
    candidate_value = float(candidate @ logits + temperature * entropy_nats(candidate))
    gap = optimum_value - candidate_value
    predicted_gap = temperature * kl_nats(candidate, optimum)

    assert np.isclose(optimum_value, direct_value, atol=1e-12)
    assert gap >= -1e-12
    assert np.isclose(gap, predicted_gap, atol=1e-12)

    print("\n=== Experiment 3: softmax variational identity ===")
    print("softmax optimum =", np.array2string(optimum, precision=9))
    print(f"tau * logsumexp(z/tau) = {optimum_value:.12f}")
    print(f"objective at optimum    = {direct_value:.12f}")
    print(f"gap for random q        = {gap:.12f}")
    print(f"tau * KL(q || softmax)  = {predicted_gap:.12f}")

    return {
        "optimum_value": optimum_value,
        "candidate_gap": gap,
    }


def experiment_reward_overoptimization() -> dict[str, object]:
    """A finite-action example in which proxy reward rises while true reward eventually falls."""
    regular_action = np.linspace(-2.0, 2.0, 100)
    true_regular = -(regular_action - 0.75) ** 2
    proxy_regular = true_regular + 0.03 * np.sin(5.0 * regular_action)

    # Add a rare action that looks attractive to the proxy but is poor under the true objective.
    true_reward = np.concatenate([true_regular, np.array([-3.0])])
    proxy_reward = np.concatenate([proxy_regular, np.array([0.8])])
    reference = np.concatenate([np.full(100, 0.999 / 100.0), np.array([0.001])])

    betas = np.geomspace(3.0, 0.02, 120)
    inverse_betas = 1.0 / betas
    expected_proxy = np.empty_like(betas)
    expected_true = np.empty_like(betas)
    kl_to_reference = np.empty_like(betas)
    hack_probability = np.empty_like(betas)

    for i, beta in enumerate(betas):
        policy = normalize_log_weights(np.log(reference) + proxy_reward / beta)
        expected_proxy[i] = policy @ proxy_reward
        expected_true[i] = policy @ true_reward
        kl_to_reference[i] = kl_nats(policy, reference)
        hack_probability[i] = policy[-1]

    # As beta decreases, alpha=1/beta increases. The exact identities predict monotone
    # proxy reward and monotone KL along this finite Gibbs path.
    assert np.all(np.diff(expected_proxy) >= -1e-10)
    assert np.all(np.diff(kl_to_reference) >= -1e-10)
    peak_index = int(np.argmax(expected_true))
    assert 0 < peak_index < len(expected_true) - 1
    assert expected_true[-1] < expected_true[peak_index] - 1.0

    print("\n=== Experiment 4: reward overoptimization ===")
    print(f"true reward peaks at beta = {betas[peak_index]:.6f}")
    print(f"peak expected true reward = {expected_true[peak_index]:.9f}")
    print(f"final expected true reward = {expected_true[-1]:.9f}")
    print(f"final expected proxy reward = {expected_proxy[-1]:.9f}")
    print(f"final KL(policy || reference) = {kl_to_reference[-1]:.9f} nats")
    print(f"final reward-hacking action probability = {hack_probability[-1]:.9f}")

    plt.figure(figsize=(7.6, 4.8))
    plt.plot(inverse_betas, expected_proxy, label="Expected proxy reward")
    plt.plot(inverse_betas, expected_true, label="Expected true reward")
    plt.xscale("log")
    plt.xlabel("Optimization pressure 1/beta")
    plt.ylabel("Expected reward")
    plt.title("A possible reward-overoptimization path")
    plt.legend()
    plt.tight_layout()
    plt.savefig(OUTPUT_DIR / "M8_reward_overoptimization.png", dpi=180)
    plt.close()

    plt.figure(figsize=(7.6, 4.8))
    plt.plot(inverse_betas, kl_to_reference, label="KL to reference")
    plt.plot(inverse_betas, hack_probability, label="Probability of hacking action")
    plt.xscale("log")
    plt.xlabel("Optimization pressure 1/beta")
    plt.ylabel("Value")
    plt.title("Policy drift along the Gibbs path")
    plt.legend()
    plt.tight_layout()
    plt.savefig(OUTPUT_DIR / "M8_policy_drift.png", dpi=180)
    plt.close()

    return {
        "betas": betas,
        "expected_proxy": expected_proxy,
        "expected_true": expected_true,
        "kl_to_reference": kl_to_reference,
        "hack_probability": hack_probability,
        "peak_index": peak_index,
    }


def main() -> None:
    experiment_die()
    state_space = experiment_state_space()
    softmax_result = experiment_softmax_variational()
    reward_path = experiment_reward_overoptimization()

    print("\nGenerated figures:")
    for path in sorted(OUTPUT_DIR.glob("*.png")):
        print(" -", path.name)

    # Compact final checks that also serve as machine-readable invariants.
    assert np.isclose(state_space["count_probabilities"].sum(), 1.0)
    assert softmax_result["candidate_gap"] >= 0
    assert reward_path["expected_true"][-1] < reward_path["expected_true"][reward_path["peak_index"]]


if __name__ == "__main__":
    main()

Bonus. Gaussian MaxEnt through KL

Let $p$ be a density with finite differential entropy, mean $\mu$ , and variance $\sigma^2$ . Using

D_{\mathrm{KL}} \bigl(p\|\mathcal N(\mu,\sigma^2)\bigr) \ge0,

prove

h(p) \le \frac12\log(2\pi e\sigma^2),

and identify the equality condition.

Show answer and solution

Let

g(x) = \frac1{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right].

For any density $p$ with the same mean and variance,

D_{\mathrm{KL}}(p\|g) = \int p\log p-\int p\log g \ge0.

The first term equals $-h(p)$ . For the second,

\begin{aligned} \int p(x)\log g(x)\,dx &=-\frac12\log(2\pi\sigma^2) - \frac1{2\sigma^2} \mathbb E_p[(X-\mu)^2]\\ &=-\frac12\log(2\pi\sigma^2)-\frac12. \end{aligned}

Therefore

D_{\mathrm{KL}}(p\|g) = -h(p)+\frac12\log(2\pi e\sigma^2) \ge0,

and hence

\boxed{ h(p) \le \frac12\log(2\pi e\sigma^2) }.

Equality in Gibbs' inequality holds if and only if $p=g$ almost everywhere. Thus the Gaussian is the unique maximizer under fixed mean and variance.

Exercise 1. A MaxEnt die

1. Form of the solution

2. Why the mean is strictly increasing in λ\lambdaλ

3. Numerical answer

4. Entropy deficit as KL to uniform

Exercise 2. One mean, two state spaces

1. Uniform reference on the count space

2. Uniform reference on binary microstates

3. Induced distribution of the number of ones

4. Why the answers differ

Exercise 3. The softmax variational formula

Exercise 4. Conditional MaxEnt and moment matching

Exercise 5. A Gibbs-optimal policy

1. Optimal policy

2. Optimal value

3. Checking the objective gap

4. Zero probability under the reference policy

Exercise 6. Python: MaxEnt, softmax, and reward overoptimization

Bonus. Gaussian MaxEnt through KL

2. Why the mean is strictly increasing in $\lambda$