Information Theory for ML8 / 16

Module 8. Maximum entropy, exponential models, and KL regularization

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 29 · Correct: 0

Question 8.1.1

8.1. From partial knowledge to a distribution

What is the most general MaxEnt formulation used in this module?

One correct optionAAlways choose the uniform distributionBMinimize

D_{\mathrm{KL}}(q\|m)

over distributions compatible with the constraintsCMinimize variance at fixed meanDMaximize likelihood without a data model

A) Always choose the uniform distribution

B) Minimize $D_{\mathrm{KL}}(q\|m)$ over distributions compatible with the constraints

C) Minimize variance at fixed mean

D) Maximize likelihood without a data model

Answer and explanation

Correct answer: B. Ordinary MaxEnt on a finite set is the special case of minimum-KL projection with a uniform reference m.

Question 8.2.1

8.2. Exponential tilting and information projection

When does entropy maximization coincide with minimizing $D_{\mathrm{KL}}(q\|m)$ ?

One correct optionAOnly for Gaussian

m

BOnly with one constraintCFor every

m

, regardless of its formDWhen

m

is uniform on a finite state space

A) Only for Gaussian $m$

B) Only with one constraint

C) For every $m$ , regardless of its form

D) When $m$ is uniform on a finite state space

Answer and explanation

Correct answer: D. In that case D_KL(q‖m)=-H(q)+log |X|, so the two problems have the same optimum.

Question 8.2.2

8.2. Exponential tilting and information projection

What is the minimum-KL solution under linear moment constraints?

One correct optionA

q(x)=m(x)+\lambda^\top f(x)

q(x)\propto 1/\|f(x)-\mu\|

q_\lambda(x)=m(x)e^{\lambda^\top f(x)}/Z(\lambda)

q(x)=\delta(x-\mu)

A) $m$

B) $q(x)\propto 1/\|f(x)-\mu\|$

C) $q_\lambda(x)=m(x)e^{\lambda^\top f(x)}/Z(\lambda)$

D) $q(x)=\delta(x-\mu)$

Answer and explanation

Correct answer: C. It is an exponential tilt of the reference distribution.

Question 8.2.3

8.2. Exponential tilting and information projection

What is the Hessian of the log-partition function $A(\lambda)=\log Z(\lambda)$ ?

One correct optionAAlways the identity matrixBThe negative covariance matrixC

\operatorname{Cov}_{q_\lambda}(f(X))

, hence positive semidefiniteDFisher information only for a Gaussian

A) Always the identity matrix

B) The negative covariance matrix

C) $\operatorname{Cov}_{q_\lambda}(f(X))$ , hence positive semidefinite

D) Fisher information only for a Gaussian

Answer and explanation

Correct answer: C. Positive definiteness requires a minimal family; redundant features can make the Hessian singular.

Question 8.2.4

8.2. Exponential tilting and information projection

(optional)

What does the Pythagorean identity state for an interior I-projection $q^*$ ?

One correct optionA

D(q\|m)=D(q^*\|q)+D(m\|q^*)

D(q\|m)=D(q\|q^*)+D(q^*\|m)

for feasible

q

D(q\|m)=D(m\|q)

D(q\|q^*)=0

for every feasible

q

A) $m$

B) $D(q\|m)=D(q\|q^*)+D(q^*\|m)$ for feasible $q$

C) $D(q\|m)=D(m\|q)$

D) $D(q\|q^*)=0$ for every feasible $q$

Answer and explanation

Correct answer: B. Every other feasible distribution is farther from the reference by exactly D(q‖q*).

Question 8.2.5

8.2. Exponential tilting and information projection

(optional)

What may happen when the required moment vector lies on the boundary of the convex hull of feature values?

One correct optionAMaxEnt always becomes uniformBThe partition function must equal oneCAll Lagrange multipliers become zeroDThe optimum may have zero probabilities and parameters may diverge

A) MaxEnt always becomes uniform

B) The partition function must equal one

C) All Lagrange multipliers become zero

D) The optimum may have zero probabilities and parameters may diverge

Answer and explanation

Correct answer: D. The derivation for a strictly positive interior solution then requires a limit or KKT analysis.

Question 8.2.6

8.2. Exponential tilting and information projection

(optional)

What is the exponential order of the probability of empirical type $q$ under a general i.i.d. reference $m$ ?

One correct optionA

\exp\{-nD_{\mathrm{KL}}(q\|m)+o(n)\}

\exp\{nD_{\mathrm{KL}}(q\|m)+o(n)\}

\exp\{-nH(m)+o(n)\}

DIt does not depend on

q

A) $m$

B) $\exp\{nD_{\mathrm{KL}}(q\|m)+o(n)\}$

C) $\exp\{-nH(m)+o(n)\}$

D) It does not depend on $q$

Answer and explanation

Correct answer: A. For a uniform reference this reduces to the familiar relation between the number of sequences of type q and e^(nH(q)).

Question 8.3.1

8.3. The state space and reference measure

Which modeling choice can change the MaxEnt answer even under the same mean constraint?

One correct optionAThe elementary state space and base measureBOnly the logarithm baseCThe name of the random variableDThe order in which constraints are written

A) The elementary state space and base measure

B) Only the logarithm base

C) The name of the random variable

D) The order in which constraints are written

Answer and explanation

Correct answer: A. A uniform reference on counts and a uniform reference on binary microstates induce different distributions.

Question 8.4.1

8.4. Classical MaxEnt distributions and the AWGN channel

Which success-count distribution arises from MaxEnt on binary microstates with $\mathbb E[K]=np$ ?

One correct optionA

\operatorname{Binomial}(n,p)

BUniform on

\{0,\ldots,n\}

CPoisson for every

n

DTruncated Cauchy

A) $m$

B) Uniform on $\{0,\ldots,n\}$

C) Poisson for every $n$

D) Truncated Cauchy

Answer and explanation

Correct answer: A. The exponential tilt factorizes across coordinates, making them independent Bernoulli(p) variables.

Question 8.4.2

8.4. Classical MaxEnt distributions and the AWGN channel

Under which conditions is the exponential distribution a MaxEnt solution?

One correct optionAOn all of

\mathbb R

at fixed meanBOn

[0,1]

at fixed varianceCOn

[0,\infty)

at fixed positive meanDOn a finite set without constraints

A) On all of $m$ at fixed mean

B) On $[0,1]$ at fixed variance

C) On $[0,\infty)$ at fixed positive mean

D) On a finite set without constraints

Answer and explanation

Correct answer: C. The support is an essential part of the problem; a mean alone is insufficient on the whole real line.

Question 8.4.3

8.4. Classical MaxEnt distributions and the AWGN channel

How is the Gaussian MaxEnt property correctly used in the AWGN capacity proof?

One correct optionAOnly noise entropy is maximizedBGaussian input makes the output Gaussian and maximizes

h(Y)

under the output-variance constraintCEvery input distribution gives the same

I(X;Y)

DMaximizing

h(X)

automatically maximizes

I(X;Y)

without analyzing the output

A) Only noise entropy is maximized

B) Gaussian input makes the output Gaussian and maximizes $D_{\mathrm{KL}}(q\|m)$ under the output-variance constraint

C) Every input distribution gives the same $I(X;Y)$

D) Maximizing $h(X)$ automatically maximizes $I(X;Y)$ without analyzing the output

Answer and explanation

Correct answer: B. Since I(X;Y)=h(Y)-h(N), fixed noise means that output entropy is the quantity to maximize.

Question 8.5.1

8.5. Conditional MaxEnt and log-linear models

What is the form of a conditional MaxEnt model with features $f(x,y)$ ?

One correct optionA

q(y\mid x)=\theta^\top f(x,y)

q(y\mid x)\propto |\theta^\top f(x,y)|

q(y\mid x)

must be GaussianD

q_\theta(y\mid x)=e^{\theta^\top f(x,y)}/Z_\theta(x)

A) $m$

B) $q(y\mid x)\propto |\theta^\top f(x,y)|$

C) $q(y\mid x)$ must be Gaussian

D) $q_\theta(y\mid x)=e^{\theta^\top f(x,y)}/Z_\theta(x)$

Answer and explanation

Correct answer: D. This is a conditional log-linear model normalized separately for each input x.

Question 8.5.2

8.5. Conditional MaxEnt and log-linear models

When does a zero conditional-log-likelihood gradient give exact moment matching?

One correct optionAFor every deep network and every local minimumBOnly after label smoothingCOnly when all features are independentDAt a finite interior optimum of an unregularized log-linear model; regularization and separability change the result

A) For every deep network and every local minimum

B) Only after label smoothing

C) Only when all features are independent

D) At a finite interior optimum of an unregularized log-linear model; regularization and separability change the result

Answer and explanation

Correct answer: D. Moment matching is an exact exponential-family property under the relevant conditions, not a universal property of every neural network.

Question 8.6.1

8.6. Softmax as score-plus-entropy optimization

Which optimization problem is solved exactly by $\operatorname{softmax}(z/\tau)$ ?

One correct optionA

\max_{q\in\Delta}\{q^\top z+\tau H(q)\}

\min_q\|q-z\|_2^2

without constraintsC

\max_q H(q)

independently of

z

\min_qD_{\mathrm{KL}}(z\|q)

where

z

is a distribution

A) $m$

B) $\min_q\|q-z\|_2^2$ without constraints

C) $\max_q H(q)$ independently of $z$

D) $\min_qD_{\mathrm{KL}}(z\|q)$ where $z$ is a distribution

Answer and explanation

Correct answer: A. The maximum value is τ log Σ_i e^(z_i/τ) and the gap is τ D_KL(q‖softmax).

Question 8.6.2

8.6. Softmax as score-plus-entropy optimization

What happens to softmax as temperature $\tau$ increases for fixed logits?

One correct optionAIt always becomes one-hotBIt changes the ranking of classesCThe distribution becomes flatter and tends to uniform as

\tau\to\infty

DThe normalizing constant disappears

A) It always becomes one-hot

B) It changes the ranking of classes

C) The distribution becomes flatter and tends to uniform as $\operatorname{Cov}_{q_\lambda}(f(X))$

D) The normalizing constant disappears

Answer and explanation

Correct answer: C. As τ→0, mass concentrates on maximizers; at high temperature the entropy term has greater influence.

Question 8.7.1

8.7. Energy-based models and the partition function

What is the gradient of EBM negative log-likelihood when the reference measure is independent of $\theta$ ?

One correct optionAOnly

\nabla_\theta E_\theta(x)

\nabla E_\theta(x)-\mathbb E_{q_\theta}[\nabla E_\theta(X)]

\mathbb E_{q_\theta}[E_\theta(X)]

DIt is always zero because of the partition function

A) Only $m$

B) $\nabla E_\theta(x)-\mathbb E_{q_\theta}[\nabla E_\theta(X)]$

C) $\mathbb E_{q_\theta}[E_\theta(X)]$

D) It is always zero because of the partition function

Answer and explanation

Correct answer: B. It is the difference between the positive phase on data and a model expectation, which is often the difficult part of EBM training.

Question 8.7.2

8.7. Energy-based models and the partition function

(optional)

What is the most accurate description of the relation between score-based diffusion models and EBMs?

One correct optionADiffusion does not use probability distributionsBScores are related to gradients of log-density/energy, but diffusion usually learns time-dependent scores of noise-perturbed distributionsCEvery diffusion model is an exact MaxEnt solution with known momentsDThe partition function in diffusion is always computed analytically

A) Diffusion does not use probability distributions

B) Scores are related to gradients of log-density/energy, but diffusion usually learns time-dependent scores of noise-perturbed distributions

C) Every diffusion model is an exact MaxEnt solution with known moments

D) The partition function in diffusion is always computed analytically

Answer and explanation

Correct answer: B. The common Gibbs/score structure is real, but “diffusion = MaxEnt without normalization” hides essential details.

Question 8.8.1

8.8. A KL-regularized policy and its Gibbs optimum

What is the unconstrained optimum of a KL-regularized policy objective?

One correct optionA

\pi^*=\pi_{\mathrm{ref}}+r/\beta

\pi^*\propto e^{-r}

\pi^*

is always uniformD

\pi^*(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)e^{r(x,y)/\beta}

A) $m$

B) $\pi^*\propto e^{-r}$

C) $\pi^*$ is always uniform

D) $\pi^*(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)e^{r(x,y)/\beta}$

Answer and explanation

Correct answer: D. The formula requires a finite normalizer and optimization on the support of the reference policy.

Question 8.8.2

8.8. A KL-regularized policy and its Gibbs optimum

How should $\beta$ be interpreted in KL-regularized policy optimization?

One correct optionAOnly relative to reward scale: large

\beta

strengthens the anchor, small

\beta

strengthens the reward tiltBAs an absolute universal safety measureCAs the optimizer learning rateDAs the number of samples in the preference dataset

A) Only relative to reward scale: large $m$ strengthens the anchor, small $\beta$ strengthens the reward tilt

B) As an absolute universal safety measure

C) As the optimizer learning rate

D) As the number of samples in the preference dataset

Answer and explanation

Correct answer: A. Scaling reward is equivalent to inversely scaling β.

Question 8.8.3

8.8. A KL-regularized policy and its Gibbs optimum

Which statement about PPO and DPO is correct?

One correct optionAThey must produce identical model weightsBDPO always computes an explicit KL on every minibatchCThe Gibbs optimum enters the DPO derivation, but finite-data PPO and DPO can yield different policiesDPPO is unrelated to reward optimization

A) They must produce identical model weights

B) DPO always computes an explicit KL on every minibatch

C) The Gibbs optimum enters the DPO derivation, but finite-data PPO and DPO can yield different policies

D) PPO is unrelated to reward optimization

Answer and explanation

Correct answer: C. They differ in data, parameterization, estimators, regularization, and training dynamics.

Question 8.9.1

8.9. Reward overoptimization and the role of KL

What is the derivative of expected proxy reward along the Gibbs path with respect to $\alpha=1/\beta$ ?

One correct optionA

-\operatorname{Var}(r_{\mathrm{proxy}})

D_{\mathrm{KL}}(\pi_\alpha\|\pi_{\mathrm{ref}})

\operatorname{Var}_{\pi_\alpha}(r_{\mathrm{proxy}})\ge0

DIt is always one

A) $m$

B) $D_{\mathrm{KL}}(\pi_\alpha\|\pi_{\mathrm{ref}})$

C) $\operatorname{Var}_{\pi_\alpha}(r_{\mathrm{proxy}})\ge0$

D) It is always one

Answer and explanation

Correct answer: C. Expected proxy reward is therefore monotonically nondecreasing as optimization pressure increases.

Question 8.9.2

8.9. Reward overoptimization and the role of KL

What is the derivative of expected true reward along the same path?

One correct optionA

\operatorname{Cov}_{\pi_\alpha}(r_{\mathrm{true}},r_{\mathrm{proxy}})

BAlways the positive variance of true rewardC

1/\beta

independently of the rewardsDZero because of KL regularization

A) $m$

B) Always the positive variance of true reward

C) $1/\beta$ independently of the rewards

D) Zero because of KL regularization

Answer and explanation

Correct answer: A. The covariance can change sign, so true reward may rise, plateau, or fall.

Question 8.9.3

8.9. Reward overoptimization and the role of KL

How does $D_{\mathrm{KL}}(\pi_\alpha\|\pi_{\mathrm{ref}})$ change along the Gibbs path for a fixed proxy reward?

One correct optionAIt always decreasesBIt has no derivativeCIt always equals

1/\beta

DIts derivative is

\alpha\operatorname{Var}_{\pi_\alpha}(r_{\mathrm{proxy}})\ge0

A) It always decreases

B) It has no derivative

C) It always equals $\operatorname{Cov}_{q_\lambda}(f(X))$

D) Its derivative is $\alpha\operatorname{Var}_{\pi_\alpha}(r_{\mathrm{proxy}})\ge0$

Answer and explanation

Correct answer: D. This is an exact monotonicity law for KL to the reference along the idealized path, not a law for KL to an unknown true-optimal policy.

Question 8.10.1

8.10. Mathematical deepening: exponential families and sufficiency

(optional)

Which qualification is essential for the Pitman–Koopman–Darmois theorem?

One correct optionAIt applies only to discrete distributionsBIt requires regularity, i.i.d. sampling, and common support independent of the parameterCIt says that every family is exponentialDIt is an unconditional converse theorem to MaxEnt

A) It applies only to discrete distributions

B) It requires regularity, i.i.d. sampling, and common support independent of the parameter

C) It says that every family is exponential

D) It is an unconditional converse theorem to MaxEnt

Answer and explanation

Correct answer: B. The family Uniform(0,θ) illustrates why parameter-dependent support cannot be ignored.

Question 8.11.1

8.11. Exercises

What is $\log_2 6-H_2(q)$ for a die distribution relative to a uniform reference?

One correct optionA

D_{\mathrm{KL},2}(q\|U_6)

D_{\mathrm{KL},2}(U_6\|q)

H_2(U_6,q)

DAlways zero

A) $m$

B) $\pi^*\propto e^{-r}$

C) $H_2(U_6,q)$

D) Always zero

Answer and explanation

Correct answer: A. For a finite uniform reference, entropy deficit is exactly the forward KL to uniform.

Question 8.C.1

Computational block

A six-sided die is constrained only by the mean $\mathbb E[X]=4.5$ . The MaxEnt solution has $\lambda\approx0.371049$ in $q^*(x)\propto e^{\lambda x}$ . Compute its entropy in bits. (tolerance 0.003)

AnswerAbsolute tolerance: ±0.003

Answer and explanation

Numeric answer: 2.328 (tolerance 0.003). The entropy is 2.328 bits, against log₂ 6 = 2.585 bits for the unconstrained uniform die. The single moment constraint costs about 0.257 bits, and nothing beyond that constraint has been assumed.

Question 8.C.2

Computational block

With $\pi_{\mathrm{ref}}=(0.7,0.2,0.1)$ , rewards $r=(0,1,2)$ and $\beta=1$ , compute the probability the Gibbs-optimal policy assigns to the third action. (tolerance 0.003)

AnswerAbsolute tolerance: ±0.003

Answer and explanation

Numeric answer: 0.373 (tolerance 0.003). The tilt gives (0.353, 0.274, 0.373). The third action started at 0.1 and now leads, but the reference is not forgotten: two actions with equal reward would keep exactly the probability ratio they had under the reference.

Question 8.C.3

Computational block

For logits $z=(2,1,0)$ and temperature $\tau=0.5$ , compute the softmax probability of the first class. (tolerance 0.003)

AnswerAbsolute tolerance: ±0.003

Answer and explanation

Numeric answer: 0.867 (tolerance 0.003). The distribution is (0.867, 0.117, 0.016). Lowering the temperature magnifies logit differences, which is the same exponential tilt as in the policy above with the reward playing the role of the logits.

Question 8.C.4

Computational block

For the same logits and temperature, compute the maximum of the score-plus-entropy objective, that is $\tau\log\sum_i e^{z_i/\tau}$ , in nats. (tolerance 0.003)

AnswerAbsolute tolerance: ±0.003

Answer and explanation

Numeric answer: 2.071 (tolerance 0.003). The optimal value is 2.071 nats. Any other distribution falls short of it by exactly τ times the KL divergence to the softmax, which is why the softmax is a global optimum rather than merely a stationary point.