Information Theory for ML11 / 16

Module 11. Comparing distributions: f-divergences, variational critics, and Wasserstein distance

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 25 · Correct: 0

Question 11.2.1

11.2. The f-divergence family

What is the central object in the definition of $D_f(P\|Q)$ when $P\ll Q$ ?

One correct optionAThe density ratio

dP/dQ

, transformed by a convex function and averaged under

Q

BEuclidean distance between the meansCOnly the difference of entropiesDThe number of parameters in the two models

A) The density ratio $dP/dQ$ , transformed by a convex function and averaged under $Q$

B) Euclidean distance between the means

C) Only the difference of entropies

D) The number of parameters in the two models

Answer and explanation

Correct answer: A. An f-divergence is E_Q[f(dP/dQ)].

Question 11.2.2

11.2. The f-divergence family

Why does adding $a(t-1)$ to the generator $f(t)$ leave the divergence unchanged?

One correct optionAEvery linear function is zeroB

\mathbb E_Q[dP/dQ-1]=0

P

and

Q

must be uniformDThe convex conjugate is unchanged

A) Every linear function is zero

B) $\mathbb E_Q[dP/dQ-1]=0$

C) $P$ and $Q$ must be uniform

D) The convex conjugate is unchanged

Answer and explanation

Correct answer: B. The linear contribution integrates to a(P(Ω)-Q(Ω))=0.

Question 11.2.3

11.2. The f-divergence family

Which statement about Rényi divergence is most accurate?

One correct optionAIt always equals total variationBIt is an IPM with a Lipschitz criticCIt is defined only at

\alpha=1

DIn general it is not an f-divergence, although it is a monotone logarithmic transform of a power integral

A) It always equals total variation

B) It is an IPM with a Lipschitz critic

C) It is defined only at $\alpha=1$

D) In general it is not an f-divergence, although it is a monotone logarithmic transform of a power integral

Answer and explanation

Correct answer: D. The logarithm is applied after integration, taking Rényi divergence outside the ordinary form E_Q[f(p/q)].

Question 11.2.4

11.2. The f-divergence family

Which statement about squared Hellinger distance is correct under the convention used in the module?

One correct optionA

H^2(P,Q)=\frac12\int(\sqrt p-\sqrt q)^2

and lies in

[0,1]

BIt is infinite whenever

P\ne Q

CIt is asymmetricDIt equals forward KL

A) $dP/dQ$ and lies in $[0,1]$

B) It is infinite whenever $P\ne Q$

C) It is asymmetric

D) It equals forward KL

Answer and explanation

Correct answer: A. Another common convention omits the factor 1/2 and has maximum 2.

Question 11.3.1

11.3. What all f-divergences share—and what $f$ chooses

Which property holds for all f-divergences under standard conditions?

One correct optionASymmetryBUpper boundedness by oneCData Processing InequalityDTriangle inequality

A) Symmetry

B) Upper boundedness by one

C) Data Processing Inequality

D) Triangle inequality

Answer and explanation

Correct answer: C. Applying the same stochastic channel cannot increase an f-divergence.

Question 11.3.2

11.3. What all f-divergences share—and what $f$ chooses

Why does forward KL often show mode-covering behavior when a multimodal $P$ is approximated by one Gaussian?

One correct optionAIt averages only under the modelBAssigning tiny model probability to regions with substantial data mass is expensive under the expectation over

P

CIt must be symmetricDIt uses a ground metric

A) It averages only under the model

B) Assigning tiny model probability to regions with substantial data mass is expensive under the expectation over $\mathbb E_Q[dP/dQ-1]=0$

C) It must be symmetric

D) It uses a ground metric

Answer and explanation

Correct answer: B. A missed data mode produces a large log-loss penalty.

Question 11.3.3

11.3. What all f-divergences share—and what $f$ chooses

What is the main limitation of f-divergences on disjoint supports?

One correct optionAThey are always zeroBThey are undefined for probability measuresCThey require an RKHS kernelDThey generally do not encode the geometric distance between supports

A) They are always zero

B) They are undefined for probability measures

C) They require an RKHS kernel

D) They generally do not encode the geometric distance between supports

Answer and explanation

Correct answer: D. JS and TV saturate, while KL may become infinite, but the magnitude of the shift is not represented.

Question 11.5.1

11.5. Variational form: from convex conjugacy to a trainable critic

What does the general Fenchel variational form state?

One correct optionAEvery critic gives the exact divergenceBEvery f-divergence is a Wasserstein transport costC

D_f(P\|Q)=\sup_T\{\mathbb E_PT-\mathbb E_Qf^*(T)\}

for a sufficiently rich class under suitable conditionsDThe critic must be linear

A) Every critic gives the exact divergence

B) Every f-divergence is a Wasserstein transport cost

C) $\alpha=1$ for a sufficiently rich class under suitable conditions

D) The critic must be linear

Answer and explanation

Correct answer: C. A restricted neural critic gives only a lower bound on the population divergence.

Question 11.5.2

11.5. Variational form: from convex conjugacy to a trainable critic

What is the optimal critic for differentiable $f$ ?

One correct optionA

T^*(x)=\|x\|

T^*(x)=p(x)+q(x)

T^*(x)=f'(p(x)/q(x))

T^*(x)=0

always

A) $dP/dQ$

B) $T^*(x)=p(x)+q(x)$

C) $T^*(x)=f'(p(x)/q(x))$

D) $T^*(x)=0$ always

Answer and explanation

Correct answer: C. The critic encodes a transformed density ratio.

Question 11.5.3

11.5. Variational form: from convex conjugacy to a trainable critic

Why can a maximized empirical variational objective exceed the true divergence?

One correct optionAThe population theorem is falseBThe critic is adaptively chosen on the same finite sample and can overfitCJensen's inequality always reverses signD

f^*

must be negative

A) The population theorem is false

B) The critic is adaptively chosen on the same finite sample and can overfit

C) Jensen's inequality always reverses sign

D) $f^*$ must be negative

Answer and explanation

Correct answer: B. The lower-bound statement is a population result for a fixed critic, not an automatic guarantee after sample optimization.

Question 11.5.4

11.5. Variational form: from convex conjugacy to a trainable critic

What is the convex conjugate of $f(t)=t\log t$ ?

One correct optionA

s^2/2

-\log s

e^{s-1}

|s|

A) $dP/dQ$

B) $-\log s$

C) $e^{s-1}$

D) $|s|$

Answer and explanation

Correct answer: C. This yields the NWJ bound E_PT-E_Qe^(T-1).

Question 11.5.5

11.5. Variational form: from convex conjugacy to a trainable critic

How can Donsker–Varadhan be obtained from NWJ?

One correct optionAOptimize the NWJ objective over an additive critic constantBRestrict the critic to 1-Lipschitz functionsCSet

T=0

DReplace KL by TV

A) Optimize the NWJ objective over an additive critic constant

B) Restrict the critic to 1-Lipschitz functions

C) Set $\alpha=1$

D) Replace KL by TV

Answer and explanation

Correct answer: A. The optimal shift turns the exponential expectation into log E_Qe^T.

Question 11.6.1

11.6. GAN and f-GAN: what the adversarial game minimizes

How is the original minimax GAN linked to Jensen–Shannon divergence?

One correct optionAWith the optimal discriminator, its value is

-\log4+2\,\mathrm{JS}(P,Q)

BGAN always minimizes reverse KLCThe discriminator computes Wasserstein-2DThere is no link even in the idealized setting

A) With the optimal discriminator, its value is $dP/dQ$

B) GAN always minimizes reverse KL

C) The discriminator computes Wasserstein-2

D) There is no link even in the idealized setting

Answer and explanation

Correct answer: A. This is a population identity for an unrestricted optimal discriminator.

Question 11.6.2

11.6. GAN and f-GAN: what the adversarial game minimizes

Which statement about the non-saturating generator loss is correct?

One correct optionAIt changes the desired optimum to

P\ne Q

BIt equals MLE for every implicit generatorCIt does not use a discriminatorDIt has the same desired fixed point but is not literal gradient descent on JS with a finite discriminator

A) It changes the desired optimum to $dP/dQ$

B) It equals MLE for every implicit generator

C) It does not use a discriminator

D) It has the same desired fixed point but is not literal gradient descent on JS with a finite discriminator

Answer and explanation

Correct answer: D. The loss was introduced to provide a more informative gradient.

Question 11.6.3

11.6. GAN and f-GAN: what the adversarial game minimizes

Why cannot mode collapse be explained solely by one choice of f-divergence?

One correct optionAMode collapse occurs only in linear regressionBAll divergences are identicalCArchitecture, restricted critic capacity, alternating optimization, and the generator objective also matterDGANs do not use probabilities

A) Mode collapse occurs only in linear regression

B) All divergences are identical

C) Architecture, restricted critic capacity, alternating optimization, and the generator objective also matter

D) GANs do not use probabilities

Answer and explanation

Correct answer: C. Real adversarial dynamics are much richer than the ideal population divergence.

Question 11.7.1

11.7. MINE and InfoNCE: two different variational MI bounds

Which distributions does MINE compare to estimate $I(X;Y)$ ?

One correct optionA

P_X

and

P_Y

BTwo different priorsCTrain and test distributionsDThe joint

P_{XY}

and the product of marginals

P_XP_Y

A) $dP/dQ$ and $P_Y$

B) Two different priors

C) Train and test distributions

D) The joint $P_{XY}$ and the product of marginals $P_XP_Y$

Answer and explanation

Correct answer: D. Mutual information is KL between the joint and product-of-marginals distributions.

Question 11.7.2

11.7. MINE and InfoNCE: two different variational MI bounds

Which practical rule improves the reliability of a neural MI estimate?

One correct optionAAlways use the same batch for training and reportingBTrain the critic on one sample and evaluate the bound on held-out pairsCNever check a permutation baselineDTreat every reported number as a guaranteed lower bound

A) Always use the same batch for training and reporting

B) Train the critic on one sample and evaluate the bound on held-out pairs

C) Never check a permutation baseline

D) Treat every reported number as a guaranteed lower bound

Answer and explanation

Correct answer: B. Separating training and evaluation reduces adaptive optimistic bias.

Question 11.7.3

11.7. MINE and InfoNCE: two different variational MI bounds

What is the ceiling of the standard InfoNCE lower bound with $K$ candidates?

One correct optionA

\log K

K

1/K

DIt always equals the true MI

A) $dP/dQ$

B) $K$

C) $1/K$

D) It always equals the true MI

Answer and explanation

Correct answer: A. Since cross-entropy loss is nonnegative, log K-L_NCE≤log K.

Question 11.7.4

11.7. MINE and InfoNCE: two different variational MI bounds

Why is it inaccurate to call InfoNCE simply “MINE plus a shift”?

One correct optionAInfoNCE does not use scoresBInfoNCE is derived as a multi-sample classification bound, not as a simple plug-in DV estimatorCMINE is unrelated to MIDInfoNCE requires a tractable density

A) InfoNCE does not use scores

B) InfoNCE is derived as a multi-sample classification bound, not as a simple plug-in DV estimator

C) MINE is unrelated to MI

D) InfoNCE requires a tractable density

Answer and explanation

Correct answer: B. Both are related to density-ratio estimation, but they use different variational constructions.

Question 11.7.5

11.7. MINE and InfoNCE: two different variational MI bounds

What exactly is limited by the McAllester–Stratos result?

One correct optionADistribution-free high-confidence lower bounds from

N

samples, which generally cannot reliably be much larger than order

\log N

BThe true MI between finite-dimensional variablesCThe size of the learned representationDEvery numerical value produced by any estimator

A) Distribution-free high-confidence lower bounds from $dP/dQ$ samples, which generally cannot reliably be much larger than order $\log N$

B) The true MI between finite-dimensional variables

C) The size of the learned representation

D) Every numerical value produced by any estimator

Answer and explanation

Correct answer: A. A point estimator may output a larger number, but without a universal guarantee.

Question 11.8.1

11.8. IPMs, Wasserstein, and MMD: when geometry matters

What is the dual definition of Wasserstein-1?

One correct optionAThe number of mismatched model parametersBKL between a joint distribution and product of marginalsCMinimum cross-entropyDThe supremum of expectation differences over 1-Lipschitz functions

A) The number of mismatched model parameters

B) KL between a joint distribution and product of marginals

C) Minimum cross-entropy

D) The supremum of expectation differences over 1-Lipschitz functions

Answer and explanation

Correct answer: D. This is Kantorovich–Rubinstein duality.

Question 11.8.2

11.8. IPMs, Wasserstein, and MMD: when geometry matters

Which statement about WGAN-GP is correct?

One correct optionAIt uses a gradient penalty as a practical approximation to the Lipschitz constraintBIt exactly certifies the Lipschitz constant over the entire spaceCIt minimizes JS with the optimal discriminatorDIt has no critic

A) It uses a gradient penalty as a practical approximation to the Lipschitz constraint

B) It exactly certifies the Lipschitz constant over the entire space

C) It minimizes JS with the optimal discriminator

D) It has no critic

Answer and explanation

Correct answer: A. The penalty is applied at selected points, usually interpolations, and is not a global certificate.

Question 11.8.3

11.8. IPMs, Wasserstein, and MMD: when geometry matters

When is MMD with a characteristic kernel equal to zero?

One correct optionAOnly when the means agreeBFor any pair of distributionsCOnly for Gaussian distributionsDIf and only if

P=Q

A) Only when the means agree

B) For any pair of distributions

C) Only for Gaussian distributions

D) If and only if $f^*$

Answer and explanation

Correct answer: D. A characteristic kernel makes the kernel mean embedding injective.

Question 11.8.4

11.8. IPMs, Wasserstein, and MMD: when geometry matters

Which quantity is both an f-divergence and an IPM?

One correct optionAForward KLBTotal variationCRényi divergenceDPearson

\chi^2

A) Forward KL

B) Total variation

C) Rényi divergence

D) Pearson $f^*$

Answer and explanation

Correct answer: B. TV has both density-ratio and supremum-over-tests representations.

Question 11.9.1

11.9. Choosing a discrepancy in practice

Which principle for choosing an objective is most accurate?

One correct optionAEvery bounded divergence guarantees stable gradientsBWasserstein is always better than KLCThe choice depends on likelihood availability, sample-space geometry, sample complexity, critic class, and the desired fixed pointDInfoNCE always measures MI exactly

A) Every bounded divergence guarantees stable gradients

B) Wasserstein is always better than KL

C) The choice depends on likelihood availability, sample-space geometry, sample complexity, critic class, and the desired fixed point

D) InfoNCE always measures MI exactly

Answer and explanation

Correct answer: C. There is no universally best discrepancy; the objective is part of the full statistical specification.