Information Theory for ML5 / 16

Module 5. Mutual information: how much one object tells us about another

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 24 · Correct: 0

Question 5.2.1

5.2. One quantity, three readings

Under population-optimal probabilistic prediction with log loss, what does $I(X;Y)=H(Y)-H(Y\mid X)$ mean?

One correct optionAThe expected reduction in the minimum achievable log loss after observing

X

BThe difference between the accuracies of two classifiersCThe number of parameters that can be removed from a modelDThe reduction in log loss achieved by a particular trained model

A) The expected reduction in the minimum achievable log loss after observing $X$

B) The difference between the accuracies of two classifiers

C) The number of parameters that can be removed from a model

D) The reduction in log loss achieved by a particular trained model

Answer and explanation

Correct answer: A. Without X the minimum expected log loss is H(Y), and with X it is H(Y|X). The statement is about the population optimum, so no actual model is guaranteed to realize that gain.

Question 5.2.2

5.2. One quantity, three readings

What does $I(X;Y)=0$ imply?

One correct optionAThere is no linear relationship, but nonlinear dependence may remainB

X=Y

almost surelyC

X

and

Y

are uncorrelatedD

X

and

Y

are independent

A) There is no linear relationship, but nonlinear dependence may remain

B) $X=Y$ almost surely

C) $X$ and $Y$ are uncorrelated

D) $X$ and $Y$ are independent

Answer and explanation

Correct answer: D. Zero mutual information is equivalent to independence, which is strictly stronger than zero correlation. Options A and C describe what vanishing correlation gives, not what vanishing MI gives.

Question 5.2.3

5.2. One quantity, three readings

Let $Y=g(X)$ be a deterministic discrete function. What is $I(X;Y)$ ?

One correct optionAAlways

H(X)

BZeroC

H(X)-H(Y)

H(Y)

A) Always $X$

B) Zero

C) $H(X)-H(Y)$

D) $H(Y)$

Answer and explanation

Correct answer: D. Since H(Y|X) = 0, mutual information equals H(Y). A non-invertible g makes this strictly smaller than H(X), which is why option A holds only for invertible functions.

Question 5.3.1

5.3. PMI: the local log ratio of association

Can pointwise mutual information be negative?

One correct optionAOnly for Gaussian variablesBNo, because MI is always nonnegativeCYes, when a pair occurs less often than independence predictsDOnly when one of the marginals is zero

A) Only for Gaussian variables

B) No, because MI is always nonnegative

C) Yes, when a pair occurs less often than independence predicts

D) Only when one of the marginals is zero

Answer and explanation

Correct answer: C. The average is nonnegative, but individual terms may have either sign. A pair seen less often than chance would predict has a log ratio below zero.

Question 5.4.1

5.4. Conditional MI

In feature selection, what does $I(X;Y\mid Z)$ measure?

One correct optionAOnly the marginal entropy

H(X)

BA linear coefficient between

X

and

Z

CThe additional information

X

provides about

Y

once

Z

is knownDThe information shared by

X

and

Z

about

Y

A) Only the marginal entropy $X$

B) A linear coefficient between $X$ and $Z$

C) The additional information $X$ provides about $Y$ once $Z$ is known

D) The information shared by $X$ and $Z$ about $Y$

Answer and explanation

Correct answer: C. Under log loss it is the expected extra gain from X after Z is already available. Shared information is a different quantity, and conditional MI can be larger than the unconditional one.

Question 5.4.2

5.4. Conditional MI

In the XOR example $Y=X_1\oplus X_2$ with independent fair bits, which statement is correct?

One correct optionAEach individual feature carries one bit about

Y

I(X_1;Y)=0

, so

I(X_1;Y\mid X_2)=0

as wellCThe pair of features is independent of

Y

I(X_1;Y)=I(X_2;Y)=0

, but

I((X_1,X_2);Y)=1

bit

A) Each individual feature carries one bit about $X$

B) $I(X_1;Y)=0$ , so $I(X_1;Y\mid X_2)=0$ as well

C) The pair of features is independent of $Y$

D) $I(X_1;Y)=I(X_2;Y)=0$ , but $I((X_1,X_2);Y)=1$ bit

Answer and explanation

Correct answer: D. This is pure synergy: each feature alone is useless, and together they determine Y. Option B is the standard trap, because conditioning on X₂ raises the value from 0 to a full bit.

Question 5.4.3

5.4. Conditional MI

What can go wrong when features are ranked only by $I(X_i;Y)$ ?

One correct optionAThe procedure always selects every featureBIt can discard features that are useful only jointlyCIt guarantees the absence of leakageDIt systematically prefers features that are redundant with each other

A) The procedure always selects every feature

B) It can discard features that are useful only jointly

C) It guarantees the absence of leakage

D) It systematically prefers features that are redundant with each other

Answer and explanation

Correct answer: B. Marginal relevance cannot see interactions, so a XOR-like pair is thrown away. Redundancy is a real weakness of the method too, but it is a separate one and not what a marginal ranking is blind to.

Question 5.5.1

5.5. DPI and representation learning

What does DPI state for a Markov chain $Y\to X\to T$ ?

One correct optionA

I(Y;T)\le I(Y;X)

I(Y;T)\ge I(Y;X)

I(Y;T)=0

for every encoderD

I(Y;T)<I(Y;X)

unless

T=X

A) $X$

B) $I(Y;T)\ge I(Y;X)$

C) $I(Y;T)=0$ for every encoder

D) $I(Y;T)<I(Y;X)$ unless $T=X$

Answer and explanation

Correct answer: A. A fixed transformation cannot create target information that was not already in X. Equality is possible for a sufficient statistic, so the strict version in option D is false.

Question 5.5.2

5.5. DPI and representation learning

Why can linear-probe accuracy improve after an encoder without violating DPI?

One correct optionADPI applies only to regressionBAccuracy and MI are numerically equalCThe encoder can make already available information more accessible to a restricted readoutDThe probe is trained on more data than the encoder

A) DPI applies only to regression

B) Accuracy and MI are numerically equal

C) The encoder can make already available information more accessible to a restricted readout

D) The probe is trained on more data than the encoder

Answer and explanation

Correct answer: C. DPI limits how much information is present, not how easily a particular model class can extract it. Rearranging information is allowed; creating it is not.

Question 5.5.3

5.5. DPI and representation learning

Which condition describes a task-sufficient representation $T=f(X)$ ?

One correct optionA

T

must preserve every detail of

X

Y\perp X\mid T

, which gives

I(Y;T)=I(Y;X)

I(Y;T)

should be as small as possibleD

I(X;T)

should equal

H(X)

A) $X$ must preserve every detail of $X$

B) $H(X)-H(Y)$ , which gives $I(Y;T)\ge I(Y;X)$

C) $I(Y;T)=0$ should be as small as possible

D) $I(X;T)$ should equal $H(X)$

Answer and explanation

Correct answer: B. Sufficiency is about target-relevant information only. Preserving all of X, as in options A and D, is far more than the task requires.

Question 5.6.1

5.6. MI, correlation, and causality

Why does $\operatorname{Corr}(X,Y)=0$ not imply independence?

One correct optionACorrelation is always larger than MIBCorrelation detects linear association and can miss nonlinear dependenceCCorrelation is defined only for discrete variablesDCorrelation is undefined whenever the dependence is deterministic

A) Correlation is always larger than MI

B) Correlation detects linear association and can miss nonlinear dependence

C) Correlation is defined only for discrete variables

D) Correlation is undefined whenever the dependence is deterministic

Answer and explanation

Correct answer: B. For a symmetric X and Y = X² the covariance vanishes while the dependence is deterministic. Mutual information sees that dependence, which is exactly the difference between the two measures.

Question 5.7.1

5.7. Noisy labels

For a BSC with uniform input, what is the mutual information at $\varepsilon=1$ , when every bit is deterministically flipped?

One correct optionA1 bit, because the transformation is invertibleB0 bits, because every label is wrongC

1/2

bitDIt is undefined, because the channel is degenerate

A) 1 bit, because the transformation is invertible

B) 0 bits, because every label is wrong

C) $1/2$ bit

D) It is undefined, because the channel is degenerate

Answer and explanation

Correct answer: A. Mutual information measures preserved dependence, not agreement of symbols. A known deterministic flip is fully reversible, so nothing is lost; the channel is worst at ε = 1/2, not at ε = 1.

Question 5.8.1

5.8. Feature selection

Where should MI-based feature selection be performed during cross-validation?

One correct optionAOnce on the full dataset before splittingBInside each training foldCOn the union of the training and validation foldsDOnly on the test set

A) Once on the full dataset before splitting

B) Inside each training fold

C) On the union of the training and validation folds

D) Only on the test set

Answer and explanation

Correct answer: B. Any selection that has seen a validation fold makes the score on that fold optimistic. Options A and C both let the held-out data influence which features are kept.

Question 5.9.1

5.9. InfoNCE

Under which sampling scheme does the standard InfoNCE bound take the form $I(X;Y)\ge\log N-\mathcal L_{\mathrm{NCE}}$ ?

One correct optionAAny arbitrary candidates drawn from the same minibatchBNegatives drawn from the same conditional distribution as the positiveCDeterministic positives with no randomnessDOne positive from the conditional distribution and independent negatives from the marginal

A) Any arbitrary candidates drawn from the same minibatch

B) Negatives drawn from the same conditional distribution as the positive

C) Deterministic positives with no randomness

D) One positive from the conditional distribution and independent negatives from the marginal

Answer and explanation

Correct answer: D. The bound is derived for that specific scheme. If negatives come from the conditional instead of the marginal, or if minibatch structure creates dependence, the interpretation has to be derived again.

Question 5.9.2

5.9. InfoNCE

What is upper-bounded by $\log N$ in standard InfoNCE?

One correct optionAThe true MI of the representationsBDownstream task qualityCThe batch size that can be used in practiceDThe particular lower bound

\log N-\mathcal L_{\mathrm{NCE}}

A) The true MI of the representations

B) Downstream task quality

C) The batch size that can be used in practice

D) The particular lower bound $\log N-\mathcal L_{\mathrm{NCE}}$

Answer and explanation

Correct answer: D. The ceiling belongs to the estimator, not to the quantity it estimates. True MI may well exceed log N, in which case the bound simply stops being informative.

Question 5.10.1

5.10. MI estimation

Which statement about numerical MI estimation is most accurate?

One correct optionAPopulation MI, its estimator, and a training proxy must be kept separate, with stability and permutation baselines checkedBAny neural estimator returns exact population MICAttention entropy equals MI between tokensDPlug-in estimates are unbiased whenever the sample exceeds the alphabet size

A) Population MI, its estimator, and a training proxy must be kept separate, with stability and permutation baselines checked

B) Any neural estimator returns exact population MI

C) Attention entropy equals MI between tokens

D) Plug-in estimates are unbiased whenever the sample exceeds the alphabet size

Answer and explanation

Correct answer: A. High-dimensional MI is hard to estimate and a learned critic can overfit. Plug-in estimates carry a positive bias at small sample sizes, which is why a permutation baseline on independent data is worth running.

Question 5.2.4

5.2. Terminology

What is usually called the self-information of an individual outcome $x$ ?

One correct optionA

H(X)

I(X;X)/2

-\log p(x)

I(X;Y)

evaluated at that outcome

A) $X$

B) $X$

C) $-\log p(x)$

D) $I(X;Y)$ evaluated at that outcome

Answer and explanation

Correct answer: C. This is the surprisal of one outcome, and entropy is its expectation. The pointwise version of mutual information is PMI, which is a different quantity.

Question 5.3.2

5.3. Rare events and PMI

Why can a very rare word pair receive an extremely large PMI?

One correct optionABecause the product of marginal probabilities in the denominator is tinyBBecause PMI equals the logarithm of the frequencyCBecause PMI is upper-bounded by the pair frequencyDBecause rare events always carry more information about each other

A) Because the product of marginal probabilities in the denominator is tiny

B) Because PMI equals the logarithm of the frequency

C) Because PMI is upper-bounded by the pair frequency

D) Because rare events always carry more information about each other

Answer and explanation

Correct answer: A. A single co-occurrence of two rare words gives a huge ratio with almost no evidence behind it. Frequency thresholds, smoothing, and stability checks exist precisely for this reason.

Question 5.5.4

5.5. Information Bottleneck

What trade-off does the Information Bottleneck seek?

One correct optionAMinimize both

I(T;Y)

and

I(X;T)

as far as possibleBMaximize the size of the representationCPreserve information about

Y

while reducing unnecessary information about

X

DMaximize

I(X;T)

subject to a bound on

I(T;Y)

A) Minimize both $X$ and $I(X;T)$ as far as possible

B) Maximize the size of the representation

C) Preserve information about $Y$ while reducing unnecessary information about $X$

D) Maximize $I(X;T)$ subject to a bound on $I(T;Y)$

Answer and explanation

Correct answer: C. Relevance is maximized while rate is penalized. Option D swaps the roles of the two terms, which turns the objective into its opposite.

Question 5.9.3

5.9. Map of self-supervised methods

Which description of DINO is most accurate?

One correct optionAAn exact kNN mutual-information estimatorBSelf-distillation without labels, rather than standard InfoNCE with negative pairsCA causal discovery algorithmDContrastive learning with a very large number of negatives

A) An exact kNN mutual-information estimator

B) Self-distillation without labels, rather than standard InfoNCE with negative pairs

C) A causal discovery algorithm

D) Contrastive learning with a very large number of negatives

Answer and explanation

Correct answer: B. Not every self-supervised method is contrastive classification against negatives. Reading the whole family as MI maximization is a useful lens, not a theorem about each method.

Question 5.C.1

Computational block

A binary symmetric channel has crossover probability $\varepsilon=0.1$ and a uniform input. Compute $I(X;Y)$ in bits. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 0.531 (tolerance 0.002). Here I = 1 − h₂(0.1) = 1 − 0.469 = 0.531 bits. At ε = 0.5 this would be 0 and at ε = 1 it returns to a full bit, so the curve is not monotone in the noise level.

Question 5.C.2

Computational block

For the XOR example $Y=X_1\oplus X_2$ with independent fair bits, compute the conditional mutual information $I(X_1;Y\mid X_2)$ in bits. (tolerance 0.005)

AnswerAbsolute tolerance: ±0.005

Answer and explanation

Numeric answer: 1 (tolerance 0.005). Given X₂ the target is a deterministic function of X₁, so the conditional MI is a full bit, while the unconditional I(X₁;Y) is exactly 0. Conditioning can raise mutual information, and here it raises it from nothing to the maximum.

Question 5.C.3

Computational block

In an InfoNCE setup with $N=256$ candidates the loss is $\mathcal L_{\mathrm{NCE}}=2.0$ nats. Compute the resulting lower bound on $I(X;Y)$ in nats. (tolerance 0.005)

AnswerAbsolute tolerance: ±0.005

Answer and explanation

Numeric answer: 3.545 (tolerance 0.005). The bound is log 256 − 2.0 = 5.545 − 2.0 = 3.545 nats. Its ceiling is 5.545 nats, so if the true MI is larger the bound cannot report it however well the critic is trained.

Question 5.C.4

Computational block

For a pair with $p(x,y)=0.002$ , $p(x)=0.1$ and $p(y)=0.05$ , compute the pointwise mutual information in bits. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: -1.322 (tolerance 0.002). The ratio is 0.002/0.005 = 0.4, and log₂ 0.4 = −1.322 bits. The pair occurs less often than independence would predict, so its PMI is negative even though the average over all pairs cannot be.