Information Theory for ML9 / 16

Module 9. Rate–distortion and the Information Bottleneck: managing information loss

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 29 · Correct: 0

Question 9.2.1

9.2. Rate–distortion: the minimum rate at an allowed loss

What is minimized in the definition of the rate–distortion function $R(D)$ ?

One correct optionA

I(X;\widehat X)

over test channels satisfying the average-distortion constraintBOnly the reconstruction entropy

H(\widehat X)

CThe number of decoder parametersDThe worst single-example error independently of the setup

A) $I(X;\widehat X)$ over test channels satisfying the average-distortion constraint

B) Only the reconstruction entropy $H(\widehat X)$

C) The number of decoder parameters

D) The worst single-example error independently of the setup

Answer and explanation

Correct answer: A. After optimization, mutual information has the operational meaning of the minimum asymptotic rate.

Question 9.2.2

9.2. Rate–distortion: the minimum rate at an allowed loss

What is a test channel $P_{\widehat X\mid X}$ ?

One correct optionANecessarily a physical noisy communication channelBA conditional distribution of allowed reconstructions for each source outcomeCOnly a deterministic quantizerDThe training-data distribution

A) Necessarily a physical noisy communication channel

B) A conditional distribution of allowed reconstructions for each source outcome

C) Only a deterministic quantizer

D) The training-data distribution

Answer and explanation

Correct answer: B. It is a mathematical object in the variational definition of R(D) and need not model a physical link.

Question 9.2.3

9.2. Rate–distortion: the minimum rate at an allowed loss

When is the simple statement $R(0)=H(X)$ valid?

One correct optionAFor every continuous source under MSEBOnly for a Gaussian sourceCFor a finite discrete source when zero distortion requires exact reconstructionDAlways, independently of the distortion

A) For every continuous source under MSE

B) Only for a Gaussian source

C) For a finite discrete source when zero distortion requires exact reconstruction

D) Always, independently of the distortion

Answer and explanation

Correct answer: C. For a continuous source under squared error, R(D) typically diverges as D↓0.

Question 9.2.4

9.2. Rate–distortion: the minimum rate at an allowed loss

For $X\sim\mathcal N(0,\sigma^2)$ under squared error, what is $R(D)$ for $0<D<\sigma^2$ in bits?

One correct optionA

\log_2(\sigma^2/D)

\sigma^2-D

h_2(D)

\tfrac12\log_2(\sigma^2/D)

A) $I(X;\widehat X)$

B) $\sigma^2-D$

C) $h_2(D)$

D) $\tfrac12\log_2(\sigma^2/D)$

Answer and explanation

Correct answer: D. The factor 1/2 corresponds to one real-valued source symbol.

Question 9.2.5

9.2. Rate–distortion: the minimum rate at an allowed loss

Why is the choice of distortion part of the model?

One correct optionABecause the theory optimizes rate only after a cost of error has been specifiedBBecause all distortion measures are equivalentCBecause MSE always matches human perceptionDBecause distortion does not affect the optimal code

A) Because the theory optimizes rate only after a cost of error has been specified

B) Because all distortion measures are equivalent

C) Because MSE always matches human perception

D) Because distortion does not affect the optimal code

Answer and explanation

Correct answer: A. The theory does not decide which differences matter to the user.

Question 9.3.1

9.3. Information Bottleneck: preserve what is needed for $Y$

What does $I(X;T)-\beta I(T;Y)$ balance?

One correct optionAOptimizer speed and batch sizeBCompression of the input and preservation of target informationCReconstruction error onlyDNumber of classes and network depth

A) Optimizer speed and batch size

B) Compression of the input and preservation of target information

C) Reconstruction error only

D) Number of classes and network depth

Answer and explanation

Correct answer: B. I(X;T) plays the role of rate, while I(T;Y) measures relevance.

Question 9.3.2

9.3. Information Bottleneck: preserve what is needed for $Y$

Under the convention $I(X;T)-\beta I(T;Y)$ , what does increasing $\beta$ do?

One correct optionAIt always reduces

I(T;Y)

BNothing, because

\beta

cancelsCIt values preservation of information about

Y

more stronglyDIt automatically makes

T=X

A) It always reduces $I(X;\widehat X)$

B) Nothing, because $\beta$ cancels

C) It values preservation of information about $Y$ more strongly

D) It automatically makes $T=X$

Answer and explanation

Correct answer: C. Other parameterizations exist, so the objective itself must be read.

Question 9.3.3

9.3. Information Bottleneck: preserve what is needed for $Y$

Which distortion arises naturally in IB?

One correct optionAOnly Euclidean distance between

x

and

t

BHamming distance between cluster indicesCMSE between network weightsD

D_{\mathrm{KL}}(P(Y\mid x)\|P(Y\mid t))

A) Only Euclidean distance between $I(X;\widehat X)$ and $t$

B) Hamming distance between cluster indices

C) MSE between network weights

D) $\beta$

Answer and explanation

Correct answer: D. It measures how much the representation damages the predictive distribution of the target.

Question 9.3.4

9.3. Information Bottleneck: preserve what is needed for $Y$

When may two inputs share an IB code with little loss of relevance?

One correct optionAWhen their

P(Y\mid x)

distributions are similarBOnly when their byte representations are identicalCWhen their Euclidean distance is maximalDAlways, provided

T

is low-dimensional

A) When their $I(X;\widehat X)$ distributions are similar

B) Only when their byte representations are identical

C) When their Euclidean distance is maximal

D) Always, provided $T$ is low-dimensional

Answer and explanation

Correct answer: A. IB preserves task-relevant distinctions, not necessarily every input detail.

Question 9.3.5

9.3. Information Bottleneck: preserve what is needed for $Y$

Why is the classical IB equation self-consistent?

One correct optionAIt contains no probabilitiesB

P(t)

and

P(Y\mid t)

themselves depend on the unknown

P(t\mid x)

CIts solution is always uniqueDIt applies only to Gaussian variables

A) It contains no probabilities

B) $P(t)$ and $P(Y\mid t)$ themselves depend on the unknown $P(t\mid x)$

C) Its solution is always unique

D) It applies only to Gaussian variables

Answer and explanation

Correct answer: B. The iterative algorithm updates these quantities jointly.

Question 9.4.1

9.4. Variational Information Bottleneck: a loss we can train

What is classification cross-entropy in the VIB interpretation?

One correct optionAExactly

-I(T;Y)

for every classifierBAn upper bound on

H(Y)

CA trainable variational bound related to

H(Y\mid T)

and relevanceDThe actual number of transmitted bits

A) Exactly $I(X;\widehat X)$ for every classifier

B) An upper bound on $H(Y)$

C) A trainable variational bound related to $H(Y\mid T)$ and relevance

D) The actual number of transmitted bits

Answer and explanation

Correct answer: C. The gap is a KL divergence between the true P(Y|T) and the variational decoder.

Question 9.4.2

9.4. Variational Information Bottleneck: a loss we can train

What is the exact decomposition of the average encoder KL to a prior $r(T)$ ?

One correct optionA

I(X;T)+D_{\mathrm{KL}}(q(T)\|r(T))

BOnly

H(T)

I(T;Y)-I(X;Y)

DReconstruction error plus a constant

A) $I(X;\widehat X)$

B) Only $h_2(D)$

C) $I(T;Y)-I(X;Y)$

D) Reconstruction error plus a constant

Answer and explanation

Correct answer: A. Hence this KL upper-bounds I(X;T).

Question 9.4.3

9.4. Variational Information Bottleneck: a loss we can train

When does the VIB KL-rate equal $I(X;T)$ exactly?

One correct optionAFor every standard-normal priorBWhen the prior equals the aggregate posterior

q(T)

COnly for a deterministic encoderDWhen classifier accuracy is zero

A) For every standard-normal prior

B) When the prior equals the aggregate posterior $P(t)$

C) Only for a deterministic encoder

D) When classifier accuracy is zero

Answer and explanation

Correct answer: B. Otherwise an extra mismatch cost remains.

Question 9.5.1

9.5. VAE: a rate–distortion view of the ELBO, not literal supervised IB

How does the averaged negative ELBO decompose?

One correct optionAAccuracy plus calibration errorBMutual information onlyCEntropy plus perplexityDDistortion plus rate

A) Accuracy plus calibration error

B) Mutual information only

C) Entropy plus perplexity

D) Distortion plus rate

Answer and explanation

Correct answer: D. Distortion is defined by the decoder likelihood, while rate is the average posterior-to-prior KL.

Question 9.5.2

9.5. VAE: a rate–distortion view of the ELBO, not literal supervised IB

Which statement about VAEs and supervised IB is most accurate?

One correct optionAA VAE naturally admits a rate–distortion interpretation but is not literally supervised IB with an external

Y

BEvery VAE exactly maximizes

I(Z;Y)

CThe ELBO contains no bottleneckD

\beta

always has the same meaning in VAE and IB

A) A VAE naturally admits a rate–distortion interpretation but is not literally supervised IB with an external $I(X;\widehat X)$

B) Every VAE exactly maximizes $I(Z;Y)$

C) The ELBO contains no bottleneck

D) $\beta$ always has the same meaning in VAE and IB

Answer and explanation

Correct answer: A. The constructions are related, but their variables and optimization problems differ.

Question 9.6.1

9.6. Where rate–distortion appears in real ML code

When is $\mathbb E[-\log_2p_\psi(\widehat Z)]$ close to a real coding rate?

One correct optionAWhen the entropy model is used by a compatible arithmetic/ANS coder and overhead is accounted forBAlways, even without an entropy coderCOnly when the latent is continuous and unquantizedDNever

A) When the entropy model is used by a compatible arithmetic/ANS coder and overhead is accounted for

B) Always, even without an entropy coder

C) Only when the latent is continuous and unquantized

D) Never

Answer and explanation

Correct answer: A. A concrete entropy coder turns NLL into actual bits.

Question 9.6.2

9.6. Where rate–distortion appears in real ML code

What is not automatically included in the phrase “4 bits per parameter”?

One correct optionAThe quantized values themselvesBScales, zero-points, codebooks, padding, and metadataCThe number of model layersDTraining labels

A) The quantized values themselves

B) Scales, zero-points, codebooks, padding, and metadata

C) The number of model layers

D) Training labels

Answer and explanation

Correct answer: B. Nominal bit-width and actual rate can differ.

Question 9.6.3

9.6. Where rate–distortion appears in real ML code

Why can weight MSE be a poor distortion surrogate for an LLM?

One correct optionAWeights are always binaryBMSE cannot be computedCEqual errors in different parameter directions can affect logits, NLL, and downstream quality very differentlyDMSE is always equal to KL

A) Weights are always binary

B) MSE cannot be computed

C) Equal errors in different parameter directions can affect logits, NLL, and downstream quality very differently

D) MSE is always equal to KL

Answer and explanation

Correct answer: C. Network sensitivity is anisotropic.

Question 9.6.4

9.6. Where rate–distortion appears in real ML code

How should GPTQ, AWQ, and SmoothQuant be read in rate–distortion language?

One correct optionAAs proofs that theoretical

R(D)

is achievedBAs methods unrelated to distortionCAs universally optimal codes for i.i.d. Gaussian weightsDAs practical ways to choose a surrogate distortion and allocate a budget non-uniformly

A) As proofs that theoretical $I(X;\widehat X)$ is achieved

B) As methods unrelated to distortion

C) As universally optimal codes for i.i.d. Gaussian weights

D) As practical ways to choose a surrogate distortion and allocate a budget non-uniformly

Answer and explanation

Correct answer: D. Their engineering success is not a proof of reaching the classical limit.

Question 9.6.5

9.6. Where rate–distortion appears in real ML code

What is required for an honest size–quality comparison?

One correct optionATrace a frontier over several

\lambda

\beta

, bit-width, or capacity values and measure actual rateBCompare one random operating pointCUse nominal bit-width onlyDIgnore latency and hardware support

A) Trace a frontier over several $I(X;\widehat X)$ , $t$ , bit-width, or capacity values and measure actual rate

B) Compare one random operating point

C) Use nominal bit-width only

D) Ignore latency and hardware support

Answer and explanation

Correct answer: A. A single point does not reveal the trade-off curve.

Question 9.8.1

9.8. The information plane: a normative principle and a descriptive hypothesis

Which statement is rigorous, and which is empirical?

One correct optionAThe IB objective is empirical, while every SGD trajectory follows a theoremBIB as a specified optimization problem is rigorous; a spontaneous IB trajectory of an ordinary network requires empirical validationCBoth follow automatically from DPIDBoth are false

A) The IB objective is empirical, while every SGD trajectory follows a theorem

B) IB as a specified optimization problem is rigorous; a spontaneous IB trajectory of an ordinary network requires empirical validation

C) Both follow automatically from DPI

D) Both are false

Answer and explanation

Correct answer: B. A normative objective and a descriptive model of training are different levels of claim.

Question 9.8.2

9.8. The information plane: a normative principle and a descriptive hypothesis

Why can $I(X;T)$ be problematic for a deterministic continuous layer?

One correct optionAIt is always zeroBIt always equals layer widthCIt is often infinite, while binning or added noise changes the quantity being measuredDIt can be read exactly from training loss

A) It is always zero

B) It always equals layer width

C) It is often infinite, while binning or added noise changes the quantity being measured

D) It can be read exactly from training loss

Answer and explanation

Correct answer: C. Information-plane plots depend on the probabilistic setup and estimator.

Question 9.9.1

Optional block

What is the I–MMSE identity for $Y_\gamma=\sqrt\gamma X+N$ with information in nats?

One correct optionA

\dfrac d{d\gamma}I(X;Y_\gamma)=\dfrac12\operatorname{mmse}(\gamma)

I(X;Y_\gamma)=\operatorname{mmse}(\gamma)^2

CThe derivative equals

H(X)

\operatorname{mmse}(\gamma)

is always constant

A) $I(X;\widehat X)$

B) $I(X;Y_\gamma)=\operatorname{mmse}(\gamma)^2$

C) The derivative equals $H(X)$

D) $\operatorname{mmse}(\gamma)$ is always constant

Answer and explanation

Correct answer: A. In bits, an additional factor 1/ln 2 appears.

Question 9.9.2

Optional block

Why can we not write $h(X)=\tfrac12\int_0^\infty\operatorname{mmse}(\gamma)d\gamma$ for a continuous $X$ without qualification?

One correct optionAMMSE is negativeBThe integral corresponding to

I(X;Y_\Gamma)

generally diverges as

\Gamma\to\infty

and requires renormalizationCI–MMSE holds only for discrete variablesDDifferential entropy is always infinite

A) MMSE is negative

B) The integral corresponding to $P(t)$ generally diverges as $\Gamma\to\infty$ and requires renormalization

C) I–MMSE holds only for discrete variables

D) Differential entropy is always infinite

Answer and explanation

Correct answer: B. For a Gaussian input, the integral grows as 1/2 ln(1+Γ).

Question 9.9.3

Optional block

What does Tweedie's formula rigorously provide on a Gaussian noising path?

One correct optionAThe required number of diffusion sampling stepsBA universal perceptual noise scheduleCA proof that flow matching and IB are equivalentDA relation between the true score of the perturbed distribution and the posterior mean, the optimal MSE denoiser

A) The required number of diffusion sampling steps

B) A universal perceptual noise schedule

C) A proof that flow matching and IB are equivalent

D) A relation between the true score of the perturbed distribution and the posterior mean, the optimal MSE denoiser

Answer and explanation

Correct answer: D. The other algorithmic choices do not follow from this identity alone.

Question 9.C.1

Computational block

A Bernoulli(0.3) source is coded under Hamming distortion with $D=0.1$ . Compute $R(D)$ in bits. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 0.412 (tolerance 0.002). Here R(D) = h₂(0.3) − h₂(0.1) = 0.881 − 0.469 = 0.412 bits. For D ≥ 0.3 the rate drops to zero, because always emitting the more likely symbol already meets the distortion budget.

Question 9.C.2

Computational block

A Gaussian source with variance $\sigma^2=4$ is coded under squared error with $D=1$ . Compute $R(D)$ in bits. (tolerance 0.005)

AnswerAbsolute tolerance: ±0.005

Answer and explanation

Numeric answer: 1 (tolerance 0.005). Here R(D) = ½ log₂(4/1) = 1 bit. Every doubling of the allowed distortion saves exactly half a bit, and the rate diverges as D approaches zero, which is why a continuous source has no finite lossless rate.

Question 9.C.3

Computational block

(optional)

In the Gaussian channel $Y_\gamma=\sqrt\gamma X+N$ with $X,N\sim\mathcal N(0,1)$ , compute $\operatorname{mmse}(\gamma)$ at $\gamma=3$ . (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 0.25 (tolerance 0.002). The posterior variance is 1/(1 + γ) = 0.25. The observation has removed three quarters of the prior variance. In nats, the I–MMSE derivative is one half of this value: dI/dγ = ½ mmse(γ) = 0.125.

Question 9.C.4

Computational block

(optional)

For the same channel, compute $I(X;Y_\gamma)$ at $\gamma=3$ in nats. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 0.693 (tolerance 0.002). Here I = ½ ln(1 + 3) = ½ ln 4 = 0.693 nats. Differentiating gives 1/(2(1 + γ)) = 0.125, which is exactly half the MMSE computed above, as the I–MMSE identity requires.