Information Theory for ML14 / 16

Module 14. Computer Vision Through Information Theory

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 25 · Correct: 0

Question 14.1.1

14.1. Why computer vision is a useful testbed for information theory

Where is the connection between computer vision and information theory most operationally exact?

One correct optionAIn the claim that every vision encoder maximizes MIBIn treating an attention map as a number of transmitted bitsCIn a learned image codec whose probability model controls an entropy coder and produces a bitstreamDIn identifying FID with image entropy

A) In the claim that every vision encoder maximizes MI

B) In treating an attention map as a number of transmitted bits

C) In a learned image codec whose probability model controls an entropy coder and produces a bitstream

D) In identifying FID with image entropy

Answer and explanation

Correct answer: C. A learned codec explicitly defines a source, quantized latent, probability model, code, and physical bitstream.

Question 14.2.1

14.2. Learned image compression: entropy models and actual bitstreams

What does

\mathbb E[-\log_2p_\psi(\widehat Y)]

mean when the entropy model and coder agree?

One correct optionAAn estimate of expected code length for the quantized latentBNumber of decoder parametersCEntropy of raw RGB pixelsDA regularizer with no operational meaning

A) An estimate of expected code length for the quantized latent

B) Number of decoder parameters

C) Entropy of raw RGB pixels

D) A regularizer with no operational meaning

Answer and explanation

Correct answer: A. Cross-entropy on actual symbols determines achievable average code length up to finite overhead.

Question 14.2.2

14.2. Learned image compression: entropy models and actual bitstreams

Why is rounding often replaced by uniform noise during codec training?

One correct optionATo obtain a differentiable relaxation of quantizationBTo guarantee a smaller physical fileCTo make deployment-time encoding stochasticDTo remove the need for an entropy model

A) To obtain a differentiable relaxation of quantization

B) To guarantee a smaller physical file

C) To make deployment-time encoding stochastic

D) To remove the need for an entropy model

Answer and explanation

Correct answer: A. Additive uniform noise enables gradient-based optimization but does not replace bitstream measurement.

Question 14.2.3

14.2. Learned image compression: entropy models and actual bitstreams

Which statement about a hyperprior is correct?

One correct optionA

H(Z)+H(Y\mid Z)<H(Y)

always holdsBA hyperprior can reduce model mismatch of a restricted entropy model, while the cost of

Z

remains part of the rateCA hyperprior reduces true source entropy for freeDSending

Z

violates the chain rule but is useful in practice

A) $H(Z)+H(Y\mid Z)<H(Y)$ always holds

B) A hyperprior can reduce model mismatch of a restricted entropy model, while the cost of $Z$ remains part of the rate

C) A hyperprior reduces true source entropy for free

D) Sending $Z$ violates the chain rule but is useful in practice

Answer and explanation

Correct answer: B. The gain is relative to a limited factorization, not to the ideal joint code.

Question 14.2.4

14.2. Learned image compression: entropy models and actual bitstreams

What is the main tradeoff of an autoregressive entropy model in learned compression?

One correct optionAFewer parameters versus larger MSEBLarger batch versus smaller datasetCLower accuracy versus higher calibrationDBetter conditional prediction versus sequential decoding latency

A) Fewer parameters versus larger MSE

B) Larger batch versus smaller dataset

C) Lower accuracy versus higher calibration

D) Better conditional prediction versus sequential decoding latency

Answer and explanation

Correct answer: D. Previously decoded symbols improve prediction but introduce sequential dependence.

Question 14.3.1

14.3. Distortion and perceptual quality

What does LPIPS measure?

One correct optionAAn f-divergence between image distributionsBExact human utilityCThe number of latent-code bitsDA per-image distance in a learned feature space

A) An f-divergence between image distributions

B) Exact human utility

C) The number of latent-code bits

D) A per-image distance in a learned feature space

Answer and explanation

Correct answer: D. LPIPS compares features of a particular image pair; it is not a distributional discrepancy.

Question 14.3.2

14.3. Distortion and perceptual quality

Which description of FID is most accurate?

One correct optionASquared

W_2

distance between Gaussian approximations to distributions of Inception featuresBMSE between each generated image and its nearest real imageCMutual information between generator and datasetDAn f-divergence between raw pixel distributions

A) Squared $H(Z)+H(Y\mid Z)<H(Y)$ distance between Gaussian approximations to distributions of Inception features

B) MSE between each generated image and its nearest real image

C) Mutual information between generator and dataset

D) An f-divergence between raw pixel distributions

Answer and explanation

Correct answer: A. FID is a distribution-level feature-space metric, not a per-image fidelity measure.

Question 14.3.3

14.3. Distortion and perceptual quality

What does the rate–distortion–perception framework state?

One correct optionAPerception always improves together with MSEBA realism constraint generally shrinks the achievable rate–distortion regionCA universal formula

D(R,P)=D(R)+P

holdsDGAN loss guarantees an optimal codec

A) Perception always improves together with MSE

B) A realism constraint generally shrinks the achievable rate–distortion region

C) A universal formula $D(R,P)=D(R)+P$ holds

D) GAN loss guarantees an optimal codec

Answer and explanation

Correct answer: B. The exact frontier depends on the source, distortion, perception measure, and operational assumptions.

Question 14.4.1

14.4. Diffusion models: Gaussian noising, denoising, and I–MMSE

What is the population optimum of an MSE predictor of clean $X_0$ from $X_t$ ?

One correct optionAThe conditional mean

\mathbb E[X_0\mid X_t]

BThe MAP estimate for every distributionCThe score without any transformationDAn independent prior sample

A) The conditional mean $H(Z)+H(Y\mid Z)<H(Y)$

B) The MAP estimate for every distribution

C) The score without any transformation

D) An independent prior sample

Answer and explanation

Correct answer: A. The conditional mean minimizes expected squared error.

Question 14.4.2

14.4. Diffusion models: Gaussian noising, denoising, and I–MMSE

For

X_t=\alpha_tX_0+\sigma_t\varepsilon,

which identity is correct?

One correct optionAThe score always equals

X_t/\sigma_t

BThe score is unrelated to denoisingC

\nabla\log p_t(x_t)=-\mathbb E[\varepsilon\mid x_t]/\sigma_t

\mathbb E[\varepsilon\mid x_t]=0

for every data distribution

A) The score always equals $H(Z)+H(Y\mid Z)<H(Y)$

B) The score is unrelated to denoising

C) $\nabla\log p_t(x_t)=-\mathbb E[\varepsilon\mid x_t]/\sigma_t$

D) $\mathbb E[\varepsilon\mid x_t]=0$ for every data distribution

Answer and explanation

Correct answer: C. The Gaussian score identity connects the noise predictor, clean posterior mean, and score.

Question 14.4.3

14.4. Diffusion models: Gaussian noising, denoising, and I–MMSE

What coefficient appears in I–MMSE for information in nats?

One correct optionAThe coefficient depends only on dimensionB

\frac{dI}{d\gamma}=2\operatorname{mmse}(\gamma)

I=\operatorname{mmse}

\frac{dI}{d\gamma}=\frac12\operatorname{mmse}(\gamma)

A) The coefficient depends only on dimension

B) $\frac{dI}{d\gamma}=2\operatorname{mmse}(\gamma)$

C) $Z$

D) $\frac{dI}{d\gamma}=\frac12\operatorname{mmse}(\gamma)$

Answer and explanation

Correct answer: D. In bits, an additional factor 1/ln 2 appears.

Question 14.4.4

14.4. Diffusion models: Gaussian noising, denoising, and I–MMSE

What does I–MMSE not determine by itself?

One correct optionAThe relation between MI and optimal estimation errorBMMSE for a Gaussian inputCThe derivative of MI with respect to SNRDThe optimal diffusion noise schedule and sampling solver

A) The relation between MI and optimal estimation error

B) MMSE for a Gaussian input

C) The derivative of MI with respect to SNR

D) The optimal diffusion noise schedule and sampling solver

Answer and explanation

Correct answer: D. Those engineering choices require additional objectives and analysis.

Question 14.5.1

14.5. CLIP and contrastive image–text alignment

What is CLIP's core training task?

One correct optionAPixel reconstructionBContrastive classification of matching image–text pairs within a batchCDirect minimization of

I(X;Z)

DSegmentation-mask prediction

A) Pixel reconstruction

B) Contrastive classification of matching image–text pairs within a batch

C) Direct minimization of $D(R,P)=D(R)+P$

D) Segmentation-mask prediction

Answer and explanation

Correct answer: B. CLIP learns to distinguish matched images and captions from candidates.

Question 14.5.2

14.5. CLIP and contrastive image–text alignment

What does $\log K$ bound in InfoNCE?

One correct optionATrue mutual information of representationsBThe ceiling of the specific lower bound

\log K-\mathcal L

CMaximum downstream accuracyDEmbedding dimension

A) True mutual information of representations

B) The ceiling of the specific lower bound $\frac{dI}{d\gamma}=2\operatorname{mmse}(\gamma)$

C) Maximum downstream accuracy

D) Embedding dimension

Answer and explanation

Correct answer: B. True MI can exceed log K; the ceiling applies to the bound.

Question 14.5.3

14.5. CLIP and contrastive image–text alignment

Why is CLIP not automatically a strict Information Bottleneck objective?

One correct optionAIB applies only to discrete dataBCLIP does not use imagesCIts objective has no explicit penalty on

I(X_I;Z_I)

DSoftmax is incompatible with MI

A) IB applies only to discrete data

B) CLIP does not use images

C) Its objective has no explicit penalty on $D(R,P)=D(R)+P$

D) Softmax is incompatible with MI

Answer and explanation

Correct answer: C. Contrastive alignment is not the full IB tradeoff.

Question 14.5.4

14.5. CLIP and contrastive image–text alignment

What principally distinguishes SigLIP loss from CLIP softmax loss?

One correct optionASigLIP uses no negatives at allBSigLIP uses pairwise sigmoid objectives without one global softmax normalizationCSigLIP is a reconstruction lossDSigLIP always requires a million-example batch

A) SigLIP uses no negatives at all

B) SigLIP uses pairwise sigmoid objectives without one global softmax normalization

C) SigLIP is a reconstruction loss

D) SigLIP always requires a million-example batch

Answer and explanation

Correct answer: B. The loss is decoupled from a single global batch-wise normalization.

Question 14.6.1

14.6. Self-supervised visual representations: invariances and collapse prevention

What role do augmentations play in visual SSL?

One correct optionAThey only enlarge the datasetBThey always increase MI with the task labelCThey guarantee preservation of every downstream labelDThey define assumed representation invariances

A) They only enlarge the dataset

B) They always increase MI with the task label

C) They guarantee preservation of every downstream label

D) They define assumed representation invariances

Answer and explanation

Correct answer: D. If an augmentation removes task-relevant information, DPI prevents the encoder from recovering it.

Question 14.6.2

14.6. Self-supervised visual representations: invariances and collapse prevention

How should BYOL and DINO be classified?

One correct optionAExact MI estimatorsBSelf-distillation or cross-view prediction methods without the standard InfoNCE boundCMasked-pixel modelsDShannon source codes

A) Exact MI estimators

B) Self-distillation or cross-view prediction methods without the standard InfoNCE bound

C) Masked-pixel models

D) Shannon source codes

Answer and explanation

Correct answer: B. They can be analyzed informationally, but their objectives are not standard InfoNCE estimates.

Question 14.6.3

14.6. Self-supervised visual representations: invariances and collapse prevention

What does the VICReg variance term actually guarantee?

One correct optionAIt prevents a constant representation along the corresponding batch dimensionsBIt makes coordinates independentCIt maximizes semantic MIDIt guarantees linear separability of every target

A) It prevents a constant representation along the corresponding batch dimensions

B) It makes coordinates independent

C) It maximizes semantic MI

D) It guarantees linear separability of every target

Answer and explanation

Correct answer: A. Positive variance is not task sufficiency, and decorrelation is not general independence.

Question 14.6.4

14.6. Self-supervised visual representations: invariances and collapse prevention

What is the MAE pretext task?

One correct optionAImage–text matchingBPairwise sigmoid classificationCReconstructing masked image patches from visible contextDAdversarial generation

A) Image–text matching

B) Pairwise sigmoid classification

C) Reconstructing masked image patches from visible context

D) Adversarial generation

Answer and explanation

Correct answer: C. MAE is masked modeling rather than a standard contrastive objective.

Question 14.7.1

14.7. Compressing a CV model: quantization, pruning, and distillation

Why does nominal INT4 not necessarily mean exactly four bits per original weight in a file?

One correct optionAEvery weight becomes float32 after packingBINT4 is prohibited for vision modelsCScales, zero points, grouping, and alignment must also be storedDEntropy is always eight bits

A) Every weight becomes float32 after packing

B) INT4 is prohibited for vision models

C) Scales, zero points, grouping, and alignment must also be stored

D) Entropy is always eight bits

Answer and explanation

Correct answer: C. Actual rate is determined by the complete serialized representation.

Question 14.7.2

14.7. Compressing a CV model: quantization, pruning, and distillation

Why is weight MSE an incomplete distortion measure for model quantization?

One correct optionAEqual weight errors can affect logits and downstream quality differentlyBIt is always infiniteCMSE cannot be computedDMSE applies only to images

A) Equal weight errors can affect logits and downstream quality differently

B) It is always infinite

C) MSE cannot be computed

D) MSE applies only to images

Answer and explanation

Correct answer: A. Task-aware distortion must account for network sensitivity.

Question 14.7.3

14.7. Compressing a CV model: quantization, pruning, and distillation

Why can 50% unstructured sparsity fail to compress a model?

One correct optionAIndex and irregular-storage costs can exceed the savings in valuesBEvery zero requires 32 extra bitsCPruning doubles parameter countDSparse models cannot be serialized

A) Index and irregular-storage costs can exceed the savings in values

B) Every zero requires 32 extra bits

C) Pruning doubles parameter count

D) Sparse models cannot be serialized

Answer and explanation

Correct answer: A. Physical size and latency depend on format and kernels, not only the zero fraction.

Question 14.8.1

14.8. One CV pipeline, four different information budgets

Which statement about a retrieval pipeline is false?

One correct optionAProduct quantization can be evaluated by bytes per item and recall@KBModel quantization should be evaluated end to endCCosine distance between embeddings is an f-divergence between distributionsDA learned codec has an operational bitstream

A) Product quantization can be evaluated by bytes per item and recall@K

B) Model quantization should be evaluated end to end

C) Cosine distance between embeddings is an f-divergence between distributions

D) A learned codec has an operational bitstream

Answer and explanation

Correct answer: C. Cosine and Euclidean vector distances are not f-divergences between probability distributions.

Question 14.9.1

14.9. Mathematical deepening: exact hyperprior accounting

Why does introducing a side variable $Z$ not by itself reduce the fundamental entropy of the coded $Y$ ?

One correct optionAArithmetic coding ignores conditional probabilitiesBThe decoder never receives the hyperpriorCJoint distributions are always uniformD

H(Y,Z)=H(Y)+H(Z\mid Y)\ge H(Y)

; the hyperprior helps a restricted model class rather than reducing fundamental entropy

A) Arithmetic coding ignores conditional probabilities

B) The decoder never receives the hyperprior

C) Joint distributions are always uniform

D) $H(Y,Z)=H(Y)+H(Z\mid Y)\ge H(Y)$ ; the hyperprior helps a restricted model class rather than reducing fundamental entropy

Answer and explanation

Correct answer: D. Side information simplifies modeling, but its own cost must be paid.