Information Theory for ML4 / 16

Module 4. Jensen's inequality: when averaging meets nonlinearity

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 24 · Correct: 0

Question 4.2.1

4.2. Jensen's inequality

Which inequality is valid for a convex function $\phi$ ?

One correct optionA

\phi(\mathbb E X)\ge\mathbb E\phi(X)

\phi(\mathbb E X)\le\mathbb E\phi(X)

\phi(\mathbb E X)=\mathbb E\phi(X)

for every

X

\phi(\mathbb E X)\le\phi(\mathbb E\phi(X))

A) $\phi(\mathbb E X)\ge\mathbb E\phi(X)$

B) $\phi(\mathbb E X)\le\mathbb E\phi(X)$

C) $\phi(\mathbb E X)=\mathbb E\phi(X)$ for every $X$

D) $\phi(\mathbb E X)\le\phi(\mathbb E\phi(X))$

Answer and explanation

Correct answer: B. For a convex function the chord lies above the graph, so averaging the arguments first gives the smaller value. The direction reverses for a concave function, which is where option A comes from.

Question 4.2.2

4.2. Jensen's inequality

Under strict convexity, when can equality hold in Jensen's inequality?

One correct optionAWhen the distribution of

X

is symmetricBWhen

\mathbb E X=0

CWhen

\phi

is affine on the support of

X

DWhen

X

is almost surely constant

A) When the distribution of $\phi(\mathbb E X)\ge\mathbb E\phi(X)$ is symmetric

B) When $\mathbb E X=0$

C) When $\phi$ is affine on the support of $X$

D) When $X$ is almost surely constant

Answer and explanation

Correct answer: D. Affineness on the support is the general equality condition, but strict convexity rules it out on any interval, so only a degenerate X remains. Symmetry and a zero mean say nothing about the gap.

Question 4.2.3

4.2. Jensen's inequality

What is the Jensen gap for $\phi(x)=x^2$ ?

One correct optionA

\mathbb E X

(\mathbb E X)^2

\operatorname{Var}(X)

DThe standard deviation of

X

A) $\phi(\mathbb E X)\ge\mathbb E\phi(X)$

B) $(\mathbb E X)^2$

C) $\operatorname{Var}(X)$

D) The standard deviation of $X$

Answer and explanation

Correct answer: C. The gap is E[X²] − (E X)², which is the variance by definition. Note that it is the variance itself and not its square root: the gap has the units of φ, not of X.

Question 4.2.4

4.2. Jensen's inequality

Which function is concave on $(0,\infty)$ ?

One correct optionA

\log x

-\log x

e^x

x\log x

A) $\phi(\mathbb E X)\ge\mathbb E\phi(X)$

B) $-\log x$

C) $e^x$

D) $x\log x$

Answer and explanation

Correct answer: A. Its second derivative is −1/x², which is negative. The other three are convex, and the pair log x and −log x is exactly why the direction of every inequality in this module has to be tracked.

Question 4.3.1

4.3. Ensembles: an exact guarantee for log loss

What does probability averaging guarantee for the NLL of a fixed example?

One correct optionAThe mixture NLL is no larger than the NLL of the best memberBThe mixture improves top-1 accuracyCThe mixture NLL equals the weighted mean NLL of the membersDThe mixture NLL is no larger than the weighted mean NLL of the members

A) The mixture NLL is no larger than the NLL of the best member

B) The mixture improves top-1 accuracy

C) The mixture NLL equals the weighted mean NLL of the members

D) The mixture NLL is no larger than the weighted mean NLL of the members

Answer and explanation

Correct answer: D. This is Jensen for the convex function −log, applied to the observed-class probability. The comparison is with the average member, not with the best one: a strong member can easily beat the mixture.

Question 4.3.2

4.3. Ensembles: an exact guarantee for log loss

What distributional average corresponds to averaging logits before the softmax?

One correct optionAAn arithmetic mean of the probabilitiesBA normalized geometric mean of the probabilitiesCA componentwise maximumDA componentwise median

A) An arithmetic mean of the probabilities

B) A normalized geometric mean of the probabilities

C) A componentwise maximum

D) A componentwise median

Answer and explanation

Correct answer: B. Averaging in log space and renormalizing is a geometric mean. The arithmetic mixture is a different object and requires explicitly mixing probabilities, not logits.

Question 4.3.3

4.3. Ensembles: an exact guarantee for log loss

Why does improved NLL not guarantee improved top-1 accuracy?

One correct optionANLL uses the probability assigned to the observed class, whereas accuracy uses only the argmaxBAccuracy always equals

e^{-\mathrm{NLL}}

CNLL does not depend on the target labelDNLL is computed on the training set and accuracy on the test set

A) NLL uses the probability assigned to the observed class, whereas accuracy uses only the argmax

B) Accuracy always equals $e^{-\mathrm{NLL}}$

C) NLL does not depend on the target label

D) NLL is computed on the training set and accuracy on the test set

Answer and explanation

Correct answer: A. The two metrics read different information out of the same prediction. A mixture can raise the probability of the correct class without moving it into first place, and it can also lower a wrong maximum without changing the decision.

Question 4.3.4

4.3. Ensembles: an exact guarantee for log loss

Which statement about the two ensemble constructions is correct?

One correct optionAProbability averaging always beats logit averagingBLogit averaging always beats probability averagingCNeither universally dominates the otherDThey coincide whenever the members are equally weighted

A) Probability averaging always beats logit averaging

B) Logit averaging always beats probability averaging

C) Neither universally dominates the other

D) They coincide whenever the members are equally weighted

Answer and explanation

Correct answer: C. Each satisfies its own Jensen guarantee relative to the average member, and the module gives two examples where the winner is different. Equal weights do not make them coincide.

Question 4.4.1

4.4. The ELBO: the logarithm sits outside hidden explanations

Which function receives Jensen's inequality in the ELBO derivation shown in this module?

One correct optionAThe convex function

-\log

BThe squareCThe concave logarithmDThe exponential

A) The convex function $\phi(\mathbb E X)\ge\mathbb E\phi(X)$

B) The square

C) The concave logarithm

D) The exponential

Answer and explanation

Correct answer: C. The module uses the concave logarithm: log E[W] ≥ E[log W]. Equivalently, apply Jensen to the convex function −log and then multiply both sides by −1, reversing the inequality. This gives the same lower bound.

Question 4.4.2

4.4. The ELBO: the logarithm sits outside hidden explanations

What is the exact ELBO gap?

One correct optionA

D_{\mathrm{KL}}(q_\phi(z\mid x)\|p_\theta(z\mid x))

D_{\mathrm{KL}}(p_\theta(z\mid x)\|q_\phi(z\mid x))

D_{\mathrm{KL}}(p(z)\|q_\phi(z\mid x))

H(X)

A) $\phi(\mathbb E X)\ge\mathbb E\phi(X)$

B) $D_{\mathrm{KL}}(p_\theta(z\mid x)\|q_\phi(z\mid x))$

C) $D_{\mathrm{KL}}(p(z)\|q_\phi(z\mid x))$

D) $H(X)$

Answer and explanation

Correct answer: A. The direction matters: the approximate posterior comes first. The bound is tight exactly when the variational posterior equals the true posterior.

Question 4.4.3

4.4. The ELBO: the logarithm sits outside hidden explanations

What does the KL term in the standard VAE form of the ELBO compare?

One correct optionAThe approximate posterior with the true posteriorBTwo target labelsCThe reconstruction term with itselfDThe approximate posterior with the prior

A) The approximate posterior with the true posterior

B) Two target labels

C) The reconstruction term with itself

D) The approximate posterior with the prior

Answer and explanation

Correct answer: D. The term follows from the factorization p(x,z) = p(z)p(x|z), so it measures the distance to the prior. The KL against the true posterior is the gap of the bound, which is a different quantity that the objective never computes.

Question 4.4.4

4.4. The ELBO: the logarithm sits outside hidden explanations

What does a higher ELBO not guarantee automatically?

One correct optionAA higher lower bound on

\log p_\theta(x)

BA latent representation that is useful for every downstream taskCA no-larger KL gap when

\log p_\theta(x)

is fixedDA no-smaller value of the variational objective being optimized

A) A higher lower bound on $\phi(\mathbb E X)\ge\mathbb E\phi(X)$

B) A latent representation that is useful for every downstream task

C) A no-larger KL gap when $\log p_\theta(x)$ is fixed

D) A no-smaller value of the variational objective being optimized

Answer and explanation

Correct answer: B. The other three are restatements of what the bound says. Representation utility depends on the task, the architecture, the variational family, and the optimization, none of which appear in the inequality.

Question 4.5.1

4.5. The log-sum inequality and convexity of KL

Which equality condition for the log-sum inequality is correct when $A,B>0$ ?

One correct optionAThe normalized vectors

a_i/A

and

b_i/B

must agreeBIt is enough that

A=B

CAll

a_i

must be equalDAll

b_i

must be equal

A) The normalized vectors $\phi(\mathbb E X)\ge\mathbb E\phi(X)$ and $b_i/B$ must agree

B) It is enough that $A=B$

C) All $a_i$ must be equal

D) All $b_i$ must be equal

Answer and explanation

Correct answer: A. Equivalently, the ratio a_i/b_i must be constant on the relevant support. Matching totals is not enough, because the inequality is about how the mass is distributed, not how much of it there is.

Question 4.5.2

4.5. The log-sum inequality and convexity of KL

After normalization, what directly proves the stated log-sum inequality?

One correct optionAThe central limit theoremBMarkov's inequalityCNonnegativity of KLDThe triangle inequality for KL

A) The central limit theorem

B) Markov's inequality

C) Nonnegativity of KL

D) The triangle inequality for KL

Answer and explanation

Correct answer: C. The difference between the two sides is exactly A times D_KL(P‖Q), which is nonnegative by Gibbs' inequality. KL has no triangle inequality, which is why option D cannot be the mechanism.

Question 4.5.3

4.5. The log-sum inequality and convexity of KL

What does joint convexity of KL state?

One correct optionAKL is linear in both argumentsBThe KL between mixtures is no larger than the mixture of the corresponding KL divergencesCKL is symmetric in its two argumentsDKL is convex in the first argument and concave in the second

A) KL is linear in both arguments

B) The KL between mixtures is no larger than the mixture of the corresponding KL divergences

C) KL is symmetric in its two arguments

D) KL is convex in the first argument and concave in the second

Answer and explanation

Correct answer: B. Mixing hides the component index, and hiding information cannot increase distinguishability. Convexity holds jointly in the pair, not separately with opposite signs.

Question 4.5.4

4.5. The log-sum inequality and convexity of KL

What happens to KL when the same coarse-graining is applied to $P$ and $Q$ ?

One correct optionAIt always increases strictlyBIt necessarily becomes zeroCIt stays exactly the sameDIt cannot increase

A) It always increases strictly

B) It necessarily becomes zero

C) It stays exactly the same

D) It cannot increase

Answer and explanation

Correct answer: D. This is the deterministic special case of data processing. Equality is possible, which is why the statement is "cannot increase" rather than "strictly decreases".

Question 4.6.1

Optional mathematical and engineering deepening

In a local quadratic approximation, what controls the scalar Jensen gap for a twice-differentiable $\phi$ ?

One correct optionAOnly

\mathbb E X

BOnly the sample sizeCApproximately

\tfrac12\phi''(\mathbb E X)\operatorname{Var}(X)

DApproximately

\phi''(\mathbb E X)

alone

A) Only $\phi(\mathbb E X)\ge\mathbb E\phi(X)$

B) Only the sample size

C) Approximately $\tfrac12\phi''(\mathbb E X)\operatorname{Var}(X)$

D) Approximately $\phi''(\mathbb E X)$ alone

Answer and explanation

Correct answer: C. Both ingredients are needed: dispersion in the arguments and curvature of the function. A flat function has no gap however spread out X is, and a constant X has no gap however curved φ is.

Question 4.6.2

Optional mathematical and engineering deepening

When does a probability mixture attain equality with the mean member NLL on a fixed example?

One correct optionAWhen all neural-network parameters are identicalBWhen all models share the same argmaxCWhen the models have equal weightsDWhen all positively weighted models assign the observed class the same probability

A) When all neural-network parameters are identical

B) When all models share the same argmax

C) When the models have equal weights

D) When all positively weighted models assign the observed class the same probability

Answer and explanation

Correct answer: D. The per-example NLL sees only the probability of the observed class, so the full distributions may still differ. A shared argmax is far too weak a condition.

Question 4.7.1

Optional mathematical and engineering deepening

When does coarse-graining preserve KL exactly?

One correct optionAOnly when all groups have equal sizeBWhen

p(x)/q(x)

is constant within every merged groupCWhen

P

and

Q

have the same entropyDWhen every group contains at most two states

A) Only when all groups have equal size

B) When $e^{-\mathrm{NLL}}$ is constant within every merged group

C) When $P$ and $Q$ have the same entropy

D) When every group contains at most two states

Answer and explanation

Correct answer: B. Equivalently, the conditional distributions inside each group agree under P and under Q, so merging destroys nothing that distinguished them. Group sizes are irrelevant.

Question 4.7.2

Optional mathematical and engineering deepening

How can an arithmetic probability mixture be computed stably from model log probabilities?

One correct optionAApply logsumexp to log_probs + log_weightsBAverage the logits and treat the result as one modelCExponentiate, average, and take the logarithm againDRound every probability to zero or one

A) Apply logsumexp to log_probs + log_weights

B) Average the logits and treat the result as one model

C) Exponentiate, average, and take the logarithm again

D) Round every probability to zero or one

Answer and explanation

Correct answer: A. Averaging logits produces a normalized geometric mean instead. The naive exponentiate-average-log route is mathematically right but underflows exactly where the probabilities matter most.

Question 4.C.1

Computational block

Let $X$ take the values $1$ , $2$ and $6$ with probabilities $0.5$ , $0.25$ and $0.25$ . Compute the Jensen gap $\mathbb E[X^2]-(\mathbb E X)^2$ . (tolerance 0.01)

AnswerAbsolute tolerance: ±0.01

Answer and explanation

Numeric answer: 4.25 (tolerance 0.01). E X = 2.5 and E[X²] = 10.5, so the gap is 10.5 − 6.25 = 4.25. For the square the gap is exactly the variance, with no approximation involved.

Question 4.C.2

Computational block

Two models predict

q_1=(0.83,\ 0.01,\ 0.16), \qquad q_2=(0.11,\ 0.88,\ 0.01),

with equal weights, and class 1 is observed. Compute the NLL of the arithmetic probability mixture, rounded to three decimals. (tolerance 0.005)

AnswerAbsolute tolerance: ±0.005

Answer and explanation

Numeric answer: 0.755 (tolerance 0.005). The mixture assigns the observed class (0.83 + 0.11)/2 = 0.47, and −ln 0.47 = 0.755. The mean member NLL is 1.197, so Jensen's guarantee holds. On this example the logit average does better still, at 0.367, which is why neither construction dominates.

Question 4.C.3

Computational block

For a fixed $x$ let $p(x,z=0)=0.09$ and $p(x,z=1)=0.01$ , and take the approximation $q(z\mid x)=(0.5,0.5)$ . Compute the gap between $\log p(x)$ and the ELBO, in nats, rounded to three decimals. (tolerance 0.005)

AnswerAbsolute tolerance: ±0.005

Answer and explanation

Numeric answer: 0.511 (tolerance 0.005). Here log p(x) = −2.303 and the ELBO is −2.813. The difference equals D_KL((0.5, 0.5)‖(0.9, 0.1)) exactly; it is the price of the wrong posterior, not a penalty added by hand.

Question 4.C.4

Computational block

Let

P=(0.5,\ 0.25,\ 0.25), \qquad Q=(0.25,\ 0.25,\ 0.5).

Merge the second and third states into one and compute $D_{\mathrm{KL}}$ between the coarse-grained distributions, in nats, rounded to three decimals. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 0.144 (tolerance 0.002). After merging, P becomes (0.5, 0.5) and Q becomes (0.25, 0.75), giving 0.144 nats against 0.173 before the merge. The drop is the information that distinguished the two merged states.