Information Theory for ML3 / 16

Module 3. Cross-Entropy and KL Divergence

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 29 · Correct: 0

Question 3.2.1

3.2. Cross-entropy: the model's average logarithmic bill

What are the roles of $P$ and $Q$ in $H(P,Q)$ ?

One correct optionA

P

supplies the probabilities inside the logarithm;

Q

is the distribution the expectation is taken overB

P

and

Q

may be interchanged, because both appear inside the same sumC

P

is the distribution the expectation is taken over;

Q

supplies the probabilities inside the logarithmDBoth are model distributions, taken before and after the current gradient step

A) $P$ supplies the probabilities inside the logarithm; $Q$ is the distribution the expectation is taken over

B) $P$ and $Q$ may be interchanged, because both appear inside the same sum

C) $P$ is the distribution the expectation is taken over; $Q$ supplies the probabilities inside the logarithm

D) Both are model distributions, taken before and after the current gradient step

Answer and explanation

Correct answer: C. By definition H(P,Q) is the expectation over X drawn from P of −log q(X). Swapping the two gives H(Q,P), a different number.

Question 3.2.2

3.2. Cross-entropy: the model's average logarithmic bill

What happens if $p(x)>0$ but $q(x)=0$ for some outcome $x$ ?

One correct optionAThe term contributes zero, because the outcome is never sampled from

Q

BBoth

H(P,Q)

and

D_{\mathrm{KL}}(P\|Q)

become infiniteCCross-entropy stays finite, while KL becomes negativeDNothing changes, since the missing mass is redistributed by normalization

A) The term contributes zero, because the outcome is never sampled from $P$

B) Both $H(P,Q)$ and $D_{\mathrm{KL}}(P\|Q)$ become infinite

C) Cross-entropy stays finite, while KL becomes negative

D) Nothing changes, since the missing mass is redistributed by normalization

Answer and explanation

Correct answer: B. The model has declared an outcome possible under P to be impossible, so −log q(x) is infinite and it carries positive weight p(x). This is why smoothing or a floor on probabilities matters in practice.

Question 3.2.3

3.2. Cross-entropy: the model's average logarithmic bill

Does a one-hot label in a dataset row imply that the true $P(Y\mid X=x)$ is degenerate?

One correct optionAYes, because one-hot encoding stores the full true conditional distributionBYes, otherwise cross-entropy would be undefinedCOnly when the model ends with a softmax layerDNo, it is usually one observed draw from a conditional distribution that may well be non-degenerate

A) Yes, because one-hot encoding stores the full true conditional distribution

B) Yes, otherwise cross-entropy would be undefined

C) Only when the model ends with a softmax layer

D) No, it is usually one observed draw from a conditional distribution that may well be non-degenerate

Answer and explanation

Correct answer: D. The one-hot vector records the observed target for the per-example loss. It is evidence about the conditional law, not a proof that aleatoric uncertainty and label noise are absent.

Question 3.3.1

3.3. KL divergence: the part of the loss attributable to the model

Which identity is correct?

One correct optionA

H(P,Q)=H(P)-D_{\mathrm{KL}}(P\|Q)

H(P,Q)=H(P)+D_{\mathrm{KL}}(P\|Q)

H(P,Q)=H(Q)+D_{\mathrm{KL}}(P\|Q)

H(P,Q)=H(P)+D_{\mathrm{KL}}(Q\|P)

A) $P$

B) $H(P,Q)=H(P)+D_{\mathrm{KL}}(P\|Q)$

C) $H(P,Q)=H(Q)+D_{\mathrm{KL}}(P\|Q)$

D) $H(P,Q)=H(P)+D_{\mathrm{KL}}(Q\|P)$

Answer and explanation

Correct answer: B. Cross-entropy splits into the irreducible entropy of the source plus the surcharge caused by using Q instead of P. Note that the entropy term is H(P), not H(Q), and the KL runs in the direction P to Q.

Question 3.3.2

3.3. KL divergence: the part of the loss attributable to the model

When is $D_{\mathrm{KL}}(P\|Q)=0$ ?

One correct optionAOnly when

P=Q

almost everywhereBWhenever

P

and

Q

have the same entropyCWhenever

P

and

Q

agree on the most probable outcomeDWhenever

P

and

Q

have the same support

A) Only when $P$ almost everywhere

B) Whenever $P$ and $Q$ have the same entropy

C) Whenever $P$ and $Q$ agree on the most probable outcome

D) Whenever $P$ and $Q$ have the same support

Answer and explanation

Correct answer: A. Equal entropy, a shared argmax and a shared support are all much weaker conditions. Gibbs' inequality gives KL greater than zero for any other pair.

Question 3.3.3

3.3. KL divergence: the part of the loss attributable to the model

Why is log-loss called a strictly proper scoring rule?

One correct optionABecause it is always non-negativeBBecause it returns an integer number of bitsCBecause it guarantees the correct argmax on every individual exampleDBecause its expected value is uniquely minimized by reporting the true conditional distribution

A) Because it is always non-negative

B) Because it returns an integer number of bits

C) Because it guarantees the correct argmax on every individual example

D) Because its expected value is uniquely minimized by reporting the true conditional distribution

Answer and explanation

Correct answer: D. This follows from H(P,Q) = H(P) + D_KL(P‖Q) together with the strict positivity of KL away from P. Reporting anything other than the truth strictly increases expected loss.

Question 3.4.2

3.4. The wrong code: how KL becomes extra bits

In what sense is $D_{\mathrm{KL},2}(P\|Q)$ a number of “extra bits”?

One correct optionAIt is the expected difference between ideal code lengths when

P

-data are coded under

Q

rather than under

P

BIt is exactly one extra bit per messageCIt is the size in bits of the model's parametersDIt is the rounding overhead incurred by using integer codeword lengths

A) It is the expected difference between ideal code lengths when $P$ -data are coded under $Q$ rather than under $P$

B) It is exactly one extra bit per message

C) It is the size in bits of the model's parameters

D) It is the rounding overhead incurred by using integer codeword lengths

Answer and explanation

Correct answer: A. KL is the mismatch term in the ideal fractional-length accounting. Integer rounding is a separate cost, which adds less than one further bit per symbol in a one-shot prefix code.

Question 3.5.1

3.5. One formula for classification and language modeling

How does conditional cross-entropy decompose?

One correct optionAIt equals the model's own predictive entropy averaged over contextsBIt equals the marginal entropy

H(Y)

regardless of contextC

\mathbb E_P[-\log q(Y\mid C)]=H_P(Y\mid C)+\mathbb E_C D_{\mathrm{KL}}(P(\cdot\mid C)\|Q(\cdot\mid C))

\mathbb E_P[-\log q(Y\mid C)]=H_P(Y\mid C)+D_{\mathrm{KL}}(P(C)\|Q(C))

A) It equals the model's own predictive entropy averaged over contexts

B) It equals the marginal entropy $H(Y)$ regardless of context

C) $P$

D) $\mathbb E_P[-\log q(Y\mid C)]=H_P(Y\mid C)+D_{\mathrm{KL}}(P(C)\|Q(C))$

Answer and explanation

Correct answer: C. The same split as in the unconditional case, applied per context and then averaged. The KL is between conditional distributions, and the averaging is over contexts drawn from the data.

Question 3.5.2

3.5. One formula for classification and language modeling

Under ordinary teacher forcing, which prefixes weight the autoregressive KL decomposition?

One correct optionAPrefixes generated by the model

Q

during trainingBPrefixes drawn from the data distribution

P

CA uniform distribution over all strings of the given lengthDPrefixes from a mixture of

P

and

Q

with a scheduled weight

A) Prefixes generated by the model $P$ during training

B) Prefixes drawn from the data distribution $P$

C) A uniform distribution over all strings of the given length

D) Prefixes from a mixture of $P$ and $D_{\mathrm{KL}}(P\|Q)$ with a scheduled weight

Answer and explanation

Correct answer: B. Teacher forcing conditions on the true prefix, so the chain rule for KL between the two sequence distributions weights each position by contexts from the data.

Question 3.5.3

3.5. One formula for classification and language modeling

Why does a good teacher-forced NLL not fully describe free-running generation?

One correct optionABecause NLL discards the logarithm at inference timeBBecause softmax becomes symmetric once sampling startsCBecause generation uses a different vocabularyDBecause during generation the model conditions on its own prefixes, whose distribution may differ from

P

A) Because NLL discards the logarithm at inference time

B) Because softmax becomes symmetric once sampling starts

C) Because generation uses a different vocabulary

D) Because during generation the model conditions on its own prefixes, whose distribution may differ from $P$

Answer and explanation

Correct answer: D. An early mistake changes every later context and can move the model outside the region the training data covered well. This is the exposure-mismatch problem, not a defect of the loss.

Question 3.5.4

3.5. One formula for classification and language modeling

What does the next-token objective guarantee on its own?

One correct optionAMinimization of conditional log-loss under the data distribution, in the population formulationBFactual correctness of the generated answersCCorrect causal reasoning about the described worldDCalibrated abstention when the model does not know

A) Minimization of conditional log-loss under the data distribution, in the population formulation

B) Factual correctness of the generated answers

C) Correct causal reasoning about the described world

D) Calibrated abstention when the model does not know

Answer and explanation

Correct answer: A. The objective is a statement about log-loss and nothing else. Factuality, utility and appropriate abstention require additional assumptions, data or objectives.

Question 3.6.1

3.6. Why CrossEntropyLoss is so convenient to optimize

For a soft target $r$ , what is the gradient of softmax cross-entropy with respect to the logits?

One correct optionA

r-q

-\log q

q-r

q(1-q)(q-r)

A) $P$

B) $-\log q$

C) $q-r$

D) $q(1-q)(q-r)$

Answer and explanation

Correct answer: C. The softmax Jacobian and the logarithm cancel, leaving the difference between the predicted and the target distribution. For a one-hot target this becomes q − e_y. The sign matters: the gradient of the loss points from the target toward the prediction.

Question 3.6.2

3.6. Why CrossEntropyLoss is so convenient to optimize

What is the standard input to torch.nn.CrossEntropyLoss that avoids redundant transformations of the model output?

One correct optionAProbabilities produced by an explicit softmaxBRaw, unnormalized logitsCLog-probabilities produced by log_softmaxDClass indices of the predicted labels

A) Probabilities produced by an explicit softmax

B) Raw, unnormalized logits

C) Log-probabilities produced by log_softmax

D) Class indices of the predicted labels

Answer and explanation

Correct answer: B. The operation combines log_softmax and NLL in a numerically stable computation. Applying softmax first makes the probabilities act as new logits, changing the distribution and loss. A log_softmax output gives the same mathematical CE, but the extra normalization is redundant; NLLLoss is intended for already computed log-probabilities.

Question 3.7.1

3.7. Perplexity: returning from the logarithmic scale

If mean NLL is measured in nats, then

One correct optionA

\operatorname{PPL}=e^{\widehat L}

\operatorname{PPL}=2^{\widehat L}

\operatorname{PPL}=\ln\widehat L

\operatorname{PPL}=\widehat L

A) $P$

B) $q-r$

C) $\operatorname{PPL}=\ln\widehat L$

D) $\operatorname{PPL}=\widehat L$

Answer and explanation

Correct answer: A. Perplexity undoes the logarithm in whichever base the loss was measured: exp for nats, 2 to the power of the loss for bits. Reporting the base is part of reporting the number.

Question 3.7.2

3.7. Perplexity: returning from the logarithmic scale

What is the exact empirical interpretation of PPL?

One correct optionAThe mean predictive entropy of the modelBThe average number of candidate tokens the model consideredCThe top-1 error rate on the evaluation setDThe geometric mean of

1/q_i(y_i)

over the observed targets

A) The mean predictive entropy of the model

B) The average number of candidate tokens the model considered

C) The top-1 error rate on the evaluation set

D) The geometric mean of $P$ over the observed targets

Answer and explanation

Correct answer: D. Perplexity is the exponentiated mean log-loss, which is exactly a geometric mean of reciprocal probabilities of the observed tokens. The reading as a number of equiprobable candidates is exact only for a uniform distribution.

Question 3.7.3

3.7. Perplexity: returning from the logarithmic scale

Can a model have low predictive entropy and high perplexity at the same time?

One correct optionANo, the two quantities are equal by definitionBNo, low entropy bounds perplexity from aboveCYes, if it confidently concentrates mass on the wrong outcomesDOnly when the target distribution is uniform

A) No, the two quantities are equal by definition

B) No, low entropy bounds perplexity from above

C) Yes, if it confidently concentrates mass on the wrong outcomes

D) Only when the target distribution is uniform

Answer and explanation

Correct answer: C. Predictive entropy measures how concentrated Q is; NLL and PPL measure the probability Q assigned to the target that actually occurred. A confidently wrong model scores badly on the second while looking sharp on the first.

Question 3.7.4

3.7. Perplexity: returning from the logarithmic scale

Why are PPL values from different tokenizers not directly comparable?

One correct optionABecause tokenization does not change the likelihood of the textBBecause the tokenizers split the same text into different numbers and kinds of prediction eventsCBecause perplexity is undefined for subword vocabulariesDBecause a larger vocabulary always raises perplexity

A) Because tokenization does not change the likelihood of the text

B) Because the tokenizers split the same text into different numbers and kinds of prediction events

C) Because perplexity is undefined for subword vocabularies

D) Because a larger vocabulary always raises perplexity

Answer and explanation

Correct answer: B. Perplexity is a per-event average, and the events themselves are defined by the tokenizer. Compare under one tokenizer and protocol, or normalize per character or per byte and state the caveats.

Question 3.3.4

Optional mathematical and reference deepening

What does maximum likelihood minimize when the model family is misspecified?

One correct optionAThe true distribution

P

is recovered anyway in the large-sample limitBThe reverse KL

D_{\mathrm{KL}}(Q_\theta\|P)

CThe forward KL projection

\arg\min_\theta D_{\mathrm{KL}}(P\|Q_\theta)

at the population levelDThe symmetrized Jensen–Shannon divergence to

P

A) The true distribution $P$ is recovered anyway in the large-sample limit

B) The reverse KL $P$

C) The forward KL projection $P$ at the population level

D) The symmetrized Jensen–Shannon divergence to $P$

Answer and explanation

Correct answer: C. Maximizing likelihood is minimizing forward KL up to a constant. When P lies outside the family, the optimum is the best available approximation in that direction, and the direction is what determines which errors it prefers.

Question 3.4.1

Optional mathematical and reference deepening

Can every distribution be given a one-symbol binary prefix code with lengths exactly $-\log_2 q(x)$ ?

One correct optionAYes, alwaysBOnly if the alphabet is a power of twoCNo, because prefix codes cannot represent probabilities below one halfDNo, because codeword lengths must be integers; this works exactly only for dyadic probabilities

A) Yes, always

B) Only if the alphabet is a power of two

C) No, because prefix codes cannot represent probabilities below one half

D) No, because codeword lengths must be integers; this works exactly only for dyadic probabilities

Answer and explanation

Correct answer: D. In general the optimal average length lies between H(Q) and H(Q)+1. Block coding or arithmetic coding spreads that one bit of rounding over many symbols.

Question 3.6.3

Optional mathematical and reference deepening

What does label smoothing do?

One correct optionAIt replaces the one-hot target by a mixture of that target and a smoothing distribution, which changes the objectiveBIt only speeds up the computation, leaving the loss unchangedCIt removes the KL term from cross-entropyDIt rescales the logits before the softmax

A) It replaces the one-hot target by a mixture of that target and a smoothing distribution, which changes the objective

B) It only speeds up the computation, leaving the loss unchanged

C) It removes the KL term from cross-entropy

D) It rescales the logits before the softmax

Answer and explanation

Correct answer: A. The training loss becomes cross-entropy against a smoothed target. This can be useful regularization, but it is no longer the ordinary NLL of the observed labels.

Question 3.6.4

Optional mathematical and reference deepening

May a weighted or label-smoothed training loss be exponentiated and reported without qualification as token perplexity?

One correct optionAYes, any positive loss exponentiates to a perplexityBYes, provided the same weights are used at evaluation timeCNo, standard PPL is built from ordinary mean NLL over the observed unmasked targetsDNo, because perplexity requires a base-2 logarithm

A) Yes, any positive loss exponentiates to a perplexity

B) Yes, provided the same weights are used at evaluation time

C) No, standard PPL is built from ordinary mean NLL over the observed unmasked targets

D) No, because perplexity requires a base-2 logarithm

Answer and explanation

Correct answer: C. Weighting, smoothing, masking and the reduction mode all define a different averaged objective. Exponentiating it produces a number that is not comparable with published perplexities.

Question 3.8.1

Optional mathematical and reference deepening

When does the difference between forward and reverse KL become especially visible?

One correct optionAOnly when

P=Q

BWhen the model family cannot represent the target exactlyCOnly for Bernoulli distributionsDWhenever the target is multimodal, regardless of the family

A) Only when $P$

B) When the model family cannot represent the target exactly

C) Only for Bernoulli distributions

D) Whenever the target is multimodal, regardless of the family

Answer and explanation

Correct answer: B. If the family contains P, both directions share the same global optimum. The directions diverge when the approximation is forced to compromise, and the shape of that compromise is what the two objectives disagree about.

Question 3.8.2

Optional mathematical and reference deepening

Which statement is correct for the canonical example of a bimodal mixture approximated by a single Gaussian?

One correct optionABoth directions produce the same GaussianBReverse KL is guaranteed to have exactly one local minimumCForward KL selects one mode and reverse KL covers bothDForward KL gives a broad moment-matching Gaussian, while reverse KL has minima near the individual modes; this is an illustration, not a universal theorem

A) Both directions produce the same Gaussian

B) Reverse KL is guaranteed to have exactly one local minimum

C) Forward KL selects one mode and reverse KL covers both

D) Forward KL gives a broad moment-matching Gaussian, while reverse KL has minima near the individual modes; this is an illustration, not a universal theorem

Answer and explanation

Correct answer: D. Note the direction of the effect, which is easy to reverse in memory. In this example reverse KL also has a broader central local minimum, so even here the mode-seeking description is a tendency rather than a rule.

Question 3.8.3

Optional mathematical and reference deepening

Is posterior collapse in a VAE just another name for reverse KL selecting one mode?

One correct optionANo, collapse usually means the approximate posterior falls back to the prior and stops carrying information about

x

BYes, the two are the same phenomenonCOnly for discrete latent variablesDOnly when the decoder is deterministic

A) No, collapse usually means the approximate posterior falls back to the prior and stops carrying information about $P$

B) Yes, the two are the same phenomenon

C) Only for discrete latent variables

D) Only when the decoder is deterministic

Answer and explanation

Correct answer: A. Both belong to variational inference, but they are different failures. Mode under-coverage is about the shape of the approximation; collapse is about the latent variable becoming uninformative.

Question 3.9.1

Optional mathematical and reference deepening

Which description of DPO is most accurate?

One correct optionAIt is ordinary next-token maximum likelihood applied to preference dataBIt is a different name for PPO with the same training loopCIt is derived from a KL-regularized formulation but trained with a preference-classification loss, without evaluating a KL penalty on each batchDIt adds an explicit KL penalty to the loss of every minibatch

A) It is ordinary next-token maximum likelihood applied to preference data

B) It is a different name for PPO with the same training loop

C) It is derived from a KL-regularized formulation but trained with a preference-classification loss, without evaluating a KL penalty on each batch

D) It adds an explicit KL penalty to the loss of every minibatch

Answer and explanation

Correct answer: C. The KL enters through the optimization problem the method is derived from and through the closed-form relation between the optimal policy and the reward, not as a term computed during training.

Question 3.C.1

Computational block

Let

P=(0.5,\ 0.25,\ 0.25), \qquad Q=(0.7,\ 0.2,\ 0.1).

Compute the cross-entropy $H(P,Q)$ in bits, rounded to three decimals. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 1.668 (tolerance 0.002). H(P,Q) is the sum of p(x) times −log2 q(x), which gives 1.668. Since H(P) = 1.5 bits exactly, the model mismatch costs D_KL(P‖Q) = 0.168 bits on top of the irreducible entropy.

Question 3.C.2

Computational block

For the same $P$ and $Q$ , compute $D_{\mathrm{KL}}(Q\|P)$ in bits, rounded to three decimals. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 0.143 (tolerance 0.002). The reverse direction gives 0.143, while the forward one gives 0.168. The two numbers describe the same pair of distributions and are not interchangeable; nothing forces them to be close.

Question 3.C.3

Computational block

A model assigns probabilities $0.5$ , $0.25$ , $0.125$ and $0.5$ to the four tokens that were actually observed. Compute the perplexity, rounded to three decimals. (tolerance 0.005)

AnswerAbsolute tolerance: ±0.005

Answer and explanation

Numeric answer: 3.364 (tolerance 0.005). Mean NLL is (1 + 2 + 3 + 1)/4 = 1.75 bits, and 2 to the power 1.75 is 3.364. The same number is the geometric mean of 1/0.5, 1/0.25, 1/0.125 and 1/0.5, which is what perplexity means exactly.

Question 3.C.4

Computational block

Logits $z=(2,1,0)$ enter a softmax, and the observed class is the first one. Compute the gradient of the cross-entropy with respect to the logit of that class, rounded to three decimals. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: -0.335 (tolerance 0.002). The softmax gives approximately (0.665, 0.245, 0.090), and the gradient is q − e_y, so the first component is 0.665 − 1 = −0.335. It is negative, which means gradient descent raises the logit of the observed class.