Module 1. Introduction — quizzes and answer checking · Information Theory for ML

Question 1.1.1

Required quiz: Sections 1.1–1.4

Which question is about the surprisal of one realized outcome rather than the source's average uncertainty?

One correct optionAWhat is the source's average uncertainty before observation?BHow surprising is this particular outcome under the chosen model?CBy how much does observing

Y

reduce uncertainty about

X

on average?DHow many bits does an optimal code need per symbol on long sequences?

A) What is the source's average uncertainty before observation?

B) How surprising is this particular outcome under the chosen model?

C) By how much does observing $Y$ reduce uncertainty about $X$ on average?

D) How many bits does an optimal code need per symbol on long sequences?

Answer and explanation

Correct answer: B. Surprisal refers to one realized outcome. Entropy averages surprisal over outcomes, mutual information measures the average reduction in uncertainty, and the last option describes the entropy rate. All four are called "information" in ordinary speech.

Question 1.1.2

Required quiz: Sections 1.1–1.4

Why was separating information from meaning a source of strength for Shannon's theory?

One correct optionAIt replaced every semantic problem with a single probabilityBIt proved that meaning does not existCIt showed that semantics can always be recovered from the statistics afterwardsDIt isolated a measurable engineering problem and made rigorous limits on coding and transmission possible

A) It replaced every semantic problem with a single probability

B) It proved that meaning does not exist

C) It showed that semantics can always be recovered from the statistics afterwards

D) It isolated a measurable engineering problem and made rigorous limits on coding and transmission possible

Answer and explanation

Correct answer: D. Restricting the scope is what made a precise mathematical theory possible. The framework does not deny meaning and does not promise to reconstruct it; it simply leaves it outside its defined quantities.

Question 1.1.3

Required quiz: Sections 1.1–1.4

Why can the same message have different surprisal in different contexts?

One correct optionASurprisal depends only on the message length in charactersBMeaning changes the base of the logarithmCContext changes the conditional probability of the message under the chosen modelDSurprisal is measured after the outcome, so it cannot depend on context at all

A) Surprisal depends only on the message length in characters

B) Meaning changes the base of the logarithm

C) Context changes the conditional probability of the message under the chosen model

D) Surprisal is measured after the outcome, so it cannot depend on context at all

Answer and explanation

Correct answer: C. Surprisal is −log p relative to a particular model, and the model's probability is conditional on what is already known. The quantity is defined after the outcome but computed from a probability assigned before it.

Question 1.2.1

Required quiz: Sections 1.1–1.4

What follows correctly from the fact that a next-token objective contains no separate criterion of factual truth?

One correct optionALow cross-entropy alone does not guarantee that every answer is trueBEvery improvement in cross-entropy reduces factual accuracyCA language model is fundamentally unable to store facts or perform reasoningDFactual accuracy cannot be measured for such a model at all

A) Low cross-entropy alone does not guarantee that every answer is true

B) Every improvement in cross-entropy reduces factual accuracy

C) A language model is fundamentally unable to store facts or perform reasoning

D) Factual accuracy cannot be measured for such a model at all

Answer and explanation

Correct answer: A. This is a statement about what the objective guarantees, not a claim that factual knowledge is impossible, that accuracy degrades, or that it cannot be evaluated by other means.

Question 1.3.1

Required quiz: Sections 1.1–1.4

Which requirement leads to the functional equation $s(pq)=s(p)+s(q)$ ?

One correct optionAAll events must be equally likelyBSurprise must not depend on probabilityCSurprise must be boundedDIndependent stages should contribute additive surprise

A) All events must be equally likely

B) Surprise must not depend on probability

C) Surprise must be bounded

D) Independent stages should contribute additive surprise

Answer and explanation

Correct answer: D. Independent probabilities multiply while their information contributions should add, and the logarithm is exactly the function that converts products into sums.

Question 1.3.2

Required quiz: Sections 1.1–1.4

A sequence has token conditional probabilities $(1/2,1/2,1/4)$ . What is its negative log-likelihood in bits?

One correct optionA2 bitsB3 bitsC4 bitsD8 bits

A) 2 bits

B) 3 bits

C) 4 bits

D) 8 bits

Answer and explanation

Correct answer: C. The joint probability is 1/16, so −log₂(1/16) = 4 bits. Equivalently the token contributions 1 + 1 + 2 add up, which is the point of taking the logarithm in the first place.

Question 1.3.3

Required quiz: Sections 1.1–1.4

What is the most precise relationship between the NLL of one observation and empirical cross-entropy?

One correct optionANLL is the contribution of one observation, and the average NLL over a dataset is empirical cross-entropyBThey are unrelated quantitiesCThey are numerically identical without specifying summation or averagingDCross-entropy is the NLL of the single least probable observation

A) NLL is the contribution of one observation, and the average NLL over a dataset is empirical cross-entropy

B) They are unrelated quantities

C) They are numerically identical without specifying summation or averaging

D) Cross-entropy is the NLL of the single least probable observation

Answer and explanation

Correct answer: A. Practice often uses the two words interchangeably, but the aggregation matters: one observation contributes one term, while cross-entropy is an expectation or an empirical average of such terms.

Question 1.9.1

Optional second-pass quiz: Sections 1.9–1.10

What follows from monotonicity and $f(mk)=f(m)+f(k)$ for uniform sources?

One correct optionAThe full Shannon formula for arbitrary real-valued probabilitiesBThe logarithmic form

f(n)=c\log n

on uniform distributionsCA unique logarithm baseDContinuity of

f

on the probability simplex

A) The full Shannon formula for arbitrary real-valued probabilities

B) The logarithmic form $f(n)=c\log n$ on uniform distributions

C) A unique logarithm base

D) Continuity of $f$ on the probability simplex

Answer and explanation

Correct answer: B. This is only the first stage of the derivation. The base is fixed later by a normalization, and moving to unequal probabilities requires weighted grouping and continuity.

Question 1.10.1

Optional second-pass quiz: Sections 1.9–1.10

Which property transfers the result from uniform micro-outcomes to rational probabilities?

One correct optionASymmetry under relabeling aloneBSecond-order differentiabilityCAdditivity for independent sourcesDWeighted grouping for a two-stage choice

A) Symmetry under relabeling alone

B) Second-order differentiability

C) Additivity for independent sources

D) Weighted grouping for a two-stage choice

Answer and explanation

Correct answer: D. Rational probabilities can be written as groups of equally likely micro-outcomes, and weighted grouping is what links the two levels of the choice. Plain additivity is too weak, which the Rényi counterexample shows.

Question 1.10.2

Optional second-pass quiz: Sections 1.9–1.10

Why is continuity needed in the Shannon-entropy derivation?

One correct optionATo prove the logarithmic form on natural numbersBOnly to choose bits instead of natsCTo extend the result from rational probability vectors to arbitrary real-valued onesDTo guarantee that entropy is differentiable in the probabilities

A) To prove the logarithmic form on natural numbers

B) Only to choose bits instead of nats

C) To extend the result from rational probability vectors to arbitrary real-valued ones

D) To guarantee that entropy is differentiable in the probabilities

Answer and explanation

Correct answer: C. Rational points are dense in the probability simplex, so continuity extends the formula to its closure. Differentiability is never required.

Question 1.10.3

Optional second-pass quiz: Sections 1.9–1.10

What does the counterexample based on Rényi entropy of order $2$ show?

One correct optionAAdditivity on independent products and the value

\log n

on uniform sources do not yet single out Shannon entropyBShannon entropy is not additiveCWeighted grouping follows from symmetry aloneDRényi entropy of order 2 disagrees with Shannon entropy already on uniform distributions

A) Additivity on independent products and the value $\log n$ on uniform sources do not yet single out Shannon entropy

B) Shannon entropy is not additive

C) Weighted grouping follows from symmetry alone

D) Rényi entropy of order 2 disagrees with Shannon entropy already on uniform distributions

Answer and explanation

Correct answer: A. Rényi entropy of order 2 has both of those properties, including agreement on uniform distributions, and still fails weighted grouping. That is precisely why the derivation needs the stronger axiom.

Question 1.10.4

Optional second-pass quiz: Sections 1.9–1.10

What does the normalization $H(1/2,1/2)=1$ bit accomplish?

One correct optionAIt proves weighted groupingBIt excludes the zero solution and fixes the unit of informationCIt restricts the number of outcomes to twoDIt makes the maximum of entropy equal to one for every alphabet

A) It proves weighted grouping

B) It excludes the zero solution and fixes the unit of information

C) It restricts the number of outcomes to two

D) It makes the maximum of entropy equal to one for every alphabet

Answer and explanation

Correct answer: B. The axioms determine the form only up to a nonnegative constant, and that constant may be zero. The normalization picks a nonzero scale, which is the same as choosing base 2.

Question 1.C.1

Computational block

A source emits one of 27 equally likely symbols. How many bits does one symbol carry? (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 4.755 (tolerance 0.002). For a uniform source over n outcomes the surprisal of every outcome is log₂ n, so here it is log₂ 27 = 4.755 bits. This is Hartley's answer and the starting point of Shannon's table for printed English in Module 2.

Question 1.C.2

Computational block

For $P=(1/2,1/4,1/4)$ , compute the Rényi entropy of order $2$ , that is $-\log_2\sum_i p_i^2$ , in bits. (tolerance 0.002)

AnswerAbsolute tolerance: ±0.002

Answer and explanation

Numeric answer: 1.415 (tolerance 0.002). The sum of squares is 0.375, so H₂ = −log₂ 0.375 = 1.415 bits, while the Shannon entropy of the same distribution is exactly 1.5 bits. Two additive measures that agree on uniform distributions can still disagree here, which is the counterexample of §1.10 in numbers.