Module 2. Tokenization and Embeddings — quizzes and answer checking · Modern LLMs

Question 2.3.1

2.3. Compressing a sequence

What does bytes/token measure for a fixed tokenizer?

One correct optionAHow many input UTF-8 bytes one token ID represents on average in this textBThe full compressed-file size including vocabulary, metadata, and ID packingCThe mean probability of the selected token IDs under a language model

A) How many input UTF-8 bytes one token ID represents on average in this text

B) The full compressed-file size including vocabulary, metadata, and ID packing

C) The mean probability of the selected token IDs under a language model

Answer and explanation

Correct answer: A. Bytes/token is a sequence-length measure: the average number of input UTF-8 bytes represented by one token ID. Full bit-level compression would also count ID width, the vocabulary, and metadata.

Question 2.4.1

2.4. The tokenizer as a sequence of operations

Why does byte-level BPE need no OOV token?

One correct optionAIt removes characters absent from the training corpusBEvery UTF-8 string decomposes into base bytes already in the vocabularyCIt reserves one token for every Unicode character in advance

A) It removes characters absent from the training corpus

B) Every UTF-8 string decomposes into base bytes already in the vocabulary

C) It reserves one token for every Unicode character in advance

Answer and explanation

Correct answer: B. The base alphabet contains all 256 byte values. Merges only add longer units, so any byte string remains representable without an Out-of-Vocabulary token.

Question 2.4.2

2.4. The tokenizer as a sequence of operations

When can decode(encode(s)) fail to preserve the original spelling?

One correct optionAWhenever BPE has at least one merge, even without normalizationBWhenever a token contains several complete UTF-8 charactersCWhen normalization changes characters or spaces before tokenization

A) Whenever BPE has at least one merge, even without normalization

B) Whenever a token contains several complete UTF-8 characters

C) When normalization changes characters or spaces before tokenization

Answer and explanation

Correct answer: C. With full coverage, the recoverable text is N(s). The teaching BBPE preserves valid UTF-8 without normalization; UNK replacement can lose further information.

Question 2.5.1

2.5. Worked example: from UTF-8 to a compression curve

Why does BPE encoding follow learned merge ranks rather than current pair frequency?

One correct optionAFrequency is used during training; rank then freezes deterministic segmentationBRank encodes frequency in the new text and is recomputed for every requestCRank selects a new ID only and does not affect token boundaries

A) Frequency is used during training; rank then freezes deterministic segmentation

B) Rank encodes frequency in the new text and is recomputed for every request

C) Rank selects a new ID only and does not affect token boundaries

Answer and explanation

Correct answer: A. After training, the tokenizer must reproduce the same segmentation expected by the model weights. Frequencies in a new string do not rewrite the merge list.

Question 2.5.2

2.5. Worked example: from UTF-8 to a compression curve

Comparing only RU train (6.535 bytes/token) and EN held-out (1.317), what is supported?

One correct optionAThis isolates generalization to unseen data in the same languageBCompression differs on these corpora; language and training exposure change togetherCRussian is 4.96 times cheaper than English under every tokenizer

A) This isolates generalization to unseen data in the same language

B) Compression differs on these corpora; language and training exposure change together

C) Russian is 4.96 times cheaper than English under every tokenizer

Answer and explanation

Correct answer: B. These two values confound language with training exposure. The lecture also reports RU held-out (5.542), which exposes a within-language train/eval difference. Parallel RU/EN texts still differ in UTF-8 length; bytes/token measures neither the cost of equal meaning nor KL.

Question 2.5.3

2.5. Worked example: from UTF-8 to a compression curve

What does tokenizer fertility measure?

One correct optionAThe number of merges learned per million bytes at a given vocab_sizeBThe fraction of special and reserved IDs in the tokenizer vocabularyCThe average tokens per word or other chosen unit under an explicit protocol

A) The number of merges learned per million bytes at a given vocab_size

B) The fraction of special and reserved IDs in the tokenizer vocabulary

C) The average tokens per word or other chosen unit under an explicit protocol

Answer and explanation

Correct answer: C. Fertility belongs to a tokenizer, a particular text sample, and a word-counting rule. It should be reported with the corpus, bytes/token, and protocol.

Question 2.8.1

2.8. Reading o200k_harmony correctly

How does the o200k_harmony vocabulary size reconcile exactly?

One correct optionA199,998 mergeable IDs + 1,090 unique special IDs = 201,088B200,019 text IDs + 16 control + 1,075 reserved = 201,110C201,088 ordinary tokens with no special identifiers

A) 199,998 mergeable IDs + 1,090 unique special IDs = 201,088

B) 200,019 text IDs + 16 control + 1,075 reserved = 201,110

C) 201,088 ordinary tokens with no special identifiers

Answer and explanation

Correct answer: A. There are 1,091 special-token names but 1,090 unique IDs because <|endofprompt|> and <|reserved_200018|> alias ID 200018. Ten names are protocol-bearing and 1,081 have reserved form.

Question 2.8.2

2.8. Reading o200k_harmony correctly

What does tiktoken.encode do by default when input literally contains a registered special-token spelling?

One correct optionAAlways emits the control ID when the spelling matches exactlyBRaises by default; the application selects special or ordinary-text handlingCAlways splits the spelling into characters and ignores the special-token table

A) Always emits the control ID when the spelling matches exactly

B) Raises by default; the application selects special or ordinary-text handling

C) Always splits the spelling into characters and ignores the special-token table

Answer and explanation

Correct answer: B. Special spellings are a protocol boundary. allowed_special admits them as controls; encode_ordinary or disallowed_special=() treats the spelling as ordinary text.

Question 2.8.3

2.8. Reading o200k_harmony correctly

Why is a plain token-to-ID table insufficient to reproduce a deployed tokenizer?

One correct optionAToken identifiers change randomly whenever the program is restartedBThe embedding matrix must be stored inside the same tokenizer.json fileCNormalization, pre-tokenization, ranks, decoder, and special-token policy are also required

A) Token identifiers change randomly whenever the program is restarted

B) The embedding matrix must be stored inside the same tokenizer.json file

C) Normalization, pre-tokenization, ranks, decoder, and special-token policy are also required

Answer and explanation

Correct answer: C. A tokenizer is a deterministic program, not merely a string table. Changing any component can alter IDs and make input incompatible with model weights.

Question 2.9.1

2.9. Three vocabulary sizes that answer different questions

o200k_base defines 200,000 IDs and its maximum ID is 200018. What is n_vocab?

One correct optionA200000: always exactly the number of defined IDsB199998: the ordinary-token count onlyC200019: maximum ID plus one, including holes

A) 200000: always exactly the number of defined IDs

B) 199998: the ordinary-token count only

C) 200019: maximum ID plus one, including holes

Answer and explanation

Correct answer: C. tiktoken n_vocab is the ID range size. There are 19 holes; this field does not establish a closed model’s embedding-table row count.

Question 2.9.2

2.9. Three vocabulary sizes that answer different questions

Why does a larger vocab_size not guarantee better tokenization for every language?

One correct optionAAdditional merges follow the training data rather than being shared equally by languagesBA large vocabulary removes base bytes and therefore loses rare Unicode charactersCAbove 100K, vocabulary size no longer affects text segmentation

A) Additional merges follow the training data rather than being shared equally by languages

B) A large vocabulary removes base bytes and therefore loses rare Unicode characters

C) Above 100K, vocabulary size no longer affects text segmentation

Answer and explanation

Correct answer: A. Allocated rows can differ from the defined ID count. Even a larger ordinary-token inventory does not guarantee better segmentation: training data and algorithm rules matter.

Question 2.10.1

2.10. Fertility, language, and domain

What is a sound claim about grouping digits into runs of one to three characters?

One correct optionAThe rule by itself guarantees exact addition for arbitrary multi-digit numbersBIt regularizes segmentation but neither guarantees nor is required for arithmeticCIts only purpose is to cap the full vocabulary at exactly one thousand IDs

A) The rule by itself guarantees exact addition for arbitrary multi-digit numbers

B) It regularizes segmentation but neither guarantees nor is required for arithmetic

C) Its only purpose is to cap the full vocabulary at exactly one thousand IDs

Answer and explanation

Correct answer: B. Regular segmentation changes the inductive bias and can help some numerical tasks. Arithmetic ability still depends on data, architecture, and training.

Question 2.11.1

2.11. Embeddings: reading an ID and predicting the next one

E=[[1,0],[2,0],[0,1]], h=E[0], U=E.T. Which ID has the largest logit?

One correct optionAID 0: tied weights always invert the lookupBID 2: the last ID always has the largest logitCID 1: the logits are [1,2,0]

A) ID 0: tied weights always invert the lookup

B) ID 2: the last ID always has the largest logit

C) ID 1: the logits are [1,2,0]

Answer and explanation

Correct answer: C. Output projection computes dot products rather than inverting lookup. Decoding selected IDs is a separate operation.

Question 2.11.2

2.11. Embeddings: reading an ID and predicting the next one

A model has vocab_size=16,384 and hidden size=512. How many parameters are in one embedding matrix?

One correct optionA8,388,608B16,384C16,777,216 regardless of weight tying

A) 8,388,608

B) 16,384

C) 16,777,216 regardless of weight tying

Answer and explanation

Correct answer: A. One matrix has V·d = 16,384·512 = 8,388,608 parameters. An untied output head adds another matrix of the same size.

Question 2.11.3

2.11. Embeddings: reading an ID and predicting the next one

Let E=[[1,0],[2,0],[0,1]], IDs=[0,2,0], and all targets be zero. For L=½·sum(E[ids]²), what is the gradient of row E[0]?

One correct optionA[1,0]: each row counts once regardless of how often it is selectedB[2,0]: the two lookups contribute equal derivativesC[0,0]: selecting rows cannot train the input table

A) [1,0]: each row counts once regardless of how often it is selected

B) [2,0]: the two lookups contribute equal derivatives

C) [0,0]: selecting rows cannot train the input table

Answer and explanation

Correct answer: B. Each occurrence of ID 0 contributes E[0]=[1,0]. Summing gives [2,0]. np.add.at accumulates repeated indices correctly; grad[ids] += ... does not.

Question 2.12.1

2.12. Where the token analogy extends to other modalities

Why is a vision projector only a partial analogue of text tokenization?

One correct optionAA projector must use the same BPE vocabulary as the text decoderBPatches usually become continuous vectors rather than discrete text token IDsCA projector runs only after the language model’s final Transformer layer

A) A projector must use the same BPE vocabulary as the text decoder

B) Patches usually become continuous vectors rather than discrete text token IDs

C) A projector runs only after the language model’s final Transformer layer

Answer and explanation

Correct answer: B. Both systems create a sequence for a Transformer, but a text tokenizer emits discrete IDs while a typical vision encoder/projector emits continuous representations. Discrete audio codecs are a closer literal analogy.