Module 2. Tokenization and Embeddings

Solution. Before BPE merges, each UTF-8 byte is one base token. Привет, мир! has nine Cyrillic letters (Привет has 6 and мир has 3), hence $9\cdot2=18$ bytes, plus a comma, a space, and !: 21. Hello, world! contains 13 ASCII characters and therefore uses 13 bytes. In наивный 🚀, seven Cyrillic letters use 14 bytes, the space uses 1, and the emoji uses 4, for 19 total. A tokenizer-free check is len(text.encode("utf-8")).

Solution.

def encode_by_rank(text, merges):
    chunks = re.findall(PAT, text)
    output = []
    for chunk in chunks:
        ids = list(chunk.encode("utf-8"))
        while len(ids) > 1:
            pairs = set(zip(ids[:-1], ids[1:]))
            best = min(pairs, key=lambda p: merges.get(p, float("inf")))
            if best not in merges:
                break
            ids = merge([ids], best, merges[best])[0]
        output.extend(ids)
    return output

The regex first splits the text into pre-token chunks, which must be encoded independently. Within each chunk, enumerate adjacent pairs, choose the present pair with the smallest numeric rank, and merge all valid occurrences. Stop when no present pair belongs to merges. Corpus frequency is not recomputed at inference: training has already encoded its effect in the learned ranks.

Solution. Freeze the exact corpus snapshot and strata, normalization and word rule, tokenizer name and revision, and whether special tokens are added. For each line record UTF-8 bytes $b_i$, words $w_i$, and tokenizer IDs $t_i$ (normally excluding beginning-of-sequence and end-of-sequence markers, BOS and EOS), then compute

This micro-average weights every word equally. The alternative $\frac1N\sum_i t_i/w_i$ weights every line equally, so a one-word line affects the result as much as a long sentence; zero-word lines also need a separate policy. Publish $\sum w_i$, $\sum t_i$, $\sum b_i$, and excluded-line counts so another reader can reproduce the result.

Demo for this project exercise: English project-demo notebook for Module 2, section “1. What does the denominator count?”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

Specify suitable word segmentation for each language, or compare tokens per aligned sentence. A regex matching letter runs treats a Chinese string as one run, not one linguistic word. The demo exposes denominators and units per sentence; its wordrun_proxy is not WordPiece.

Demo for this project exercise: English project-demo notebook for Module 2, section “2. Why can names outnumber IDs?”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

Expected for the 11 October 2026 snapshot: 1,091 names, 1,090 unique special IDs, 10 protocol names, 1,081 reserved-form names, and n_vocab=201088 after adding 199,998 mergeable IDs.

Solution. Lookup gives [[1,0],[0,1],[1,0]]. The logits are [1,2,0], so ID 1 wins. There are $16384\cdot512=8388608$ parameters occupying $16777216$ BF16 bytes, or 16 MiB. Repeated ID 0 accumulates two contributions: the gradient is [[2,0],[0,0],[0,1]]. A step of 0.1 gives [[0.8,0],[2,0],[0,0.9]], lowering $L$ from 1.5 to 1.045. The notebook checks the gradient with finite differences; this is a lookup exercise rather than semantic model training.