Module 11. Speculative Decoding and Quantization

Solution. Accepted mass is the overlap

$\min(p,q)=(0.25,0.30,0.15,0.10,0.05),\qquad \alpha=0.85.$

Missing target mass is $(p-q)_+=(0.15,0,0,0,0)$. Normalize it by rejection probability $1-\alpha=0.15$ to get $r=(1,0,0,0,0)$. The emitted law is

$\min(p,q)+(1-\alpha)r=(0.40,0.30,0.15,0.10,0.05)=p.$

Normalizing this residual—rather than simply resampling from $p$—is what corrects the distribution after rejection.

Solution. Expected emitted length, counting accepted proposals plus one correction or bonus token, is

$E_\gamma=1+\alpha+\cdots+\alpha^\gamma=\frac{1-\alpha^{\gamma+1}}{1-\alpha}.$

For $\alpha=0.8$, $S=E_\gamma/(1+0.1\gamma)$ over $\gamma=1,\ldots,9$ is 1.6364, 2.0333, 2.2708, 2.4011, 2.4595, 2.4696, 2.4477, 2.4050, and 2.3490. Thus the peak is $\gamma=6$, with $E=3.951424$: beyond it, acceptance gain no longer pays for the extra draft work.

For the MTP point, $E=1+0.875=1.875$ and the denominator is 1.05, so $S=1.78571$. These are results of the stated cost model, not universal runtime measurements.

Solution. Eight billion weights occupy 16 GB in bf16, 8 GB in FP8, 4.5 GB in the NVFP4-like scenario with overhead, 4.25 GB in the MXFP4-like row, and 4 GB at ideal four-bit packing. With $Q=3.35\cdot10^{12}/W$:

Applying the scenario factors gives $209.375\cdot2.83790769\approx594.19$ tok/s and $\frac{3.35\cdot10^{12}}{4.5\cdot10^9}\cdot(1.875/1.05)\approx1329.37$ tok/s. These combine a weight-stream ceiling with a fixed-relative-cost scenario; real kernels, KV traffic, and acceptance can prevent the gains from composing multiplicatively.

Demo for this project exercise: English project-demo notebook for Module 11, section “Demo 1”. It supplies the starting experiment; the exercise asks you to vary its assumptions and explain what changes.

Demo for this project exercise: English project-demo notebook for Module 11, section “Demo 2”. It supplies the starting experiment; the exercise asks you to vary its assumptions and explain what changes.