Module 15. Reasoning Models and Test-Time Compute

Solution. Under independent attempts, no candidate is correct with probability $(1-p)^N$, so oracle coverage is $1-(1-p)^N$.

This is the probability that a correct candidate exists; calculating it needs no verifier at all. It would equal system accuracy under perfect selection. The table specifies no real verifier TPR/FPR and therefore does not predict which candidate will be selected.

Solution. Make the comparison concrete by keeping $p_{correct}=0.4$ and allocating the remaining 0.6 as (0.20, 0.20, 0.05, 0.05, 0.05, 0.05); the uniform baseline has six distractors at 0.10. In the multinomial-composition enumeration, replace only the probability vector and keep the tie rule fixed.

Common distractors consolidate wrong votes and compete more often with the correct mode. Equal one-shot correctness therefore does not imply equal self-consistency gains.

Solution. Use the module scenario $p_{correct}=0.4$: independently accept a correct candidate with probability 0.85 and an incorrect one with 0.10; select uniformly among accepted candidates and fall back to a random generated candidate if none are accepted. Exact enumeration gives:

Gains diminish sharply around $N=8$. The limit is $q=0.4\cdot0.85/(0.4\cdot0.85+0.6\cdot0.10)=0.85$. Its equality to TPR is a numerical coincidence; the limit is correctness conditional on acceptance. §7 already provides a counterexample: TPR $=0.90$ gives a limit of $12/13$.

Solution. FP8 uses one byte per element, and K and V are stored separately:

$M_{token}=2\cdot48\cdot512\cdot1=49152\ \text{bytes}=48\ \text{KiB}.$

This is $128/48=2.667$ times smaller than the original 128 KiB/token scenario. It is payload only, excluding scales, alignment, allocator overhead, and runtime buffers.

Demo for this project exercise: English project-demo notebook for Module 15, sections “Demo 1” and “Demo 2” as starting points for this module’s projects. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

This is an independent project task. Use the module’s demonstration notebooks as guidance.

Module notebooks

This is an independent project task. Use the module’s demonstration notebooks as guidance.

Module notebooks

Solution. With fixed inputs and a deterministic environment, a test reproducibly checks a stated requirement. Answer position and verbosity do not change its verdict, and a failure supplies a counterexample. A suite nevertheless checks only its recorded inputs and assertions. A program may overfit visible examples or violate uncovered edge cases, security requirements, or performance constraints. A nondeterministic environment can also make a test flaky. Passing therefore supports correctness within tested coverage; hidden cases, property tests, fuzzing, and review of the specification provide additional evidence.