Module 19. Evaluating Modern LLM Systems: From Benchmarks to Decisions

Solution. Under the rough independent-rate approximation,

$SE=\sqrt{\frac{0.955(1-0.955)}{500}+\frac{0.950(1-0.950)}{500}}=0.013452,$ $z=\frac{0.955-0.950}{SE}=0.3717.$

The assumed 0.005 difference is much smaller than the planned half-width $1.96SE$; these settings cannot promise fine discrimination. An observed interval requires actual outcomes. In the paired table, accuracies are $(150+20)/200=0.85$ and $(150+10)/200=0.80$, but McNemar uses only 30 discordant outcomes. Under the null, A-only count is $\operatorname{Bin}(30,0.5)$; the exact two-sided value is $2P[X\le10]\approx0.09874$. Pairing preserves information lost in two aggregate rates.

Solution. Of $n$ generated programs, $c$ are correct. The probability that a size-$k$ subset contains no correct program is $\binom{n-c}{k}/\binom nk$, hence

$\widehat{pass@k}=1-\frac{\binom{n-c}{k}}{\binom nk}.$

For $n=20,c=4$, values are 0.2000, 0.718266, and 0.956656 at $k=1,5,10$. The plug-in approximation $1-(1-c/n)^k$ gives 0.2000, 0.672320, and 0.892626 because substituting the noisy observed proportion into a nonlinear function introduces bias. The combinatorial estimator is unbiased under independent draws from one fixed sampling policy. The rise from one to ten samples measures candidate-generation headroom with a selector, not reliability of one deployment call.

Solution. Candidate A's two-order average is $(650/1000+350/1000)/2=0.5$; a fixed first position would show 0.65 versus 0.35 and create a false 30-point gap.

For $a(x)=0.95-0.35\sin^2(\pi x)$, mean accuracy on $[0,1]$ is $0.95-0.35/2=0.775$. The condition $a(x)\ge0.9$ is $\sin^2(\pi x)\le1/7$; combining the intervals at both edges gives share

$\frac2\pi\arcsin\frac1{\sqrt7}=0.246752.$

The normal planning approximation gives $n\approx1.96^2\cdot0.8\cdot0.2/0.03^2=682.95$, so round up to 683. A final interval should use a method such as Wilson; this formula explains the planning size.

The project notebook saves outputs from two rules on 200 fictional refund requests. A ignores packaging integrity; B mishandles the 30-day boundary. A passes the ordinary suite but fails critical exceptions; B passes exceptions but misses four ordinary cases. The trajectory suite performs repeated ledger writes and inspects final state. Neither system passes all predeclared gates. These are deterministic teaching rules, not LLM results or sampled customer traffic.

In this experiment A gets 200/200 ordinary requests right and B gets 196/200. The difference is 0.02; a paired percentile bootstrap with 5000 resamples gives [0.005, 0.04], while exact McNemar gives $p=0.125$. There are only four disagreements, all in one direction. The empirical bootstrap cannot create a B-only success absent from its source data and may give an overconfident interval near this boundary. Do not declare a win from that interval alone; repeated fictional templates also provide no evidence of customer-level generalization.

Demo for this project exercise: English project-demo notebook for Module 19, section “Demo 1”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.

Demo for this project exercise: English project-demo notebook for Module 19, section “Demo 2”. It shows a compact reproducible example on small inputs, not a completed solution to the full-scale project.