Modern LLMs19 / 20

Module 19. Evaluating Modern LLM Systems: From Benchmarks to Decisions

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Checked: 0 / 20 · Correct: 0

Question 19.4.1

19.4. How a benchmark loses measurement power

Why can a public benchmark lose discriminating power over time?

One correct optionAItem exposure combines with strong systems approaching the benchmark ceilingBA public metric stops reflecting human judgment after its first yearCPublished items can no longer be audited or corrected by experts

A) Item exposure combines with strong systems approaching the benchmark ceiling

B) A public metric stops reflecting human judgment after its first year

C) Published items can no longer be audited or corrected by experts

Answer and explanation

Correct answer: A. Exposure can inflate scores, while saturation compresses differences among strong systems. Both effects motivate refreshed, private, or harder evaluation sets.

Question 19.5.1

19.5. Build an evaluation map before choosing benchmark names

Why should a model-selection decision not be based on one aggregate score?

One correct optionAAggregates are valid only for tasks with one correct classBThe mean hides critical slices, cost, latency, and riskCEvery aggregate metric has a systematic downward bias

A) Aggregates are valid only for tasks with one correct class

B) The mean hides critical slices, cost, latency, and risk

C) Every aggregate metric has a systematic downward bias

Answer and explanation

Correct answer: B. A product decision is multidimensional. Two systems with the same mean can differ sharply on a critical slice, cost, or tail latency.

Question 19.5.2

19.5. Build an evaluation map before choosing benchmark names

A confidence threshold retains 4 correct responses from 20 requests. What follows?

One correct optionAThe system is proven error-free on all future requestsBCoverage is 20%; observed risk is zero over four responsesCEach token probability equals the probability of a correct response

A) The system is proven error-free on all future requests

B) Coverage is 20%; observed risk is zero over four responses

C) Each token probability equals the probability of a correct response

Answer and explanation

Correct answer: B. Risk uses accepted responses as its denominator; coverage uses all requests. Four successes do not establish future reliability. If no responses were accepted, risk would be undefined.

Question 19.6.1

19.6. Worked example A: what does half a percentage point mean?

At planned N=500, assume success probabilities 0.955 and 0.950. What does the independent approximation imply?

One correct optionASystem A is proven better under every item-level error patternBThe gap is exactly five extra successes and is therefore significantCThe planned half-width is substantially larger than the probability gap

A) System A is proven better under every item-level error pattern

B) The gap is exactly five extra successes and is therefore significant

C) The planned half-width is substantially larger than the probability gap

Answer and explanation

Correct answer: C. These are assumed probabilities: 95.5% cannot be observed in one N=500 binary run. The normal half-width is about 2.64 points, larger than the 0.5-point gap; an observed comparison on shared tasks requires paired outcomes.

Question 19.6.2

19.6. Worked example A: what does half a percentage point mean?

How many tasks per system target a normal 95% difference-interval half-width of 0.5 pp for independent rates near 95%?

One correct optionAAbout 14,599 items per systemBAbout 3,650 items per systemCAbout 500 items per system

A) About 14,599 items per system

B) About 3,650 items per system

C) About 500 items per system

Answer and explanation

Correct answer: A. Rounding 2·0.95·0.05·(1.96/0.005)² up gives 14,599. This plans interval precision, not a specified statistical power.

Question 19.6.3

19.6. Worked example A: what does half a percentage point mean?

Two N=200 tables give 85%/80%. Which has the smaller exact McNemar p-value?

One correct optionA20 A-only successes, 10 B-onlyB10 A-only successes, 0 B-onlyCEqual p-values because accuracies match

A) 20 A-only successes, 10 B-only

B) 10 A-only successes, 0 B-only

C) Equal p-values because accuracies match

Answer and explanation

Correct answer: B. All ten disagreements favor A in the second table: p=0.001953, versus about 0.098737 in the first. Aggregates are insufficient.

Question 19.7.1

19.7. Difficulty has several sources: HLE, FrontierMath, and ARC-AGI

Why use several hard sets—HLE, FrontierMath, and ARC-AGI—instead of one “hardest benchmark”?

One correct optionATo obtain one averaged coefficient of general intelligenceBTo separate expert knowledge, mathematics, and abstract generalizationCBecause each set supports only one architecture family

A) To obtain one averaged coefficient of general intelligence

B) To separate expert knowledge, mathematics, and abstract generalization

C) Because each set supports only one architecture family

Answer and explanation

Correct answer: B. A low score can have different causes. The sets probe expert knowledge, original mathematical problems, and transfer of rules to novel visual structures.

Question 19.8.1

19.8. pass@k: finding one solution is not the same as being reliable

With n=20 generations and c=4 successes, what is the unbiased pass@10 estimate?

One correct optionA0.8926 — the plug-in estimate with independent repeated attemptsB0.2000 — the observed success rate of one random generationC0.9567 — the finite-sample estimate without replacement

A) 0.8926 — the plug-in estimate with independent repeated attempts

B) 0.2000 — the observed success rate of one random generation

C) 0.9567 — the finite-sample estimate without replacement

Answer and explanation

Correct answer: C. The formula 1−C(n−c,k)/C(n,k) gives 0.9567. Unbiasedness for at least one success requires independent draws from a fixed policy. Plugging c/n into the nonlinear formula introduces bias.

Question 19.8.2

19.8. pass@k: finding one solution is not the same as being reliable

What does a large gap between pass@1 and pass@10 mean?

One correct optionAThere is solution-finding headroom, but the first attempt remains unreliableBThe model necessarily switches among ten different solution algorithmsCpass@10 penalizes errors more strictly than a one-attempt metric

A) There is solution-finding headroom, but the first attempt remains unreliable

B) The model necessarily switches among ten different solution algorithms

C) pass@10 penalizes errors more strictly than a one-attempt metric

Answer and explanation

Correct answer: A. The gap separates search capability from single-attempt reliability. A product that permits one try may gain little from a high pass@10.

Question 19.9.1

19.9. Human arenas and composite indices answer someone else’s question

What is the main limitation of a composite index or arena rating?

One correct optionAAn arena cannot collect human pairwise preferences at scaleBThe index inherits someone else’s task weights, audience, and aggregationCA composite is always statistically less stable than a single metric

A) An arena cannot collect human pairwise preferences at scale

B) The index inherits someone else’s task weights, audience, and aggregation

C) A composite is always statistically less stable than a single metric

Answer and explanation

Correct answer: B. A composite is useful as a broad map, but its coordinate system is determined by its users, tasks, weights, and aggregation protocol. Product decisions need private slices as well.

Question 19.10.1

19.10. LLM judges: a synthetic 30-point illusion

How can position bias in pairwise LLM judging be reduced?

One correct optionAAlways show the new system first in every comparisonBRaise judge temperature until order dependence disappearsCEvaluate A/B and B/A and aggregate by candidate identity

A) Always show the new system first in every comparison

B) Raise judge temperature until order dependence disappears

C) Evaluate A/B and B/A and aggregate by candidate identity

Answer and explanation

Correct answer: C. Swapping exposes preference for the first or second position. A stronger protocol also controls rubric, length, judge family, and cost.

Question 19.10.2

19.10. LLM judges: a synthetic 30-point illusion

How many calls per pair are required by three LLM judges evaluating both A/B and B/A orders?

One correct optionA6 calls per pairB3 calls per pairC2 calls per pair

A) 6 calls per pair

B) 3 calls per pair

C) 2 calls per pair

Answer and explanation

Correct answer: A. Each of the three judges evaluates two orderings, so the protocol uses 3×2=6 calls. Bias control has a measurable cost.

Question 19.11.1

19.11. Multimodal capability, safety, truthfulness, and honesty are not one axis

Why should safety, truthfulness, and general capability not be collapsed into one number?

One correct optionASafety cannot be measured quantitatively on a fixed evaluation setBThe axes describe different risks and need different failure criteriaCTruthfulness requires humans, while capability requires only tests

A) Safety cannot be measured quantitatively on a fixed evaluation set

B) The axes describe different risks and need different failure criteria

C) Truthfulness requires humans, while capability requires only tests

Answer and explanation

Correct answer: B. A system can be capable but unsafe, or factually strong yet vulnerable to adversarial prompts. These axes should be measured separately and combined only through an explicit product utility.

Question 19.12.1

19.12. Long context: accepted length versus usable evidence

How does working context differ from a nominal context window?

One correct optionAWorking context is always half of the advertised context lengthBNominal length belongs to prefill, while working length belongs to decodeCIt is the region where evidence is used above a chosen threshold

A) Working context is always half of the advertised context length

B) Nominal length belongs to prefill, while working length belongs to decode

C) It is the region where evidence is used above a chosen threshold

Answer and explanation

Correct answer: C. Accepting one million tokens does not guarantee uniform use of the beginning, middle, and end. Working context depends on task, evidence position, and quality threshold.

Question 19.12.2

19.12. Long context: accepted length versus usable evidence

What does 0.246752 mean for the synthetic U-shaped profile?

One correct optionAThe combined measure of two edge regionsBA contiguous 246752-token usable windowCMeasured accuracy of a particular LLM

A) The combined measure of two edge regions

B) A contiguous 246752-token usable window

C) Measured accuracy of a particular LLM

Answer and explanation

Correct answer: A. The 0.9 threshold selects two disjoint edges under a uniform position distribution and an assumed curve.

Question 19.13.1

19.13. A minimal evaluation harness: one call, one observed output

Why should a harness call the system once per item within a fixed run and trial?

One correct optionASo output, metrics, latency, and cost refer to one observed responseBBecause a second call deterministically reproduces the first responseCTo prohibit computing several different metrics on the same example

A) So output, metrics, latency, and cost refer to one observed response

B) Because a second call deterministically reproduces the first response

C) To prohibit computing several different metrics on the same example

Answer and explanation

Correct answer: A. A second call to a stochastic or stateful model can produce a different answer and add cost. All metrics should be derived from one stored observation.

Question 19.13.2

19.13. A minimal evaluation harness: one call, one observed output

A metric raises after a successful model response. What does the full teaching notebook do?

One correct optionADrops the row and reduces the denominatorBSilently assigns a model failureCRetains the response and invalidates the aggregate

A) Drops the row and reduces the denominator

B) Silently assigns a model failure

C) Retains the response and invalidates the aggregate

Answer and explanation

Correct answer: C. An evaluator bug is not a model failure. A None aggregate prevents publication of a misleading quality result.

Question 19.15.1

19.15. A private golden set in seven deliberate steps

Why keep a private golden set versioned?

One correct optionATo avoid documenting criteria, slices, and evaluation rulesBTo reduce test-specific tuning and track dataset versionsCTo prevent the model from solving any publicly released task

A) To avoid documenting criteria, slices, and evaluation rules

B) To reduce test-specific tuning and track dataset versions

C) To prevent the model from solving any publicly released task

Answer and explanation

Correct answer: B. Privacy reduces direct tuning to the test; versioning ties a result to specific items, rules, slices, and date. The set becomes reproducible change control.

Question 19.15.2

19.15. A private golden set in seven deliberate steps

A private set repeatedly guides prompt tuning. What changes?

One correct optionAIt automatically becomes weight trainingBIt becomes development evidence; a separate final audit is neededCSecrecy alone preserves its independence

A) It automatically becomes weight training

B) It becomes development evidence; a separate final audit is needed

C) Secrecy alone preserves its independence

Answer and explanation

Correct answer: B. Selection based on results creates adaptation without gradients or public task disclosure.

Question 19.17.1

19.17. Read two passports: the benchmark and the reported run

What should a reported-result passport contain beyond benchmark name and score?

One correct optionAOnly parameter count, context length, and model nameBOnly run date, GPU type, and the final aggregate scoreCVersion, protocol, attempts, tools, cost, and uncertainty

A) Only parameter count, context length, and model name

B) Only run date, GPU type, and the final aggregate score

C) Version, protocol, attempts, tools, cost, and uncertainty

Answer and explanation

Correct answer: C. The same benchmark name does not guarantee the same protocol. Without a full passport, the result cannot be reproduced or attributed to model versus system scaffolding.