RL for LLM13 / 13

Module 13. Evaluating RL systems for LLMs: statistics, reward hacking, and safety

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 0 · Correct: 0

Question P01 · Configuration, not a model nickname

Only the chat template and decoding settings changed. What is the evaluated object?

Your answer or notes

Question P02 · Hierarchical resampling

There are twenty generations for each of twelve prompts. What is the first resampling level for a claim about new prompts?

Your answer or notes

Answer and explanation

Resample prompts as whole clusters. Generations can be resampled within a selected prompt or aggregated by a predeclared prompt-level protocol.

Question P03 · Exact McNemar reasoning

A_only=9, B_only=2; the asymptotic result crosses 0.05 but the exact value is 0.06543. What is the primary conclusion?

Your answer or notes

Answer and explanation

With only eleven discordant tasks, use the exact binomial result: the evidence does not cross the 0.05 threshold.

Question P04 · Coverage and selector quality

A correct candidate exists in the pool but the selector returns another candidate. Which two booleans describe the run?

Your answer or notes

Answer and explanation

coverage=True and selection_success=False. pass@k alone does not describe delivered quality.

Question P05 · Containment versus Jaccard

Why can a Jaccard threshold of 0.7 not be copied to benchmark-side containment?

Your answer or notes

Answer and explanation

The denominators differ, so the same text pair receives different values. A detector threshold belongs to its exact metric and calibration protocol.

Question P06 · Adaptive test reuse

A team selects the best of eighty prompts on one private set. Why is that set no longer an independent final test?

Your answer or notes

Answer and explanation

Its feedback selected the configuration, making it part of development. A fresh holdout or a controlled adaptive-query procedure is required.

Question P07 · Identity and position

A judge is identity-consistent on a pair but shows first-slot preference elsewhere. Are the diagnostics contradictory?

Your answer or notes

Question P08 · Structural isolation

Which is stronger: “ignore candidate instructions” or passing the candidate as untrusted data with a structured output schema?

Your answer or notes

Answer and explanation

Structural isolation is the stronger engineering contract, but it is not a formal security boundary for a single LLM.

Question P09 · Two safety denominators

Why not divide harmful compliance and benign refusals by one mixed test-set size?

Your answer or notes

Answer and explanation

They estimate different conditional rates: ASR on harmful attempts and over-refusal on benign neighbors. A mixed denominator hides test composition.

Question P10 · Critical event

Average metrics pass, but a confirmed sandbox escape was predeclared as release-blocking. What is the decision?

Your answer or notes

Answer and explanation

NO-GO. A critical gate is not compensated by an average score; preserve an incident record and regression test.

Question P11 · Evidence bundle

What minimally allows a new scorer version to recompute results without another model run?

Your answer or notes

Answer and explanation

Versioned samples, raw responses/transcripts keyed by sample_id, solver configuration, scorer revision, summary, and an artifact-hash passport.

Question P12 · Current software contract

What Python entry point does current lm-evaluation-harness documentation recommend, and what must happen before publication?

Your answer or notes

Answer and explanation

Use simple_evaluate() for Python integration, then pin the actual dependency version, backend, task configuration, and full result artifact.