RL for LLM7 / 13

Module 7. Reward models and preference data

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 0 · Correct: 0

Question P01 · Original pair orientation

A log keeps only chosen/rejected; the original A/B order is lost. Which checks become impossible?

Your answer or notes

Answer and explanation

You can no longer measure positional bias honestly, reconstruct the observed 0/1 label in the original orientation, or build a sound calibration set. Store the original order, candidate identifiers, and the annotator decision.

Question P02 · Comparison-graph connectivity

Two sets of responses were never compared across sets. Can their absolute BT scores be compared?

Your answer or notes

Answer and explanation

No. Each connected component admits an independent additive score shift. Bridging comparisons or an external anchor are required; centering within components does not create cross-component information.

Question P03 · Row split without prompt isolation

Why can a random row split overstate reward-model quality?

Your answer or notes

Answer and explanation

Pairs from the same prompt may appear in both train and test. The model can exploit prompt, generator, or template identity instead of transferring the criterion. Use a prompt-disjoint split and add temporal/domain slices.

Question P04 · Calibration on the test set

Temperature scaling is fitted on the same set used to report ECE. What is wrong?

Your answer or notes

Answer and explanation

The estimate is optimistic because the test set tuned the calibrator. Use train, calibration, and evaluation splits; fit temperature on calibration and report metrics on untouched evaluation data.

Question P05 · Accuracy without calibration

Two RMs have the same pair accuracy but different log loss. What does that show?

Your answer or notes

Answer and explanation

They get the sign right equally often but differ in probability quality and confidence. Lower log loss usually indicates better probabilistic predictions; accuracy cannot reveal that.

Question P06 · Averaging raw ensemble scores

Why is raw-score averaging across independently trained RMs unsafe?

Your answer or notes

Answer and explanation

Each score has an arbitrary origin and scale. Calibrate each model on a common calibration set, convert margins to probabilities, and aggregate probabilities. Ensemble spread is a diagnostic, not automatically a confidence interval.

Question P07 · Proxy overoptimization

Proxy score rises with Best-of-N while independent utility falls after N=32. What follows?

Your answer or notes

Answer and explanation

Selection pressure is exploiting a proxy direction. Cap the search pressure, add independent metrics and fresh labels, test distribution shift, and never select N using the proxy alone.

Question P08 · Judge swap test

An LLM judge chooses position A 70% of the time and preserves the same textual winner after swapping only 55% of the time. What is diagnosed?

Your answer or notes

Answer and explanation

Strong position sensitivity. Evaluate both orientations, aggregate explicitly, inspect ties, and include independent human audit.

Question P09 · Soft tie

Annotators split 6:4. Why does a hard label of 1 discard information?

Your answer or notes

Answer and explanation

It makes a weak majority indistinguishable from unanimity. A soft target of 0.6 or an annotator model preserves uncertainty and supports calibration analysis.

Question P10 · Reward model versus cost model

Can a separate cost model always be replaced by negative reward?

Your answer or notes

Answer and explanation

No. Utility and risk may depend on different features, thresholds, and error costs. Log and validate them separately; critical constraints should not be compensated by high utility.

Question P11 · Fresh policy responses

Why is RM evaluation only on old candidates insufficient before RL?

Your answer or notes

Answer and explanation

Optimization shifts the response distribution into regions not covered by the old set. Evaluate fresh current-policy generations, generator/domain slices, and independent labels.

Question P12 · Outcome and process reward

When are whole-answer and step-level scores not interchangeable?

Your answer or notes

Answer and explanation

An outcome score does not locate the faulty step, while local scores need not compose into a correct whole-answer value. A PRM needs its own annotation semantics and validation.