RL for LLM2 / 13

Module 2. State, Value, and Bellman Equations

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 8 · Correct: 0

Question P01 · State sufficiency as a testable claim

Two visible state records are identical and the action is the same, yet a tool's next response depends on a hidden authorization token. What follows?

One correct option1The Markov property holds because the visible text matches.2The representation is incomplete; the authorization state or a sufficient summary must be represented.3Any randomness violates the Markov property.4Reward, rather than state, must be changed.

The Markov property holds because the visible text matches.

The representation is incomplete; the authorization state or a sufficient summary must be represented.

Any randomness violates the Markov property.

Reward, rather than state, must be changed.

Answer and explanation

Answer: 2.

A Markov model may be stochastic, but the conditional next-transition distribution must be determined by the represented state and action. A hidden authorization variable that changes that distribution makes the visible representation insufficient.

Question P02 · Reward on a terminal transition

An agent enters the goal and receives $+5$ . The terminal-state value is zero. What target should the predecessor receive for any $\gamma$ ?

TargetAbsolute tolerance: ±0.000001

Answer and explanation

Answer: $5$ .

y=5+\gamma\cdot0=5.

Zero terminal value removes future continuation; it does not remove reward paid on entry.

Question P03 · The numerical cost of a wrong mask

Let $r=1$ , $\gamma=0.9$ , and $V(s')=2$ . Compute the target for a true terminal and for external truncation. How much does the wrong terminal mask understate the truncated target?

True terminalAbsolute tolerance: ±0.000001External truncationAbsolute tolerance: ±0.000001Target underestimationAbsolute tolerance: ±0.000001

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

Answer and explanation

The targets are $1$ and $1+0.9\cdot2=2.8$ . Treating truncation as terminal understates the target by $1.8$ . If continuation value were negative, the direction of the bias would reverse.

Question P04 · Policy-weighted zero advantage

A policy has probabilities $[0.2,0.3,0.5]$ and advantages $[-2,1,x]$ . Find $x$ if the advantages are correctly centered under the policy.

xAbsolute tolerance: ±0.000001

Answer and explanation

Solve

5

which gives $x=0.2$ . The unweighted sum need not vanish.

Question P05 · Bellman residual versus value error

For a fixed policy, $\gamma=0.95$ and $\|T_\pi\widehat V-\widehat V\|_\infty=0.02$ . What upper bound follows for the value error? Is it the exact error?

Error upper boundAbsolute tolerance: ±0.000001

Answer and explanation

\|\widehat V-V^\pi\|_\infty \le \frac{0.02}{1-0.95}=0.4.

This is an upper bound, not an equality; the actual error may be much smaller.

Question P06 · What changes at gamma = 1

Choose the most accurate statement.

One correct option1No episodic problem can have finite value at

\gamma=1

.2The standard strict-contraction proof no longer applies; extra properties of the episodic process are needed.3Bellman equations become invalid.4Adding a terminal state guarantees that every policy reaches it almost surely.

No episodic problem can have finite value at $5$ .

The standard strict-contraction proof no longer applies; extra properties of the episodic process are needed.

Bellman equations become invalid.

Adding a terminal state guarantees that every policy reaches it almost surely.

Answer and explanation

Answer: 2.

The contraction coefficient is no longer below one. Proper episodic tasks may still be well posed, but absorption and nonsingularity must be established separately.

Question P07 · Reading FrozenLake value

Reward $+1$ is paid on entry to the goal, all other rewards are zero, and $\gamma=0.9$ . What is the contribution to $G_0$ of success after 4 transitions? After 7 transitions?

After 4 transitionsAbsolute tolerance: ±0.000001After 7 transitionsAbsolute tolerance: ±0.000001

Answer and explanation

Success after $1$ transitions contributes $1+0.9\cdot2=2.8$ .

$T=4$ : $0.9^3=0.729$ .
$T=7$ : $0.9^6=0.531441$ .

Thus discounted value combines success probability with a preference for earlier success.

Question P08 · Where bootstrapping begins

A target contains three observed rewards followed by $\gamma^3V(S_{t+3})$ . Which part is bootstrapped, and why?

Your answer or notes

Answer and explanation

The term $1$ is bootstrapped because the unobserved continuation is replaced by a current estimate. The preceding rewards were observed directly.

Question P09 · Another multi-step target

Rewards are $[2,0,-1]$ , $\gamma=0.9$ , and $V(S_{t+3})=3$ . Compute the three-step target for a non-terminal end of the window and for a true terminal on the third transition.

Nonterminal window endAbsolute tolerance: ±0.000001True terminalAbsolute tolerance: ±0.000001

Answer and explanation

Non-terminal:

5

Terminal:

2-0.81=1.19.

Question P10 · On-policy without defining it by a replay buffer

Behavior is $\varepsilon$ -greedy. The update target represents the same $\varepsilon$ -greedy policy, while transitions are kept for one training epoch. Is the method on-policy or off-policy in the module's conceptual sense?

Your answer or notes

Answer and explanation

It is on-policy as long as the target and behavior policies match and the stored data have not become stale relative to a changed policy. Buffer presence alone does not define the category; the degree of reuse and policy drift should be stated separately.

Question P11 · Interpreting CliffWalking without a contradiction

In one run, Q-learning has a greedy return of $-13$ and SARSA has $-17$ , yet SARSA obtains better average return during $\varepsilon$ -greedy training. Is this contradictory?

Your answer or notes

Answer and explanation

No. Greedy evaluation removes exploration, while training return includes exploratory deviations. Q-learning can learn the short cliff-edge route and suffer costly falls during exploration. SARSA accounts for future exploratory actions and may prefer a longer but safer route.

Question P12 · Best-of-many under a noisy evaluator

Ten equally good responses receive independent zero-mean noisy scores, and the maximum-scoring response is selected. Is the expected maximum score equal to true quality?

Your answer or notes

Answer and explanation

Generally not. The maximum preferentially selects a positive noise realization, so the expected winning score is optimistic. Independent reevaluation can reduce this selection effect, but does not make the evaluator error-free.