RL for LLM11 / 13

Module 11. RL for LLM agents: tools, environments, and long trajectories

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 4 · Correct: 0

Question P01 · Full history and hidden state

Consider a POMDP with a fixed environment model and initial distribution. A complete log contains every action, observation, and received reward, but an access flag has never been directly observed. Does uncertainty about the flag prevent the complete log from being a Markov information state? How does a belief state differ, and what can change when the log is truncated?

Your answer or notes

Answer and explanation

It does not. Under these assumptions, the complete history $H_t$ determines the conditional distribution of the future and can serve as a growing Markov information state without identifying the hidden flag. The belief $b_t(s)=P(s_t=s\mid H_t)$ is a distribution over hidden states that summarizes history for prediction; it does not add a missing observation. A text log is not automatically an explicitly computed $b_t$ . A truncated window or summary can discard relevant observations and therefore need not remain sufficient. Different hidden realizations compatible with one full history express uncertainty, not a violation of the Markov property.

Question P02 · A controller-generated token sequence

The model emits the symbolic action OPEN_RED, after which a controller serializes it into long JSON. Which tokens belong in the policy loss?

Your answer or notes

Answer and explanation

If the model chose only the symbolic action, the learned action is OPEN_RED and its recorded probability. The JSON tokens were produced by a deterministic controller and receive no policy-gradient term. A stochastic serializer treated as part of a composite policy would need its own logged probability.

Question P03 · Collector truncation

A transition gives reward 0, the next value is 0.7, and $\gamma=0.9$ . Collection stops, but the environment has not terminated. Compute the one-step critic target.

TargetAbsolute tolerance: ±0.000001

Answer and explanation

Bootstrap through the collector boundary: $H_t$ . A zero target would correspond to true task termination or a separately declared no-continuation objective.

Question P04 · Semantic action probability

Two JSON strings parse to the same action and have probabilities 0.25 and 0.35. What is the semantic-action probability?

Action probabilityAbsolute tolerance: ±0.000001

Answer and explanation

The mutually exclusive serializations add: $H_t$ . A standard token loss still scores the sampled string unless it explicitly sums over all equivalent serializations.

Question P05 · Turn probability ratio

The turn log-probability sums under current and behavior policies are $-2.1$ and $-2.4$ . Compute the sequence ratio.

Probability ratioAbsolute tolerance: ±0.00001

Answer and explanation

$r=\exp[-2.1-(-2.4)]=e^{0.3}\approx1.34986$ . Products across many turns can become heavy-tailed even when each local ratio looks modest.

Question P06 · Synchronous straggler cost

Four rollouts take 1, 1, 2, and 8 seconds. What is utilization for four synchronous workers?

Utilization fractionAbsolute tolerance: ±0.000001

Answer and explanation

Useful work is 12 worker-seconds; batch capacity is $H_t$ . Utilization is $12/32=0.375$ and idle fraction is $0.625$ . Module 12 develops the asynchronous systems consequences.

Question P07 · Branching without a state snapshot

An agent branches after a search call, but every branch queries the live search service again. Are the actions compared from one state?

Your answer or notes

Answer and explanation

Not necessarily. Index contents, ranking, and hidden session state may differ. A local comparison requires a restorable snapshot or deterministic replay, fixed randomness, and the same shared prefix; otherwise action effects are confounded with environment differences.

Question P08 · A homogeneous group

All eight rewards in a rollout group equal 1. What remains after subtracting the group mean?

Your answer or notes

Answer and explanation

Every relative advantage is zero. A larger learning rate cannot create missing variation. Change the task distribution, group design, or signal source. KL and entropy terms may still have gradients, but they are different signals.

Question P09 · Repeatable inspect reward

Why is $+0.1$ for every inspect more dangerous than a first-visit bonus?

Your answer or notes

Answer and explanation

A repeatable bonus turns a loop into an independent source of return. First-visit payment limits repetition but still lacks the general invariance theorem. Potential-based shaping has a stricter form under its assumptions.

Question P10 · What an agent benchmark measures

A browser-controller replacement improves success without changing model weights. Can the gain be attributed to RL?

Your answer or notes

Answer and explanation

No. The score entangles the model, controller, observation representation, budget, retries, environment, and verifier. Attribution requires ablations with a fixed execution contract.

Question P11 · Online or offline first

The environment sends real emails and has no safe reset. Which initial training route is preferable?

Your answer or notes

Answer and explanation

Start with a simulator, shadow mode, expert logs, or state-conditioned offline preferences. Online RL with irreversible production actions is unacceptable without sandboxing, least privilege, and a confirmation protocol.

Question P12 · Experiment passport

Name at least six fields needed to reproduce a multi-turn rollout.

Your answer or notes

Answer and explanation

At minimum: task/environment ID and version; initial state or seed; actions and observations; policy token IDs and provenance mask; behavior log-probabilities; policy version per turn; decoding settings; rewards and verifier version; stop reason with terminated/truncated/infra status; duration and cost. External tools make exact versions and state snapshots even more important.