RL for LLM12 / 13

Module 12. RL infrastructure for LLMs: memory, rollouts, and asynchrony

Choose your answers and select “Check”. Numbers accept a decimal point, comma, or simple fraction. Answers and drafts are saved in this browser and shared between RU/EN.

Use the author’s worked answers to review open questions and code tasks yourself. Code does not run on this site.

Checked: 0 / 10 · Correct: 0

Question P01 · Placement does not determine synchrony

The rollout engine is a separate vLLM server and the trainer is another process. Which statement is correct?

One correct option1The system is asynchronous by definition.2It may remain synchronous if the trainer waits for the whole batch and the next generation phase starts only after the update.3Disaggregation automatically removes off-policy data.4Disaggregation guarantees a smaller total cluster-memory peak.

The system is asynchronous by definition.

It may remain synchronous if the trainer waits for the whole batch and the next generation phase starts only after the update.

Disaggregation automatically removes off-policy data.

Disaggregation guarantees a smaller total cluster-memory peak.

Answer and explanation

The correct answer is B.

Placement asks where roles execute; synchrony asks when generation and updates are allowed to proceed. A separate server can still participate in a strict batch-barrier loop. Disaggregation also neither removes policy lag nor guarantees lower total memory, because state may be duplicated across pools.

Question P02 · Logical full-training memory

A model has 7 billion parameters. The profile uses 2 bytes for each model parameter, 2 bytes for its gradient, and 8 bytes for the two Adam moments. There is no master copy.

Compute the logical full-training state:

in decimal GB;
in binary GiB.

Round GiB to two decimal places.

Memory, GBAbsolute tolerance: ±0.000001Memory, GiBAbsolute tolerance: ±0.005

Answer and explanation

Each trainable parameter uses

2+2+8=12\ \text{bytes}.

Therefore,

7\cdot10^9\cdot12 = 84\cdot10^9\ \text{bytes} = 84.00\ \text{GB}.

In GiB,

\frac{84\cdot10^9}{2^{30}} \approx78.23\ \text{GiB}.

This is a logical model-state ledger. Activations, communication buffers, fragmentation, and temporary full-parameter materialization are excluded.

Question P03 · KV cache for long responses

A teaching architecture has:

32 layers;
8 KV heads;
head dimension 128;
bf16 values, or 2 bytes per element;
16 sequences with 8192 retained tokens each.

Estimate the useful KV-cache payload in GiB. Exclude pages, block tables, and temporary buffers.

KV cache, GiBAbsolute tolerance: ±0.000001

Answer and explanation

Use

2+2+8=12\ \text{bytes}.

The retained-token count is

N_{\mathrm{tok}}=16\cdot8192=131072.

Thus,

B_{\mathrm{KV}} = 2\cdot32\cdot8\cdot128\cdot131072\cdot2 = 17179869184\ \text{bytes} = 16\ \text{GiB}.

The leading factor two accounts for keys and values.

Question P04 · Two architectural axes

Match each description to one combination.

One GPU pool; generation finishes before training starts.
Separate GPU pools with a shared batch barrier.
One pool releases memory between phases and overlaps only limited preparation.
A queue connects separate pools and the trainer may consume multiple policy versions.

Combinations:

A. colocated + synchronous;
B. disaggregated + synchronous;
C. colocated + limited overlap;
D. disaggregated + asynchronous.

Match each item to an answer1234

A) colocated + synchronous;

B) disaggregated + synchronous;

C) colocated + limited overlap;

D) disaggregated + asynchronous.

Answer and explanation

The mapping is:

1 → A;
2 → B;
3 → C;
4 → D.

The exercise separates role placement from the order in which data may be produced and consumed.

Question P05 · Pipeline fill and drain

Generating one batch takes $G=45$ s and training it takes $T=30$ s. There are $n=8$ batches.

Compute:

sequential time;
ideal two-stage pipeline time;
speedup.

Round the speedup to three decimal places.

Sequential time, sAbsolute tolerance: ±0.000001Pipeline time, sAbsolute tolerance: ±0.000001SpeedupAbsolute tolerance: ±0.001

Answer and explanation

Sequential time is

2+2+8=12\ \text{bytes}.

The ideal two-stage pipeline takes

t_{\mathrm{pipe}} = G+T+(n-1)\max(G,T) = 45+30+7\cdot45 = 390\ \text{s}.

Hence,

\frac{600}{390}\approx1.538.

This assumes independent resources and omits communication, queues, and duration variance.

Question P06 · A straggling response

Four rollouts run concurrently for 4, 4, 4, and 12 seconds. The batch closes only after the slowest rollout.

Compute useful utilization of the four-worker capacity.

Utilization fractionAbsolute tolerance: ±0.000001

Answer and explanation

Useful work is

2+2+8=12\ \text{bytes}.

Capacity available before the barrier is

4\cdot12=48\ \text{worker-seconds}.

Therefore,

U=\frac{24}{48}=0.5.

Half of the capacity is lost while workers wait for the slow response; a mean duration alone hides this loss.

Question P07 · First queue check

A queue receives 8 thousand tokens per second on average, while the trainer processes 10 thousand tokens per second.

Compute $\rho=\lambda/\mu$ .
Does the system pass the first-order stability check?
Why does this not guarantee short delays?

ρAbsolute tolerance: ±0.000001

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

Answer and explanation

We obtain

2+2+8=12\ \text{bytes}.

Because $\rho<1$ , mean service capacity exceeds mean arrival rate and the first-order check passes.

It is not a latency guarantee. Bursts, correlated long completions, heavy tails, synchronization pauses, and bounded buffers can still create large queues. Measure the age distribution and enforce explicit backpressure.

Question P08 · Effective sample size

Importance weights for four rollouts are $(1,1,1,5)$ .

Compute:

ESS;
normalized ESS.

Round to four decimal places.

ESSAbsolute tolerance: ±0.0001Normalized ESSAbsolute tolerance: ±0.0001

Answer and explanation

Use

2+2+8=12\ \text{bytes}.

The weight sum is 8 and the sum of squares is 28:

\mathrm{ESS} = \frac{64}{28} = \frac{16}{7} \approx2.2857.

Normalized ESS is

\frac{2.2857}{4} = \frac47 \approx0.5714.

This diagnoses weight concentration; it does not prove absence of bias.

Question P09 · Version age

The current trainer version is 112. Rollouts were collected by versions $(108,109,110,110)$ .

Compute:

version lags;
mean lag;
maximum lag.

Mean version lagAbsolute tolerance: ±0.000001Maximum version lagAbsolute tolerance: ±0

Only your selection or numeric part is checked automatically. Compare your reasoning with the author’s worked answer.

Answer and explanation

The lags are

2+2+8=12\ \text{bytes}.

Their mean is

\bar L=\frac{4+3+2+2}{4}=2.75,

and the maximum is 4. This metric requires a monotone, unambiguous version scheme; one scalar is insufficient for branched runs.

Question P10 · Checkpoint interval

Writing a checkpoint takes $C=120$ s and the mean time between failures in the simplified model is $M=8$ hours.

Using Young's formula

I^*=\sqrt{2CM},

compute the interval in seconds and minutes.

Interval, sAbsolute tolerance: ±0.01Interval, minAbsolute tolerance: ±0.01

Answer and explanation

Convert eight hours:

2+2+8=12\ \text{bytes}.

Then

7\cdot10^9\cdot12 = 84\cdot10^9\ \text{bytes} = 84.00\ \text{GB}.

This is a first-order approximation. Recovery time, multilevel storage, correlated failures, and asynchronous writes change the practical policy.

Question P11 · Zero lag but nonzero mismatch

Rollout and trainer recomputation use the same weight version, so policy lag is zero. Sequence log-probabilities still differ.

Name at least four possible causes and give the correct diagnostic order.

Your answer or notes

Answer and explanation

Possible causes include:

different tokenization or chat templates;
different logit processors or decoding rules;
different response masks or causal shifts;
precision and reduction-order differences;
different kernels or parallel layouts;
incorrect selected-token reconstruction.

First freeze tokens and masks, compare post-processed logits and log-probabilities at zero lag, and locate the first diverging token. Only after the contract matches should one study off-policy drift. Importance correction must not conceal a contract bug.

Question P12 · Which rollout can be accepted

A collector sends tokens, a response mask, reward, and policy_version, but omits:

tokenizer and chat-template versions;
decoding parameters;
verifier version;
termination versus truncation.

May the trainer use this record? What should happen?

Your answer or notes

Answer and explanation

The record must not be accepted automatically. A weight version does not fully determine the behavior policy: tokenization, templating, and decoding matter. Without a verifier version, reward provenance is unknown; without stop semantics, bootstrap targets and masks cannot be reconstructed safely.

Reject or quarantine the record. Recover fields only from an immutable authoritative log, then rerun schema validation. Filling them from the current configuration would silently rewrite history.