RL for LLM
Reinforcement learning, preference optimization, verifiable rewards, agents, and safety.
Contents
- Module 1. Introduction: Reinforcement Learning for LLMs · Quizzes · Exercises · Notebooks
- Module 2. State, Value, and Bellman Equations · Quizzes · Exercises · Notebooks
- Module 3. Bandits, exploration, and one-step decisions · Quizzes · Exercises · Notebooks
- Module 4. Policy gradients: from REINFORCE to PPO · Quizzes · Exercises · Notebooks
- Module 5. Learning from fixed data: imitation, deep Q-methods, and offline RL · Quizzes · Exercises · Notebooks
- Module 6. The LLM as a policy: tokens, log-probabilities, and KL · Quizzes · Exercises · Notebooks
- Module 7. Reward models and preference data · Quizzes · Exercises · Notebooks
- Module 8. RLHF and direct preference optimization · Quizzes · Exercises · Notebooks
- Module 9. RLVR and GRPO: verifiable rewards, group-relative estimates, and stability · Quizzes · Exercises · Notebooks
- Module 10. Search, verification, and test-time compute · Quizzes · Exercises · Notebooks
- Module 11. RL for LLM agents: tools, environments, and long trajectories · Quizzes · Exercises · Notebooks
- Module 12. RL infrastructure for LLMs: memory, rollouts, and asynchrony · Quizzes · Exercises · Notebooks
- Module 13. Evaluating RL systems for LLMs: statistics, reward hacking, and safety · Quizzes · Exercises · Notebooks