Information Theory for ML
Information theory from entropy and coding to NLP, LLMs, computer vision, and reinforcement learning.
Contents
- Module 1. Introduction · Quizzes
- Module 2. Entropy · Quizzes · Exercises
- Module 3. Cross-Entropy and KL Divergence · Quizzes · Exercises
- Module 4. Jensen's inequality: when averaging meets nonlinearity · Quizzes · Exercises
- Module 5. Mutual information: how much one object tells us about another · Quizzes · Exercises
- Module 6. Source coding: from entropy to real bits · Quizzes · Exercises
- Module 7. Channel and capacity · Quizzes · Exercises
- Module 8. Maximum entropy, exponential models, and KL regularization · Quizzes · Exercises
- Module 9. Rate–distortion and the Information Bottleneck: managing information loss · Quizzes · Exercises
- Module 10. Algorithmic Information Theory: Shortest Descriptions, MDL, and Universal Prediction · Quizzes · Exercises
- Module 11. Comparing distributions: f-divergences, variational critics, and Wasserstein distance · Quizzes · Exercises
- Module 12. Information-theoretic generalization bounds: PAC-Bayes and mutual information · Quizzes · Exercises
- Module 13. Reasoning in LLMs: computation, search, and information · Quizzes · Exercises
- Module 14. Computer Vision Through Information Theory · Quizzes · Exercises
- Module 15. Information Geometry: Fisher, Natural Gradient, and Distribution Flows · Quizzes · Exercises
- Module 16. Course Reference Sheet