M/W 4pm to 5:20pm, 4025 CIF | Please refresh the page twice to make sure you are seeing the latest updates

First day information

Slides, Notes, Homework and MPs

Tentative course calendar:

🟢 = must read. 🔵 = additional reading.

• Generative models overview | - Lecture 1 slides | 🔵 Gen AI overview youtube (Stanford) 🔵 History of diffusion youtube (Yang Song) | ✍️ HW 1 (foundations) | | | 2. Wed Aug 27 | Review1: Probability
• Bayes, MLE, Multivariate, …
• Conditional independence, Markov
• Expectation Maximization (EM) | - Lecture 2 slides
• EM_notes.pdf | 🔵 Probability review (online book) | | | | 3. Mon Sep 1 | Review2: Probability, Linear Algebra, DL
• EM continued
• PCA, SVD, …
• Basic deep learning | - Lecture 3 slides
• PCA, SVD notes | 🔵 Gilbert Strang’s lecture # 🔵 EM tutorial by D. Lin PDF | | 🖥️ MP 1 | | 4. Wed Sep 3 | Variational Inference, VAE
• AutoEncoder (AE)
• Semantics
• VAE (moon eclipse), ELBO | - Lecture 4 slides | 🟢 Intro. to VAEs (Kingma): chap. 1,2 | | | | 5. Mon Sep 8 | VAE visualization and problems
• Entropy push pull
• Prior hole
• Posterior collapse | - Lecture 5 slides | 🟢 Prior hole paper (UIUC) PDF 🔵 IMUV paper (UIUC) 🔵  VQ-VAE paper (NeuRIPS) 🔵 Learning latent prior paper (UCLA) | 🚫 HW 1 | | | 6. Wed Sep 10 | Diffusion Models 1
• Hierarchical VAEs
• Variational diffusion models (VDM) | - Lecture 6 slides | 🟢 Luo’s tutorial, chapters 1-2 | | | | 7. Mon Sep 15 | Diffusion Models 2
• Gaussian encoders
• Updated ELBO | - Lecture 7 slides | 🟢 Luo’s tutorial, chapters 3-4 🔵 Stanley Chan tutorial | | | | 8. Wed Sep 17 | Diffusion Models 3
• Langevin dynamics
• Tweedie’s formula
• Score function
• Noise annealing | - Lecture 8 slides | 🟢 Grad. of data distribution (Song) blog 🟢 Score Matching (Helsinki) PDF 🔵 Elad’s lecture 3 video youtube | | | | 9. Mon Sep 22 | Diffusion Models 4
• Zoom out / big picture
• Visuals of Gaussians on straight line
• Distribution contracting from t=0 → T | - Lecture 9 slides | 🔵 Learning from thermodynamics PDF | | | | 10. Wed Sep 24 | Recent papers on Diffusion
• Papers
• DDPM
• ?? | None | 🟢 DDPM paper (Berkeley) PDF 🔵 Improved DDPM paper (OpenAI) PDF 🔵 DDIM paper (Stanford) paper PDF 🔵 Diffusion language models video | | | | 11. Mon Sep 29 | Guidance
• Classifier based guidance
• Derivative of logits
• Classifier free guidance
• Scaling factor $s$ | - Lecture 11 slides | 🟢 GLIDE T2I models (OpenAI) PDF 🔵 Classifier free guidance (OpenAI) PDF | | | | 12. Wed Oct 1 | CLIP and Text-to-Image (T2I) Models
• T2I problem statement
• CLIP
• Latent Diffusion Model (LDM) | - Lecture 12 slides | 🟢 CLIP: Visual from language PDF 🔵 Latent Diffusion paper (Heidelberg) | | | | 13. Mon Oct 6 | Inverse problems + Posterior sampling
• IP problem statement
• Approximate conditional score
• Diffusion based IP
• DPS
• Pi-GDM | - Lecture 13 slides | 🟢 Diffusion posterior sampling (KAIST) PDF 🔵 Pi-GDM (NVidia) paper | | | | 14. Wed Oct 8 | Recent papers on inverse problems
• Colorization
• Deblurring
• Inpainting
• Source (speech) separation
• Motion tracking and mapping | - Lecture 14 slides | 🟢 RePaint and inpaint paper (ETH) 🟢 ArrayDPS paper (UIUC) 🟢 Image super resolution paper (Google) 🔵 MapDiffusion paper (UIUC) | | | | 15. Mon Oct 13 | Introduction to differential equations
• DE problem statement
• GD as ODE
• Brownian / Wiener process
• Vector fields
• ODE/SDE | - Lecture 15 slides | 🟢 Chan’s tutorial (Purdue), Chapter 4 | | | | 16. Wed Oct 15 | Flows 1
• Diffusion and ODE/SDE
• Reverse ODE/SDE
• Fokker Planck and Kolmogorov
• Solvers: Euler and Runga Kutta | - Lecture 16 slides | 🟢 Flow and Diffusion tutorial (MIT), chap. 1 | | | | 17. Mon Oct 20 | Flows 2
• Conditional and marginal probability path
• Designing target vector field
• Continuity equation
• Zoom out → visualize
• Score functions | - Lecture 17 slides | 🟢 Flow and Diffusion tutorial (MIT), chap. 2 🔵 Divergence video (Khan Academy) | | | | 18. Wed Oct 22 | Flow and score matching
• Conditional flow matching
• Conditional to marginal
• CFM is surrogate loss for FM
• Score matching extension
• Special case: Gaussian
• For Gaussian, score comes free | - Lecture 18 slides | 🟢 Flow tutorial (MIT), chapters 3 and 4 🔵 Lilian’s blog 🔵 Flows_foundations_slides.pdf (UIUC) | | | | 19. Mon Oct 27 | Recent papers in flows and flow matching
• Distillation of ideas
• Flow matching paper | - Lecture 19 slides | 🟢 Flow matching paper (Meta) 🔵 Rectified flows paper (UT Austin) | | | | 20. Wed Oct 29 | Composition
• Problem statement
• Decomposing conditional score (Toralba)
• Progressive guidance paper
• MultiDiffusion | - Lecture 20 slides | 🟢 Composing diffusion models paper (MIT) 🟢 Diffusion seeds blog (Reddit) 🔵 Progressive guidance paper (Sydney) | | | | 21. | Editing
• Problem statement
• Jacobian of Denoiser
• LOCO Edit | - Lecture 21 slides | 🟢 LOCO Edit paper (Michigan) 🔵 LOCO seminar youtube (Michigan) | | | | 22. | Recent papers on controlled diffusion
• RB Modulation
• Tweedie Mix
• Minority focussed T2I models | - Lecture 22 slides | 🟢 MultiDiffusion paper (Weizmann) 🟢 RB_Modulation paper (UT Austin) 🟢 TweedieMix paper (KAIST) | | | | 23. | Review, big picture, and plugging holes | - Lecture 23 slides | 🔵 Comp. imaging seminar youtube (KAIST) | | | | 24. | MIDTERM exam | | | | | | 25. | Generative Adversarial Networks (GANs)
• Problem statement
• Loss function design
• Comparison with MLE
• Properties
• Results | - Lecture 25 slides | 🟢 NeurIPS tutorial by Goodman 🔵 Adversarial AutoEncoders paper (Toronto) | | | | 26. | Sampling methods:
• Inverse Transform Sampling (ITS),
• Accept / reject sampling,
• Importance sampling, MCMC | - Lecture 26 slides | 🟢 First 20 pages of this tutorial 🔵 MCMC for ML Tutorial | | | | 27. | Topics not covered:
• Transformers
• LLMs
• NeRFs
• RL | - Lecture 27 slides | 🔵 How LLMs work youtube (3Brown1Blue) 🟢 NeRF paper (Berkeley) 🔵 RL lectures 2 and 3 youtube (Berkeley) | | | | 28. | Wrap up and zoom out | - Lecture 28 slides | | | | | 29. | Buffer | | | | | | 30. | Buffer | | | | | | | FINAL Exam | | | | |