M/W 4pm to 5:20pm, 4025 CIF | Please refresh the page twice to make sure you are seeing the latest updates
First day information
Homework and MPs
Tentative course calendar:
🟢 = must read. 🔵 = additional reading.
| Lecture |
Topics |
Lecture material (web folder) |
Reading |
Homework |
MP |
| 1. Mon |
|
|
|
|
|
| Aug 25 |
Course logistics + big picture |
|
|
|
|
- Generative models overview | - lec1a_slides_overview.pptx
- lec1_video
| 🔵 Gen AI overview youtube (Stanford)
🔵 History of diffusion youtube (Yang Song) | ✍️ HW 1
(foundations) | |
| 2. Wed
Aug 27 | Review1: Probability
- Bayes, MLE, Multivariate, …
- Conditional independence, Markov
- Expectation Maximization (EM) | - lec2a_slides_prob_EM1.pptx
- lec2b_annotated.pdf
- lec2c_notes_EM.pdf
- Lec2_video | 🔵 Probability review (online book) | | |
| 3. Wed
Sep 3 | Review2: Probability, Linear Algebra, DL
- EM continued
- PCA, SVD, …
- Basic deep learning | - lec3a_slides_EM2_Eigen.pptx
- lec3b_annotated.pdf
- lec3c_notes_Eigen.pdf
- lec3_video | 🔵 Gilbert Strang’s lectures on null space, rank, etc.
🔵 EM tutorial by D. Lin PDF
🔵 SVD video here | | |
| 4. Mon
Sep 8 | Variational Inference, VAE
- AutoEncoder (AE)
- Semantics
- VAE (moon eclipse), ELBO | - lec4a_notes_PCA.pdf
- lec4b_slides_AE_VAE.pptx
- lec4_video | 🟢 Intro. to VAEs (Kingma): chap. 1,2 | | 🖥️ MP 1 |
| 5. Wed
Sep 10 | VAE visualization and problems
- Entropy push pull
- Prior hole
- Posterior collapse | - lec5a_slides_VAE_ELBO.pptx
- lec5b_annotated.pdf
- lec5_video | 🟢 Prior hole paper (UIUC) PDF
🔵 IMUV paper (UIUC)
🔵 VQ-VAE paper (NeuRIPS)
🔵 Learning latent prior paper (UCLA) PDF | 🚫 HW 1
| |
| 6. Mon
Sep 15 | Diffusion Models 1
- Hierarchical VAEs
- Variational diffusion models (VDM) | - lec6a_slides_Diffusion1.pptx
- lec6b_annotated.pdf
- lec6_video | 🟢 Luo’s tutorial, chapters 1-2 | | |
| 7. Wed
Sep 17 | Diffusion Models 2
- Gaussian encoders
- Updated ELBO | - lec7a_slides_Diffusion2.pptx
- lec7b_annotated.pdf
- lec7_video | 🟢 Luo’s tutorial, chapters 3-4
🔵 Stanley Chan tutorial | | |
| 8. Mon
Sep 22 | Diffusion Models 3
- 3 different interpretations
- Image prediction
- Noise prediction
- Tweedie’s formula → score matching | - lec8a_slides_diffusion3.pptx
- lec8b_annotated.pdf
- lec8_video | | | |
| Wed
Sep 24 | No class | | | | |
| 9. Mon
Sep 29 | Diffusion Models 4
- Score as a MMSE estimator
- Langevin dynamics
- Noise annealing
| - lec9a_slides_score.pptx
- lec9b_annotated.pdf
- lec9_video | 🟢 Grad. of data distribution (Song) blog
🟢 Score Matching (Helsinki) PDF
🔵 Elad’s lecture 3 video youtube
🔵 Learning from thermodynamics PDF
| ✍️ HW2
(EM, GMM, VAE) | 🚫 MP1 |
| 10. Wed
Oct 1 | Diffusion Models 5
- Zoom out / big picture
- Annealed Langevin dynamics
- Connect the dots: score, denoising, DDPM
- Time varying force field = differential equations | - lec10a_bigpicture.pptx
- lec10b_annotated.pdf
- lec10_video | 🟢 DDPM paper (Berkeley) PDF
🔵 Improved DDPM paper (OpenAI) PDF
🔵 DDIM paper (Stanford) paper PDF
🔵 Diffusion language models video
| | |
| 11. Mon Oct 6 | Guidance
- Classifier based guidance
- Derivative of logits
- Classifier free guidance
- Scaling factor $s$ | - lec11a_slides_guidance.pptx
- lec11b_annotated.pdf
- lec11_video | 🟢 GLIDE T2I models (OpenAI) PDF
🔵 Classifier free guidance (OpenAI) PDF
| | |
| 12. Wed Oct 8 | CLIP and Text-to-Image (T2I) Models
- T2I problem statement
- CLIP
- Latent Diffusion Model (LDM) | -lec12a_slides_FID-CLIP.pptx
- lec12b_annotated.pdf
- lec12_video | 🟢 CLIP: Visual from language PDF
🔵 Latent Diffusion paper (Heidelberg) | 🚫 HW 2 | 🖥️ MP 2 |
| 13. Mon
Oct 14 | Inverse problems + Posterior sampling
- IP problem statement
- Approximate conditional score
- Diffusion based IP
- DPS | - lec13a_slides_inverse.pptx
- lec13b_annotated.pdf
- lec13_video | 🟢 Diffusion posterior sampling (KAIST) PDF
🔵 Pi-GDM (NVidia) paper | | |
| 14. Wed
Oct 15 | Recent papers on inverse problems
- Pi-GDM
- Colorization
- Deblurring
- Inpainting
- Source (speech) separation
- Motion tracking and mapping | - Lecture 14 slides
- lec14_video | 🟢 RePaint and inpaint paper (ETH)
🟢 ArrayDPS paper (UIUC)
🟢 Image super resolution paper (Google)
🔵 MapDiffusion paper (UIUC)
| | |
| | Introduction to differential equations
- DE problem statement
- GD as ODE
- Brownian / Wiener process
- Vector fields
- ODE/SDE | - Lecture 15 slides | 🟢 Chan’s tutorial (Purdue), Chapter 4
| | |
| | Flows 1
- Diffusion and ODE/SDE
- Reverse ODE/SDE
- Fokker Planck and Kolmogorov
- Solvers: Euler and Runga Kutta | - Lecture 16 slides | 🟢 Flow and Diffusion tutorial (MIT), chap. 1 | | |
| | Flows 2
- Conditional and marginal probability path
- Designing target vector field
- Continuity equation
- Zoom out → visualize
- Score functions | - Lecture 17 slides
| 🟢 Flow and Diffusion tutorial (MIT), chap. 2
🔵 Divergence video (Khan Academy)
| | |
| Wed,
Oct 29 | MIDTERM exam (in class) | | | | |
| | Flow and score matching
- Conditional flow matching
- Conditional to marginal
- CFM is surrogate loss for FM
- Score matching extension
- Special case: Gaussian
- For Gaussian, score comes free | - Lecture 18 slides | 🟢 Flow tutorial (MIT), chapters 3 and 4
🔵 Lilian’s blog
🔵 Flows_foundations_slides.pdf (UIUC) | | |
| | Recent papers in flows and flow matching
- Distillation of ideas
- Flow matching paper | - Lecture 19 slides | 🟢 Flow matching paper (Meta)
🔵 Rectified flows paper (UT Austin) | | |
| | Composition
- Problem statement
- Decomposing conditional score (Toralba)
- Progressive guidance paper
- MultiDiffusion | - Lecture 20 slides | 🟢 Composing diffusion models paper (MIT)
🟢 Diffusion seeds blog (Reddit)
🔵 Progressive guidance paper (Sydney) | | |
| | Editing
- Problem statement
- Jacobian of Denoiser
- LOCO Edit | - Lecture 21 slides | 🟢 LOCO Edit paper (Michigan)
🔵 LOCO seminar youtube (Michigan) | | |
| | Recent papers on controlled diffusion
- RB Modulation
- Tweedie Mix
- Minority focussed T2I models | - Lecture 22 slides | 🟢 MultiDiffusion paper (Weizmann)
🟢 RB_Modulation paper (UT Austin)
🟢 TweedieMix paper (KAIST) | | |
| | Review, big picture, and plugging holes | - Lecture 23 slides | 🔵 Comp. imaging seminar youtube (KAIST) | | |
| | Generative Adversarial Networks (GANs)
- Problem statement
- Loss function design
- Comparison with MLE
- Properties
- Results | - Lecture 25 slides | 🟢 NeurIPS tutorial by Goodman
🔵 Adversarial AutoEncoders paper (Toronto) | | |
| | Sampling methods:
- Inverse Transform Sampling (ITS),
- Accept / reject sampling,
- Importance sampling, MCMC | - Lecture 26 slides | 🟢 First 20 pages of this tutorial
🔵 MCMC for ML Tutorial | | |
| | Topics not covered:
- Transformers
- LLMs
- NeRFs
- RL | - Lecture 27 slides | 🔵 How LLMs work youtube (3Blue1Brown)
🟢 NeRF paper (Berkeley)
🔵 RL lectures 2 and 3 youtube (Berkeley) | | |
| | Wrap up and zoom out | - Lecture 28 slides | | | |
| | Buffer | | | | |
| | Buffer | | | | |
| | FINAL Exam: Monday, Dec 15, 7-10pm. | | | | |