M/W 4:00pm to 5:20pm, 4025 CIF | Please refresh the page twice to make sure you are seeing the latest updates
First day information
Homework and MPs
Tentative course calendar:
🟢 = must read. 🔵 = additional reading.
<aside>
All slides + notes are in this web folder
</aside>
.
| Lecture |
Topics |
Lecture material |
Reading |
Homework |
MP |
| 1. Wed |
|
|
|
|
|
| Jan 21 |
Course logistics + big picture |
|
|
|
|
- Generative models overview | - lec1_overview.pptx
- lec1_video (poor audio in 2nd half. Will fix next lecture onwards) | 🔵 Gen AI overview youtube (Stanford)
🔵 History of diffusion youtube (Yang Song) | | |
| 2. Mon
Jan 26 | Review1: Probability
- Bayes, MLE, Multivariate, …
- Conditional independence, Markov
- Expectation Maximization (EM) | - lec2_review1.pptx
- lec2_EM_notes.pdf
- lec2_video | 🔵 Probability review (online book) | ✅ HW1
(Foundations) | |
| 3. Wed
Jan 28 | Review2: Probability, Linear Algebra, DL
- EM continued
| - lec3_review2.pptx
- lec3_video | 🔵 Gilbert Strang’s lectures on null space, rank, etc.
🔵 EM tutorial by D. Lin PDF
🔵 SVD video here | | |
| 4. Mon
Feb 2 | Review3: Linear Algebra, Covariance matrices
- PCA, SVD, …
| - lec4_Eigen_Cov_PCA.pdf
- lec4_video | | | 🖥️ MP1
(VAE)
|
| 5. Wed
Feb 4 | Variational Inference, VAE, visualization
- AutoEncoder (AE)
- Semantics
- VAE (moon eclipse), ELBO
- Entropy push pull
- Prior hole
- Posterior collapse | - lec5_AE_VAE.pptx
- lec5_video | 🟢 Intro. to VAEs (Kingma): chap. 1,2
🟢 Prior hole paper (UIUC) PDF
🔵 IMUV paper (UIUC)
🔵 VQ-VAE paper (NeuRIPS)
🔵 Learning latent prior paper (UCLA) PDF | | |
| 6. Mon
Feb 9 | Diffusion Models 1
- Hierarchical VAEs
- Variational diffusion models (VDM) | | 🟢 Luo’s tutorial, chapters 1-2 | 🚫 HW1 | |
| 7. Wed
Feb 11 | Diffusion Models 2
- Gaussian encoders
- Updated ELBO | | 🟢 Luo’s tutorial, chapters 3-4
🔵 Stanley Chan tutorial | ‣ Quiz 1
✅ HW2
(VAE) | |
| 8. Mon
Feb 16 | Diffusion Models 3
- 3 different interpretations
- Image prediction
- Noise prediction
- Tweedie’s formula → score matching | | | | |
| 9. Wed
Feb 18 | Diffusion Models 4
- Score as a MMSE estimator
- Langevin dynamics
- Noise annealing
| | 🟢 Grad. of data distribution (Song) blog
🟢 Score Matching (Helsinki) PDF
🔵 Elad’s lecture 3 video youtube
🔵 Learning from thermodynamics PDF
| | 🚫 MP1 |
| 10. Mon
Feb 23 | Diffusion Models 5
- Zoom out / big picture
- Annealed Langevin dynamics
- Connect the dots: score, denoising, DDPM
- Time varying force field = differential equations | | 🟢 DDPM paper (Berkeley) PDF
🔵 Improved DDPM paper (OpenAI) PDF
🔵 DDIM paper (Stanford) paper PDF
🔵 Diffusion language models video
| | 🖥️ MP2
(DDPM) |
| 11. Wed
Feb 25 | Guidance
- Classifier based guidance
- Derivative of logits
- Classifier free guidance
- Scaling factor $s$ | | 🟢 GLIDE T2I models (OpenAI) PDF
🔵 Classifier free guidance (OpenAI) PDF
| 🚫 HW2 | |
| 12. Mon
Mar 2 | CLIP and Text-to-Image (T2I) Models
- T2I problem statement
- CLIP
- Latent Diffusion Model (LDM) | | 🟢 CLIP: Visual from language PDF
🔵 Latent Diffusion paper (Heidelberg) | ✅ HW3
(Diffusion) | |
| 13. Wed
Mar 4 | Inverse problems + Posterior sampling
- IP problem statement
- Approximate conditional score
- Diffusion based IP
- DPS | | 🟢 Diffusion posterior sampling (DPS) PDF
| ‣ Quiz 2
| |
| 14. Mon
Mar 9 | DPS and Pi-GDM
- Approximating likelihood with single sample
- Approximating likelihood with Gaussian
- Colorization
- Deblurring
- Inpainting
| | 🟢 RePaint and inpaint paper (ETH)
🟢 Pi-GDM (NVidia) paper | | |
| 15. Wed
Mar 11 | Applications of Inverse Problems
- Source (speech) separation
- Motion tracking and mapping
- Inverse path planning | | 🟢 ArrayDPS paper (UIUC)
🟢 Image super resolution paper (Google)
🔵 Pose estimation paper (UIUC)
🔵 Inverse path planning paper (UIUC) | | |
| | SPRING BREAK 🎉 | | | | |
| 16. Mon
Mar 23 | Buffer class | | | 🚫 HW3 | 🚫 MP2 |
| 17. Wed
Mar 25 | Mid-term review | | | | |
| 18. Mon
Mar 30 | Introduction to differential equations
- DE problem statement
- GD as ODE, Reverse ODE
- SDE, Brownian / Wiener process
- Reverse SDE | | 🟢 Chan’s tutorial (Purdue), Chapter 4
| | |
| 19. Wed
Apr 1 | ‣ MIDTERM exam (in class) | | | ‣ Midterm | |
| 20. Mon
Apr 6 | SDE, ODE, and Flows 1
- Diffusion and ODE/SDE
- Reverse ODE/SDE
- Fokker Planck and Kolmogorov
- Solvers: Euler and Runge-Kutta | | 🟢 Flow and Diffusion tutorial (MIT), chap. 1 | | 🖥️ MP3
(Inverse) |
| 21. Wed
Apr 8 | Flows 2
- Conditional and marginal probability path
- Designing target vector field
- Continuity equation
- Zoom out → visualize
- Score functions | | 🟢 Flow and Diffusion tutorial (MIT), chap. 2
🔵 Divergence video (Khan Academy)
| ✅ HW4
(Flows) | 🖥️ MP4
(Flows, for 4 credit students) |
| 22. Mon
Apr 13 | Flows 3: ODE to SDE to score matching
- Conditional flow matching
- Conditional to marginal
- CFM is surrogate loss for FM
- Score matching extension
- Special case: Gaussian
- For Gaussian, score comes free | | 🟢 Flow tutorial (MIT), chapters 3 and 4
🔵 Lilian’s blog
🔵 Flows_foundations_slides.pdf (UIUC)
🟢 Flow matching paper (Meta)
🔵 Rectified flows paper (UT Austin) | | |
| 23. Wed
Apr 15 | Surgery of diffusion models
- Inpainting and outpainting
- Generalization and reproducability
- Low dimensional sub-spaces in diffusion
- Precise editing of diffusion models | | 🟢 RePaint paper PDF
🟢 LOCO paper PDF
🟢 ICML tutorial video by Qing Qu
| | |
| 24. Mon
Apr 20 | Low Rank and Editing Diffusion Models
- Problem statement
- Jacobian of Denoiser
- LOCO Edit | | 🟢 LOCO Edit paper (Michigan)
| | |
| 25. Wed
Apr 22
| NeRFs (vision and beyond)
- Radiance fields
- Rendering function and inverse problem
- Wireless NeRF | | 🟢 NeRF paper (Berkeley)
🟢 Wireless NeRF paper (UIUC)
🟢 NeRFs + Diffusion paper (Berkeley) | | 🚫 MP3 |
| 26. Mon
Apr 27 | Recent papers on controlled diffusion
- DreamFusion
- Progressive Guidance
- MultiDiffusion
- RB Modulation
- TRAP
| | 🟢 MultiDiffusion paper (Weizmann)
🟢 RB_Modulation paper (UT Austin)
🟢 TweedieMix paper (KAIST)
🟢 Composing diffusion models paper (MIT)
🟢 Diffusion seeds blog (Reddit)
🔵 Progressive guidance paper (Sydney) | 🚫 HW4 | |
| 27. Wed
Apr 29 | Last class:
- Composition of Diffusion
- Seed tutorial
- LLM based material generation
- Parallel diffusion
- SoTTA (test time adaptation)
- Uncovered topics
- Philosophical advise (ignore 🙂)
| | 🔵 How LLMs work youtube (3Blue1Brown)
🔵 RL lectures 2 and 3 youtube (Berkeley) | | |
| 28. Mon
May 4 | | | | | |
| 29. Wed
May 6 | | | | | 🚫 MP4 |
| Tentative:
May 13 | ‣ FINAL Exam: | | | ‣ Finals | |
<aside>
❄️ Zoom lecture today: here is the link
Password: 4 digits, this year.
</aside>