Date |
Topic |
Day Schedule |
Homeworks |
8/16 |
Introduction and background |
Introduction and overview |
|
8/17 |
|
Data and dimensionality |
|
8/22 |
Data representation space and structures |
Dimension reduction, metric learning, PCA, MDS |
|
8/23 |
|
Deep learning workflow and useful programming tools |
|
8/24 |
|
Structures in data spaces, manifolds, subspaces, sparse coding |
|
8/29 |
Visual representations |
Pixels, 3D points, and cameras |
|
8/30 |
|
Image operations |
|
8/31 |
|
Semantics |
HW1 assignment |
9/5 |
|
Videos |
|
9/6 |
|
Project and research discussion, how to do research? |
|
9/7 |
|
Image subspaces and manipulations |
|
9/12 |
Language representations |
Representing words and sentences |
|
9/13 |
|
Project and research discussion, how to read a paper? |
|
9/14 |
|
Language model pretraining |
|
9/19 |
|
NLP tasks |
|
9/20 |
|
Project and research discussion, how to write a paper? |
|
9/21 |
|
Zero-shot and in-context learning |
HW1 due |
9/26 |
Audio representations |
Representing sound |
|
9/27 |
|
Project and research discussion |
|
9/28 |
|
Audio generation and editing |
|
10/3 |
Graphs |
Graphs and neural networks |
|
10/4 |
|
Project and research discussion |
|
10/5 |
|
GNN applications |
HW2 assignment |
10/10 |
Multi-modal representations |
Midterm |
Midterm |
10/11 |
|
Project and research discussion |
|
10/12 |
|
Overview of multi-modal learning: Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions |
|
10/17 |
|
Multimodal representation alignment: reference |
|
10/18 |
|
Project and research discussion |
|
10/19 |
|
Multimodal reasoning: reference |
|
10/24 |
Advanced Topics - Implicit representation |
Implicit Neural Representations with Periodic Activation Functions |
|
10/25 |
|
Project and research discussion |
|
10/26 |
|
Neural Ordinary Differential Equations |
|
10/31 |
Advanced Topics - Meta-learning and multi-domain learning |
Meta-Learning in Neural Networks: A Survey |
|
11/1 |
|
Project and research discussion |
|
11/2 |
|
Domain Generalization: A Survey |
|
11/7 |
Advanced Topics - Adapter approach |
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention |
|
11/8 |
|
Project and research discussion |
|
11/9 |
|
LoRA: Low-Rank Adaptation of Large Language Models |
|
11/14 |
Advanced Topics - Beyond perception |
PaLM-E: An Embodied Multimodal Language Model |
|
11/15 |
|
Project and research discussion |
|
11/16 |
|
Representation Learning for Autonomous Robots |
HW2 due (11/19) |
11/21 |
Class review and summary |
Class wrap-up and discussion |
|
|
--Thanksgiving-- |
|
|
|
--Thanksgiving-- |
|
|
11/28 |
Project |
Project presentation and discussion |
|
11/29 |
|
Project and research discussion |
|
11/30 |
|
Project and research discussion |
|
12/5 |
No class |
No class (Friday Schedule) |
|