CMSC 35246 Deep Learning
Spring 2017

In many real world Machine Learning tasks, in particular those with perceptual input, such as Vision and Speech, the mapping from raw data to the output is often a complicated function with many factors of variation. Prior to 2010, to achieve decent performance on such tasks, significant effort had to be put to engineer hand crafted features. Deep Learning algorithms aim to learn feature hierarchies with features at higher levels in the hierarchy formed by the composition of lower level features. This automatic learning of features has been demonstrated to uncover underlying structure in the data leading to the state-of-the-art results in tasks in Vision, Speech and in other domains as well.

This course aims to cover the basics of Deep Learning and some of the underlying theory with a particular focus on supervised Deep Learning, with a good coverage of unsupervised methods.

Instructors: Shubhendu Trivedi and Risi Kondor

Time: Mondays and Wednesdays, 3.00pm-4.20pm, Ryerson 277

Office hours:

Prerequisites

  1. Graduate Machine Learning courses at the level of STAT 37710/CMSC 35400 or TTIC 31020 (STAT 27725/CMSC 25400 should be OK).
  2. Familiarity with basic Probability Theory, Linear Algebra, Calculus
  3. Programming proficiency in Python (although you should be fine if you have extensive experience in some other high level language)

Syllabus

Course Work

Grading

This course is experimental, so the course plan and weighing is subject to revision:

Maximum
Assignments 4. Total contribution to grade 40%
In-Class Midterm Total contribution to grade 20-25%
Project Total contribution to grade: 25-30%
Quizzes + Participation 5. Total contribution to grade: 10%

Books and Resources

Deep learning is a rapidly evolving field and so we will freely move from using recent research papers to materials from older books etc. However, there are three recent books that ground a lot of the fundamentals. In particular, the book by Goodfellow, Bengio and Courville is highly recommended, not only for the quality of its discussions, but also given that it has widest coverage of topics. It is also the most up-to-date and will be followed in most of the lectures.

Textbooks

Deep Learning By Ian Goodfellow and Yoshua Bengio and Aaron Courville MIT Press, 2016
Neural Networks and Deep Learning By Michael Nielsen Online book, 2016
Learning Deep Architectures for AI (slightly dated) By Yoshua Bengio NOW Publishers, 2009

Tools

We recommend the use of PyTorch or TensorFlow for projects

Logistics, Submissions etc.

Please ensure the course is added to your canvas, as we would be using it for announcements, assignment submissions etc.

Schedule

Week Date Topics Lecture notes/Slides Additional Resources, Suggested Readings, etc.
1 March 27
  • Introduction to Deep Learning
  • Course logistics
1 March 29
  • Machine Learning Review
2 April 3
  • Feedforward Deep Networks
  • Backpropagation Start
  • Assignment 1 Out (on Canvas)
2 April 5
  • Backpropagation
  • Quiz 1
  • Suggested Reading: Chapter 6 of Goodfellow, Bengio and Courville
  • Suggested Reading: Chapter 2 of Nielsen
  • Suggested Reading: Calculus on Computational Graphs, Chris Olah
  • Suggested Reading: Andrej Karpathy's notes (linked below in "Additional Linkage")
3 April 10
  • Regularization Methods
  • slides
  • slides(flat)
3 April 12
  • Optimization Methods --
  • SGD, Momentum method
  • Adaptive first order methods
  • Batch Normalization
  • Initialization Strategies
  • Polyak Averaging
  • Assignment 1 Due (13 April; 23:59)
4 April 17
  • Convolutional Neural Networks
  • Assignment 2 Out (April 16)
  • Quiz 2
4 April 19
  • Visualizing Convolutional Networks
  • Variants (Locally Connected Networks, Tiled CNNs, Dilated Convolutions)
  • Motivations: Neuroscientific
  • Motivations: Efficiency, Equivariance, Invariance, parameter tying
  • Project Proposals Due at midnight
5 April 24
  • Ultra Deep Architectures
  • Residual Networks
  • ResNet in ResNet
  • Highway Networks
  • Fractal Networks
  • Some Analysis of Residual Learning
  • DenseNets
  • Similarity Learning and Siamese Networks
  • Assignment 2 Due (April 27; 23:59)
5 April 26
  • CNNs on Graphs Introduction
  • CNNs on Non-Euclidean Domains
  • Locally Connected Networks
  • Spectral Networks
  • Graph Embedding: Basic
  • Quiz 3
6 May 01
  • Sequence Learning with NNs
  • Recurrent Neural Networks
  • Design Patterns for RNNs
  • Intro to long term dependencies and gated architectures
6 May 03
  • Recap of RNNs
  • Challenges with Long Term Dependencies
  • Reservoir Computing (basic idea)
  • Long Short Term Memory and GRU
  • Attention Models: Introduction using applications in Machine Translation and Caption Generation
7 May 08
  • Mid-term
7 May 10
  • Neural Networks with External Memory
  • Memory Networks
  • Neural Turing Machines
  • Differentiable Neural Computers
  • slides
  • slides(flat)
  • (slides on canvas for now)
8 May 15
  • Mid-Term Solutions Review
  • Introduction to Deep Unsupervised Learning
  • Linear Projections and Linear Autoencoders
  • Sparse Coding
  • Autoencoders
  • Assignment 3 Due (16 May 23:59)
  • slides
  • slides(flat)
  • (slides on canvas for now)
8 May 17
  • Recap of PCA, ICA, Sparse Coding
  • Ditributed versus Localist Representations
  • Autoencoders wrap-up (sans training)
  • Quiz 4
  • slides
  • slides(flat)
  • (slides on canvas for now)
9 May 22
  • Linear Factor Models and Probabilistic PCA as motivation for Deep Factor Models
  • Energy Based Methods
  • Sigmoid Belief Networks and Deep Belief Networks
  • Boltzmann Machines and Restricted Boltzmann Machines set-up
9 May 24
  • Linear Factor Models, SBNs, DBNs, RBMs recap
  • Approximate Learning in Restricted Boltzmann Machines
  • Contrastive Divergence
  • Boltzmann Machines
  • Implicit density models: Intro
  • slides
  • slides(flat)
  • (slides on canvas for now)
10 May 29
  • Memorial day holiday: No Class
10 May 31
  • Implicit density models
  • Generative adversarial networks
  • Wasserstein GANs
  • Bonus topic: AlphaGo
  • Quiz 5
  • slides
  • slides(flat)
  • (slides on canvas for now)
11-12 June 6-10
  • Project presentations and discussions
  • Final project report and code due 10 June 23:59

Additional Linkage

Related Courses

Notes, Surveys and Pedagogical Material