TTIC 31230: Fundamentals of Deep Learning

David McAllester

Winter 2018

  1. Introduction and Historical Notes
  2. Multi-Layer Perceptrons (MLPs) and Stochastic Gradient Descent (SGD)
  3. Feed-Forward Computation Graphs, Backpropagation, and the Educational Framework (EDF)
  4. Convolutional Neural Networks (CNNs)
  5. Invariant Theory
  6. Controling Gradients: Initialization, Batch Normalization, Resnets and Gated RNNs
  7. Language Modeling and Machine Translation
  8. First Order Stochastic Gradient Descent (SGD)
  9. Gradients as Dual Vectors, Hessian-Vector Products, and Information Geometry
  10. Regularization
  11. Interpretation
  12. Information Theory
  13. Fully Observed Graphical Models I: Exponential Softmax, Sufficient Statistics, and Belief Propagation
  14. Fully Observed Graphical Models II: Approximate SGD Algorithms
  15. Partially Observed Graphical Models: Expectation Maximization (EM), Expected Gradient (EG), and CTC
  16. Variational Autoencoders (VAEs)
  17. Rate-Distortion Autoencoders
  18. Generative Adversarial Networks (GANs)
  19. Reinforcement Learning (RL)
  20. AlphaZero
  21. The Quest for Artificial General Intelligence (AGI)