20114221: Introduction to Statistical Machine Learning
Winter 2012
Instructor: Greg Shakhnarovich, greg@ttic.edu
Tuesday, 0900-1100, Ziskind, Faculty Room (1st floor) note change in location
Thursday, 0900-1100, FGS, Rm C
First lecture: Thursday Dec 29, 2011
Scope
This courses covers supervised learning, primarily classification and regreession. Some topics in unsupervised learning that are central to supervised tasks, such as density estimation, are covered as well. The emphasis is on the fundamental principles in design of statistical methods, such as bias-variance tradeoff and overfitting, and on computational issues in applying these methods to real data sets. In the process of eploring these fundamental topics the course covers in detail modern machine learning tools: generative and discriminative regression and classification models, support vector machines, decision trees, neural networks, ensemble methods etc.
There are no formal prerequisites, but working knowledge of linear algebra and basic probability are necessary.
Books
There is no official textbook for the class, however the following books will be very helpful:
|
| C. M. Bishop
Pattern Recognition
and Machine Learning |
| T. Hastie, R. Tibshirani and
J. H. Friedman
The Elements of Statistical Learning
(the most recent edition is available online)
|
Syllabus, lecture slides, homework assignments, etc.
Note: All handouts and lecture slides are in PDF format. Any
information posted for future dates should be treated as
tentative.
Thursday Dec 29
- General introduction & administrivia
- Loss and risk
- Linear regression; least squares
- Decomposition of squared error
- Lecture material: slides (with animations),
slides in handout format (no animations)
Tuesday Jan 3
- Statistical model for regression; maximum likelihood
- Elementary estimation theory
- Bias/variance dilemma
- Shrinkage regularization
- Lecture material: slides (with animations),
slides in handout format (no animations)
- Notes on bias/variance decomposition in regression
Thursday Jan 5
- Regularization
- Introduction to classification, elementary decision theory
- Logistic regression
- Lecture material: slides (with animations),
slides in handout format (no animations)
Thursday Jan 5
Tuesday Jan 10
- Learning for logistic regression
- ML vs MAP and regularization
- Stepwise regression and boosting
- Lecture material: slides (with animations),
slides in handout format (no animations)
Thursday Jan 12
Tuesday Jan 17
Tuesday Jan 17
Thursday Jan 19
- Kernels
- Intro to generative models
- Density estimation and discriminative analysis
- Lecture material: slides (with animations),
slides in handout format (no animations)
Tuesday Jan 24
Thursday Jan 26
- KL-divergence and learning
- Hierarchical models: mixtures of experts, neural networks
- Lecture material: slides (with animations),
slides in handout format (no animations)
Monday January 30
Tuesday January 31
Thursday February 2
Final exam: Tuesday February 21, 9:00-12:00
Feinberg, room TBA