20114221: Introduction to Statistical Machine Learning
Winter 2012
Instructor: Greg Shakhnarovich, greg@ttic.edu
Tuesday, 09001100, Ziskind, Faculty Room (1st floor) note change in location
Thursday, 09001100, FGS, Rm C
First lecture: Thursday Dec 29, 2011
Scope
This courses covers supervised learning, primarily classification and regreession. Some topics in unsupervised learning that are central to supervised tasks, such as density estimation, are covered as well. The emphasis is on the fundamental principles in design of statistical methods, such as biasvariance tradeoff and overfitting, and on computational issues in applying these methods to real data sets. In the process of eploring these fundamental topics the course covers in detail modern machine learning tools: generative and discriminative regression and classification models, support vector machines, decision trees, neural networks, ensemble methods etc.
There are no formal prerequisites, but working knowledge of linear algebra and basic probability are necessary.
Books
There is no official textbook for the class, however the following books will be very helpful:
 C. M. Bishop Pattern Recognition
and Machine Learning 
 T. Hastie, R. Tibshirani and
J. H. Friedman
The Elements of Statistical Learning
(the most recent edition is available online)

Syllabus, lecture slides, homework assignments, etc.
Note: All handouts and lecture slides are in PDF format. Any
information posted for future dates should be treated as
tentative.
Thursday Dec 29
 General introduction & administrivia
 Loss and risk
 Linear regression; least squares
 Decomposition of squared error
 Lecture material: slides (with animations),
slides in handout format (no animations)
Tuesday Jan 3
 Statistical model for regression; maximum likelihood
 Elementary estimation theory
 Bias/variance dilemma
 Shrinkage regularization
 Lecture material: slides (with animations),
slides in handout format (no animations)
 Notes on bias/variance decomposition in regression
Thursday Jan 5
 Regularization
 Introduction to classification, elementary decision theory
 Logistic regression
 Lecture material: slides (with animations),
slides in handout format (no animations)
Thursday Jan 5
Tuesday Jan 10
 Learning for logistic regression
 ML vs MAP and regularization
 Stepwise regression and boosting
 Lecture material: slides (with animations),
slides in handout format (no animations)
Thursday Jan 12
Tuesday Jan 17
Tuesday Jan 17
Thursday Jan 19
 Kernels
 Intro to generative models
 Density estimation and discriminative analysis
 Lecture material: slides (with animations),
slides in handout format (no animations)
Tuesday Jan 24
Thursday Jan 26
 KLdivergence and learning
 Hierarchical models: mixtures of experts, neural networks
 Lecture material: slides (with animations),
slides in handout format (no animations)
Monday January 30
Tuesday January 31
Thursday February 2
Final exam: Tuesday February 21, 9:0012:00
Feinberg, room TBA