Topics in Artificial Intelligence
Generative and Discriminative Approaches
Graphical Models

CMCS 35900 Friday 3:30-6:00 TTI  Room 201 (Press Building)

Instructor: Yasemin Altun
Office Hours: Friday 10am-Noon

Short Course Description

This course provides an introduction to graphical models, exact and approximate inference and learning algorithms and applications.  We will cover Bayesian belief networks and Markov Random Fields and focus on  recent advances in learning with graphical models, such as loopy belief propagation, variational approximations, Conditional Random Fields, large margin methods and kernel methods for graphical models. Topics will be motivated by applications in computation linguistics, speech, bioinformations and computer vision.

The course is mainly geared towards graduate students, but may also be suitable for advanced undergraduate students with a solid mathematical background. There are no strictly enforced prerequisites, but familiarity with probability theory, calculus, and linear algebra is a plus.

Textbook & References

There is no textbook for this class. Reading material (tutorials, book chapters, journal/conference papers) will be provided either electronically or as hard-copies.

Course Organisation and Grading Policy



General Outline (subject to change)

Jan 3
Admistrivia and Course Overview
Directed and Undirected graphical models
Tutorial on GM [Murphy01]

Generative Approaches

Jan 17
No class - BIRS Workshop 07w5078 Mathematical
Programming in Data Mining and Machine Learning

Jan 22
Exact Inference
Tutorial on Junction Tree Algorithm [Barber03]
Chap3 of Graphical Models [Lauritzen96]
(copies available at my office)
Jan 26
Parameter Learning
Tutorial on EM Algorithm [NeaHin98]
Generative vs Discriminative Methods [NgJor01]
Discriminative Training of HMMs [McCFrePer00]
Discriminative Methods


Feb 2
Conditional Random Fields, Perceptron learning for GM
Chap 2.1,3 Introduction to SVMs [Cristianini, Shaw-Taylor2000]

Feb 9
SVM based approaches for GM
Chap 6 Introduction to SVMs [Cristianini, Shaw-Taylor2000]

Feb 16
Boosting based approaches for GM
Tutorial on Boosting [FreSch99]

Feb 23
Decompositional approaches
Mar 2
Semi-supervised/Unsupervised Learning
[DauMar05], [AltMcCBel05], [BreSch06], [XuWilSouSch06]

Mar 9
Project Presentations


Authors: Andrew McCallum, Dayne Freitag, and Fernando Pereira
Title: Maximum entropy Markov models for information extraction and segmentation
Proceedings: International Conference on Machine Learning (ICML 2000),  2000.
Presenter: Yasemin

CRF and Perceptron approaches

Authors: John Lafferty, Andrew McCallum, Fernando Pereira
Title: Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data.
Proceedings: International Conference on Machine Learning (ICML-2001), 2001.

Authors: John Lafferty, Xiaojin Zhu, Yan Liu
Title: Kernel Conditional Random Fields: Representation and Clique Selection
Proceedings: International Conference on Machine Learning (ICML-2004), 2004.

Author: Michael Collins
Title: Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms.
Proceedings: EMNLP 2002.
Presenter: Ozgur

SVM approaches

Authors: I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun
Large Margin Methods for Structured and Interdependent Output Variables,
Journal: Journal of Machine Learning Research (JMLR),
6(Sep):1453-1484, 2005.
Presenter: Vikas

Authors: Ben Taskar, Carlos Guestrin and Daphne Koller
Title: Max-Margin Markov Networks.
Proceedings: In Advances in Neural Information Processing Systems 16 (NIPS 2003), 2004.
Presenter: Irina

Authors: David McAllester
Generalization Bounds and Consistency for Structured Labeling
to appear in Predicting Structured Data,
edited by G. BakIr, T. Hofmann, B. Scholkopf, A. Smola, B. Taskar, and S. V. N. Vishwanathan. 2006
MIT Press.
Presenter: David

Boosting Approaches

Authors: Yasemin Altun, Thomas Hofmann & Mark Johnson
Title: Discriminative Learning for Label Sequences via Boosting
Proceedings: Advances in Neural Information Processing Systems (NIPS*15), 2003.
Presenter: Ozgur

Authors: Antonio Torralba, Kevin Murphy and William Freeman 
Title: Contextual Models for Object Detection using Boosted Random Fields
Proceedings: Advances in Neural Information Processing Systems (NIPS*17), 2005.
Presenter: Allie

Authors: Michael Collins
Title: Discriminative Reranking for Natural Language Parsing.
Proceedings: International Conference on Machine Learning (ICML-2000), 2000.
Presenter: Irina

Decompositional Approaches

Authors: Hal Daume and Daniel Marcu
Title: Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction
Proceedings: International Conference on Machine Learning (ICML), 2005.
Presenter: Yasemin

Authors: D. Roth and W. Yih
Title: Integer Linear Programming Inference for Conditional Random Fields.
Proceedings: International Conference on Machine Learning (ICML)  (2005) pp. 737--744 

Presenter: Karthik

Authors: LeCun and Huang
Title: Loss Functions for Discriminative Training of Energy-Based Models
Proceedings: AI-Stats, 2005
Presenter: Allie

Authors: J. Weston, O. Chapelle, A. Elisseeff, B. Schoelkopf and V. Vapnik
Title: Kernel Dependency Estimation
Proceedings: NIPS 2002.

Presenter: Vikas

Semi-Supervised/Unsupervised Learning

Authors: Yasemin Altun, David McAllester, Misha Belkin.
Title: Maximum Margin Semi-Supervised Learning for Structured Variables
Proceedings: NIPS 2005.

Authors: Ulf Brefeld, Tobias Scheffer.
Title: Semi-Supervised Learning for Structured Output Variables,
Proceedings: ICML 2006.

Authors: Linli Xu, Dana Wilkinson, Finnegan Southey, Dale Schuurmans
Title: Discriminative Unsupervised Learning of Structured Predictors
Proceedings: ICML 2006.