Topics in Artificial Intelligence
Generative and Discriminative Approaches
for
Graphical Models

CMCS 35900 Friday 3:30-6:00 TTI  Room 201 (Press Building)

Instructor: Yasemin Altun
Office Hours: Friday 10am-Noon




Short Course Description

This course provides an introduction to graphical models, exact and approximate inference and learning algorithms and applications.  We will cover Bayesian belief networks and Markov Random Fields and focus on  recent advances in learning with graphical models, such as loopy belief propagation, variational approximations, Conditional Random Fields, large margin methods and kernel methods for graphical models. Topics will be motivated by applications in computation linguistics, speech, bioinformations and computer vision.

The course is mainly geared towards graduate students, but may also be suitable for advanced undergraduate students with a solid mathematical background. There are no strictly enforced prerequisites, but familiarity with probability theory, calculus, and linear algebra is a plus.

Textbook & References

There is no textbook for this class. Reading material (tutorials, book chapters, journal/conference papers) will be provided either electronically or as hard-copies.

Course Organisation and Grading Policy

Organization

Grading


General Outline (subject to change)

Jan 3
 Admistrivia and Course Overview
Directed and Undirected graphical models 		Tutorial on GM [Murphy01]
 slides

Generative Approaches

Jan 17
 No class - BIRS Workshop 07w5078 Mathematical
Programming in Data Mining and Machine Learning

Jan 22
 Exact Inference
 Tutorial on Junction Tree Algorithm [Barber03]
Chap3 of Graphical Models [Lauritzen96]
(copies available at my office) 		slides
Jan 26
 Parameter Learning
 Tutorial on EM Algorithm [NeaHin98]
Generative vs Discriminative Methods [NgJor01]
Discriminative Training of HMMs [McCFrePer00] 		slides
 
Discriminative Methods

 

Feb 2
 Conditional Random Fields, Perceptron learning for GM
 Chap 2.1,3 Introduction to SVMs [Cristianini, Shaw-Taylor2000]
[LafMcCPer01],[LafZhuLiu04],[Collins02]
Feb 9
 SVM based approaches for GM
 Chap 6 Introduction to SVMs [Cristianini, Shaw-Taylor2000]
[TsoJoaHofAlt05],[TasGueKol04],[McAllester06]
Feb 16
 Boosting based approaches for GM
 Tutorial on Boosting [FreSch99]
[AltHofJoh03],[TorMurFre05],[Collins04]
Feb 23
 Decompositional approaches
 [RotYih05],[LeCHua05],[WesChaEliSchVap02]
Mar 2
 Semi-supervised/Unsupervised Learning
[DauMar05], [AltMcCBel05], [BreSch06], [XuWilSouSch06]
Mar 9
 Project Presentations



References

[McCFrePer00]
Authors: Andrew McCallum, Dayne Freitag, and Fernando Pereira
Title: Maximum entropy Markov models for information extraction and segmentation
Proceedings: International Conference on Machine Learning (ICML 2000),  2000.
Presenter: Yasemin

CRF and Perceptron approaches

[LafMcCPer01]
 Authors: John Lafferty, Andrew McCallum, Fernando Pereira
Title: Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data.
Proceedings: International Conference on Machine Learning (ICML-2001), 2001.
Presenter:Karthik

[LafZhuLiu04]
 Authors: John Lafferty, Xiaojin Zhu, Yan Liu
Title: Kernel Conditional Random Fields: Representation and Clique Selection
Proceedings: International Conference on Machine Learning (ICML-2004), 2004.
Presenter:Karthik

[Collins02]
Author: Michael Collins
Title: Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms.
Proceedings: EMNLP 2002.
Presenter: Ozgur

SVM approaches

[TsoJoaHofAlt05]
 Authors: I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun
Title: Large Margin Methods for Structured and Interdependent Output Variables,
Journal: Journal of Machine Learning Research (JMLR), 6(Sep):1453-1484, 2005.
Presenter: Vikas

[TasGueKol04]
 Authors: Ben Taskar, Carlos Guestrin and Daphne Koller
Title: Max-Margin Markov Networks.
Proceedings: In Advances in Neural Information Processing Systems 16 (NIPS 2003), 2004.
Presenter: Irina

[McAllester06]
Authors: David McAllester
Generalization Bounds and Consistency for Structured Labeling
to appear in Predicting Structured Data,
edited by G. BakIr, T. Hofmann, B. Scholkopf, A. Smola, B. Taskar, and S. V. N. Vishwanathan. 2006
MIT Press.
Presenter: David


Boosting Approaches

[AltHofJoh03]
Authors: Yasemin Altun, Thomas Hofmann & Mark Johnson
Title: Discriminative Learning for Label Sequences via Boosting
Proceedings: Advances in Neural Information Processing Systems (NIPS*15), 2003.
Presenter: Ozgur

[TorMurFre05]
Authors: Antonio Torralba, Kevin Murphy and William Freeman 
Title: Contextual Models for Object Detection using Boosted Random Fields
Proceedings: Advances in Neural Information Processing Systems (NIPS*17), 2005.
Presenter: Allie

[Collins04]
Authors: Michael Collins
Title: Discriminative Reranking for Natural Language Parsing.
Proceedings: International Conference on Machine Learning (ICML-2000), 2000.
Presenter: Irina

Decompositional Approaches

[DauMar05]
 Authors: Hal Daume and Daniel Marcu
Title: Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction
Proceedings: International Conference on Machine Learning (ICML), 2005.
Presenter: Yasemin

[RotYih05]
Authors: D. Roth and W. Yih
Title: Integer Linear Programming Inference for Conditional Random Fields.
Proceedings: International Conference on Machine Learning (ICML)  (2005) pp. 737--744 
Presenter: Karthik

[LeCHua05]
Authors: LeCun and Huang
Title: Loss Functions for Discriminative Training of Energy-Based Models
Proceedings: AI-Stats, 2005
Presenter: Allie

[WesChaEliSchVap02]
Authors: J. Weston, O. Chapelle, A. Elisseeff, B. Schoelkopf and V. Vapnik
Title: Kernel Dependency Estimation
Proceedings: NIPS 2002.
Presenter: Vikas

Semi-Supervised/Unsupervised Learning

[AltMcCBel05]
Authors: Yasemin Altun, David McAllester, Misha Belkin.
Title: Maximum Margin Semi-Supervised Learning for Structured Variables
Proceedings: NIPS 2005.

[BreSch06]
Authors: Ulf Brefeld, Tobias Scheffer.
Title: Semi-Supervised Learning for Structured Output Variables,
Proceedings: ICML 2006.

[XuWilSouSch06]
Authors: Linli Xu, Dana Wilkinson, Finnegan Southey, Dale Schuurmans
Title: Discriminative Unsupervised Learning of Structured Predictors
Proceedings: ICML 2006.