Generative and Discriminative Approaches

for

Graphical Models

Instructor: Yasemin Altun

Office Hours: Friday 10am-Noon

This course provides an introduction to graphical models, exact and approximate inference and learning algorithms and applications. We will cover Bayesian belief networks and Markov Random Fields and focus on recent advances in learning with graphical models, such as loopy belief propagation, variational approximations, Conditional Random Fields, large margin methods and kernel methods for graphical models. Topics will be motivated by applications in computation linguistics, speech, bioinformations and computer vision.

The course is mainly geared towards graduate students, but may also be suitable for advanced undergraduate students with a solid mathematical background. There are no strictly enforced prerequisites, but familiarity with probability theory, calculus, and linear algebra is a plus.

There is no textbook for this class. Reading material (tutorials, book chapters, journal/conference papers) will be provided either electronically or as hard-copies.

Organization

- Standard lecture format
- Additional discussion sessions
- Final student presentations

Grading

Participation: mandatory

Presentation:

40% Final Projects: 60%

Jan 3

Admistrivia and Course Overview

Directed and Undirected graphical modelsTutorial on GM [Murphy01]

slides

Generative Approaches

Jan 17

No class - BIRS Workshop 07w5078 Mathematical

Programming in Data Mining and Machine Learning

Jan 22

Exact Inference

Tutorial on Junction Tree Algorithm [Barber03]

Chap3 of Graphical Models [Lauritzen96]

(copies available at my office)slides Jan 26

Parameter Learning

Tutorial on EM Algorithm [NeaHin98]

Generative vs Discriminative Methods [NgJor01]

Discriminative Training of HMMs [McCFrePer00]slides

Discriminative Methods

Feb 2

Conditional Random Fields, Perceptron learning for GM

Chap 2.1,3 Introduction to SVMs [Cristianini, Shaw-Taylor2000]

[LafMcCPer01],[LafZhuLiu04],[Collins02]

Feb 9

SVM based approaches for GM

Chap 6 Introduction to SVMs [Cristianini, Shaw-Taylor2000]

[TsoJoaHofAlt05],[TasGueKol04],[McAllester06]

Feb 16

Boosting based approaches for GM

Tutorial on Boosting [FreSch99]

[AltHofJoh03],[TorMurFre05],[Collins04]

Feb 23

Decompositional approaches

[RotYih05],[LeCHua05],[WesChaEliSchVap02]

Mar 2

Semi-supervised/Unsupervised Learning

[DauMar05], [AltMcCBel05], [BreSch06], [XuWilSouSch06]

Mar 9

Project Presentations

References

[McCFrePer00]

Authors: Andrew McCallum, Dayne Freitag, and Fernando Pereira

Title: Maximum entropy Markov models for information extraction and segmentation

Proceedings: International Conference on Machine Learning (ICML 2000), 2000.

Presenter: Yasemin

CRF and Perceptron approaches

[LafMcCPer01]

Title: Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data.

Proceedings: International Conference on Machine Learning (ICML-2001), 2001.

Presenter:Karthik

[LafZhuLiu04]

Title: Kernel Conditional Random Fields: Representation and Clique Selection

Proceedings: International Conference on Machine Learning (ICML-2004), 2004.

Presenter:Karthik

[Collins02]

Title: Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms.

Proceedings: EMNLP 2002.

Presenter: Ozgur

SVM approaches

Title: Large Margin Methods for Structured and Interdependent Output Variables,

Journal: Journal of Machine Learning Research (JMLR), 6(Sep):1453-1484, 2005.

Presenter: Vikas

[TasGueKol04]

Title: Max-Margin Markov Networks.

Proceedings: In Advances in Neural Information Processing Systems

Presenter: Irina

[McAllester06]

Authors: David McAllester

to appear in Predicting Structured Data,

edited by G. BakIr, T. Hofmann, B. Scholkopf, A. Smola, B. Taskar, and S. V. N. Vishwanathan. 2006

MIT Press.

Presenter: David

Boosting Approaches

Title: Discriminative Learning for Label Sequences via Boosting

Proceedings: Advances in Neural Information Processing Systems (NIPS*15), 2003.

Presenter: Ozgur

[TorMurFre05]

Authors: Antonio Torralba, Kevin Murphy and William Freeman

Title: Contextual Models for Object Detection using Boosted Random Fields

Proceedings: Advances in Neural Information Processing Systems (NIPS*17), 2005.

Presenter: Allie

[Collins04]

Title: Discriminative Reranking for Natural Language Parsing.

Presenter: Irina

Decompositional Approaches

[DauMar05]

Title: Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction

Proceedings:

Presenter: Yasemin

[RotYih05]

Title: Integer Linear Programming Inference for Conditional Random Fields.

Proceedings: International Conference on Machine Learning (ICML) (2005) pp. 737--744

Presenter: Karthik

[LeCHua05]

Authors: LeCun and Huang

Title: Loss Functions for Discriminative Training of Energy-Based Models

Proceedings: AI-Stats, 2005

Presenter: Allie

[WesChaEliSchVap02]

Title: Kernel Dependency Estimation

Proceedings: NIPS 2002.

Presenter: Vikas

Semi-Supervised/Unsupervised Learning

[AltMcCBel05]

Title: Maximum Margin Semi-Supervised Learning for Structured Variables

Proceedings: NIPS 2005.

[BreSch06]

Title: Semi-Supervised Learning for Structured Output Variables,

Proceedings: ICML 2006.

[XuWilSouSch06]

Title: Discriminative Unsupervised Learning of Structured Predictors

Proceedings: ICML 2006.