Mathematical Foundations of Computational Linguistics
MWF 10:30 Ry 276
This course will cover various mathematical concepts relevant to computational
linguistics. The course covers both statistical and logical concepts.
The statistical material will include hidden Markov models (HMMs); probabilistic
context free
grammars; n-gram language modeling; expectation-maximization (EM);
exponential models (maximum entropy methods); and various formal
generalization guarantees from machine learning theory. The logical
material will include Montague grammar, the simply type lambda calculus, and a
type-denotational approach to tense and aspect. The relationship
between statistics and logic is left as a major open problem of computational
linguistics.
This page contains lecture slides in postscript format. The last slide in each lecture
(other than the first) contains an assignment. For registered students the assignments
from Monday, Wednesday and Friday are due the following Wednesday. The first assignment
(the two problems from the second and third lecture) is due at Lecture Oct. 8.
- Lecture 1: What is Language For?
- Lecture 2: Probabilistic Context Free Grammars
- Lecture 3: Eisner and Sata's Cubic Parser for Bilexical PCFGs
- Lecture 4: HMMs and A* Parsing
- Lecture 5: Smoothing in n-gram Models and PCFGs
- Lecture 6: Good-Turing, Leave-One-Out Estimation, and Language Modeling
- Lecture 7: EM for Gaussian Mixtures, General EM
- Lecture 8: A guest lecture by Gina Levow
- Lecture 9: EM for PCFGs (The Inside-Outside Algorithm)
- Lecture 10: Proof of the Main EM Theorem, MAP EM, Structural EM, Leave-One-Out EM
- Lecture 11: Least Squares Regression
- Lecture 12: Logistic Regression (also known as log-linear models, maxent models, Gibbs models, and the exponential family)
- Lecture 13: Optimization in Logistic Regression
- Lecture 14: Gaussian Priors in Logistic Regression
- Lecture 15: Some PAC-Bayesian Theorems
- Lecture 16:
- Lecture 17: Concentration Inequalities
- Lecture 18: SVMs: Margin Bounds
- Lecture 19: SVMs: Kernels
- Lecture 20: SVMs: Convex Duality
- Lecture 21: Boosting
- Lecture 22: What is NL Semantics? Why Study Logic? Truth vs. Proof. Godel's Theorem. Soundness and Completeness.
- Lecture 23: Inference Rules as Algorithms
- Lecture 24: Natural Language as Formal Notation --- Chapters 2 and 3 of "Type-Logical Semantics" (TTS) by Robert Carpenter
- Lecture 25: The Lambek Calculus --- Capters 4 and 5 of TTS
- Lecture 26: Coordination --- Chapter 6 of TTS
- Lecture 27: Quantification and Plurals --- Chapters 7 and 8 of TTS
- Lecture 28: Possible world Semantics, Propositional Attitudes, Tense and Aspect --- Chapters 10, 11, and 12 of TTS.
- Lecture 29: Statistics in Semantics.