Schedule     Assignments/Quiz
TTIC 31190: Natural Language Processing (Autumn 2023)
Instructors: Freda Shi <freda [AT] ttic.edu> and Joe Zhou <jzhou [AT] ttic.edu>
TA: Jiamin Yang <jiaminy [AT] uchicago.edu>
Grader: Kangrui Wang <kangrui [AT] uchicago.edu>
Location: TTIC Room 530
Time: Monday/Wednesday 3:00-4:20pm Central Time

Description

This course will introduce fundamental concepts in natural language processing (NLP). NLP includes a range of research problems that involve computing with natural language. Some are user-facing applications, such as spam classification, question answering, summarization, and machine translation. Others serve supporting roles, such as part-of-speech tagging and syntactic parsing. Solutions to these challenges are derived from a combination of machine learning (especially deep learning) techniques, algorithms, and principles from linguistics.

Topics

  • Words: definition, tokenization, morphology, word senses
  • Lexical semantics: distributional semantics, word embeddings, word clustering
  • Text Classification: classifiers, linear models, features, naive Bayes, training linear classifiers via loss function optimization, loss functions, stochastic gradient descent
  • Neural networks: MLP, CNN, RNN and Transformers, finetuning
  • Language Modeling: n-gram models, smoothing, neural network--based language modeling
  • Sequence Labeling: part-of-speech tagging, named entity recognition, hidden Markov models, conditional random fields, dynamic programming, Viterbi
  • Syntax: weighted context-free grammars, dependency syntax, inference algorithms
  • Semantics: compositionality, semantic role labeling, frame semantics, lambda calculus, semantic parsing, grounded semantics
  • Pragmatics: phenomena, rational speech act model
  • Cross-lingual NLP: translation, decoding, lexicon induction, unsupervised translation
  • Large language models: challenges, prompting

Expected Outcomes

  • Understand key challenges of computing with natural language
  • Understand and apply solutions to standard NLP tasks
  • Be able to implement basic neural network architectures for core NLP tasks using deep learning toolkits
  • Be able to derive dynamic programming algorithms to perform inference in structured output spaces, and to analyze their computational properties
  • Understand common types of syntactic and semantic analysis, and how they are used in downstream applications
  • Recognize and characterize the errors made by NLP systems

Prerequisites

Basic knowledge of calculus, linear algebra, and probability; programming proficiency (no specific language required but Python is preferred); a machine learning course (e.g., TTIC 31020) is recommended but not required.

Schedule (Tentative)

All readings are optional. Click on the topic for the slides (in PDF format).

Week Date Topics Readings Instructor
1 Wed (09/27) Lec 1: Introduction SLP2: Chapter 1 Freda
2 Mon (10/02) Lec 2: Words SLP3: Chapter 2; SLP3: Chapter 23 Joe
2 Wed (10/04) Lec 3: Distributional Semantics SLP3: Chapter 6; Levy and Goldberg (2014) Joe
3 Mon (10/09) Lec 4: Text Classification I Bishop PRML Chapters 4 and 7.1 Freda
3 Wed (10/11) Lec 5: Text Classification II Freda
4 Mon (10/16) Lec 6: Neural Networks I Chris Olah's blog on LSTMs; Socher et al. (2011); Kim (2014) Freda
4 Wed (10/18) Lec 7: Neural Networks II & Sequence Labeling I Vaswani et al. (2017); SLP3: Chapter 8 Freda
5 Mon (10/23) Lec 8: Sequence Labeling II Freda
5 Wed (10/25) Lec 9: Language Modeling Joe
6 Mon (10/30) Lec 10: Sequence to Sequence I Joe
6 Wed (11/01) Lec 11: Syntax I Freda
7 Mon (11/06) Lec 12: Syntax II Freda
7 Wed (11/08) Lec 13: Semantics I Joe
8 Mon (11/13) Lec 14: Semantics II Joe
8 Wed (11/15) Lec 15: Pragmatics Freda
9 Thanksgiving break
10 Mon (11/27) Lec 16: LLM -- Pretraining and Finetuning Joe
10 Wed (11/29) Lec 17: LLM -- Prompting Freda

Assignments/Quiz