Information and Coding Theory - Autumn 2022


TTIC 31200/CMSC 37220

T Th 9:30-10:50 (TTIC 530)

Discussion: T 1-2 pm (TTIC 530)

Office hours: F 2-3 pm (TTIC 505 and Zoom)

Instructors: Madhur Tulsiani and Omar Montasser

TAs: Max Ovsiankin and Shashank Srivastava



 

This course is meant to serve as an introduction to some basic concepts in information theory and error-correcting codes, and some of their applications in computer science and statistics. We plan to cover the following topics:

  • Introduction to entropy and source coding. Some applications of entropy to counting problems.
  • Mutual information and KL-divergence. Method of types and hypothesis testing. Minimax rate bounds.
  • I-projections, maximum entropy, exponential families and applications.
  • Introduction to error-correcting codes. Unique and list decoding of Reed-Solomon and Reed-Muller codes.
  • Applications of information theory to problems in theoretical computer science.

The course will have 4-5 homeworks (60 percent) and a final (40 percent).


There is no textbook for this course, since no single book covers all the topics discussed in the course. A very useful reference is ``Elements of Information Theory'' by T. M. Cover and J. A. Thomas. We will also post links to other materials in the resources section below.


Lectures will be streamed via Panopto, office hours can also be joined via Zoom. Please see the Canvas page of the course for links.



Homeworks and Announcements



Lecture Plan and Notes

  • 9/27: Reminder on convexity. Entropy of a random variable.
    [Notes]
  • 9/29: Prefix-free codes and Kraft's inequality. Conditional and joint entropy.
    [Notes]
  • 10/04: Fundamental source coding theorem. Subadditivity of entropy and combinatorial applications. Shearer's lemma.
    [Notes]
  • 10/06: Proof and applications of Shearer's lemma. Mutual Information.
    [Notes]
  • 10/11: Data-processing inequality and Fano's inequality. KL-divergence and some properties.
    [Notes]
  • 10/13: Pinsker's inequality and its application to lower bounds for distinguishing biased coins.
    [Notes]
  • 10/18: Differential entropy and KL-divergence.
    [Notes]
  • 10/20: Gaussian computations.
    [Notes]
  • 10/25: Method of types and Sanov's theorem.
    [Notes]
  • 10/27: Hypothesis testing.
    [Notes]
  • 11/1: Minimax risk bounds.
    [Notes]
  • 11/3: Minimax risk bounds (continued).
    [Notes]
  • 11/8: Sparse mean estimation. I-projections.
    [Notes]
  • 11/10: Linear families, I-projections, and maximum entropy distributons. Matrix scaling.
    [Notes]
  • 11/15: Channel coding and channel capacity.
    [Notes]
  • 11/17: Random codes for the Binary Symmetric Channel.
    [Notes]
  • 11/29: Linear Codes. Explicit capacity-achieving codes via entropy polarization.
    [Notes]
  • 12/1: Error-correction in the Hamming model. Reed-Solomon codes.
    [Notes]


Resources