EESC 6341: Information Theory

Information theory has been responsible for the key ideas behind cell phones, high-definition cable and sattelite television, streaming of music and video, cloud storage, as well as enabling error-free operation of high-capacity storage such as hard disks and semiconductor memory. The operation of all these devices started from ideas developed in infomration theory.

This course is designed for the beginning graduate student in the areas of communications or signal processing. It will illuminate the underlying reasons for many of the topics learned in other courses, and prepare the student for high-quality research. Grading is based on homework/quizzes, midterm(s), and a course project.

Instructor Aria Nosratinia,
ECSN 4.504, Tel: 972-883-2894
Time Mon-Wed. 1:00-2:15pm
Place Virtual/Remote
Textbooks Cover and Thomas, Elements of Information Theory, 2nd Edition, John Wiley (required)
El Gamal and Kim, Network Information Theory, Cambridge University Press (recommended)
Grading Midterms (40%), Class Project (40%), Homework & Quizzes (10%), class participation (10%)
Important Dates: Midterm 1 and 2: TBA, projects due Nov. 25
Office Hours By appointment
Course Notes Will be made available via elearning

Suggested Projects

Available on WebCT

An interesting movie about Shannon, the father of information theory.

Contents:

  • Information, Entropy, and AEP
    • Entropy and mutual information
    • Chain rules and information inequalities
    • Markov Chains and entropy rate
    • Typical sets and Asymptotic Equipartition Property (AEP)

  • Data Compression
    • Compression and entropy
    • Kraft inequality
    • Huffman Codes
    • Arithmetic coding

  • Channel Capacity
    • Jointly typical sequences
    • Discrete channel capacity and Shannon’s 2nd theorem
    • Source-channel separation
    • Differential Entropy
    • Gaussian channel capacity
    • Continuous (waveform) channels
    • Capacity and Feedback
    • Error exponent analysis

  • The Wireless Channel
    • Implications of the Gaussian Capacity Formula
    • Fading and Outage Capacity
    • Diversity and Multiplexing
    • The MIMO channel

  • Introduction to Network Information Theory
    • The multiple-access channel
    • The broadcast channel
    • The relay channel
    • The Slepian-Wolf problem
    • The max-flow min-cut bound