Course Syllabus

Jump to Today

Course Overview

A graduate-level course covering technical foundations of modern natural language processing (NLP). The course will cast NLP as an application of machine learning, in particular deep learning, and focus on deriving general mathematical principles that underlie state-of-the-art NLP systems today. 

Course survey Links to an external site. (due May 6)

Prerequisites for undergraduates: linear algebra (M250), probability (CS206, or M477/S379), data structures (CS112). Recommended: multivariable calculus (M251), machine learning (533).

Syllabus Page Links to an external site.

Project Google Sheet Links to an external site. (15-minute meeting scheduling link Links to an external site.)

Instructor: Karl Stratos (karl.stratos@rutgers.edu)

Instructor Office Hours: Tuesday 4-5pm (Zoom link Links to an external site., passcode "rutgersnlp") 

Teaching Assistant: Wenyue Hua (wh302@scarletmail.rutgers.edu)

Teaching Assistant Office Hours: Thursday 3-4pm (Zoom link Links to an external site., passcode "nlpta")

Textbooks (for optional reading):

LaTeX templates:

Course Schedule

Week 1

Tuesday, January 19

Lecture: General introduction (video Links to an external site., slides Download slides)

Entrance Quiz: 3:20-4pm (TA available in the office hour Zoom link during this time window) 

Optional reading: Chapter 1 (Eisenstein); linear algebra review Links to an external site. (Kolter)

Week 2

Tuesday, January 26

Lecture: Linear classification (video Links to an external site., slides Download slides)

Optional reading: Chapter 2.5, 2.6  (Eisenstein) 

Assignment 1 assigned (due in 3 weeks) 

Jupyter Notebook on projections Download Jupyter Notebook on projections 

Week 3

Tuesday, February 2

Lecture: Optimization, introduction to deep learning (video Links to an external site., slides Download slides)

Optional reading: Notes on feedforward networks Links to an external site. (Collins), notes on backpropagation Links to an external site.

Jupyter Notebook on separable encodings Download Jupyter Notebook on separable encodings 

Week 4 

Tuesday, February 9 

Lecture: Feedforward networks, universality, backpropagation (video Links to an external site., slides Download slides)

Optional reading: Chapter 3.1-3.3 (Eisenstein), notes on Xavier initialization Links to an external site. (Stanford), notes on gradient-based optimization algorithms Links to an external site. (Ruder)

Thursday, February 11

Quiz 1: 30 minutes (available 1-6pm)

Week 5

Tuesday, February 16  

Lecture: Convolutional, recurrent and attention-based architectures (video Links to an external site., slides Download slides)

Optional reading: Chapter 3.4 (Eisenstein), Olah's blogs on LSTMs Links to an external site. and attention Links to an external site., notes on transformers Links to an external site. 

Assignment 2 assigned

Assignment 1 due

Week 6

Tuesday, February 23

Lecture: Language models, beam search, text generation (video Links to an external site., slides Download slides)

Optional reading: RNN LM Links to an external site. PyTorch example, generate Links to an external site. function in Hugging Face transformers, top-p/top-k sampling Links to an external site. implementation

Week 7 

Tuesday, March 2

Lecture: Conditional language models, machine translation (video Links to an external site., slides Download slides)

Optional reading: Chapter 18.1 (Eisenstein), Google NMT Links to an external site. and multilingual translation Links to an external site. papers, T5 Links to an external site. paper  

Thursday, March 4

Quiz 2: 40 minutes (available 1-6pm)

Week 8

Tuesday, March 9

Lecture: Copy mechanism, relation-aware self-attention, hidden Markov models (video Links to an external site., slides Download slides)

Optional reading: Gulcehre et al. (2016) Links to an external site., Shaw et al. (2018) Links to an external site., notes on hidden Markov models Links to an external site. (Collins), example of neural HMM (Chui and Rush, 2020) Links to an external site. 

Assignment 3 assigned

Assignment 2 due 

Spring Recess (March 12-20)

Week 9

Tuesday, March 23

Lecture: Marginal decoding, conditional random fields (video Links to an external site., slides) 

Optional reading: Chapter 7.5.3 (Eisenstein), Lample et al. (2016) Links to an external site., notes on graphical models Links to an external site. (Blei), notes on belief propagation Links to an external site.

Week 10

Tuesday, March 30

Lecture: Natural language understanding, pretrained language models (video Links to an external site., slides) 

Optional reading: The word2vec paper Links to an external site. (also a blog Links to an external site.), the ELMo paper, Links to an external site. the BERT paper Links to an external site., paper analyzing commonsense reasoning performance (Trichelair et al., 2019 Links to an external site.), paper about effects of pretraining scale (Zhang et al., 2020 Links to an external site.) 

Jupyter notebook on how to use BERT 

Thursday, April 1

Quiz 3: 30 minutes (available 1-6pm)

Week 11 

Tuesday, April 6

Lecture: More pretrained transformers, latent-variable generative models (video Links to an external site., slides)

Optional reading: The BART paper Links to an external site., Section 1 and Appendix A of this note Links to an external site., additional notes Links to an external site., VAEs applied to text generation Links to an external site. and document hashing Links to an external site.   

Project proposal due 

Assignment 3 due 

Week 12 

Tuesday, April 12

Lecture: More variational autoencoders, discrete latent variables (video Links to an external site., slides)

Optional reading: Notes on Gumbel (Appendix A of this note Links to an external site., you may have to refresh the page), Li et al. (2019) Links to an external site. 

Week 13

Tuesday, April 20

Lecture: Knowledge-intensive language tasks (video Links to an external site., slides) 

Optional reading: Notes on noise contrastive estimation Links to an external site., Lee et al. (2019) Links to an external site., Cheng et al. (2020) Links to an external site., Wu et al. (2020) Links to an external site. 

Milestone due

Week 14

Tuesday, April 26

Lecture: Coreference resolution, review (video Links to an external site., slides) 

Optional reading: Section 4.2 of Marquez et al. (2012) Links to an external site., LEA Links to an external site., end-to-end coref (Lee et al., 2017) Links to an external site. and its extension Links to an external site., coref with BERT Links to an external site. and SpanBERT Links to an external site., CorefQA Links to an external site.    

Monday, May 10

Project presentation video and final report due 

Course Summary:

Date Details Due