Applications of NLP CS 5964/6964 Fall 2007 Instructor: Hal Daumé III: Office Hours: MEB 3126; TBA (or by appointment) Schedule: Monday/Wednesday, 9:10 - 10:30am Location: MEB 3147 Mailing list: Cs5964 -- PLEASE subscribe (but don't post)! Teach-Cs5964 -- send questions here! TA: Scott Alfeld (office hours T 11-2, R 11-1 in Cade Lab)

This course is about developing systems for solving high-level natural language processing problems. It complements the Natural Language Processing course that has been offered in the past -- students are encouraged to take both, in either order (there is no prerequisite structure). At the end of the course, you will have built four substantial NLP applications by combining learned linguistic- and data-processing skills with useful tools.

Grading: This course is entirely homework- and project-based. There are three small homework assignments and three projects, one each for each of the segments. Homeworks are worth one point each; projects, three. There are no exams.

Readings: There are no required textbooks for this class. Occasionally we will make use of handouts and online tutorials. (Result: you save money!)

Prerequisites: There are none! Well, you have to know how to program. Since this course is largely project-based, you're expected to write a fair amount of code. However, we'll also make heavy use of existing tools (such as finite state toolkits), which means that the majority of the programming will be creating input files from text. I recommend using Python or Perl or something like that that deals well with text, but I don't care. There will be a small amount of math, but nothing beyond basic probabilities: chain rule, Bayes rule (which we'll review).

• Language as a Sequence: We will begin by considering simple high-level NLP tasks to get a handle on what words are and how best to process them. This will include some basic linguistics, and some basic string processing. We will then move on to language modeling: methodologies for generating fluent text. This will naturally lead into applications for problems such as automatic speech recognition. Finally, we will apply these skills to problems involving identifying people, places and things in running text.
• Machine Translation: Next, we will discuss the problem of translating documents from one language to another automatically. This is perhaps the oldest problem in NLP, if not AI in general (this is what Turing wanted to solve during WWII). Amazingly, one can achieve remarkably good translations with very little human knowledge. We'll primarily consider corpus-based approaches, where we build translation systems based only on examples of translated documents.
• NLP on the Web: We'll discuss the role of NLP technology in enabling web usage. In particular, we'll cover basic information retrieval, summarization and question answering. Question answering systems aim to pin-point answers to factoid-style questions, such as "Who shot JFK?" We will discuss knowledge-rich and knowledge-poor approaches to solving this problem. Another take on the information overload problem is automatic document summarization: given an input document (or collection of documents), produce a short summary. We will discuss methods of summarization based on lexical items, discourse structure and coherence.

Date	Topics	Readings	HW	Notes
20 Aug	Introduction to natural language processing Overview of class	-	HW1 out
LANGUAGE AS A SEQUENCE
22 Aug	Basic linguistic theory Words, sentences, morphology, tagging Corpora and tools	Unix for Poets POS tag list	-
27 Aug	String processing techniques Text-to-sound conversion Finite state machines for language	NLTK 1	HW1A due	-
29 Aug	Probability 101 Conditional, Bayes rule, chain rule Estimating probabilities from data	Carmel	-	-
5 Sep	Probability in strings Noisy-channel framework Probabilistic automata	-	P1 out	-
10 Sep	Language modeling Distinguishing good strings from bad	-	HW1B due	-
12 Sep	Language modeling II Sparse data problem, smoothing	Goodman	-	-
17 Sep	Probabilistic string transformations Entity tagging	-	-	-
19 Sep	Probabilistic string transformations II Automatic speech recognition	-	-	-
24 Sep	Catch-up	-	HW1C due	-
MACHINE TRANSLATION
26 Sep	Incomplete data Cryptanalysis, tranliteration	-	HW2 out	-
1 Oct	Incomplete data II EM algorithm	EM notes	-	-
3 Oct	Word-based alignment models IBM models 1 and 2	SMT (pp.11-26)	HW2A due	-
15 Oct	Word-based alignment models II HMM model IBM models 3 and 4	SMT (pp.30-45)	P1 due	-
17 Oct	Machine translation decoding Integration with language models	-	P2 out	-
22 Oct	Catch-up	-	-	-
24 Oct	Toward phrase-based translation Combination of alignments	SMT (pp.61-71)	HW2B due	-
29 Oct	Phrase-based translation Beam search Discriminative training	SMT (pp.89-99)	-	-
31 Oct	Evaluation BLEU score	SMT (pp.157-175)	-
5 Nov	Syntax-based translation Current research directions	-	-	-
7 Nov	Catch-up	-	HW2C due	-
NLP ON THE WEB
12 Nov	Information Retrieval Inverted indices, TF-IDF	-	HW3 out P3 out
14 Nov	Single-document summarization Vector space model Sentence extraction	-	P2 due	-
19 Nov	Headline generation Keyword extraction using automata	-	HW3A due
21 Nov	Single-document summarization Discourse and coherence Responding to queries	-	-	-
26 Nov	Question answering I Knowledge-lean approaches	-	-
28 Nov	Question answering II Knowledge-rich approaches	-	-
3 Dec	Tree-tranducers and syntatic transformations	-	-	-
5 Dec	CLASS CANCELLED	-	P3, HW3B due	-

Segment	Assignment	Topic
1	Homework 1	Basic linguistics and probability (Solution; hw1c-wordbigram.pl and hw1c-cblm.pl -- rename .txt to .pl)
	Project 1	Language modeling and tagging (Solution and code)
2	Homework 2	Incomplete data (Solution and hw2b-make-ej.txt)
	Project 2	Machine translation (Solution and my outputs and code)
3	Homework 3	NLP on the Web (Solution)
	Project 3	Headline generation (data) (Solution and my outputs and code)

• USC (Kevin Knight and Daniel Marcu)
• Cambridge (Bill Byrne)
• Edinburgh (Philipp Koehn)
• Berkeley (Dan Klein)

• Carmel
  This is installed on the Cade machines in ~5964/bin

Cheating: Any assignment that is handed in must be your own work. However, talking with one another to understand the material better is strongly encouraged. Recognizing the distinction between cheating and cooperation is very important. If you copy someone else's solution, you are cheating. If you let someone else copy your solution, you are cheating. If someone dictates a solution to you, you are cheating. Everything you hand in must be in your own words, and based on your own understanding of the solution. If someone helps you understand the problem during a high-level discussion, you are not cheating. We strongly encourage students to help one another understand the material presented in class, in the book, and general issues relevant to the assignments. Any student who is caught cheating will be given an E in the course and referred to the University Student Behavior Committee. Please don't take that chance - if you're having trouble understanding the material, please let us know and we will be more than happy to help.

ADA: The University of Utah conforms to all standards of the Americans with Disabilities Act (ADA). If you wish to qualify for exemptions under this act, notify the Center for Disabled Students Services, 160 Union.