Machine Learning CS 5350/CS 6350 Spring 2007 Instructor: Hal Daume III: Office Hours: MEB 3126; Tue 10:30-noon (or by appointment) Schedule: Tuesday/Thursday, 9:10 - 10:30am Locantion: NS 207 (NEW ROOM!) Mailing list: Cs5350 -- PLEASE subscribe (but don't post)! TA: Eun Yong Kang (office hours Wed 2-5pm, MEB 4158)

The catalog lists CS 3510 as a prerequisite; this can be waived if you have or can quickly acquire reasonable programming skills (in C, Matlab, Java, R, Perl, whatever; talk to me) or if you are a graduate student. There will be a fair amount of math in this class, but all I'll really expect you to know coming in is how to take multivariate derivatives. Some assignments will probably go quicker if you have some background in continuous math (probability and/or linear algebra), but we'll cover in class all you need to know there. See also Reading 0 for some more background.

• Supervised Learning: Linear classifiers (maximum entropy, perceptron, naive Bayes, support vector machines); non-linear classifiers (kNN, decision trees); kernels; bagging and boosting; model selection and feature selection.
• Unsupervised Learning: Clustering, EM, PCA, factor analysis.
• Structured Prediction: HMMs, CRFs, M3Ns, Searn.
• Bayesian Learning: Decision theory, exact inference, linear regression, logistic regression, sampling, latent Dirichlet allocation.

Each homework is worth 1 point (for a total of 5 points). The midterm is worth 4 points. The project is worth 4 points if you're enrolled in 5350 and 6 points if you're enrolled in 6350. The final exam is worth 6 points if you're enrolled in 5350 and 4 points if you're enrolled in 6350. One additional point is withheld for class participation, etc. There will be limited opportunity for extra credit on some homework assignments, midterm and final. I do not curve, but do adjust cutoffs based on overall performance (though the cutoffs for 5350 and 6350 will be determined separately).

Information Theory, Inference and Learning Algorithms by David MacKay (ISBN 0521642981)

Machine Learning by Tom Mitchell (ISBN 0070428077)

An Introduction to Computational Learning Theory by Michael Kearns and Umesh Vazirani (ISBN 0262111934)

Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani and Jerome Friedman (ISBN 0387952845)

Date	Topics	Readings	HW	Notes
09 Jan	What is machine learning? Learning theory, Bias, Overfitting	-	0 out
11 Jan	Math refresher Statistic, linear algebra and calculus	1.1-1.4, [0]	-
SUPERVISED LEARNING
16 Jan	Linear models for regression Least-squares techniques Bias/variance trade-off	3.1-3.2	0 due
18 Jan	Linear models for classification Logistic regression, naive Bayes	4.1-4.3, [1]	1 out
23 Jan	Linear models for classification (cont'd) Perceptron algorithm, linear SVMs	7.1.1-7.1.3, [2]	-
25 Jan	Linear models (cont'd) Linear SVMs, VC dimension	6.1-6.2, [2]	-	(same)
30 Jan	Non-linear models Nearest neighbors, decision trees	[3]	-
01 Feb	Non-linear models Non-linear SVMs, Kernels	[4]	1 due 2 out
06 Feb	Combining models Boosting, bagging and reductions	14.2, 14.3, [6], [7]	-
08 Feb	Feature selection Discussion of projects	[8]	-
UNSUPERVISED LEARNING
13 Feb	Clustering K-means, agglomerative	9.1, [9]	-
15 Feb	Clustering (cont'd) Mixtures of Gaussians	9.2, [10]	2 due 3 out
20 Feb	Expectation maximization	9.4, [11]	-
22 Feb	Expectation maximization (cont'd) Semi-supervised learning	[12]	-
27 Feb	Low-dimensional representations Principle component analysis Locally linear embedding	12.1, [13]	-
STRUCTURED PREDICTION
01 Mar	Sequence labeling Hidden Markov Models	13.1-13.2	-
06 Mar	Sequence labeling and beyond Maximum Entropy Markov Models Conditional Random Fields	[14], [15]	-
08 Mar	Sequence labeling and beyond (cont'd) Structured Perceptron Search-based Structured Prediction	[16], [18]	3 due 4 out
13 Mar	Catch-up	-	PP due	-
15 Mar	MIDTERM	-	-	-
20 Mar	Spring break	-	-	-
22 Mar	Spring break	-	-	-
BAYESIAN LEARNING
27 Mar	Introduction to Bayesian learning Probability distributions, graphical models	1.5, 2.1-2.4, 8.1	-
29 Mar	Cancelled (Hal out of town)	-	-	-
03 Apr	Inference Exact inference, basic sampling	8.2, 8.4.1-8.4.2	4 due 5 out
05 Apr	Inference (cont'd) Markov Chain Monte Carlo Latent Dirichlet allocation	11.1-11.3, [19], [20]	-
10 Apr	Classification and regression Bayesian linear/logistic regression Laplace approximation	3.3, 4.4-4.5	-
12 Apr	Catch-up	-	-	-
PROJECT PRESENTATIONS
17 Apr	Project presentations Virost; Hansen/Oh/Zhu; Kim/Rai; Lanka	-	-	-
19 Apr	Project presentations Gilbert/Bresee; Tandon; Valentine/Alfeld; Ha/Santos/Wang	-	-	-
24 Apr	Project presentations Gerber; Abbasi/Quist; Hetrick/Milyavskaya; Andrade	-	5 due PW due	-

0. Linear Algebra Review and Reference by Andrew Ng.
   Don't be afraid if not everything is 100% familiar. You can ignore Sections 3.6-3.13 and 4.2-4.6.
1. A Brief Maxent Tutorial by Adam Berger.
2. A Tutorial on Support Vector Machines for Pattern Recognition by Chris Burges. KDDM, 1998. (Also as PDF)
   You only need to read sections 1-3 for now.
3. Induction of Decision Trees by J.R. Quinlan. MLJ, 1986.
4. A Tutorial on Support Vector Machines for Pattern Recognition by Chris Burges. KDDM, 1998.
   Now, read sections 4-10 (the rest).
5. Computational Learning Theory by Sally Goldman.
   You only need to read sections 1-3 (though you are encouraged to read the whole thing).
6. Reductions Between Classification Tasks by Alina Beygelzimer, Varsha Dani, Tom Hayes, John Langford and Bianca Zadronzny. ICML, 2005.
7. A brief introduction to boosting by Robert Schapire. IJCAI, 1999.
8. An Introduction to Variable and Feature Selection by Isabelle Guyon and Andre Elisseeff. JMLR, 2003.
9. Handout: Chapter from Hastie et al.
10. Using EM To Estimate A Probablity Density With A Mixture Of Gaussians by Aaron D'Souza.
11. The Expectation Maximization Algorithm: A short tutorial by Sean Borman.
12. Semi-supervised Text Classification Using EM by Kamal Nigam, Andrew McCallum and Tom Mitchell. In Semi-supervised Learning, 2006.
13. An Introduction to Locally Linear Embedding by Lawrence Saul and Sam Roweis. 2001.
14. Maximum Entropy Markov Models for Information Extraction and Segmentation by Andrew McCallum, Dayne Freitag and Fernando Pereira. ICML, 2000.
15. An Introduction to Conditional Random Fields for Relational Learning by Charles Sutton and Andrew McCallum. In Intro to SRL, 2006.
16. Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms by Michael Collins. EMNLP, 2002.
17. Max-Margin Markov Networks by Ben Taskar, Carlos Guestrin and Daphne Koller. NIPS, 2003.
18. Search-based Structured Prediction by Hal Daume III, John Langford and Daniel Marcu. MLJ, under review.
19. Latent Dirichlet allocation by Dave Blei, Andrew Ng and Michael Jordan. JMLR, 2003.
    You can ignore Section 5 (Inference and Parameter Estimation).
20. Finding scientific topics by Tom Griffiths and Mark Steyvers. PNAS, 2004.

• HW0: Basic machine learning concepts, statistics and linear algebra (solution)
• HW1: Linear models (download train, dev and test data; also a Matlab shell); solution and implementation
• HW2: Non-linear models (download train, dev and test data; also a Matlab shell); solution and matlab code for kNN and perceptron and perl script for FastDT input
• HW3: Clustering and dimensionality reduction (download data, shell for EM and PCA code); solution and matlab code.
• HW4: Structured prediction (download hw4hmm.m, train.data and test.data); solution and matlab code and searn code
• HW5: Bayesian learning (download hw5.m); solution and code

There is more detail about the project and some canned ideas here. For some example projects from similar courses in previous years, see here, here, here or here.

The following software may be of interest:

• SVM packages: libSVM, SVMlight, SVMsequel
• Maximum entropy packages: MegaM
• CRFs: Mallet