Machine Learning CS 5350/CS 6350 Fall 2008 Instructor: Hal Daume III: Office Hours: MEB 3126; Tue 9:30-10:30, Wed 4-5 or (preferably!) by appointment Schedule: Tuesday/Thursday, 3:40 - 5:00pm Location: MEB 1208 Mailing list: cs5350@list.eng.utah.edu -- PLEASE subscribe (but don't post)! teach-cs5350@list.eng.utah.edu -- PLEASE post (by don't subscribe)! RSS feed TA: Scott Alfeld (office hours Mon 11-3, Thr 12:10-2, in the CADE lab, or by appointment)

There are a few cool things about machine learning that I hope to get across in class. The first is that it's broadly applicable. These techniques have led to significant advances in many fields, including stock trading, robotics, machine translation, computer vision, medicine, etc. The second is that there is a very close connection between theory and practice. While this course is more on the "practical" side of things, almost everything we will talk about has a huge amount of accompanying theory. The third is that once you understand the basics of machine learning technology, it's a very open field and lots of progress can be made quickly, effectively by figuring out ways to formalize whatever we can figure out about the world.

The catalog lists CS 3510 as a prerequisite; this can be waived if you have or can quickly acquire reasonable programming skills (in Matlab) or if you are a graduate student. There will be a fair amount of math in this class, but all I'll really expect you to know coming in is how to take derivatives. Some assignments will probably go quicker if you have some background in continuous math (probability and/or linear algebra), but we'll cover in class all you need to know there.

• Homework assignments. There are ten written homework assignments. Each is worth one point. These are due before class on the date listed on the syllabus. Often, there will be questions on the homeworks that are only for students in 6350. Students in 5350 may answer these questions for extra credit.
• Programming projects. There are five programming projects. Each is worth four points. These are also due before class. These will also include extra questions for students in 6350; students in 5350 may do these as extra credit.
• Midterm exam. On 23 Oct, there will be a midterm exam covering everything up through (and including) 7 Oct. This is required for both 5350 and 6350 and is worth 6 points (one and a half projects).
• Final exam. At the scheduled final exam slot, there will be a final exam covering the entire semester, but focusing on topics after the midterm. This is required for both 5350 and 6350 and is worth 6 points.
• Learning competition. This is a required project for everyone. It is not worth many points (just two!), but is also relatively low overhead and hopefully a bit fun. See here for more details.

Your overall score will be out of 1*10+4*5+6+6+2=44. You can view the current grades (indexed by the last three digits of your student ID, "++" means "bonus").

• Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani and Jerome Friedman (ISBN 0387952845)
• Pattern Recognition and Machine Learning by Chris Bishop (SBN 0387310738)
• Information Theory, Inference and Learning Algorithms by David MacKay (ISBN 0521642981)
• Machine Learning by Tom Mitchell (ISBN 0070428077)
• An Introduction to Computational Learning Theory by Michael Kearns and Umesh Vazirani (ISBN 0262111934)

Homework assignments are due by before class (by 3:40pm) on the date listed on the syllabus. Programming assignments are to be completed in Matlab.

Date	Topics	Readings	HW	Notes
26 Aug	Course overview, general classes of problems Supervised, unsupervised, online and reinforcement learning Background in discrete probability	-	-
Model Fitting
28 Aug	Decision trees Entropy and information gain Overfitting and pruning	dtrees.pdf (p1-10)	-
2 Sep	Learning: It's all about generalization Evaluating learned hypotheses Cross validation and statistical significance	eval.pdf	HW0 due	-
4 Sep	Nearest neighbors Geometry of data, feature vectors	knn.pdf	-
9 Sep	Neural networks Network structure and perceptron	nnet.pdf (p1-5)	HW1 due
11 Sep	Matlab tutorial and Q/A session (in CADE Lab 2)	-	-
Function Approximation
16 Sep	Fitting parametric functions Gradient descent and back-propagation	nnet.pdf (p6-p14)	HW2 due	-
18 Sep	Support vector machines Optimizing the mistake bound: margins	Burges98.pdf (sec 3-3.4)	-
23 Sep	Catch-up	-	HW3 due P1 due	-
25 Sep	Loss functions and regularization 0/1 loss in NP-hard Hinge loss, log loss, exponential loss	Burges98.pdf (sec 3.5-3.7)	-	-
30 Sep	Loss functions and regularization II L0, L1, L2 and Linfinity penalties	-	HW4 due	-
2 Oct	Kernels Dual formulation of SVMs	Burges98.pdf (sec 4)	-	-
7 Oct	Kernels II From linear to non-linear learning	-	-
9 Oct	Catch-up and Midterm review	-	-	-
Learning Theory
21 Oct	PAC learning Definition of PAC learning	pac.pdf (through 1.3)	HW5 due P2 due
23 Oct	MIDTERM	-	-	-
28 Oct	PAC II: VC dimension	pac.pdf (rest)	-
30 Oct	Boosting Weak learners	schapire99boosting.pdf	-
4 Nov	Reductions Multiclass to binary: OVA, AVA	langford05wap.pdf	-
6 Nov	Reductions II Cost-sensitive classification, ranking	-	HW7 due	-
Bayesian Machine Learning
11 Nov	Conditional models Logistic and linear regression, revisited	prob.pdf linreg.pdf (esp p1-5)	P3 due
13 Nov	Inference in Bayesian models Maximum a posterior and clustering	-	-	-
18 Nov	Inference in Bayesian models II (Markov Chain) Monte Carlo	-	HW8 due	-
20 Nov	Inference in Bayesian models III Expectation maximization	mix_gauss.pdf	-
25 Nov	Latent variable models II Matching words and pictures	slides-bayes.pdf	P4 due	-
Assorted Topics
2 Dec	Structured prediction Conditional random fields and max-margin Markov networks	-	-
4 Dec	Collaborative Filtering The NetFlix Challenge	slides-cf.pdf	HW9 due	-
9 Dec	Natural language processing (Riloff)	-	-	-
11 Dec	Statistical image processing (Fletcher)	-	P5 due HW10 due	-

Written Homeworks

• HW0: Basic concepts and survey ( LaTeX)
• HW1: Decision Trees ( LaTeX); solution
• HW2: Data Geometry ( LaTeX); solution
• HW3: Neural Networks ( LaTeX); solution
• HW4: Loss and Optimization ( LaTeX); solution
• HW5: Support Vector Machines ( LaTeX); solution
• HW6: Cancelled (instead, your lowest grade on a homework will be dropped!)
• HW7: Learning Theory ( LaTeX); solution
• HW8: Probabilistic Modeling ( LaTeX); solution
• HW9: Expectation Maximization ( LaTeX); solution
• HW10: Assorted Topics ( LaTeX); solution

• P1: Decision Trees and Nearest Neighbors ( LaTeX) with data+scripts (solution)
• P2: Function Approximation ( LaTeX) with data+scripts (solution)
• P3: Learning Theory ( LaTeX) with data+scripts (solution))
• P4: Gaussian Classifiers [important: fixed version] ( LaTeX) with data+scripts (solution)
• P5: Latent Variable Models ( LaTeX) with data+scripts (solution)

This course is similar to several other machine learning courses, taught at other universities: CMU (Tom Mitchell and Andrew Moore), Stanford (Andrew Ng), Cornell (Thorsten Joachims) and Edinburgh (Sethu Vijayakumar). There have also been a series of summer schools on machine learning, some of which have videos up.

Although you won't need to use any of this software for your homeworks/projects, there are a large number of open-source machine learning toolkits out there. (Some of these may be useful for the competition.) A small sample:

• Torch3: a generic machine learning library, particularly good for neural networks, but also a lot more!
• MegaM: Optimization software for maximum entropy models, uses conjugate gradient for binary/binomial problems and LM-BFGS for multiclass problems
• FastDT: Very fast decision tree learner that implements bagging and boosting
• libSVM: a very efficient library for SVMs
• SVM-Light: another efficient library for SVMs
• Weka: the "defacto" machine learning/datamining library
• Mallet: a library for structured prediction with CRFs (plus other stuff)

ADA: The University of Utah conforms to all standards of the Americans with Disabilities Act (ADA). If you wish to qualify for exemptions under this act, notify the Center for Disabled Students Services, 160 Union.

College guidelines: Document concerning adding, dropping, etc. here.