Machine Learning CS 5350/CS 6350 Fall 2009 Schedule: Tue/Thr 3:40-5:00pm Location: MEB 1208 Instructor: Hal Daume III: Office Hours: MEB 3126; Thr 9-10:30 or by appointment Mailing list: cs5350@list.eng.utah.edu -- PLEASE subscribe (but don't post)! teach-cs5350@list.eng.utah.edu -- PLEASE post (by don't subscribe)! RSS feed TA: Seth Juarez (office hours: Thr 1:30-3:30 and by appt)

This class will showcase machine learning technology in the context of recommender systems, ala what you see on Amazon or NetFlix (or eHarmony). The data we'll be working with is recommendations for CS courses at the U! But, for part of the final project, you are encouraged to work on the NetFlix prize, from which you could (probably no longer :() win $1,000,000.

There are a few cool things about machine learning that I hope to get across in class. The first is that it's broadly applicable. These techniques have led to significant advances in many fields, including stock trading, robotics, machine translation, computer vision, medicine, etc. The second is that there is a very close connection between theory and practice. While this course is more on the "practical" side of things, almost everything we will talk about has a huge amount of accompanying theory. The third is that once you understand the basics of machine learning technology, it's a very open field and lots of progress can be made quickly, effectively by figuring out ways to formalize whatever we can figure out about the world.

Prerequisites: I take prerequisites seriously. There will be a lot of math in this class and if you do not come prepared, life will be rough. You should be able to take derivatives by hand (preferably of multivariate functions), you should know what dot products are and how they are related to projections onto subspaces, you should know what Bayes' rule is and you should know that it's okay for the density of a Gaussian probability distribution to be greater than one. I've provided some reading material to refresh these issues in your head, but if you haven't at least seen these things before, you should beef up your math background before class begins. On the programming side, projects will be in Python; you should understand basic computer science concepts (like recursion), basic data structures (trees, graphs), and basic algorithms (search, sorting, etc.). (If you know matlab, here's a nice cheat sheet.)

Your responsibilities are as follows:

• Read the assigned reading assignments before class. It will be very helpful if you write down a short list of questions before class, though this isn't actually required. I'm serious about reading; to demonstrate that, most reading assignments are about 10 pages long (some are 15ish).
• Complete the assigned weekly homework assignments before class. Some will be "starred", meaning that we will spend the first part of class time going over the solutions. Students will present the solutions: you will be chosen to present uniformly at random (without replacement). We have our own handin system.
• Participate actively in class discussions.

Late homeworks are not allowed (without prior approval). This is because I need to put solutions up on the web page. You may hand any project in up to 48 hours late; however, once it is late by one minute, your final score will be halved.

We will post solutions to homeworks and projects quickly after the due dates. We will also email you your scores once grading has been completed. If you handed something in and do not get a score for an assignment, you have a one week moritorium on complaints.

You can view the current grades (indexed by the key you provided us in HW01).

The textbook is the new-ish book by Chris Bishop, Pattern Recognition and Machine Learning (ISBN 0387310738). Other recommended (but not required) books: Machine Learning by Tom Mitchell (ISBN 0070428077) Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani and Jerome Friedman (ISBN 0387952845) Information Theory, Inference and Learning Algorithms by David MacKay (ISBN 0521642981) An Introduction to Computational Learning Theory by Michael Kearns and Umesh Vazirani (ISBN 0262111934)

One thing that students have pointed out in the past that I'll point out to you is that Wikipedia has a bunch of good articles related to machine learning and statistics. Especially basic statistics stuff (various distributions, rules of probability, etc.) are very well explained there. I highly recommend it as an alternative source of information.

Date	Topics	Readings	Due	Notes
25 Aug	Welcome to Machine Learning What is ML, what is this class and some history	-	-
27 Aug	Decision trees, parameterized models Fitting parameters to data, probabilistic view of learning	PRML 1-1.1,1.3-1.4	HW01
Linear Models
01 Sep	Math you've forgotten Refresher on linear algebra and probability	PRML 1.2-1.2.4, App C	P0
03 Sep	Linear regression Least squares and probabilistic interpretation	PRML 3-3.1.4, 1.2.5	HW02	py
08 Sep	Linear classification Logistic and hinge regression, gradient descent	PRML 4-4.1.2, 4.3-4.3.2	-
10 Sep	Linear classification Perceptron algorithm	PRML 4.1.7, Perceptron	HW03
15 Sep	Generative classification Naive Bayes and generative vs discriminative	PRML 4.2 (optional: ng)	-
17 Sep	Practical issues Features and evaluation	eval	HW04	-
Non-linear Models
22 Sep	Neural networks Multiple layers and back propagation	PRML 5-5.2	-
24 Sep	Neural networks II Back propagation and invariances	PRML 5.3-5.3.2, 5.5-5.5.3 (optional: 5.5.5)	P1,HW05
29 Sep	Instance-Based Learning Nearest neighbors and locally weighted regression	mitchell-instance 8-8.4	-
01 Oct	Kernel methods Optimizing in the dual	PRML 6-6.2	HW06
06 Oct	Support vector machines Support vectors and hinge loss	PRML 7-7.1.2	-
08 Oct	Learning theory PAC learning and VC dimension	pac 7-7.4	HW07
Unsupervised Learning
20 Oct	Hierarchical clustering Algorithms and measures	hier (skip 14.1.3)	P2
22 Oct	K-means clustering Clustering as optimization	PRML 9-9.1.1	HW08
27 Oct	Mixture of Gaussians Soft clustering and psuedo-counts	PRML 9.2-9.3.2	-
29 Oct	Expectation maximization Jensen's inequality and majorization	PRML 9.4	HW09
03 Nov	Linear dimensionality reduction Principle components analysis and factor analysis	PRML 12-12.1.4, 12.2.4	-
05 Nov	Non-linear dimensionality reduction ISOMAP, LLE, kernel PCA and Laplacian Eigenmaps	PRML 12.3, 12.4.3, manifold	HW10
10 Nov	Topic modeling for text Latent semantic analysis and pLSA	LSA pLSA	-	-
Learning in Complex Settings
12 Nov	Sequence modeling Hidden markov models and Viterbi	PRML 13-13.2 (stop before 13.2.1) viterbi	HW11
17 Nov	Sequence modeling II Forward-backward algorithm	PRML 13.2.1, 13.2.2	P3, PP
19 Nov	Statistical relational learning Conditional random fields	crf (esp. 1-1.3)	HW12
24 Nov	Collaborative filtering Matrix factorization	factorize	-	-
01 Dec	Semi-supervised learning Expectation maximization and generative/discriminative	nigam	P4	-
03 Dec	Bayesian inference I Thinking Bayesian, linear regression revisited	PRML 1.2.3, 1.2.6, 3.3, (optional: 3.5)	HW13	-
08 Dec	Multimodal learning Matching words and pictures	wordspics	-	-
10 Dec	Bayesian inference II Approximation methods: variational EM and sampling	PRML 10.1-10.1.1, 11-11.1.2, 11.2	HW14	-
15 Dec	Final exam party (3:30 pm - 5:30 pm)	-	PW	-

Written Homeworks

• HW01: Survey and basic concepts (tex) -- Course-Rec survey
• HW02: Math and linear regression (tex); solution
• HW03: Linear classification (tex); solution
• HW04: Generative/discriminative (tex); solution
• HW05: Neural networks (tex); solution
• HW06: Kernel methods (tex); solution
• HW07: Support vector machines (tex); solution
• HW08: Distance-based clustering (tex); solution
• HW09: Expectation maximization (tex); solution
• HW10: Dimensionality reduction (tex); solution
• HW11: Text modeling and HMMs (tex)
• HW12: Forward-backward (tex)
• HW13: Semi-supervised learning
• HW14: Bayesian learning

Programming Projects

• P0: Unix/Python/NumPy tutorial (all files)
• P1: Linear models ; solution
• P2: Features and Kernels
• P3: Unsupervised learning
• P4: Text modeling

Brain Teasers

• World's best recommender system. During the course, we will be investigating machine learning primarily through the lens of recommendations. By the time the end of the semester rolls around, you'll have a lot of tools for doing recommendations. However, we will have been working exclusively with a relatively small and unique data set (course recommendations at the U). You should turn around and try to do as well as possible on the NetFlix challenge. I'll help you get most of the data in a similar format to what you've been using so far, but after that you're on your own. You're free to explore better feature engineering, data from alternative sources, more clever learning or optimization, etc.
• A project Hal finds particularly interesting. Here is a list of some possible project ideas that I think would be reasonable. I also maintain a list of open problems (only accessible from inside utah.edu), many of which have to do with machine learning. You should select one that sounds interesting, discuss it with me, and carry it out. You should pick one of them that is categorized under the ML keyword and probably one that's listed as "Months" for a time-frame. (Aside: most of these are things that, with a bit of extra work after class is over, could probably turn into a paper at a machine learning conference.)
• A project of your own choosing. If neither of the above options appeals to you, then you may propose a different topic to me. I will be quite picky with such projects: you have to convince me that you're actually going to be using your newly aquired expertise in machine learning to carry out the project. ("I will run XXX machine learning algorithm to some specific data set I care about" is not an acceptable project idea.)

• Project Proposal: This is a one to two page proposal that you (and your team, if you have one) will write and submit in early November (see schedule for details -- it's called "PP"). It should describe the problem you're working on, your approach to solving it, why you expect it to work, and how it relates to things we've talked about so far during the semester. This is worth 25% of your project grade.
• Project Writeup: An end-of-semester report on the project that you undertook. You should discuss what you did and any obstacles you encountered. You should present empirical results. You should discuss what worked and what didn't and why you think certain things worked and certain other things didn't. Be sure to have a section dedicated to talking about how this relates to what we learned in class and, even more importantly, what you learned in the process of doing the project. This is officially due the last day of class, but unofficially due on 18 December. This is worth 50% of your project grade.
• Project Presentation: All teams that include a 6350 student are responsible for giving a "spotlight" overview of their project at the final day party. Spotlights are a tradition in NIPS where you get two slides and two minutes to explain your idea. I'll maybe give you another minute or two depending on how many teams there are. Logistically, you must hand in your slides in PDF format by noon on the day of the party. This is worth 25% of your project grade.

This course is similar to several other machine learning courses, taught at other universities: CMU (Tom Mitchell and Andrew Moore), Stanford (Andrew Ng), Cornell (Thorsten Joachims) and Edinburgh (Sethu Vijayakumar). There have also been a series of summer schools on machine learning, some of which have videos up.

Although you won't need to use any of this software for your homeworks/projects, there are a large number of open-source machine learning toolkits out there. (Some of these may be useful for the competition.) A small sample:

• Torch3: a generic machine learning library, particularly good for neural networks, but also a lot more!
• MegaM: Optimization software for maximum entropy models, uses conjugate gradient for binary/binomial problems and LM-BFGS for multiclass problems
• FastDT: Very fast decision tree learner that implements bagging and boosting
• libSVM: a very efficient library for SVMs
• SVM-Light: another efficient library for SVMs
• Weka: the "defacto" machine learning/data-mining library
• Mallet: a library for structured prediction with CRFs (plus other stuff)

ADA: The University of Utah conforms to all standards of the Americans with Disabilities Act (ADA). If you wish to qualify for exemptions under this act, notify the Center for Disabled Students Services, 160 Union.

College guidelines: Document concerning adding, dropping, etc. here.