From ResearchWiki

Contents 1 Organizing the Updated Machine Learning Class 1.1 Key Concepts 1.2 Topics Covered 1.3 Projects 1.4 Misc

Organizing the Updated Machine Learning Class

Key Concepts

• Model fitting
• Noise, error, overfitting, underfitting
• Feature extraction
• Probabilistic view of data
• Geometric view of data
• Decision boundaries

Topics Covered

1. Welcome, what is ML, what is this class, a little history
2. Decision trees, parameterized models, fitting parameters to data (probabilistic view of learning)
3. Linear regression: least squares and probabilistic interpretation
4. Linear classification: logistic regression (perceptron?)
5. Generative classification: naive Bayes
6. Practical issues: overfitting, cross-validation, regularization/priors, feature engineering (leading toward kernels)
7. Filler: Neural networks
8. Filler: Locally weighted regression and radial basis functions (maybe drop? would help understanding RBF kernels)
9. SVMs: optimizing in the dual, support vectors
10. SVMs: kernels
11. VC dimension and learning theory
12. Clustering: Agglomerative
13. Clustering: K-means
14. Mixture of Gaussians
15. Dimensionality reduction: PCA
16. EM
17. Filler: Dimensionality reduction: kPCA, LLE, Isomap, etc.
18. Filler: Semi-supervised learning (Nigam and McCallum-style)
19. Topic modeling for text (eg., pLSI)
20. Syntactic modeling for text (eg., HMMs)
21. Density estimation for text (eg., language models)
22. Filler: active learning
23. Filler: reductions
24. Applications: SRL
25. Applications: matrix factorization
26. Applications: vision

Projects

1. Linear models for classification and regression
   1. Linear regression (maybe we give this to you?)
   2. Logistic regression
   3. Naive Bayes (this would make document classification much more straightforward)
   4. Feature extraction (extra credit for doing something cool?)
   5. Due sometime around days 7/8
   6. Note: feature extraction *could* be a separate project, but timing might be difficult
   7. Data turn collaborative filtering into classification/regression problem... mostly text stuff, but could also use other people's predictions
   8. Code need infrastructure to view classification boundaries in 2D and view regressors in 2D; plus, something like the "color and resize words in text" for text classification, maybe in an online fashion as GD is happening
2. Kernels
   1. Kernel linear regression (one liner)
   2. Kernel classification? Logistic regression would be most natural fit; perceptron is easier, but we haven't talked about perceptron
   3. Combinations of kernels
   4. Design your own kernel (extra credit)
   5. Due sometime around day 12 (maybe we need some filler in here)
   6. Data same as P1
   7. Code again, need infrastructure to view classification boundaries and regressors in 2D, even though they're not linear (this is a bit of a pain)
3. Clustering and dimensionality reduction
   1. K-means (we give it to you)
   2. Mixture of multinomials (would fit better with text view, but not with lectures... we could make them do mixture of X, and have plug ins for multinomial and Gaussian and really anything else we want)
   3. PCA
   4. Would be cool if we can provide LLE/Isomap/something non-linear other than kPCA
   5. Design your own kernel for kPCA or something like that (extra credit)... or maybe turn kPCA into LLE or Isopmap
   6. Due some time around day 19
   7. Data real CF data this time (is missing data going to be a problem?)
   8. Code need to be able to view users and movies in the 2D projection, maybe color-coded by cluster ID... also could use something like the P1 text viz to do mixture of multinomials (but now we'd need multiclass -- not sure how to do this for >3 (RBG) classes)
4. Text modeling
   1. pLSI
   2. HMMs
   3. Is there anything fun extra credit here?
   4. Due some time around day 24
   5. Data same as P1
   6. Code Similar to P3 for visualizing projected documents; maybe something for visualizing an HMM lattice?
5. Final Contest
   1. Putting it all together
   2. Do better feature engineering with everything we know
   3. Code/Data Nothing new

Misc