Computer Science 499-007 (9235) at Northern Arizona University, Spring 2019

Optimization Algorithms for Machine Learning

TuTh 9:35-10:50

Coding projects

• Project 1: nearest neighbors
• Project 2: linear models
• Project 3: neural networks

How to ace this class

Before class you should read the suggested book chapters. When you read, write a summary in your own words of every paragraph or section. Also write questions that you have during your reading so you can ask in class (if related to the lecture topic) or in office hours.

During class, instead of copying what I write on the whiteboard, I would suggest to try to write what you understood in your own words. Also I would suggest to ask questions in class as soon as you need clarification.

After class, you should review your notes with one of your classmates (ask one of the students who seem to be correctly answering a lot of questions). Ask each other questions and try to teach/summarize some of the material with each other – that is one of the best ways to learn.

Finally after doing all of the above, please email me to ask for an office hour.

Lectures

Abbreviated notes PDF

• Lecture 1 Tue Jan 15: Applications of machine learning, syllabus, calculus/C quiz.
• Lecture 2 Thu Jan 17: supervised learning.
• Lecture 3 Tue Jan 22: nearest neighbors.
  • 1d regression one train/validation split viz, code.
  • 1d classification one train/validation split viz, code.
• Lecture 4 Thu Jan 24: nearest neighbors CV data visualizations:
  • 1d regression 4-fold cross-validation viz, code.
  • 2d regression 3-fold cross-validation viz, code.
  • 2d classification ex from Hastie book viz+code.
• Lecture 5 Tue Jan 29: R package development.
  • Software you need to download:
    • C++ compiler.
      • Windows: Rtools, blog.
      • Mac: blog which explains how to install xcode/clang.
      • Debian/ubuntu: sudo apt-get install r-base r-base-dev.
      • redhat/centos: sudo yum install R
    • R-3.5.2 for running/executing R code.
    • If you already know how to use Emacs, I recommend using Emacs Speaks Statistics (ESS), which provides an R-mode for editing and running R/C++ code.
      • Easiest way to install emacs+ess is Vincent Goulet’s distribution.
      • Otherwise you can get ESS via the emacs package manager.
      • or manually download/install (my instructions).
      • Tips: I put (global-set-key [f9] 'compile) into my ~/.emacs so I can compile by hitting the F9 key.
      • See the first line of knn.cpp to see an example of how to customize compilation – typically R CMD INSTALL can be used to re-compile the C++ code (hit TAB key to scroll to next error in compilation buffer), then I run an R script that tests if my C++ code is working properly.
    • Otherwise, you can use RStudio IDE for writing R/C++ code.
    • git version control software.
  • Using RStudio with Git and GitHub.
  • Video screencast tutorials.
• Lecture 6 Thu Jan 31: Coding nearest neighbors prediction in C++
  • My tutorial about how to write R/C++ interface code.
  • Eigen C++ linear algebra library reference manual.
    • To use Eigen C++ code in an R package: install.packages(“RcppEigen”)
  • std::sort reference manual.
• Lecture 7 Tue Feb 05: linear regression, gradient descent with early stopping.
  • gradient descent for linear regression viz, code.
  • gradient descent choose step size viz, code.
• Lecture 8 Thu Feb 07: probabilistic interpretation of linear regression.
• Lecture 9 Tue Feb 12: logistic regression.
• Lecture 10 Thu Feb 14: L2 regularization.
  • gradient descent for L2 regularized (ridge) linear regression, viz, code.
• Lecture 11 Tue Feb 19: project review
• Lecture 12 Thu Feb 21: scaling
• Lecture 13 Tue Feb 26: exact line search
• Lecture 14 Thu Feb 28: backtracking line search
• Lecture 15 Tue Mar 05: vignette tutorial
  • Example vignette created in class: 2019-03-05-vignette/report.Rmd
  • previous quizzes exam-1-review.pdf
• Lecture 16 Thu Mar 07: exam review 1
• Lecture 17 Tue Mar 12: exam review 2
• Lecture 18 Thu Mar 14: mid-term

MIDTERM EXAM

• Lecture 19 Tue Mar 19: spring
• Lecture 20 Thu Mar 21: break
• Lecture 21 Tue Mar 26: Neural networks
• Lecture 22 Thu Mar 28: Backpropagation algorithm
  • Lecture 23 Tue Apr 02:
• Lecture 24 Thu Apr 04:
• Lecture 25 Tue Apr 09:
• Lecture 26 Thu Apr 11:
• Lecture 27 Tue Apr 16: paper review?
• Lecture 28 Thu Apr 18:
• Lecture 29 Tue Apr 23:
• Lecture 30 Thu Apr 25:
• Lecture 31 Tue Apr 30:
• Lecture 32 Thu May 02:
• Lecture 33 Tue May 07:
• Lecture 34 Thu May 09:

FINAL EXAM

Books

Murphy

• https://www.cs.ubc.ca/~murphyk/MLbook/
• NAU library https://arizona-nau-primo.hosted.exlibrisgroup.com/primo-explore/fulldisplay?docid=01NAU_ALMA51166833980003842&context=L&vid=01NAU&lang=en_US&search_scope=Everything&adaptor=Local%20Search%20Engine&tab=default_tab&query=any,contains,murphy%20machine%20learning&sortby=rank&mode=Basic
• e-book http://eds.a.ebscohost.com/ehost/ebookviewer/ebook?sid=f4a68ba6-099a-4015-9d63-a342771786f3%40sdc-v-sessmgr02&vid=0&format=EB

Hastie, Tibshirani, Friedman

• https://web.stanford.edu/~hastie/ElemStatLearn/

Press, et al.

• http://numerical.recipes/