update

Author: tmp-coder

CMU-ML/10701/HW1_Tex_v3.zip

@@ -0,0 +1,105 @@
+% Alex' default definitions and packages. Saves time
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% ams tools
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\usepackage{amsmath}
+\usepackage{amssymb}
+%\usepackage{amsthm}
+
+\usepackage{mdwlist}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% theorem and friends
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\usepackage{theorem}
+\newcommand{\BlackBox}{\rule{1.5ex}{1.5ex}}  % end of proof
+\newtheorem{example}{Example} 
+\newtheorem{theorem}{Theorem}
+\newtheorem{lemma}[theorem]{Lemma} 
+\newtheorem{assumption}[theorem]{Assumption} 
+\newtheorem{proposition}[theorem]{Proposition} 
+\newtheorem{remark}[theorem]{Remark}
+\newtheorem{corollary}[theorem]{Corollary}
+\newtheorem{definition}[theorem]{Definition}
+\newtheorem{conjecture}[theorem]{Conjecture}
+\newtheorem{axiom}[theorem]{Axiom}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% fancy fonts
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%\usepackage{eucal}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% graph drawing
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\usepackage[all]{xy}
+\CompileMatrices
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% math symbols and commands
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\newcommand{\eq}[1]{(\ref{#1})}
+\newcommand{\mymatrix}[2]{\left[\begin{array}{#1} #2 \end{array}\right]}
+
+%brackets
+\newcommand{\inner}[2]{\left\langle #1,#2 \right\rangle}
+\newcommand{\rbr}[1]{\left(#1\right)}
+\newcommand{\sbr}[1]{\left[#1\right]}
+\newcommand{\cbr}[1]{\left\{#1\right\}}
+\newcommand{\nbr}[1]{\left\|#1\right\|}
+\newcommand{\abr}[1]{\left|#1\right|}
+
+%sets
+\newcommand{\RR}{\mathbb{R}}
+\newcommand{\NN}{\mathbb{N}}
+\newcommand{\ZZ}{\mathbb{Z}}
+
+%matrix stuff
+\newcommand{\one}{\mathbf{1}}
+
+%domains
+\newcommand{\Bcal}{\mathcal{B}}
+\newcommand{\Mcal}{\mathcal{M}}
+\newcommand{\Dcal}{\mathcal{D}}
+\newcommand{\Qcal}{\mathcal{Q}}
+\newcommand{\Tcal}{\mathcal{T}}
+\newcommand{\Fcal}{\mathcal{F}}
+\newcommand{\Pcal}{\mathcal{P}}
+\newcommand{\Ccal}{\mathcal{C}}
+\newcommand{\Ocal}{\mathcal{O}}
+\newcommand{\Xcal}{\mathcal{X}}
+\newcommand{\Ical}{\mathcal{I}}
+\newcommand{\Jcal}{\mathcal{J}}
+\newcommand{\Ycal}{\mathcal{Y}}
+\newcommand{\Gcal}{\mathcal{G}}
+\newcommand{\Zcal}{\mathcal{Z}}
+\newcommand{\Ncal}{\mathcal{N}}
+\newcommand{\Ucal}{\mathcal{U}}
+\newcommand{\Ecal}{\mathcal{E}}
+\newcommand{\Hcal}{\mathcal{H}}
+\newcommand{\tildi}{\tilde{\imath}}
+\newcommand{\tildj}{\tilde{\jmath}}
+\newcommand{\bari}{\bar{\imath}}
+\newcommand{\barj}{\bar{\jmath}}
+
+%boldface
+\newcommand{\Eb}{\mathbf{E}}
+
+%more operators
+\DeclareMathOperator*{\argmax}{\mathrm{argmax}}
+\DeclareMathOperator*{\argmin}{\mathrm{argmin}}
+\DeclareMathOperator*{\sgn}{\mathrm{sgn}}
+\DeclareMathOperator*{\tr}{\mathrm{tr}}
+\DeclareMathOperator*{\cov}{\mathrm{Cov}}
+\DeclareMathOperator*{\var}{\mathrm{Var}}
+\DeclareMathOperator*{\mini}{\mathrm{minimize}}
+\DeclareMathOperator*{\maxi}{\mathrm{maximize}}
+\DeclareMathOperator*{\E}{\mathbb{E}}
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% comment tools
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\newcommand{\intset}[1]{\cbr{1..n}}
+\newcommand{\flexset}[2]{\cbr{{#1}_1, \ldots, {#1}_{#2}}}

CMU-ML/10701/HW1_Tex_v3/Tex Folder_v3/hw.sty

@@ -0,0 +1,97 @@
+\documentclass{article}
+
+\usepackage[usenames,dvipsnames]{color}
+\usepackage{hw}
+\usepackage{bm}
+\usepackage{amsmath}
+\usepackage{graphicx}
+\usepackage[colorlinks=true,urlcolor=blue]{hyperref}
+\usepackage{geometry}
+\geometry{margin=1in}
+\usepackage{float}
+\setlength{\marginparwidth}{2.15cm}
+\usepackage{booktabs}
+\usepackage{enumitem}
+\usepackage{epsfig}
+\usepackage{setspace}
+\usepackage{parskip}
+\usepackage[]{algorithm2e}
+\usepackage{comment}
+
+%\newcommand{\comment}[1]{\textcolor{blue}{\textsc{\textbf{[#1]}}}}
+
+% Some commands to allow solutions to be embedded in the assignment file.
+\ifcsname issoln\endcsname \else \def\issoln{1} \fi
+\newcommand{\soln}[1]
+{
+  \if\issoln 1
+  {\color{red}
+  \textbf{Solution:}
+  #1
+  }
+  \fi
+}
+
+% SOLUTIONS ENVIRONMENT DEFAULT.
+\newenvironment{soln2}{\begin{comment}}{\end{comment}}
+% TO SHOW SOLUTIONS, include following (else comment out):
+%\renewenvironment{soln2}{
+%    \leavevmode\color{red}\ignorespaces
+%    \textbf{Solution:}
+%}{}
+
+
+\newcommand{\norm}[1]{\lVert #1 \rVert}
+\newcommand{\st}{\mathrm{s.t.}}
+
+\makeatletter
+\newcommand{\removelatexerror}{\let\@latex@error\@gobble}
+\makeatother
+
+\begin{document}
+
+\section*{}
+\begin{center}
+  \centerline{\textsc{\LARGE Homework 1{\if\issoln 1 Solutions \else \fi}}}
+  \vspace{0.5em}
+  \centerline{\textsc{\Large Probability, Maximum Likelihood Estimation (MLE), Bayes Rule, kNN}}
+  \vspace{1em}
+  \textsc{\large CMU 10-701: Machine Learning (Fall 2016)} \\
+  \url{https://piazza.com/class/is95mzbrvpn63d}
+  \centerline{OUT: September 13th}
+  \centerline{DUE: September 26th, 11:59 PM}
+\end{center}
+
+
+\section*{START HERE: Instructions}
+
+
+\begin{itemize*}
+
+\item \textbf{Collaboration policy:} Collaboration on solving the homework is allowed, after you have thought about the problems on your own.  It is also OK to get clarification (but not solutions) from books or online resources, again after you have thought about the problems on your own.  There are two requirements: first, cite your collaborators fully and completely (e.g., ``Jane explained to me what is asked in Question 3.4'').  Second, write your solution {\em independently}: close the book and all of your notes, and send collaborators out of the room, so that the solution comes from you only.
+
+\item\textbf{Late Submission Policy:} You will be allowed 2 total late days without penalty for the entire semester. You may be late by 1 day on two different homeworks or late by 2 days on one homework. Weekends and holidays are also counted as late days. Late submissions are automatically considered as using late days.
+\\Once those days are used, you will be penalized according to the following policy:
+\begin{itemize}
+\item Homework is worth full credit before the deadline.
+\item It is worth half credit for the next 24 hours.
+\item It is worth zero credit after that.
+\end{itemize}
+You must turn in at least $n-1$ of the $n$ homeworks, even if for zero credit, in order to pass the course.
+
+
+\item\textbf{Submitting your work:} Assignments should be submitted as PDFs using Gradescope unless explicitly stated otherwise.  Each derivation/proof should be completed on a separate page. Submissions can be handwritten, but should be labeled and clearly legible.  Else, submissions can be written in LaTeX.  Upon submission, label each question using the template provided by Gradescope.  
+
+ \item \textbf{Programming}: All programming portions of the assignments should be submitted to Autolab.  We will not be using this for autograding, but rather for plagiarism detection, meaning you may use any language which you like to submit.
+\end{itemize*}
+
+\newpage
+
+\input{problem1}
+\input{problem2}
+\input{problem3}
+\input{problem4}
+\input{problem5}
+
+
+\end{document}

CMU-ML/10701/HW1_Tex_v3/Tex Folder_v3/main.pdf

@@ -0,0 +1,31 @@
+
+\section*{Problem 1: Background Material [20pts] (Devendra \& Hyun-Ah)}
+
+\subsection*{1.1 Probability review (Devendra)}
+
+Consider a naive classifier which stochastically assigns labels to data points; specifically, for each label $l$, it assigns that label with probability proportional to the number of times that label appears in the training dataset. Let the size of training set be $N$, total number of classes be $C$ and $n_i$ be number of datapoints of class $i$ ($\sum_i^C{n_i} = N$), then the probability of labelling a datapoint with class $i$ is $n_i/N$.
+
+1) Consider a training set with $N=100$, $C=2$, $n_1 = 50$, $n_2=50$ and a test set with, $N=100$, $C= 2$, $n_1 = 30$, $n_2 = 70$.\\
+a) What is the expected accuracy of the classifier on the training set?\\
+b) What is the expected accuracy of the classifier on the test set?\\
+
+2) Now consider a training set with $N=100$, $C=3$, $n_1 = 50$, $n_2=20$, $n_3=30$ and a test set with, $N=100$, $C= 3$, $n_1 = 20$, $n_2 = 20$, $n_3 = 60$.\\
+a) What is the expected accuracy of the classifier on the training set?\\
+b) What is the expected accuracy of the classifier on the test set?\\
+
+This kind of analysis can be used to calculate a baseline accuracy for various Machine Learning algorithms. 
+
+\subsection* {Problem 1.2: Bayes Rule [10 pts] (Hyun-Ah) }
+
+Smith owns a retail store for selling phones. The phones are manufactured at three different factories: $A, B, C$. Factory A, B, and C produces $20\%, 30\%,$ and $50\%$ of the phone being sold at Smith's store. The probabilities of the defective phones from stores $A, B,$ and $C$ are $2\%, 1\%,$ and $0.05\%$, respectively. The total number of phones being sold at Smith's store is $10000$. 
+One day, a customer walks up to Smith's store, and ask for a refund for a defective phone.
+
+a) What is the probability of a phone being defective?
+
+b) What is the probability that this defective phone is manufactured at factory $A$?
+
+c) What is the probability that this defective phone is manufactured at factory $B$?
+
+d) What is the probability that this defective phone is manufactured at factory $C$?
+
+\newpage

CMU-ML/10701/HW1_Tex_v3/Tex Folder_v3/main.tex

@@ -0,0 +1,24 @@
+\section*{Problem 2: kNN [30 pts] (Hemank)}
+
+In this problem, you will implement K-Nearest Neighbor approach on a synthetic dataset and on the provided dataset\footnote{\url{http://archive.ics.uci.edu/ml/datasets/Iris}}. We will figure out how does various parameters affect the performance of K-NN.
+
+\subsection*{$k$-Nearest Neighbor}
+Implement the $k$-nn algorithm and answer the following questions. We will use Euclidean distance for our implementation. For validation, we use $N$-fold cross validation($N=10$). 
+
+\textbf{Input Data}: We will generate synthetic input data. The input data will be generated using the code provided in the folder. The code is provided in R, Matlab and Python syntax. The code takes as input the following:
+
+$n$ is the number of samples, $p$ is the dimensionality of the data and $sigma$ determines the amount of noise.
+
+\begin{enumerate}
+\item{Why is it suitable to choose odd-$k$?}
+
+\item{Generate $n=1000$ points with $p=10$ and $sigma=0.001$. For different values of $k$, ranging from 1 to 50 (in increments of 5), plot the cross-validation train-error and test-error. What can you say about the effect of $k$? What is the optimal value of $k$, which you can infer from the plot? How does performance changes when you move from smaller values of $k$ to larger values?}
+
+\item{Generate $n=400$ points with $sigma=0.001$. Vary $p$ in range of $2$ to $50$. Run $k$-NN with $k=9$. Plot the cross-validation test-error and comment about effect of dimensionality on the algorithm.}
+
+\item{Generate $n=1000$ points with $sigma=0.001$ and $p=10$. Run $k$-NN with $k=9$. Vary the $N$ for $N$-fold validation from $2$ to $10$ and plot the cross-validation test error. What do you observe from the plot?}
+
+\item{On the given iris dataset, plot with varying $K=1,3,5,7$, the $N$-fold cross-validation test-error.}
+
+\end{enumerate}
+\newpage