Machine Learning using Scikit-Learn.ipynb

{
 "metadata": {
  "name": "Machine Learning using Scikit-Learn"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Machine learning overview\n",
      "Machine learning is to develop algorithms that make decisions using a model fitted on data.\n",
      "\n",
      "The goal is to build a model which makes decisions automatically\n",
      "based on new information, whose performance\n",
      "improves with experience. Initially train the model with a subset of the input.\n",
      "\n",
      "- [Scikit-Learn Documentation](http://scikit-learn.org/dev/)\n",
      "\n",
      "### Scikit-Learn\n",
      "scikit-learn package is a collection of machine learning algorithms\n",
      "which share this common usage pattern:\n",
      "\n",
      "- Load data\n",
      "- Pick model\n",
      "- Fit model parameters to data\n",
      "- Predict using fitted model"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn import datasets, neighbors\n",
      "# data sets consists of 3 different types of irises\u2019 (Setosa, Versicolour, and Virginica) \n",
      "# petal and sepal length, stored in a 150x4 numpy.ndarray\n",
      "iris = datasets.load_iris()\n",
      "\n",
      "# Classifier implementing the k-nearest neighbors vote.\n",
      "model = neighbors.KNeighborsClassifier()\n",
      "\n",
      "# Fit linear model\n",
      "model.fit(iris.data, iris.target)\n",
      "\n",
      "# Predict using the linear model, ordinary least squares Linear Regression\n",
      "predicted_arr = model.predict([7.5, 3, 6.5, 2.1])\n",
      "print predicted_arr"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[2]\n"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "- [Scikit-Learn Tutorials](http://scikit-learn.org/dev)\n",
      "- [Scikit-Learn Examples](http://scikit-learn.org/dev/auto_examples)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Cut & paste code from gallery and press CTRL-ENTER"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Cut & paste code from an example and press CTRL-ENTER "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Which model do we use?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.datasets import load_digits\n",
      "digits = load_digits()\n",
      "X, y = digits.data, digits.target\n",
      "trainingSet = X[:-100], y[:-100]\n",
      "testSet = X[-100:], y[-100:]\n",
      "\n",
      "def model_eval(model):\n",
      "    return model.fit(*trainingSet).score(*testSet)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.gaussian_process import GaussianProcess\n",
      "model_eval(GaussianProcess())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "0.74548917199754894"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.svm import SVC\n",
      "model_eval(SVC(kernel='linear', C=0.001))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 16,
       "text": [
        "0.98999999999999999"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Evaluate model performance with [cross-validation](http://scikit-learn.org/dev/model_selection.html)\n",
      "\n",
      "Is a model validation technique for assessing how the results of a statistical analysis will generalize to an independent data set\n",
      "\n",
      "Mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.cross_validation import cross_val_score\n",
      "from sklearn.datasets import load_iris\n",
      "from sklearn.linear_model import LogisticRegression\n",
      "\n",
      "iris = load_iris()\n",
      "model = LogisticRegression()\n",
      "\n",
      "# Array of scores of the estimator for each run of the cross validation\n",
      "# If cv None, a 3-fold cross validation is used\n",
      "cross_val_score(model, iris.data, iris.target)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 24,
       "text": [
        "array([ 0.94,  0.98,  0.98])"
       ]
      }
     ],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cross_val_score(model, iris.data, iris.target, cv=4)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "array([ 0.92105263,  0.92105263,  0.91891892,  0.97297297])"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.cross_validation import LeaveOneOut\n",
      "\n",
      "# use a single observation from the original sample as the validation data, \n",
      "# and the remaining observations as the training data\n",
      "cross_val_score(model, iris.data, iris.target, cv=LeaveOneOut(len(iris.target)))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 27,
       "text": [
        "array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  0.,  1.,  1.,  1.,  0.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  0.,\n",
        "        1.,  1.,  1.,  0.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,\n",
        "        1.,  1.,  1.,  1.,  1.,  1.,  1.])"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Evaluate performance with [pipelining](http://scikit-learn.org/dev/modules/pipeline.html)\n",
      "\n",
      "Chain multiple estimators into one. \n",
      "\n",
      "Useful as there is often a fixed sequence of steps in processing the data, for example feature selection, normalization and classification."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "from sklearn.pipeline import Pipeline\n",
      "from sklearn.decomposition import PCA\n",
      "from sklearn.linear_model import LogisticRegression\n",
      "from sklearn.cross_validation import cross_val_score\n",
      "from sklearn.datasets import load_digits; digits = load_digits()\n",
      "\n",
      "model = Pipeline([\n",
      "    ('pca', PCA()), \n",
      "    ('logistic', LogisticRegression()),\n",
      "])\n",
      "np.mean(cross_val_score(model, digits.data, digits.target))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 29,
       "text": [
        "0.95770728992765708"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### [Transform data](http://scikit-learn.org/dev/data_transforms.html) "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.preprocessing import StandardScaler\n",
      "\n",
      "# StandardScaler: Standardize features by removing the mean and scaling to unit variance\n",
      "# Principal component analysis: Linear dimensionality reduction using Singular Value Decomposition of the data and keeping only the most \n",
      "#     significant singular vectors to project the data to a lower dimensional space.\n",
      "\n",
      "# Logistic Regression: used for predicting the outcome of a categorical dependent variable (a dependent variable that can take on a \n",
      "#     limited number of values, whose magnitudes are not meaningful but whose ordering of magnitudes may or may not be meaningful) \n",
      "#     based on one or more predictor variables\n",
      "\n",
      "model = Pipeline([\n",
      "    ('scaler', StandardScaler()),\n",
      "    ('pca', PCA()), \n",
      "    ('logistic', LogisticRegression()),\n",
      "])\n",
      "np.mean(cross_val_score(model, digits.data, digits.target))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 30,
       "text": [
        "0.96160267111853093"
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.feature_extraction.text import CountVectorizer\n",
      "\n",
      "# CountVectorizer: Convert a collection of text documents to a matrix of token counts\n",
      "vectorizer = CountVectorizer(min_df=1)\n",
      "documents = [\n",
      "    'I ', \n",
      "    'She',\n",
      "    'She ',\n",
      "    'I ', \n",
      "]\n",
      "\n",
      "# fit_transform: Learn the vocabulary dictionary and return the count vectors.\n",
      "X = vectorizer.fit_transform(documents)\n",
      "documentVectors = X.toarray()\n",
      "documentVectors"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 50,
       "text": [
        "array([[0],\n",
        "       [1],\n",
        "       [1],\n",
        "       [0]])"
       ]
      }
     ],
     "prompt_number": 50
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "featureNames = vectorizer.get_feature_names()\n",
      "for bagOfWords in documentVectors:\n",
      "    print zip(featureNames, bagOfWords)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[(u'she', 0)]\n",
        "[(u'she', 2)]\n",
        "[(u'she', 1)]\n",
        "[(u'she', 0)]\n"
       ]
      }
     ],
     "prompt_number": 49
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Identifier for Lady Gaga"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from archiveIO import Archive, TemporaryFolder\n",
      "\n",
      "archive = Archive('data/ladygaga.tar.gz')\n",
      "documents = []\n",
      "categories = []\n",
      "with TemporaryFolder() as temporaryFolder:\n",
      "    for documentPath in archive.load(temporaryFolder):\n",
      "        text = open(documentPath).read()\n",
      "        documents.append(text)\n",
      "        categories.append('lady' in text)\n",
      "        \n",
      "print categories"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[True, True, True, True]\n"
       ]
      }
     ],
     "prompt_number": 56
    }
   ],
   "metadata": {}
  }
 ]
}