notebooks/1-hr-lecture.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![DrivenData Logo](https://s3.amazonaws.com/drivendata-public-assets/logo-white-blue.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/bull/dd-dsis\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import os\n",
    "import sys\n",
    "\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "PROJ_ROOT = os.path.join(os.pardir)\n",
    "\n",
    "print(os.path.abspath(PROJ_ROOT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The [watermark](https://github.com/rasbt/watermark) extension\n",
    "\n",
    "Tell everyone when your notebook was run, and with which packages. This is especially useful for nbview, blog posts, and other media where you are not sharing the notebook as executable code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: watermark in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages\n",
      "Requirement already satisfied: ipython in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from watermark)\n",
      "Requirement already satisfied: setuptools>=18.5 in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: pickleshare in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: pygments in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: jedi>=0.10 in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: simplegeneric>0.8 in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: pexpect; sys_platform != \"win32\" in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: decorator in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: appnope; sys_platform == \"darwin\" in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: traitlets>=4.2 in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from ipython->watermark)\n",
      "Requirement already satisfied: six>=1.9.0 in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->watermark)\n",
      "Requirement already satisfied: wcwidth in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->watermark)\n",
      "Requirement already satisfied: ipython-genutils in /Users/bull/anaconda3/envs/dd-dsis/lib/python3.6/site-packages (from traitlets>=4.2->ipython->watermark)\n"
     ]
    }
   ],
   "source": [
    "!pip install watermark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# once it is installed, you'll just need this in future notebooks:\n",
    "%load_ext watermark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%watermark -a \"Peter Bull\" -d -t -v -p numpy,pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%watermark?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Laying the foundation\n",
    "\n",
    "Continuum's `conda` tool provides a way to create [isolated environments](http://conda.pydata.org/docs/using/envs.html). In fact, you've already seen this at work if you followed the [pydata setup](https://github.com/drivendata/pydata-setup) instructions to setup your machine for this tutorial. The `conda env` functionality let's you created an isolated environment on your machine for \n",
    "\n",
    " - Start from \"scratch\" on each project\n",
    " - Choose Python 2 or 3 as appropriate\n",
    "\n",
    "To create an empty environment:\n",
    "\n",
    " - `conda create -n <name> python=3`\n",
    "\n",
    "**Note: `python=2` will create a Python 2 environment; `python=3` will create a Python 3 environment.**\n",
    "\n",
    "\n",
    "To work in a particular virtual environment:\n",
    "\n",
    " - `source activate <name>`\n",
    " \n",
    "To leave a virtual environment:\n",
    "\n",
    " - `source deactivate`\n",
    "\n",
    "**Note: on Windows, the commands are just `activate` and `deactivate`, no need to type `source`.**\n",
    "\n",
    "There are other Python tools for environment isolation, but none of them are perfect. If you're interested in the other options, [`virtualenv`](https://virtualenv.pypa.io/en/stable/) and [`pyenv`](https://github.com/yyuu/pyenv) both provide environment isolation. There are _sometimes_ compatibility issues between the Anaconda Python distribution and these packages, so if you've got Anaconda on your machine you can use `conda env` to create and manage environments.\n",
    "\n",
    "-------------------\n",
    " \n",
    "**`#lifehack`: create a new environment for every project you work on**\n",
    "\n",
    "------------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `pip` [requirements.txt](https://pip.readthedocs.org/en/1.1/requirements.html) file\n",
    "\n",
    "It's a convention in the Python ecosystem to track a project's dependencies in a file called `requirements.txt`. We recommend using this file to keep track of your MRE, \"Minimum reproducible environment\".\n",
    "\n",
    "Conda\n",
    "\n",
    "-----------\n",
    "\n",
    "**`#lifehack`: never again run `pip install <package>`. Instead, update `requirements.txt` and run `pip install -r requirements.txt`**\n",
    "\n",
    "-------\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# what does requirements.txt look like?\n",
    "print(open(os.path.join(PROJ_ROOT, 'requirements.txt')).read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The format for a line in the requirements file is:\n",
    "\n",
    " | Syntax | Result |\n",
    " | --- | --- |\n",
    " | `package_name` | for whatever the latest version on PyPI is |\n",
    " | `package_name==X.X.X` | for an exact match of version X.X.X |\n",
    " | `package_name>=X.X.X` | for at least version X.X.X |\n",
    " \n",
    "Now, contributors can create a new virtual environment (using conda or any other tool) and install your dependencies just by running:\n",
    "\n",
    " - `pip install -r requirements.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "-------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's get to the data!\n",
    "\n",
    "We've got our environment set up, we're tracking the packages that we use, and we've got a standard folder structure. Now that all of that is working we can \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "## Try adding parameter index=0\n",
    "pump_data_path = os.path.join(PROJ_ROOT,\n",
    "                              \"data\",\n",
    "                              \"raw\",\n",
    "                              \"pumps_train_values.csv\")\n",
    "\n",
    "df = pd.read_csv(pump_data_path, index=0)\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.read_csv?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Paste for 'construction_year' and plot\n",
    "## Paste for 'gps_height' and plot\n",
    "plot_data = df['amount_tsh']\n",
    "sns.kdeplot(plot_data, bw=100)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def kde_plot(dataframe, variable, upper=None, lower=None, bw=0.1):\n",
    "    \"\"\" Plots a density plot for a variable with optional upper and\n",
    "        lower bounds on the data (inclusive).\n",
    "    \"\"\"\n",
    "    plot_data = dataframe[variable]\n",
    "    \n",
    "    if upper is not None:\n",
    "        plot_data = plot_data[plot_data <= upper]\n",
    "    if lower is not None:\n",
    "        plot_data = plot_data[plot_data >= lower]\n",
    "\n",
    "    sns.kdeplot(plot_data, bw=bw)\n",
    "    \n",
    "    plt.savefig(os.path.join(PROJ_ROOT, 'reports', 'figures', '{}.png'.format(variable)))\n",
    "    \n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "kde_plot(df, 'amount_tsh', bw=100, lower=0)\n",
    "kde_plot(df, 'construction_year', bw=1, lower=1000, upper=2016)\n",
    "kde_plot(df, 'gps_height', bw=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------\n",
    "\n",
    "\n",
    "-------\n",
    "\n",
    "\n",
    "# Writing code for reproducibility\n",
    "\n",
    "So, we've got some invalid data in this dataset. For example, water pumps installed before in the year 0. We'll want to have a function to load and clean this data since we will probably be using this data in multiple datasets.\n",
    "\n",
    "Here's a first pass at a function that will do that for us. Now, we've got the function implemented in the notebook, but let's bring it to a standalone file.\n",
    "\n",
    "We'll copy these functions into:\n",
    "`src/features/build_features.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def awesome_function(s):\n",
    "    from IPython.display import display, HTML\n",
    "    css = \"\"\"\n",
    "        .blink {\n",
    "            animation-duration: 1s;\n",
    "            animation-name: blink;\n",
    "            animation-iteration-count: infinite;\n",
    "            animation-timing-function: steps(2, start);\n",
    "        }\n",
    "        @keyframes blink {\n",
    "            80% {\n",
    "                visibility: hidden;\n",
    "            }\n",
    "        }\"\"\"\n",
    "\n",
    "    to_show = HTML(\n",
    "        '<style>{}</style>'.format(css) +\n",
    "        '<p class=\"blink\"> {} IS AWESOME!!!!! </p>'.format(s)\n",
    "    )\n",
    "    display(to_show)\n",
    "\n",
    "\n",
    "def remove_invalid_data(path):\n",
    "    \"\"\" Takes a path to a water pumps csv, loads in pandas, removes\n",
    "        invalid columns and returns the dataframe.\n",
    "    \"\"\"\n",
    "    df = pd.read_csv(path, index_col=0)\n",
    "\n",
    "    # preselected columns\n",
    "    useful_columns = ['amount_tsh',\n",
    "                      'gps_height',\n",
    "                      'longitude',\n",
    "                      'latitude',\n",
    "                      'region',\n",
    "                      'population',\n",
    "                      'construction_year',\n",
    "                      'extraction_type_class',\n",
    "                      'status_group',\n",
    "                      'management_group',\n",
    "                      'quality_group',\n",
    "                      'source_type',\n",
    "                      'waterpoint_type']\n",
    "\n",
    "    df = df[useful_columns]\n",
    "\n",
    "    invalid_values = {\n",
    "        'amount_tsh': {0: np.nan},\n",
    "        'longitude': {0: np.nan},\n",
    "        'installer': {0: np.nan},\n",
    "        'construction_year': {0: np.nan},\n",
    "    }\n",
    "\n",
    "    # drop rows with invalid values\n",
    "    df.replace(invalid_values, inplace=True)\n",
    "\n",
    "    # drop any rows in the dataset that have NaNs\n",
    "    df.dropna(how=\"any\")\n",
    "\n",
    "    # create categorical columns\n",
    "    for c in df.columns:\n",
    "        if df[c].dtype == 'object':\n",
    "            df[c] = df[c].astype('category')\n",
    "\n",
    "    df.drop('status_group')\n",
    "\n",
    "    return pd.get_dummies(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading local development files\n",
    "\n",
    "If I'm just loading local python files that I expect to use in this project, I often just add the `src` folder to the Python path using `sys.path.append`. This tells Python to look in that folder for modules that we can import. This works well for local code and notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# add local python functions\n",
    "import sys\n",
    "\n",
    "# add the 'src' directory as one where we can import modules\n",
    "src_dir = os.path.join(PROJ_ROOT, \"src\")\n",
    "sys.path.append(src_dir)\n",
    "\n",
    "# import my method from the source code\n",
    "from features.build_features import remove_invalid_data\n",
    "from features.build_features import awesome_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# edit function in file!\n",
    "awesome_function(\"ODSC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = remove_invalid_data(pump_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%debug"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned in the slides, using `sys.path.append` is not the best way to distribute code that you want to run on other machines. For that, create a real Python package that can be separately developed, maintained, and deployed.\n",
    "\n",
    "We can build a python package to solve that! In fact, there is a cookiecutter to create Python packages.\n",
    "Once we create this package, we can install it in \"editable\" mode, which means that as we change the code the changes will get picked up if the package is used. The process looks like:\n",
    "\n",
    "    cookiecutter https://github.com/wdm0006/cookiecutter-pipproject\n",
    "    cd package_name\n",
    "    pip install -e .\n",
    "\n",
    "Now we can have a separate repository for this code and it can be used across projects without having to maintain code in multiple places."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------------\n",
    "\n",
    "------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's train a model!\n",
    "\n",
    "Now, we'll use `sklearn` to train a machine learning model. We'll just do a simple logistic regression model, and for fun we'll use `PolynomialFeatures` to generate interaction terms. \n",
    "\n",
    "\n",
    "#### #lifehack: if something goes wrong use `%debug` !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "df = remove_invalid_data(pump_data_path)\n",
    "\n",
    "labels = (pd.read_csv(os.path.join(PROJ_ROOT, 'data', 'raw', 'pumps_train_labels.csv'),\n",
    "                     index_col=0)\n",
    "            .loc[df.index])\n",
    "\n",
    "pl = Pipeline([\n",
    "    ('interactions', PolynomialFeatures(degree=2)),\n",
    "    ('clf', LogisticRegression())\n",
    "])\n",
    "\n",
    "pl.fit(df, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pl.predict(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### #lifehack: if something takes a long time use `%%prun` !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import itertools\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    \n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, cm[i, j],\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    \n",
    "    plt.savefig(os.path.join(PROJ_ROOT, \"reports\", \"figures\", \"confusion_matrix.png\"))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cm = confusion_matrix(labels, clf.predict(df),\n",
    "                 labels=['functional', 'non functional', 'functional needs repair'])\n",
    "\n",
    "plot_confusion_matrix(cm,\n",
    "                      ['functional', 'non functional', 'functional needs repair'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now let's see what we've put together!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!tree {PROJ_ROOT}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}