data_programming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data programming: Training data without hand labeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2018 term\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "0. [Overview](#Overview)\n",
    "0. [Set-up](#Set-up)\n",
    "0. [Motivation](#Motivation)  \n",
    "0. [The data programming model](#The-data-programming-model)\n",
    "0. [Basic implementation](#Basic-implementation)\n",
    "0. [Simple example: cheese vs. disease](#Simple-example:-cheese-vs.-disease)\n",
    "  0. [Cheese/disease data](#Cheese/disease-data)\n",
    "  0. [Cheese/disease labeling functions](#Cheese/disease-labeling-functions)\n",
    "  0. [Applying the cheese/disease labelers](#Applying-the-cheese/disease-labelers)\n",
    "  0. [Fitting the generative model to obtain cheese/disease labels](#Fitting-the-generative-model-to-obtain-cheese/disease-labels)\n",
    "  0. [Training discriminative models for cheese/disease prediction](#Training-discriminative-models-for-cheese/disease-prediction)\n",
    "0. [In-depth example: Stanford Sentiment Treebank](#In-depth-example:-Stanford-Sentiment-Treebank)\n",
    "  0. [SST training set](#SST-training-set)\n",
    "  0. [Lexicon-based labeling functions](#Lexicon-based-labeling-functions)\n",
    "  0. [Other SST labeling function ideas](#Other-SST-labeling-function-ideas)\n",
    "  0. [Applying the SST labeling functions](#Applying-the-SST-labeling-functions)\n",
    "  0. [Fitting the SST generative model](#Fitting-the-SST-generative-model)\n",
    "  0. [Direct assessment of the inferred labels against the gold ones](#Direct-assessment-of-the-inferred-labels-against-the-gold-ones)\n",
    "  0. [Fitting a discriminative model on the noisy labels](#Fitting-a-discriminative-model-on-the-noisy-labels)\n",
    "0. [Extra-credit bake-off](#Extra-credit-bake-off)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook provides an overview of the __data programming__ model pioneered by [Ratner et al. 2016](https://papers.nips.cc/paper/6523-data-programming-creating-large-training-sets-quickly):\n",
    "\n",
    "* This model synthesizes a bunch of noisy labeling functions into a set of (binary) supervised labels for examples. These labels are then used for __training__. \n",
    "\n",
    "* Thus, on this model, one need only have gold labels for assessment, thereby greatly reducing the burden of labeling examples.\n",
    "\n",
    "* The researchers open-sourced their code as [Snorkel](https://github.com/HazyResearch/snorkel). For ease of use and exploration, we'll work with a simplified version derived from [this excellent blog post](https://hazyresearch.github.io/snorkel/blog/dp_with_tf_blog_post.html). This is implemented in our course repository as `tf_snorkel_lite.py`.\n",
    "\n",
    "* Project teams that find this direction useful are encouraged to use the real Snorkel, as it will better handle the complex relationships that inevitably arise in a set of real labeling functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Set-up\n",
    "\n",
    "The set-up steps are [the same as those required for working with the Stanford Sentiment Treebank materials](sst_01_overview.ipynb#Set-up), since we'll be revisiting that dataset as an in-depth use-case. Make sure you've done a recent pull of the repository so that you have the latest code release."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Applications/anaconda/envs/nlu/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from tf_snorkel_lite import TfSnorkelGenerative, TfLogisticRegression\n",
    "import sst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Motivation\n",
    "\n",
    "Have newer methods reduced the need for labels? Has crowdsourcing made it easy enough to get labels at scale?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Types of learning\n",
    "\n",
    "1. __Supervised learning__: Individual examples _from the domain you care about_ labeled in a way that you think/assume/hope is aligned with your actual real-world objective. The model objective is to minimize error between predicted and actual.\n",
    "\n",
    "2. __Distantly supervised learning__: Exactly like supervised learning, but with individual examples _from a domain that is different from the one you care about_.\n",
    "\n",
    "3. __Semi-supervised learning__: A fundamentally supervised method that can make use of unlabeled data. \n",
    "\n",
    "3. __Reinforcement learning__: The data are in some sense labeled, but not at the level of individual examples. The model objective is essentially as in supervised learning.\n",
    "\n",
    "4. __Unsupervised learning__: No labels that you can make use of directly. The model objective is thus set independently of the data but is presumably tied to something intuitive.\n",
    "\n",
    "In almost all domains right now, __effective learning is supervised learning__ – somewhere in 1–4. However, representations from unsupervised learning are very common as inputs to supervised deep learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The cost of labeling\n",
    "\n",
    "[Ratner et al. 2016](https://papers.nips.cc/paper/6523-data-programming-creating-large-training-sets-quickly):\n",
    "> In many applications, we would like to use machine learning, but we face the following challenges: \n",
    ">\n",
    "> (i) hand-labeled training data is not available, and is prohibitively expensive to obtain in sufficient quantities as it requires expensive domain expert labelers; \n",
    ">\n",
    "> (ii) related external knowledge bases are either unavailable or insufficiently specific, precluding a traditional distant supervision or co-training approach; \n",
    ">\n",
    "> (iii) application specifications are in flux, changing the model we ultimately wish to learn.\n",
    "\n",
    "In addition, the annotator will register the same judgment repeatedly.\n",
    "\n",
    "Point (iii) is subtle but very important: labels tend to be brittle, useful only for a narrow range of tasks, and thus they can quickly become irrelevant where one's scientific or business goals are evolving."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: SNLI\n",
    "\n",
    "SNLI ([Bowman et al. 2015](http://aclweb.org/anthology/D/D15/D15-1075.pdf)) represents one of the largest labeling efforts in NLP to date. It provides reasonable coverage for a __very__ narrow domain. The most frequent complaint is that it is too specialized.\n",
    "\n",
    "\n",
    "### Example: I2B2\n",
    "\n",
    "From [Uzuner 2009](https://academic.oup.com/jamia/article-abstract/16/4/561/766997):\n",
    " \n",
    "> To define the Obesity Challenge task, two experts from the Massachusetts General Hospital Weight Center studied 50 (25 each) random pilot discharge summaries from the Partners HealthCare Research Patient Data Repository.\n",
    ">\n",
    "> [...]\n",
    ">\n",
    "> The data for the challenge were annotated by two obesity experts from the Massachusetts General Hospital Weight Center. The experts were given a _textual task_, which asked them to classify each disease (see list of diseases above) as Present, Absent, Questionable, or Unmentioned based on explicitly documented information in the discharge summaries [...]. The experts were also given an _intuitive task_, which asked them to classify each disease as Present, Absent, or Questionable by applying their intuition and judgment to information in the discharge summaries,\n",
    "\n",
    "Extrapolate these costs, in money and time, to the +1M records we'd need for reasonable coverage of obesity patient experiences.\n",
    "\n",
    "### Example: THYME\n",
    "\n",
    "From [Styler et al. 2017](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5657277/):\n",
    "\n",
    "> The THYME colon cancer corpus, which includes clinical notes and pathology reports for 35 patients diagnosed with colon cancer for a total of 107 documents. Each note was annotated by a pair of graduate or undergraduate students in Linguistics at the University of Colorado, then adjudicated by a domain expert.\n",
    "\n",
    "Again, extrapolate these costs to a dataset that would provide reasonable coverage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The data programming model\n",
    "\n",
    "1. Suppose we have some raw set of $m$ examples $T$. To help keep the concepts straight, assume that these are just raw examples, not representations for machine learning.\n",
    "<br /><br />\n",
    "2. We write a set of $n$ labeling functions $\\Lambda$:\n",
    "    * Each $\\lambda \\in \\Lambda$ maps each $t \\in T$ to a label in $\\{-1, 0, 1\\}$. \n",
    "    \n",
    "    * These labeling functions need not be mutually consistent.\n",
    "    \n",
    "    * We expect each $\\lambda$ to be high precision and low recall. We hope that $\\Lambda$ in aggregate is high precision and high recall.\n",
    "<br /><br /> \n",
    "3. Think of $\\Lambda$ as mapping each $t$ to a vector of labels of dimension $n$ – e.g., $\\Lambda(t) = [-1, 1, 1, 0]$. Let $\\Lambda(T)$ be the $m \\times n$ matrix of these representations.\n",
    "<br /><br />\n",
    "4. We fit a generative model to $\\Lambda(T)$ that returns a binary vector $\\widehat{y}$ of length $m$. These are the labels for the examples in $T$ we'll use for __training__.\n",
    "<br /><br />\n",
    "5. From here, it's just supervised learning as usual. A feature function will map $T$ to a matrix of representations $X$, and you can pick your favorite supervised model. It will learn from $(X, \\widehat{y})$. In this way, you're doing supervised learning without any actual labeled data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Basic implementation\n",
    "\n",
    "Our implementation is in `tf_snorkel_lite.py`, as `TfSnorkelGenerative`. It works well, but it is mainly for illustrative purposes. As noted above, its primary limitation is that it makes the \"naive Bayes\" assumption that the labeling functions are independent. Since real-world labeling functions you want to write will likely have many complex dependencies between them, this is strictly speaking an incorrect model. (In practice, and like Naive Bayes classifiers, the model might nonetheless work well!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Simple example: cheese vs. disease\n",
    "\n",
    "Let's start with a toy example modeled on the cheese/disease problem that is distributed with the [Stanford MaxEnt classifier](https://nlp.stanford.edu/software/classifier.shtml)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cheese/disease data\n",
    "\n",
    "The first three examples are diseases, and the rest are cheeses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "T = [\"gastroenteritis\", \"gaucher disease\", \"blue sclera\",\n",
    "     \"cure nantais\", \"charolais\", \"devon blue\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = [1, 1, 1, 0, 0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cheese/disease labeling functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first two positively label diseases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def contains_biological_word(text):\n",
    "    disease_words = {'disease', 'syndrome', 'cure'}\n",
    "    return 1.0 if {w for w in disease_words if w in text} else 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ends_in_itis(text):\n",
    "    \"\"\"Positively label diseases\"\"\"\n",
    "    return 1.0 if text.endswith('itis') else 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These positively label cheeses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sounds_french(text):\n",
    "    return -1.0 if text.endswith('ais') else 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def contains_color_word(text):\n",
    "    colors = {'red', 'blue', 'purple'}\n",
    "    return -1.0 if {w for w in colors if w in text} else 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying the cheese/disease labelers\n",
    "\n",
    "We apply all the labeling functions to form the $\\Lambda(T)$ matrix [described in the model overview above](#The-data-programming-model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_labelers(T, labelers):\n",
    "    return np.array([[l(t) for l in labelers] for t in T])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "labelers = [contains_biological_word, ends_in_itis,\n",
    "            sounds_french, contains_color_word]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "L = apply_labelers(T, labelers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a look at $\\Lambda(T)$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>contains_biological_word</th>\n",
       "      <th>ends_in_itis</th>\n",
       "      <th>sounds_french</th>\n",
       "      <th>contains_color_word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>gastroenteritis</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gaucher disease</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>blue sclera</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cure nantais</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>charolais</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>devon blue</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 contains_biological_word  ends_in_itis  sounds_french  \\\n",
       "gastroenteritis                       0.0           1.0            0.0   \n",
       "gaucher disease                       1.0           0.0            0.0   \n",
       "blue sclera                           0.0           0.0            0.0   \n",
       "cure nantais                          1.0           0.0           -1.0   \n",
       "charolais                             0.0           0.0           -1.0   \n",
       "devon blue                            0.0           0.0            0.0   \n",
       "\n",
       "                 contains_color_word  \n",
       "gastroenteritis                  0.0  \n",
       "gaucher disease                  0.0  \n",
       "blue sclera                     -1.0  \n",
       "cure nantais                     0.0  \n",
       "charolais                        0.0  \n",
       "devon blue                      -1.0  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(L, columns=[x.__name__ for x in labelers], index=T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Fitting the generative model to obtain cheese/disease labels\n",
    "\n",
    "Now we get to the heart of it – using `TfSnorkelGenerative` to synthesize these label-function vectors into a single set of (probabilistic) labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "snorkel = TfSnorkelGenerative(max_iter=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration 100: loss: 5.951983451843262"
     ]
    }
   ],
   "source": [
    "snorkel.fit(L)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the predicted probabilistic labels, along with their non-probabilistic counterparts (derived from mapping scores above 0.5 to 1 and scores at or below 0.5 to 0):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_proba = snorkel.predict_proba(L)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = snorkel.predict(L)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predict_proba</th>\n",
       "      <th>texts</th>\n",
       "      <th>true</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.916934</td>\n",
       "      <td>gastroenteritis</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.836139</td>\n",
       "      <td>gaucher disease</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.083066</td>\n",
       "      <td>blue sclera</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.500000</td>\n",
       "      <td>cure nantais</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.163861</td>\n",
       "      <td>charolais</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.083066</td>\n",
       "      <td>devon blue</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   predict_proba            texts  true\n",
       "0       0.916934  gastroenteritis     1\n",
       "1       0.836139  gaucher disease     1\n",
       "2       0.083066      blue sclera     1\n",
       "3       0.500000     cure nantais     0\n",
       "4       0.163861        charolais     0\n",
       "5       0.083066       devon blue     0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'texts':T, 'true': y, 'predict_proba': pred_proba})\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we did pretty well. Only `blue sclera` tripped this model up. If we wanted to address that, we could write a labeling function to correct it. But let's retain this mistake to see what impact it has."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training discriminative models for cheese/disease prediction\n",
    "\n",
    "At this point, it's just training classifiers as usual. The only difference is that we're using the potentially noisy labels created by the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To round it out, I define a feature function `character_ngram_phi`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def character_ngram_phi(s, n=4):\n",
    "    chars = list(s)\n",
    "    chars = [\"<w>\"] + chars + [\"</w>\"]\n",
    "    data = []\n",
    "    for i in range(len(chars)-n+1):\n",
    "        data.append(\"\".join(chars[i: i+n]))\n",
    "    return Counter(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create a feature matrix in the usual way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = DictVectorizer(sparse=False)\n",
    "\n",
    "feats = [character_ngram_phi(s) for s in T]\n",
    "\n",
    "X = vec.fit_transform(feats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And then we fit a model. The real data programming way is to fit this model with the predicted probability values rather than the 1/0 versions of them. The `sklearn` class `LogisticRegression` doesn't support this, but this is an easy extension of [our core TensorFlow framework](tensorflow_models.ipynb):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod = TfLogisticRegression(max_iter=5000, l2_penalty=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration 5000: loss: 0.47204923629760743"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf_snorkel_lite.TfLogisticRegression at 0x1a1c5f9358>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mod.fit(X, pred_proba)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd_pred = mod.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['predicted'] = cd_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predict_proba</th>\n",
       "      <th>texts</th>\n",
       "      <th>true</th>\n",
       "      <th>predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.916934</td>\n",
       "      <td>gastroenteritis</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.836139</td>\n",
       "      <td>gaucher disease</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.083066</td>\n",
       "      <td>blue sclera</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.500000</td>\n",
       "      <td>cure nantais</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.163861</td>\n",
       "      <td>charolais</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.083066</td>\n",
       "      <td>devon blue</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   predict_proba            texts  true  predicted\n",
       "0       0.916934  gastroenteritis     1          1\n",
       "1       0.836139  gaucher disease     1          1\n",
       "2       0.083066      blue sclera     1          0\n",
       "3       0.500000     cure nantais     0          0\n",
       "4       0.163861        charolais     0          0\n",
       "5       0.083066       devon blue     0          0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That looks good, but the model's ability to generalize seem not so great:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "tests = ['maconnais', 'dermatitis']\n",
    "\n",
    "X_test = vec.transform([character_ngram_phi(s) for s in tests])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 0]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mod.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use a standard `sklearn` `LogisticRegression` on the 1/0 labels. It works better for the test cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['negative', 'positive'], dtype='<U8')"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "\n",
    "lr.fit(X, pred)\n",
    "\n",
    "lr.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## In-depth example: Stanford Sentiment Treebank\n",
    "\n",
    "The toy illustration shows how the model works and suggests it should work. Let's see how we do in practice by returning to the Stanford Sentiment Treebank (SST) – but this time without using any of the training labels!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### SST training set\n",
    "\n",
    "Here we just load in the SST training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "sst_train = list(sst.train_reader(class_func=sst.binary_class_func))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll keep the training labels as `sst_train_y` for a comparison, but they won't be used for training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "sst_train_texts, sst_train_y = zip(*sst_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Lexicon-based labeling functions\n",
    "\n",
    "The `vsmdata` distribution contains an excellent multidimensional sentiment lexicon, `Ratings_Warriner_et_al.csv`. The following function loads it into a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_warriner_lexicon(src_filename, df=None):\n",
    "    \"\"\"Read in 'Ratings_Warriner_et_al.csv' and optionally restrict its \n",
    "    vocabulary to items in `df`.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    src_filename : str\n",
    "        Full path to 'Ratings_Warriner_et_al.csv'\n",
    "    df : pd.DataFrame or None\n",
    "        If this is given, then its index is intersected with the \n",
    "        vocabulary from the lexicon, and we return a lexicon \n",
    "        containing only values in both vocabularies.\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    pd.DataFrame\n",
    "    \n",
    "    \"\"\"\n",
    "    lexicon = pd.read_csv(src_filename, index_col=0)\n",
    "    lexicon = lexicon[['Word', 'V.Mean.Sum', 'A.Mean.Sum', 'D.Mean.Sum']]\n",
    "    lexicon = lexicon.set_index('Word').rename(\n",
    "        columns={'V.Mean.Sum': 'Valence', \n",
    "                 'A.Mean.Sum': 'Arousal', \n",
    "                 'D.Mean.Sum': 'Dominance'})\n",
    "    if df is not None:\n",
    "        shared_vocab = sorted(set(lexicon.index) & set(df.index))\n",
    "        lexicon = lexicon.loc[shared_vocab]\n",
    "    return lexicon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "lex = load_warriner_lexicon(\n",
    "    os.path.join('vsmdata', 'Ratings_Warriner_et_al.csv'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The lexicon contains scores, rather than classes, so I create positive and negative sets from the words that are one standard deviation above and below the mean, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "sd_high = lex['Valence'].mean() + lex['Valence'].std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "sd_low = lex['Valence'].mean() - lex['Valence'].std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_words = set(lex[lex['Valence'] > sd_high].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_words = set(lex[lex['Valence'] < sd_low].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lex_pos_labeler(tree):\n",
    "    return 1 if set(tree.leaves()) & pos_words else 0    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lex_neg_labeler(tree):\n",
    "    return -1 if set(tree.leaves()) & neg_words else 0    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other SST labeling function ideas\n",
    "\n",
    "* More lexicon-based features: http://sentiment.christopherpotts.net/lexicons.html\n",
    "\n",
    "* Position-sensitive lexicon features. For example, perhaps core lexicon features should be reversed if there is a preceding negation or a following _but_.\n",
    "\n",
    "* Features for near-neighbors of lexicon words, in a VSM derived from, say, `imdb5` or `imdb20` from our VSM unit.\n",
    "\n",
    "* Feature identifying specific actors and directors, building in assumptions that their moves are good or bad.\n",
    "\n",
    "* Negations like _not_, _never_, _no one_, and _nothing_ as signals of negativity in the evaluative sense ([Potts 2010](https://journals.linguisticsociety.org/proceedings/index.php/SALT/article/view/2565)); universal quantifiers like _always_, _all_, and _every_ as signals of positivity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying the SST labeling functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "sst_train_labels = apply_labelers(\n",
    "    sst_train_texts, \n",
    "    [lex_neg_labeler, lex_pos_labeler])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Fitting the SST generative model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration 1000: loss: 2.0862767696380615"
     ]
    }
   ],
   "source": [
    "nb = TfSnorkelGenerative(max_iter=1000)\n",
    "\n",
    "nb.fit(sst_train_labels)\n",
    "\n",
    "sst_train_predicted_y = nb.predict(sst_train_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Direct assessment of the inferred labels against the gold ones\n",
    "\n",
    "Since we have the labels, we can see how we did in reconstructing them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "   negative       0.60      0.62      0.61      3310\n",
      "   positive       0.64      0.62      0.63      3610\n",
      "\n",
      "avg / total       0.62      0.62      0.62      6920\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(sst_train_y, sst_train_predicted_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pretty good! With more labeling functions we could do better. It's tempting to hill-climb on this directly, but that's not especially realistic. However, it does suggest that, when doing data programming, one does well to have labels that are used strictly to improve the labeling functions (which can be run on a much larger dataset to create the training data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Fitting a discriminative model on the noisy labels\n",
    "\n",
    "And now we slip back into the usual SST classifier workflow. As a reminder, `unigrams_phi` gets 0.77 average F1 on the `dev` set when we train on the actual gold labels. Can we approach that performance by writing excellent labeling functions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unigrams_phi(tree):\n",
    "    return Counter(tree.leaves())    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = sst.build_dataset(\n",
    "    sst.train_reader, \n",
    "    phi=unigrams_phi,\n",
    "    class_func=sst.binary_class_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we swap the true labels for the predicted ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "train['y'] = sst_train_predicted_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assess against the `dev` set, which is unchanged – that is, for assessment, we use the gold labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev = sst.build_dataset(\n",
    "    sst.dev_reader,\n",
    "    phi=unigrams_phi,\n",
    "    class_func=sst.binary_class_func,\n",
    "    vectorizer=train['vectorizer'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cheese/disease example, `LogisticRegression` worked best, so we'll continue to use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mod.fit(train['X'], sst_train_predicted_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "snorkel_dev_preds = mod.predict(dev['X'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "   negative       0.60      0.59      0.60       428\n",
      "   positive       0.61      0.62      0.61       444\n",
      "\n",
      "avg / total       0.61      0.61      0.61       872\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(dev['y'], snorkel_dev_preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we might [return to writing more labeling functions](#Other-SST-labeling-function-ideas), in the hope of improving our dev-set results. We got this far with only two simple lexicon-based feature functions, so there is reason to be optimistic that we can train effective models without showing our models any gold labels!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra-credit bake-off\n",
    "\n",
    "This is a fast, optional bake-off intended to be done in class on May 2:\n",
    "    \n",
    "__Question__: How good an F1 score can you get with the function call in [Direct assessment of the inferred labels against the gold ones](#Direct-assessment-of-the-inferred-labels-against-the-gold-ones) above? This just compares the actual gold labels in the train set against the ones you're creating with data programming.\n",
    "\n",
    "__To submit__:\n",
    "\n",
    "1. Your average F1 score from this assessment.\n",
    "1. A description of the labeling functions you wrote to get this score.\n",
    "\n",
    "To get full credit, you just need to write at least one new labeling function and try it out.\n",
    "\n",
    "Submission URL: https://goo.gl/forms/MtyQHoWDHmU5oEyt1\n",
    "\n",
    "The close-time for this is May 2, 11:59 pm."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}