hw2_sst.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 2: Stanford Sentiment Treebank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2019\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Methodological note](#Methodological-note)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [A softmax baseline](#A-softmax-baseline)\n",
    "1. [RNNClassifier wrapper](#RNNClassifier-wrapper)\n",
    "1. [Error analysis](#Error-analysis)\n",
    "1. [Homework questions](#Homework-questions)\n",
    "  1. [Sentiment words alone [2 points]](#Sentiment-words-alone-[2-points])\n",
    "  1. [A more powerful vector-summing baseline [3 points]](#A-more-powerful-vector-summing-baseline-[3-points])\n",
    "  1. [Your original system [4 points]](#Your-original-system-[4-points])\n",
    "1. [Bake-off [1 point]](#Bake-off-[1-point])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This homework and associated bake-off are devoted to the Stanford Sentiment Treebank (SST). The homework questions ask you to implement some baseline systems, and the bake-off challenge is to define a system that does extremely well at the SST task.\n",
    "\n",
    "We'll focus on the ternary task as defined by `sst.ternary_class_func`.\n",
    "\n",
    "The SST test set will be used for the bake-off evaluation. This dataset is already publicly distributed, so we are counting on people not to cheat by develping their models on the test set. You must do all your development without using the test set at all, and then evaluate exactly once on the test set and turn in the results, with no further system tuning or additional runs. __Much of the scientific integrity of our field depends on people adhering to this honor code__. \n",
    "\n",
    "Our only additional restriction is that __you cannot make any use of the subtree labels__. This corresponds to the 'Root' condition in the paper. As we discussed in class, the subtree labels are a really interesting feature of SST, but bringing them in results in a substantially different learning problem.\n",
    "\n",
    "One of our goals for this homework and bake-off is to encourage you to engage in __the basic development cycle for supervised models__, in which you\n",
    "\n",
    "1. Write a new feature function. We recommend starting with something simple.\n",
    "1. Use `sst.experiment` to evaluate your new feature function, with at least `fit_softmax_classifier`.\n",
    "1. If you have time, compare your feature function with `unigrams_phi` using `sst.compare_models` or `sst.compare_models_mcnemar`. (For discussion, see [this notebook section](sst_02_hand_built_features.ipynb#Statistical-comparison-of-classifier-models).)\n",
    "1. Return to step 1, or stop the cycle and conduct a more rigorous evaluation with hyperparameter tuning and assessment on the `dev` set.\n",
    "\n",
    "[Error analysis](#Error-analysis) is one of the most important methods for steadily improving a system, as it facilitates a kind of human-powered hill-climbing on your ultimate objective. Often, it takes a careful human analyst just a few examples to spot a major pattern that can lead to a beneficial change to the feature representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methodological note\n",
    "\n",
    "You don't have to use the experimental framework defined below (based on `sst`). However, if you don't use `sst.experiment` as below, then make sure you're training only on `train`, evaluating on `dev`, and that you report with \n",
    "\n",
    "```\n",
    "from sklearn.metrics import classification_report\n",
    "classification_report(y_dev, predictions)\n",
    "```\n",
    "where `y_dev = [y for tree, y in sst.dev_reader(class_func=sst.ternary_class_func)]`. We'll focus on the value at `macro avg` under `f1-score` in these reports."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up\n",
    "\n",
    "See [the first notebook in this unit](sst_01_overview.ipynb#Set-up) for set-up instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import random\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import sst\n",
    "import torch.nn as nn\n",
    "from torch_rnn_classifier import TorchRNNClassifier\n",
    "from torch_tree_nn import TorchTreeNN\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "SST_HOME = os.path.join('data', 'trees')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A softmax baseline\n",
    "\n",
    "This example is here mainly as a reminder of how to use our experimental framework with linear models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unigrams_phi(tree):\n",
    "    \"\"\"The basis for a unigrams feature function.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    tree : nltk.tree\n",
    "        The tree to represent.\n",
    "    \n",
    "    Returns\n",
    "    -------    \n",
    "    Counter\n",
    "        A map from strings to their counts in `tree`. (Counter maps a \n",
    "        list to a dict of counts of the elements in that list.)\n",
    "    \n",
    "    \"\"\"\n",
    "    return Counter(tree.leaves())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thin wrapper around `LogisticRegression` for the sake of `sst.experiment`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_softmax_classifier(X, y):        \n",
    "    mod = LogisticRegression(\n",
    "        fit_intercept=True,\n",
    "        solver='liblinear',\n",
    "        multi_class='ovr')\n",
    "    mod.fit(X, y)\n",
    "    return mod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The experimental run with some notes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.628     0.689     0.657       428\n",
      "     neutral      0.343     0.153     0.211       229\n",
      "    positive      0.629     0.750     0.684       444\n",
      "\n",
      "   micro avg      0.602     0.602     0.602      1101\n",
      "   macro avg      0.533     0.531     0.518      1101\n",
      "weighted avg      0.569     0.602     0.575      1101\n",
      "\n"
     ]
    }
   ],
   "source": [
    "softmax_experiment = sst.experiment(\n",
    "    SST_HOME,\n",
    "    unigrams_phi,                      # Free to write your own!\n",
    "    fit_softmax_classifier,            # Free to write your own!\n",
    "    train_reader=sst.train_reader,     # Fixed by the competition.\n",
    "    assess_reader=sst.dev_reader,      # Fixed until the bake-off.\n",
    "    class_func=sst.ternary_class_func) # Fixed by the bake-off rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`softmax_experiment` contains a lot of information that you can use for analysis; see [this section below](#Error-analysis) for starter code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNNClassifier wrapper\n",
    "\n",
    "This section illustrates how to use `sst.experiment` with RNN and TreeNN models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To featurize examples for an RNN, we just get the words in order, letting the model take care of mapping them into an embedding space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rnn_phi(tree):\n",
    "    return tree.leaves()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model wrapper gets the vocabulary using `sst.get_vocab`. If you want to use pretrained word representations in here, then you can have `fit_rnn_classifier` build that space too; see [this notebook section for details](sst_03_neural_networks.ipynb#Pretrained-embeddings)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_rnn_classifier(X, y):    \n",
    "    sst_glove_vocab = utils.get_vocab(X, n_words=10000)     \n",
    "    mod = TorchRNNClassifier(\n",
    "        sst_glove_vocab, \n",
    "        eta=0.05,\n",
    "        embedding=None,\n",
    "        batch_size=1000,\n",
    "        embed_dim=50,\n",
    "        hidden_dim=50,\n",
    "        max_iter=50,\n",
    "        l2_strength=0.001,\n",
    "        bidirectional=True,\n",
    "        hidden_activation=nn.ReLU())\n",
    "    mod.fit(X, y)\n",
    "    return mod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finished epoch 50 of 50; error is 2.3075825870037083"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.581     0.734     0.649       428\n",
      "     neutral      0.310     0.192     0.237       229\n",
      "    positive      0.656     0.619     0.637       444\n",
      "\n",
      "   micro avg      0.575     0.575     0.575      1101\n",
      "   macro avg      0.516     0.515     0.508      1101\n",
      "weighted avg      0.555     0.575     0.559      1101\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rnn_experiment = sst.experiment(\n",
    "    SST_HOME,\n",
    "    rnn_phi,\n",
    "    fit_rnn_classifier, \n",
    "    vectorize=False,  # For deep learning, use `vectorize=False`.\n",
    "    assess_reader=sst.dev_reader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error analysis\n",
    "\n",
    "This section begins to build an error-analysis framework using the dicts returned by `sst.experiment`. These have the following structure:\n",
    "\n",
    "```\n",
    "'model': trained model\n",
    "'train_dataset':\n",
    "   'X': feature matrix\n",
    "   'y': list of labels\n",
    "   'vectorizer': DictVectorizer,\n",
    "   'raw_examples': list of raw inputs, before featurizing   \n",
    "'assess_dataset': same structure as the value of 'train_dataset'\n",
    "'predictions': predictions on the assessment data\n",
    "'metric': `score_func.__name__`, where `score_func` is an `sst.experiment` argument\n",
    "'score': the `score_func` score on the assessment data\n",
    "```\n",
    "The following function just finds mistakes, and returns a `pd.DataFrame` for easy subsequent processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_errors(experiment):\n",
    "    \"\"\"Find mistaken predictions.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    experiment : dict\n",
    "        As returned by `sst.experiment`.\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    pd.DataFrame\n",
    "    \n",
    "    \"\"\"\n",
    "    raw_examples = experiment['assess_dataset']['raw_examples']\n",
    "    raw_examples = [\" \".join(tree.leaves()) for tree in raw_examples]\n",
    "    df = pd.DataFrame({\n",
    "        'raw_examples': raw_examples,\n",
    "        'predicted': experiment['predictions'],\n",
    "        'gold': experiment['assess_dataset']['y']})\n",
    "    df['correct'] = df['predicted'] == df['gold']\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax_analysis = find_errors(softmax_experiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_analysis = find_errors(rnn_experiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we merge the sotmax and RNN experiments into a single DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis = softmax_analysis.merge(\n",
    "    rnn_analysis, left_on='raw_examples', right_on='raw_examples')\n",
    "\n",
    "analysis = analysis.drop('gold_y', axis=1).rename(columns={'gold_x': 'gold'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code collects a specific subset of examples; small modifications to its structure will give you different interesting subsets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examples where the softmax model is correct, the RNN is not,\n",
    "# and the gold label is 'positive'\n",
    "\n",
    "error_group = analysis[\n",
    "    (analysis['predicted_x'] == analysis['gold'])\n",
    "    &\n",
    "    (analysis['predicted_y'] != analysis['gold'])    \n",
    "    &\n",
    "    (analysis['gold'] == 'positive')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "77"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "error_group.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================\n",
      "An operatic , sprawling picture that 's entertainingly acted , magnificently shot and gripping enough to sustain most of its 170-minute length .\n",
      "======================================================================\n",
      "It all adds up to good fun .\n",
      "======================================================================\n",
      "A fitfully amusing romp that , if nothing else , will appeal to fans of Malcolm in the Middle and its pubescent star , Frankie Muniz .\n",
      "======================================================================\n",
      "Trademark American triteness and simplicity are tossed out the window with the intelligent French drama that deftly explores the difficult relationship between a father and son .\n",
      "======================================================================\n",
      "The film 's welcome breeziness and some unbelievably hilarious moments -- most portraying the idiocy of the film industry -- make it mostly worth the trip .\n"
     ]
    }
   ],
   "source": [
    "for ex in error_group['raw_examples'].sample(5):\n",
    "    print(\"=\"*70)\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework questions\n",
    "\n",
    "Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment words alone [2 points]\n",
    "\n",
    "NLTK includes an easy interface to [Minqing Hu and Bing Liu's __Opinion Lexicon__](https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html), which consists of a list of positive words and a list of negative words. How much of the ternary SST story does this lexicon tell?\n",
    "\n",
    "For this problem, submit code to do the following:\n",
    "\n",
    "1. Create a feature function `op_unigrams` on the model of `unigrams_phi` above, but filtering the vocabulary to just items that are members of the Opinion Lexicon. Submit this feature function.\n",
    "\n",
    "1. Evaluate your feature function with `sst.experiment`, with all the same parameters as were used to create `softmax_experiment` in [A softmax baseline](#A-softmax-baseline) above, except of course for the feature function.\n",
    "\n",
    "1. Use `utils.mcnemar` to compare your feature function with the results in `softmax_experiment`. The information you need for this is in `softmax_experiment` and your own `sst.experiment` results. Submit your evaluation code. You can assume `softmax_experiment` is already in memory, but your code should create the other objects necessary for this comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import opinion_lexicon\n",
    "\n",
    "# Use set for fast membership checking:\n",
    "positive = set(opinion_lexicon.positive())\n",
    "negative = set(opinion_lexicon.negative())\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A more powerful vector-summing baseline [3 points]\n",
    "\n",
    "In [Distributed representations as features](sst_03_neural_networks.ipynb#Distributed-representations-as-features), we looked at a baseline for the ternary SST problem in which each example is modeled as the sum of its 50-dimensional GloVe representations. A `LogisticRegression` model was used for prediction. A neural network might do better with these representations, since there might be complex relationships between the input feature dimensions that a linear classifier can't learn. \n",
    "\n",
    "To address this question, rerun the experiment with `TorchShallowNeuralClassifier` as the classifier. Specs:\n",
    "* Use `sst.experiment` to conduct the experiment. \n",
    "* Using 3-fold cross-validation, exhaustively explore this set of hyperparameter combinations:\n",
    "  * The hidden dimensionality at 50, 100, and 200.\n",
    "  * The hidden activation function as `nn.Tanh` or `nn.ReLU`.\n",
    "* (For all other parameters to `TorchShallowNeuralClassifier`, use the defaults.)\n",
    "\n",
    "For this problem, submit code to do the following:\n",
    "\n",
    "1. Your model wrapper function around `TorchShallowNeuralClassifier`. This function should implement the requisite cross-validation; see [this notebook section](sst_02_hand_built_features.ipynb#Hyperparameter-search) for examples.\n",
    "1. The classification report as printed by `sst.experiment`. (This will print out when you run `sst.experiment`. That print-out suffices.)\n",
    "2. The optimal hyperparameters chosen in your experiment. (This too will print out when you run `sst.experiment`. The print-out again suffices.)\n",
    "\n",
    "We're not evaluating the quality of your model. (We've specified the protocols completely, but there will still be variation in the results.) However, the primary goal of this question is to get you thinking more about this strikingly good baseline feature representation scheme for SST, so we're sort of hoping you feel compelled to try out variations on your own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your original system [4 points]\n",
    "\n",
    "Your task is to develop an original model for the SST ternary problem, using only the root-level labels (again, __you cannot make any use of the subtree labels__). There are many options. If you spend more than a few hours on this homework problem, you should consider letting it grow into your final project! Here are some relatively manageable ideas that you might try:\n",
    "\n",
    "1. We didn't systematically evaluate the `bidirectional` option to the `TorchRNNClassifier`. Similarly, that model could be tweaked to allow multiple LSTM layers (at present there is only one), and you could try adding layers to the classifier portion of the model as well.\n",
    "\n",
    "1. We've already glimpsed the power of rich initial word representations, and later in the course we'll see that smart initialization usually leads to a performance gain in NLP, so you could perhaps achieve a winning entry with a simple model that starts in a great place.\n",
    "\n",
    "1. The [practical introduction to contextual word representations](contextualreps.ipynb) (to be discussed later in the quarter) covers pretrained representations and interfaces that are likely to boost the performance of any system.\n",
    "\n",
    "1. The `TreeNN` and `TorchTreeNN` don't perform all that well, and this could be for the same reason that RNNs don't peform well: the gradient signal doesn't propagate reliably down inside very deep trees. [Tai et al. 2015](https://aclanthology.info/papers/P15-1150/p15-1150) sought to address this with TreeLSTMs, which are fairly easy to implement in PyTorch.\n",
    "\n",
    "1. In the [distributed representations as features](#Distributed-representations-as-features) section, we just summed  all of the leaf-node GloVe vectors to obtain a fixed-dimensional representation for all sentences. This ignores all of the tree structure. See if you can do better by paying attention to the binary tree structure: write a function `glove_subtree_phi` that obtains a vector representation for each subtree by combining the vectors of its daughters, with the leaf nodes again given by GloVe (any dimension you like) and the full representation of the sentence given by the final vector obtained by this recursive process. You can decide on how you combine the vectors. \n",
    "\n",
    "1. If you have a lot of computing resources, then you can fire off a large hyperparameter search over many parameter values. All the model classes for this course are compatible with the `scikit-learn` and [scikit-optimize](https://scikit-optimize.github.io) methods, because they define the required functions for getting and setting parameters.\n",
    "\n",
    "We want to emphasize that this needs to be an __original__ system. It doesn't suffice to download code from the Web, retrain, and submit. You can build on others' code, but you have to do something new and meaningful with it.\n",
    "\n",
    "__Please include a brief prose description of your system along with your code, to help the teaching team understand the structure of your system.__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bake-off [1 point]\n",
    "\n",
    "The bake-off will begin on April 22. The announcement will go out on Piazza. As we said above, the bake-off evaluation data is the official SST test set release. For this bake-off, you'll evaluate your original system from the above homework problem on the test set, using the ternary class problem. Rules:\n",
    "\n",
    "1. Only one evaluation is permitted.\n",
    "1. No additional system tuning is permitted once the bake-off has started.\n",
    "1. As noted above, __you cannot make any use of the subtree labels__.\n",
    "\n",
    "To enter the bake-off, upload this notebook on Canvas:\n",
    "\n",
    "https://canvas.stanford.edu/courses/99711/assignments/187246\n",
    "\n",
    "The cells below this one constitute your bake-off entry.\n",
    "\n",
    "Systems that enter will receive the additional homework point, and systems that achieve the top score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.\n",
    "\n",
    "The bake-off will close at 4:30 pm on April 24. Late entries will be accepted, but they cannot earn the extra 0.5 points. Similarly, you cannot win the bake-off unless your homework is submitted on time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your bake-off assessment code in this cell. \n",
    "# Please do not remove this comment.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On an otherwise blank line in this cell, please enter\n",
    "# your macro-average F1 value as reported by the code above. \n",
    "# Please enter only a number between 0 and 1 inclusive.\n",
    "# Please do not remove this comment.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}