hw_sst.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework and bake-off: Stanford Sentiment Treebank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Fall 2020\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Methodological note](#Methodological-note)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [A softmax baseline](#A-softmax-baseline)\n",
    "1. [RNNClassifier wrapper](#RNNClassifier-wrapper)\n",
    "1. [Error analysis](#Error-analysis)\n",
    "1. [Homework questions](#Homework-questions)\n",
    "  1. [Sentiment words alone [2 points]](#Sentiment-words-alone-[2-points])\n",
    "  1. [A more powerful vector-averaging baseline [2 points]](#A-more-powerful-vector-averaging-baseline-[2-points])\n",
    "  1. [Sentiment shifters [2 points]](#Sentiment-shifters-[2-points])\n",
    "  1. [Your original system [3 points]](#Your-original-system-[3-points])\n",
    "1. [Bake-off [1 point]](#Bake-off-[1-point])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This homework and associated bake-off are devoted to the Stanford Sentiment Treebank (SST). The homework questions ask you to implement some baseline systems and some original feature functions, and the bake-off challenge is to define a system that does extremely well at the SST task.\n",
    "\n",
    "We'll focus on the ternary task as defined by `sst.ternary_class_func` This isn't used in the literature but I think it is the best version of the SST problem for the reasons given [here](sst_01_overview.ipynb#Modeling-the-SST-labels).\n",
    "\n",
    "The SST test set will be used for the bake-off evaluation. This dataset is already publicly distributed, so we are counting on people not to cheat by develping their models on the test set. You must do all your development without using the test set at all, and then evaluate exactly once on the test set and turn in the results, with no further system tuning or additional runs. __Much of the scientific integrity of our field depends on people adhering to this honor code__. \n",
    "\n",
    "Our only additional restriction is that you cannot use any of the subtree labels as input features. You can have your system learn to predict them (as intended), but no feature function can make use of them.\n",
    "\n",
    "One of our goals for this homework and bake-off is to encourage you to engage in __the basic development cycle for supervised models__, in which you\n",
    "\n",
    "1. Write a new feature function. We recommend starting with something simple.\n",
    "1. Use `sst.experiment` to evaluate your new feature function, with at least `fit_softmax_classifier`.\n",
    "1. If you have time, compare your feature function with `unigrams_phi` using `sst.compare_models` or `utils.mcnemar`. (For discussion, see [this notebook section](sst_02_hand_built_features.ipynb#Statistical-comparison-of-classifier-models).)\n",
    "1. Return to step 1, or stop the cycle and conduct a more rigorous evaluation with hyperparameter tuning and assessment on the `dev` set.\n",
    "\n",
    "[Error analysis](#Error-analysis) is one of the most important methods for steadily improving a system, as it facilitates a kind of human-powered hill-climbing on your ultimate objective. Often, it takes a careful human analyst just a few examples to spot a major pattern that can lead to a beneficial change to the feature representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methodological note\n",
    "\n",
    "You don't have to use the experimental framework defined below (based on `sst`). However, if you don't use `sst.experiment` as below, then make sure you're training only on `train`, evaluating on `dev`, and that you report with \n",
    "\n",
    "```\n",
    "from sklearn.metrics import classification_report\n",
    "classification_report(y_dev, predictions)\n",
    "```\n",
    "where `y_dev = [y for tree, y in sst.dev_reader(class_func=sst.ternary_class_func)]`. We'll focus on the value at `macro avg` under `f1-score` in these reports."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up\n",
    "\n",
    "See [the first notebook in this unit](sst_01_overview.ipynb#Set-up) for set-up instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from nltk.tree import Tree\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import random\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import sst\n",
    "import torch.nn as nn\n",
    "from torch_rnn_classifier import TorchRNNClassifier\n",
    "from torch_tree_nn import TorchTreeNN\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SST_HOME = os.path.join('data', 'trees')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A softmax baseline\n",
    "\n",
    "This example is here mainly as a reminder of how to use our experimental framework with linear models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unigrams_phi(tree):\n",
    "    \"\"\"The basis for a unigrams feature function.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    tree : nltk.tree\n",
    "        The tree to represent.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    Counter\n",
    "        A map from strings to their counts in `tree`. (Counter maps a\n",
    "        list to a dict of counts of the elements in that list.)\n",
    "\n",
    "    \"\"\"\n",
    "    return Counter(tree.leaves())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thin wrapper around `LogisticRegression` for the sake of `sst.experiment`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_softmax_classifier(X, y):\n",
    "    mod = LogisticRegression(\n",
    "        fit_intercept=True,\n",
    "        solver='liblinear',\n",
    "        multi_class='ovr')\n",
    "    mod.fit(X, y)\n",
    "    return mod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The experimental run with some notes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax_experiment = sst.experiment(\n",
    "    SST_HOME,\n",
    "    unigrams_phi,                      # Free to write your own!\n",
    "    fit_softmax_classifier,            # Free to write your own!\n",
    "    train_reader=sst.train_reader,     # Fixed by the competition.\n",
    "    assess_reader=sst.dev_reader,      # Fixed until the bake-off.\n",
    "    class_func=sst.ternary_class_func) # Fixed by the bake-off rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`softmax_experiment` contains a lot of information that you can use for analysis; see [this section below](#Error-analysis) for starter code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNNClassifier wrapper\n",
    "\n",
    "This section illustrates how to use `sst.experiment` with `TorchRNNClassifier`. The same basic patterns hold for using `TorchTreeNN`; see [sst_03_neural_networks.ipynb](sst_03_neural_networks.ipynb) for additional discussion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To featurize examples for an RNN, we just get the words in order, letting the model take care of mapping them into an embedding space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rnn_phi(tree):\n",
    "    return tree.leaves()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model wrapper gets the vocabulary using `sst.get_vocab`. If you want to use pretrained word representations in here, then you can have `fit_rnn_classifier` build that space too; see [this notebook section for details](sst_03_neural_networks.ipynb#Pretrained-embeddings). See also [torch_model_base.py](torch_model_base.py) for details on the many optimization parameters that `TorchRNNClassifier` accepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_rnn_classifier(X, y):\n",
    "    sst_glove_vocab = utils.get_vocab(X, mincount=2)\n",
    "    mod = TorchRNNClassifier(\n",
    "        sst_glove_vocab,\n",
    "        early_stopping=True)\n",
    "    mod.fit(X, y)\n",
    "    return mod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_experiment = sst.experiment(\n",
    "    SST_HOME,\n",
    "    rnn_phi,\n",
    "    fit_rnn_classifier,\n",
    "    vectorize=False,  # For deep learning, use `vectorize=False`.\n",
    "    assess_reader=sst.dev_reader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error analysis\n",
    "\n",
    "This section begins to build an error-analysis framework using the dicts returned by `sst.experiment`. These have the following structure:\n",
    "\n",
    "```\n",
    "'model': trained model\n",
    "'phi': the feature function used\n",
    "'train_dataset':\n",
    "   'X': feature matrix\n",
    "   'y': list of labels\n",
    "   'vectorizer': DictVectorizer,\n",
    "   'raw_examples': list of raw inputs, before featurizing   \n",
    "'assess_dataset': same structure as the value of 'train_dataset'\n",
    "'predictions': predictions on the assessment data\n",
    "'metric': `score_func.__name__`, where `score_func` is an `sst.experiment` argument\n",
    "'score': the `score_func` score on the assessment data\n",
    "```\n",
    "The following function just finds mistakes, and returns a `pd.DataFrame` for easy subsequent processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_errors(experiment):\n",
    "    \"\"\"Find mistaken predictions.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    experiment : dict\n",
    "        As returned by `sst.experiment`.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    pd.DataFrame\n",
    "\n",
    "    \"\"\"\n",
    "    raw_examples = experiment['assess_dataset']['raw_examples']\n",
    "    raw_examples = [\" \".join(tree.leaves()) for tree in raw_examples]\n",
    "    df = pd.DataFrame({\n",
    "        'raw_examples': raw_examples,\n",
    "        'predicted': experiment['predictions'],\n",
    "        'gold': experiment['assess_dataset']['y']})\n",
    "    df['correct'] = df['predicted'] == df['gold']\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax_analysis = find_errors(softmax_experiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_analysis = find_errors(rnn_experiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we merge the sotmax and RNN experiments into a single DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis = softmax_analysis.merge(\n",
    "    rnn_analysis, left_on='raw_examples', right_on='raw_examples')\n",
    "\n",
    "analysis = analysis.drop('gold_y', axis=1).rename(columns={'gold_x': 'gold'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code collects a specific subset of examples; small modifications to its structure will give you different interesting subsets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examples where the softmax model is correct, the RNN is not,\n",
    "# and the gold label is 'positive'\n",
    "\n",
    "error_group = analysis[\n",
    "    (analysis['predicted_x'] == analysis['gold'])\n",
    "    &\n",
    "    (analysis['predicted_y'] != analysis['gold'])\n",
    "    &\n",
    "    (analysis['gold'] == 'positive')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "error_group.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ex in error_group['raw_examples'].sample(5):\n",
    "    print(\"=\"*70)\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework questions\n",
    "\n",
    "Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment words alone [2 points]\n",
    "\n",
    "NLTK includes an easy interface to [Minqing Hu and Bing Liu's __Opinion Lexicon__](https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html), which consists of a list of positive words and a list of negative words. How much of the ternary SST story does this lexicon tell?\n",
    "\n",
    "For this problem, submit code to do the following:\n",
    "\n",
    "1. Create a feature function `op_unigrams_phi` on the model of `unigrams_phi` above, but filtering the vocabulary to just items that are members of the Opinion Lexicon. Submit this feature function. You can use `test_op_unigrams_phi` to check your work.\n",
    "\n",
    "1. Evaluate your feature function with `sst.experiment`, with all the same parameters as were used to create `softmax_experiment` in [A softmax baseline](#A-softmax-baseline) above, except of course for the feature function.\n",
    "\n",
    "1. Use `utils.mcnemar` to compare your feature function with the results in `softmax_experiment`. The information you need for this is in `softmax_experiment` and your own `sst.experiment` results. Submit your evaluation code. You can assume `softmax_experiment` is already in memory, but your code should create the other objects necessary for this comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import opinion_lexicon\n",
    "\n",
    "# Use set for fast membership checking:\n",
    "positive = set(opinion_lexicon.positive())\n",
    "negative = set(opinion_lexicon.negative())\n",
    "\n",
    "def op_unigrams_phi(tree):\n",
    "    pass\n",
    "    ##### YOUR PART 1 CODE HERE\n",
    "\n",
    "\n",
    "##### YOUR PART 2 CODE HERE\n",
    "\n",
    "\n",
    "##### YOUR PART 3 CODE HERE\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_op_unigrams_phi(func):\n",
    "    tree = Tree.fromstring(\"\"\"(4 (2 NLU) (4 (2 is) (4 amazing)))\"\"\")\n",
    "    expected = {\"amazing\": 1}\n",
    "    result = func(tree)\n",
    "    assert result == expected, \\\n",
    "        (\"Error for `op_unigrams_phi`: \"\n",
    "         \"Got `{}` which differs from `expected` \"\n",
    "         \"in `test_op_unigrams_phi`\".format(result))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_op_unigrams_phi(op_unigrams_phi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A more powerful vector-averaging baseline [2 points]\n",
    "\n",
    "In [Distributed representations as features](sst_03_neural_networks.ipynb#Distributed-representations-as-features), we looked at a baseline for the ternary SST problem in which each example is modeled as the sum of its GloVe representations. A `LogisticRegression` model was used for prediction. A neural network might do better with these representations, since there might be complex relationships between the input feature dimensions that a linear classifier can't learn. \n",
    "\n",
    "To address this question, we want to get set up to run the experiment with a shallow neural classifier. Thus, your task is to write and submit a model wrapper function around `TorchShallowNeuralClassifier`. This function should implement hyperparameter search according to this specification:\n",
    "\n",
    "* Set `early_stopping=True` for all experiments.\n",
    "* Using 3-fold cross-validation, exhaustively explore this set of hyperparameter combinations:\n",
    "  * The hidden dimensionality at 50, 100, and 200.\n",
    "  * The hidden activation function as `nn.Tanh()` and `nn.ReLU()`.\n",
    "* For all other parameters to `TorchShallowNeuralClassifier`, use the defaults.\n",
    "\n",
    "\n",
    "See [this notebook section](sst_02_hand_built_features.ipynb#Hyperparameter-search) for examples. You are not required to run a full evaluation with this function using `sst.experiment`, but we assume you will want to.\n",
    "\n",
    "We're not evaluating the quality of your model. (We've specified the protocols completely, but there will still be variation in the results.) However, the primary goal of this question is to get you thinking more about this strong baseline feature representation scheme for SST, so we're sort of hoping you feel compelled to try out variations on your own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch_shallow_neural_classifier import TorchShallowNeuralClassifier\n",
    "\n",
    "def fit_shallow_neural_classifier_with_hyperparameter_search(X, y):\n",
    "    pass\n",
    "    ##### YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment shifters [2 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some words have greater power than others to shift sentiment around. Because the SST has sentiment labels on all of its subconstituents, it provides an opportunity to study these shifts in detail. This question takes a first step in that direction by asking you to identify some of these sentiment shifters automatically.\n",
    "\n",
    "More specifically, the task is to identify words that effect a particularly large shift between the value of their sibling node and the value of their mother node. For instance, in the tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = Tree.fromstring(\n",
    "    \"\"\"(1 (2 Astrology) (1 (2 is) (1 (2 not) (4 enlightening))))\"\"\")\n",
    "\n",
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we have the shifter calculations:\n",
    "    \n",
    "* *not*: `1 - 4 = -3`\n",
    "* *enlightening*: `1 - 2 = -1`\n",
    "* *is*: `1 - 1 = 0`\n",
    "* *Astrology*: `1 - 1 = 0`.\n",
    "    \n",
    "__Your task__: write a function `sentiment_shifters` that accepts a `tree` argument and returns a dict mapping words to their list of shifts in `tree`. You can then run `view_top_shifters` to see the results. In addition, you can use `test_sentiment_shifters` to test your function directly. It uses the above example as the basis for the test.\n",
    "\n",
    "__Tips__:\n",
    "\n",
    "* You'll probably want to use `tree.subtrees()` to inspect all of the subtrees in each tree.\n",
    "* `len(tree)` counts the number of children (immediate descendants) of `tree`.\n",
    "* `isinstance(subtree[0][0], str)` will test whether the left daughter of subtree has a lexical child.\n",
    "* `tree.label()` gives the label for any tree or subtree.\n",
    "* Your SST reader should use `replace_root_score=False` so that you keep the root node label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "from operator import itemgetter\n",
    "\n",
    "def sentiment_shifters(tree, diffs=defaultdict(list)):\n",
    "    \"\"\"\n",
    "    Calculates the shifts in `tree`.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    tree : nltk.tree.Tree\n",
    "\n",
    "    diffs: defaultdict(list)\n",
    "        This accumulates the results for `tree`, and `view_top_shifters`\n",
    "        accumulates all these results into a single dict.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    defaultdict mapping words to their list of shifts in `tree`.\n",
    "\n",
    "    \"\"\"\n",
    "    pass\n",
    "    ### YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_sentiment_shifters(func):\n",
    "    \"\"\"func should be `sentiment_shifters`\"\"\"\n",
    "    tree = Tree.fromstring(\n",
    "        \"\"\"(1 (2 Astrology) (1 (2 is) (1 (2 not) (4 enlightening))))\"\"\")\n",
    "    expected = {\"not\": [-3], \"enlightening\": [-1], \"is\": [0], \"Astrology\": [0]}\n",
    "    result = func(tree)\n",
    "    assert result == expected, \\\n",
    "        (\"Error for `sentiment_shifters`: \"\n",
    "         \"Got\\n\\n\\t{}\\n\\nwhich differs from `expected` \"\n",
    "         \"in `test_sentiment_shifters`\".format(result))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_sentiment_shifters(sentiment_shifters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following utility will let you use `sentiment_shifters`. The resulting insights could inform new feature functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def view_top_shifters(top_n=10, mincount=100):\n",
    "    diffs = defaultdict(list)\n",
    "    for tree, label in sst.train_reader(SST_HOME) :\n",
    "        these_diffs = sentiment_shifters(tree, diffs=diffs)\n",
    "    diffs = {key: np.mean(vals) for key, vals in diffs.items()\n",
    "             if len(vals) >= mincount}\n",
    "    diffs = sorted(diffs.items(), key=itemgetter(1))\n",
    "    segs = ((\"Negative\", diffs[:top_n]), (\"Positive\", diffs[-top_n:]))\n",
    "    for label, seg in segs:\n",
    "        print(\"\\nTop {} {} shifters:\\n\".format(top_n, label))\n",
    "        for key, val in seg:\n",
    "            print(key, val)\n",
    "\n",
    "\n",
    "view_top_shifters()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your original system [3 points]\n",
    "\n",
    "Your task is to develop an original model for the SST ternary problem, predicting only the root-level labels. There are many options. If you spend more than a few hours on this homework problem, you should consider letting it grow into your final project! Here are some relatively manageable ideas that you might try:\n",
    "\n",
    "1. We didn't systematically evaluate the `bidirectional` option to the `TorchRNNClassifier`. Similarly, that model could be tweaked to allow multiple LSTM layers (at present there is only one), and you could try adding layers to the classifier portion of the model as well.\n",
    "\n",
    "1. We've already glimpsed the power of rich initial word representations, and later in the course we'll see that smart initialization usually leads to a performance gain in NLP, so you could perhaps achieve a winning entry with a simple model that starts in a great place.\n",
    "\n",
    "1. Our [practical introduction to contextual word representations](contextualreps.ipynb) covers pretrained representations and interfaces that are likely to boost the performance of any system.\n",
    "\n",
    "1. The `TreeNN` and `TorchTreeNN` don't perform all that well, and this could be for the same reason that RNNs don't peform well: the gradient signal doesn't propagate reliably down inside very deep trees. [Tai et al. 2015](https://www.aclweb.org/anthology/P15-1150/) sought to address this with TreeLSTMs, which are fairly easy to implement in PyTorch.\n",
    "\n",
    "We want to emphasize that this needs to be an __original__ system. It doesn't suffice to download code from the Web, retrain, and submit. You can build on others' code, but you have to do something new and meaningful with it.\n",
    "\n",
    "In the cell below, please provide a brief technical description of your original system, so that the teaching team can gain an understanding of what it does. This will help us to understand your code and analyze all the submissions to identify patterns and strategies.  We also ask that you report the best score your system got during development, just to help us understand how systems performed overall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PLEASE MAKE SURE TO INCLUDE THE FOLLOWING BETWEEN THE START AND STOP COMMENTS:\n",
    "#   1) Textual description of your system.\n",
    "#   2) The code for your original system.\n",
    "#   3) The score achieved by your system in place of MY_NUMBER.\n",
    "#        With no other changes to that line.\n",
    "#        You should report your score as a decimal value <=1.0\n",
    "# PLEASE MAKE SURE NOT TO DELETE OR EDIT THE START AND STOP COMMENTS\n",
    "\n",
    "# NOTE: MODULES, CODE AND DATASETS REQUIRED FOR YOUR ORIGINAL SYSTEM \n",
    "# SHOULD BE ADDED BELOW THE 'IS_GRADESCOPE_ENV' CHECK CONDITION. DOING\n",
    "# SO ABOVE THE CHECK MAY CAUSE THE AUTOGRADER TO FAIL.\n",
    "\n",
    "# START COMMENT: Enter your system description in this cell.\n",
    "# My peak score was: MY_NUMBER\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    pass\n",
    "\n",
    "# STOP COMMENT: Please do not remove this comment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bake-off [1 point]\n",
    "\n",
    "As we said above, the bake-off evaluation data is the official SST test-set release. For this bake-off, you'll evaluate your original system from the above homework problem on the test set, using the ternary class problem. Rules:\n",
    "\n",
    "1. Only one evaluation is permitted.\n",
    "1. No additional system tuning is permitted once the bake-off has started.\n",
    "\n",
    "The cells below this one constitute your bake-off entry.\n",
    "\n",
    "Systems that enter will receive the additional homework point, and systems that achieve the top score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.\n",
    "\n",
    "Late entries will be accepted, but they cannot earn the extra 0.5 points. Similarly, you cannot win the bake-off unless your homework is submitted on time.\n",
    "\n",
    "The announcement will include the details on where to submit your entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your bake-off assessment code in this cell.\n",
    "# Place your code in the scope of the 'IS_GRADESCOPE_ENV'\n",
    "# conditional.\n",
    "# Please do not remove this comment.\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    pass\n",
    "    # Please enter your code in the scope of the above conditional.\n",
    "    ##### YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On an otherwise blank line in this cell, please enter\n",
    "# your macro-average F1 value as reported by the code above.\n",
    "# Please enter only a number between 0 and 1 inclusive.\n",
    "# Please do not remove this comment.\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    pass\n",
    "    # Please enter your score in the scope of the above conditional.\n",
    "    ##### YOUR CODE HERE\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}