sst_01_overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Supervised sentiment: overview of the Stanford Sentiment Treebank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2022\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview of this unit](#Overview-of-this-unit)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [Data readers](#Data-readers)\n",
    "    1. [Train split](#Train-split)\n",
    "    1. [Dev and test splits](#Dev-and-test-splits)\n",
    "1. [Tokenization](#Tokenization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Overview of this unit\n",
    "\n",
    "We have a few inter-related goals for this unit:\n",
    "\n",
    "* Provide a basic introduction to supervised learning in the context of a problem that has long been central to academic research and industry applications: __sentiment analysis__.\n",
    "\n",
    "* Explore and evaluate a diverse array of methods for modeling sentiment:\n",
    "  * Hand-built feature functions with (mostly linear) classifiers\n",
    "  * Dense feature representations derived from VSMs as we built them in the previous unit\n",
    "  * Recurrent neural networks (RNNs)\n",
    "  \n",
    "* Begin discussing and implementing responsible methods for __hyperparameter optimization__ and __classifier assessment and comparison__.\n",
    "\n",
    "The unit is built around the [Stanford Sentiment Treebank (SST)](http://nlp.stanford.edu/sentiment/), a widely-used resource for evaluating supervised NLU models, and one that provides rich linguistic representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Set-up\n",
    "\n",
    "*  Make sure your environment includes all the requirements for [the cs224u repository](https://github.com/cgpotts/cs224u).\n",
    "\n",
    "* If you haven't already, download [the course data](http://web.stanford.edu/class/cs224u/data/data.tgz), unpack it, and place it in the directory containing the course repository – the same directory as this notebook. (If you want to put it somewhere else, change `SST_HOME` below.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize.treebank import TreebankWordDetokenizer\n",
    "from nltk.tokenize.treebank import TreebankWordTokenizer\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "import sst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "SST_HOME = os.path.join('data', 'sentiment')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data readers\n",
    "\n",
    "Our SST distribution is the ternary version of the problem (SST-3). It consists of train/dev/test files with the following columns:\n",
    "\n",
    "1. `example_id`: a string with the format 'N-S' where N is the example number and S is the index for the subtree in example N. Both N and S are five-digit numbers with 0-padding.\n",
    "2. `sentence`: a string giving the example sentence.\n",
    "3. `label`: a string giving the label: `'positive'`, `'negative'`, or `'neutral'`. This value is derived from the original SST by mapping labels 0 and 1 to `'negative'`, label 2 to `'neutral'`, and labels 3 and 4 to `'positive'`.\n",
    "4. `is_subtree`: the integer `1` if the example is a (proper) subtree, else `0`. This affects only the train file. Our dev and test splits contain no subtrees – full examples only – and hence `is_subtree` is always `0` for them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When reading in the train split, you have a few options. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Root-only formulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default will include only full examples and retain duplicate examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = sst.train_reader(SST_HOME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'example_id': '04162-00001',\n",
       "  'sentence': \"One can only assume that the jury who bestowed star Hoffman 's brother Gordy with the Waldo Salt Screenwriting award at 2002 's Sundance Festival were honoring an attempt to do something different over actually pulling it off\",\n",
       "  'label': 'negative',\n",
       "  'is_subtree': 0},\n",
       " {'example_id': '05626-00001',\n",
       "  'sentence': \"Comedy troupe Broken Lizard 's first movie is very funny but too concerned with giving us a plot .\",\n",
       "  'label': 'neutral',\n",
       "  'is_subtree': 0},\n",
       " {'example_id': '02940-00001',\n",
       "  'sentence': \"Imagine O. Henry 's The Gift of the Magi relocated to the scuzzy underbelly of NYC 's drug scene .\",\n",
       "  'label': 'negative',\n",
       "  'is_subtree': 0}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.sample(3, random_state=1).to_dict(orient=\"records\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8544"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This yields the following label distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "positive    3610\n",
       "negative    3310\n",
       "neutral     1624\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might want to remove the duplicate examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dup_train_df = sst.train_reader(SST_HOME, dedup=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8534"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dup_train_df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This removes only ten examples for this setting so it is unlikely to be a significant choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our CSV-based distribution should make it easy to do basic analysis of the dataset to inform system development. \n",
    "\n",
    "Here's a look at the distribution of examples by length in characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAD4CAYAAAAdIcpQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbs0lEQVR4nO3df7DV9X3n8ecrJEEUUVziHQJsIBk0RUn8cUPtppu51GxEjWKduEsWG0zZZSdD1GTJNFB3xkx3mVI7pGNqtUOiK1brDbU2sjGmIWzO2E4lBKx6RUrFQCxCYFN/cdUQr3nvH9/vbU9Ozr3f7733fM+P73k9Zu7c8/2cz/l+3+/5Hu6b76/PRxGBmZnZaN7W6gDMzKz9uViYmVkmFwszM8vkYmFmZplcLMzMLNPbWx1AUWbMmBFz587N1fe1117jlFNOKTagFihrXlDe3MqaF5Q3t7LltXv37p9ExLtq20tbLObOncuuXbty9a1UKvT19RUbUAuUNS8ob25lzQvKm1vZ8pL0o3rtPg1lZmaZXCzMzCyTi4WZmWVysTAzs0wuFmZmlsnFwszMMrlYmJlZJhcLMzPL5GJhZmaZSvsEt3WGuWsfHtfn1iwc4rpxfnbYwQ2XT+jzZt3ERxZmZpbJxcLMzDK5WJiZWSYXCzMzy+RiYWZmmVwszMwsk4uFmZllKqxYSLpL0jFJT9e0Xy9pn6Q9km6pal8naX/63iVV7RdKGkjf+4okFRWzmZnVV+RDeXcDtwH3DDdIWgwsBT4QEScknZm2LwCWAecA7wa+K+msiHgLuANYBewAvgUsAR4pMO6uNN6H48ysOxR2ZBERjwIv1jR/BtgQESfSPsfS9qVAf0SciIgDwH5gkaSZwLSIeCwigqTwXFVUzGZmVl+zh/s4C/j3ktYDPwW+EBE/AGaRHDkMO5S2vZm+rm2vS9IqkqMQenp6qFQquYIaHBzM3beTjCWvNQuHig2mwXqmTDzmdtznZf0uQnlzK2tetZpdLN4OTAcuAj4EbJH0XqDedYgYpb2uiNgEbALo7e2Nvr6+XEFVKhXy9u0kY8lrouMsNduahUNsHJjY1/fg8r7GBNNAZf0uQnlzK2tetZp9N9Qh4MFI7AR+DsxI2+dU9ZsNHE7bZ9dpNzOzJmp2sfgG8BsAks4C3gn8BNgKLJM0WdI8YD6wMyKOAMclXZTeBfUp4KEmx2xm1vUKOw0l6X6gD5gh6RBwM3AXcFd6O+3PgBXphes9krYAzwBDwOr0TihILorfDUwhuQvKd0KZmTVZYcUiIj45wlvXjtB/PbC+Tvsu4NwGhmZmZmPkJ7jNzCyTi4WZmWVysTAzs0wuFmZmlsnFwszMMrlYmJlZJhcLMzPL5GJhZmaZXCzMzCyTi4WZmWVysTAzs0wuFmZmlsnFwszMMrlYmJlZJhcLMzPLVFixkHSXpGPpREe1731BUkiaUdW2TtJ+SfskXVLVfqGkgfS9r6Qz5pmZWRMVeWRxN7CktlHSHOA/AM9XtS0AlgHnpJ+5XdKk9O07gFUkU63Or7dOMzMrVmHFIiIeBV6s89YfAb8DRFXbUqA/Ik5ExAFgP7BI0kxgWkQ8lk6/eg9wVVExm5lZfYVNq1qPpCuBFyLiyZqzSbOAHVXLh9K2N9PXte0jrX8VyVEIPT09VCqVXHENDg7m7ttJxpLXmoVDxQbTYD1TJh5zO+7zsn4Xoby5lTWvWk0rFpJOBm4CPlbv7TptMUp7XRGxCdgE0NvbG319fbliq1Qq5O3bScaS13VrHy42mAZbs3CIjQMT+/oeXN7XmGAaqKzfRShvbmXNq1YzjyzeB8wDho8qZgOPS1pEcsQwp6rvbOBw2j67TnspzW3wH+w1C4c6rgiYWXtq2q2zETEQEWdGxNyImEtSCC6IiB8DW4FlkiZLmkdyIXtnRBwBjku6KL0L6lPAQ82K2czMEoUdWUi6H+gDZkg6BNwcEXfW6xsReyRtAZ4BhoDVEfFW+vZnSO6smgI8kv6YTVijj+TyOrjh8pZs12wiCisWEfHJjPfn1iyvB9bX6bcLOLehwZmZ2Zj4CW4zM8vkYmFmZplcLMzMLJOLhZmZZXKxMDOzTC4WZmaWycXCzMwyZRYLSR+WdEr6+lpJX5b0nuJDMzOzdpHnyOIO4HVJHyQZWvxHJEOFm5lZl8hTLIbSuSSWArdGxK3AqcWGZWZm7STPcB/HJa0DrgU+ks5g945iwzIzs3aS58jiPwEngJXpCLGzgD8sNCozM2srox5ZpEcR90bER4fbIuJ5fM3CzKyrjHpkkQ4T/rqk05oUj5mZtaE81yx+CgxI2ga8NtwYETcUFpWZmbWVPMXi4fTHzMy6VGaxiIjNkqYA/zYi9uVdsaS7gI8DxyLi3LTtD4ErgJ8BzwGfjoiX0/fWASuBt4AbIuKv0/YL+deZ8r4F3JjeymtmZk2S5wnuK4AngG+ny+dJ2ppj3XcDS2ratgHnRsQHgH8E1qXrXAAsA85JP3N7enEdkocCV5HMyz2/zjrNzKxgeW6d/RKwCHgZICKeAOZlfSgiHgVerGn7TkQMpYs7gNnp66VAf0SciIgDwH5gkaSZwLSIeCw9mrgHuCpHzGZm1kB5rlkMRcQrkqrbGnEa6LeBr6evZ5EUj2GH0rY309e17XVJWkVyFEJPTw+VSiVXIIODg7n7FmnNwqHsTmPQM6Xx62wXnZzbaN+1dvkuFqGsuZU1r1p5isXTkv4zMEnSfOAG4O8mslFJNwFDwH3DTXW6xSjtdUXEJmATQG9vb/T19eWKp1KpkLdvka5b29j7CNYsHGLjQJ5d3Hk6ObeDy/tGfK9dvotFKGtuZc2rVp7TUNeTXEs4Afw58Apw43g3KGkFyYXv5VUXqg8Bc6q6zQYOp+2z67SbmVkT5SkWl0fETRHxofTnfwBXjmdjkpYAXwSujIjXq97aCiyTNFnSPJIL2Tsj4gjJ2FQXKTkP9ingofFs28zMxi9PsViXs+0XSLofeAw4W9IhSSuB20hGrN0m6QlJfwoQEXuALcAzJHddrU6fHgf4DPA1kovezwGP5IjZzMwaaMSTvpIuBS4DZkn6StVb00iuN4wqIj5Zp/nOUfqvB9bXad8FnJu1PTMzK85oVwgPA7tITjntrmo/Dny+yKDMzKy9jFgsIuJJ4ElJfwW8NnxaKH1YbnKT4jMzszaQ55rFd0iG2hg2BfhuMeGYmVk7ylMsToqIweGF9PXJxYVkZmbtJk+xeE3SBcML6cB+bxQXkpmZtZs8j8B+DvgLScMPw80kmWrVzMy6RJ4hyn8g6f3A2STDb/xDRLxZeGRmZtY28g6uczawADgJOF8SEeF5uM3MukRmsZB0M9BHUiy+BVwK/C3JcOFmZtYF8lzg/gRwMfDjiPg08EH8nIWZWVfJUyzeiIifA0OSpgHHgPcWG5aZmbWTPNcsdkk6HfgqybAfg8DOIoMyM7P2MmqxSIcF//2IeBn4U0nfJpnm9KlmBGdmZu1h1NNQ6eRE36haPuhCYWbWffJcs9gh6UOFR2JmZm0rT7FYDDwm6TlJT0kakJR5dCHpLknHJD1d1XaGpG2Snk1/T696b52k/ZL2Sbqkqv3CdJv7JX0lPTVmZmZNlKdYXAq8D/gN4AqS+bOvyPG5u4ElNW1rge0RMR/Yni4jaQGwjGSu7yXA7elQ6AB3AKtIplqdX2edZmZWsMxiERE/iogfkQweGFU/WZ97FHixpnkpsDl9vRm4qqq9PyJORMQBkilUF0maSXJB/bH0+sk9VZ8xM7MmyfME95XARuDdJM9YvAfYS3IUMFY9EXEEICKOSDozbZ8F7KjqdyhtezN9Xds+UqyrSI5C6OnpoVKp5ApqcHAwd98irVmYOVvtmPRMafw620Un5zbad61dvotFKGtuZc2rVp7nLP4ncBHw3Yg4X9JioN782hNR7zpEjNJeV0RsAjYB9Pb2Rl9fX66NVyoV8vYt0nVrH27o+tYsHGLjQN7hvzpLJ+d2cHnfiO+1y3exCGXNrax51cpzzeLNiPhn4G2S3hYR3wPOG+f2jqanlkh/H0vbDwFzqvrNJpkD/FD6urbdzMyaKE+xeFnSVOBR4D5JtwLjPf7fCqxIX68AHqpqXyZpsqR5JBeyd6anrI5Luii9C+pTVZ8xM7MmyXMcvxT4KfB5YDlwGvB7WR+SdD/JaLUzJB0CbgY2AFskrQSeB64BiIg9krYAz5AUotUR8Va6qs+Q3Fk1BXgk/THrWHNHOd24ZuFQw09HVju44fLC1m3llmfyo9eqFjeP2PGXPzfSdY2LR+i/Hlhfp30XcG7e7ZqZWeNlnoaSdHX6EN0rkl6VdFzSq80IzszM2kOe01C3AFdExN6igzEzs/aU5wL3URcKM7PuNuKRhaSr05e7JH2dZPTZE8PvR8SDxYZmZmbtYrTTUNXjP70OfKxqOQAXCzOzLjFisUjn2zYzM8t1N9TmdFrV4eXpku4qNCozM2sreS5wfyCdVhWAiHgJOL+wiMzMrO3kKRZvq5mk6Azy3XJrZmYlkeeP/kbg7yQ9QHJh+z9S50lrMzMrrzzDfdwjaRfJTHkCro6IZwqPzMzM2kau00lpcXCBMDPrUnmuWZiZWZdzsTAzs0wuFmZmlslDlJuZWaY8Rxa3AFdGxGkRMS0iTo2IaRPZqKTPS9oj6WlJ90s6SdIZkralhWlbzbMd6yTtl7RP0iUT2baZmY1d04colzQLuAHojYhzgUnAMmAtsD0i5gPb02UkLUjfPwdYAtwuaVKj4jEzs2x5bp0tYojytwNTJL0JnAwcBtaRzNkNyfStFeCLJHOA90fECeCApP3AIuCxCWzfzMzGQBExegfpf9dpjoj47XFvVLqR5CnwN4DvRMRySS9HxOlVfV6KiOmSbgN2RMS9afudwCMR8UCd9a4CVgH09PRc2N/fnyuewcFBpk6dOt50GmbghVcaur6eKXD0jYausm2UNbei81o467TiVp6hXf6dNVrZ8lq8ePHuiOitbc/zBHdDhypPr0UsBeYBLwN/Iena0T5SL6x6HSNiE7AJoLe3N/r6+nLFVKlUyNu3SNetfbih61uzcIiNA+UcxqusuRWd18HlfYWtO0u7/DtrtLLmVWu0mfJ+JyJukfTH1PnjHBE3jHObHwUORMT/S7fzIPDvgKOSZkbEEUkzgWNp/0PAnKrPzyY5bWVmZk0y2n9hhi9q72rwNp8HLpJ0MslpqIvTbbwGrAA2pL8fSvtvBf5c0peBdwPzgZ0NjsnMzEYx2kx5/yf9vbmRG4yI76cj2D4ODAF/T3LqaCqwRdJKkoJyTdp/j6QtJGNTDQGrI+KtRsZkZmaja8lJ34i4Gbi5pvkEyVFGvf7r8bDoZmYt4+E+zMwsk4uFmZllyjwNJeldwH8F5lb3n8hzFmZm1lnyXLN4CPgb4LuALyybmXWhPMXi5Ij4YuGRmJlZ28pzzeKbki4rPBIzM2tboz3BfZzkyW0BvyvpBPBmuhwTHabczMw6x2gP5Z3azEDMzKx95Zkpb3ueNjMzK6/RTkOdBJwCzEhHih0e/XUayRhNZmbWJUa7G+q/AZ8jKQyPV7W/CvxJgTGZmVmbGe2axa3ArZKuj4g/bmJMZmbWZvI8Z/GCpKtr2l4BBiLiWL0PmJlZueQpFiuBXwO+ly73ATuAsyT9XkT8WUGxmZlZm8hTLH4O/EpEHAWQ1APcAfwq8CjgYmFmVnJ5nuCeO1woUseAsyLiRZKH9MZM0umSHpD0D5L2Svo1SWdI2ibp2fT39Kr+6yTtl7RP0iXj2aaZmY1fnmLxN5K+KWmFpOHpTh+VdArw8ji3eyvw7Yh4P/BBkilc1wLbI2I+sD1dRtICYBlwDrAEuF3SpHFu18zMxiFPsVgN3A2cB5wP3EMytelrEbF4rBuUNA34CHAnQET8LCJeBpYCw1O4bgauSl8vBfoj4kREHAD2A4vGul0zMxs/RURzNyidRzLn9jMkRxW7gRuBFyLi9Kp+L0XEdEm3ATsi4t60/U7gkYh4oM66VwGrAHp6ei7s7+/PFdPg4CBTp06dSFoNMfDCKw1dX88UOPpGQ1fZNsqaW9F5LZx1WnErz9Au/84arWx5LV68eHdE9Na255n86GrgD4AzSZ7inuhAgm8HLgCuj4jvS7qV9JTTSCHUaatb4SJiE0khore3N/r6+nIFVKlUqO47d+3DuT7XeI2dEn3NwiE2DrRkmvXClTW3ovM6uLyvsHVnqf13VhZlzatWntNQtwBXRsRpETEtIk6d4Iizh4BDEfH9dPkBkuJxVNJMgPT3sar+c6o+Pxs4PIHtm5nZGOUpFkcjYm+jNhgRPwb+SdLZadPFJKektgIr0rbhC+mk7cskTZY0D5gP7GxUPGZmli3P8e4uSV8HvgGcGG6MiAcnsN3rgfskvRP4IfBpksK1RdJK4HngmnQ7eyRtISkoQyQX1z29q5lZE+UpFtOA14GPVbUFMO5iERFPAL90AYXkKKNe//XA+vFuz8zMJiazWETEp5sRiJmZta88d0OdRTK8R09EnCvpAyQXvP9X4dGZWUO16k6/gxsub8l2rXHyXOD+KrCOdGiPiHiK5IlqMzPrEnmKxckRUXv30VARwZiZWXvKUyx+Iul9pA/CSfoEcKTQqMzMrK3kuRtqNclT0e+X9AJwAFheaFRmZtZWMo8sIuKHEfFR4F3A+yPi14HfLDwyMzNrG3lOQwGQjjJ7PF387wXFY2ZmbSh3sahRb3A/MzMrqfEWi+aOa25mZi014gVuScepXxQETCksIjMzazsjFouIOLWZgZiZWfsa72koMzPrIi4WZmaWycXCzMwyuViYmVmmlhULSZMk/b2kb6bLZ0jaJunZ9Pf0qr7rJO2XtE/SJa2K2cysW7XyyOJGoHpu77XA9oiYD2xPl5G0gGRI9HOAJcDtkiY1OVYzs67WkmIhaTZwOfC1qualwOb09Wbgqqr2/og4EREHgP3AoiaFamZmgCKa/zC2pAeA3wdOBb4QER+X9HJEnF7V56WImC7pNmBHRNybtt8JPBIRD9RZ7ypgFUBPT8+F/f39ueIZHBxk6tSp/7I88MIr486tnfRMgaNvtDqKYpQ1t7LmtXDWab/076wsypbX4sWLd0dEb217niHKG0rSx4FjEbFbUl+ej9Rpq1vhImITyXDq9Pb2Rl9fntVDpVKhuu91LZp6stHWLBxi40DTd3FTlDW3suZ1cHnfL/07K4uy5lWrFd/KDwNXSroMOAmYJule4KikmRFxRNJM4Fja/xAwp+rzs4HDTY3YzKzLNf2aRUSsi4jZETGX5ML1/42Ia4GtwIq02wrgofT1VmCZpMmS5gHzgdppXs3MrEDtdLy7AdgiaSXwPHANQETskbQFeIZk7u/VEfFW68I0M+s+LS0WEVEBKunrfwYuHqHfemB90wIzM7Nf4Ce4zcwsk4uFmZllcrEwM7NMLhZmZpbJxcLMzDK5WJiZWSYXCzMzy+RiYWZmmVwszMwsk4uFmZllcrEwM7NMLhZmZpbJxcLMzDK10xDlZlZSc9c+zJqFQ02fhfLghsubur0y85GFmZllanqxkDRH0vck7ZW0R9KNafsZkrZJejb9Pb3qM+sk7Ze0T9IlzY7ZzKzbteLIYghYExG/AlwErJa0AFgLbI+I+cD2dJn0vWXAOcAS4HZJk1oQt5lZ12rFHNxHIuLx9PVxYC8wC1gKbE67bQauSl8vBfoj4kREHAD2A4uaGrSZWZdTRLRu49Jc4FHgXOD5iDi96r2XImK6pNuAHRFxb9p+J/BIRDxQZ32rgFUAPT09F/b39+eKY3BwkKlTp/7L8sALr4w3pbbSMwWOvtHqKIpR1tzKmhe0JreFs04rfBu1fz863eLFi3dHRG9te8vuhpI0FfhL4HMR8aqkEbvWaatb4SJiE7AJoLe3N/r6+nLFUqlUqO7b7Ds2irJm4RAbB8p5w1tZcytrXtCa3A4u7yt8G7V/P8qqJXdDSXoHSaG4LyIeTJuPSpqZvj8TOJa2HwLmVH18NnC4WbGamVlr7oYScCewNyK+XPXWVmBF+noF8FBV+zJJkyXNA+YDO5sVr5mZteY01IeB3wIGJD2Rtv0usAHYImkl8DxwDUBE7JG0BXiG5E6q1RHxVtOjNjPrYk0vFhHxt9S/DgFw8QifWQ+sLywoMzMblZ/gNjOzTC4WZmaWycXCzMwyuViYmVkmFwszM8vkYmFmZpnKOa6AmRnJpEtFG2lSp7JNvOQjCzMzy+RiYWZmmVwszMwsk4uFmZllcrEwM7NMLhZmZpbJxcLMzDK5WJiZWSYXCzMzy9QxT3BLWgLcCkwCvhYRG1ockpnZiJrx9Hg9RT053hFHFpImAX8CXAosAD4paUFrozIz6x4dUSyARcD+iPhhRPwM6AeWtjgmM7OuoYhodQyZJH0CWBIR/yVd/i3gVyPiszX9VgGr0sWzgX05NzED+EmDwm0nZc0LyptbWfOC8uZWtrzeExHvqm3slGsWqtP2S1UuIjYBm8a8cmlXRPSOJ7B2Vta8oLy5lTUvKG9uZc2rVqechjoEzKlang0cblEsZmZdp1OKxQ+A+ZLmSXonsAzY2uKYzMy6RkechoqIIUmfBf6a5NbZuyJiTwM3MeZTVx2irHlBeXMra15Q3tzKmtcv6IgL3GZm1lqdchrKzMxayMXCzMwydXWxkLRE0j5J+yWtbXU8EyXpoKQBSU9I2pW2nSFpm6Rn09/TWx1nFkl3STom6emqthHzkLQu3Yf7JF3SmqjzGSG3L0l6Id1vT0i6rOq9jshN0hxJ35O0V9IeSTem7R2930bJq+P32ZhFRFf+kFwofw54L/BO4ElgQavjmmBOB4EZNW23AGvT12uBP2h1nDny+AhwAfB0Vh4kw788CUwG5qX7dFKrcxhjbl8CvlCnb8fkBswELkhfnwr8Yxp/R++3UfLq+H021p9uPrLoliFElgKb09ebgataF0o+EfEo8GJN80h5LAX6I+JERBwA9pPs27Y0Qm4j6ZjcIuJIRDyevj4O7AVm0eH7bZS8RtIReY1HNxeLWcA/VS0fYvQvQScI4DuSdqdDnwD0RMQRSL74wJkti25iRsqjLPvxs5KeSk9TDZ+q6cjcJM0Fzge+T4n2W01eUKJ9lkc3F4tcQ4h0mA9HxAUko/OulvSRVgfUBGXYj3cA7wPOA44AG9P2jstN0lTgL4HPRcSro3Wt09a2udXJqzT7LK9uLhalG0IkIg6nv48Bf0Vy+HtU0kyA9Pex1kU4ISPl0fH7MSKORsRbEfFz4Kv862mLjspN0jtI/qDeFxEPps0dv9/q5VWWfTYW3VwsSjWEiKRTJJ06/Br4GPA0SU4r0m4rgIdaE+GEjZTHVmCZpMmS5gHzgZ0tiG/chv+Ypn6TZL9BB+UmScCdwN6I+HLVWx2930bKqwz7bMxafYW9lT/AZSR3NzwH3NTqeCaYy3tJ7sJ4EtgznA/wb4DtwLPp7zNaHWuOXO4nObR/k+R/aitHywO4Kd2H+4BLWx3/OHL7M2AAeIrkj83MTssN+HWS0y1PAU+kP5d1+n4bJa+O32dj/fFwH2ZmlqmbT0OZmVlOLhZmZpbJxcLMzDK5WJiZWSYXCzMzy+RiYWZmmVwszMws0/8Hh3wpcvklCRgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = train_df.sentence.str.len().hist().set_ylabel(\"Length in characters\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And by word count, assuming a very simple tokenization strategy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAD4CAYAAAAdIcpQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZ9UlEQVR4nO3dfZBdBZnn8e9vomIghJcFejJJdjtOBRVodKVFXJzZjqhkhDGsjrOxcAwOs9m1ouJMLE1mtopdq7LDOou7iANTWUDC6BAjomS1UKloL+4uiImgISBLlAwTEpNxEaHViXT47R/nNHNtb+fc7ty3vvf3qerqc57z9jzV9+bJeZdtIiIijuTXOp1ARER0vzSLiIiolGYRERGV0iwiIqJSmkVERFR6QacTaJVTTjnFg4ODAPz0pz/luOOO62xCbdZvNfdbvZCa+0W7a96xY8ePbJ86Od6zzWJwcJDt27cDMDo6ysjISGcTarN+q7nf6oXU3C/aXbOkv60Xz2GoiIiolGYRERGV0iwiIqJSmkVERFRKs4iIiEppFhERUSnNIiIiKqVZREREpTSLiIio1LN3cMfsMLjuS01Zz9qhcS6b5rr2XHVRU7Yd0Q/SLAJo3j/aEdGbchgqIiIqpVlERESlljULSTdJOijpwUnx90l6RNIuSR+tia+XtLucdmFN/BxJO8tpH5ekVuUcERH1tXLP4mZgeW1A0jJgBXC27TOB/1LGzwBWAmeWy1wnaU652PXAamBp+fNL64yIiNZrWbOwfTfw5KTwe4CrbB8q5zlYxlcAm20fsv0YsBs4V9ICYL7te2wbuAW4pFU5R0REfe2+Gup04LckbQD+Afig7W8BC4F7a+bbW8aeLYcnx+uStJpiL4SBgQFGR0cBGBsbe364X0y35rVD461Lpg0G5k6/htn+mcjnuj90S83tbhYvAE4CzgNeDWyR9BKg3nkIHyFel+2NwEaA4eFhT7xdKm/XqjbdexS6zdqhca7eOb2P855LR1qTTJvkc90fuqXmdl8NtRe43YX7gOeAU8r44pr5FgH7yviiOvGIiGijdjeLLwCvB5B0OvAi4EfAVmClpGMkLaE4kX2f7f3AM5LOK6+CehdwR5tzjojoey07DCXpVmAEOEXSXuBK4CbgpvJy2l8Aq8oT17skbQEeAsaBNbYPl6t6D8WVVXOBO8ufiIhoo5Y1C9vvmGLSO6eYfwOwoU58O3BWE1OLiIhpyh3cERFRKc0iIiIqpVlERESlNIuIiKiUZhEREZXSLCIiolKaRUREVEqziIiISmkWERFRKc0iIiIqpVlERESlNIuIiKiUZhEREZXSLCIiolKaRUREVEqziIiISi1rFpJuknSwfCve5GkflGRJp9TE1kvaLekRSRfWxM+RtLOc9vHy9aoREdFGrdyzuBlYPjkoaTHwRuDxmtgZwErgzHKZ6yTNKSdfD6ymeC/30nrrjIiI1mpZs7B9N/BknUn/FfgQ4JrYCmCz7UO2HwN2A+dKWgDMt31P+a7uW4BLWpVzRETU17J3cNcj6S3AE7a/M+lo0kLg3prxvWXs2XJ4cnyq9a+m2AthYGCA0dFRAMbGxp4f7hfTrXnt0HjrkmmDgbnTr+HaT9/RomyObGjhCU1ZTz7X/aFbam5bs5B0LPBnwJvqTa4T8xHiddneCGwEGB4e9sjICACjo6NMDPeL6dZ82bovtS6ZNlg7NM7VO9v6f58Z23PpSFPWk891f+iWmtv57fpNYAkwsVexCPi2pHMp9hgW18y7CNhXxhfViUdERBu17dJZ2zttn2Z70PYgRSN4le0fAluBlZKOkbSE4kT2fbb3A89IOq+8CupdQGeOHURE9LFWXjp7K3AP8FJJeyVdPtW8tncBW4CHgC8Da2wfLie/B7iB4qT394E7W5VzRETU17LDULbfUTF9cNL4BmBDnfm2A2c1NbmIiJiW3MEdERGV0iwiIqJSmkVERFRKs4iIiEppFhERUWl23PLaJwabeBf12qHxWX9XdkR0j+xZREREpTSLiIiolGYRERGV0iwiIqJSmkVERFRKs4iIiEppFhERUSnNIiIiKqVZREREpTSLiIio1Mo35d0k6aCkB2tifyHpe5K+K+nzkk6smbZe0m5Jj0i6sCZ+jqSd5bSPl69XjYiINmrlnsXNwPJJsbuAs2yfDfxfYD2ApDOAlcCZ5TLXSZpTLnM9sJrivdxL66wzIiJarGXNwvbdwJOTYl+1PV6O3gssKodXAJttH7L9GMX7ts+VtACYb/se2wZuAS5pVc4REVFfJ586+4fAZ8rhhRTNY8LeMvZsOTw5Xpek1RR7IQwMDDA6OgrA2NjY88PdbO3QePVMDRqY29z1dbvZVG+zPouz5XPdTKm5czrSLCT9GTAOfHoiVGc2HyFel+2NwEaA4eFhj4yMAMWXc2K4mzXzkeJrh8a5emf/PIF+NtW759KRpqxntnyumyk1d07bv12SVgEXAxeUh5ag2GNYXDPbImBfGV9UJx4REW00rXMWkk6SdPZMNyZpOfBh4C22f1YzaSuwUtIxkpZQnMi+z/Z+4BlJ55VXQb0LuGOm24+IiJmpbBaSRiXNl3Qy8B3gk5I+1sBytwL3AC+VtFfS5cAngOOBuyQ9IOmvAGzvArYADwFfBtbYPlyu6j3ADRQnvb8P3DndIiMi4ug0chjqBNtPS/oj4JO2r5T03aqFbL+jTvjGI8y/AdhQJ74dOKuBPCMiokUaOQz1gvIS1t8HvtjifCIiogs10iw+AnwF2G37W5JeAjza2rQiIqKbVB6Gsv1Z4LM14z8A3tbKpCIiortM2SwkXcuR72l4f0syioiIrnOkw1DbgR3Ai4FXURx6ehR4JXB46sUiIqLXTLlnYXsTgKTLgGW2ny3H/wr4aluyi4iIrtDICe7foLg3YsK8MhYREX2ikfssrgLul/T1cvxfAv+hZRlFRETXOWKzkPRrwCPAa8ofgHW2f9jqxCIionscsVnYfk7S1bZfS57JFBHRtxo5Z/FVSW/L60wjIvpXI+cs/gQ4Djgs6R/KmG3Pb11aERHRTRq5g/v4qnkiIqK3NfTyI0lvAX67HB21nQcKRkT0kUbeZ3EVcAXFuyYeAq4oYxER0Sca2bN4M/BK288BSNoE3A+sa2ViERHRPRp9reqJNcMnNLKApJskHZT0YE3sZEl3SXq0/H1SzbT1knZLekTShTXxcyTtLKd9PFdlRUS0XyPN4s8p7uC+udyr2AH8pwaWuxlYPim2DthmeymwrRxH0hnASuDMcpnrJM0pl7keWE3xXu6lddYZEREtVtksbN8KnAfcXv681vbmBpa7G3hyUngFsKkc3gRcUhPfbPuQ7cco3rd9bvmGvvm277Ft4JaaZSIiok0qz1lI+mvgbuAbtr93lNsbsL0fwPZ+SaeV8YXAvTXz7S1jz5bDk+NT5bqaYi+EgYEBRkdHARgbG3t+uJutHRpv2roG5jZ3fd1uNtXbrM/ibPlcN1Nq7pxGTnB/EngdcG35StUHgLttX9PEPOqdh/AR4nXZ3ghsBBgeHvbIyAhQfDknhrvZZeu+1LR1rR0a5+qdDV0Z3RNmU717Lh1pynpmy+e6mVJz5zRyU97XJP1P4NXAMuDfUZxbmEmzOCBpQblXsQA4WMb3Aotr5lsE7Cvji+rEIyKijRq5z2Ib8L+Bf03xBNpX237ZDLe3FVhVDq/iHx9OuBVYKekYSUsoTmTfVx6yekbSeeVVUO8iDzSMiGi7Rq6G+i7wC+As4GzgLElzqxaSdCtwD/BSSXslXU7xbow3SnoUeGM5ju1dwBaKm/6+DKyxPfHq1vcAN1Cc9P4+cGfj5UVERDM0chjqjwEkzQPeTXEO49eBYyqWe8cUky6YYv4NwIY68e0UjSoiIjqkkauh3gv8FnAO8LfATcA3WpxXRER0kUYuH5kLfAzYYXt2XJsYERFN1chhqL9oRyIR/WKwSZdIrx0an/bl1nuuuqgp247+0+izoSIioo+lWURERKU0i4iIqNTITXlvLR8p/hNJT0t6RtLT7UguIiK6QyNXQ30U+F3bD7c6mYiI6E6NHIY6kEYREdHfGtmz2C7pM8AXgEMTQdu3tyqpiIjoLo00i/nAz4A31cRM8SKkiIjoA43clPfudiQSERHda8pmIelDtj8q6VrqvHDI9vtbmllERHSNI+1ZTJzU3t6ORCIiontN2Sxs/4/y96b2pRMREd0od3BHRESljjQLSX8saZekByXdKunFkk6WdFd5t/hdkk6qmX+9pN2SHpF0YSdyjojoZ21vFpIWAu8Hhm2fBcwBVgLrgG22lwLbynEknVFOPxNYDlwnaU67846I6GeNvCnvVODfAIO189v+w6Pc7lxJzwLHAvuA9cBIOX0TMAp8GFgBbLZ9CHhM0m7gXIr3e0dERBvI/pWrYn95Bun/ULxGdQdweCJu+3Mz3qh0BcX7tn8OfNX2pZKesn1izTw/tn2SpE8A99r+VBm/EbjT9m111rsaWA0wMDBwzubNmwEYGxtj3rx5M023bXY+8ZOmrWtgLhz4edNW1/X6rV6YWc1DC09oTTJtMlu+y83U7pqXLVu2w/bw5Hgjd3Afa/vDzUqkPBexAlgCPAV8VtI7j7RInVjdDmd7I7ARYHh42CMjIwCMjo4yMdzNpvvWsyNZOzTO1Tsb+fP2hn6rF2ZW855LR1qTTJvMlu9yM3VLzY2cs/iipDc3cZtvAB6z/fe2n6V4bMi/AA5IWgBQ/j5Yzr8XWFyz/CKKw1YREdEmUzaLmvdWXEHRMH7epPdZPA6cJ+lYSQIuoLgBcCuwqpxnFXBHObwVWCnpGElLgKXAfUex/YiImKYj3ZR3fCs2aPubkm4Dvg2MA/dTHDqaB2yRdDlFQ3l7Of8uSVuAh8r519g+XHflERHREo1cDbXN9gVVsemwfSVw5aTwIYq9jHrzb6A4IR4RER1wpAcJvhg4DjilPCk9caJ5PvAbbcgtIiK6xJH2LP4t8AGKxvDtmvjTwF+2MKeIiOgyRzpncQ1wjaT32b62jTlFRESXaeQi7SckvXVS7CfATtsH6y0QERG9pZFmcTnwWuDr5fgIcC9wuqSP2P7rFuUWERFdopFm8RzwctsHACQNANcDrwHuBtIsIiJ6XCN3cA9ONIrSQeB0208Cz7YmrYiI6CaN7Fl8Q9IXgc+W428D7pZ0HMWznSIiosc10izWUDSI8ynutbgF+JyLx9Uua2FuERHRJSqbRdkUbit/IiKiDzXyuI+3Av8ZOI1iz0IUPWR+i3PrmMEmPio8IqIXNHIY6qPA79p+uNXJREREd2rkaqgDaRQREf2tkT2L7ZI+A3yB4smwANi+vVVJRUREd2mkWcwHfga8qSZmijfcRUREH2jkaqh3tyORiIjoXpXnLCSdLmmbpAfL8bMl/fuj2aikEyXdJul7kh6W9FpJJ0u6S9Kj5e+TauZfL2m3pEckXXg0246IiOlr5AT3fwfWUz7aw/Z3gZVHud1rgC/bfhnwCop3cK8DttleCmwrx5F0Rrm9M4HlwHWS5hzl9iMiYhoaaRbH2r5vUmx8phuUNB/4beBGANu/sP0UsALYVM62CbikHF4BbLZ9yPZjwG7g3JluPyIipq+RE9w/kvSbFCe1kfR7wP6j2OZLgL8HPinpFcAO4ApgwPZ+ANv7JZ1Wzr+Q4pHoE/aWsV8haTWwGmBgYIDR0VEAxsbGnh9uxNqhGffCrjEwtzfqaFS/1Qszq3k634NuNN3vci/olpobfTbURuBlkp4AHgMuPcptvgp4n+1vSrqG8pDTFFQn5noz2t5Y5srw8LBHRkaA4gsyMdyIy3rgDu61Q+NcvbORP29v6Ld6YWY177l0pDXJtMl0v8u9oFtqrjwMZfsHtt8AnAq8zPbrgH91FNvcC+y1/c1y/DaK5nFA0gKA8vfBmvkX1yy/CNh3FNuPiIhpauScBQC2f2r7mXL0T2a6Qds/BP5O0kvL0AXAQ8BWYFUZWwXcUQ5vBVZKOkbSEmApMPkcSkREtNBM99vrHRqajvcBn5b0IuAHwLspGtcWSZcDjwNvB7C9S9IWioYyDqyxffgotx/Rlzr1kMw9V13Uke1G88y0WdQ9Z9DwwvYDwHCdSRdMMf8GYMPRbDMiImZuymYh6RnqNwUBc1uWUUREdJ0pm4Xt49uZSEREdK+GT3BHRET/SrOIiIhKaRYREVEpzSIiIiqlWURERKU0i4iIqJRmERERldIsIiKiUppFRERUSrOIiIhKaRYREVEpzSIiIiqlWURERKU0i4iIqNSxZiFpjqT7JX2xHD9Z0l2SHi1/n1Qz73pJuyU9IunCTuUcEdGvOrlncQXwcM34OmCb7aXAtnIcSWcAK4EzgeXAdZLmtDnXiIi+1pFmIWkRcBFwQ014BbCpHN4EXFIT32z7kO3HgN3AuW1KNSIiANlH9TrtmW1Uug34c+B44IO2L5b0lO0Ta+b5se2TJH0CuNf2p8r4jcCdtm+rs97VwGqAgYGBczZv3gzA2NgY8+bNazi/nU/8ZMa1dYuBuXDg553Oon36rV6YXTUPLTyhKeuZ7ne5F7S75mXLlu2wPTw5PuVrVVtF0sXAQds7JI00skidWN0OZ3sjsBFgeHjYIyPF6kdHR5kYbsRl677U8Lzdau3QOFfvbPuft2P6rV6YXTXvuXSkKeuZ7ne5F3RLzZ34pJ0PvEXSm4EXA/MlfQo4IGmB7f2SFgAHy/n3Aotrll8E7GtrxhERfa7t5yxsr7e9yPYgxYnrr9l+J7AVWFXOtgq4oxzeCqyUdIykJcBS4L42px0R0de6aR/2KmCLpMuBx4G3A9jeJWkL8BAwDqyxfbhzaUZE9J+ONgvbo8BoOfz/gAummG8DsKFtiUVExC/JHdwREVEpzSIiIiqlWURERKU0i4iIqJRmERERldIsIiKiUppFRERUSrOIiIhKaRYREVEpzSIiIiqlWURERKU0i4iIqJRmERERldIsIiKiUppFRERUSrOIiIhKbX/5kaTFwC3ArwPPARttXyPpZOAzwCCwB/h92z8ul1kPXA4cBt5v+yvtzjsiZm5w3Zeasp61Q+NcNo117bnqoqZsNzqzZzEOrLX9cuA8YI2kM4B1wDbbS4Ft5TjltJXAmcBy4DpJczqQd0RE32p7s7C93/a3y+FngIeBhcAKYFM52ybgknJ4BbDZ9iHbjwG7gXPbmnRERJ+T7c5tXBoE7gbOAh63fWLNtB/bPknSJ4B7bX+qjN8I3Gn7tjrrWw2sBhgYGDhn8+bNAIyNjTFv3ryG89r5xE9mWlLXGJgLB37e6Szap9/qhdTciKGFJ7QumTaZ7r9fR2vZsmU7bA9Pjrf9nMUESfOAzwEfsP20pClnrROr2+FsbwQ2AgwPD3tkZASA0dFRJoYbMZ1jot1q7dA4V+/s2J+37fqtXkjNjdhz6UjrkmmT6f771SoduRpK0gspGsWnbd9ehg9IWlBOXwAcLON7gcU1iy8C9rUr14iI6ECzULELcSPwsO2P1UzaCqwqh1cBd9TEV0o6RtISYClwX7vyjYiIzhyGOh/4A2CnpAfK2J8CVwFbJF0OPA68HcD2LklbgIcorqRaY/tw27OOiOhjbW8Wtv8X9c9DAFwwxTIbgA0tSyoiIo4od3BHRESlNIuIiKiUZhEREZXSLCIiolKaRUREVEqziIiISmkWERFRqb8eLBMRfaVZ79GYiV57l0b2LCIiolKaRUREVEqziIiISmkWERFRKc0iIiIqpVlERESlNIuIiKiU+ywiIlqgWfd4rB0a57JprKtV93fMmj0LScslPSJpt6R1nc4nIqKfzIpmIWkO8JfA7wBnAO+QdEZns4qI6B+zolkA5wK7bf/A9i+AzcCKDucUEdE3ZLvTOVSS9HvActt/VI7/AfAa2++dNN9qYHU5+lLgkXL4FOBHbUq3W/Rbzf1WL6TmftHumv+Z7VMnB2fLCW7Vif1Kl7O9Edj4KwtL220PtyKxbtVvNfdbvZCa+0W31DxbDkPtBRbXjC8C9nUol4iIvjNbmsW3gKWSlkh6EbAS2NrhnCIi+sasOAxle1zSe4GvAHOAm2zvmsYqfuXQVB/ot5r7rV5Izf2iK2qeFSe4IyKis2bLYaiIiOigNIuIiKjU082iHx4RIukmSQclPVgTO1nSXZIeLX+f1Mkcm03SYklfl/SwpF2SrijjPVu3pBdLuk/Sd8qa/2MZ79maoXh6g6T7JX2xHO/pegEk7ZG0U9IDkraXsY7X3bPNoo8eEXIzsHxSbB2wzfZSYFs53kvGgbW2Xw6cB6wp/7a9XPch4PW2XwG8Elgu6Tx6u2aAK4CHa8Z7vd4Jy2y/sub+io7X3bPNgj55RIjtu4EnJ4VXAJvK4U3AJe3MqdVs77f97XL4GYp/TBbSw3W7MFaOvrD8MT1cs6RFwEXADTXhnq23Qsfr7uVmsRD4u5rxvWWsHwzY3g/FP6zAaR3Op2UkDQL/HPgmPV53eUjmAeAgcJftXq/5vwEfAp6rifVyvRMMfFXSjvIRRtAFdc+K+yxmqKFHhMTsJWke8DngA7aflur9yXuH7cPAKyWdCHxe0lkdTqllJF0MHLS9Q9JIh9Npt/Nt75N0GnCXpO91OiHo7T2Lfn5EyAFJCwDK3wc7nE/TSXohRaP4tO3by3DP1w1g+ylglOJcVa/WfD7wFkl7KA4hv17Sp+jdep9ne1/5+yDweYpD6h2vu5ebRT8/ImQrsKocXgXc0cFcmk7FLsSNwMO2P1YzqWfrlnRquUeBpLnAG4Dv0aM1215ve5HtQYrv7tdsv5MerXeCpOMkHT8xDLwJeJAuqLun7+CW9GaK454TjwjZ0NmMmk/SrcAIxWOMDwBXAl8AtgD/FHgceLvtySfBZy1JrwO+AezkH49n/ynFeYuerFvS2RQnNudQ/Cdvi+2PSPon9GjNE8rDUB+0fXGv1yvpJRR7E1CcJvgb2xu6oe6ebhYREdEcvXwYKiIimiTNIiIiKqVZREREpTSLiIiolGYRERGV0iwiIqJSmkVERFT6/0vTwuimwKgQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_df['word_count'] = train_df.sentence.str.split().apply(len)\n",
    "\n",
    "_ = train_df['word_count'].hist().set_ylabel(\"Length in words\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEcCAYAAAAmzxTpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgfElEQVR4nO3de5gdVZnv8e8vFzqdhJA0SNtBsI8JaJMecSbtKAMOyURAAYeMAiMTNJg4ATxmRsc5EiecIx7FJx6H45wnzogBkXjEKMLILcgloTsYHkESiZLQXDwQAUlAAzHm0p3be/6o6ri70+nr7uzuvX6f59lPV9WuqvXWXr3fXXvVqrUVEZiZWXkbVuoAzMxs4DnZm5klwMnezCwBTvZmZglwsjczS4CTvZlZApzs7bCRFJImlzqOUpI0TdJLXTxflNdI0qWSVvdw3aslfbeP5fR5Wzu8nOwTJGmjpF2Stkt6XdJySceXOq42vUlUZtYzTvbp+kBEjAVqgFeAxSWOZ8BIGlHqGMxKzck+cRHRAtwKnNy2TNJRkr4j6beSfi3pKknDJFVJeknSB/L1xkr6laSP5vM3SbpO0gOS/iBplaQ3d1ZuF2XUAdcBp+bfPLYeYvv/IumhvJwVkv69rTlBUm3eHDJX0gvAg/m+r8rLejUv+6h8/YOaVvJvP+/Np6+WdKukH+Tl/VzSKQXrTpR0W34sz0v6h4LnKvPX5XVJTwLv7EG1nCPpOUm/k/TVPPYKSa9J+pOCfR+bf0N7Q3c7lPR/JL0oaZuktZLe02GVUX05Phs6nOwTJ2k08LfAIwWLFwNHAW8BzgA+CnwsIl4D5gDXSzoW+BqwLiK+U7DtLOCLwDHAOuDmQxR9qDKagcuBn0bE2IgYf4jtvwf8DDgauBr4SCfrnAHUAWcDl+aP6XmZY4GvH2LfnTkf+CFQlZd9u6SRkoYBdwG/AI4DZgCfknR2vt3ngUn542xgdg/K+hugAfizvNw5EdEKfB+4pGC9i4EVEfHbHuzzMeAdBfH/UNKoIhyfDRUR4UdiD2AjsB3YCuwFXgb+JH9uONAKnFyw/mVAU8H8YuCJfLujC5bfBHy/YH4ssA84Pp8PYHJ3ZZAl5dVdxH9CHvfogmXfBb6bT9fmZb2l4PmVwCcK5t8K7AFGANOAlzp5jd6bT18NPFLw3DBgE/Ae4F3ACx22/Rzw7Xz6OeB9Bc/N61hWh22jw/qfAFbm0+8CXgSG5fNrgIsOsZ/uXsPXgVOKcHxXt73ufgzuh9sy0zUzIlZIGk52VrdK0slkyeYI4NcF6/6a7KyuzRLgk8CXI2JLh/2+2DYREdslvQZMLFxOdtbfXRldmQi8FhE7O5Tb8SJzYZkTOylvBFDdwzILj2t/3uwzkez1mtihuWk48JOCcgvjKIyh27Ly9Sfm5T4qaQdwhqRNZB+cd/YkeEmfAT5eEPM4sno4qMxeHp8NEW7GSVxE7IuI/yQ7Az8d+B3ZGW9hW/sJwG8A8g+HbwLfAa7Qwd0EDyRcSWPJmgVe7rBOl2WQJZiubAKq8iaog8otPLyC6Zc7KW8v2cXpHcCBfeXH2LEdvPC4hgFvyvf5IvB8RIwveBwZEecUxFoY2wndHFvHYzmB9q/fUrKmnI8At0Z2zaVLefv8lcBFwITImsZ+D6gIx2dDhJN94pQ5H5gANEfEPuAW4BpJR+YXWP+JrJkE4F/yv3OAfwW+kyfHNudIOl3SEWRt949GROGZKj0o4xXgTfk+DhIRvyZrwrha0hGSTgU+0M2hLgM+nV/YHQt8GfgBsAJ4N9kFynMljQSuAio6bD9V0geV9ez5FFkz1CNk1w22Sboyvxg7XFK9pLYLsbcAn5M0QdKbgPndxAnw3/L1jwf+MY+zzf8la9O/hOwDtyeOJPtg+y0wQtL/IDuzL8bx2RDhZJ+uuyRtB7YB1wCzI2JD/tx8srPd54DVZBfsbpQ0lSwpfzRP2F8hO3teULDf75FdlHwNmEp2wbYznZaRP/cgsAHYLOl3h9h+FnAqsAX4EllCbO3ieG8kS5QPAc8DLfwx8e4iaxu/gezbxQ6g441Pd5BdyH6d7Kz6gxGxJ38dPkB28fN5sm8tN5BdfAb4AllTzPPA/XkM3bkDeJrstVkOfKvtiYh4Cfg52eve06aU+4AfA8/ksbTQvqmoP8dnQ4Qi/OMlVhySbiK7+HhVCcr+AfBURHz+EM+L7P99f4flTWQXGG/oYt9XA5Mj4pJDrVNski4FPh4Rp3fy3I3Ay6V4nW3o8pm9DUqSPibproL5X0m6pWD+FUnnSTpN0tNk7dEXSPqLgnWaJF0j6WFgJ/AWSWdKekrS7yV9nfbt1l2ZLKk574f+pKQ/y8uoy8vZKmmDpL/uUP7HC+bb3Rms7F6AyyU9q6wf/r/nzWqHvNdAUi3wQQrO9s16wsneBqtVwHuU3VBUA4wETgOQ9BZgDNmF4tVk7c8fJ2vOWS7p6IL9fISsu+ORZBclbyNrkz8G+H9t++zGycDbye4FGAf8NbAlb9+/i6x55liyZqGbJb21F8d5HtmNVqeQfWCdHYe410DSF4H1wFcj4vlelGHmZG/FExGXFqtpISKeA/5A1lZ8Blm7828kvS2ff5DsWsFjEVETEd+KiGXAU7S/WHtTRGyIiL3A+4EnI+LWiNgD/BuwuQfhHAV8LiIei8yv8ovE7ya7l2BRROyOiAeBu8luduqpRRGxNSJeABrz4+1URPz3PPlf04v9mwG4n70NaqvIbnianE9vJUv0p+bzHfvOw8H99Tv2tS/sTx6SOl6o7MzxZN8COpoIvNjhOkBv7heA9h82O8k+PMyKzmf2Npi1Jfv35NOryJL9Gfl0x77z0L6/PrTva9+uz3t+0bYno32+SDbcQUcvA8fn/dI7K79d/33gjT0oq417TlhROdnbYLaKbCybyrzL4U+A95GNh/M4cA9wkqS/kzRC0t+Sta/ffYj9LQemFPQn/wd6loBvAP5Z0tT8Aurk/N6AR8kS+meVjSMzjawJ6fv5duuAD0oarezms7m9OPYu7zUw6y0nexu0IuIZsjF8fpLPbyPre/5wfufvFrILnJ8h62//WeC8iOi0b36+/EJgUb7+icDDPYjjh2T3InyP7DrC7UBVROwmu1j7frL+5/9Bdg/CU/mmXwN2kyXupRx6ULjO9OReA7Mecz97M7ME+MzezCwBTvZmgLIfXdneyeO6UsdmVgxuxjEzS4DP7M3MEnBYb6o65phjora29nAWWRI7duxgzJgxpQ7DisB1WT5Sqcu1a9f+LiIO+l3iw5rsa2trWbNmzeEssiSampqYNm1aqcOwInBdlo9U6lJSp7+G5mYcM7MEONmbmSXAyd7MLAFO9mZmCXCyNzNLgJN9ES1btoz6+npmzJhBfX09y5YtK3VIZsmbP38+o0aNYvr06YwaNYr58+d3v1EZ8o+XFMmyZctYuHAh3/rWt9i3bx/Dhw9n7txsRNuLL+7NDxeZWbHMnz+f6667jq985SucfPLJPPnkk1x55ZUALF68uMTRHWYRcdgeU6dOjXI1ZcqUePDBByMiorGxMSIiHnzwwZgyZUoJo7L+aqtLG5oqKiri2muvjYg/1uW1114bFRUVJYxqYAFropP862acImlubub0009vt+z000+nubm5RBGZWWtrK5dffnm7ZZdffjmtra0liqh0nOyLpK6ujosuuqhd2+BFF11EXV1dqUOzPvD1l/JQUVHBvHnz2tXlvHnzqKioKHVoh53b7IvkuOOO4/bbb+eKK67gnHPO4Z577uEb3/gGZ511VqlDs17y9ZfyccYZZ3DzzTdzxRVXsGjRorTfl5217QzUo5zb7CsqKmLWrFkxZcqUGDZsWEyZMiVmzZpV1m2D5crXX8rHlClTYubMmVFRURFAVFRUxMyZM8u6LulPm72kjZKekLRO0pp8WZWkByQ9m/+dMJAfSoNda2srM2bMaLdsxowZSbYNDnW+/lI+mpub2b59O7t37wZg9+7dbN++Pcm67E0zzvRo/0POC4CVEbFI0oJ8/sqiRjeEjBgxgs985jPcdtttB776f+hDH2LECLeUDTV1dXWsXr2a6dOnH1i2evVqX38ZgiorK1mxYsVBzaspDHXcUX8u0J4PLM2nlwIz+x3NEDZu3Di2bdvG448/zt69e3n88cfZtm0b48aNK3Vo1ksLFy5k7ty5NDY2snfvXhobG5k7dy4LFy4sdWjWSzt27GDs2LFceOGFjBo1igsvvJCxY8eyY8eOUod22PX0tDOA+yUF8M2IWAJUR8QmgIjYJOnYzjaUNA+YB1BdXU1TU1P/ox6Etm7dynnnnceCBQvYs2cPI0eO5Nxzz+Xuu+8u22MuVzU1NcyaNYs5c+bwwgsvcMIJJ3DJJZdQU1PjuhyCLrvssnZ1edlll3HttdcmV5c9+g1aSRMj4uU8oT8AzAfujIjxBeu8HhFdtts3NDREuf54SX19PSeeeCI//vGPaW1tpaKigve///08++yzrF+/vtThWR+l8oMX5WrYsGEcccQR7a6dVVRUsHv3bvbv31/CyAaOpLUR0dBxeY+acSLi5fzvq8CPgD8HXpFUk++8Bni1eOEOPW1dL+fMmcNdd93FnDlzuP322znuuONKHZpZsoYPH05raysTJkzg+uuvZ8KECbS2tjJ8+PBSh3bYdZvsJY2RdGTbNHAWsB64E5idrzYbuGOgghwKVq1axaxZs3jooYc4//zzeeihh5g1axarVq0qdWhmydq7dy+jR49m69at/P3f/z1bt25l9OjR7N27t9ShHXY9ObOvBlZL+gXwM2B5RNwLLALOlPQscGY+n6zW1laWLFnC+vXrWblyJevXr2fJkiXuemlWYs3Nzezfv5/Gxkb279+fZLdL6MEF2oh4Djilk+VbgBkHb5GmiooKJk2axObNmw8se+Mb35jkbdnlYNmyZVxzzTU0NzdTV1fHwoULfffsEFVXV8euXbuICCRRWVlZ6pBKwmPjFMmYMWPYvHkzU6ZMYdmyZUyZMoXNmzcn2Z93qGsbLmHx4sXcd999LF68mIULF3p8nCFoxIgR7Ny5k/Hjx3P99dczfvx4du7cmeT9Lz3qjVMs5dwbRxK1tbVs2rTpQG+cmpoaNm7cyOF8ja3/6uvrWbx4MdOnTz/QG6exsZH58+e7Z9UQ4944f+Qz+yJ67LHHaGlpobGxkZaWFh577LFSh2R94OESykdE8MorrxARNDY2tptPTXrfZQbQO9/5zoPO7G3o8XAJ5UMSF1xwAZs2bTpw/aWmpgZJpQ7tsPOZfZFUVVWxceNGJk+ezLJly5g8eTIbN26kqqqq1KFZL3m4hPJRX1/PihUrmDRpErfddhuTJk1ixYoV1NfXlzq0w85t9kUyatQoJkyYcFBvnNdff52WlpYSRmZ94d445aG+vp7KykrWrl17oDfO1KlT2bVrV9lefzlUm72TfZFIYseOHYwePfrARb2dO3cyZsyYJNsHy4WHSxjahg8fTktLCyNHjjxQl3v27GHUqFHs27ev1OENiEMle7fZF0lFRQVnnXUWa9asOdBm39DQ4H72ZiVUV1fHmDFj2LNnz4FlI0eOTPL6i9vsi+Skk07i4Ycf5uyzz+ZHP/oRZ599Ng8//DAnnXRSqUMzS9bTTz/Nnj172o2Ns2fPHp5++ulSh3bY+cy+SJ555hlOO+007rvvPu68804qKio47bTTKNdmK7OhYO/evYwbN46JEydy2WWXUVdXx759+9i2bVupQzvsnOyLpLW1lfvvv7/TNnsbnIrV/c7XZAa3Rx99lLe97W0H3pdPPfVUks04TvZFUlFRwTHHHMOuXbsOLKusrHSb/SDWkyRdu2A5GxedexiisYGSYmLvjNvsi2jXrl1UV1fz7W9/m+rq6naJ38xK66qrrip1CCXlZF8kbT+QsHXrVj72sY+xdevWAz+UYGal96UvfanUIZSUk30RrVu3rt3YOOvWrSt1SGbJe+SRR9qNjfPII4+UOqSScLIvonPOOafLeTM7/DreFJfqTXJO9kVSVVXFhg0bqK+vZ/PmzdTX17NhwwaPjWNWQpJoaWmhsrKSDRs2UFlZSUtLS5IDobk3TpFs2bKFo48+mg0bNhwYQ6WqqootW7aUODKzdO3fv59hw4bR0tLCJz/5SSD7ACjXsey74mTfS705I3jttdcOub77ZpsVT2/el20DonX1fDlyM04vRUS3jzdfeXe365hZ8RTrfVnO700nezOzBDjZm5klwMnezCwBTvZmZglwsjczS4CTvZlZApzszcwS4GRvZpYAJ3szswQ42ZuZJaDHyV7ScEmPS7o7n6+S9ICkZ/O/EwYuTDMz64/enNn/I9BcML8AWBkRJwIr83kzMxuEepTsJb0JOBe4oWDx+cDSfHopMLOokZmZWdH09Mz+34DPAoWDQFdHxCaA/O+xxQ3NzMyKpdvx7CWdB7waEWslTettAZLmAfMAqquraWpq6u0uhqRUjjMFrsvykXJd9uTHS04D/lrSOcAoYJyk7wKvSKqJiE2SaoBXO9s4IpYASwAaGhoiid9/vHd5sr9zWXZcl+Uj8brsthknIj4XEW+KiFrgw8CDEXEJcCcwO19tNnDHgEVpZmb90p9+9ouAMyU9C5yZz5uZ2SDUq9+gjYgmoCmf3gLMKH5IZmZWbL6D1swsAU72ZmYJcLI3M0uAk72ZWQKc7M3MEuBkb2aWACd7M7MEONmbmSXAyd7MLAFO9mZmCXCyNzNLgJO9mVkCnOzNzBLgZG9mlgAnezOzBDjZm5klwMnezCwBTvZmZglwsjczS4CTvZlZApzszcwS4GRvZpYAJ3szswQ42ZuZJcDJ3swsAU72ZmYJcLI3M0uAk72ZWQKc7M3MEuBkb2aWgG6TvaRRkn4m6ReSNkj6Qr68StIDkp7N/04Y+HDNzKwvenJm3wr8VUScArwDeJ+kdwMLgJURcSKwMp83M7NBqNtkH5nt+ezI/BHA+cDSfPlSYOZABGhmZv3XozZ7ScMlrQNeBR6IiEeB6ojYBJD/PXbAojQzs34Z0ZOVImIf8A5J44EfSarvaQGS5gHzAKqrq2lqaupDmENPKseZAtdl+Ui5LnuU7NtExFZJTcD7gFck1UTEJkk1ZGf9nW2zBFgC0NDQENOmTetfxEPBvctJ4jhT4LosH4nXZU9647whP6NHUiXwXuAp4E5gdr7abOCOAYrRzMz6qSdn9jXAUknDyT4cbomIuyX9FLhF0lzgBeDCAYzTzMz6odtkHxG/BP60k+VbgBkDEZSZmRWX76A1M0uAk72ZWQKc7M3MEuBkb2aWACd7M7MEONmbmSXAyd7MLAFO9mZmCXCyNzNLgJO9mVkCnOzNzBLgZG9mlgAnezOzBDjZm5klwMnezCwBiojDVlhDQ0OsWbPmsJXXW6d84X5+v2tPqcMA4KjKkfzi82eVOowhy3VZXgZLfQ6FupS0NiIaOi7v1W/Qlrvf79rDxkXn9ns/TU1N/f6ty9oFy/sdR8pcl+WlGPWZel26GcfMLAFO9mZmCXCyNzNLgJO9mVkCnOzNzBLgZG9mlgAnezOzBDjZm5klwMnezCwBTvZmZglwsjczS4CTvZlZApzszcwS4GRvZpaAbpO9pOMlNUpqlrRB0j/my6skPSDp2fzvhIEP18zM+qInZ/Z7gc9ERB3wbuC/SjoZWACsjIgTgZX5vJmZDULdJvuI2BQRP8+n/wA0A8cB5wNL89WWAjMHKEYzM+unXv1SlaRa4E+BR4HqiNgE2QeCpGMPsc08YB5AdXU1TU1N/Yl3wBUjvu3btxdlP4P9tRrsXJflpb+vYfJ1GRE9egBjgbXAB/P5rR2ef727fUydOjUGszdfeXdR9tPY2NjvfRQrllS5LstLMV7DVOoSWBOd5N8e9caRNBK4Dbg5Iv4zX/yKpJr8+Rrg1aJ+CpmZWdH0pDeOgG8BzRHxvwueuhOYnU/PBu4ofnhmZlYMPWmzPw34CPCEpHX5sn8BFgG3SJoLvABcOCARmlnyjqxbwJ8sLUKHv6Xdr9J1HADn9j+OEug22UfEakCHeHpGccMxMzvYH5oXsXFR/5JsU1MT06ZN69c+ahcs79f2peQ7aM3MEuBkb2aWgF71sy93RWsXhKTbBgcD16VZe072BYrRLghuGxwMXJdm7bkZx8wsAU72ZmYJcLI3M0uAk72ZWQKc7M3MEuBkb2aWACd7M7MEONmbmSXAyd7MLAFO9mZmCXCyNzNLgJO9mVkCnOzNzBLgZG9mlgAnezOzBDjZm5klwMnezCwBTvZmZglwsjczS4CTvZlZApzszcwS4GRvZpaAEaUOYLCpXbC8ODu6t3/7OapyZHHiMCsTRXlvJvy+dLIvsHHRuUXZT+2C5UXbl5kV572Z+vvSzThmZglwsjczS0C3zTiSbgTOA16NiPp8WRXwA6AW2AhcFBGvD1yYZr3n6y9mf9STNvubgK8D3ylYtgBYGRGLJC3I568sfnhmfePrL2btdduMExEPAa91WHw+sDSfXgrMLG5YZmZWTH1ts6+OiE0A+d9jixeSmZkV24B3vZQ0D5gHUF1dTVNT00AXOSikcpwpcF2Wj5Trsq/J/hVJNRGxSVIN8OqhVoyIJcASgIaGhpg2bVofixxC7l1OEseZAtdl+Ui8LvvajHMnMDufng3cUZxwzMxsIHSb7CUtA34KvFXSS5LmAouAMyU9C5yZz5uZ2SDVbTNORFx8iKdmFDkWMzMbIL6D1swsAU72ZmYJcLI3M0uAk72ZWQKc7M3MEuBkb2aWACd7M7MEONmbmSXAyd7MLAFO9mZmCXCyNzNLgJO9mVkCnOzNzBLgZG9mlgAnezOzBDjZm5klwMnezCwBTvZmZglwsjczS4CTvZlZApzszcwS4GRvZpYAJ3szswQ42ZuZJcDJ3swsAU72ZmYJGFHqAIYaST1b7ytdPx8RRYjG+sN1WT6KVZdQvvXpM/teiohuH42Njd2uY6XnuiwfxarLcq5PJ3szswQ42ZuZJaBfyV7S+yQ9LelXkhYUKygzMyuuPid7ScOBfwfeD5wMXCzp5GIFZmZmxdOfM/s/B34VEc9FxG7g+8D5xQnLzMyKqT/J/jjgxYL5l/JlZmY2yPSnn31nHVsP6rckaR4wD6C6upqmpqZ+FDk0bN++PYnjTIHrsnykXpf9SfYvAccXzL8JeLnjShGxBFgC0NDQENOmTetHkUNDU1MTKRxnClyX5SP1ulRfbyKQNAJ4BpgB/AZ4DPi7iNjQxTa/BX7dpwKHlmOA35U6CCsK12X5SKUu3xwRb+i4sM9n9hGxV9IngfuA4cCNXSX6fJuDAihHktZEREOp47D+c12Wj9Trsl9j40TEPcA9RYrFzMwGiO+gNTNLgJP9wFhS6gCsaFyX5SPpuuzzBVozMxs6fGZvZpYAJ/sBJmm8pE8UzE+UdGspY7LekVQr6e/6uO32YsdjvSPpckkfzacvlTSx4LkbUhnTy804A0xSLXB3RNSXOhbrG0nTgH+OiPM6eW5EROztYtvtETF2AMOzXpDURFaXa0ody+GW/Jl9ftbWLOl6SRsk3S+pUtIkSfdKWivpJ5Lelq8/SdIjkh6T9D/bztwkjZW0UtLPJT0hqW1QuEXAJEnrJH01L299vs2jkqYUxNIkaaqkMZJuzMt4vGBf1gt9qNubJF1QsH3bWfki4D15HX46Pzv8oaS7gPu7qHvrp7wOn5K0VNIvJd0qabSkGfl744n8vVKRr79I0pP5uv+aL7ta0j/nddsA3JzXZWX+nmuQdIWk/1VQ7qWSFufTl0j6Wb7NN/MRf4eenvxMVzk/gFpgL/COfP4W4BJgJXBivuxdwIP59N3Axfn05cD2fHoEMC6fPgb4Fdn4QbXA+g7lrc+nPw18IZ+uAZ7Jp78MXJJPjye7U3lMqV+rofboQ93eBFxQsH1b3U4j+3bWtvxSsuFCqrqq+8J9+NGvOgzgtHz+RuAqskEYT8qXfQf4FFAFPF3w2o/P/15NdjYP0AQ0FOy/iewD4A1ko/i2Lf8xcDpQB9wFjMyX/wfw0VK/Ln15JH9mn3s+Itbl02vJ/sH+AvihpHXAN8mSMcCpwA/z6e8V7EPAlyX9ElhBNgJodTfl3gJcmE9fVLDfs4AFedlNwCjghN4dkuV6U7e98UBEvJZP96XuredejIiH8+nvkg3R8nxEPJMvWwr8JbANaAFukPRBYGdPC4iI3wLPSXq3pKOBtwIP52VNBR7L/19mAG/p/yEdfv26g7aMtBZM7yN7o26NiHf0Yh+zyM4OpkbEHkkbyZL0IUXEbyRtkfR24G+By/KnBHwoIp7uRfnWud7U7V7ypk1JAo7oYr87CqZ7XffWKz26sBjZEC5/TpaQPwx8EvirXpTzA7KTrqeAH0VE5P8HSyPic72MedDxmX3ntgHPS7oQsje+pFPy5x4BPpRPf7hgm6OAV/M3+3TgzfnyPwBHdlHW94HPAkdFxBP5svuA+fk/GpL+tL8HZAd0Vbcbyc7iIPshnpH5dHd1eKi6t+I4QdKp+fTFZN+eaiVNzpd9BFglaSzZ++gesmadd3Syr67q8j+BmXkZP8iXrQQukHQsgKQqSUOyfp3sD20WMFfSL4AN/PFXuD4F/JOkn5F9/f99vvxmoEHSmnzbpwAiYgvwsKT1kr7aSTm3kn1o3FKw7ItkieaX+cXcLxbzwOyQdXs9cEZet+/ij2fvvwT2SvqFpE93sr9O696KphmYnTeTVQFfAz5G1hT3BLAfuI4sid+dr7eK7JpYRzcB17VdoC18IiJeB54kGzXyZ/myJ8muEdyf7/cB+tbsV3LuetlLkkYDu/KveB8mu1jr3hdmA0Duulw0brPvvanA1/Mmlq3AnNKGY2bWPZ/Zm5klwG32ZmYJcLI3M0uAk72ZWQKc7C056mYkShWMX9SLfbYbV8dssHGyNzNLgJO9Jaub0SpHdBxpMd9mqqRVykbMvE/SkLzBxtLjZG8pawH+JiL+DJgOXNs2RAXZQFhLIuLtZEMsfELSSGAx2ciYU8lGYLymBHGb9ZpvqrKUtY1W+Zdkt9wXjlbZcaTFfwDuBeqBB/LPhOHApsMasVkfOdlbyroarbLj3YZB9uGwISJOxWyIcTOOpayr0So7jrS4muyHMd7QtlzSSBX80pjZYOZkbynrarTKjiMtfiMidgMXAF/JR8xcR/ZDKGaDnsfGMTNLgM/szcwS4GRvZpYAJ3szswQ42ZuZJcDJ3swsAU72ZmYJcLI3M0uAk72ZWQL+P3ZRNzmFFDxDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = train_df.boxplot(\"word_count\", by=\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Including subtrees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much of the special interest of the SST is that it includes labels, not just for full examples, but also for all the constituent words and phrases in those examples. You might also want to try training on this expanded dataset. It's much larger and so experiments will be more costly in terms of time and compute resources, but it could be worth it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "subtree_train_df = sst.train_reader(SST_HOME, include_subtrees=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "318582"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subtree_train_df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>example_id</th>\n",
       "      <th>sentence</th>\n",
       "      <th>label</th>\n",
       "      <th>is_subtree</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00001-00001</td>\n",
       "      <td>The Rock is destined to be the 21st Century 's...</td>\n",
       "      <td>positive</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00001-00002</td>\n",
       "      <td>The Rock</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00001-00003</td>\n",
       "      <td>The</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00001-00004</td>\n",
       "      <td>Rock</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00001-00005</td>\n",
       "      <td>is destined to be the 21st Century 's new `` C...</td>\n",
       "      <td>positive</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    example_id                                           sentence     label  \\\n",
       "0  00001-00001  The Rock is destined to be the 21st Century 's...  positive   \n",
       "1  00001-00002                                           The Rock   neutral   \n",
       "2  00001-00003                                                The   neutral   \n",
       "3  00001-00004                                               Rock   neutral   \n",
       "4  00001-00005  is destined to be the 21st Century 's new `` C...  positive   \n",
       "\n",
       "   is_subtree  \n",
       "0           0  \n",
       "1           1  \n",
       "2           1  \n",
       "3           1  \n",
       "4           1  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subtree_train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAD4CAYAAADRuPC7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYK0lEQVR4nO3df7DldX3f8ecrYJEgEH7oFlnaJRWriAbLBsmQttdggWoq/oBmHSqL0uI42GBDJwGnHVIYpmiqNDqRdg3IQo2IqIGqVDfgHUxHgUVJAJFhq0RXKIxdArtmJCy++8f5XDms5957WPdz795zn4+ZM+ec9/l+vufzXoZ97ffnSVUhSdKu9guLPQFJ0mQyYCRJXRgwkqQuDBhJUhcGjCSpiz0XewK7i4MPPrhWrVo173I/+tGP2GefffpPaDewnHoF+51ky6lXWNh+77zzzh9W1QtHfWbANKtWrWLjxo3zLjc9Pc3U1FT/Ce0GllOvYL+TbDn1Cgvbb5K/mu0zd5FJkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrrwSv5dZNX5X1iU733w0jcsyvdK0nzcgpEkdWHASJK6MGAkSV10C5gkhyX5SpL7ktyb5NxW//0kP0hyV3u8fmjMBUk2Jbk/yUlD9WOS3N0++3CStPpeST7V6rclWTU0Zm2SB9pjba8+JUmj9TzIvx04r6q+kWRf4M4kG9pnl1XVfxleOMmRwBrgFcCLgT9L8tKqehq4HDgb+DrwReBk4CbgLOCxqnpJkjXA+4HfSnIgcCGwGqj23TdW1WMd+5UkDem2BVNVD1fVN9rrrcB9wKFzDDkFuLaqnqyq7wKbgGOTHALsV1Vfq6oCrgbeNDRmfXt9PXBC27o5CdhQVVtaqGxgEEqSpAWyIKcpt11XrwZuA44H3pPkDGAjg62cxxiEz9eHhm1utafa6x3rtOfvA1TV9iSPAwcN10eMGZ7X2Qy2jFixYgXT09Pz9rJt27aRy533yu3zju1hnDnvrNl6nVT2O7mWU6+w+/TbPWCSvAD4DPDeqnoiyeXAxQx2XV0MfBB4J5ARw2uOOjs55plC1TpgHcDq1atrnF+Am+2X4s5crOtgTp/qtm5/BXCyLad+l1OvsPv02/UssiTPYxAun6iqzwJU1SNV9XRV/QT4GHBsW3wzcNjQ8JXAQ62+ckT9WWOS7AnsD2yZY12SpAXS8yyyAFcA91XVh4bqhwwt9mbgnvb6RmBNOzPscOAI4PaqehjYmuS4ts4zgBuGxsycIXYqcEs7TvMl4MQkByQ5ADix1SRJC6TnLrLjgbcDdye5q9XeB7wtydEMdlk9CLwLoKruTXId8C0GZ6Cd084gA3g3cBWwN4Ozx25q9SuAa5JsYrDlsqata0uSi4E72nIXVdWWLl1KkkbqFjBV9eeMPhbyxTnGXAJcMqK+EThqRP3HwGmzrOtK4Mpx5ytJ2rW8kl+S1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSF90CJslhSb6S5L4k9yY5t9UPTLIhyQPt+YChMRck2ZTk/iQnDdWPSXJ3++zDSdLqeyX5VKvflmTV0Ji17TseSLK2V5+SpNF6bsFsB86rqpcDxwHnJDkSOB+4uaqOAG5u72mfrQFeAZwMfDTJHm1dlwNnA0e0x8mtfhbwWFW9BLgMeH9b14HAhcBrgGOBC4eDTJLUX7eAqaqHq+ob7fVW4D7gUOAUYH1bbD3wpvb6FODaqnqyqr4LbAKOTXIIsF9Vfa2qCrh6hzEz67oeOKFt3ZwEbKiqLVX1GLCBZ0JJkrQA9lyIL2m7rl4N3AasqKqHYRBCSV7UFjsU+PrQsM2t9lR7vWN9Zsz327q2J3kcOGi4PmLM8LzOZrBlxIoVK5ienp63l23bto1c7rxXbp93bA/jzHlnzdbrpLLfybWceoXdp9/uAZPkBcBngPdW1RPt8MnIRUfUao76zo55plC1DlgHsHr16pqampptbj81PT3NqOXOPP8L847t4cHTp7qte7ZeJ5X9Tq7l1CvsPv12PYssyfMYhMsnquqzrfxI2+1Fe3601TcDhw0NXwk81OorR9SfNSbJnsD+wJY51iVJWiA9zyILcAVwX1V9aOijG4GZs7rWAjcM1de0M8MOZ3Aw//a2O21rkuPaOs/YYczMuk4FbmnHab4EnJjkgHZw/8RWkyQtkJ67yI4H3g7cneSuVnsfcClwXZKzgO8BpwFU1b1JrgO+xeAMtHOq6uk27t3AVcDewE3tAYMAuybJJgZbLmvaurYkuRi4oy13UVVt6dSnJGmEbgFTVX/O6GMhACfMMuYS4JIR9Y3AUSPqP6YF1IjPrgSuHHe+kqRdyyv5JUldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHXxnAKmXRn/ql6TkSRNjnkDJsl0kv3ab6z8BfDxJB+ab5wkaXkbZwtm/6p6AngL8PGqOgZ4Xd9pSZKWunECZs921+N/CXy+83wkSRNinIC5iMGdiDdV1R1Jfhl4oO+0JElL3bw3u6yqTwOfHnr/HeCtPSclSVr6Zg2YJB9hxK9Azqiq3+4yI0nSRJhrF9lG4E7g+cA/YrBb7AHgaODp2YdJkjTHFkxVrQdIcibw2qp6qr3/b8CXF2R2kqQla5yD/C8G9h16/4JWkyRpVuP8ouWlwDeTfKW9/6fA73ebkSRpIswZMEl+AbgfeE17AJxfVf+398QkSUvbnAFTVT9J8sGq+jXghgWakyRpAoxzDObLSd6aJN1nI0maGOMcg/kdYB/g6SQ/brWqqv36TUuStNSNcyX/vvMtI0nSjsbZgiHJG4F/0t5OV5U3vZQkzWmc34O5FDgX+FZ7nNtqkiTNapwtmNcDR1fVTwCSrAe+CZzfc2KSpKVt3J9M/qWh1/t3mIckacKMswXzn3nmSv4wOBZzQddZSZKWvHHOIvtkkmngVxkEzO95Jb8kaT7zBkySa4Bbga9W1bf7T0mSNAnGOQbzceAQ4CNJ/k+SzyQ5t/O8JElL3LwBU1W3AJcA/xH4Y2A18O75xiW5MsmjSe4Zqv1+kh8kuas9Xj/02QVJNiW5P8lJQ/VjktzdPvvwzC1rkuyV5FOtfluSVUNj1iZ5oD3WjvdHIUnalca5DuZm4H8Dv8Xgzsq/WlUvG2PdVwEnj6hfVlVHt8cX23ccCawBXtHGfDTJHm35y4GzgSPaY2adZwGPVdVLgMuA97d1HQhcyODuz8cCFyY5YIz5SpJ2oXF2kf0l8LfAUcCrgKOS7D3foKq6Fdgy5jxOAa6tqier6rvAJuDYJIcA+1XV16qqgKuBNw2NWd9eXw+c0LZuTgI2VNWWqnoM2MDooJMkdTTOWWT/DiDJC4B3MDgm83eBvXbyO9+T5AxgI3BeC4FDga8PLbO51Z5qr3es056/3+a4PcnjwEHD9RFjniXJ2Qy2jlixYgXT09PzTn7btm0jlzvvldvnHdvDOHPeWbP1Oqnsd3Itp15h9+l3nLPI3gP8Y+AY4K+AK4Gv7uT3XQ5cDFR7/iDwTganP++o5qizk2OeXaxaB6wDWL16dU1NTc0x9YHp6WlGLXfm+V+Yd2wPD54+1W3ds/U6qex3ci2nXmH36XecCy33Bj4E3FlVP9c/06vqkZnXST4GzNw0czNw2NCiK4GHWn3liPrwmM1J9mRwh4EtrT61w5jpn2fekqTnbpyzyP6gqm77ecMFoB1TmfFmYOYMsxuBNe3MsMMZHMy/vaoeBrYmOa4dXzmDZ35Z80Zg5gyxU4Fb2nGaLwEnJjmgHdw/sdUkSQtorNv174wkn2SwJXFwks0MzuyaSnI0g11WDwLvAqiqe5Ncx+BuzduBc6rq6baqdzM4I21v4Kb2ALgCuCbJJgZbLmvaurYkuRi4oy13UVWNe7KBJGkX6RYwVfW2EeUr5lj+EgbX2+xY38jgDLYd6z8GTptlXVcyOFYkSVok495NWZKk52ScCy3f0q6IfzzJE0m2JnliISYnSVq6xtlF9gHgX1TVfb0nI0maHOPsInvEcJEkPVfjbMFsTPIp4E+BJ2eKVfXZXpOSJC194wTMfsDfMLieZEYBBowkaVbj3IvsHQsxEUnSZJk1YJL8blV9IMlHGHEvr6r67a4zkyQtaXNtwcwc2N+4EBORJE2WWQOmqv5ne14/2zKSJM3GK/klSV0YMJKkLgwYSVIX4/yi5QuBfwOsGl6+qt7Zb1qSpKVunAstb2DwE8l/Bjw9z7KSJAHjBcwvVtXvdZ+JJGmijHMM5vNJXt99JpKkiTLXlfxbGVzBH+B9SZ4Enmrvq6r2W5gpSpKWorkutNx3ISciSZos4/yi5c3j1CRJGjbXLrLnA/sAByc5gMGuMRjcvv/FCzA3SdISNtdZZO8C3ssgTL4xVH8C+KOOc5IkTYC5jsH8IfCHSf5tVX1kAeckSZoA41wH84Mkb9mh9jhwd1U92mFOkqQJME7AnAX8GvCV9n4K+Drw0iQXVdU1neYmSVrCxgmYnwAvr6pHAJKsAC4HXgPcChgwkqSfMc6V/KtmwqV5FHhpVW1hcOGlJEk/Y5wtmK8m+Tzw6fb+rcCtSfYB/rrXxCRJS9s4AXMOg1A5nsG1MFcDn6mqAl7bcW6SpCVs3oBpQXJ9e0iSNJZxbhXzliQPJHk8yRNJtiZ5YiEmJ0lausY5yP8B4I1VtX9V7VdV+45zJ+UkVyZ5NMk9Q7UDk2xogbWh3YJm5rMLkmxKcn+Sk4bqxyS5u3324SRp9b2SfKrVb0uyamjM2vYdDyRZO+afhSRpFxonYB6pqvt2Yt1XASfvUDsfuLmqjgBubu9JciSwBnhFG/PRJHu0MZcDZwNHtMfMOs8CHquqlwCXAe9v6zoQuJDBadTHAhcOB5kkaWGMEzAb25bC29rusreMuLL/Z1TVrcCWHcqnAOvb6/XAm4bq11bVk1X1XWATcGySQ4D9qupr7VjQ1TuMmVnX9cAJbevmJGBDVW2pqseADfxs0EmSOhvnLLL9gL8BThyqFfDZnfi+FVX1MEBVPZzkRa1+KIO7A8zY3GpPtdc71mfGfL+ta3uSx4GDhusjxjxLkrMZbB2xYsUKpqen521g27ZtI5c775Xb5x3bwzhz3lmz9Tqp7HdyLadeYffpd5yzyN6xAPPIiFrNUd/ZMc8uVq0D1gGsXr26pqam5p3o9PQ0o5Y78/wvzDu2hwdPn+q27tl6nVT2O7mWU6+w+/Q7zllkL01y88zB+iSvSvIfdvL7Hmm7vWjPMzfL3AwcNrTcSuChVl85ov6sMUn2BPZnsEtutnVJkhbQOMdgPgZcQLstTFX9JYMD8jvjRmDmrK61wA1D9TXtzLDDGRzMv73tTtua5Lh2fOWMHcbMrOtU4JZ2nOZLwIlJDmgH909sNUnSAhrnGMwvVtXt7ezgGfMecEjySQZ3Xj44yWYGZ3ZdClyX5Czge8BpAFV1b5LrgG+1dZ9TVU+3Vb2bwRlpewM3tQfAFcA1STYx2HJZ09a1JcnFwB1tuYvafdMkSQtonID5YZJ/QDuOkeRU4OH5BlXV22b56IRZlr8EuGREfSNw1Ij6j2kBNeKzK4Er55ujJKmfce9Ftg54WZIfAN8FTu86K0nSkjfvMZiq+k5VvQ54IfCyqvp14M3dZyZJWtLGOcgPQFX9qKq2tre/02k+kqQJMXbA7GDUtSaSJP3UzgbMyAsXJUmaMetB/iRbGR0kYXDKsCRJs5o1YKpq34WciCRpsuzsLjJJkuZkwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqYlECJsmDSe5OcleSja12YJINSR5ozwcMLX9Bkk1J7k9y0lD9mLaeTUk+nCStvleST7X6bUlWLXiTkrTMLeYWzGur6uiqWt3enw/cXFVHADe39yQ5ElgDvAI4Gfhokj3amMuBs4Ej2uPkVj8LeKyqXgJcBrx/AfqRJA3ZnXaRnQKsb6/XA28aql9bVU9W1XeBTcCxSQ4B9quqr1VVAVfvMGZmXdcDJ8xs3UiSFsaei/S9BXw5SQH/varWASuq6mGAqno4yYvasocCXx8au7nVnmqvd6zPjPl+W9f2JI8DBwE/HJ5EkrMZbAGxYsUKpqen5534tm3bRi533iu3zzu2h3HmvLNm63VS2e/kWk69wu7T72IFzPFV9VALkQ1Jvj3HsqO2PGqO+lxjnl0YBNs6gNWrV9fU1NSck4bBX+ijljvz/C/MO7aHB0+f6rbu2XqdVPY7uZZTr7D79Lsou8iq6qH2/CjwOeBY4JG224v2/GhbfDNw2NDwlcBDrb5yRP1ZY5LsCewPbOnRiyRptAUPmCT7JNl35jVwInAPcCOwti22Frihvb4RWNPODDucwcH829vutK1JjmvHV87YYczMuk4FbmnHaSRJC2QxdpGtAD7XjrnvCfxJVf2vJHcA1yU5C/gecBpAVd2b5DrgW8B24Jyqerqt693AVcDewE3tAXAFcE2STQy2XNYsRGOSpGcseMBU1XeAXxlR/3/ACbOMuQS4ZER9I3DUiPqPaQElSVocu9NpypKkCWLASJK6MGAkSV0YMJKkLgwYSVIXBowkqQsDRpLUhQEjSerCgJEkdWHASJK6MGAkSV0YMJKkLhbrB8e0i6zq+ENn571y+5w/pPbgpW/o9t2Slj63YCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXXglv3Zaz7sIzMU7CEhLg1swkqQuDBhJUhcGjCSpCwNGktSFB/m15PQ6ucCfJ5B2LbdgJEldGDCSpC7cRSaNabGu+wF3z2lpmugtmCQnJ7k/yaYk5y/2fCRpOZnYLZgkewB/BPwzYDNwR5Ibq+pbizsz6bnrsfU030kN4JaTfj4TGzDAscCmqvoOQJJrgVMAA0Ya02LuFtyVxgnTGYbqrpOqWuw5dJHkVODkqvrX7f3bgddU1XuGljkbOLu9/YfA/WOs+mDgh7t4urur5dQr2O8kW069wsL2+/er6oWjPpjkLZiMqD0rTatqHbDuOa002VhVq3+eiS0Vy6lXsN9Jtpx6hd2n30k+yL8ZOGzo/UrgoUWaiyQtO5McMHcARyQ5PMnfAdYANy7ynCRp2ZjYXWRVtT3Je4AvAXsAV1bVvbtg1c9pl9oSt5x6BfudZMupV9hN+p3Yg/ySpMU1ybvIJEmLyICRJHVhwIxp0m87k+TKJI8muWeodmCSDUkeaM8HLOYcd5UkhyX5SpL7ktyb5NxWn9R+n5/k9iR/0fr9T60+kf3C4E4eSb6Z5PPt/cT2CpDkwSR3J7krycZWW/SeDZgxDN125p8DRwJvS3Lk4s5ql7sKOHmH2vnAzVV1BHBzez8JtgPnVdXLgeOAc9p/z0nt90ngN6rqV4CjgZOTHMfk9gtwLnDf0PtJ7nXGa6vq6KHrXxa9ZwNmPD+97UxV/S0wc9uZiVFVtwJbdiifAqxvr9cDb1rIOfVSVQ9X1Tfa660M/iI6lMntt6pqW3v7vPYoJrTfJCuBNwB/PFSeyF7nseg9GzDjORT4/tD7za026VZU1cMw+EsZeNEiz2eXS7IKeDVwGxPcb9tldBfwKLChqia53/8K/C7wk6HapPY6o4AvJ7mz3QILdoOeJ/Y6mF1s3tvOaOlJ8gLgM8B7q+qJZNR/5slQVU8DRyf5JeBzSY5a5Cl1keQ3gUer6s4kU4s8nYV0fFU9lORFwIYk317sCYFbMONarredeSTJIQDt+dFFns8uk+R5DMLlE1X12Vae2H5nVNVfA9MMjrdNYr/HA29M8iCDXdm/keR/MJm9/lRVPdSeHwU+x2C3/qL3bMCMZ7neduZGYG17vRa4YRHnsstksKlyBXBfVX1o6KNJ7feFbcuFJHsDrwO+zQT2W1UXVNXKqlrF4P/TW6rqXzGBvc5Isk+SfWdeAycC97Ab9OyV/GNK8noG+3ZnbjtzyeLOaNdK8klgisFtvh8BLgT+FLgO+HvA94DTqmrHEwGWnCS/DnwVuJtn9tO/j8FxmEns91UMDvLuweAflddV1UVJDmIC+53RdpH9+6r6zUnuNckvM9hqgcFhjz+pqkt2h54NGElSF+4ikyR1YcBIkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktTF/wctIWOJjhSLAgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "subtree_train_df['word_count'] = subtree_train_df.sentence.str.split().apply(len)\n",
    "\n",
    "_ = subtree_train_df['word_count'].hist().set_ylabel(\"Length in words\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this setting, removing duplicates has a large effect, since many subtrees are repeated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "subtree_dedup_train_df = sst.train_reader(SST_HOME, include_subtrees=True, dedup=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(159274, 4)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subtree_dedup_train_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neutral     81658\n",
       "positive    42672\n",
       "negative    34944\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subtree_dedup_train_df.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dev and test splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the dev and test splits, we include only the root-level examples, and we do not deduplicate to remain aligned with the original paper. (The dev set has one repeated example, and the test set has none.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_df = sst.dev_reader(SST_HOME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1101, 4)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dev_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "positive    444\n",
       "negative    428\n",
       "neutral     229\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dev_df.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is an associated `sst.test_reader(SST_HOME)` with 2,210 (root-only) examples and no duplicates. As always in our field, you should use the test set only at the very end of your system development, and you should never, ever develop a system on the basis of test-set scores. \n",
    "\n",
    "In a similar vein, you should use the dev set only very sparingly. This will give you a clearer picture of how you will ultimately do on test; over-use of a dev set can lead to over-fitting on that particular dataset with a resulting loss of performance at test time.\n",
    "\n",
    "In the homework and associated bake-off for this course, we will introduce a second dev/test pair involving sentences about restaurants. The goal there is to have a fresh test set, and to push you to develop a system that works both for the SST movie domain and this new domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD4CAYAAAAD6PrjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZDElEQVR4nO3df5DddX3v8efLqBhZCFDg3DTQLnQiNrAazbm0d2i9Z4tXAa9G8Ueh1EmU3tUZrNqmU4N0rkwdZpA2OF6p9oaBMRZlQX4IBbWmGdZoW4obGtiEGAmw0Py4myKQZENurhve94/z3S+H5Zyz3+zu93x3z3k9ZnbO+X6+3/P9vN/5bs57v78+X0UEZmZmAK8pOgAzM5s9XBTMzCzlomBmZikXBTMzS7komJlZ6rVFBzAdJ598cnR3dzdd5uDBgxx77LGtCWgW6cS8nXNncM7Tt2nTpmcj4pR68+Z0Ueju7mZwcLDpMgMDA1QqldYENIt0Yt7OuTM45+mT9HSjeT58ZGZmKRcFMzNLuSiYmVnKRcHMzFIuCmZmlnJRMDOzlIuCmZmlcisKkk6X9ICkbZK2SvpM0n6SpPWSHk9eT6z5zJWSdkjaLundecVmZmb15bmnMAasiojfBH4buELSEmA1sCEiFgMbkmmSeZcAZwMXAF+TNC/H+MzMbILc7miOiD3AnuT9AUnbgEXAcqCSLLYOGAA+l7T3R8Rh4ClJO4BzgX/JK0Zrne7V9+fex6qeMVZO6Gf42vfk3q9ZO1ErnrwmqRvYCJwDPBMRJ9TMez4iTpR0A/BgRNyStN8EfD8i7piwrj6gD6BUKi3r7+9v2vfo6ChdXV0zmM3cMNvyHtq1L/c+SvNh5NAr23oWLci93yLNtu3cCs55+np7ezdFRLnevNzHPpLUBdwJfDYi9ktquGidtldVrIhYC6wFKJfLMdl4IJ04TgrMvrwn/gWfh1U9Y6wZeuWv9PBlldz7LdJs286t4JzzlevVR5JeR7UgfCsi7kqaRyQtTOYvBPYm7TuB02s+fhqwO8/4zMzslfK8+kjATcC2iLi+Zta9wIrk/Qrgnpr2SyQdI+kMYDHwUF7xmZnZq+V5+Og84KPAkKTNSdvngWuB2yVdDjwDfBggIrZKuh14jOqVS1dExJEc4zMzswnyvProJ9Q/TwBwfoPPXANck1dMZmbWnO9oNjOzlIuCmZmlXBTMzCzlomBmZikXBTMzS7komJlZKvdhLsyK1IqB+BrxYHw2F3lPwczMUt5T6CBF/tVsZnOD9xTMzCzlomBmZikXBTMzS7komJlZykXBzMxSLgpmZpZyUTAzs1Sej+O8WdJeSVtq2m6TtDn5GR5/IpukbkmHaub9bV5xmZlZY3nevPYN4Abgm+MNEfH74+8lrQH21Sz/REQszTEeMzObRJ6P49woqbvePEkCPgL8Xl79m5nZ0SvqnMLvAiMR8XhN2xmS/k3SjyT9bkFxmZl1NEVEfiuv7incFxHnTGj/OrAjItYk08cAXRHxC0nLgO8CZ0fE/jrr7AP6AEql0rL+/v6mMYyOjtLV1TUD2cwt9fIe2rWvwdLtoTQfRg4VHcXLehYtyL2PTvz9ds7T19vbuykiyvXmtXxAPEmvBS4Glo23RcRh4HDyfpOkJ4A3AYMTPx8Ra4G1AOVyOSqVStP+BgYGmGyZdlQv75VtPiDeqp4x1gzNnjEehy+r5N5HJ/5+O+d8FXH46J3AzyJi53iDpFMkzUvenwksBp4sIDYzs46W5yWptwL/Apwlaaeky5NZlwC3Tlj8HcCjkh4B7gA+GRHP5RWbmZnVl+fVR5c2aF9Zp+1O4M68YjEzs2x8R7OZmaVcFMzMLOWiYGZmKRcFMzNLuSiYmVnKRcHMzFIuCmZmlnJRMDOzlIuCmZmlXBTMzCzlomBmZikXBTMzS7komJlZykXBzMxSLgpmZpZyUTAzs9SkRUHSeZKOTd7/oaTrJf16/qGZmVmrZdlT+DrwoqS3An8OPA18c7IPSbpZ0l5JW2rarpa0S9Lm5OeimnlXStohabukd08hFzMzm6YsRWEsIgJYDnwlIr4CHJfhc98ALqjT/uWIWJr8fA9A0hKqz24+O/nM1yTNy5KAmZnNnCxF4YCkK4E/BO5PvqxfN9mHImIj8FzGOJYD/RFxOCKeAnYA52b8rJmZzRBVdwKaLCD9J+APgJ9GxI8l/RpQiYgsh5C6gfsi4pxk+mpgJbAfGARWRcTzkm4AHoyIW5LlbgK+HxF31FlnH9AHUCqVlvX39zeNYXR0lK6urslCbTv18h7ata+gaFqjNB9GDhUdxct6Fi3IvY9O/P12ztPX29u7KSLK9ea9ttkHk72CWyLineNtEfEMGc4pNPB14ItAJK9rgI8DqrNs3WoVEWuBtQDlcjkqlUrTDgcGBphsmXZUL++Vq+8vJpgWWdUzxpqhpr/SLTV8WSX3Pjrx99s556vp4aOIOEL1JPOM/MkTESMRcSQiXgJu5OVDRDuB02sWPQ3YPRN9mplZdln+rPq/wJCk9cDB8caI+PTRdiZpYUTsSSY/AIxfmXQv8G1J1wO/CiwGHjra9ZuZ2fRkKQr3Jz9HRdKtQAU4WdJO4AtARdJSqoeGhoFPAETEVkm3A48BY8AVyV6KmZm10KRFISLWSZoP/FpEbM+64oi4tE7zTU2Wvwa4Juv6zcxs5mW5o/m9wGbgB8n0Ukn35hyXmZkVIMvho6upnhAeAIiIzZLOyDEms7bQ3YKrvVb1jL3qqrLha9+Te7/WvrLe0TzxAvfmNzeYmdmclGVPYYukPwDmSVoMfBr453zDMjOzImTZU/hjqmMSHQa+DewDPpNnUGZmVowsewrviYirgKvGGyR9GPhOblGZmVkhsuwpXJmxzczM5riGewqSLgQuAhZJ+l81s46neoOZTVFRV6WYmU2m2eGj3VRHMn0fsKmm/QDwJ3kGZWZmxWhYFCLiEeARSXcDB8eHnUhGTj2mRfGZmVkLZTmn8ENgfs30fOAf8wnHzMyKlKUovCEiRscnkvdvzC8kMzMrSpaicFDS28cnJC0DZtHzrczMbKZkuU/hs8B3JI0/9GYh8Pu5RWRmZoXJMnT2TyW9GTiL6mMzfxYRv8w9MjMza7msD7Q9C1gCvAF4myQiYqrPaTYzs1lq0qIg6QtUn6C2BPgecCHwE8BFwcyszWQ50fwh4Hzg/0TEx4C3kuE+BUk3S9oraUtN219J+pmkRyXdLemEpL1b0iFJm5Ofv51aOmZmNh1ZisKhiHgJGJN0PLAXODPD574BXDChbT1wTkS8Bfg5rxxD6YmIWJr8fDLD+s3MbIZlKQqDyV/0N1Id7uJh4KHJPhQRG4HnJrT9MCLGx016EDjtqKI1M7NcKaLxQ9QkCTgtIv49me4Gjo+IRzOtvLr8fRFxTp15fw/cFhG3JMttpbr3sB/4i4j4cYN19gF9AKVSaVl/f3/TGEZHR+nq6soSbssM7Zr4ILuZV5oPIx12N4lzrupZtKCYYFpkNv6fzttM59zb27spIsr15jUtCgCSNkXEsql03KgoSLoKKAMXR0RIOgboiohfJDfHfRc4OyL2N1t/uVyOwcHBpjEMDAxQqVSmEn5uWjVK6pqhrBeXtQfnXNXuz2iejf+n8zbTOSff63WLQpbDRw9K+s8zGMwK4L8Dl0VSkSLicET8Inm/CXgCeNNM9WlmZtlk+bOqF/iEpKeBg1RvYIvkZPFRkXQB8Dngv0bEizXtpwDPRcQRSWcCi4Enj3b9ZmY2PVmKwoVTWbGkW6ne33CypJ3AF6hebXQMsL56uoIHkyuN3gH8paQx4AjwyYh4ru6KzcwsN1mGuXgaQNKpVO9oziQiLq3TfFODZe8E7sy6bjMzy8ek5xQkvU/S48BTwI+AYeD7OcdlZmYFyHKi+YvAbwM/j4gzqN7d/E+5RmVmZoXIUhR+mVwZ9BpJr4mIB4Cl+YZlZmZFyHKi+QVJXcBG4FuS9gJjk3zGzMzmoCx7CsupPmntT4AfUL2H4L15BmVmZsXIcvXRwZrJdTnGYmZmBcty9dHFkh6XtE/SfkkHJDUdfsLMzOamLOcUrgPeGxHb8g7GzMyKleWcwogLgplZZ2i4pyDp4uTtoKTbqI5cenh8fkTclW9oZjYVrRiFt5F2H6G1EzQ7fFR7hdGLwLtqpgNwUTAzazMNi0LyPGYzM+sgWa4+Wpc8jnN8+kRJN+calZmZFSLLiea3RMQL4xMR8TzwttwiMjOzwmQpCq+RdOL4hKSTyHYpq5mZzTFZvtzXAP8s6Q6qJ5g/AlyTa1RmZlaILMNcfFPSIPB7VB/FeXFEPJZ7ZGZm1nJZDh8REY9FxA0R8dWsBUHSzZL2StpS03aSpPXJsBnrJxyWulLSDknbJb376FMxM7PpylQUpugbwAUT2lYDGyJiMbAhmUbSEuAS4OzkM1+TNC/H2MzMrI7cikJEbASem9C8nJdHWl0HvL+mvT8iDkfEU8AO4Ny8YjMzs/oUEfmtXOoG7ouIc5LpFyLihJr5z0fEiZJuAB6MiFuS9puA70fEHXXW2Qf0AZRKpWX9/f1NYxgdHaWrq2uGMpoZQ7v25d5HaT6MHMq9m1nFORevZ9GC3PuYjf+n8zbTOff29m6KiHK9eZOeaE7GQPoScCrVE80CIiKOn7EIq+ucqG61ioi1wFqAcrkclUql6YoHBgaYbJlWW9mCsWlW9YyxZqizrhx2zsUbvqySex+z8f903lqZc5bDR9cB74uIBRFxfEQcN42CMCJpIUDyujdp3wmcXrPcacDuKfZhZmZT1Oqhs+8FViTvVwD31LRfIukYSWcAi4GHZqhPMzPLKMt+55SGzpZ0K1ABTpa0E/gCcC1wu6TLgWeADyfr2irpduAxYAy4IiKOHHU2ZmY2LVmKwvFMYejsiLi0wazzGyx/Db5T2sysUFnuaPYQ2mZmHaLZk9f+PCKuk/RV6lwJFBGfzjUyMzNruWZ7CuMnlwdbEYiZmRWv2ZPX/j55XddoGTMzay95jn1kZmZzjIuCmZmlXBTMzCyVZeyjU4D/AXTXLh8RH88vLDMzK0KWm9fuAX4M/CPgu4zNzNpYlqLwxoj4XO6RmJlZ4bKcU7hP0kW5R2JmZoVrdkfzAap3Mgv4vKTDwC/J53kKZmY2CzS7ee24VgZiZmbFm/TwkaQNWdrMzGzua3b46A3AsVSfh3AiLz8y83jgV1sQm5mZtVizq48+AXyWagF4uKZ9P/A3OcZkZmYFaXZO4SvAVyT9cUR8daY6lHQWcFtN05nA/wROoHqT3H8k7Z+PiO/NVL9mZja5LPcp7JJ08YS2fcBQROw92g4jYjuwFEDSPGAXcDfwMeDLEfHXR7tOMzObGVmKwuXAfwEeSKYrwIPAmyT9ZUT83TT6Px94IiKeljTpwmZmlq8sN6+9BPxmRHwwIj4ILAEOA78FTPdO50uAW2umPyXpUUk3Jye3zcyshRTxqidtvnIBaSgiemqmRfXQ0TmS/i0i3jaljqXXA7uBsyNiRFIJeJbqDXNfBBbWG3RPUh/QB1AqlZb19/c37Wd0dJSurq6phJiboV37cu+jNB9GDuXezazinIvXs2hB7n3Mxv/TeZvpnHt7ezdFRLnevCyHj34s6T7gO8n0B4GNko4FXphGXBcCD0fECMD4K4CkG4H76n0oItYCawHK5XJUKpWmnQwMDDDZMq22cvX9ufexqmeMNUNZNm/7cM7FG76sknsfs/H/dN5amXOW36YrqBaC86jeq/BN4M6o7mL0TqPvS6k5dCRpYUTsSSY/AGyZxrrNzGwKJi0KyZf/HcnPjJD0RuC/Ub0XYtx1kpZSPXw0PGGemZm1QJaH7FwMfAk4leqewrQHxIuIF4FfmdD20amuz8zMZkaWw0fXAe+NiG15B2NmZsXKcknqiAuCmVlnyLKnMCjpNuC7VO9PACAi7sorKDMzK0aWonA88CLwrpq2AFwUzMzaTJarjz7WikDMzKx4WR6y8yZJGyRtSabfIukv8g/NzMxaLcuJ5huBK6k+n5mIeJTqmEVmZtZmshSFN0bEQxPaxvIIxszMipWlKDwr6TeonlxG0oeAPc0/YmZmc1HWsY/WAm+WtAt4Crgs16jMzKwQk+4pRMSTEfFO4BTgzRHxO1QHrDMzszaT5fARABFxMCIOJJN/mlM8ZmZWoMxFYQI/O9PMrA1N9ekczR/XNkd0t+BhN2Zmc0nDoiDpAPW//AXMzy0iMzMrTMOiEBHHtTIQMzMr3lTPKZiZWRsq5InfkoaBA8ARYCwiypJOAm4Duqk+jvMjEfF8EfGZmXWqIvcUeiNiaUSUk+nVwIaIWAxsSKbNzKyFZtPho+XAuuT9OuD9xYViZtaZFNH6q0slPQU8T/Xqpv8dEWslvRARJ9Qs83xEnFjns31AH0CpVFrW39/ftK/R0VG6urrqzhvatW/KOcx2pfkwcqjoKFrLOXeGejn3LFpQTDAt0ux7bCp6e3s31RyleYVCzikA50XEbkmnAusl/SzrByNiLdWxmCiXy1GpVJouPzAwQKNlVrbxfQqresZYM1TU5i2Gc+4M9XIevqxSTDAt0ux7bKYVcvgoInYnr3uBu4FzgRFJCwGS171FxGZm1slaXhQkHSvpuPH3VJ/9vAW4F1iRLLYCuKfVsZmZdboi9jtLwN2Sxvv/dkT8QNJPgdslXQ48A3y4gNjMzDpay4tCRDwJvLVO+y+A81sdj5mZvWw2XZJqZmYFc1EwM7OUi4KZmaVcFMzMLOWiYGZmKRcFMzNLuSiYmVnKRcHMzFIuCmZmlnJRMDOzlIuCmZmlXBTMzCzlomBmZikXBTMzS7komJlZykXBzMxSLgpmZpYq4hnNp0t6QNI2SVslfSZpv1rSLkmbk5+LWh2bmVmnK+IZzWPAqoh4WNJxwCZJ65N5X46Ivy4gJjMzo5hnNO8B9iTvD0jaBixqdRxmZvZqiojiOpe6gY3AOcCfAiuB/cAg1b2J5+t8pg/oAyiVSsv6+/ub9jE6OkpXV1fdeUO79k09+FmuNB9GDhUdRWs5585QL+eeRQuKCaZFmn2PTUVvb++miCjXm1dYUZDUBfwIuCYi7pJUAp4FAvgisDAiPt5sHeVyOQYHB5v2MzAwQKVSqTuve/X9U4h8bljVM8aaoSKODhbHOXeGejkPX/uegqJpjWbfY1MhqWFRKOTqI0mvA+4EvhURdwFExEhEHImIl4AbgXOLiM3MrJO1/E8MSQJuArZFxPU17QuT8w0AHwC2tDo2M5ubitrrb8c9lCL2O88DPgoMSdqctH0euFTSUqqHj4aBTxQQm5lZRyvi6qOfAKoz63utjsXMzF7JdzSbmVnKRcHMzFIuCmZmlnJRMDOzlIuCmZmlXBTMzCzlomBmZikXBTMzS7komJlZykXBzMxSLgpmZpZyUTAzs5SLgpmZpVwUzMws1VnP8TMzm0GterjPqp4xVk7oK68H/HhPwczMUi4KZmaWmnVFQdIFkrZL2iFpddHxmJl1kllVFCTNA/4GuBBYQvW5zUuKjcrMrHPMqqIAnAvsiIgnI+L/Af3A8oJjMjPrGIqIomNISfoQcEFE/FEy/VHgtyLiUzXL9AF9yeRZwPZJVnsy8GwO4c52nZi3c+4Mznn6fj0iTqk3Y7Zdkqo6ba+oWhGxFlibeYXSYESUpxvYXNOJeTvnzuCc8zXbDh/tBE6vmT4N2F1QLGZmHWe2FYWfAoslnSHp9cAlwL0Fx2Rm1jFm1eGjiBiT9CngH4B5wM0RsXWaq818qKnNdGLezrkzOOcczaoTzWZmVqzZdvjIzMwK5KJgZmapti4KnTJkhqRhSUOSNksaTNpOkrRe0uPJ64lFxzkdkm6WtFfSlpq2hjlKujLZ7tslvbuYqKenQc5XS9qVbOvNki6qmdcOOZ8u6QFJ2yRtlfSZpL1tt3WTnIvZ1hHRlj9UT1Q/AZwJvB54BFhSdFw55ToMnDyh7TpgdfJ+NfClouOcZo7vAN4ObJksR6pDpDwCHAOckfwezCs6hxnK+Wrgz+os2y45LwTenrw/Dvh5klvbbusmOReyrdt5T6HTh8xYDqxL3q8D3l9cKNMXERuB5yY0N8pxOdAfEYcj4ilgB9XfhzmlQc6NtEvOeyLi4eT9AWAbsIg23tZNcm4k15zbuSgsAv69Znonzf+h57IAfihpUzIMCEApIvZA9ZcOOLWw6PLTKMd23/afkvRocnhp/DBK2+UsqRt4G/CvdMi2npAzFLCt27koTDpkRhs5LyLeTnV02SskvaPogArWztv+68BvAEuBPcCapL2tcpbUBdwJfDYi9jdbtE7bnMy7Ts6FbOt2LgodM2RGROxOXvcCd1PdlRyRtBAged1bXIS5aZRj2277iBiJiCMR8RJwIy8fNmibnCW9juqX47ci4q6kua23db2ci9rW7VwUOmLIDEnHSjpu/D3wLmAL1VxXJIutAO4pJsJcNcrxXuASScdIOgNYDDxUQHwzbvyLMfEBqtsa2iRnSQJuArZFxPU1s9p2WzfKubBtXfSZ95zP6l9E9Uz+E8BVRceTU45nUr0S4RFg63iewK8AG4DHk9eTio51mnneSnUX+pdU/1K6vFmOwFXJdt8OXFh0/DOY898BQ8CjyZfDwjbL+XeoHgp5FNic/FzUztu6Sc6FbGsPc2FmZql2PnxkZmZHyUXBzMxSLgpmZpZyUTAzs5SLgpmZpVwUzMws5aJgZmap/w9Pai2slbe1NgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = dev_df.sentence.str.len().hist().set_ylabel(\"Length in characters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD4CAYAAAAD6PrjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAS1ElEQVR4nO3df5BlZX3n8fcn6CphkB8BehHYNNnCKDLKLh01pdntiSllJQlGooVlUmBMJqnCDdlMKhmsVGlMUTvlFiYpjFs1WdDZaBzxJyysG8nEDqYqRmfUOCBSzOKE5cfOlCsCYyzcwW/+uKcPl6F75kz33Hu6+75fVV19znN/nO95ZuZ+5jznPuekqpAkCeCH+i5AkrRyGAqSpJahIElqGQqSpJahIElqPavvApbjtNNOq+npaQC++93vcsIJJ/RbUI8mff/BPgD7AOyDLvu/a9eub1XV6Qs9tqpDYXp6mp07dwIwNzfH7OxsvwX1aNL3H+wDsA/APuiy/0n+cbHHHD6SJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLVW9YxmHZ3pzbf1tu29Wy7pbduSuvNIQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSa2RhUKSc5J8LsndSe5KcnXTfmqS25Pc2/w+Zeg11yTZk+SeJK8dVW2SpIWN8kjhILCpql4EvAK4Ksn5wGZgR1WdB+xo1mkeuxx4MXAx8P4kx42wPknSIUYWClX1cFV9uVl+HLgbOAu4FNjWPG0b8Ppm+VJge1U9UVXfBPYALxtVfZKkZ0pVjX4jyTRwB3ABcH9VnTz02CNVdUqS9wFfqKoPNe03AJ+pqo8f8l4bgY0AU1NTF23fvh2AAwcOsG7dupHvy0rVZf93P/jomKp5pvVnnTTybUz63wGwD8A+6LL/GzZs2FVVMws99qyRVDUkyTrgE8BvVdVjSRZ96gJtz0isqtoKbAWYmZmp2dlZAObm5phfnkRd9v/KzbeNp5gF7H3L7Mi3Mel/B8A+APtgufs/0m8fJXk2g0D4cFV9smnel+TM5vEzgf1N+wPAOUMvPxt4aJT1SZKebpTfPgpwA3B3Vb136KFbgCua5SuAm4faL0/ynCTnAucBXxxVfZKkZxrl8NErgV8Gdif5atP2DmALcFOStwH3A28EqKq7ktwEfJ3BN5euqqonR1ifJOkQIwuFqvpbFj5PAPDqRV5zLXDtqGqSJB2eM5olSa2Rf/tI6tN0n9+42nJJb9uWlsojBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy/spaCzGcV+DTesPcmWP90+Q1gKPFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktQ6qlBIckqSl4yqGElSv44YCknmkjwvyanAPwAfSPLe0ZcmSRq3LhfEO6mqHkvyq8AHquqdSb426sLWslFcHM6LwUk6FroMHz0ryZnAm4BbR1yPJKlHXULh3cBfAnuq6ktJfgy4d7RlSZL6cMTho6r6GPCxofX7gMtGWZQkqR+LhkKS64Fa7PGq+s2RVCRJ6s3hho92AruA5wL/lsGQ0b3AhcCTI69MkjR2i4ZCVW2rqm3AecCGqrq+qq4HXs0gGA4ryY1J9ie5c6jtXUkeTPLV5ud1Q49dk2RPknuSvHZZeyVJWpIuJ5qfD5w4tL6uaTuSDwIXL9D+R1V1YfPzPwGSnA9cDry4ec37kxzXYRuSpGOoyzyFLcBXknyuWf/3wLuO9KKquiPJdMc6LgW2V9UTwDeT7AFeBvxdx9dLko6BVC16LpkkPwS8ArgPeHnT/PdV9X87vfkgFG6tqgua9XcBVwKPMThnsamqHknyPuALVfWh5nk3AJ+pqo8v8J4bgY0AU1NTF23fvh2AAwcOsG7dui5l9W73g48e8/ecOh72fe+Yv+2qstL6YP1ZJ419m6vp38GoTHofdNn/DRs27KqqmYUeO+yRQlX9IMl1VfWTwM1LL7P1X4E/ZPCtpj8ErgN+BchCm1+kpq3AVoCZmZmanZ0FYG5ujvnllW4UM483rT/Idbu7HPitXSutD/a+ZXbs21xN/w5GZdL7YLn73+WcwmeTXJZkoQ/uo1JV+6rqyar6AfBnDIaIAB4Azhl66tnAQ8vdniTp6HQJhd9mMHnt+0keb34eW8rGmstlzPsFYP6bSbcAlyd5TpJzGXzj6YtL2YYkaem6zGg+8UjPWUiSjwCzwGlJHgDeCcwmuZDB0NBe4NebbdyV5Cbg68BB4Kqqci6EJI1ZpwHYJD8P/Ltmda6qjnhhvKp68wLNNxzm+dcC13apR5I0GkcMhSRbgJ8APtw0XZ3kVVW1eaSVSavcKC6RfiSb1h9kduxb1VrS5UjhdcCFzclhkmwDvgIYCpK0xnS9HefJQ8vj//K1JGksuhwp/GeemtEcBucWrhlpVZKkXnT59tFHkswxOK8Q4Pe6zmiWJK0uXU40/zlwB/D5qvrG6EuSJPWlyzmFDwBnAtcn+d9JPpHk6hHXJUnqQZfho79O8jcMho82AL/B4BLXfzLi2iRJY9Zl+GgHcAKDy1h/HviJqto/6sIkSePXZfjoa8D3gQuAlwAXJDl+pFVJknrRZfjoPwEkWQe8lcE5hn8JPGe0pUmSxq3L8NHbgZ8CLgL+EbiRwTCSJGmN6TJ57XjgvcCuqjo44nokST3qMnz0X8ZRiCSpf12vfSRJmgCGgiSpZShIklpHDIUkb0hyb5JHkzy2nHs0S5JWti7fPnoP8HNVdfeoi5Ek9avL8NE+A0GSJkOXI4WdST4KfBp4Yr6xqj45qqIkSf3oEgrPA/4JeM1QWwGGgiStMV0mr711HIVIkvq3aCgk+d2qek+S6xkcGTxNVf3mSCuTJI3d4Y4U5k8u7xxHIZKk/i0aClX1P5rf28ZXjiSpT85oliS1DAVJUstQkCS1utx57XTg14Dp4edX1a+MrixJUh+6TF67mcHtN/8KeHK05UiS+tQlFH64qn5v5JVIknrX5ZzCrUleN/JKJEm9O9yM5scZzGQO8I4kTwD/v1mvqnreeEqUJI3L4SavnTjOQiRJ/ety57UdXdokSavfoqGQ5LlJfgQ4LckpSU5tfqaB5x/pjZPcmGR/kjuH2k5Ncntze8/bk5wy9Ng1SfYkuSfJa5e5X5KkJTjckcKvM7gY3guBLwO7mp+bgT/t8N4fBC4+pG0zsKOqzgN2NOskOR+4HHhx85r3Jzmu815Iko6JRUOhqv6kqs4Ffqeqzh36eWlVve9Ib1xVdwDfPqT5UmD+AnvbgNcPtW+vqieq6pvAHuBlR7kvkqRlStUzbpXw9Cckb1ig+VFgd1XtP8Jrp4Fbq+qCZv07VXXy0OOPVNUpSd4HfKGqPtS03wB8pqo+vsB7bgQ2AkxNTV20fft2AA4cOMC6desOuy8rxe4HHz3m7zl1POz73jF/21XFPhj0wRmnntR3Gb1aTZ8Fo9Bl/zds2LCrqmYWeqzL5LW3AT8JfK5ZnwW+ALwgybur6s+7l7uoLNC2YFpV1VZgK8DMzEzNzs4CMDc3x/zySnfl5tuO+XtuWn+Q63Z3+eNcu+yDQR+8aZX8OxiV1fRZMArL3f8uk9d+ALyoqi6rqsuA84EngJcDRzvTeV+SMwGa3/NHGg8A5ww972zgoaN8b0nSMnUJhemq2je0vh94QVV9m8FktqNxC3BFs3wFg5PW8+2XJ3lOknOB84AvHuV7S5KWqcux9ueT3Ap8rFm/DLgjyQnAdxZ7UZKPMBhqOi3JA8A7gS3ATUneBtwPvBGgqu5KchPwdeAgcFVVefE9SRqzLqFwFYMgeCWDsf//DnyiBmeoNyz2oqp68yIPvXqR518LXNuhHknSiBwxFJoP/483P5KkNazLZS7e0MxAfjTJY0keT/LYOIqTJI1Xl+Gj9wA/V1V3j7oYSVK/unz7aJ+BIEmTocuRws4kHwU+zWB+AgBV9clRFSVJ6keXUHge8E/Aa4baCjAUJGmN6fLto7eOoxBJUv+6fPvoBUl2zN8XIclLkvz+6EuTJI1blxPNfwZcQ3NJi6r6GoN7H0iS1pguofDDVXXodYgOjqIYSVK/uoTCt5L8a5pLWSf5ReDhkVYlSepF12sfbQVemORB4JvAW0ZalSSpF0c8Uqiq+6rqZ4DTgRdW1auAXxh5ZZKksesyfARAVX23qh5vVn97RPVIknrUORQOsdDtMyVJq9xSQ2HB+ydLkla3RU80J3mchT/8Axw/sookLcv05tt62/beLZf0tm0dG4uGQlWdOM5CJEn9W+rwkSRpDTIUJEktQ0GS1Ooyo3nN6vOEnCStRB4pSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqdXLpbOT7AUeB54EDlbVTJJTgY8C08Be4E1V9Ugf9UnSpOrzSGFDVV1YVTPN+mZgR1WdB+xo1iVJY7SSho8uBbY1y9uA1/dXiiRNpr5CoYDPJtmVZGPTNlVVDwM0v8/oqTZJmlipqvFvNHl+VT2U5AzgduA/ArdU1clDz3mkqk5Z4LUbgY0AU1NTF23fvh2AAwcOsG7duqOqY/eDjy55H1aaqeNh3/f6rqJf9kH/fbD+rJP623hjKZ8Fa0mX/d+wYcOuoaH7p+klFJ5WQPIu4ADwa8BsVT2c5Exgrqp+/HCvnZmZqZ07dwIwNzfH7OzsUW17Ld2jedP6g1y3e6JvuW0f0H8f7N1ySW/bnreUz4K1pMv+J1k0FMY+fJTkhCQnzi8DrwHuBG4BrmiedgVw87hrk6RJ18d/KaaATyWZ3/5fVNX/SvIl4KYkbwPuB97YQ22SNNHGHgpVdR/w0gXa/x/w6nHXI0l6ykr6SqokqWeGgiSpZShIklqGgiSpZShIklqTPdNH0jHV14TQlTBpbq3wSEGS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEmtZ/VdgCQt1/Tm29rlTesPcuXQ+ijt3XLJWLYzTh4pSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJazmiWpCWaHtPM6YWMajb1ijtSSHJxknuS7Emyue96JGmSrKhQSHIc8KfAfwDOB96c5Px+q5KkybGiQgF4GbCnqu6rqu8D24FLe65JkiZGqqrvGlpJfhG4uKp+tVn/ZeDlVfX2oedsBDY2qz8O3NMsnwZ8a4zlrjSTvv9gH4B9APZBl/3/0ao6faEHVtqJ5izQ9rTUqqqtwNZnvDDZWVUzoypspZv0/Qf7AOwDsA+Wu/8rbfjoAeCcofWzgYd6qkWSJs5KC4UvAeclOTfJvwAuB27puSZJmhgravioqg4meTvwl8BxwI1VdVfHlz9jSGnCTPr+g30A9gHYB8va/xV1olmS1K+VNnwkSeqRoSBJaq36UJjEy2IkuTHJ/iR3DrWdmuT2JPc2v0/ps8ZRSnJOks8luTvJXUmubtonqQ+em+SLSf6h6YM/aNonpg/mJTkuyVeS3NqsT1QfJNmbZHeSrybZ2bQtuQ9WdShM8GUxPghcfEjbZmBHVZ0H7GjW16qDwKaqehHwCuCq5s99kvrgCeCnq+qlwIXAxUlewWT1wbyrgbuH1iexDzZU1YVD8xOW3AerOhSY0MtiVNUdwLcPab4U2NYsbwNeP86axqmqHq6qLzfLjzP4QDiLyeqDqqoDzeqzm59igvoAIMnZwCXAfxtqnqg+WMSS+2C1h8JZwP8ZWn+gaZtEU1X1MAw+NIEzeq5nLJJMA/8G+HsmrA+aYZOvAvuB26tq4voA+GPgd4EfDLVNWh8U8Nkku5rLAMEy+mBFzVNYgiNeFkNrV5J1wCeA36qqx5KF/jqsXVX1JHBhkpOBTyW5oOeSxirJzwL7q2pXktmey+nTK6vqoSRnALcn+cZy3my1Hyl4WYyn7EtyJkDze3/P9YxUkmczCIQPV9Unm+aJ6oN5VfUdYI7BeaZJ6oNXAj+fZC+DoeOfTvIhJqsPqKqHmt/7gU8xGFZfch+s9lDwshhPuQW4olm+Ari5x1pGKoNDghuAu6vqvUMPTVIfnN4cIZDkeOBngG8wQX1QVddU1dlVNc3g3/5fV9UvMUF9kOSEJCfOLwOvAe5kGX2w6mc0J3kdg3HF+ctiXNtvRaOX5CPALINL5O4D3gl8GrgJ+FfA/cAbq+rQk9FrQpJXAZ8HdvPUWPI7GJxXmJQ+eAmDE4jHMfjP3U1V9e4kP8KE9MGwZvjod6rqZyepD5L8GIOjAxicDviLqrp2OX2w6kNBknTsrPbhI0nSMWQoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqfXPqcrNUmBi8m4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dev_df['word_count'] = dev_df.sentence.str.split().apply(len)\n",
    "\n",
    "_ = dev_df['word_count'].hist().set_ylabel(\"Length in words\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEcCAYAAAAmzxTpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbA0lEQVR4nO3de5hdVZ3m8e9LEuUSBCJSk8ilBqHtpKOgqRFoQlPpiKKipGlQg0DQ+KRpRxQvY0LDjNAaDWPP2POgPRhJhjBouNlyiw2EkELDI7fINalwGcJN0iB3gtyCv/ljrSInRV1O1TmVU3XW+3me85y999l7r7XPqvOeXWtfjiICMzNrbts0ugJmZjb0HPZmZgVw2JuZFcBhb2ZWAIe9mVkBHPZmZgVw2NtWIykk7dPoejSSpHZJj/Xxel3eI0knSlpV5bxnSLpgkOUMelnbuhz2BZL0kKSXJW2U9KykZZL2aHS9ugwkqMysOg77cn0yIsYC44EngLMbXJ8hI2l0o+tg1mgO+8JFxCvApcCkrmmSdpJ0vqQ/SHpY0umStpE0TtJjkj6Z5xsr6QFJJ+Tx8ySdI2m5pBcl3SBpr57K7aOMicA5wEH5P4/neln+P0r6dS7nOkk/7upOkNSau0NmS3oEuD6v+/Rc1pO57J3y/G/pWsn//Xw4D58h6VJJF+Xyfidpv4p5J0j6Rd6W9ZK+UvHadvl9eVbSWuA/VdEsH5f0oKSnJP0g1/3tkp6R9L6Kde+W/0N7V38rlPS/JD0q6QVJqyUd0m2WbQezfTZyOOwLJ2l74DPATRWTzwZ2AvYGDgVOAD4fEc8AXwB+Kmk34IfAHRFxfsWynwO+A+wK3AH8rJeieyujEzgJ+G1EjI2InXtZ/ufALcA7gTOA43uY51BgIvBR4MT8mJbLHAv8qJd19+RI4BJgXC77MkljJG0DXAncCbwbmA6cIumjeblvA+/Jj48Cs6oo62+ANuCDudwvRMSrwIXAcRXzzQSui4g/VLHOW4H9K+p/iaRt67B9NlJEhB+FPYCHgI3Ac8Am4HHgffm1UcCrwKSK+f8O6KgYPxu4Oy/3zorp5wEXVoyPBd4A9sjjAezTXxmkUF7VR/33zPXevmLaBcAFebg1l7V3xesrgC9VjL8XeB0YDbQDj/XwHn04D58B3FTx2jbABuAQ4ADgkW7Lngr8nzz8IHB4xWtzupfVbdnoNv+XgBV5+ADgUWCbPH4b8Ole1tPfe/gssF8dtu+Mrvfdj+H9cF9muWZExHWSRpH26m6QNIkUNm8DHq6Y92HSXl2XhcCXge9FxNPd1vto10BEbJT0DDChcjppr7+/MvoyAXgmIv7YrdzuB5kry5zQQ3mjgZYqy6zcrj/lbp8JpPdrQrfuplHAbyrKraxHZR36LSvPPyGXe7Okl4BDJW0gfXFeUU3lJX0D+GJFnd9Baoe3lDnA7bMRwt04hYuINyLiX0l74FOBp0h7vJV97XsCvwfIXw4/Ac4H/l5vPU3wzcCVNJbULfB4t3n6LIMUMH3ZAIzLXVBvKbdy8yqGH++hvE2kg9MvAW+uK29j937wyu3aBtg9r/NRYH1E7Fzx2DEiPl5R18q67dnPtnXflj3Z8v1bQurKOR64NNIxlz7l/vm5wKeBXSJ1jT0PqA7bZyOEw75wSo4EdgE6I+IN4GJgvqQd8wHWr5O6SQD+IT9/Afgn4Pwcjl0+LmmqpLeR+u5vjojKPVWqKOMJYPe8jreIiIdJXRhnSHqbpIOAT/azqUuBr+UDu2OB7wEXAdcBB5IOUH5C0hjgdODt3ZafIukopTN7TiF1Q91EOm7wgqS5+WDsKEmTJXUdiL0YOFXSLpJ2B07up54A/yXPvwfw1VzPLv+X1Kd/HOkLtxo7kr7Y/gCMlvTfSHv29dg+GyEc9uW6UtJG4AVgPjArItbk104m7e0+CKwiHbBbLGkKKZRPyIF9FmnveV7Fen9OOij5DDCFdMC2Jz2WkV+7HlgD/Lukp3pZ/nPAQcDTwHdJgfhqH9u7mBSUvwbWA6+wOXhfJvWNn0v67+IloPuFT5eTDmQ/S9qrPioiXs/vwydJBz/Xk/5rOZd08BngTFJXzHrg2lyH/lwO3Et6b5YBi7peiIjHgN+R3vdqu1KuAf4NuC/X5RW27CqqZftshFCEf7zE6kPSeaSDj6c3oOyLgHUR8e1eXhfp7/1P3aZ3kA4wntvHus8A9omI43qbp94knQh8MSKm9vDaYuDxRrzPNnJ5z96GJUmfl3RlxfgDki6uGH9C0hGSDpZ0L6k/+mhJf1kxT4ek+ZJuBP4I7C3pMEnrJD0v6Uds2W/dl30kdebz0NdK+mAuY2Iu5zlJayR9qlv5X6wY3+LKYKVrAU6SdL/Sefg/zt1qvV5rIKkVOIqKvX2zajjsbbi6AThE6YKi8cAY4GAASXsDO5AOFK8i9T9/kdSds0zSOyvWczzpdMcdSQclf0Hqk98V+H9d6+zHJOD9pGsB3gF8Cng69+9fSeqe2Y3ULfQzSe8dwHYeQbrQaj/SF9ZHo5drDSR9B7gH+EFErB9AGWYOe6ufiDixXl0LEfEg8CKpr/hQUr/z7yX9eR6/nnSs4NaIGB8RiyJiKbCOLQ/WnhcRayJiE/AxYG1EXBoRrwP/DPx7FdXZCTg1Im6N5IF8kPhA0rUECyLitYi4HriKdLFTtRZExHMR8QiwMm9vjyLiv+bwnz+A9ZsB+Dx7G9ZuIF3wtE8efo4U9Afl8e7nzsNbz9fvfq595fnkIan7gcqe7EH6L6C7CcCj3Y4DDOR6Adjyy+aPpC8Ps7rznr0NZ11hf0gevoEU9ofm4e7nzsOW5+vDlufab3HOez5oW83dPh8l3e6gu8eBPfJ56T2Vv8X5+8B/qKKsLj5zwurKYW/D2Q2ke9lsl085/A1wOOl+OLcDvwL+TNKxkkZL+gypf/2qXta3DPiLivPJv0J1AXwu8E1JU/IB1H3ytQE3kwL9W0r3kWkndSFdmJe7AzhK0vZKF5/NHsC293mtgdlAOext2IqI+0j38PlNHn+BdO75jfnK36dJBzi/QTrf/lvAERHR47n5efoxwII8/77AjVXU4xLStQg/Jx1HuAwYFxGvkQ7Wfox0/vm/kK5BWJcX/SHwGim4l9D7TeF6Us21BmZV83n2ZmYF8J69mVkBHPZmgNKPrmzs4XFOo+tmVg/uxjEzK4D37M3MCrBVL6raddddo7W1dWsW2RAvvfQSO+ywQ6OrYXXgtmwepbTl6tWrn4qIt/wu8VYN+9bWVm677batWWRDdHR00N7e3uhqWB24LZtHKW0pqcdfQ6sq7CU9RDq/+A1gU0S0SRpHuod4K+n3Oj8dEc/Wo7JmZlZfA+mznxYR+0dEWx6fR/oh5H1JP+Y8r/dFzcyskWo5QHsk6apA8vOMmmtjZmZDoqpTLyWtJ/1cWQA/iYiFkp7rus92nufZiNilh2XnkO4nTktLy5QLL7yw+yxNZ+PGjYwd65sXNgO3ZfMopS2nTZu2uqIH5k3VHqA9OCIel7QbsFzSun6XyCJiIbAQoK2tLUo4QFLKgaASuC2bR+ltWVU3TkQ8np+fBH4JfAh4Iv+CEPn5yaGqpNnWtnTpUiZPnsz06dOZPHkyS5cubXSVzGrS7569pB2AbSLixTz8EeAfgSuAWaQ7CM4i/Tq92Yi3dOlSTjvtNBYtWsQbb7zBqFGjmD073Z145syB/AiV2fBRzZ59C7BK0p3ALcCyiLiaFPKHSbofOCyPm4148+fPZ9GiRUybNo3Ro0czbdo0Fi1axPz5/jVAG7n63bPPvwW6Xw/TnwamD0WlzBqps7OTqVOnbjFt6tSpdHZ2NqhGZrXzvXHMupk4cSKrVq3aYtqqVauYOHFig2pkVjuHvVk3p512GrNnz2blypVs2rSJlStXMnv2bE477bRGV81s0LbqvXHMRoKug7Ann3wynZ2dTJw4kfnz5/vgrI1oDnuzHsycOZOZM2cWf262NQ9345hZU/M1E4n37M2safmaic28Z29mTcvXTGzmsDezpuVrJjZzN46ZNa2JEydy5plnctlll715ZtWMGTOKvGbCYW9mTWvatGmcddZZnHXWWUyaNIm1a9cyd+5cTjrppEZXbatz2JtZ01q5ciVz585l8eLFb+7Zz507l8suu6zRVdvqHPZm1rQ6Ozu5/fbb+e53v/vmNROvv/463//+9xtdta3OB2jNrGn5PkebOezNrGn5PkebuRvHzJqW73O0mcPezEY8SVXNt2bNGo499liOPfbYXueJiHpVa1hxN46ZjXgR0e9jr7lXVTVfs3LYm5kVwGFvZlYAh72ZWQF8gHaAqj0Q1J9m7hs0s+HHe/YDVK8DQWZmW5PD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MC+Dx7K5avmbCSeM/eiuVrJqwkDnszswI47M3MCuCwNzMrQNVhL2mUpNslXZXHx0laLun+/LzL0FXTzMxqMZA9+68CnRXj84AVEbEvsCKPm5nZMFRV2EvaHfgEcG7F5COBJXl4CTCjrjUzM7O6qfY8+38GvgXsWDGtJSI2AETEBkm79bSgpDnAHICWlhY6OjoGXdmRpJTtLIHbsnmU3Jb9hr2kI4AnI2K1pPaBFhARC4GFAG1tbdHePuBVjDxXL6OI7SyB27J5FN6W1ezZHwx8StLHgW2Bd0i6AHhC0vi8Vz8eeHIoK2pmZoPXb599RJwaEbtHRCvwWeD6iDgOuAKYlWebBVw+ZLU0M7Oa1HKe/QLgMEn3A4flcTMzG4YGdCO0iOgAOvLw08D0+lfJzMzqzVfQmpkVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBeg37CVtK+kWSXdKWiPpzDx9nKTlku7Pz7sMfXXNzGwwqtmzfxX464jYD9gfOFzSgcA8YEVE7AusyONmZjYM9Rv2kWzMo2PyI4AjgSV5+hJgxlBU0MzMaldVn72kUZLuAJ4ElkfEzUBLRGwAyM+7DVktzcysJqOrmSki3gD2l7Qz8EtJk6stQNIcYA5AS0sLHR0dg6jmyFPKdpbAbdk8Sm7LqsK+S0Q8J6kDOBx4QtL4iNggaTxpr7+nZRYCCwHa2tqivb29thqPBFcvo4jtLIHbsnkU3pbVnI3zrrxHj6TtgA8D64ArgFl5tlnA5UNURzMzq1E1e/bjgSWSRpG+HC6OiKsk/Ra4WNJs4BHgmCGsp5mZ1aDfsI+Iu4AP9DD9aWD6UFTKzMzqy1fQmpkVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBRjQ/ezNzBphvzOv5fmXX695Pa3zltW0/E7bjeHOb3+k5no0gsPezIa9519+nYcWfKKmdXR0dNT84yW1flk0krtxzMwK4LA3MyuAu3Eq1KtfEMruGxwO3JZmW3LYV6hHvyC4b3A4cFuabcndOGZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlaAfsNe0h6SVkrqlLRG0lfz9HGSlku6Pz/vMvTVNTOzwahmz34T8I2ImAgcCPxnSZOAecCKiNgXWJHHzcxsGOo37CNiQ0T8Lg+/CHQC7waOBJbk2ZYAM4aojmZmVqMB9dlLagU+ANwMtETEBkhfCMBuda+dmZnVRdW/QStpLPAL4JSIeEFStcvNAeYAtLS00NHRMYhqbj31qN/GjRvrsp7h/l4Nd27L5lLre1h8W0ZEvw9gDHAN8PWKafcC4/PweODe/tYzZcqUGM72mntVXdazcuXKmtdRr7qUym3ZXOrxHpbSlsBt0UP+VnM2joBFQGdE/M+Kl64AZuXhWcDldfr+MTOzOqumG+dg4Hjgbkl35Gn/ACwALpY0G3gEOGZIamhmZjXrN+wjYhXQWwf99PpWp7F2nDiP9y2p0xmkS/qfpe+6AHyiHjUxG/Hq9tks+HNZ9QHaErzYuYCHFtTekB0dHbS3t9e0jtZ5y2quh1mzqMdns/TPpW+XYGZWAIe9mVkBHPZmZgVw2JuZFcBhb2ZWAIe9mVkBfOqlNSVfM2G2JYe9NSVfM2G2JXfjmJkVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQEc9mZmBXDYm5kVwGFvZlYAh72ZWQF8P3szGxHq8rsAV9e2jp22G1N7HRrEYW9mw149foimdd6yuqxnpHI3jplZARz2ZmYFcDdON3X7vdCC+waHC7el2WYO+wr16s8rvW9wOHBbmm3J3ThmZgVw2JuZFcBhb2ZWgH7DXtJiSU9Kuqdi2jhJyyXdn593GdpqmplZLarZsz8POLzbtHnAiojYF1iRx83MbJjqN+wj4tfAM90mHwksycNLgBn1rZaZmdXTYPvsWyJiA0B+3q1+VTIzs3ob8vPsJc0B5gC0tLTQ0dEx1EUOC6VsZwncls2j5LYcbNg/IWl8RGyQNB54srcZI2IhsBCgra0t2tvbB1nkCHL1MorYzhK4LZtH4W052G6cK4BZeXgWcHl9qmNmZkOhmlMvlwK/Bd4r6TFJs4EFwGGS7gcOy+NmZjZM9duNExEze3lpep3rYmZmQ8RX0JqZFcBhb2ZWAIe9mVkBHPZmZgVw2JuZFcBhb2ZWAIe9mVkBHPZmZgVw2JuZFcBhb2ZWAIe9mVkBHPZmZgVw2JuZFcBhb2ZWAIe9mVkBHPZmZgVw2JuZFcBhb2ZWAIe9mVkBHPZmZgVw2JuZFcBhb2ZWAIe9mVkBHPZmZgUY3egKmDWKpOrmO6vv1yOiDrUxG1res7diRUS/j5UrV/Y7j9lI4LA3MyuAw97MrADusx8g9/OaDT/1+lxC8342vWc/QO7nNRt+6vW5bObPpsPezKwADnszswI47M3MClBT2Es6XNK9kh6QNK9elTIzs/oadNhLGgX8GPgYMAmYKWlSvSpmZmb1U8ue/YeAByLiwYh4DbgQOLI+1TIzs3qq5Tz7dwOPVow/BhzQfSZJc4A5AC0tLXR0dNRQ5MiwcePGIrazBG7L5lF6W9YS9j1dxfCWk1QjYiGwEKCtrS3a29trKHJk6OjooITtLIHbsnmU3pa1hP1jwB4V47sDj/e1wOrVq5+S9HANZY4UuwJPNboSVhduy+ZRSlvu1dNEDfaKMUmjgfuA6cDvgVuBYyNizWBr2Cwk3RYRbY2uh9XObdk8Sm/LQe/ZR8QmSV8GrgFGAYsd9GZmw1NNN0KLiF8Bv6pTXczMbIj4CtqhsbDRFbC6cVs2j6LbctB99mZmNnJ4z97MrAAO+yEmaWdJX6oYnyDp0kbWyQZGUqukYwe57MZ618cGRtJJkk7IwydKmlDx2rml3ObF3ThDTFIrcFVETG50XWxwJLUD34yII3p4bXREbOpj2Y0RMXYIq2cDIKmD1Ja3NbouW1vxe/Z5r61T0k8lrZF0raTtJL1H0tWSVkv6jaQ/z/O/R9JNkm6V9I9de26SxkpaIel3ku6W1HWfoAXAeyTdIekHubx78jI3S/qLirp0SJoiaQdJi3MZt1esywZgEG17nqSjK5bv2itfAByS2/Bree/wEklXAtf20fZWo9yG6yQtkXSXpEslbS9pev5s3J0/K2/P8y+QtDbP+0952hmSvpnbtg34WW7L7fJnrk3S30v67xXlnijp7Dx8nKRb8jI/yTeBHHmq+ZmuZn4ArcAmYP88fjFwHLAC2DdPOwC4Pg9fBczMwycBG/PwaOAdeXhX4AHSLSVagXu6lXdPHv4acGYeHg/cl4e/BxyXh3cmXby2Q6Pfq5H2GETbngccXbF8V9u2k/4765p+IukK8nF9tX3lOvyoqQ0DODiPLwZOJ92X68/ytPOBU4BxwL0V7/3O+fkM0t48QAfQVrH+DtIXwLtIN3bsmv5vwFRgInAlMCZP/xfghEa/L4N5FL9nn62PiDvy8GrSH9hfApdIugP4CSmMAQ4CLsnDP69Yh4DvSboLuI50o7iWfsq9GDgmD3+6Yr0fAeblsjuAbYE9B7ZJlg2kbQdieUQ8k4cH0/ZWvUcj4sY8fAHpqv31EXFfnrYE+CvgBeAV4FxJRwF/rLaAiPgD8KCkAyW9E3gvcGMuawpwa/57mQ7sXfsmbX01XVTVRF6tGH6D9EF9LiL2H8A6PkfaO5gSEa9LeogU0r2KiN9LelrS+4HPAH+XXxLwtxFx7wDKt54NpG03kbs2JQl4Wx/rfalieMBtbwNS1YHFSFf1f4gUyJ8Fvgz89QDKuYi007UO+GVERP47WBIRpw6wzsOO9+x79gKwXtIxkD74kvbLr90E/G0e/mzFMjsBT+YP+zQ234zoRWDHPsq6EPgWsFNE3J2nXQOcnP/QkPSBWjfI3tRX2z5E2ouD9NsMY/Jwf23YW9tbfewp6aA8PJP031OrpH3ytOOBGySNJX2OfkXq1tm/h3X11Zb/CszIZVyUp60Ajpa0G4CkcZJGZPs67Hv3OWC2pDuBNWz+YZZTgK9LuoX07//zefrPgDZJt+Vl1wFExNPAjZLukfSDHsq5lPSlcXHFtO+QguaufDD3O/XcMOu1bX8KHJrb9gA2773fBWySdKekr/Wwvh7b3uqmE5iVu8nGAT8EPk/qirsb+BNwDinEr8rz3UA6JtbdecA5XQdoK1+IiGeBtcBeEXFLnraWdIzg2rze5Qyu26/hfOrlAEnaHng5/4v3WdLBWp99YTYE5FOX68Z99gM3BfhR7mJ5DvhCY6tjZtY/79mbmRXAffZmZgVw2JuZFcBhb2ZWAIe9FUf93IlSFfcvGsA6t7ivjtlw47A3MyuAw96K1c/dKkd3v9NiXmaKpBuU7ph5jaQReYGNlcdhbyV7BfibiPggMA34H123qCDdCGthRLyfdIuFL0kaA5xNujPmFNIdGOc3oN5mA+aLqqxkXXer/CvSJfeVd6vsfqfFrwBXA5OB5fk7YRSwYavW2GyQHPZWsr7uVtn9asMgfTmsiYiDMBth3I1jJevrbpXd77S4ivTDGO/qmi5pjCp+acxsOHPYW8n6ultl9zst/u+IeA04Gjgr3zHzDtIPoZgNe743jplZAbxnb2ZWAIe9mVkBHPZmZgVw2JuZFcBhb2ZWAIe9mVkBHPZmZgVw2JuZFeD/A/qF4wLic3e+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = dev_df.boxplot(\"word_count\", by=\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SST began as a collection of sentences from [Rotten Tomatoes](https://www.rottentomatoes.com/) that were released as a corpus by [Pang and Lee 2004](https://doi.org/10.3115/1218955.1218990). The data were parsed as part of the SST project, and we are now releasing them in a flat format similar to what one sees in benchmarks like [GLUE](https://gluebenchmark.com). Along this journey, the sentences have acquired a tokenization scheme that is reminiscent of what one sees in standard [Penn Treenbank](https://catalog.ldc.upenn.edu/docs/LDC95T7/cl93.html) formats, with some additional quirks. This makes the tokens different in sigificant respects from what one sees in most standard English texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The Rock is destined to be the 21st Century 's new `` Conan '' and that he 's going to make a splash even greater than Arnold Schwarzenegger , Jean-Claud Van Damme or Steven Segal .\""
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex = train_df.iloc[0].sentence\n",
    "\n",
    "ex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can address some of this using the NLTK `TreebankWordDetokenizer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "detokenizer = TreebankWordDetokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detokenize(s):\n",
    "    return detokenizer.detokenize(s.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The Rock is destined to be the 21st Century\\'s new``Conan\"and that he\\'s going to make a splash even greater than Arnold Schwarzenegger, Jean-Claud Van Damme or Steven Segal.'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "detokenize(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there is additional clean-up one could do, but this is a start."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option would be to go in the reverse – for outside data, one could try to bring it into the SST format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = TreebankWordTokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def treebank_tokenize(s):\n",
    "    return tokenizer.tokenize(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The',\n",
       " 'Rock',\n",
       " 'is',\n",
       " \"n't\",\n",
       " 'the',\n",
       " 'new',\n",
       " '``',\n",
       " 'Conan',\n",
       " \"''\",\n",
       " '–',\n",
       " 'he',\n",
       " \"'s\",\n",
       " 'this',\n",
       " 'generation',\n",
       " \"'s\",\n",
       " 'Olivier',\n",
       " '!']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treebank_tokenize(\"The Rock isn't the new ``Conan'' – he's this generation's Olivier!\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}