ulmfit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stance Classification of Tweets using Transfer Learning\n",
    "This notebook shows how *transfer learning*, an extension of deep learning, can be used for predicting Tweet stance toward a particular topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Motivation\n",
    "\n",
    "The traditional approach to applying deep learning methods in NLP have involved feeding a model large amounts of labeled training data, and fitting the model's parameters to this data. In practice, natural language data is highly variable and can come in a variety of forms (tweets, blog posts, reviews etc.), and hence, a model trained for a particular language task does not generalize well to new data from another distribution. In addition, many natural language applications do not come with an abundance of labeled examples, and human annotation can get very expensive as the datasets get larger.\n",
    "\n",
    "This offers good motivation to explore the notion of [transfer learning](http://ruder.io/transfer-learning/index.html#whatistransferlearning) - a machine-learning technique that has the ability to transfer knowledge to novel scenarios not encountered during training. While transfer learning has been ubiquitous throughout computer vision applications since the advent of huge datasets such as ImageNet, it is only since 2017-18 that significant progress has been made for transfer learning in NLP applications. There have been a string of interesting papers in 2018 that discuss the power of language models in natural language understanding and how they can be used to provide pre-trained representations of a language's syntax, which can be far more useful when training a neural network for previously unseen tasks.\n",
    "\n",
    "Twitter data is a very interesting use case for transfer learning, mainly because the typical language syntax seen in Tweets is quite different from that which is used to train language models. For these reasons, the **2016 SemEval Stance Detection task** is chosen for studying the effectiveness of our transfer learning approach. The dataset, experiments and the evaluation criteria used are explained in below sections. \n",
    "\n",
    "The aim of this notebook is to highlight the development of a model that can help answer the following questions:\n",
    "- How does our approach generalize to Twitter-specific language syntax?\n",
    "- Are we able to achieve reasonable results (comparable to the winning team of SemEval 2016 Task 6) with *limited amounts of training data* and *limited computing resources*?\n",
    "- How much fine-tuning effort is required to achieve reasonable results?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Approach\n",
    "This section describes the 3-Layer LSTM architecture used for transfer learning, **ULMFit**. This is the architecture described in the well known paper by [Howard and Ruder, 2018](https://arxiv.org/pdf/1801.06146.pdf), that has been proven to generalize\n",
    "well to unseen distributions. The goal is to see how *ULMFit* can perform in this particular Tweet stance classification task.\n",
    "\n",
    "\n",
    "## 2.1 ULMFit Model Architecture\n",
    "![title](assets/ulmfit_arch.png)\n",
    "\n",
    "Source: [Howard and Ruder, 2018](https://arxiv.org/pdf/1801.06146.pdf)\n",
    "\n",
    "To classify out-of-domain data (such as our Twitter data), the following techniques are applied from the ULMFit paper:\n",
    "\n",
    "### Discriminative Fine-tuning\n",
    "From the *ULMFit* paper, each layer of the model captures *different types of information*. Hence, it makes sense to fine-tune each layer differently, and this is done in *ULMFit* after extensive empirical testing and implementation updates. Rather than change each layer manually, we can use the implemented ratios, but adapt the multiplier that influences the learning rate for each layer. \n",
    "\n",
    "*ULMFit* also uses discriminative fine-tuning with regard to the SGD update as follows:\n",
    "\n",
    "$\\theta_{t}^{l} = \\theta_{t-1}^{l} - \\eta^{l} \\cdot \\nabla_{\\theta^l} J{\\theta} $\n",
    "\n",
    "### 1-cycle learning rate policy\n",
    "In the fine-tuning stage, a *1-cycle learning rate* policy is applied, which comes from this [report by Leslie Smith](https://arxiv.org/abs/1803.09820). It is a modification of the cyclical learning rate policy, which has been around for a long time, but the 1-cycle policy allows a large initial learning rate ($LR_{max}=10^{-3}$, for example), but decreases it by several orders of magnitude just at the last epoch. This seems to provide greater final accuracy. In the ULMFit implementation,  this 1-cycle policy has been tweaked and is referred to as *slanted triangular learning rate*.\n",
    "\n",
    "### Gradual unfreezing\n",
    "Rather than training all the layers at once during classification, the layers are \"frozen\" and the last layer is fine-tuned first, followed by the next layer before it, and so on. This avoids the phenomenon known as *catastrophic forgetting* (by fine-tuning all layers too aggressively). \n",
    "\n",
    "### Concatenated pooling\n",
    "Because an input text can consist of hundreds or thousands of words, information might get lost if we only consider the last hidden state.\n",
    "\n",
    "Hence, the hidden state at the last time step, $h_T$ is concatenated with *both* the max-pooled and mean-pooled representation of the hidden states over as many time steps as can fit in GPU memory.\n",
    "\n",
    "$h_C = [h_T, maxpool(H), meanpool(H)]$\n",
    "\n",
    "Where $H$ is the vector of all hidden states.\n",
    "\n",
    "### 3-stage fine-tuning methodology\n",
    "The classification task is done in a 3-stage process:\n",
    "1. General-domain LM pretraining: ULMFit has a pretrained model generated using an AWD-LSTM (as per [Merity et al., 2017]((https://arxiv.org/pdf/1708.02182.pdf))) to develop a language model called ```Wikitext-103``` and was trained of 28,595 preprocessed Wikipedia articles, totalling to 103 million words. \n",
    "2. Target task *LM* fine-tuning: Since the data for the target will likely come from a different distribution, ULMFit allows us to use the pre-trained language model anf fine-tune it (using the above techniques) to adapt to the idiosyncrasies of the target data.\n",
    "3. Target task *classifier* fine-tuning: Once we save the updated weights from the language model fine-tuning step, we can fine-tune the classifier with gradual unfreezing and the other techniques described above to perform  task-specific class prediction. \n",
    "\n",
    "Note that for this SemEval Tweet stance classification task, we only perform steps 2 and 3, and utilize the pretrained language model from the ```fastai``` database. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Data\n",
    "\n",
    "The training and test data consist of Tweets pertaining to five distinct topics, shown in [SemEval 2016: Task 6](http://alt.qcri.org/semeval2016/task6/). We only look at Task A: \"Supervised Framework\" in this notebook. The train and test data (including the gold) are in the the ```data/``` directory provided along with this repository.\n",
    "\n",
    "The five topics for which Tweets are provided as part of the training data are given below. \n",
    "\n",
    "| Topic        \n",
    "|:------------ | \n",
    "| Atheism     |\n",
    "| Climate Change is a Real Concern  | \n",
    "| Feminist Movement | \n",
    "| Hillary Clinton  |\n",
    "| Legalization of Abortion  | \n",
    "\n",
    "A more detailed breakdown of the tweets for this shared task is provided in [this link](http://www.saifmohammad.com/WebPages/StanceDataset.htm). \n",
    "\n",
    "## Size of dataset\n",
    "The total number of Tweets (in the training set) available for this task is roughly 2700, which amounts to roughly 500-600 Tweets per topic. Thus, this can be considered a small dataset. \n",
    "\n",
    "![title](assets/stance_balance.png)\n",
    "\n",
    "Upon inspecting the training data, it is clear that there is quite a large variance in terms of the number of Tweets in favor vs. those against a topic. There is quite a large variance *within* classes as well as *between* classes in the overall data. \n",
    "\n",
    "## 3.1 Pretrained language models\n",
    "\n",
    "*ULMFit* uses its own pretrained language model ```wikitext-103``` that can be conveniently downloaded  from the ```fastai``` database hosted on AWS. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Preprocess SemEval training data\n",
    "This section discusses the steps used to prepare the data for transfer learning utilizing the ```fastai``` framework. In particular, the ```fastai.text``` library is designed to work very well with tabular data, hence, Pandas DataFrames are used to organize and filter the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the path to train and test data for SemEval 2016 Task 6\n",
    "The full data and evaluation files for this task is available on the [source page](http://alt.qcri.org/semeval2016/task6/index.php?id=data-and-tools). However, these have been included along with this repository for future reference. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = Path('./data')\n",
    "trainfile = 'semeval2016-task6-trainingdata.txt'\n",
    "testfile = 'SemEval2016-Task6-subtaskA-testdata.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the characters in the training set are not *utf-8* encoded, hence we make sure that we only treat the characters that appear in the English language for this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_ascii(text):\n",
    "    # function to remove non-ASCII chars from data\n",
    "    return ''.join(i for i in text if ord(i) < 128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_orig = pd.read_csv(path/trainfile, delimiter='\\t', header=0, encoding = \"latin-1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cuda available\n",
      "PyTorch version:  1.0.0.dev20181203\n"
     ]
    }
   ],
   "source": [
    "# Check that the latest 1.0.0 build of PyTorch has been installed \n",
    "# alongside fastai\n",
    "import torch\n",
    "print(\"Cuda available\" if torch.cuda.is_available() is True else \"CPU\")\n",
    "print(\"PyTorch version: \", torch.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAERCAYAAABy/XBZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGAlJREFUeJzt3X20XVV97vHvYyIIqBAgUpoEgzW2F/EFegZg1cqVFkGp0XGRAVWJmjb1ivW1WsTRonJVuKKovZaOKEi49YKKWqKiGAFLeytoQEHelIhAEnk58qbC9QX53T/WTN3GhCRnn5ydZH0/Y5yx15pr7r3nZpP1rDXXXHOnqpAk9c8jRt0ASdJoGACS1FMGgCT1lAEgST1lAEhSTxkAktRTBoAk9dQGAyDJmUnuTHLNOra9OUkl2b2tJ8mHk6xIcnWS/QfqLkhyY/tbMLkfQ5K0qTbmDOAs4LC1C5PMAQ4Fbh0oPhyY1/4WAae3ursCJwIHAgcAJyaZMUzDJUnD2WAAVNWlwN3r2HQa8FZg8Fbi+cDZ1bkM2CXJnsDzgGVVdXdV3QMsYx2hIkmaOtMn8qQk84HVVXVVksFNs4CVA+urWtn6yh/W7rvvXnPnzp1IEyWpt6644oofVdXMDdXb5ABIsiNwAl33z6RLsoiu+4i99tqL5cuXb463kaRtVpJbNqbeREYB/R6wN3BVkpuB2cCVSX4HWA3MGag7u5Wtr/y3VNXiqhqrqrGZMzcYYJKkCdrkAKiq71TV46pqblXNpevO2b+qbgeWAse20UAHAfdV1W3AhcChSWa0i7+HtjJJ0ohszDDQc4CvA7+fZFWShQ9T/QLgJmAF8FHgNQBVdTdwEvDN9veuViZJGpFsyb8HMDY2Vl4DkKRNk+SKqhrbUD3vBJaknjIAJKmnDABJ6ikDQJJ6akJ3Am+r5h7/xVE3YbO6+eQXjLoJkrYgngFIUk8ZAJLUUwaAJPWUASBJPWUASFJPGQCS1FMGgCT1lAEgST1lAEhSTxkAktRTBoAk9ZQBIEk9ZQBIUk8ZAJLUUwaAJPWUASBJPWUASFJPbTAAkpyZ5M4k1wyUvS/JDUmuTvK5JLsMbHtbkhVJvpvkeQPlh7WyFUmOn/yPIknaFBtzBnAWcNhaZcuAfavqqcD3gLcBJNkHOBp4cnvOPyaZlmQa8BHgcGAf4JhWV5I0IhsMgKq6FLh7rbKvVNWDbfUyYHZbng+cW1U/r6ofACuAA9rfiqq6qap+AZzb6kqSRmQyrgG8CvhSW54FrBzYtqqVra9ckjQiQwVAkrcDDwKfmJzmQJJFSZYnWT4+Pj5ZLytJWsuEAyDJK4AjgJdWVbXi1cCcgWqzW9n6yn9LVS2uqrGqGps5c+ZEmydJ2oAJBUCSw4C3Ai+sqgcGNi0Fjk6yfZK9gXnAN4BvAvOS7J1kO7oLxUuHa7okaRjTN1QhyTnAwcDuSVYBJ9KN+tkeWJYE4LKqenVVXZvkU8B1dF1Dx1XVr9rrvBa4EJgGnFlV126GzyNJ2kgbDICqOmYdxWc8TP13A+9eR/kFwAWb1DpJ0mbjncCS1FMGgCT1lAEgST1lAEhSTxkAktRTBoAk9ZQBIEk9ZQBIUk8ZAJLUUwaAJPWUASBJPWUASFJPGQCS1FMGgCT1lAEgST1lAEhSTxkAktRTBoAk9ZQBIEk9ZQBIUk8ZAJLUUwaAJPXUBgMgyZlJ7kxyzUDZrkmWJbmxPc5o5Uny4SQrklydZP+B5yxo9W9MsmDzfBxJ0sbamDOAs4DD1io7HrioquYBF7V1gMOBee1vEXA6dIEBnAgcCBwAnLgmNCRJo7HBAKiqS4G71yqeDyxpy0uAFw2Un12dy4BdkuwJPA9YVlV3V9U9wDJ+O1QkSVNootcA9qiq29ry7cAebXkWsHKg3qpWtr5ySdKIDH0RuKoKqEloCwBJFiVZnmT5+Pj4ZL2sJGktEw2AO1rXDu3xzla+GpgzUG92K1tf+W+pqsVVNVZVYzNnzpxg8yRJGzLRAFgKrBnJswA4f6D82DYa6CDgvtZVdCFwaJIZ7eLvoa1MkjQi0zdUIck5wMHA7klW0Y3mORn4VJKFwC3AUa36BcDzgRXAA8ArAarq7iQnAd9s9d5VVWtfWJYkTaENBkBVHbOeTYeso24Bx63ndc4Eztyk1kmSNhvvBJaknjIAJKmnDABJ6ikDQJJ6ygCQpJ4yACSppwwASeopA0CSesoAkKSeMgAkqacMAEnqKQNAknrKAJCknjIAJKmnDABJ6ikDQJJ6ygCQpJ4yACSppwwASeopA0CSesoAkKSeMgAkqaeGCoAkb0xybZJrkpyT5FFJ9k5yeZIVST6ZZLtWd/u2vqJtnzsZH0CSNDETDoAks4DXAWNVtS8wDTgaOAU4raqeCNwDLGxPWQjc08pPa/UkSSMybBfQdGCHJNOBHYHbgOcC57XtS4AXteX5bZ22/ZAkGfL9JUkTNOEAqKrVwKnArXQ7/vuAK4B7q+rBVm0VMKstzwJWtuc+2OrvNtH3lyQNZ5guoBl0R/V7A78L7AQcNmyDkixKsjzJ8vHx8WFfTpK0HsN0Af0J8IOqGq+qXwKfBZ4J7NK6hABmA6vb8mpgDkDbvjNw19ovWlWLq2qsqsZmzpw5RPMkSQ9nmAC4FTgoyY6tL/8Q4DrgEuDIVmcBcH5bXtrWadsvrqoa4v0lSUMY5hrA5XQXc68EvtNeazHwt8Cbkqyg6+M/oz3lDGC3Vv4m4Pgh2i1JGtL0DVdZv6o6EThxreKbgAPWUfdnwEuGeT9J0uTxTmBJ6ikDQJJ6ygCQpJ4yACSpp4a6CCxtUd6x86hbsHm9475Rt0DbGANA0hbhKUueMuombDbfWfCdUTdhnewCkqSeMgAkqacMAEnqKQNAknrKAJCknjIAJKmnDABJ6ikDQJJ6ygCQpJ4yACSppwwASeopA0CSesoAkKSeMgAkqacMAEnqKQNAknpqqABIskuS85LckOT6JM9IsmuSZUlubI8zWt0k+XCSFUmuTrL/5HwESdJEDHsG8CHgy1X1B8DTgOuB44GLqmoecFFbBzgcmNf+FgGnD/nekqQhTDgAkuwM/DFwBkBV/aKq7gXmA0tatSXAi9ryfODs6lwG7JJkzwm3XJI0lGHOAPYGxoGPJ/lWko8l2QnYo6pua3VuB/Zoy7OAlQPPX9XKfkOSRUmWJ1k+Pj4+RPMkSQ9nmACYDuwPnF5V+wH38+vuHgCqqoDalBetqsVVNVZVYzNnzhyieZKkhzNMAKwCVlXV5W39PLpAuGNN1057vLNtXw3MGXj+7FYmSRqBCQdAVd0OrEzy+63oEOA6YCmwoJUtAM5vy0uBY9tooIOA+wa6iiRJU2z6kM//a+ATSbYDbgJeSRcqn0qyELgFOKrVvQB4PrACeKDVlSSNyFABUFXfBsbWsemQddQt4Lhh3k+SNHm8E1iSesoAkKSeMgAkqacMAEnqKQNAknrKAJCknjIAJKmnDABJ6ikDQJJ6ygCQpJ4yACSppwwASeopA0CSesoAkKSeMgAkqacMAEnqKQNAknrKAJCknjIAJKmnDABJ6ikDQJJ6augASDItybeSfKGt753k8iQrknwyyXatfPu2vqJtnzvse0uSJm4yzgBeD1w/sH4KcFpVPRG4B1jYyhcC97Ty01o9SdKIDBUASWYDLwA+1tYDPBc4r1VZAryoLc9v67Tth7T6kqQRGPYM4IPAW4GH2vpuwL1V9WBbXwXMasuzgJUAbft9rb4kaQQmHABJjgDurKorJrE9JFmUZHmS5ePj45P50pKkAcOcATwTeGGSm4Fz6bp+PgTskmR6qzMbWN2WVwNzANr2nYG71n7RqlpcVWNVNTZz5swhmidJejgTDoCqeltVza6qucDRwMVV9VLgEuDIVm0BcH5bXtrWadsvrqqa6PtLkoazOe4D+FvgTUlW0PXxn9HKzwB2a+VvAo7fDO8tSdpI0zdcZcOq6mvA19ryTcAB66jzM+Alk/F+kqTheSewJPWUASBJPWUASFJPGQCS1FMGgCT1lAEgST1lAEhSTxkAktRTBoAk9ZQBIEk9ZQBIUk8ZAJLUUwaAJPWUASBJPWUASFJPGQCS1FMGgCT1lAEgST1lAEhSTxkAktRTBoAk9ZQBIEk9NeEASDInySVJrktybZLXt/JdkyxLcmN7nNHKk+TDSVYkuTrJ/pP1ISRJm26YM4AHgTdX1T7AQcBxSfYBjgcuqqp5wEVtHeBwYF77WwScPsR7S5KGNOEAqKrbqurKtvwT4HpgFjAfWNKqLQFe1JbnA2dX5zJglyR7TrjlkqShTMo1gCRzgf2Ay4E9quq2tul2YI+2PAtYOfC0Va1MkjQCQwdAkkcDnwHeUFU/HtxWVQXUJr7eoiTLkywfHx8ftnmSpPUYKgCSPJJu5/+JqvpsK75jTddOe7yzla8G5gw8fXYr+w1VtbiqxqpqbObMmcM0T5L0MIYZBRTgDOD6qvrAwKalwIK2vAA4f6D82DYa6CDgvoGuIknSFJs+xHOfCbwc+E6Sb7eyE4CTgU8lWQjcAhzVtl0APB9YATwAvHKI95YkDWnCAVBV/w5kPZsPWUf9Ao6b6PtJkiaXdwJLUk8ZAJLUUwaAJPWUASBJPWUASFJPGQCS1FMGgCT1lAEgST1lAEhSTxkAktRTBoAk9ZQBIEk9ZQBIUk8ZAJLUUwaAJPWUASBJPWUASFJPGQCS1FMGgCT1lAEgST1lAEhSTxkAktRTUx4ASQ5L8t0kK5IcP9XvL0nqTGkAJJkGfAQ4HNgHOCbJPlPZBklSZ6rPAA4AVlTVTVX1C+BcYP4Ut0GSxNQHwCxg5cD6qlYmSZpi00fdgLUlWQQsaqs/TfLdUbZnM9sd+NFUvVlOmap36o0p/f54Z6bsrXpgav/tvWLKv7vHb0ylqQ6A1cCcgfXZrew/VdViYPFUNmpUkiyvqrFRt0MT4/e39fK760x1F9A3gXlJ9k6yHXA0sHSK2yBJYorPAKrqwSSvBS4EpgFnVtW1U9kGSVJnyq8BVNUFwAVT/b5bqF50dW3D/P62Xn53QKpq1G2QJI2AU0FIUk8ZAFupJHOSPHLU7ZC09TIAtjJJdkryAeBLwF6jbo/WL8nMJPNG3Q5NXJLfGVje5m7EMAC2IkleA9wAPKGq9q2q74+6TVq3JCcA/xd44qjbok2X5FlJLgY+nuRDSabXNnjB1ADYCiSZl+RK4DDgU8DVA9v8DrcgSf40yU3AHsChVfWlUbdJGy/JtCQnA/9EN3HlG4F5wAkjbdhm4s5jC5bkiUneCfwKOL6qXgicCeyaZAFAVT00yjaqk+RxbfGJwL1V9fqqujnJs5Mc12bC1Rauqn4F7ARcXFWfqaobgH8EdtgWD7a2uQ+0LUjyiCTvAc4DdgR+WlVfaZtvAb4FPCPJnFZ/m+ub3Bqs+e+e5EnAF5LsDfwz8B9JTkpyKvBh4P62Y9EWKMkxSV6X5IhW9C5g3yTPaeuH0u0rdx9JAzcjA2DL9ALgKcB+VfWWqrpzzYaq+ilwGXAfcFQr2+b6JrcSa26kvAP4GvC6qvoJ3fQmfwHsXFX7VdVZo2meHk6Sxye5FHg18EvgA+3A65HAx4G3JfkK8EfAfwG+muSoJI8ZWaMnmQGwhWjzI72nrW4P3FVV1bqBDmwXpXZu268DvkE3r9KB7fmeBUyRJE9I8g3grCR7VtV9dDuMpyb543a2tgS4c63n7ZtktxE0WQMG/q38GfBvVfWcqjqd7rdJfg68o6r+N3A3sKqqxlr367vogv1d20p30DbxIbYRPwX+MskY8D3gV0muA86muxD1OeCkJE9vR/zLgR8D/xU8C5hi99BdlxkD/iHJK4Ab6X7g6A2tzrnAPkmeluRxST4PnMgWOAV7nyQ5EnhdW92PNhtxkkdU1fXAZ4GdkxxMdyF4t3YQNr2qzgMWAm/eVq69GQAjlORvkuwPUFXjwMnAKVV1NXA83SiE19LtOA6mOzN4aqt/C3B6VZ08gqb3SpIdk7wjyZFJ9q2qe4D3A9fS7eiPBk6hC4H7kxxTVd8GLgUuoZv76oKqeklV3TGij9FrA0f9OwGvaMuPAW5eU6U93gRcD8ypqkvpuoYW0vaVVbVyW9n5gwEwEkmmJ/kj4N103QgnA1TV+4FHJXl1VY1X1Ueq6sqq+m6bNXV3uovAtPo/GMkH6JEkLwe+DfwucBDw0ST/vR0NQjer7WuAu+iGCm4HvDzJDsAngf8BHNy6GDTFkuy5ZrE9/iuwovXjXw78PXSjf9pR/v10/f1rflDlBOC89hO22xwDYAol2SHJG4Bjq+o/gP8FLKMb0XNakj+hO+J/45phg0l2SfKc1oXwSGDFqNrfN0lm0PUTH1VVi6rqb+j6gZ+dZD5dgP8lML2q3gN8EJhBd7/GoVX1w6r6QLtwrykyMDrrccAVSY4GdmibHw08FtihHXDtluSvksxo09XvRHfU/yWAqvpeVV0x9Z9iahgAU6iq/h/dEeOTk+wFfJruf8wP0R2ZnEY3jvw+ft1P+d+A9wGfq6oXVtXq33phTZoWuIe31R3pfrXuzoGLfl8DvgC8jG447vXAy5I8qk11/ud0R/znT23LNeCxAG303BuBF9J1qVJV19D9mzuy1X018CxgabvnZjlwLwM3W27LDIDNLMmxSZ42UPQvQAHzq+oyut8lfTrdmcCrgT8A5gLvbfOQnAs8o6rOnNKG99eLgZckeTpdH/EDwB1r+n1biF9Dd9H+0XTXaV4M7J8kVfWj1nesKZbkMUk+TbcznwNQVZ8ETgX2S7I4yaOAs4C9kkyrqouA4+i66n4CHF1Vr62qX47mU0wtA2AzajeSnAX8zyRPb32M3weuBJ7ULgB/lO6o/0+Br1fVSXQXg88BfllV3kS0mSV5bpI1c/b8K7AKeF67C3RXum4e8uvZV38IHEI3zv8G4O3AVY7EGrn7gT3pzgAWJnkZQFVdSTd8cwbwd8BzgR+1fv9pVfXjqrqwqk6tqqtG1fhRMAAm2Vrj8S8F/p3uaPHlwFtb+ZfpTjNfQLczuYSu3/hpAK3f+JVVdddUtbuv2lnWV4FPJHkq3UX2S4DfS7IfXVfc25LMGzgq3A+4mDbOv6qWtouHmkLp5u15d5JntrOvh+i6Va+iC/L3JjkoyaOr6na6C7630nXTvSbJdn0/uHJM8iRJsiPweuD6JJdW1d10XQSX0PUn3kA34udGuikevggcCzwP+D/AH9LdhKIpVFW3J3kfcAzwfLpx/G8EDgSOrKq3J1kMnJrkHroL8U8GTqoqv68RSfIXwEvp7sK+ZuDsaxy4paouSfJ24L10d2af1sb5X59uyo5fdS+T9PnMzZ+EnARJ/hp4Jd0UDV+l+x/wirbtHGBJVX25jeTZB/h8Vb0hyd/Tna7+HfDzbWl88dakhfcq4Al0E3+tpPtebgOurqp/aX3KzwIeV1UfGlljtWZ0z+3AU9tFXZLsXFX3JXkxsAh4C93v/j5ANzPrZ4HFVbW6dcU+OKLmb1EMgCG1oZtvAd7QjjDW3v4q4J10p54P0I0rfj/wTbqJw37ozUGjl+63Fp7UgvkQuiG6DwI/oJvj5+ZRtk+/KcnHgAur6tNJzgbGq+rNbdsquu7tt1bVPyfZB3gTcHJVOYx6gAEwpCQfp7t4u7jdTv5QG+L5ArpbyZ9MNy/MB9v8IrTtj6+qfxtZw/Ub2jDPW4HnV9XVbeTWX9H197+49SFrC9HG699LNwz3E8Cp7aLuY+nu1fhSVV3Y9y6eDfEawCZKcgpdV8+ydoPPXXQjRWg7/1fRHW08QDfc8zPAzwZ2/ttV1a10OxttIdp39xLgY8ABbTTIa0bcLK1HVd2fZBFwRFWdAl2HflX9uF3Mvxm4cJRt3Bo4CmgTpJvJ8c/oLhK+txXfCjwmydy2/i26YWYL6Ub+PADs1LoV2FZvKd8WVNXXgQfbDkRbviXAgQPf13bt8T100zw4SeIG2AW0CZJsD3ye7uLSnwNXAN+nmyPmuqpaPFD3COAIurOB7dsEYtrCtXHhvR4auDVJ8gy6ET4HjbotWyPPADZS69//Od1OfwxYQDfB1MF0Uzf8YZJ/SPKMJGfR7fg/X1UPuPPferjz37q0s7aHPGubGANgIw0M0VxO9yMRP6GbMfAoujH8nwV+Rtdv/MOqem5VfXEkjZX65dnVTaGuTWQAbLqHgOOSXAPsAjyJbiTCS+mGdb6iqk4YYfukXvGsbeK8BjABSa6i+zGWf2rruwKPqqofjrZlkrTxHAa6iZJMp5ve4ea2Pq1N+yBJWxW7gDZRu4X8EbR5ezz9lLS1sgtoAhwqKGlbYABIUk/ZBSRJPWUASFJPGQCS1FMGgCT1lAEgST1lAEhSTxkAktRT/x++Pm+opW9jgwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot value counts for rach stance in the dataset\n",
    "train_orig['Stance'].value_counts().plot.bar(rot=30);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SemEval 2016 Task 6 (subtask A) Tweet topics are:\n",
      "\n",
      "------------------------------------------\n",
      "Atheism\n",
      "Climate Change is a Real Concern\n",
      "Feminist Movement\n",
      "Hillary Clinton\n",
      "Legalization of Abortion\n"
     ]
    }
   ],
   "source": [
    "print(\"SemEval 2016 Task 6 (subtask A) Tweet topics are:\\n\")\n",
    "print(\"------------------------------------------\")\n",
    "for item in train_orig['Target'].unique():\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply the cleaning function to the 'Tweet' column of the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_orig['Tweet'] = train_orig['Tweet'].apply(clean_ascii)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Target</th>\n",
       "      <th>Tweet</th>\n",
       "      <th>Stance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>101</td>\n",
       "      <td>Atheism</td>\n",
       "      <td>dear lord thank u for all of ur blessings forg...</td>\n",
       "      <td>AGAINST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>102</td>\n",
       "      <td>Atheism</td>\n",
       "      <td>Blessed are the peacemakers, for they shall be...</td>\n",
       "      <td>AGAINST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>103</td>\n",
       "      <td>Atheism</td>\n",
       "      <td>I am not conformed to this world. I am transfo...</td>\n",
       "      <td>AGAINST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>104</td>\n",
       "      <td>Atheism</td>\n",
       "      <td>Salah should be prayed with #focus and #unders...</td>\n",
       "      <td>AGAINST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>105</td>\n",
       "      <td>Atheism</td>\n",
       "      <td>And stay in your houses and do not display you...</td>\n",
       "      <td>AGAINST</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ID   Target                                              Tweet   Stance\n",
       "0  101  Atheism  dear lord thank u for all of ur blessings forg...  AGAINST\n",
       "1  102  Atheism  Blessed are the peacemakers, for they shall be...  AGAINST\n",
       "2  103  Atheism  I am not conformed to this world. I am transfo...  AGAINST\n",
       "3  104  Atheism  Salah should be prayed with #focus and #unders...  AGAINST\n",
       "4  105  Atheism  And stay in your houses and do not display you...  AGAINST"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_orig.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store the relevant information from training data \n",
    "ULMFit requires just the stance and the text data (i.e. tweets) for the language-model finetuning and classification steps, hence these are stored in a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Stance</th>\n",
       "      <th>Tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2809</th>\n",
       "      <td>AGAINST</td>\n",
       "      <td>There's a law protecting unborn eagles, but no...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2810</th>\n",
       "      <td>AGAINST</td>\n",
       "      <td>I am 1 in 3... I have had an abortion #Abortio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2811</th>\n",
       "      <td>AGAINST</td>\n",
       "      <td>How dare you say my sexual preference is a cho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2812</th>\n",
       "      <td>AGAINST</td>\n",
       "      <td>Equal rights for those 'born that way', no rig...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2813</th>\n",
       "      <td>AGAINST</td>\n",
       "      <td>#POTUS seals his legacy w/ 1/2 doz wins. The #...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Stance                                              Tweet\n",
       "2809  AGAINST  There's a law protecting unborn eagles, but no...\n",
       "2810  AGAINST  I am 1 in 3... I have had an abortion #Abortio...\n",
       "2811  AGAINST  How dare you say my sexual preference is a cho...\n",
       "2812  AGAINST  Equal rights for those 'born that way', no rig...\n",
       "2813  AGAINST  #POTUS seals his legacy w/ 1/2 doz wins. The #..."
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.concat([train_orig['Stance'], train_orig['Tweet']], axis=1)\n",
    "train.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write train to csv\n",
    "train.to_csv(path/'train.csv', index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Experimental Setup\n",
    "\n",
    "Stance detection uses a *semi-supervised approach* where we reuse weights from a pretrained language model (in this case ```wikitext-103```) and perform multi-class classification on the training data over the three classes (*FAVOR*, *AGAINST* and *NONE*).\n",
    "\n",
    "### Evaluation\n",
    "The metric used to score the stance classification is **F-score**. The event organizers provide an [evaluation script](http://alt.qcri.org/semeval2016/task6/index.php?id=data-and-tools) that calculates the macro-average of F-score (FAVOR) and F-score (AGAINST) for task A. This compares our model's predicted stance for each Tweet against the gold reference.\n",
    "\n",
    "The *perl* script provided by the organizers is used to generate an F- score. The evaluation script is in ```data/eval/``` and has the following usage:\n",
    "    \n",
    "    cd data/eval\n",
    "    perl eval.pl -u\n",
    "\n",
    "    ---------------------------\n",
    "    Usage:\n",
    "    perl eval.pl goldFile guessFile\n",
    "\n",
    "    goldFile:  file containing gold standards;\n",
    "    guessFile: file containing your prediction.\n",
    "\n",
    "### Reference result\n",
    "To have a benchmark to compare our results against, it makes sense to look at the winning paper for this shared task, from team *MITRE*, who [published their methodology and approach](https://arxiv.org/pdf/1606.03784.pdf). From this paper, it can be seen that *MITRE* used an LSTM-based deep learning approach and performed per-topic classification of Tweets to achieve an average macro F-score of **0.67**. \n",
    "\n",
    "### Stance Prediction\n",
    "The predicted output stances on the test dataset is written out according to the format expected by the evaluation *perl* script, and the F-scores are published as per this evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Transfer Learning Stages\n",
    "\n",
    "This notebook utilizes the refactored and updated version (```v1```) of *ULMFit* as implemented in the [fastai](https://github.com/fastai/fastai/tree/master/fastai)  GitHub repository. This requires an install of **Pytorch 1.0.0** ([released on December 7, 2018](https://developers.facebook.com/blog/post/2018/05/02/announcing-pytorch-1.0-for-research-production/)) to run since its development intended to move alongside PyTorch's own development cycle in the future. It is recommended to run this entire notebook in a virtual environment to avoid clashes with pre-existing installs of PyTorch.\n",
    "\n",
    "The refactored code for *ULMFit* is much more high-level and abstracted than the original version that was used in the paper, so it doesn't make sense to rewrite the data manipulation code for text handling and tokenization from the ground up. We can take advantage of the powerful transfer learning functionality that is baked into ULMFit (obtained after months of fine-tuning and refactoring the source code by Fast.ai) instead! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Language model finetuning: (SemEval Tweets only)\n",
    "The pre-trained language model used is from [this work by Merity et al.](https://arxiv.org/pdf/1708.02182.pdf). Although we are working with Twitter whose content's structure that can be quite dissimilar to Wikipedia's sentence formulation, the hope is that the language understanding developed by the model, in combination with the fine-tuning techniques described in [Howard and Ruder's paper](https://arxiv.org/pdf/1801.06146.pdf), will allow us to at least transfer some knowledge for the purposes of classification of Tweet stance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess data for language model fine-tuning\n",
    "The ```TextLMDataBunch```[ [source]](https://docs.fast.ai/text.data.html#TextLMDataBunch) class returns a \"Databunch\" type of object that is a custom object defined in fastai.text's API. Note that according to this class definition, a significant amount of data pre-processing is going on under the hood. \n",
    "\n",
    "- Initialize the language model with the pre-trained vocabulary's weights\n",
    "- Create a custom data loader that \"batchifies\" the target task's vocabulary, i.e. split it up into batches for processing on the GPU\n",
    "- Introduce randomness into the batches, but without breaking up the sentence sequences using techniques [borrowed from language modeling](https://arxiv.org/pdf/1708.02182.pdf).\n",
    "- Tokenize the target task's vocabulary and increase the efficiency of tokenization (for large tasks) using multi-threading\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Language model data\n",
    "data_lm = TextLMDataBunch.from_csv(path, 'train.csv', min_freq=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in this case, we specify a minimum word frequency of 1 (i.e. words that appear just once in the data are given the tag ```<unk>``` during tokenization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the language and classifier model data for re-use\n",
    "data_lm.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Reload LM and clas data from file if doing so for large cases\n",
    "# data_lm = TextLMDataBunch.load(path)\n",
    "# data_clas = TextClasDataBunch.load(path, bs=32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='5%'>  <col width='95%'>  <tr>\n",
       "    <th>idx</th>\n",
       "    <th>text</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>xxbos xxfld 1 xxmaj if ' xxunk not human beings in the womb , how do we xxunk and xxunk their xxunk onto human beings ? # semst xxbos xxfld 1 # xxmaj tolerance ... i wonder how much more # xxmaj mother # xxmaj earth has at # methodology levels re our # xxunk # ideologies ? # semst xxbos xxfld 1 # xxmaj xxunk # heatwave continues .</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>it . xxmaj such an amazing xxunk . # semst xxbos xxfld 1 xxunk xxunk xxmaj hillary 's xxmaj new xxmaj campaign slogan is : xxup hillary - xxup nothing xxup to xxup hide xxup but xxup the xxup truth ! # tcot # wakeupamerica # semst xxbos xxfld 1 xxmaj hillary has lied , deleted xxmaj benghazi emails , and betrayed the trust of xxmaj americans scandal after scandal</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>much as boring as mine . # xxunk # semst xxbos xxfld 1 xxmaj great that @hillaryclinton xxunk # xxunk 's xxunk today ! ! # semst xxbos xxfld 1 xxmaj good xxmaj friday morning # patriots . xxmaj make it ur best one ever . # xxmaj smile all day long . # xxup xxunk # xxunk # xxup xxunk # xxup xxunk # xxunk # semst xxbos xxfld</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th># xxmaj xxunk just raised 45 million ! # semst xxbos xxfld 1 xxmaj how can she live with herself ? # xxmaj benghazi # semst xxbos xxfld 1 xxmaj my vote is definitely for xxmaj hillary . i do n't want xxmaj bush 's dumb ass . xxmaj do n't want xxmaj trump either . # hillaryclinton # semst xxbos xxfld 1 xxmaj women are strong , women are</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>the right to choose to kill . # semst xxbos xxfld 1 xxunk xxunk xxunk xxunk / xxunk xxmaj xxunk face . # semst xxbos xxfld 1 xxmaj if you 're a feminist , stop whining on the internet and step up your game . xxmaj guys in xxmaj baltimore riot for their cause . xxmaj get shit done . # semst xxbos xxfld 1 xxmaj ok now how about</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only need to run the above tokenization step once, and save the data. All the data goes to a local folder named ```tmp/```. For a large added vocabulary, the above tokenization step can take a while, hence we save them and restart the notebook from the next cell (if we close our session).\n",
    "\n",
    "The tokenization technique used by ```fastai.text``` is quite advanced, and uses quite a few tricks to capture semantic meaning from the text. Unlike typical NLTK-type tokenization, here we are not converting the Tweets to lowercase and removing information from the base text - that would result in a tremendous loss of information that our model could use to gather an understanding of our new task's vocabulary.\n",
    "\n",
    "Instead, a number of added tokens are applied so that minimal information is lost. A full list of all the tokenization rules used are [here](https://docs.fast.ai/text.transform.html#Rules). \n",
    "\n",
    "As an example, the ```xxmaj``` token [[source]](https://github.com/fastai/fastai/blob/1c5c007e10a187f4a048a3ee8137016534259768/fastai/text/transform.py#L71) indicates that there is capitalization of the word, either in part or in full. \"The\" will be tokenized as \"xxmaj the\". For words that are full capitalized, such as \"I AM SHOUTING\"; it is tokenized as \"xxup i xxup am xxup shouting\". The tokenizer uses spaCy's underlying tokens, but adds tags in a very smart way that balances capturing semantic meaning while reducing the number of overall tokens, so it is both powerful and efficient. \n",
    "\n",
    "The tokenization process described above is highly efficient, in part thanks to the power of *SpaCy* and also due to the multi-threading wrappers implemented by Fast.ai on top of Spacy's tokenizer. As a result, a really large language model's vocabulary (of millions of words) can also be effectively tokenized in a reasonable time. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define learner object for language model finetuning\n",
    "We define a learner object that uses the tokenized language model data, that is organized into batches for the GPU, and feed it a pre-trained language model. \n",
    "\n",
    "We use the ```wt_103_v1``` model that was refined after re-training using the more efficient ```fastai v1``` library. The data (```lstm_wt103.pth```) is downloaded through an AWS-hosted link (accessed through the ```fastai.datasets``` interface), and stored locally. All the data in the language model file is numericalized and stored in a way that the learner can efficiently pick up tokens in float form. In addition to the language model, there is also an additional pickle object ```itos_wt103.pkl``` that is similar to the ```torchtext``` definition of \"integer-to-string\" - this contains the list of unique tokens in word form, so that we can map the tokens back to strings if necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, pretrained_model=URLs.WT103_1, drop_mult=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print the structure of the language model RNN \n",
    "We can examine the architecture of the 3-layer LSTM used in ULMFit as shown below. For the language model fine-tuning, we define the embeddings based on the the target task vocabulary size. A single linear output layer is defined  that updates the weights once we pass the input through the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[RNNCore(\n",
       "   (encoder): Embedding(2760, 400, padding_idx=1)\n",
       "   (encoder_dp): EmbeddingDropout(\n",
       "     (emb): Embedding(2760, 400, padding_idx=1)\n",
       "   )\n",
       "   (rnns): ModuleList(\n",
       "     (0): WeightDropout(\n",
       "       (module): LSTM(400, 1150)\n",
       "     )\n",
       "     (1): WeightDropout(\n",
       "       (module): LSTM(1150, 1150)\n",
       "     )\n",
       "     (2): WeightDropout(\n",
       "       (module): LSTM(1150, 400)\n",
       "     )\n",
       "   )\n",
       "   (input_dp): RNNDropout()\n",
       "   (hidden_dps): ModuleList(\n",
       "     (0): RNNDropout()\n",
       "     (1): RNNDropout()\n",
       "     (2): RNNDropout()\n",
       "   )\n",
       " ), LinearDecoder(\n",
       "   (decoder): Linear(in_features=400, out_features=2760, bias=True)\n",
       "   (output_dp): RNNDropout()\n",
       " )]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(learn.model.children())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find the optimum learning rate \n",
    "```fastai.train``` [source](https://github.com/fastai/fastai/blob/050080be574cb1260462bbd03e9600e43e7a54b1/fastai/train.py#L23) provides a convenient utility to search through a range of learning rates to find the optimum one for our dataset. \n",
    "\n",
    "The idea is that our optimization function needs to use a learning rate that is an order of magnitude below the point at which the loss starts to diverge. We can quickly find this as shown below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcXFWd9/HPr5Zek05n6UBWAiRssgTShLAKxmGA4WFRcOAZF+QRRBHE9aXj60EfHdxFRQYwoozMIKNsigwKuARQCBhCAoGEEMKSnaST3reqrt/zR90uiraTdKf7Vt3q/r5fr/uqqntP3fvL7Ur96pxz7znm7oiIiADEih2AiIhEh5KCiIjkKCmIiEiOkoKIiOQoKYiISI6SgoiI5CgpiIhIjpKCiIjkKCmIiEhOotgBDNakSZN81qxZxQ5DRKSkPPPMM9vdvW5P5UouKcyaNYulS5cWOwwRkZJiZq8PpJyaj0REJEdJQUREcpQUREQkR0lBRERylBRERCRHSUFERHKUFEREJEdJQUSkBPzgD2t4/OVtoR9HSUFEJOLcnR/9aS1PrdsR+rGUFEREIq69u4eejDO2IvxBKJQUREQirqUzDcDYimTox1JSEBGJuJbOFIBqCiIiAs1BUqipVE1BRGTUa841H6mmICIy6vX2KdSUelIws0+a2Uoze8HMruln+6lm1mRmy4Pl2jDjEREpRW/1KYTffBRa2jGzw4HLgPlAN/B7M3vA3df2Kfq4u58dVhwiIqWuuWNkNB8dCjzl7u3ungYeBd4T4vFEREakls4UiZhRmYyHfqwwk8JK4GQzm2hmVcBZwIx+yh1vZivM7Hdm9o4Q4xERKUktnWnGViQws9CPFVpdxN1Xmdm3gIeBNmA50NOn2DJgP3dvNbOzgF8Dc/ruy8wuBy4HmDlzZlghi4hEUktnqiD9CRByR7O7/9Td57n7KcBOYE2f7c3u3ho8fxBImtmkfvazyN3r3b2+rq4uzJBFRCKnt6ZQCGFffTQ5eJxJtj/hF32272tBfcjM5gfxNIQZk4hIqWnuTBUsKYR9lHvMbCKQAq5090YzuwLA3W8BLgA+ZmZpoAO4yN095JhEREpKS2eamROqCnKsUJOCu5/cz7pb8p7fCNwYZgwiIqUu23w0AvoURERk6ArZfKSkICISYZmM09qVLsgQF6CkICISaa3dadwLM8QFKCmIiERabjC8StUURERGvUIOhgdKCiIikdZSwLkUQElBRCTSVFMQEZGcQg6bDUoKIiKR1ltTqFFNQURECjk/MygpiIhEWktnmrJ4jIoCTLADSgoiIpHWUsAhLkBJQUQk0poLOJcCKCmIiERaS2eKmsrCdDKDkoKISKQVctY1UFIQEYm0ls4UY8tHSE3BzD5pZivN7AUzu6af7WZmN5jZWjN7zsyOCTMeEZFSM2JqCmZ2OHAZMB84CjjbzGb3KXYmMCdYLgduDiseEZFS1NyRKtgQFxBuTeFQ4Cl3b3f3NPAo8J4+Zc4FbvesJUCtmU0JMSYRkZLRk3HaunsKNmw2hJsUVgInm9lEM6sCzgJm9CkzDVif93pDsE5EZNRrzd3NXLiaQmjpx91Xmdm3gIeBNmA50LM3+zKzy8k2LzFz5sxhi1FEJMqacyOkjoyaAu7+U3ef5+6nADuBNX2KbOTttYfpwbq++1nk7vXuXl9XVxdewCIiEZKbdW2kJAUzmxw8ziTbn/CLPkXuBz4YXIW0AGhy981hxiQiUiqaCzyXAoTYfBS4x8wmAingSndvNLMrANz9FuBBsn0Na4F24MMhxyMiUjLeqimMkKTg7if3s+6WvOcOXBlmDCIipaplpPUpiIjI3iv0/MygpCAiElmFnp8ZlBRERCKruTNNeSJGWaJwX9VKCiIiEVXoYbNBSUFEJLIKPcEOKCmIiERWdoRU1RRERITsCKmFvJsZlBRERCKrpTOl5iMREclq6UwX9G5mUFIQEYmsQs+6BkoKIiKRlOrJ0JHqUUeziIgUZ4gLUFIQEYmkYgxxAUoKIiKRVIwJdkBJQUQkkooxwQ4oKYiIRNKI7FMws0+Z2QtmttLM7jSzij7bLzGzbWa2PFg+EmY8IiKlorkjW1MYMfcpmNk04Gqg3t0PB+LARf0U/aW7zw2WW8OKR0SklIzImgLZ6T4rzSwBVAGbQj6eiMiIMOKSgrtvBL4LvAFsBprc/eF+ir7XzJ4zs7vNbEZY8YiIlJKWzhRVZXES8cJ2/YbZfDQeOBfYH5gKVJvZ+/sU+y0wy92PBB4Bfr6LfV1uZkvNbOm2bdvCCllEJDKKMcQFhNt89G7gVXff5u4p4F7ghPwC7t7g7l3By1uBef3tyN0XuXu9u9fX1dWFGLKISDQ0d6YKfjkqhJsU3gAWmFmVmRmwEFiVX8DMpuS9PKfvdhGR0aq5s/BzKUC2IzgU7v6Umd0NLAPSwLPAIjP7KrDU3e8Hrjazc4LtO4BLwopHRKSUNLR2M318VcGPG2oacvcvA1/us/ravO1fBL4YZgwiIqVoR1s3R02vLfhxdUeziEjEuDs72rqZOKas4MdWUhARiZjmjjTpjDNxTHnBj62kICISMdvbshdlTqxWTUFEZNRraO0GUPORiIjAjqCmMEE1BRER2R7UFCapT0FERHa0ZZPC+CrVFERERr2G1i5qKhKUJQr/Fa2kICISMdvbuovSdARKCiIikbOjtbsoncygpCAiEjkNbV1FuRwVlBRERCInO8SFmo9EREa9nkww7pGaj0REpLG9m4wXZ4gLUFIQEYmU3nsUJqj5SEREcnczj8Sagpl9ysxeMLOVZnanmVX02V5uZr80s7Vm9pSZzQozHhGRqOutKYy4jmYzmwZcDdS7++FAHLioT7H/A+x099nA94FvhRWPiEgpaCjiYHgQfvNRAqg0swRQBWzqs/1c4OfB87uBhWZmIcckIhJZ21u7MYPxVcmiHD+0pODuG4HvAm8Am4Emd3+4T7FpwPqgfBpoAib23ZeZXW5mS81s6bZt28IKWUSk6Ha0dVFbmSQRL06Xb5jNR+PJ1gT2B6YC1Wb2/r3Zl7svcvd6d6+vq6sbzjBFRCKlobV4N65BuM1H7wZedfdt7p4C7gVO6FNmIzADIGhiGgc0hBiTiEikNRTxxjUINym8ASwws6qgn2AhsKpPmfuBDwXPLwD+5O4eYkwiIpHW0Fq8cY8g3D6Fp8h2Hi8Dng+OtcjMvmpm5wTFfgpMNLO1wKeBL4QVj4hIKcgOcVG85qPEQAqZ2YHABnfvMrNTgSOB2929cXfvc/cvA1/us/ravO2dwIWDilhEZIRK92TY2Z4q2uWoMPCawj1Aj5nNBhaR7Qf4RWhRiYiMQjvae+dmjn5SyASXjJ4P/MjdPwdMCS8sEZHRp9h3M8PAk0LKzC4m2yn8QLCuOHdWiIiMUA3BuEel0Hz0YeB44Dp3f9XM9gf+M7ywRERGn4a24jcfDaij2d1fJDuOUe9NaWPdXeMUiYgMo4bW3nGPIt58ZGaLzazGzCaQvcT0J2Z2fbihiYiMLg2t3cQMaiuL1zo/0Oajce7eDLyH7KWox5G9Y1lERIZJQ1s3E6rLicWKNy7oQJNCwsymAO/jrY5mEREZRg2tXUUd4gIGnhS+CjwEvOLufzOzA4CXwwtLRGT02dHWXdQhLmDgHc13AXflvV4HvDesoERERqOGtm7eMbWmqDEMtKN5upndZ2ZvBss9ZjY97OBEREaT7a1dTCrijWsw8Oaj28iOaDo1WH4brBMRkWHQnc7Q0pkumT6FOne/zd3TwfIfgGa7EREZJr1DXEwocp/CQJNCg5m938ziwfJ+NBmOiMiwaWjL3rhWzGGzYeBJ4VKyl6NuITvf8gXAJSHFJCIy6vSOe1Tsq48GlBTc/XV3P8fd69x9srufh64+EhEZNm/VFEogKezCp3e30cwONrPleUuzmV3Tp8ypZtaUV+baXe1PRGQke6umUNzmowHdp7ALu70P291fAuYCmFkc2Ajc10/Rx9397CHEISJS8t5s6aIsEaOmYihfy0M3lJqCD6LsQrJ3Q78+hOOJiIxYGxs7mFZbiVnxxj2CPdQUzKyF/r/8DagcxHEuAu7cxbbjzWwFsAn4rLu/0E8clwOXA8ycOXMQhxURKQ2bGzuYWltR7DB2X1Nw97HuXtPPMtbdB1THMbMy4BzyhsnIswzYz92PAn4E/HoXcSxy93p3r6+r0+0RIjLybGrsZMq4wfzWDsdQmo8G6kxgmbtv7bvB3ZvdvTV4/iCQNLNJBYhJRCQyUj0ZtrZ0MrV2dCSFi9lF05GZ7WtBA5qZzQ/i0U1xIjKqbG3uxB2mRaD5KNRubjOrBv4B+GjeuisA3P0WsjfBfczM0kAHcJG7D6YDW0Sk5G1q7ASIRPNRqEnB3duAiX3W3ZL3/EbgxjBjEBGJuk2NHQCjpvlIRER2Y2MuKRS/+UhJQUSkyDY3dVBblaSqrLg3roGSgohI0W1q7GRqBPoTQElBRKToNjV2RKI/AZQURESKblNjRyQuRwUlBRGRomrpTNHcmWaKagoiIrK5KXuPgpqPREQkd4+Cmo9ERCRSdzODkoKISFFtauwgHjMmjy3ujGu9lBRERIpoU2MH+9ZUkIhH4+s4GlGIiIxSm5o6mDIuGv0JoKQgIlJUmxqjMY9CLyUFEZEiyWSczU3RuZsZlBRERIpme1sXqR6PzOWooKQgIlI0UbscFUJMCmZ2sJktz1uazeyaPmXMzG4ws7Vm9pyZHRNWPCIiUROlyXV6hTZ4t7u/BMwFMLM4sBG4r0+xM4E5wXIccHPwKCIy4r11N3N0kkKhmo8WAq+4++t91p8L3O5ZS4BaM5tSoJhERIpqU2MnVWVxaiqLP7lOr0IlhYuAO/tZPw1Yn/d6Q7DubczscjNbamZLt23bFlKIIiKF1TuPgpkVO5Sc0JOCmZUB5wB37e0+3H2Ru9e7e31dXd3wBSciUkSbInY5KhSmpnAmsMzdt/azbSMwI+/19GCdiMiIt6mxM1KXo0JhksLF9N90BHA/8MHgKqQFQJO7by5ATCIiRdWZ6mF7a1ekLkeFEK8+AjCzauAfgI/mrbsCwN1vAR4EzgLWAu3Ah8OMR0QkKrZEbHKdXqEmBXdvAyb2WXdL3nMHrgwzBhGRKFq/sx2I1uWooDuaRUSK4qUtLQActM+YIkfydkoKIiJFsGpzC5PHljNxTDQm1+kVnTsmQvba9jYee3kbNRVJaioTjK1IUpmMk8446Z4M6YzT2plmZ3s3O9u7aWxPUZmMM766jAnVZYyrTGIG+Fv7jMWMmBnxGFQmE9RWJRlfVUZlWbxo/04Ad6e5M01zRwozSMZjJGKGmdHenaaju4e27h4MGFeZZFxlkprKJADd6Qxd6R46Uxk6Uz10pnvo6M4urV1pWrvStHWl6e5xknEjHjOSsVhwLsAMYmZk3OnJZEeBdJxELEYibiRiMarKgvNaVcaEMWVUJLK/TcwMd6cl93dI0dqVJtvKmN1uZPefPZZRkYwxtiLBmPIk1eVxKpLx3L9VJMpWb2nmkCk1xQ7j74yapLBiQyPX/uaFAZePx4yejO+5YD/KEzHGViQZUx6nujxBRTJOW1eals40LZ0putIZKsviVCXjVJZlv8jKEjHK4jHKEjE6Uz20BF/qrV1pksH68kT2y7crlf3i7kplcHjbe9u70zS2p0jvZewjgRmUxWNUJOOMKU8ESSPBmIoE1WUJqsqyf5exFQnGV5UxvjpJbVUZFYk4ybiRCJJozAzLS3Sx3KNRWRanpiJJRTKmBCSDlu7J8PLWVk6aPanYofydUZMU/umIKZw0e1LuF3RzZ4qO7h6S8RjxmJGIGdXlCSZUlzG+uozqsjjdPRma2lPsaO+mqT2VqyT0Vhgy7mQy0ONOR3eane2pXC2jpTP7i7qtK01HqofxVVXUVCSoqUzmvvjbg1/gnakeunsydKUyNHemqUrGmTmhiprKJGPKE6Qz2W3dQY2mIhGnPBmjIpGtkXT39NCdztCdzlBZlmBCdbbGUlOZBIdUJkO6x3F3qsoSVJbFqS6Pk8lAU0eKpuB8GEZ5Mpt8yhIxKpNxKpPZpFWRjOe+XKvLE5TFY6Qz2XhSPRncg/Ph2ZpK75dnLJb9Rd/T46QyGXoyTltXtibQ0NrNjrZuutOZ/ApY7st6XFWSmopEUIPIbnP37LnPOD3udKUztHZmE25rVyp3nrI1nkwuEbd2pWlo7WZ9d/vbaj1DzZ2JmFFTma119p638uB571JZlqC2MkltVTb5TBpTxpRxlUwZV8E+NRWUJdSKO9q8ur2N7p4Mh0wZW+xQ/s6oSQqJeIyJYwbXfleeiDO5Js7kmmjdXCLDI5NxmjtTuWTelcqQ6smQzmRI9TjueUkoL+H1ZJzOVIbmzlTuB0Zvc1tXOvvYnc4EiShDR6qHxvZumjpS/SahidVlTK6pYPLYcvapKWf6+CpmTKhkxvgqZk2qZlLE2pxl6FYFncyH7KvmI5HIiMWM2qoyaqvK2J/q0I+XyWT7S7a1drKpsZMtTZ1saupga3MX21o6ebOlixc3N7Otpett75s5oYp5+43nmP3Gc/SMWg7aZ6xqFyVu9eZmEjHjwLpoXXkESgoiBROLGeOqkoyrSjJ78q6bDTpTPWzY2cH6ne28vLWFZa838pe127nv2ewIMGXxGIdMGcsR08Yxf/8JnDR7UuSuYJHdW72lhdmTx0QyuSspiERMRTLO7MljmD15DKcdPBnINltt2NnBig2NPL+hiec3NnH/8k3c8dQbmMHhU8dx8pxJLDx0MnNnjCceU+d3lK3e3Mz8/ScUO4x+KSmIlAAzY8aEKmZMqOLsI6cC0JNxnt/YxONrtvH4y9tZ9Ng6blr8CpPGlPGuQyZz+mH78s6D60jGo/drdDRrak+xqakzkpejgpKCSMmKx4y5M2qZO6OWqxbOoakjxaNrtvGHF7fyu5Vb+NXSDdSNLefCedP552NnsN/E8PtNZM9Wb2kG4JB9o3flESgpiIwY4yqTnHPUVM45aiqpngyPvrSN//7bG9zy6CvctPgVTpo9iQ8cvx8LD5lMQrWHolkdXHl0qGoKIlIoyXiMdx+2D+8+bB+2NHXyq6XrufPpN/jofz7D1HEV/MuC/bh4/kwmVJcVO9RRZ/WWZsZXJZk8NpoXB+jngsgIt++4Cq5eOIfHP38at7x/HvvXVfOdh17ixG/+ia898GJuCGcpjFWbWzhk35rI3gmvmoLIKJGIxzjj8H054/B9WbO1hVsefYX/eOI1bn/yNS6YN52PvXM2MydWFTvMES2TcV7a0sJF82fsuXCRhFpTMLNaM7vbzFab2SozO77P9lPNrMnMlgfLtWHGIyJZB+0zluvfN5fFnz2Vi46dyT3LNnLa9xbz2btW8Or2tmKHN2K9saOdjlQPh0bwTuZeYdcUfgj83t0vMLMyoL+fIY+7+9khxyEi/ZgxoYqvnXc4V71rNj9+bB13PPU69y7bwHlzp/Hp0w9i+njVHIZT7sqjCI551Cu0moKZjQNOAX4K4O7d7t4Y1vFEZO9Nrqng/559GI9//l185OQD+J/nN7Pwe4/y3Ydeoq0rXezwRoxVm1uIGczZzR3txRZm89H+wDbgNjN71sxuDeZs7ut4M1thZr8zs3eEGI+I7EHd2HL+9axD+dNnT+WMw/flxj+v5bTvLuaeZzbk5rWQvbd6SzOzJlUXfc6V3QkzKSSAY4Cb3f1ooA34Qp8yy4D93P0o4EfAr/vbkZldbmZLzWzptm3bQgxZRCA7b/APLzqaez9+AlNrK/nMXSv4wE+f5o2G9mKHVtJWb2mJdH8ChJsUNgAb3P2p4PXdZJNEjrs3u3tr8PxBIGlmfzfrhLsvcvd6d6+vq6sLMWQRyXfMzPHc+7ET+Np5h7N8fSOn/+BRfvLYOtI9mWKHVnIa27t5vaGdw6aO0qTg7luA9WZ2cLBqIfBifhkz29eCi3XNbH4QT0NYMYnI4MVixgcW7Mcjnz6Fk2ZP4roHV3Hhj5/k9QZdpTQYT726AyCyA+H1CvvmtauAO8zsOWAu8HUzu8LMrgi2XwCsNLMVwA3ARa6GS5FImjKukp98sJ4bLj6atW+2ctYPH1dfwyAsWddARTLGkdPHFTuU3bJS+4PW19f70qVLix2GyKi2sbGDT/1yOU+/uoOzj5zC199zBDUVyWKHFWln/OAxJo0p578+clxRjm9mz7h7/Z7KaZgLERm0abWV3HnZAj73jwfz+5VbOOdHf8ldgy9/b2dbN6u3tLDggGg3HYGSgojspXjMuPK02fz35Qto7+7hvH//K78OZoeTt+vtT1hwwMQiR7JnSgoiMiT1sybwwNUnceT0Wq755XKu/c1KUro66W2WrGugMhnnyOm1xQ5lj5QURGTIJo+t4I6PHMdlJ+/P7U++zod+9jSN7d3FDisylqxroH7W+EjOydxX9CMUkZKQjMf40j8dxvcuPIqlr+3k/JueYN221mKHVXQNrV1Bf0L0m45ASUFEhtl7503njsuOo6kjxfk3PcETa7cXO6SierqE+hNASUFEQnDsrAn8+uMnMnlsOR/82dPc/cyGYodUNG/1J0T7/oReSgoiEoqZE6u45+MnsOCAiXz2rhVc/8iaUXmj25NBf0KyRObFLo0oRaQk1VQkue3Dx3LhvOnc8MeX+cyvVtCdHj1XJm1v7WLN1taSaToCTccpIiFLxmN8+4Ij2W9iFd99eA0bdnZw0/uPYdKYaE5cP5yeWpftTzj+wNJJCqopiEjozIxPvGsON1x8NCs2NHLujX/lhU1NxQ4rdEvWNVBVFueIaaXRnwBKCiJSQOccNZW7rjienoxzwc1P8j/PbS52SKHpyTh/WLWVBQdMLJn+BFBSEJECO3J6LfdfdSKHThnLlb9YxvUPv0QmM/I6oB9d8yabmzq5cN70YocyKEoKIlJwk8dWcOflC7Id0H9ay8fueGbEzQV959PrmTSmjIWH7lPsUAZFSUFEiqI8EefbFxzJtWcfxiMvbuW9Nz/B+h0jY7rPrc2d/Gn1m1wwb0ZJDG2Rr7SiFZERxcy49KT9+fml89nU2ME5N/6Fx9aU/jzsdy1dT0/GuejYGcUOZdBCTQpmVmtmd5vZajNbZWbH99luZnaDma01s+fM7Jhd7UtERq6T59Txm0+cxOSxFXzotqe54Y8vl2w/Qybj/HLpeo4/YCKzJlUXO5xBC7um8EPg9+5+CHAUsKrP9jOBOcFyOXBzyPGISETtP6ma+648gfPmTuP6R9Zw6c//VpIjrf71le2s39HBxcfNLHYoeyW0pGBm44BTgJ8CuHu3uzf2KXYucLtnLQFqzWxKWDGJSLRVlSW4/n1H8bXzDueva7fzTzf8hSXrGood1qDc+fQbjK9K8o/vKK0O5l5h1hT2B7YBt5nZs2Z2q5n1rUtNA9bnvd4QrBORUcrM+MCC/bj7ihMoS8S4+CdL+PqDq+hM9RQ7tD3a3tqV7TQ/ZjrliXixw9krYSaFBHAMcLO7Hw20AV/Ymx2Z2eVmttTMlm7bVvqdUCKyZ0fNqOV/rj6J/z1/JoseW1cSd0H/15LXSfU4F80vvQ7mXmEmhQ3ABnd/Knh9N9kkkW8jkH/2pgfr3sbdF7l7vbvX19XVhRKsiERPVVmC684/gtsuOZYd7d2cc+Nf+caDq2jvjt49Da9ub+Omxa9w5uH7Mnvy2GKHs9dCSwruvgVYb2YHB6sWAi/2KXY/8MHgKqQFQJO7j9z73kVkr5x2yGQe+dQpXDhvOj9+bB2nf/8xFr/0ZrHDynF3vnTf85QnYnzlnHcUO5whCfvqo6uAO8zsOWAu8HUzu8LMrgi2PwisA9YCPwE+HnI8IlKiaqvK+OZ7j+SXly+gPBHjktv+xmW3L43ElJ/3LNvIE6808IUzD2GfmopihzMkVmqTXtTX1/vSpUuLHYaIFFFXuodbH3+Vm/68lq50hvcv2I+rF85hQnVZwWPZ3trFu69/lNl1Y/jVR48nFrOCxzAQZvaMu9fvqZzuaBaRklOeiHPlabNZ/LnT+OdjZ3D7k6/xzu/8mZsXv1Lwq5T+7YEXaetK8433HBHZhDAYSgoiUrLqxpZz3flH8NA1p3DsrAl86/eredd3F3Pvsg0FuSP6zqff4NfLN/GxU2czZ5/S7VzOp6QgIiVvzj5j+dklx/KLy45j4phyPv2rFZzz73/h2Td2hnbMn/3lVb547/OcenAdV552YGjHKTQlBREZMU44cBK/ufJEfnjRXLa3dPOem5/gX+97ftiHy/j3P6/lqw+8yBnv2JdFH6gv2RvV+qM5mkVkRInFjHPnTmPhofvwg0fWcNsTr/HQyi188t1zuHDeDCrL9v4LvDud4fpH1nDLo69w/tHT+M4FR5IooVnVBkJXH4nIiPbipma+fP9K/vbaTmqrkvzLcTP54PGzBnXpaCbj/Pa5TXzv4TW8saOdi+fP5LrzDi+pjuWBXn2kpCAiI567s/T1ndz6+DoefnEriZhxzMzxHDtrAsfuP4G5M2qpqUhg9taXfEtnile2tbFmSwu3PfEaqzY3c9iUGj5/xsG886C6t5UtBUoKIiL9eL2hjV88/QZLXmlg5aZmeoKrlBIxY0xFguqyBOlMhq3NXbn3zJxQxWdOP4j/deTUkqod5BtoUlCfgoiMKvtNrOaLZx4KQFtXmuXrG3l+YxPNHSnautK0dKWJmXFAXTUH1o1h9uQxzJpYTbxEk8FgKSmIyKhVXZ7gxNmTOHH2pGKHEhkjq9tcRESGRElBRERylBRERCRHSUFERHKUFEREJEdJQUREcpQUREQkR0lBRERySm6YCzNrAl7OWzUOaNrN8/zHJLB9EIfL39+e1vddt6dY8h8nDSKuXcU0kLgGc66GK66hnqvedQxjTAOJqxjnqr9te3OuRuvnfSAx7Sq+0XCuat29bo97cfeSWoBFu3rd3/P8R2DpUI61u/UDiWs3jwOOa1cxDSSuwZyr4YprqOcq72834s/Vns7NQM9V3jkbVZ93fTcM7vO+q6UUm49+u5vX/T3v+ziUY+1u/UDiCjOmgcRViueq9/loOFf9bdubczXccZXK513fDXu37W1KrvloKMxanJtqAAAHnUlEQVRsqQ9glMBCU1wDF8WYIJpxRTEmiGZcUYwJihNXKdYUhmJRsQPYBcU1cFGMCaIZVxRjgmjGFcWYoAhxjaqagoiI7N5oqymIiMhulGxSMLOfmdmbZrZyL947z8yeN7O1ZnaDBfPqmdkvzWx5sLxmZsujEFew7SozW21mL5jZt4sdk5l9xcw25p2vs/Zi36Gcq2D7Z8zMzWzQA+WHdL6+ZmbPBefqYTObGoGYvhN8pp4zs/vMrHYv9h1GXBcGn/OMmQ24PX0osexifx8ys5eD5UN7irvIMV1nZuvNrHXIBxnMZVhRWoBTgGOAlXvx3qeBBYABvwPO7KfM94BroxAXcBrwB6A8eD05AjF9BfhsFP+GwAzgIeB1YFIU4gJq8spcDdwSgZhOBxLB828B34rIuToUOBhYDNSHHUtwnFl91k0A1gWP44Pn4/f02StiTAuAKUDrYP8OfZeSrSm4+2PAjvx1Znagmf3ezJ4xs8fN7JC+7zOzKWT/gy7x7Nm8HTivTxkD3gfcGZG4PgZ80927gmO8GYGYhizEuL4PfB7Yqw6zMOJy9+a8otWDjS2kmB5293RQdAkwfTAxhRjXKnd/qVCx7MI/Ao+4+w533wk8Apwx2P8ThYgpOM4Sd988wP3sVskmhV1YBFzl7vOAzwI39VNmGrAh7/WGYF2+k4Gt7v4yw2OocR0EnGxmT5nZo2Z2bARiAvhE0PTwMzMbPwwxDTkuMzsX2OjuK4YpnmGJK4jtOjNbD/wLcG0UYspzKdlfvcNhOOMqRCz9mQasz3vdG99wxD3cMQ2rETNHs5mNAU4A7spr4ivfy91dzF7UEkKMK0G2yrgAOBb4lZkdEPxSKVZMNwNfI/uL92tkm9su3Zt4hisuM6sC/pVss8iwGa7Plrt/CfiSmX0R+ATw5WLHFOzrS0AauGNv4wkjrjBjMbMPA58M1s0GHjSzbuBVdz9/NMXU14hJCmRrPY3uPjd/pZnFgWeCl/eT/TLLryZPBzbmlU8A7wHmRSiuDcC9QRJ42swyZMdE2VasmNx9a977fgI8sJexDGdcBwL7AyuC/3DTgWVmNt/dtxQxrr7uAB5kCElhuGIys0uAs4GFe/sjI4y4hkm/sQC4+23AbUFsi4FL3P21vCIbgVP7xLc4WD+UuMOIaXgNtVOimAswi7wOHOAJ4MLguQFH7eJ9fTuKzsrbdgbwaJTiAq4Avho8P4hsFdKKHNOUvDKfAv47CueqT5nX2IuO5pDO15y8MlcBd0cgpjOAF4G6KH3e87YvZhAdzXsbC7vu1H2VbIfu+OD5hIF+9godU16ZIXc0D+nNxVzINu9sBlJkf0n/H7K/En8PrAg+7P1ePQTUAyuBV4AbyfuCBf4DuCJKcQFlwH8F25YB74pATP8JPA88R/aX35TBxBTm3zCvzGvs3dVHYZyve4L1z5Edh2ZaBGJaS/YHxvJgGdQVUSHGdX6wry5gK/BQmLHQzxdwsP7S4BytBT48mM9eEWL6drD/TPD4lcH+LXsX3dEsIiI5I+3qIxERGQIlBRERyVFSEBGRHCUFERHJUVIQEZEcJQUZEYZldMjBHe9WMztsmPbVY9nRU1ea2W9tD6OVmlmtmX18OI4t0pcuSZURwcxa3X3MMO4v4W8NFheq/NjN7OfAGne/bjflZwEPuPvhhYhPRhfVFGTEMrM6M7vHzP4WLCcG6+eb2ZNm9qyZPWFmBwfrLzGz+83sT8AfzexUM1tsZndbdt6BO8xyY/4vtmCsfzNrDQa7W2FmS8xsn2D9gcHr583s3wZYm3mStwb3G2NmfzSzZcE+zg3KfBM4MKhdfCco+7ng3/icmf2/YTyNMsooKchI9kPg++5+LPBe4NZg/WrgZHc/muxopV/Pe88xwAXu/s7g9dHANcBhwAHAif0cpxpY4u5HAY8Bl+Ud/4fufgRvH1mzX8H4QAvJ3iEO0Amc7+7HkJ1T43tBUvoC8Iq7z3X3z5nZ6cAcYD4wF5hnZqfs6Xgi/RlJA+KJ9PVu4LC80ShrglEqxwE/N7M5ZEd5Tea95xF3zx///ml33wBg2Zn4ZgF/6XOcbt4aEPAZ4B+C58fz1lj7vwC+u4s4K4N9TwNWkR0nH7Lj4nw9+ILPBNv36ef9pwfLs8HrMWSTxGO7OJ7ILikpyEgWAxa4e2f+SjO7Efizu58ftM8vztvc1mcfXXnPe+j//0zK3+qc21WZ3elw97nB0N8PAVcCN5Cdd6EOmOfuKTN7Dajo5/0GfMPdfzzI44r8HTUfyUj2MNkRSQEws97hisfx1nDHl4R4/CVkm60ALtpTYXdvJztV52eCIdzHAW8GCeE0YL+gaAswNu+tDwGXBrUgzGyamU0epn+DjDJKCjJSVJnZhrzl02S/YOuDztcXyQ5BDtkRJb9hZs8Sbm35GuDTZvYc2UlTmvb0Bnd/luxIqheTnXeh3syeBz5Iti8Ed28A/hpcwvodd3+YbPPUk0HZu3l70hAZMF2SKhKSoDmow93dzC4CLnb3c/f0PpFiUp+CSHjmATcGVww1MsTpSkUKQTUFERHJUZ+CiIjkKCmIiEiOkoKIiOQoKYiISI6SgoiI5CgpiIhIzv8HzoHE1UYmlgAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find(start_lr=1e-8, end_lr=1e2)\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the ```language_model_learner``` class\n",
    "The pre-trained language model quickly downloads from the ```fastai```'s AWS link, so there is no need to download it separately. [```drop_mult```] is the multiplier applied to the dropout in the language learner. As per the ULMFit paper, we use a custom dropout for each language model layer, and the ```drop_mult```just scales a [numpy array of these dropouts](https://github.com/fastai/fastai/blob/1c5c007e10a187f4a048a3ee8137016534259768/fastai/text/learner.py#L15) which remain in the same relative ratio to one another. These numbers have been obtained through empirical tuning for a range of language tasks, and we took note of this during our extensive reading on this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:01 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>6.777056</th>\n",
       "    <th>5.760276</th>\n",
       "    <th>0.184697</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = language_model_learner(data_lm, pretrained_model=URLs.WT103_1, \n",
    "                               drop_mult=0.5)\n",
    "learn.fit_one_cycle(cyc_len=1, max_lr=1e-3, moms=(0.8, 0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unfreeze the entire model \n",
    "To train the language model during fine-tuning, we unfreeze all the layers and then perform training for a reasonable time until we obtain a low enough validation loss. For a larger target vocabulary, these hyperparameters might need to be modified, and teh training time would be significantly longer, but for this SemEval task, we have a very small additional vocabulary to fine-tune the language model on, so it is really quick.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:31 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>6.387111</th>\n",
       "    <th>5.049564</th>\n",
       "    <th>0.213630</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>5.968547</th>\n",
       "    <th>4.324110</th>\n",
       "    <th>0.293626</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>5.577562</th>\n",
       "    <th>4.065256</th>\n",
       "    <th>0.318037</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>5.292692</th>\n",
       "    <th>3.894169</th>\n",
       "    <th>0.334843</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>5.035873</th>\n",
       "    <th>3.689974</th>\n",
       "    <th>0.332790</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>4.764125</th>\n",
       "    <th>3.326344</th>\n",
       "    <th>0.374279</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>4.451447</th>\n",
       "    <th>3.148137</th>\n",
       "    <th>0.390625</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>4.185637</th>\n",
       "    <th>3.142953</th>\n",
       "    <th>0.393342</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>3.969743</th>\n",
       "    <th>3.105297</th>\n",
       "    <th>0.398438</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>3.790563</th>\n",
       "    <th>3.118295</th>\n",
       "    <th>0.395987</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>11</th>\n",
       "    <th>3.629449</th>\n",
       "    <th>3.101075</th>\n",
       "    <th>0.399016</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>12</th>\n",
       "    <th>3.488647</th>\n",
       "    <th>3.109549</th>\n",
       "    <th>0.398317</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>13</th>\n",
       "    <th>3.365237</th>\n",
       "    <th>3.113755</th>\n",
       "    <th>0.400229</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>14</th>\n",
       "    <th>3.258392</th>\n",
       "    <th>3.125626</th>\n",
       "    <th>0.399956</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>15</th>\n",
       "    <th>3.173512</th>\n",
       "    <th>3.125749</th>\n",
       "    <th>0.395706</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>16</th>\n",
       "    <th>3.098583</th>\n",
       "    <th>3.135684</th>\n",
       "    <th>0.400078</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>17</th>\n",
       "    <th>3.037213</th>\n",
       "    <th>3.127744</th>\n",
       "    <th>0.399381</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>18</th>\n",
       "    <th>2.980722</th>\n",
       "    <th>3.129555</th>\n",
       "    <th>0.398513</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>19</th>\n",
       "    <th>2.940534</th>\n",
       "    <th>3.132044</th>\n",
       "    <th>0.398257</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>20</th>\n",
       "    <th>2.901994</th>\n",
       "    <th>3.139766</th>\n",
       "    <th>0.398245</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(cyc_len=20, max_lr=1e-3, moms=(0.8, 0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the fine-tuned encoder\n",
    "learn.save_encoder('ft_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Train the Classifier: SemEval Tweets only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For classification, the [winning paper for the SemEval 2016](https://www.aclweb.org/anthology/S/S16/S16-1074.pdf) task (team MITRE) describe in section $5$ of their paper that they achieved their highest average F1 score when they trained 5 *distinct* classifiers, i.e. a separate training task for each of the 5 topics in the training data. \n",
    "\n",
    "Hence, in this notebook we only perform classification for one *topic* at a time. Evaluation of the predicted classes is also done on a per-topic basis. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify the 5 topics in our training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Atheism\n",
      "Climate Change is a Real Concern\n",
      "Feminist Movement\n",
      "Hillary Clinton\n",
      "Legalization of Abortion\n"
     ]
    }
   ],
   "source": [
    "for item in train_orig.Target.unique():\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify the topic we want to classify for\n",
    "We specify the topic name exactly as the training data does, and use it to subset the Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic = \"Feminist Movement\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get only those tweets that pertain to a single topic in the training data\n",
    "train_topic = train.loc[train_orig['Target'] == topic]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write train to csv\n",
    "train_topic.to_csv(path/'train_topic.csv', index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create classification dataloader\n",
    "For classification, the ```TextClasDataBunch```  [[source](https://docs.fast.ai/text.data.html#TextClasDataBunch)] dataloader used is slightly different. Just like the language model custom data loader described in the previous section, it does a lot under the hood. The classification data loader tokenizes the training data, but batchifies the data differently from the language model data loader - for classification, we perform sorting so as to group all large sentences together, and smaller sentences together - and pad them to have all batches of roughly the same length, followed by randomizing these batches so that we don't introduce additional bias during training.\n",
    "\n",
    "Just like before, we define the learner object used for classification. Note that we feed in the entire vocabulary of the training data to the classifier. We once more specify a minimum word frequency of 1 (i.e. words that appear just once in the data are given the tag ```<unk>```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Classifier model data\n",
    "data_clas = TextClasDataBunch.from_csv(path, 'train_topic.csv', vocab=data_lm.train_ds.vocab,\n",
    "                                       min_freq=1, bs=32)\n",
    "data_clas.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(data_clas, drop_mult=0.5)\n",
    "learn.load_encoder('ft_enc')\n",
    "learn.freeze()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we begin the training with all the outer layers as frozen. The fine-tuned language model is read in, and we use a dropout multiplier as shown above. \n",
    "\n",
    "For training the classifier, we apply \"gradual unfreezing\", to avoid \"catastrophic forgetting\" as explained in section $3.3$ of [Howard and Ruder's paper](https://arxiv.org/pdf/1801.06146.pdf). In addition to gradual unfreezing, "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Just as before, find optimum learning rate for the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl4VOX1wPHvyQIhAQJkYQtL2BcBgaggKrhVqlarVisudUGprdZqrbXWX8XWtnbRtrYWKaWItS51QesuWmVRQAggIYR9T1gSsu/r+f0xEx1CJpmQmcyd5HyeZx5m7nvv3JNLMmfe5b6vqCrGGGNMc8KCHYAxxpjQYAnDGGOMTyxhGGOM8YklDGOMMT6xhGGMMcYnljCMMcb4xBKGMcYYnwQsYYjIIhHJFpF0L+XXi0iaiGwWkVUiMsGj7F4R2SIi6SLyoohEBSpOY4wxvglkDWMxMLOJ8r3AdFUdBzwKLAAQkf7A3UCKqp4ChAPXBjBOY4wxPogI1Bur6goRGdxE+SqPl2uAJI/XEUAXEakGooFDvpwzPj5eBw/2ekpjjDENrF+//piqJviyb8ASRgvNBt4DUNUsEXkcOACUA0tVdakvbzJ48GBSU1MDF6UxxrQzIrLf132D3uktIufiShgPuF/3BC4HkoF+QIyI3NDE8XNEJFVEUnNyctoiZGOM6ZCCmjBEZDywELhcVXPdmy8A9qpqjqpWA0uAM729h6ouUNUUVU1JSPCpVmWMMeYkBC1hiMhAXMngRlXd4VF0AJgiItEiIsD5wNZgxGiMMeYrAevDEJEXgRlAvIhkAnOBSABVnQ88DMQB81x5gRp3TeFzEXkV2ADUABtxj6AyxhgTPNKe1sNISUlR6/Q2xhjfich6VU3xZd+gd3obY4wJDZYwjDHG+MQShjHGhLAPM44yf/nuNjmXJQxjjAlhH2w5wrOr9rXJuSxhGGNMCCsoq6JndKc2OZclDGOMCWF5pVX0jIlsk3NZwjDGmBCWX1ZtNQxjjDHNyy+roleMJQxjjDFNqKmto7C8mh5WwzDGGNOUwvJqVKFXtPVhGGOMaUJ+WTUAPa1JyhhjTFPyy6oArNPbGGNM0/JKXQnDOr2NMcY0qaC+hmEJwxhjTFPySt19GNbpbYwxpikFZVV0jgijS2R4m5wvYAlDRBaJSLaIpHspv15E0kRks4isEpEJHmU9RORVEdkmIltFZGqg4jTGmFCVV+q6ac+9amnABbKGsRiY2UT5XmC6qo4DHuX4ZVifBN5X1VHABGxNb2OMOUF+WVWb3bQHAVzTW1VXiMjgJspXebxcAyQBiEgscA5ws3u/KqAqUHEaY0yoyi+rplcbTTwIzunDmA28536eDOQAz4jIRhFZKCIxwQvNGGOcKb+07aY2BwckDBE5F1fCeMC9KQKYBDytqhOBUuCnTRw/R0RSRSQ1Jycn4PEaY4xT5LXhWhgQ5IQhIuOBhcDlqprr3pwJZKrq5+7Xr+JKII1S1QWqmqKqKQkJCYEN2BhjHKK2Tiksr26zezAgiAlDRAYCS4AbVXVH/XZVPQIcFJGR7k3nAxlBCNEYYxyrrScehAB2eovIi8AMIF5EMoG5QCSAqs4HHgbigHnuIWE1qpriPvwHwPMi0gnYA9wSqDiNMSYU1U8L0pY1jECOkprVTPltwG1eyr4AUhorM8YY4zEtSEfpwzDGGHNy2nriQbCEYYwxIal+avMebdiHYQnDGGNCUP3iSVbDMMYY06T80radeBAsYRhjTEjKc9/l3VYTD4IlDGOMCUn5ZW170x5YwjDGmJCUX1bVphMPgiUMY4wJSfmlbTu1OVjCMMaYkJRfVkUvSxjGGGOaUlunFLTxxINgCcMYY0JO/cSDPdvwpj2whGGMMSGn/i7vtrxpDyxhGGNMyMkvbfuJB8EShjHGhJw8SxjGGGN8UeCeR6qn3YdhjDGmKXntrQ9DRBaJSLaIpHspv15E0kRks4isEpEJDcrDRWSjiLwdqBiNMSYU5ZdW0amNJx6EwNYwFgMzmyjfC0xX1XHAo8CCBuU/BLYGJjRjjAld9TftteXEgxDAhKGqK4C8JspXqWq+++UaIKm+TESSgEuAhYGKzxhjQlVeadvftAfO6cOYDbzn8frPwE+AuuCEY4wxzpVfVtXmN+2BAxKGiJyLK2E84H59KZCtqut9PH6OiKSKSGpOTk4AIzXGGGfIL6vqeDUMERmPq9npclXNdW+eBlwmIvuAl4DzROTf3t5DVReoaoqqpiQkJAQ8ZmOMCbb80rafeBCCmDBEZCCwBLhRVXfUb1fVB1U1SVUHA9cCH6vqDUEK0xhjHCVYEw8CRATqjUXkRWAGEC8imcBcIBJAVecDDwNxwDx3T3+NqqYEKh5jjGkPioI08SAEMGGo6qxmym8Dbmtmn2XAMv9FZYwxoS1YN+2BAzq9jTHG+C5YEw+CJQxjjAkp+fXzSFnCMMYY05QvaxhtPPEgWMIwxpiQYn0YxhhjfJJfFpyJB8EShjHGhJT6m/baeuJBsIRhjDEhJVgTD4IlDGOMCSkFQZp4ECxhGGNMSMkL0sSDYAnDGGNChqqSW2I1DGOMMc3YnVNKYXk1o/t2D8r5LWEYY0yIWL7DtebPOcODs5SDJQxjjAkRK3bkMCQhhgG9ooNyfksYxhgTAiqqa1mzJzdotQuwhGGMMSFh7d48KmvqmD7SEoYxxpgmrNiRQ6eIMKYkxwUtBksYxhgTApbvyOH0wb3o0qnt55CqF7CEISKLRCRbRNK9lF8vImkisllEVonIBPf2ASLyiYhkiMgWEflhoGI0xphQcKignJ3ZJUwfEbzmKAhsDWMxMLOJ8r3AdFUdBzwKLHBvrwHuU9UxwBTgThEZE8A4jTHG0VbudA+nba8JQ1VXAHlNlK9S1Xz3yzVAknv7YVXd4H5eDGwF+gcqTmOMcbrlO3Lo0z2KEb27BjUOp/RhzAbea7hRRAYDE4HPvR0oInNEJFVEUnNycgIWoDHGBENNbR2f7jzGOSPigzKluaegJwwRORdXwnigwfauwGvAPapa5O14VV2gqimqmpKQENzqmjHG+NumzAKKKmqC3hwFEBHMk4vIeGAh8HVVzfXYHokrWTyvqkuCFZ8xxgTb8h3HCBM4a1h8sEMJXg1DRAYCS4AbVXWHx3YB/glsVdU/Bis+Y4xxghU7cpgwoAc9ooMzpbmnQA6rfRFYDYwUkUwRmS0id4jIHe5dHgbigHki8oWIpLq3TwNuBM5zb/9CRC4OVJzGGONU+aVVbMosCOp0IJ4C1iSlqrOaKb8NuK2R7Z8Cwe3ZMcYYB/h4WzaqBHU6EE9B7/Q2xhjTuCUbMxnYK5qJA3oEOxTAEoYxxjjSoYJyVu3O5cpJ/YM+nLaeJQxjjHGg1zdmoQpXTkwKdihfsoRhjDEOo6os2ZDJ6YN7MTAuOIslNcYShjHGOMymzEJ255Ry5SRnzYpkCcMYYxxmyYZMOkeEcfH4vsEO5TiWMIwxxkEqa2p5c9Mhvja2D92jIoMdznEsYRhjjIN8si2HgrJqxzVHgSUMY4xxlCUbMkno1pmzHTB3VEOWMIwxxiHySqv4ZHs23zy1HxHhzvt4dl5ExhjTQb216RDVtcpVk51z74UnSxjGGOMQa/fmMbBXNKP6dA92KI2yhGGMMQ6RV1pFYrfOwQ7DK0sYxhjjEPllVY5Y98IbSxjGGOMQ+WVV9Ipx1r0XnixhGGOMA6gq+WXV9OyINQwRWSQi2SKS7qX8ehFJE5HNIrJKRCZ4lM0Uke0isktEfhqoGI0xxinKqmqpqqmjZ0yIJwwRGSoind3PZ4jI3SLS3Ioei4GZTZTvBaar6jjgUWCB+/3Dgb8BXwfGALNEZIwvcRpjTKjKL6sCoGd06DdJvQbUisgwXB/sA4AXmjpAVVcAeU2Ur1LVfPfLNUD9wOPTgV2qukdVq4CXgMt9jNMYY0JSfmk1QLtokqpT1RrgCuCvqno/4M9pFGcD77mf9wcOepRlurc1SkTmiEiqiKTm5OT4MSRjjGk7X9YwQr1JCqgWkVnATcDb7m1+qTeJyLm4EsYDJ3O8qi5Q1RRVTUlIcMZC6cYY01JfNUmFfsK4BZgK/FpV94pIMvBca08uIuOBhcDlqprr3pyFq8mrXpJ7mzHGtFv5pc7vw4jwZSdVzQDuBhCRnkA3Vf1da04sIgOBJcCNqrrDo2gdMNydlLKAa4HrWnMuY4xxuvyyakQgtkuIJwwRWQZc5t5/PZAtIp+p6o+aOOZFYAYQLyKZwFzczViqOh94GIgD5okIQI27aalGRO4CPgDCgUWquuXkfjxjjAkN+WVVdI+KdOQstfV8ShhArKoWichtwL9Uda6IpDV1gKrOaqb8NuA2L2XvAu/6GJsxxoS8/LJqejm4wxt878OIEJG+wDV81eltjDHGT/JLq+jh4P4L8D1h/BJXE9FuVV0nIkOAnYELyxhjOpb8sip6OXiEFPje6f0K8IrH6z3AVYEKyhhjOpr80irHroNRz9epQZJE5HX33FDZIvKaiDhzSShjjAlBrj6M9tEk9QzwJtDP/XjLvc0YY0wrVVTXUl5d6+i1MMD3hJGgqs+oao37sRiw26qNMcYP6u/ybi+jpHJF5AYRCXc/bgBymz3KGGNMs/JC4C5v8D1h3IprSO0R4DDwLeDmAMVkjDEdSkGZa6badtEkpar7VfUyVU1Q1URV/SY2SsoYY/yivobRXpqkGuN1WhBjjDG+K3D3YbSXG/caI36LwhhjOrC8EFg8CVqXMNRvURhjTAeWX1ZFt84RRDp44kFo5k5vESmm8cQgQJeARGSMMR1MflmVo1faq9dkwlDVbm0ViDHGdFT5ZdWOH1ILrWuSMsYY4wf5paFRw7CEYYwxQZZfVuX4Dm8IYMIQkUXuiQrTvZSPEpHVIlIpIj9uUHaviGwRkXQReVFEogIVpzHGBFtBWXXHThjAYmBmE+V5uNYJf9xzo4j0d29PUdVTcC3Tem2AYjTGmKCqqqmjpLKmY/dhqOoKXEnBW3m2qq4DqhspjgC6iEgEEA0cCkyUxhgTXPU37VkfxklQ1SxctY4DuOatKlTVpd72F5E5IpIqIqk5OTltFaYxxvhFXn3C6OBNUidFRHoClwPJuNbeiHHPjtsoVV2gqimqmpKQYDOuG2NCS379Xd4OXzwJHJgwgAuAvaqao6rVwBLgzCDHZIwxAZFvNYxWOQBMEZFoERHgfGBrkGMyxpiACJXFk6CZO71bQ0ReBGYA8SKSCcwFIgFUdb6I9AFSge5AnYjcA4xR1c9F5FVgA1ADbAQWBCpOY4wJpvzS0JipFgKYMFR1VjPlR4AkL2VzcSUYY4xp1/LLqonpFE7niPBgh9IsJzZJGWNMh5FfWuX4lfbqWcIwxpggyi+rCon+C7CEYYwxQZVXVh0S/RdgCcMYY4KqIEQmHgRLGMYYE1R5pdYkZYwxphnVtXUUV9RYk5QxxpimFZS5pgWxGoYxxpgm1c9Ua8NqjTHGNCm/voZhCcMYY0xT8kJoWhCwhGGMMUFTEEITD4IlDGOMCZpQWjwJLGEYY0zQFJRVExUZRpdOzp94ECxhGGNM0OSVhs5d3mAJwxhjgiaUpgUBSxjGGBM0eaVVIbGWd72AJQwRWSQi2SKS7qV8lIisFpFKEflxg7IeIvKqiGwTka0iMjVQcRpjTLAUlFVbDcNtMTCzifI84G7g8UbKngTeV9VRwARsTW9jTDuUZ01SLqq6AldS8FaerarrgGrP7SISC5wD/NO9X5WqFgQqTmOMCYayqhoKy6uJ79o52KH4zIl9GMlADvCMiGwUkYUiEuNtZxGZIyKpIpKak5PTdlEGyebMQhau3ENdnQY7FGNMK2w9XIwqjOnXPdih+MyJCSMCmAQ8raoTgVLgp952VtUFqpqiqikJCQltFWNQvLzuIFfNX8Wv3tnK7z/YHuxwjDGtsOVQIQCn9A+dhBER7AAakQlkqurn7tev0kTC6Aiqaur4xVtbeP7zA5w1LJ6+sVHMX76boQkxXJ0yINjhGWNOwubMQuJiOtGne1SwQ/GZ4xKGqh4RkYMiMlJVtwPnAxnBjitYjhZV8P3nN7B+fz53TB/K/ReNpE6VQ4Xl/Oz1zQzsFc0ZQ+KCHaYxpoXSDxUxtn8sIhLsUHwWsIQhIi8CM4B4EckE5gKRAKo6X0T6AKlAd6BORO4BxqhqEfAD4HkR6QTsAW4JVJxO9k7aYX7+33Qqqmt56rqJXDq+HwDhCPOum8wVT3/GHf9ezxt3TmNQnNduHnMSVJVlO3J4bvV+iiuqqVOoU6VOYWy/7tx+9hCS4+2am5NTUV3LzqPFnDcqtJrRA5YwVHVWM+VHgCQvZV8AKYGIKxTkllTy8JtbeCftMOP6x/LHayYwvHe34/aJjY5k0U2n8c15n3Hr4nUs+f40YrsE9wagTQcLWLxqHz+6cAQDekUHNZbWWLX7GE8s3cH6/fn0i41iUFwMYWEQJoIqvLo+kxfXHuDiU/pyx/ShjEuKDXbIQVNSWcPOo8XklVZx9vAEOkU4sVvUeXYcLaamTjmlX2j97jiuSaqjez/9MA+9nk5RRTX3XzSS754zhIjwxv8IB8fHMP+GyVy/8HMee3crv71qfBtH+5X1+/O5edFaiitrWLnzGItuTmF8Uo+gxXMy1u/P54ml21m1O5c+3aP49RWncPXkASd8CGYXV/DMZ/v49+r9vLP5MNNHJPDIZWO91jhS9+VRXl3L1CFxXv8vT8bKnTkMS+xK39gufntPX6zYkcOzq/ax/WgxmfnlX26fOiSOp2+YFDKrxwXT5qz6Du/QShii2n6GZ6akpGhqamqwwzhp85bt4vfvb2dc/1gev3oCI/t0a/4g4DfvbmXBij28esdUUgb3CnCUJ1q7N49bnllLYvcoHr38FH66JI3ckiqeum4i54/u3ebxtNSWQ4U8sXQHH2/LJr5rJ74/YxjXnTGQqMimZxAtqqjm+TUHmLdsF1U1dfzowhHMPiv5y6SwJ6eE37y7lY+2ZgOQ0K0zV0zsz5WT+jOqT9MjY2rrlDDBa/v285/v56HX0+kUEcbNZw7me9OH0rMN1lT4ZFs2c55LJbFbFJMG9WRk766M6N2NYyVVPPLmFvr37MI/b0phSELXgMcSyh5cspl3Nx/mi4cvDHofhoisV1WfWnQsYTjEUx/v5PGlO7hsQj+euGYCkS34JlpWVcOFf1xB184RvH33WS06trVW787l1sXr6Ncjihdun0Lv7lFkF1cwe3EqWw4V8ovLT+HGKYPaLJ6W2JNTwhNLd/DO5sPEdonku9OHcPOZg4nu1LKK99GiCv7vjXQ+zDjKuP6x/PzSMSzdcoRnV++jc0Q4d503jEG9onltQxbLtmdTU6cMS+zKsISuJPXsQlLPLvSJjeJQQQVbDxex7UgxO44WMyShKwtunHxC897yHTncungd04bFk9C1M0s2ZtK1cwR3TB/KLdNaHr+njQfyeeC1NG6Zlsy3UwYQFvbVh9lnu45xy+J1jOjdledvm3JCE2jqvjzmPLee2jrl6esnceaw+JOOo7277KlP6RYVwfO3TQl2KJYwnKi8qpYDeWXEd+1EXIM7O//80Q7+/NFOrpzYnz9cPYHwsJZ/4/gw4yi3/yuVB2aO4nszhrb4+IrqWrKLKhHB/RBiOoV7bV6oqa3jnc2HeeC1NAb0jOaF26eQ0O2rn6usqoYfvLCR/23L5oqJ/Xlg5ij6xDpn+GB6ViGzFqyhTpXZZyUz++whreoDUlXe3XyEuW+mc6ykChG4ZvIA7rtoBIndvvq5c0sqeXPTIZbvyOFgXhmZ+eVU1tR9WR7ftROj+3ZnaEJXXtuQSeeIMBbedBqnDnA17209XMTV81czsFc0L98xla6dI9h+pJg/fLCNj7Zm06d7FI9cNoaLxvZp8TfXmto6vvHUZ+w4WkxtnXJGci8eu3IcQxK68vmeXG56Zi2D42J48fYpXmszB/PKuHXxOvYeK+XHF43k1mnJQe3XqK6t4/EPtnPdGQMdMzCkuraOsQ9/wC3TBvPgxaODHY4lDCfIOFTEC2v3s/NoCftySzlaVAm4PoxPHdCD80Ymct7oRD5IP8JfPt7FtyYn8burxp9Usqg351+prNiZw4f3Tve503l/bin/Wr2fl1MPUlxRc0L5hAE9uGhsb2aO7cOQhK5kFZTzn7UH+E/qQY4WVTK6b3eem316o9Mb1NTW8eT/dvL35XsIDxO+N2Moc84Z0mxTT6DtPVbK1fNX0TkinJfvmEr/Hv7rA8gvreLfa/Zz7qhEn9qnVZVjJVUcKaygT2zUcUl3V3YxtyxeR3ZRJX/+9qlMGtSTb/7tM1ThjTunnZCA1+3L4+dvpLPtSDHnj0rkkcvGtmjwwbOr9jH3zS387bpJlFRW86t3tlJZU8eNUwbx0toD9ImN4qU5U4+LsTFFFdX8+OVNLM04ypCEGOZ+YyzTRwRnNNCqXce4buHnnDk0judvOyPozT/gagK95C+f8pdZE7lsQr9gh2MJoy0czCujqraOwXExx33Ip2cV8uT/dvJhxlGiO4Uzum93BsfFkBwfzYBe0ew7VsbH246yKbPwy2O+nTKAx64cd1z1/2QcKijnwj8u5/TkXiy6+TRKKmt4L/0Ir2/IYnNWIYPiohmW2JXhiV3pE9uFdzcf5pPt2YSL8PVxfTl7uLsJQUFRcoor+TDjq1gH9OpCVn45CkwfkcB1pw/kvFGJzXbkHsgt47H3tvJe+hH6xUbxs0tGc8m4vkH54z1aVMFVT6+irKqWV+6YylCHt7XnllRy+79S2XCggH6xURSUV/PKHVMZ62V0TU1tHc98to8/fbSDOlV+eP4Ibjs7udlmymMllZz3+DLGJ/XgudmnIyJkF1Uw980tvJd+hEFx0fxnztQW1RI/3naUX76Vwb7cMi4c05s7pg+htg6KK6oprqihorqWEX26cUq/2IDVQh57byt/X74HgPk3TGbmKX0Ccp6WeHndQX7yWhof3zfdEX09ljAC7EhhBRf8cTkllTV0iQxnVN9ujOnbnaNFFXy0NZvuURHcelYyt0xL9trMkV1cwbJtOdSqntBW3BoLV+7hV+9s5ezh8azbl0dFdR2D4qKZNiyerPxydmWXkFXgGtkS37Uz150xkOvPGEjvJu42zSooZ+mWI6zceYwxfbvz7dMGnNSw2TV7cvnlWxlkHC7igtGJ/Oqb49q0maqwrJpr/r6azPwyXpwzJWRGcVVU13L/q2m8u/kw//jOZM4b1fxAgqyCcn7x5haWZhxldN/u/O6qcU3+vD95dRNLNmTx/j1nMyzx+MEWa/fmkRwf02zNojGVNbUs+nQff/14J2VVtY3u0zkijAlJPZg0qCfXpCT59UP04idXEt0pnOKKGsqqa/jw3ulBr+E+/N90lmzIIm3u1/z2d98aljAC7K4XNrA04yj/d8lo9h4rJeNQERmHiwgTYfZZydw8bTDdo4JzT0RNbR1XzV/NvmOlXDq+L1dOSmLSwB7HfZsvrazhYH4ZyfExdI5o2z+e2jrlmc/28vjS7USGhfGzS0Zz7WkDAl7bKKqo5tZn1pGWWcgzt5zGtBDrkFVVispriI1u2e/V++lHePi/6RwrqeS2s4dw7wUjTlg/esOBfK6ct4rvnjMkYG3q2UUVbDhQQNfOEXSLcj0iwsLYcqiQ1P35rN+fz5ZDhcR26cQbd55JUs/W38eTU1zJab/+iPsvGsmpA3pw/cLPuf+ikdx57jA//EQn78p5nxERHsbL33XGMj+WMALos13HuH7h5/zw/OHce+GIL7erKqo44htDda2rE7UtR0u11P7cUn762mZW78nlzKFxPH3D5IDdeLjlUCHff34Dmfnl/HXWRC4e1zcg53GqwvJqfvveNl5ce4CBvaK5YcpAJg7sybj+sUSGh3H53z4lp7iS/903g66dg3dr1q7sYq6Yt4p+sV145XtTW/2l6/WNmdz7n028dddZjEuK5bvPpbJy5zE+vm9G0AZg1NYpY+e+z3WnD+Lhb4wJSgwNtSRhOPcTxYEqa2r5+X/TGRQXfcJIJBFxRLIAV6JwcrIAGBQXwwu3n8FjV45j7d48fvLqJvz95UVVeWntAa6Yt4rK6jr+M2dKh0sWALFdInnsynG8NGcK0Z3C+c2727h6/mpOmfsBF/xxOelZRTx0yZigJguAYYndmH/DZHbnlHDn8xu+/OJzslbuOEavmE6MdU8f/tDFY6ipVX7//jZ/hHtSdueUUFFdF1Iz1Hpy9qeKwyxcuZc9OaU8ctnYoLeDtgciwqzTB/LTr4/igy1H+eene/323uVVtfz4lTR+umQzZyT34p27zwrKTY1OMmVIHO/fcw7rHrqAf3wnhTnnDKFP9yiumpTEN8Y7I5FOGxbPr684hZU7jzH3zS3HfYk4VFDOJ9uyySutavZ96uqUFTuPcdaw+C+/yA2Mi+a2s5NZsjGLT7Znc6ykkvKqWr9/UWlKeoje4V3PpgbxUWZ+GX/9eCcXje3NuSMTgx1OuzL7rGTW7s3jt+9tY+LAnkwe1LNV73e0qIJbF68j43ARPzx/OHefP7xVw5Xbm4RunblwTG8uHOPMu/C/fdpA9uWW8fSy3dTVKaVVtazfl8ehwgoAIsOF80YlctWkJM4dldhobXrbkWKOlVR+NfLP7c5zh/HahkxueWbdl9vCBIYmdOX528847p6ZQEjPKiIqMszxo/O8sYTho1++lYEgPPyNscEOpd0REf5w9QQu/etK7nphA+/cfTa9Yjq5ZozdnsO8Zbs4VFDBpRP6cvXkAQxL9P7HlnGoiNnPrqOovJqF30kJialJzInu/9pIDuSW8dK6g/TpHsXkwT25fVBPhid2Y9n2bN74IosPthwlLqYTD3x9FNc0WBdm5U7X6pvnNLj/I6ZzBEu+P401u3Mpq6qhtKqW4opq/rFyL794K4O/XTcpoD9X+qFCxvTtHrJfYCxh+OCLgwUszTjKT2aO9OtNXuYrsV0imXfdZK56ehX3/ucLrj1tAE99sosth4ro36MLI3p3ZeHKvfx9+R4mDezBVZOTOCM5jiHxMV82OXyyLZu7XthAt6hIXrnjzJBa+tIcLyxM+MuP1mwKAAARDElEQVSsicz9xhgSGwz5Pmt4PA98fRQrduQwb9lu/u/1dCYP6nnct/YVO3MY2btbo8PF+/fowlWTj58oOyoinCc+3MGVE48G7EtGXZ2ScaiIKyf1D8j7twVLGD5YuzcX4IRvMca/xiXF8vNvjOHnb6SzfEcOyfEx/P5b47liYn8iw8PILq7gjY1ZvJKayUOvpwMQ0ymcsf1iSerVhTc2ZjG6b3f+edNpjpqGxJyc8DA5IVnUiwwP4/zRvRmXFMsFTyznZ0s289KcKYgI5VW1rNubz3em+j6H2XenD+XttMP83xvpnDEkLiADAPblllJSWRNyU5p7soThg7TMQvr36NLo9BfGv244YyDVNXUkdOvMxeP6Hld1T+wWxZxzhnL72UPYmV3CpoMFpGcVkpZVyHubj3DR2D48fvUEYoI82se0ncRuUTx48WgeXLKZV1Izuea0AazZm0tVbd0JzVFN6RQRxm+uHMe35q/i8Q+288hl/m96Tj9UBMDYEB0hBYFdcW8RcCmQraqnNFI+CngGmAQ8pKqPNygPx7UiX5aqXhqoOH2RllnI+A68SE5bEhFuPSu52X1G9O7GiN7dvlzTXFUdMU+QaXvfThnA6xuy+PW7Wzl3VCIrdxyjc0QYpye3bFTc5EE9uXHKIJ5dvY/LTu3HpIGtG3zRUMahIiLDheGJvi1b4ESBHFa7GJjZRHkecDfwuJfyHwJb/RxTixWUVXEgr6xDr6oWCixZdFxhYcJvrjyF8qpaHn07g5U7czg9uddJDX2//6KR9O4WxYOvbaaqpnX3gTSUcbiI4YndQnpVwoBFrqorcCUFb+XZqroOqG5YJiJJwCXAwkDF56s098R7E0Jk3iFjOqJhid343oyhvLnpEDuzSzhn+MnNjtstKpJHv3kK248W8+jbGX69RyPjUFHID8Rwaqr7M/AToNkULyJzRCRVRFJzcnL8HkhaZgEQujfaGNNRfP/coQxJcK15cfaIk58r7MIxvbn97GSeW7OfP3200y+xZRdXcKykkjF9QzthOK53UETq+z3Wi8iM5vZX1QXAAnDNJeXveNIyC0mOjwnYPEfGGP/oHBHOX2dN5L3NRxjZu3X9BD+7eDSF5dX85X876dElstl+teZsPVwMEPI1DMclDGAacJmIXAxEAd1F5N+qekMwgknLLOSMIR17SgljQsXYfrFe1wppCRHhN1eMo6i8hl++nUFsl8gT7t1oiQz3CKnRzazl7nSOa5JS1QdVNUlVBwPXAh8HK1lkF1VwpKiCcdYcZUyHExEexpOzTmXasDh+8loaTy/bzSfbstlyqJBjJZXU1fneoJFx2HUDakunp3eaQA6rfRGYAcSLSCYwF4gEUNX5ItIH17DZ7kCdiNwDjFHVokDF1FJfdngPsA5vYzqizhHhLLgxhZsWreV3DWa5jYvpxHOzz/CpmSnjUGHIN0dBABOGqs5qpvwI0GQdT1WXAcv8F1XLpGUVEiZ8OT2yMabjiekcwcvfncrhogqOFlW4Wh4KK5i/fA+3PbuO/951VpOrEZZV1bDnWCmXjg/++t2t5cQ+DMdIyyxgeGI3ojvZZTKmIwsLE/r36HLcXHIpg3vxrfmr+O5zqbxw+xSv931sP1KMauh3eIMD+zCcQlXtDm9jjFen9I/lT9ecyoYDBTy4ZLPXezYyDrta2UN9SC1YwvAqq6CcvNIqxlv/hTHGi6+P68t9F47g9Y1ZzFu2u9F9Mg4V0S0qgqSeoT/TtSUML+o7vMfbCCljTBPuOm8Yl03oxx8+2M7yHSfePJxxuIgxfbu3i+lrLGF4kZZZSGS4MKpv6E4UZowJPBHh998az8Be0Tz50Y7jymrrlG2Hi9tF/wVYwvAqLbOAUX260znC1u42xjQtKjKcW6cNZsOBAjYcyP9y+/7cUsqra9tF/wVYwmhUXZ2y2Tq8jTEtcHXKALpFRfDPT/d+ua2+w3u0JYz2a19uKcWVNTZDrTHGZzGdI7jujIG8t/kwmfllgKvDOyJMGN7b+zr0ocQSRiPqO7xtDQxjTEvcNHUwIsKzq/YBrhrGsMSu7aZp2xJGI744WEBUZBjDE9vHtwJjTNvo16MLl4zry0trD1JSWdMu1sDwZAmjEZ/uOsZpg3sREW6XxxjTMrPPSqa4soZ5n+wiuzj018DwZJ+IDRzMK2NXdgkzRiYGOxRjTAiaMKAHpw3uyYIVe4D2MSVIPUsYDSzbng3AuSNPbolHY4yZfVYyNe7pz9tTDcNm1Wtg2fYcBsVFkxwfE+xQjDEh6sIxfRjQqwt1ddAjulOww/EbSxgeKqpr+Wz3Mb6dMqBd3MZvjAmO8DDhqVmTKKmsCXYofmUJw8Pne/OoqK5jxijrvzDGtE57XHgtYH0YIrJIRLJFJN1L+SgRWS0ilSLyY4/tA0TkExHJEJEtIvLDQMXY0CfbsukcEcbUIXFtdUpjjAkZgez0XgzMbKI8D7gbeLzB9hrgPlUdA0wB7hSRMQGJsIFl27M5c2ic14VQjDGmIwtYwlDVFbiSgrfybFVdB1Q32H5YVTe4nxcDW4H+gYqz3t5jpezLLeNca44yxphGOXpYrYgMBiYCnzexzxwRSRWR1JycE+ei91X9cNoZIyxhGGNMYxybMESkK/AacI+qFnnbT1UXqGqKqqYkJJz8vROfbM9hSEIMA+OiT/o9jDGmPXNkwhCRSFzJ4nlVXRLo85VX1bJmTy7n2t3dxhjjleMShrhugPgnsFVV/9gW51y95xhVNXWWMIwxpgkBuw9DRF4EZgDxIpIJzAUiAVR1voj0AVKB7kCdiNwDjAHGAzcCm0XkC/fb/UxV3w1UrJ9syyG6UzinJfcM1CmMMSbkBSxhqOqsZsqPAEmNFH0KtNlt1qrKJ9uzOXNofLuZs94YYwKhw9/pXVFdx5lD45g2LD7YoRhjjKN1+ITRpVM4v//WhGCHYYwxjue4Tm9jjDHOZAnDGGOMTyxhGGOM8YklDGOMMT6xhGGMMcYnljCMMcb4xBKGMcYYn1jCMMYY4xNR1WDH4DcikgPsB2KBQvfm5p7X/xsPHGvhKT3fz9fyhtuaet0wxmDH6i2+5uL2d7zeyny9tk78PWi4za5t87F6Kz+Za9vYto5ybQepqm9rQ6hqu3sAC3x97vFvamvO42t5w21NvW4kxqDG6pRr663M12vrxN8Du7bBvbZetnXoa9vYo702Sb3Vguee21pzHl/LG25r6nXDGIMda8Ntwbq23sp8vbZO/D1ouM2urW/H+uvaeitvqfZ0bU/QrpqkWkNEUlU1Jdhx+CKUYoXQijeUYoXQijeUYoXQiretYm2vNYyTsSDYAbRAKMUKoRVvKMUKoRVvKMUKoRVvm8RqNQxjjDE+sRqGMcYYn7TLhCEii0QkW0TST+LYySKyWUR2ichf3GuMIyL/EZEv3I99HsvHOi5Wd9kPRGSbiGwRkd/7I9ZAxSsij4hIlsf1vdipsXqU3yciKiJ+W3krQNf2URFJc1/XpSLSz8Gx/sH9O5smIq+LSA8Hx3q1+2+rTkT80nfQmji9vN9NIrLT/bjJY3uTv9tNaumwqlB4AOcAk4D0kzh2LTAF1zKx7wFfb2SfJ4CHnRorcC7wEdDZ/TrRydcWeAT4caj8HgADgA9w3fMT7+R4ge4e+9wNzHdwrF8DItzPfwf8zsGxjgZGAsuAlGDG6Y5hcINtvYA97n97up/3bO53u7lHu6xhqOoKIM9zm4gMFZH3RWS9iKwUkVENjxORvrj+wNao68r+C/hmg30EuAZ40cGxfg/4rapWus+R7Y9YAxhvQAQw1j8BPwH82gEYiHhVtchj1xh/xRygWJeqao171zVAkoNj3aqq2/0RX2vj9OIi4ENVzVPVfOBDYGZr/w7bZcLwYgHwA1WdDPwYmNfIPv2BTI/Xme5tns4GjqrqzoBE6dLaWEcAZ4vI5yKyXEROC2Cs4J9re5e7KWKRiPQMXKiti1VELgeyVHVTAGP01OprKyK/FpGDwPXAw06O1cOtuL79Boo/Yw0kX+JsTH/goMfr+thb9TN1iDW9RaQrcCbwikdzXeeTfLtZ+Kl20Rg/xRqBqyo6BTgNeFlEhri/UfiVn+J9GngU17ffR3E1+d3qrxjrtTZWEYkGfoar6STg/PV7q6oPAQ+JyIPAXcBcvwXp5s+/MRF5CKgBnvdPdCe8vz8/DwKmqThF5Bbgh+5tw4B3RaQK2KuqVwQqpg6RMHDVpApU9VTPjSISDqx3v3wT1weXZzU4Ccjy2D8CuBKY7PBYM4El7gSxVkTqcM01k+PEeFX1qMdx/wDeDkCc/oh1KJAMbHL/AScBG0TkdFU94sB4G3oeeJcAJAz89zd2M3ApcH4gvuD4M9Y20GicAKr6DPAMgIgsA25W1X0eu2QBMzxeJ+Hq68iiNT+TPzprnPgABuPReQSsAq52PxdggpfjGnYIXexRNhNY7vRYgTuAX7qfj8BVNRUHx9vXY597gZecGmuDffbhx07vAF3b4R77/AB41cGxzgQygAR/XtNA/h7gx07vk40T753ee3F1ePd0P+/l6++21/j8/R/jhAeuJqPDQDWub9uzcX0zfB/Y5P6lbHSUE5ACpAO7gafw+KAFFgN3OD1WoBPwb3fZBuA8h8f7HLAZSMP1za6vU2NtsM8+/DtKKhDX9jX39jRccwf1d3Csu3B9ufnC/fDXiK5AxHqF+70qgaPAB8GKk0YShnv7re5rugu4pSW/294edqe3McYYn3SkUVLGGGNawRKGMcYYn1jCMMYY4xNLGMYYY3xiCcMYY4xPLGGYdk1EStr4fAtFZIyf3qtWXLPMpovIW9LM7K0i0kNEvu+PcxvTGBtWa9o1ESlR1a5+fL8I/WqCvIDyjF1EngV2qOqvm9h/MPC2qp7SFvGZjsdqGKbDEZEEEXlNRNa5H9Pc208XkdUislFEVonISPf2m0XkTRH5GPifiMwQkWUi8qq41m943j2LMe7tKe7nJe6J/zaJyBoR6e3ePtT9erOI/MrHWtBqvpoAsauI/E9ENrjf43L3Pr8FhrprJX9w73u/+2dME5Ff+PEymg7IEobpiJ4E/qSqpwFXAQvd27cBZ6vqRFyzuv7G45hJwLdUdbr79UTgHmAMMASY1sh5YoA1qjoBWAHc7nH+J1V1HMfPHNoo9xxH5+O6Cx6gArhCVSfhWvvkCXfC+imwW1VPVdX7ReRrwHDgdOBUYLKInNPc+YzxpqNMPmiMpwuAMR4zgHZ3zwwaCzwrIsNxzZwb6XHMh6rquVbBWlXNBBDX6ouDgU8bnKeKryZSXA9c6H4+la/WIHgBeNxLnF3c790f2IprTQNwzQH0G/eHf527vHcjx3/N/djoft0VVwJZ4eV8xjTJEobpiMKAKapa4blRRJ4CPlHVK9z9Acs8iksbvEelx/NaGv9bqtavOgm97dOUclU91T2t+gfAncBfcK1rkQBMVtVqEdkHRDVyvACPqerfW3heYxplTVKmI1qKa+ZWAESkfvroWL6a6vnmAJ5/Da6mMIBrm9tZVctwLa96n3uK/Vgg250szgUGuXctBrp5HPoBcKu79oSI9BeRRD/9DKYDsoRh2rtoEcn0ePwI14dvirsjOAPXdPAAvwceE5GNBLb2fQ/wIxFJw7X4TWFzB6jqRlwzzs7Cta5FiohsBr6Dq+8FVc0FPnMPw/2Dqi7F1eS12r3vqxyfUIxpERtWa0wbczcxlauqisi1wCxVvby544wJNuvDMKbtTQaeco9sKiAAy9EaEwhWwzDGGOMT68MwxhjjE0sYxhhjfGIJwxhjjE8sYRhjjPGJJQxjjDE+sYRhjDHGJ/8PX2+sjlihUDYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find(start_lr=1e-8, end_lr=1e2)\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Carefully train the classifier\n",
    "During classification, we gradually unfreeze layers as shown below. This helps us obtain a better classification accuracy than if we were to aggressively train all the layers at once. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:00 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>1.124041</th>\n",
       "    <th>1.034543</th>\n",
       "    <th>0.511278</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(cyc_len=1, max_lr=1e-3, moms=(0.8, 0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:00 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>1.031176</th>\n",
       "    <th>1.056985</th>\n",
       "    <th>0.488722</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "learn.fit_one_cycle(1, slice(1e-4,1e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:00 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.951832</th>\n",
       "    <th>1.075786</th>\n",
       "    <th>0.473684</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "learn.fit_one_cycle(1, slice(1e-5,5e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:04 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.888372</th>\n",
       "    <th>1.051242</th>\n",
       "    <th>0.413534</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>0.859089</th>\n",
       "    <th>1.085068</th>\n",
       "    <th>0.451128</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>0.853765</th>\n",
       "    <th>1.041656</th>\n",
       "    <th>0.436090</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>0.808356</th>\n",
       "    <th>1.068729</th>\n",
       "    <th>0.406015</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(4, slice(1e-5,1e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-tabulate the predictions for each class\n",
    "The below table shows the predicted classes for the topic in concern. The actual evaluation of the F1 score of the classification result is done at a later step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>col_0</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row_0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>30</td>\n",
       "      <td>45</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "col_0   0   1   2\n",
       "row_0            \n",
       "0      30  45  23\n",
       "1       6  18   1\n",
       "2       0   4   6"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get predictions\n",
    "preds, targets = learn.get_preds()\n",
    "predictions = np.argmax(preds, axis=1)\n",
    "pd.crosstab(predictions, targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the predicted results from the classifier (SemEval tweets only)\n",
    "We read in the test dataset and store in a Pandas DataFrame as shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Target</th>\n",
       "      <th>Tweet</th>\n",
       "      <th>Stance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Atheism</td>\n",
       "      <td>He who exalts himself shall      be humbled; a...</td>\n",
       "      <td>UNKNOWN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Atheism</td>\n",
       "      <td>RT @prayerbullets: I remove Nehushtan -previou...</td>\n",
       "      <td>UNKNOWN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Atheism</td>\n",
       "      <td>@Brainman365 @heidtjj @BenjaminLives I have so...</td>\n",
       "      <td>UNKNOWN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Atheism</td>\n",
       "      <td>#God is utterly powerless without Human interv...</td>\n",
       "      <td>UNKNOWN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Atheism</td>\n",
       "      <td>@David_Cameron   Miracles of #Multiculturalism...</td>\n",
       "      <td>UNKNOWN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Target                                              Tweet   Stance\n",
       "0  Atheism  He who exalts himself shall      be humbled; a...  UNKNOWN\n",
       "1  Atheism  RT @prayerbullets: I remove Nehushtan -previou...  UNKNOWN\n",
       "2  Atheism  @Brainman365 @heidtjj @BenjaminLives I have so...  UNKNOWN\n",
       "3  Atheism  #God is utterly powerless without Human interv...  UNKNOWN\n",
       "4  Atheism  @David_Cameron   Miracles of #Multiculturalism...  UNKNOWN"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = pd.read_csv(path/testfile, delimiter='\\t', header=0, encoding = \"latin-1\")\n",
    "test = test.drop(['ID'], axis=1)\n",
    "# test['Tweet'] = test['Tweet'].apply(clean_ascii)\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we do not yet know the stance for the test set, it is marked as \"UNKNOWN\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filter the test data set based on topic we are classifying for"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Target</th>\n",
       "      <th>Tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>669</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>@Maisie_Williams is our hero with her #LikeAGi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>670</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>Rather be an \"ugly\" feminist then be these sad...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>671</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>iamNovaah: RT ChrzOC: Bitches be running wild....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>672</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>@angerelle you disagree that people should str...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>673</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>#Rapeculture is basically a FABLE. It has almo...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Target                                              Tweet\n",
       "669  Feminist Movement  @Maisie_Williams is our hero with her #LikeAGi...\n",
       "670  Feminist Movement  Rather be an \"ugly\" feminist then be these sad...\n",
       "671  Feminist Movement  iamNovaah: RT ChrzOC: Bitches be running wild....\n",
       "672  Feminist Movement  @angerelle you disagree that people should str...\n",
       "673  Feminist Movement  #Rapeculture is basically a FABLE. It has almo..."
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_pred = test[['Target', 'Tweet']]\n",
    "test_pred = test_pred.loc[test_pred['Target'] == topic]\n",
    "test_pred.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply the learner's ```predict``` method to produce our prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred['Stance'] = test_pred['Tweet'].apply(lambda row: str(learn.predict(row)[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Target</th>\n",
       "      <th>Tweet</th>\n",
       "      <th>Stance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>669</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>@Maisie_Williams is our hero with her #LikeAGi...</td>\n",
       "      <td>FAVOR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>670</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>Rather be an \"ugly\" feminist then be these sad...</td>\n",
       "      <td>FAVOR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>671</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>iamNovaah: RT ChrzOC: Bitches be running wild....</td>\n",
       "      <td>AGAINST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>672</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>@angerelle you disagree that people should str...</td>\n",
       "      <td>AGAINST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>673</th>\n",
       "      <td>Feminist Movement</td>\n",
       "      <td>#Rapeculture is basically a FABLE. It has almo...</td>\n",
       "      <td>AGAINST</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Target                                              Tweet  \\\n",
       "669  Feminist Movement  @Maisie_Williams is our hero with her #LikeAGi...   \n",
       "670  Feminist Movement  Rather be an \"ugly\" feminist then be these sad...   \n",
       "671  Feminist Movement  iamNovaah: RT ChrzOC: Bitches be running wild....   \n",
       "672  Feminist Movement  @angerelle you disagree that people should str...   \n",
       "673  Feminist Movement  #Rapeculture is basically a FABLE. It has almo...   \n",
       "\n",
       "      Stance  \n",
       "669    FAVOR  \n",
       "670    FAVOR  \n",
       "671  AGAINST  \n",
       "672  AGAINST  \n",
       "673  AGAINST  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_pred.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output the predicted dataset to a text file for comparison with the gold reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred.to_csv(path/'eval'/'predicted.txt', sep='\\t', index=True,\n",
    "                 header=['Target', 'Tweet', 'Stance'], index_label='ID')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have the predicted data, we use the *perl* script provided by the SemEval organizers to generate the F1 score. \n",
    "\n",
    "The evaluation script is in ```data/eval/``` and has the following usage:\n",
    "\n",
    "    perl eval.py gold_feminism.txt predicted.txt\n",
    "    \n",
    "    ============\n",
    "    Results\t\t\t\t \n",
    "    ============\n",
    "    FAVOR     precision: 0.3265 recall: 0.5517 f-score: 0.4103\n",
    "    AGAINST   precision: 0.7259 recall: 0.5355 f-score: 0.6164\n",
    "    ------------\n",
    "    Macro F: 0.5133"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a relatively small input vocabulary from the Twitter training data, we do not do that well in the F-score for the \"Feminist Movement\" topic.  On trying out a similar approach for Tweets from the other topics (Atheism, Hillary Clinton, etc.), we still do not see an F-score of above 0.60 - this is well below the winning score by *MITRE* of 0.67.\n",
    "\n",
    "To improve the score, some more fine-tuning is needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 Train the Classifier: Augmented Vocabulary\n",
    "Since the initial approach (despite multiple attempts at fine-tuning both the language model and the classifier) did not an F-score as high as *MITRE's* (which was ~0.67), it can be reasoned that this could be because the pre-trained language model did not generalize to Tweets specifically (since the language syntax of tweets are quite different than the Wikipedia text that the LSTMs were trained on). \n",
    "\n",
    "The provided training data of 2,800 Tweets can be augmented with the Kaggle [Twitter Sentiment140 dataset](https://www.kaggle.com/kazanova/sentiment140) of 1.6 million tweets. By fine-tuning the language model on this larger Twitter dataset, we might better learn the structure of Twitter conversations. \n",
    "\n",
    "**NOTE**: Since 1.6 million tweets is quite a large dataset and can add a significant number of unique, one-time-only words, in this notebook, we only sampled a subset of 200,000 from the original Sentiment140 dataset for the language model fine-tuning to save on language model fine-tuning time. It is possible that fine-tuning the language model with a larger subset of the augmentation vocabulary could further improve the language model fine-tuned results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Load Kaggle Twitter Dataset\n",
    "kaggle = pd.read_csv(path/\"training.1600000.processed.noemoticon.csv\",header=None,encoding = \"latin-1\")\n",
    "kaggle.columns = ['col1', 'col2',\"col3\",\"Stance\",\"col5\",\"Tweet\"]\n",
    "kaggle = kaggle[['Stance','Tweet']]\n",
    "# Clean Data and Remove URL's from the tweets\n",
    "kaggle['Tweet'] = kaggle['Tweet'].apply(clean_ascii)\n",
    "kaggle['Tweet'] = kaggle['Tweet'].replace(r'http\\S+', '', regex=True).replace(r'www\\S+', '', regex=True)\n",
    "kaggle_sub = kaggle.sample(n=200000,random_state=77)\n",
    "kaggle_sub = kaggle_sub.append(train, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# The Lines are commented out as this takes a long time to run\n",
    "# Write train to csv\n",
    "kaggle_sub.to_csv(path/'kaggle_train_sub_v2.csv', index=False, header=False)\n",
    "twitter_lm = TextLMDataBunch.from_csv(path, 'kaggle_train_sub_v2.csv', min_freq=1)\n",
    "kaggle_lm.save()\n",
    "twitter_lm.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "learn_twitter = language_model_learner(twitter_lm, pretrained_model=URLs.WT103_1, drop_mult=0.1)\n",
    "learn_twitter.lr_find(start_lr=1e-8, end_lr=1e2)\n",
    "learn_twitter.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_twitter = language_model_learner(twitter_lm, pretrained_model=URLs.WT103_1,drop_mult=0.5)\n",
    "learn_twitter.fit_one_cycle(cyc_len=1, max_lr=1e-3, moms=(0.8, 0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn_twitter.unfreeze()\n",
    "learn_twitter.fit_one_cycle(cyc_len=10, max_lr=1e-3, moms=(0.8, 0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the fine-tuned encoder (Uncomment the line given below)\n",
    "learn_twitter.save_encoder('ft_enc_kaggle_v1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Vocabulary\n",
    "This assumes that we have a fine-tuned language model's numericalized vocabulary ```vocab.pkl``` in the same directory as the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Save the Vocab to a Pickle file for later use\n",
    "# pickle.dump(twitter_lm.train_ds.vocab, open(f'vocab.pkl','wb'))\n",
    "vocab_load = pickle.load(open('vocab.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Various values of momentum, dropout and learning rates were tried to find the best values of these parameters. The below parameters are applied in a for loop to train on each topic one by one, and then appending the predictions to a common file for comparison with the gold reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for topic in train_orig.Target.unique():\n",
    "    print(\"Topic: \",topic)\n",
    "    #learn_twitter=learn_twitter_org\n",
    "        # Get only those tweets that pertain to a single topic in the training data\n",
    "    train_topic = train.loc[train_orig['Target'] == topic]\n",
    "    # Write train to csv\n",
    "    train_topic.to_csv(path/'train_topic.csv', index=False, header=False)\n",
    "    # Classifier model data\n",
    "    data_clas = TextClasDataBunch.from_csv(path, 'train_topic.csv', vocab=vocab_load, min_freq=7, bs=64)\n",
    "    data_clas.save()\n",
    "    learn = text_classifier_learner(data_clas, drop_mult=0.5)\n",
    "    learn.load_encoder('ft_enc_kaggle_v1')\n",
    "    learn.freeze()\n",
    "\n",
    "    #Find Optimum Learning Rate\n",
    "    learn.lr_find(start_lr=1e-6, end_lr=1e2)\n",
    "    learn.recorder.plot()\n",
    "\n",
    "    #Train the Classifier by Gradually Unfreezing the Layers\n",
    "    learn.fit_one_cycle(cyc_len=1, max_lr=1e-3, moms=(0.8, 0.7))\n",
    "    #Unfreeze Layer\n",
    "    learn.freeze_to(-2)\n",
    "    learn.fit_one_cycle(3, slice(1e-4,1e-2), moms=(0.8,0.7))\n",
    "    #Unfreeze Layer\n",
    "    learn.freeze_to(-3)\n",
    "    learn.fit_one_cycle(3, slice(1e-5,5e-3), moms=(0.8,0.7))\n",
    "    #Unfreeze All Layers\n",
    "    learn.unfreeze()\n",
    "    learn.fit_one_cycle(3, slice(1e-5,1e-3), moms=(0.8,0.7))\n",
    "    # get predictions\n",
    "    preds, targets = learn.get_preds()\n",
    "    predictions = np.argmax(preds, axis=1)\n",
    "    pd.crosstab(predictions, targets)\n",
    "\n",
    "    test = pd.read_csv(path/testfile, delimiter='\\t', header=0, encoding = \"latin-1\")\n",
    "    # test = test.drop(['ID'], axis=1)\n",
    "    # test['Tweet'] = test['Tweet'].apply(clean_ascii)\n",
    "    test.head()\n",
    "\n",
    "    test_pred = test[['Target', 'Tweet']]\n",
    "    test_pred = test_pred.loc[test_pred['Target'] == topic]\n",
    "    test_pred.tail()\n",
    "\n",
    "    test_pred['Stance'] = test_pred['Tweet'].apply(lambda row: str(learn.predict(row)[0]))\n",
    "    file_to_save=\"predicted_\"+str(topic.replace(\" \", \"_\"))+\".txt\"\n",
    "    test_pred.to_csv(path/'eval'/file_to_save, sep='\\t', index=True,\n",
    "                     header=['Target', 'Tweet', 'Stance'], index_label='ID')\n",
    "    if(topic==\"Atheism\"):\n",
    "        preds_document=test_pred\n",
    "    else:\n",
    "        preds_document=preds_document.append(test_pred, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output the predicted dataset to a text file for comparison with the gold reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_document.to_csv(path/'eval'/'predicted.txt', sep='\\t', index=True,\n",
    "                      header=['Target', 'Tweet', 'Stance'], index_label='ID')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The evaluation script is in ```data/eval/``` is run with the following command:\n",
    "\n",
    "    perl eval.py gold.txt predicted.txt\n",
    "\n",
    "    ============\n",
    "    Results                          \n",
    "    ============\n",
    "    FAVOR     precision: 0.5842 recall: 0.5362 f-score: 0.5592\n",
    "    AGAINST   precision: 0.7404 recall: 0.7301 f-score: 0.7352\n",
    "    ------------\n",
    "    Macro F: 0.6472"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the above result, we can see that augmenting the training data with additional domain-specific data (i.e., Tweets) helps us obtain better F-scores for the stance classification task.\n",
    "\n",
    "This result is much closer to the winning result by *MITRE* (of 0.67). The improvement can be largely attributed to the language model fine-tuning step, where we augmented the language model parameters with the tokenized form of a much larger Tweet vocabulary (of 200,000 Tweets downloaded from Kaggle). It can be reasoned that using an even larger Tweet vocabulary (say a million words) could result in a further improvement. However, this would also significantly increase the training time during fine-tuning. In addition to vocabulary augmentation, more extensive hyperparameter tuning and careful classifier fine-tuning could result in further gains. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Conclusions\n",
    "\n",
    "This notebook showed a training and classification pipeline for a PyTorch-based framework (*ULMFit*) for evaluating stance of Tweets towards a particular topic. It appears that with some basic data cleaning and augmentation with additional Twitter language data, the transfer learning approach using *ULMFit* can provide good classification accuracy F-scores (compared with the winning results from the SemEval 2016 winning team *MITRE*). \n",
    "\n",
    "The main benefit of using *ULMFit* is that it can perform classifier re-training with a *very limited amount of data* (fewer than 500 Tweets per topic in this dataset). However, since Twitter data is of a very different distribution from the pre-trained language model, the language model fine-tuning step is rather expensive (and can take significant time even on GPU). Once the language model is fine-tuned adequately, the classifier can perform per-topic classification relatively easily and with a reasonable level of accuracy comparable with the best score by *MITRE*. \n",
    "\n",
    "We can likely further improve the classification accuracy across all topics with the below steps:\n",
    "\n",
    "1.  Augment the language model vocabulary with a larger subset of Twitter Sentiment140 dataset to improve the model's domain understanding of Tweet syntax\n",
    "\n",
    "2.  Perform additional fine-tuning on the training hyperparameters in both the language model fune-tuning and the classifier fine-tuning stages. \n",
    "\n",
    "3.  Plug in a pre-trained language model that was trained on a large Twitter dataset (if such a model becomes available in the future!)\n",
    "\n",
    "Tweets are sufficiently different from typical language data used to generate pre-trained language models, and  hence are an interesting usecase for analyzing the effectiveness of transfer learning techniques. It will be interesting to see how transfer learning techniques for NLP tasks evolve with time."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}