lm3-portuguese-classifier-olist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (MultiFiT) Portuguese Sentiment Classifier on Brazilian E-Commerce Public Dataset by Olist\n",
    "- **Architecture 4 QRNN with 1550 hidden parameters by layer, SentencePiece tokenizer (15 000 tokens)**\n",
    "- **Hyperparameters and training method from the MultiFiT paper (except the Label Smoothing for the loss)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n",
    "- Date: September 2019\n",
    "- Post in medium: [link](https://medium.com/@pierre_guillou/nlp-fastai-portuguese-language-model-980c8ec75362)\n",
    "- Ref: [Fastai v1](https://docs.fast.ai/) (Deep Learning library on PyTorch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Information**\n",
    "\n",
    "According to this new article \"[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)\" (September 10, 2019), the QRNN architecture and the SentencePiece tokenizer give better results than AWD-LSTM and the spaCy tokenizer respectively. \n",
    "\n",
    "Therefore, they have been used in this notebook to **fine-tune a Portuguese Bidirectional Language Model** by Transfer Learning of a Portuguese Bidirectional Language Model (with the QRNN architecture and the SentencePiece tokenizer, too) trained on a Wikipedia corpus of 100 millions tokens ([lm3-portuguese.ipynb](https://github.com/piegu/language-models/blob/master/lm3-portuguese.ipynb)). \n",
    "\n",
    "This Portuguese Bidirectional Language Model has been **fine-tuned on \"[Brazilian E-Commerce Public Reviews Dataset by Olist](https://www.kaggle.com/olistbr/brazilian-ecommerce/metadata)\"** and **its encoder part has been transfered to a sentiment classifier which has been finally trained on this Olist corpus**.\n",
    "\n",
    "This process **LM General --> LM fine-tuned --> Classifier fine-tuned** is called [ULMFiT](http://nlp.fast.ai/category/classification.html) but we trained our 3 models with the hyperparameters values of the [MultiFiT](https://arxiv.org/abs/1909.04761) method given at the end of the MultiFiT article.\n",
    "\n",
    "**Hyperparameters values**\n",
    "\n",
    "- Language Model\n",
    "    - (batch size) bs = 50\n",
    "    - (QRNN) 4 QRNN (default: 3) with 1550 hidden parameters each one (default: 1152)\n",
    "    - (SentencePiece) vocab of 15000 tokens\n",
    "    - (dropout) mult_drop = 1.0\n",
    "    - (weight decay) wd = 0.1\n",
    "    - (number of training epochs) 17 epochs\n",
    "    - (learning rate) modified version of 1-cycle learning rate schedule (Smith, 2018) that uses cosine instead of linear annealing, cyclical momentum and discriminative finetuning\n",
    "    - (loss) FlattenedLoss of CrossEntropyLoss\n",
    "    \n",
    "\n",
    "- Sentiment Classifier\n",
    "    - (batch size) bs = 18\n",
    "    - (SentencePiece) vocab of 15000 tokens\n",
    "    - (dropout) mult_drop = 1.0 \n",
    "    - (weight decay) wd = 0.1\n",
    "    - (number of training epochs) 9 epochs\n",
    "    - (learning rate) modified version of 1-cycle learning rate schedule (Smith, 2018) that uses cosine instead of linear annealing, cyclical momentum and discriminative finetuning\n",
    "    - (loss) FlattenedLoss of weighted CrossEntropyLoss\n",
    "    \n",
    "**Note**: the MultiFiT values of mult_drop and weight decay are respectively 0.5 and 0.01 but as the Olist dataset after pre-processing is small, we need to increase regularization. Therefore, we did test and we finally chose mult_drop = 1.0 and wd = 0.1 both for fine-tuning the Bidirectional LM and the Sentiment Classifier. More, we trained the Bidirectional LM on 17 epochs and not 20 for the same reason (our models overfitted beacuse of the small size of teh dataset on 20 epochs). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We can conclude that this Bidirectional Portuguese LM model using the MultiFiT configuration is a good model to perform text classification but with only 46 millions of parameters, it is far from being a LM that can gan compete with [GPT-2](https://openai.com/blog/better-language-models/) or [BERT](https://arxiv.org/abs/1810.04805) in NLP tasks like text generation.**\n",
    "\n",
    "- **About the data**: the dataset \"Brazilian E-Commerce Public Dataset by Olist\", even after pre-processing (deleting empty reviews, neutral ones, etc.), is unbalanced. Therefore, we used a weighted loss function (FlattenedLoss of weighted CrossEntropyLoss).\n",
    "    - initial number of reviews: 100 000\n",
    "    - number of reviews after pre-processing: 37 770\n",
    "        - neg: 11270 (29.84%)\n",
    "        - pos: 26500 (70.16%)\n",
    "\n",
    "\n",
    "- **(fine-tuned) Language Model**   \n",
    "    - forward : (accuracy) 34.74% | (perplexity) 27.42\n",
    "    - backward: (global) 38.24% | (perplexity) 27.84\n",
    "    \n",
    "\n",
    "- **(fine-tuned) Sentiment Classifier**\n",
    "\n",
    "    - **Accuracy**\n",
    "        - forward : (global) **92.80%** | (neg) 95.10% | **(pos) 91.80%**\n",
    "        - backward: (global) 91.08% | **(neg) 96.33%** | (pos) 88.80%\n",
    "        - ensemble: (global) 92.30% | (neg) 95.80% | (pos) 90.77%\n",
    "\n",
    "    - **f1 score**\n",
    "        - forward: **0.9467**\n",
    "        - backward: 0.9328\n",
    "        - ensemble: 0.9426\n",
    "    \n",
    "\n",
    "(neg = negative reviews | pos = positive reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To be improved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lost function FlattenedLoss of LabelSmoothingCrossEntropy should be tested as it is used in the MultiFiT method (see the notebook [lm3-portuguese-classifier-TCU-jurisprudencia.ipynb](https://github.com/piegu/language-models/blob/master/lm3-portuguese-classifier-TCU-jurisprudencia.ipynb) to get the code)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "@np_func\n",
    "def f1(inp,targ): return f1_score(targ, np.argmax(inp, axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bs=48\n",
    "# bs=24\n",
    "bs=50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.set_device(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = Config.data_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents. (For other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang = 'pt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)\n",
    "\n",
    "lm_fns3 = [f'{lang}_wt_sp15_multifit', f'{lang}_wt_vocab_sp15_multifit']\n",
    "lm_fns3_bwd = [f'{lang}_wt_sp15_multifit_bwd', f'{lang}_wt_vocab_sp15_multifit_bwd']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Brazilian E-Commerce Public Dataset by Olist](https://www.kaggle.com/olistbr/brazilian-ecommerce/metadata)\n",
    "- File: olist_order_reviews_dataset.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/olist/olist_products_dataset.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/olist_sellers_dataset.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/olist_order_items_dataset.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/7'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/olist_reviews_tobechecked_pt.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/brazilian-ecommerce.zip'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/product_category_name_translation.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/olist_orders_dataset.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/olist_reviews_filtered_pt.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/olist_geolocation_dataset.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/olist_order_payments_dataset.csv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/olist/olist_order_reviews_dataset.csv')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "name = 'olist'\n",
    "path_data = data_path/name\n",
    "path_data.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run this code the first time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from converter import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to solve display error of pandas dataframe\n",
    "get_ipython().config.get('IPKernelApp', {})['parent_appname'] = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_id</th>\n",
       "      <th>order_id</th>\n",
       "      <th>review_score</th>\n",
       "      <th>review_comment_title</th>\n",
       "      <th>review_comment_message</th>\n",
       "      <th>review_creation_date</th>\n",
       "      <th>review_answer_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>7bc2406110b926393aa56f80a40eba40</td>\n",
       "      <td>73fc7af87114b39712e6da79b0a377eb</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2018-01-18 00:00:00</td>\n",
       "      <td>2018-01-18 21:46:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>80e641a11e56f04c1ad469d5645fdfde</td>\n",
       "      <td>a548910a1c6147796b98fdf73dbeba33</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2018-03-10 00:00:00</td>\n",
       "      <td>2018-03-11 03:05:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>228ce5500dc1d8e020d8d1322874b6f0</td>\n",
       "      <td>f9e4b658b201a9f2ecdecbb34bed034b</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2018-02-17 00:00:00</td>\n",
       "      <td>2018-02-18 14:36:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>e64fb393e7b32834bb789ff8bb30750e</td>\n",
       "      <td>658677c97b385a9be170737859d3511b</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Recebi bem antes do prazo estipulado.</td>\n",
       "      <td>2017-04-21 00:00:00</td>\n",
       "      <td>2017-04-21 22:02:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>f7c4243c7fe1938f181bec41a392bdeb</td>\n",
       "      <td>8e6bfb81e283fa7e4f11123a3fb894f1</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Parabéns lojas lannister adorei comprar pela I...</td>\n",
       "      <td>2018-03-01 00:00:00</td>\n",
       "      <td>2018-03-02 10:26:53</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          review_id                          order_id  \\\n",
       "0  7bc2406110b926393aa56f80a40eba40  73fc7af87114b39712e6da79b0a377eb   \n",
       "1  80e641a11e56f04c1ad469d5645fdfde  a548910a1c6147796b98fdf73dbeba33   \n",
       "2  228ce5500dc1d8e020d8d1322874b6f0  f9e4b658b201a9f2ecdecbb34bed034b   \n",
       "3  e64fb393e7b32834bb789ff8bb30750e  658677c97b385a9be170737859d3511b   \n",
       "4  f7c4243c7fe1938f181bec41a392bdeb  8e6bfb81e283fa7e4f11123a3fb894f1   \n",
       "\n",
       "   review_score review_comment_title  \\\n",
       "0             4                  NaN   \n",
       "1             5                  NaN   \n",
       "2             5                  NaN   \n",
       "3             5                  NaN   \n",
       "4             5                  NaN   \n",
       "\n",
       "                              review_comment_message review_creation_date  \\\n",
       "0                                                NaN  2018-01-18 00:00:00   \n",
       "1                                                NaN  2018-03-10 00:00:00   \n",
       "2                                                NaN  2018-02-17 00:00:00   \n",
       "3              Recebi bem antes do prazo estipulado.  2017-04-21 00:00:00   \n",
       "4  Parabéns lojas lannister adorei comprar pela I...  2018-03-01 00:00:00   \n",
       "\n",
       "  review_answer_timestamp  \n",
       "0     2018-01-18 21:46:59  \n",
       "1     2018-03-11 03:05:13  \n",
       "2     2018-02-18 14:36:24  \n",
       "3     2017-04-21 22:02:06  \n",
       "4     2018-03-02 10:26:53  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path_data/'olist_order_reviews_dataset.csv', encoding='utf-8')\n",
    "print(len(df))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# columns names\n",
    "reviews = \"review_comment_message\"\n",
    "idx = \"review_id\"\n",
    "rating = \"review_score\"\n",
    "label = \"label\"\n",
    "\n",
    "# keep columns\n",
    "df2 = df[[idx,reviews,rating]].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(orginal csv) number of all reviews: 100000\n",
      "58274 empty reviews were deleted\n",
      "from the 322 identical reviews ids, only the first one has been kept.\n",
      "3634 neutral reviews (rating = 3) were deleted\n",
      "0 reviews with nan label were deleted\n",
      "\n",
      "number of neg reviews (rating = 1 or 2): 11270 (29.84%)\n",
      "number of pos reviews (rating = 4 or 5): 26500 (70.16%)\n",
      "\n",
      "(final) number of all reviews neg and pos: 37770\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_id</th>\n",
       "      <th>review_comment_message</th>\n",
       "      <th>review_score</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>e64fb393e7b32834bb789ff8bb30750e</td>\n",
       "      <td>Recebi bem antes do prazo estipulado.</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>f7c4243c7fe1938f181bec41a392bdeb</td>\n",
       "      <td>Parabéns lojas lannister adorei comprar pela I...</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>8670d52e15e00043ae7de4c01cc2fe06</td>\n",
       "      <td>aparelho eficiente. no site a marca do aparelh...</td>\n",
       "      <td>4</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>4b49719c8a200003f700d3d986ea1a19</td>\n",
       "      <td>Mas um pouco ,travando...pelo valor ta Boa.\\r\\n</td>\n",
       "      <td>4</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>3948b09f7c818e2d86c9a546758b2335</td>\n",
       "      <td>Vendedor confiável, produto ok e entrega antes...</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>9314d6f9799f5bfba510cc7bcd468c01</td>\n",
       "      <td>GOSTARIA DE SABER O QUE HOUVE, SEMPRE RECEBI E...</td>\n",
       "      <td>2</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>373cbeecea8286a2b66c97b1b157ec46</td>\n",
       "      <td>Péssimo</td>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>d21bbc789670eab777d27372ab9094cc</td>\n",
       "      <td>Loja nota 10</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>0e0190b9db53b689b285d3f3916f8441</td>\n",
       "      <td>obrigado pela atençao amim dispensada</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>fe3db7c069d694bab50cc43463f91608</td>\n",
       "      <td>A compra foi realizada facilmente.\\r\\nA entreg...</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           review_id  \\\n",
       "3   e64fb393e7b32834bb789ff8bb30750e   \n",
       "4   f7c4243c7fe1938f181bec41a392bdeb   \n",
       "9   8670d52e15e00043ae7de4c01cc2fe06   \n",
       "12  4b49719c8a200003f700d3d986ea1a19   \n",
       "15  3948b09f7c818e2d86c9a546758b2335   \n",
       "16  9314d6f9799f5bfba510cc7bcd468c01   \n",
       "19  373cbeecea8286a2b66c97b1b157ec46   \n",
       "22  d21bbc789670eab777d27372ab9094cc   \n",
       "24  0e0190b9db53b689b285d3f3916f8441   \n",
       "27  fe3db7c069d694bab50cc43463f91608   \n",
       "\n",
       "                               review_comment_message  review_score label  \n",
       "3               Recebi bem antes do prazo estipulado.             5   pos  \n",
       "4   Parabéns lojas lannister adorei comprar pela I...             5   pos  \n",
       "9   aparelho eficiente. no site a marca do aparelh...             4   pos  \n",
       "12    Mas um pouco ,travando...pelo valor ta Boa.\\r\\n             4   pos  \n",
       "15  Vendedor confiável, produto ok e entrega antes...             5   pos  \n",
       "16  GOSTARIA DE SABER O QUE HOUVE, SEMPRE RECEBI E...             2   neg  \n",
       "19                                            Péssimo             1   neg  \n",
       "22                                       Loja nota 10             5   pos  \n",
       "24              obrigado pela atençao amim dispensada             5   pos  \n",
       "27  A compra foi realizada facilmente.\\r\\nA entreg...             5   pos  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of reviews\n",
    "print(f'(orginal csv) number of all reviews: {len(df2)}')\n",
    "\n",
    "# keep not null reviews\n",
    "## delete nan reviews\n",
    "empty_nan = (df2[reviews].isnull()).sum()\n",
    "df2 = df2[df2[reviews].notnull()]\n",
    "## delete empty reviews\n",
    "list_idx_none = []\n",
    "for idxs, row in df2.iterrows():\n",
    "    if row[reviews].strip() == \"\":\n",
    "        df2.drop(idxs, axis=0, inplace=True)\n",
    "        list_idx_none.append(idxs)\n",
    "empty_none = len(list_idx_none)\n",
    "## print results\n",
    "empty = empty_nan+empty_none\n",
    "if empty != 0:\n",
    "    print(f'{empty} empty reviews were deleted')\n",
    "else:\n",
    "    print('there is no empty review.')\n",
    "\n",
    "# check that there is no twice the same review\n",
    "# keep the first of unique review_id reviews\n",
    "same = len(df2) - len(df2[idx].unique())\n",
    "if same != 0:\n",
    "    df2.drop_duplicates(subset=[idx], inplace=True)\n",
    "    print(f'from the {same} identical reviews ids, only the first one has been kept.')\n",
    "else:\n",
    "    print('there is no identical review id.')\n",
    "\n",
    "# categorify reviews in 2 classes neg, pos in the label column (rating != 3)\n",
    "neutral = len(df2[df2[rating] == 3])\n",
    "df2 = df2[df2[rating] != 3]\n",
    "df2[label] = 'neg'\n",
    "df2.loc[df2[rating] > 3, label] = 'pos'\n",
    "print(f'{neutral} neutral reviews (rating = 3) were deleted')\n",
    "\n",
    "## delete nan labels\n",
    "empty_label_nan = (df2[label].isnull()).sum()\n",
    "df2 = df2[df2[label].notnull()]\n",
    "print(f'{empty_label_nan} reviews with nan label were deleted')\n",
    "\n",
    "# number of reviews neg or pos\n",
    "num_neg = len(df2[df2[label] == 'neg'])\n",
    "num_pos = len(df2[df2[label] == 'pos'])\n",
    "num_neg_pos = len(df2)\n",
    "pc_neg = round((num_neg/num_neg_pos)*100,2)\n",
    "pc_pos = round((num_pos/num_neg_pos)*100,2)\n",
    "print(f'\\nnumber of neg reviews (rating = 1 or 2): {num_neg} ({pc_neg}%)')\n",
    "print(f'number of pos reviews (rating = 4 or 5): {num_pos} ({pc_pos}%)')\n",
    "print(f'\\n(final) number of all reviews neg and pos: {num_neg_pos}')  \n",
    "\n",
    "# convert HTML caracters to normal letters\n",
    "df2[reviews] = df2[reviews].apply(convert)\n",
    "\n",
    "df2.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Delete the non French reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**FastText**\n",
    "\n",
    "- library source: https://github.com/facebookresearch/fastText/tree/master/python\n",
    "- Blog post about using fasttext for language detection: https://amitness.com/2019/07/identify-text-language-python/\n",
    "- \"New release of python module\" (june 2019): https://fasttext.cc/blog/2019/06/25/blog-post.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fasttext\n",
    "import urllib.request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(PosixPath('/home/jupyter/tutorials/fastai/course-nlp/lid.176.bin'),\n",
       " <http.client.HTTPMessage at 0x7f87f0e91890>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download the file from url and save it locally under file_name\n",
    "# source: https://fasttext.cc/docs/en/language-identification.html\n",
    "\n",
    "path_ft = Path.cwd() # /home/jupyter/tutorials/fastai/course-nlp\n",
    "url = 'https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin'\n",
    "file_name = 'lid.176.bin'\n",
    "url_file = path_ft/file_name\n",
    "urllib.request.urlretrieve(url, url_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# get the model trained on 176 languages\n",
    "model = fasttext.load_model(file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "([['__label__pt']], array([[0.990301]]))\n"
     ]
    }
   ],
   "source": [
    "# test\n",
    "sentences = ['Eu me chamo Roberto.']\n",
    "predictions = model.predict(sentences)\n",
    "print(predictions[0][0][0].replace('__label__','') == 'pt')\n",
    "print(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of non-Portuguese reviews: 2157\n",
      "CPU times: user 4.75 s, sys: 16 ms, total: 4.77 s\n",
      "Wall time: 4.77 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "list_idx = []\n",
    "for idx, row in df2.iterrows():\n",
    "    try:\n",
    "        string = str(row[reviews]).replace('\\r',' ').replace('\\n','').replace('<br />', ' ').lower()\n",
    "        predictions = model.predict(string)\n",
    "        language = predictions[0][0].replace('__label__','')\n",
    "    except:\n",
    "        language = \"error\"\n",
    "    if not (language == 'pt'):\n",
    "        list_idx.append(idx)\n",
    "        \n",
    "print(f'number of non-Portuguese reviews: {len(list_idx)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19                      Péssimo\n",
       "55                          boa\n",
       "67     Recomendo o vendedor... \n",
       "90                Super rápido.\n",
       "108                OK RECOMENDO\n",
       "142                       OTIMA\n",
       "160                          OK\n",
       "204             nada a declarar\n",
       "228                   Recomendo\n",
       "231                decepcionado\n",
       "Name: review_comment_message, dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[reviews][list_idx][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[reviews][list_idx].to_csv (path_data/'olist_reviews_tobechecked_pt.csv', index = None, header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After reading the 2 157 reviews labeled non Portuguese by fastText, we discovered that the majority was written in Portuguese. Therefore, we decided to keep all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df2.drop(list_idx, axis=0, inplace=True)\n",
    "df_trn_val = df2.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save filtered reviews dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of neg reviews (rating = 1 or 2): 11270 (29.84%)\n",
      "number of pos reviews (rating = 4 or 5): 26500 (70.16%)\n",
      "number of all reviews: 37770\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAO8UlEQVR4nO3df6zddX3H8edrrRidOopcCGm7XcJuNgubVbpSZ7I4SUrBP4qbZG0WaRhJjSmLTJdY/acGJcFkaiRDtjobSvxR8VdotFo7QnQuAr1o11Ir6w0we20Dl5Vfhk0DvvfH+TSelXN7b+9t77n0Ph/JyTnnfb7fcz8nufDs+Z7vaVNVSJLmtt/q9wIkSf1nDCRJxkCSZAwkSRgDSRIwv98LmKpzzz23BgcH+70MSXpZefDBB5+sqoHj5y/bGAwODjI8PNzvZUjSy0qS/+o19zCRJMkYSJKMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkngZfwNZOpMNbvxWv5egWeqxW95xWp7XdwaSJGMgSTIGkiSMgSQJYyBJwhhIkjAGkiQmEYMki5Pcm+RAkv1J3tfmH0ny8yR72uWqrn0+lGQkycNJruiar2qzkSQbu+YXJrk/ycEkX05y1ql+oZKk8U3mncELwAeq6g3ACmBDkiXtsU9V1dJ22QHQHlsDXAysAj6TZF6SecBtwJXAEmBt1/N8vD3XEPAUcP0pen2SpEmYMAZVdaSqftRuPwccABaeYJfVwLaq+mVVPQqMAMvbZaSqHqmqXwHbgNVJArwd+Grbfytw9VRfkCTp5J3UZwZJBoE3Afe30Q1J9ibZkmRBmy0EDnXtNtpm481fDzxdVS8cN+/189cnGU4yPDY2djJLlySdwKRjkOQ1wNeAG6vqWeB24CJgKXAE+MSxTXvsXlOYv3RYtbmqllXVsoGBgckuXZI0gUn9RXVJXkEnBF+oqq8DVNXjXY9/FvhmuzsKLO7afRFwuN3uNX8SODvJ/PbuoHt7SdIMmMzZRAE+Bxyoqk92zS/o2uydwEPt9nZgTZJXJrkQGAIeAHYDQ+3MobPofMi8vaoKuBd4V9t/HXD39F6WJOlkTOadwVuBdwP7kuxpsw/TORtoKZ1DOo8B7wGoqv1J7gJ+QudMpA1V9SJAkhuAncA8YEtV7W/P90FgW5KPAT+mEx9J0gyZMAZV9QN6H9ffcYJ9bgZu7jHf0Wu/qnqEztlGkqQ+8BvIkiRjIEkyBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJKYRAySLE5yb5IDSfYneV+bn5NkV5KD7XpBmyfJrUlGkuxN8uau51rXtj+YZF3X/NIk+9o+tybJ6XixkqTeJvPO4AXgA1X1BmAFsCHJEmAjcE9VDQH3tPsAVwJD7bIeuB068QA2AZcBy4FNxwLStlnftd+q6b80SdJkTRiDqjpSVT9qt58DDgALgdXA1rbZVuDqdns1cGd13AecneQC4ApgV1UdraqngF3AqvbY66rqh1VVwJ1dzyVJmgEn9ZlBkkHgTcD9wPlVdQQ6wQDOa5stBA517TbaZieaj/aYS5JmyKRjkOQ1wNeAG6vq2RNt2mNWU5j3WsP6JMNJhsfGxiZasiRpkiYVgySvoBOCL1TV19v48XaIh3b9RJuPAou7dl8EHJ5gvqjH/CWqanNVLauqZQMDA5NZuiRpEiZzNlGAzwEHquqTXQ9tB46dEbQOuLtrfm07q2gF8Ew7jLQTWJlkQfvgeCWwsz32XJIV7Wdd2/VckqQZMH8S27wVeDewL8meNvswcAtwV5LrgZ8B17THdgBXASPA88B1AFV1NMlHgd1tu5uq6mi7/V7gDuBVwLfbRZI0QyaMQVX9gN7H9QEu77F9ARvGea4twJYe82HgkonWIkk6PfwGsiTJGEiSjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkJhGDJFuSPJHkoa7ZR5L8PMmedrmq67EPJRlJ8nCSK7rmq9psJMnGrvmFSe5PcjDJl5OcdSpfoCRpYpN5Z3AHsKrH/FNVtbRddgAkWQKsAS5u+3wmybwk84DbgCuBJcDati3Ax9tzDQFPAddP5wVJkk7ehDGoqu8DRyf5fKuBbVX1y6p6FBgBlrfLSFU9UlW/ArYBq5MEeDvw1bb/VuDqk3wNkqRpms5nBjck2dsOIy1os4XAoa5tRttsvPnrgaer6oXj5pKkGTTVGNwOXAQsBY4An2jz9Ni2pjDvKcn6JMNJhsfGxk5uxZKkcU0pBlX1eFW9WFW/Bj5L5zAQdP5kv7hr00XA4RPMnwTOTjL/uPl4P3dzVS2rqmUDAwNTWbokqYcpxSDJBV133wkcO9NoO7AmySuTXAgMAQ8Au4GhdubQWXQ+ZN5eVQXcC7yr7b8OuHsqa5IkTd38iTZI8iXgbcC5SUaBTcDbkiylc0jnMeA9AFW1P8ldwE+AF4ANVfVie54bgJ3APGBLVe1vP+KDwLYkHwN+DHzulL06SdKkTBiDqlrbYzzu/7Cr6mbg5h7zHcCOHvNH+M1hJklSH/gNZEmSMZAkGQNJEsZAkoQxkCRhDCRJGANJEsZAkoQxkCRhDCRJGANJEsZAkoQxkCRhDCRJGANJEsZAkoQxkCRhDCRJGANJEsZAkoQxkCRhDCRJGANJEsZAkgTM7/cC+mFw47f6vQTNUo/d8o5+L0HqC98ZSJKMgSTJGEiSMAaSJIyBJIlJxCDJliRPJHmoa3ZOkl1JDrbrBW2eJLcmGUmyN8mbu/ZZ17Y/mGRd1/zSJPvaPrcmyal+kZKkE5vMO4M7gFXHzTYC91TVEHBPuw9wJTDULuuB26ETD2ATcBmwHNh0LCBtm/Vd+x3/syRJp9mEMaiq7wNHjxuvBra221uBq7vmd1bHfcDZSS4ArgB2VdXRqnoK2AWsao+9rqp+WFUF3Nn1XJKkGTLVzwzOr6ojAO36vDZfCBzq2m60zU40H+0xlyTNoFP9AXKv4/01hXnvJ0/WJxlOMjw2NjbFJUqSjjfVGDzeDvHQrp9o81Fgcdd2i4DDE8wX9Zj3VFWbq2pZVS0bGBiY4tIlScebagy2A8fOCFoH3N01v7adVbQCeKYdRtoJrEyyoH1wvBLY2R57LsmKdhbRtV3PJUmaIRP+RXVJvgS8DTg3ySids4JuAe5Kcj3wM+CatvkO4CpgBHgeuA6gqo4m+Siwu213U1Ud+1D6vXTOWHoV8O12kSTNoAljUFVrx3no8h7bFrBhnOfZAmzpMR8GLploHZKk08dvIEuSjIEkyRhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkphmDJI8lmRfkj1JhtvsnCS7khxs1wvaPEluTTKSZG+SN3c9z7q2/cEk66b3kiRJJ+tUvDP486paWlXL2v2NwD1VNQTc0+4DXAkMtct64HboxAPYBFwGLAc2HQuIJGlmnI7DRKuBre32VuDqrvmd1XEfcHaSC4ArgF1VdbSqngJ2AatOw7okSeOYbgwK+G6SB5Osb7Pzq+oIQLs+r80XAoe69h1ts/HmL5FkfZLhJMNjY2PTXLok6Zj509z/rVV1OMl5wK4kPz3BtukxqxPMXzqs2gxsBli2bFnPbSRJJ29a7wyq6nC7fgL4Bp1j/o+3wz+06yfa5qPA4q7dFwGHTzCXJM2QKccgyW8nee2x28BK4CFgO3DsjKB1wN3t9nbg2nZW0QrgmXYYaSewMsmC9sHxyjaTJM2Q6RwmOh/4RpJjz/PFqvpOkt3AXUmuB34GXNO23wFcBYwAzwPXAVTV0SQfBXa37W6qqqPTWJck6SRNOQZV9Qjwxh7z/wYu7zEvYMM4z7UF2DLVtUiSpsdvIEuSjIEkyRhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJwhhIkjAGkiSMgSQJYyBJYhbFIMmqJA8nGUmysd/rkaS5ZFbEIMk84DbgSmAJsDbJkv6uSpLmjlkRA2A5MFJVj1TVr4BtwOo+r0mS5oz5/V5AsxA41HV/FLjs+I2SrAfWt7u/SPLwDKxtLjgXeLLfi5gN8vF+r0Dj8He0OQW/o7/XazhbYpAes3rJoGozsPn0L2duSTJcVcv6vQ5pPP6Onn6z5TDRKLC46/4i4HCf1iJJc85sicFuYCjJhUnOAtYA2/u8JkmaM2bFYaKqeiHJDcBOYB6wpar293lZc4mH3jTb+Tt6mqXqJYfmJUlzzGw5TCRJ6iNjIEkyBpIkYyBJwhic8ZIMJjmQ5LNJ9if5bpJXJbkoyXeSPJjk35L8Ydv+oiT3Jdmd5KYkv+j3a9CZr/2e/jTJ1iR7k3w1yauTXJ7kx0n2JdmS5JVt+1uS/KRt+w/9Xv+ZwBjMDUPAbVV1MfA08Jd0TtX726q6FPh74DNt208Dn66qP8Ev/mlm/QGwuar+GHgWeD9wB/BXVfVHdE6Ff2+Sc4B3Ahe3bT/Wp/WeUYzB3PBoVe1ptx8EBoE/Bb6SZA/wz8AF7fG3AF9pt784k4vUnHeoqv693f48cDmd393/bLOtwJ/RCcX/Av+S5C+A52d8pWegWfGlM512v+y6/SJwPvB0VS3t03qkXib1paf2JdXldGKxBrgBePvpXNhc4DuDuelZ4NEk1wCk443tsfvoHEaCzn9o0kz53SRvabfXAv8KDCb5/TZ7N/C9JK8BfqeqdgA3Av6h5hQwBnPXXwPXJ/kPYD+/+fcjbgTen+QBOoeOnunT+jT3HADWJdkLnAN8CriOzuHMfcCvgX8CXgt8s233PeDv+rTeM4p/HYX+nySvBv6nqirJGmBtVfkPDem0SjIIfLOqLunzUuYsPzPQ8S4F/jFJ6Jx59Dd9Xo+kGeA7A0mSnxlIkoyBJAljIEnCGEiSMAaSJOD/AChEDCoq9Hg/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# number of reviews neg or pos\n",
    "num_neg = len(df_trn_val[df_trn_val[label] == 'neg'])\n",
    "num_pos = len(df_trn_val[df_trn_val[label] == 'pos'])\n",
    "num_neg_pos = len(df_trn_val)\n",
    "pc_neg = round((num_neg/num_neg_pos)*100,2)\n",
    "pc_pos = round((num_pos/num_neg_pos)*100,2)\n",
    "print(f'number of neg reviews (rating = 1 or 2): {num_neg} ({pc_neg}%)')\n",
    "print(f'number of pos reviews (rating = 4 or 5): {num_pos} ({pc_pos}%)')\n",
    "print(f'number of all reviews: {num_neg_pos}') \n",
    "\n",
    "# plot histogram\n",
    "x= [1,2]\n",
    "keys = list(df_trn_val[label].value_counts().keys())\n",
    "values = list(df_trn_val[label].value_counts().array)\n",
    "plt.bar(x, values[::-1]) \n",
    "plt.xticks(x, keys[::-1])\n",
    "# print(df_trn_val['label'].value_counts())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_id</th>\n",
       "      <th>review_comment_message</th>\n",
       "      <th>review_score</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>e64fb393e7b32834bb789ff8bb30750e</td>\n",
       "      <td>Recebi bem antes do prazo estipulado.</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>f7c4243c7fe1938f181bec41a392bdeb</td>\n",
       "      <td>Parabéns lojas lannister adorei comprar pela I...</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>8670d52e15e00043ae7de4c01cc2fe06</td>\n",
       "      <td>aparelho eficiente. no site a marca do aparelh...</td>\n",
       "      <td>4</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>4b49719c8a200003f700d3d986ea1a19</td>\n",
       "      <td>Mas um pouco ,travando...pelo valor ta Boa.\\r\\n</td>\n",
       "      <td>4</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>3948b09f7c818e2d86c9a546758b2335</td>\n",
       "      <td>Vendedor confiável, produto ok e entrega antes...</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           review_id  \\\n",
       "3   e64fb393e7b32834bb789ff8bb30750e   \n",
       "4   f7c4243c7fe1938f181bec41a392bdeb   \n",
       "9   8670d52e15e00043ae7de4c01cc2fe06   \n",
       "12  4b49719c8a200003f700d3d986ea1a19   \n",
       "15  3948b09f7c818e2d86c9a546758b2335   \n",
       "\n",
       "                               review_comment_message  review_score label  \n",
       "3               Recebi bem antes do prazo estipulado.             5   pos  \n",
       "4   Parabéns lojas lannister adorei comprar pela I...             5   pos  \n",
       "9   aparelho eficiente. no site a marca do aparelh...             4   pos  \n",
       "12    Mas um pouco ,travando...pelo valor ta Boa.\\r\\n             4   pos  \n",
       "15  Vendedor confiável, produto ok e entrega antes...             5   pos  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_trn_val.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trn_val.to_csv (path_data/'olist_reviews_filtered_pt.csv', index = None, header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the csv of pre-processed data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_trn_val = pd.read_csv(path_data/'olist_reviews_filtered_pt.csv', encoding='utf-8')\n",
    "# df_trn_val.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning \"forward LM\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.74 s, sys: 1.46 s, total: 9.21 s\n",
      "Wall time: 7.75 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = (TextList.from_df(df_trn_val, path, cols=reviews, \n",
    "                            processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save(f'{path}/{lang}_databunch_lm_aws_sp15_multifit')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, f'{lang}_databunch_lm_aws_sp15_multifit', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 756 ms, sys: 164 ms, total: 920 ms\n",
      "Wall time: 509 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, config=config, pretrained_fnames=lm_fns3, drop_mult=1., \n",
    "                                  metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dfZBcV3nn8e/TPd3zPtK8SRpLFrIsWbZjG2MLg3Fw7ICJ4iIQA9nCFWodYOPN5oUQQrJJuYpkQxEgkM0mULvEC8ZkAadiDBXsJCAHYgx+AUtGMvKLbMnyiyyNZjQz0nSPpt+f/ePelsbjGWkszb3dPf37VHVN9+l773mmp+fp0+eee465OyIi0jwStQ5ARETipcQvItJklPhFRJqMEr+ISJNR4hcRaTIttQ5gIQYGBnzdunW1DkNEpKFs3779sLsPzi5viMS/bt06tm3bVuswREQaipk9P1e5unpERJqMEr+ISJNR4hcRaTKRJX4zu83MRsxs1xzPfdTM3MwGoqpfRETmFmWL/3Zgy+xCMzsbuA54IcK6RURkHpElfne/Hxif46m/Af4Y0OxwIiI1EGsfv5m9A3jJ3XcuYNubzWybmW0bHR2NIToRkeYQW+I3sw7gFuBjC9ne3W91983uvnlw8BXXH4iILGnDR3P89dbdPDuaXfRjx9niPxc4B9hpZs8Ba4BHzWxVjDGIiDSEFyeO8bnv7+HAkdyiHzu2K3fd/WfAiurjMPlvdvfDccUgItIoMrkiAF1ti5+moxzOeQfwELDJzPab2QejqktEZKnJ5EoAdLUufuKPrMXv7jee4vl1UdUtItLosvkg8fc0UotfREROX7ba4lfiFxFpDtl8iYRBeyq56MdW4hcRqUOZXImu1hbMbNGPrcQvIlKHMrkS3W2pSI6txC8iUoey+WIkI3pAiV9EpC5l8yW6IzixC0r8IiJ1KZsrRTKiB5T4RUTqUiZfUlePiEgzCU7uKvGLiDSNbE4tfhGRplEqV5gululq1XBOEZGmMJUvA6irR0SkWWTy0U3JDEr8IiJ1pzozZ7f6+EVEmkMmwpk5QYlfRKTuZCNchAWU+EVE6k6m2tWjFr+ISHOotvg1O6eISJPIVkf1qKtHRKQ5ZHMlzKAjvfirb4ESv4hI3ZmMcPUtUOIXEak72XwpsjH8EGHiN7PbzGzEzHbNKPu4mT1mZjvMbKuZnRVV/SIijSob4bKLEG2L/3Zgy6yyz7j7Je5+KXAP8LEI6xcRaUjZfHSLsECEid/d7wfGZ5VNznjYCXhU9YuINKooF2EBiO7I8zCzTwD/GTgKXHuS7W4GbgZYu3ZtPMGJiNSBbK7Imt72yI4f+8ldd7/F3c8Gvgb87km2u9XdN7v75sHBwfgCFBGpsUyuQU/uLsDXgXfXsH4RkbqUjbirJ9bEb2YbZzx8B/BUnPWLiNS7csU5VihHOqonso8UM7sDuAYYMLP9wJ8B15vZJqACPA/8VlT1i4g0oupc/FGO6onsyO5+4xzFX4qqPhGRpSDqRVhAV+6KiNSVTC7aZRdBiV9EpK5EvQgLKPGLiNSVTAx9/Er8IiJ1pNri71HiFxFpDsdH9bQ25iRtIiLyKh3v41eLX0SkOWRyxWD1rVQ0q2+BEr+ISF3J5Et0pVtIJKJZfQuU+EVE6ko2F+1c/KDELyJSV7L5Et1K/CIizSPqmTlBiV9EpK5kciW6IpyZE5T4RUTqSiZXjHSCNlDiFxGpK+rqERFpMhrVIyLSRMoVZ6pQ1qgeEZFmMVWIfkpmUOIXEakb1Xl61OIXEWkSmVz0M3OCEr+ISN3I5qNfdhGU+EVE6kYmhmUXQYlfRKRuVBdhiXL1LYgw8ZvZbWY2Yma7ZpR9xsyeMrPHzOxbZrY8qvpFRBpNHIuwQLQt/tuBLbPK7gUucvdLgKeBP42wfhGRhtLwXT3ufj8wPqtsq7uXwocPA2uiql9EpNFkwq6eznSDJv4F+ADwbzWsX0SkrmRzwTw9Ua6+BTVK/GZ2C1ACvnaSbW42s21mtm10dDS+4EREaiSbL0bezQM1SPxmdhPwduDX3d3n287db3X3ze6+eXBwML4ARURqJI7VtwCir2EGM9sC/HfgF9z9WJx1i4jUu0wMM3NCtMM57wAeAjaZ2X4z+yDweaAbuNfMdpjZF6KqX0Sk0WRy0c/FDxG2+N39xjmKvxRVfSIijS6bL3HW8rbI69GVuyIidSKTW6Ind0VE5JXcnYmpIn2drZHXpcQvIlIHsvkShXKFvs5op2QGJX4RkbowPlUAUItfRKRZVBN/f2c68rqU+EVE6kA18fcq8YuINIcxtfhFRJrLxPE+fiV+EZGmMD5VIN2SoCOdjLwuJX4RkTowNlWgvzONWbRTMoMSv4hIXZiYKsTSzQNK/CIidWFMiV9EpLmMK/GLiDQXdfWIiDSRfKlMJl+ir0OJX0SkKUxMFQHo61LiFxFpCmNTeQC1+EVEmsXxFr/6+EVEmkO1xd+vrh4RkeZwfGZOdfWIiDSHiakCZrBciV9EpDmMTRXo7UiTTEQ/Tw8o8YuI1Nz4VIHejujX2q2KLPGb2W1mNmJmu2aU/ZqZPW5mFTPbHFXdIiKNZHyqQH8Ma+1WRdnivx3YMqtsF/Au4P4I6xURaShxztMDC0z8ZnaumbWG968xsw+Z2fKT7ePu9wPjs8qedPfdpx2tiMgSND5ViGWt3aqFtvjvAspmtgH4EnAO8PXIogLM7GYz22Zm20ZHR6OsSkSkZioVZ+JYIZa1dqsWmvgr7l4CbgD+l7v/ATAUXVjg7re6+2Z33zw4OBhlVSIiNXN0ukjF47tqFxae+ItmdiNwE3BPWBbfKWgRkSVqLMZF1qsWmvjfD1wJfMLd95nZOcBXowtLRKQ5TByLP/G3LGQjd38C+BCAmfUC3e7+qZPtY2Z3ANcAA2a2H/gzgpO9nwMGgX8xsx3u/kunH76ISGMby9Zp4jez+4B3hNvvAEbN7Afu/pH59nH3G+d56luvNkgRkaVqvI67epa5+yTBGPwvu/vlwFujC0tEpDnUoqtnoYm/xcyGgP/EiZO7IiJyhsayBTrTSdpSydjqXGji/wvgu8Bed3/EzNYDz0QXlohIcxifysd68RYs/OTuncCdMx4/C7w7qqBERJrF+LFirBdvwcKnbFhjZt8KJ107ZGZ3mdmaqIMTEVnqxqfysfbvw8K7er4MfBs4C1gN3B2WiYjIGRjPxjtPDyw88Q+6+5fdvRTebicYiy8iImdgPOZ5emDhif+wmb3PzJLh7X3AWJSBiYgsdccKJXLFSt22+D9AMJRzGDgIvIdgGgcRETlN1at267LF7+4vuPs73H3Q3Ve4+68SXMwlIiKn6cTFW/GtvgVntgLXvNM1iIjIqZ2YmTPeyY7PJPHHsxy8iMgSNZ5tvBa/L1oUIiJNqBbz9MAprtw1swxzJ3gD2iOJSESkSYxNFWhJGD1tC5pEYdGctDZ3744rEBGRZjMymWewuxWzeHvOz6SrR0REzsDw5DQre9pir1eJX0SkRoaP5hhapsQvItI0ho/m1OIXEWkWmVyRqUJZLX4RkWYxfDQHwColfhGR5jA8GSZ+dfWIiDSHg2GLf2hZ/JdERZb4zey2cMWuXTPK+szsXjN7JvzZG1X9IiL17FCY+Ff0xDtdA0Tb4r8d2DKr7E+A77n7RuB74WMRkaZzcDJHX2eatlQy9rojS/zufj8wPqv4ncBXwvtfAX41qvpFROrZoRoN5YT4+/hXuvtBgPDnivk2NLObzWybmW0bHR2NLUARkTgcrNHFW1DHJ3fd/VZ33+zumwcHtbyviCwthyabp8V/yMyGAMKfIzHXLyJSc/lSmbGpQtO0+L8N3BTevwn455jrFxGpuZHJPFCbMfwQ7XDOO4CHgE1mtt/MPgh8CrjOzJ4Brgsfi4g0lYM1vGoXTjEf/5lw9xvneeotUdUpItIIqlftNktXj4hI0xs+Og3ASiV+EZHmMHw0T2c6SXdrvEsuVinxi4jEbHhympXL2mJfcrFKiV9EJGa1WnmrSolfRCRmtVp5q0qJX0QkRuWKcyiTV4tfRKRZjGXzlCtes4u3QIlfRCRWJy7ein8BliolfhGRGNX64i1Q4hcRiVV1kXWd3BURaRLDkzlSSaO/M12zGJT4RURiNHw0x4ruNhKJ2ly8BUr8IiKxqvXFW6DELyISq+HJXM0mZ6tS4hcRiYm7By3+Gp7YBSV+EZHYTE6XmC6Wa7YAS5USv4hITKpj+JX4RUSaxLOjWQDW9nXUNA4lfhGRmDw1nMEMNq7ormkcSvwiIjHZPZxhXX8n7elkTeNQ4hcRicnuQxk2raxtax+U+EVEYjFdKPPc2BSbVjVp4jez3zezXWb2uJl9uBYxiIjE6ZmRDO5wwVATJn4zuwj4TeAK4LXA281sY9xxiIjE6anhDACbVvXUOJLatPgvAB5292PuXgJ+ANxQgzhERGKzezhDWypR86GcUJvEvwu42sz6zawDuB44e/ZGZnazmW0zs22jo6OxBykisph2D2c4b2U3yRrOylkVe+J39yeBTwP3At8BdgKlOba71d03u/vmwcHBmKMUEVlcTw1P1sWIHqjRyV13/5K7X+buVwPjwDO1iENEJA6Hs3kOZwt1MaIHoKUWlZrZCncfMbO1wLuAK2sRh4hIHHaHJ3bPr4MTu1CjxA/cZWb9QBH4HXefqFEcIiKROzGip4lb/O7+5lrUKyJSC7uHJ+nvTDPY3VrrUABduSsiErndw5m6ae2DEr+ISKQqFefpQ1klfhGRZvHC+DGmi2XOV+IXEWkO9TRVQ5USv4hIhHaHi6+ct7Kr1qEcp8QvIhKh3YcmWdvXQUe6VqPnX0mJX0QkQk8erI/FV2ZS4hcRicjIZI59h6e47DW9tQ7lZZT4RUQi8uDeMQCuOnegxpG8nBK/iEhEHthzmGXtKS48q35G9IASv4hIJNydB/eOceX6/rqYg38mJX4RkQg8P3aMl45Mc9WG/lqH8gpK/CIiEXhg72EA3rShvvr3QYlfRCQSD+4ZY1VPG+sHOmsdyiso8YuILLJKxXlw72HetKEfs/rq3wclfhGRRffk8CQTx4p1N4yzSolfRGSRPbgnHL9fh/37ULulF2WJKJYrTBfLTBfKHCuUSZrRnk7SkU7SnkqSmDWMrVJxCuUKxXKFlkSCtlTilF+FyxWnXHHMIGGGAQ5U3Kl48Fyx7BTD45YrjnvwvGH0d6XpbNVbXeLzwN7DrB/sZNWytlqHMif9N9SRQqnCkekCPW0pWltOnhCL5QpHp4tUKk7Foewe3g8SoQPdrS30tKdoSyWpVJyRTJ4Xxo/x4vgx8qUKLQkjkTASdiK5lipOJlfipSPHeGlimpeOTDOVL1MoVyiVK5TKwTblilOsVHA/+e9UTdZJM5wgQc/WnkrSlkqQTBhmQTwVh1yxTK5YnnOfV6u7tYWVy9ro7UjRkkjQkjSSCaMlYbQkEiSTwf3qh0r1Nc7my2RzRbL5EhUn2D5pJBMJWltO3NItiePHTSUSOCdep1LFSVqwX7B/db8k6er+yQSpZPBc8EFWoVAOXt/2VJL28IN0WXuK3s40/Z1p+jr1gVaPCqUKP9k3zrsvW1PrUObVlO+a0Uye/3hqhO3PT7C6t50Lhnq4YKib1cvb50y2xXKFyekiECSxhBnplle2Vt2dqUKZiakCuWKZfKlCvlQhmy9x6GiOg0dzHMrkSJrR25FieUea1lSCpw5meGz/EZ48mKFQrgCQSho9bSm621roDn+2p5IczuY5cDTH4Wz+lEm3Kt2SAOf4sRdieUeK1cvbWdffSU97ilSYmJIJI5VMhAkzSFQd6ROJqVzx498ApotlKhUPPpTCWNPJIEmmkwlK4ba5cPtKuJ170LpvSyWDW0uSZALcT7T0E+EHhNmJmNJh4kwm7Pjz5YpzOFvg0GSO4aM5jkwXKFUq5EtBQg4+yCrHk3SVAcmE0dWWoru1hcHuVpIJo1Sufug5hVKZbL7EWLZy/IOxGB7POJHoEwmjUpnxgVmuHH9vFEoL/5vMZaCrlfWDnawf6OTsvg56O9LH31sDXWkGulpZ1p56xTcvic7O/Uc4VijX5fj9qiWd+D/73d3c89gBBrpaGexupa8zzRMHJ9nx4hHcoaethclc6fj26WSC7rYWutpa6GptIVcsMzZV4Mix4pzHTyWNZe0petpT5IsVDmfz5E/xj9zXmabiztHp4vHE3dXawkWre3j/VetY09tOJl8ikytxdLpIJlcikwt+HjlWpL8rzfmreli1rI2+zvTxJJdMhC3r8DFANh8cYzJXBIc1fR2s7evg7N52OtItx78llCsetH7DxN6RDn5/iV4l/OZULDvFUtgFFn7IppLBKbh8MehOO1YI/p7jUwXGpgoczuZ57vAUz45OsfWJQ4xPFeasoyVhDHS1sqa3nbX9wXvgrGXt9LSn6GkLvhUOdrcy2NWqD4hF8MOnRzGDN65X4q+Jc1d0csma5Yxm8jwzkmU0k2dtXwcffst5vPXCFVw41MNUoczu4QxPHpzkxYljZHMlsmHibUsl6O9spb8rzfKw1VTtWsmXgq6Wo9MFjk4XaUslGehqpb8zTW9nmvZU8DU+3ZKgq7WFVT1trOhppbUlCQQt0UyuyFShzFBPm/7hmlQiYbQmkrS2AK1zb9OWSrKM1CmPNV0oc2S6wMRUkSPHgg+H0Uyew9k8I5k8L44f4+G9Y3zrpy/N+W0xnUwwtLyN1cvbOWegk3MHuzh3RRcbVnRx1rK2uhyWWG/ypTJ3PPIiP79hgOUd6VqHMy/zhfYXLGalZn8A/BeCb+4/A97v7rn5tt+8ebNv27YtrvBElrRcscxoJk8mV2IyV+TodJGRyRz7j0zz0sQ0+yemeXY0+7Jvw53pJBtWdrNxRRc/d1YPr1vbywVD3ccbMhK4a/t+/vDOnfzDB67g6vMGax0OZrbd3TfPLo+9xW9mq4EPARe6+7SZ/RPwXuD2uGMRaUZtqSRn93WcdBt3Z2yqwN6RLM+MZNkzkuWZkQz37R7lG9v3A8E3hJ9b3cMV5/TxxvX9vH5dX1N3Ebo7X/zRPs5b2cWbN9bnMM6qWv2VWoB2MysCHcCBGsUhInMwC84LDHS18oZZfdUHj06z44Uj7HjxCNufn+C2H+3j73/wLMmEcfHqZVy9cYCrzxvk0rOX05JsnkuFHto7xpMHJ/n0uy+u+26x2BO/u79kZp8FXgCmga3uvjXuOETk9Awta2fo4nZ++eIhIDi38OgLEzy0d4wH9h7m8/+xh7/7/h66W1t4w/o+rjinj9ev6+Oi1cuOn7Beir74o330d6Z556Wrax3KKdWiq6cXeCdwDnAEuNPM3ufuX5213c3AzQBr166NO0wRWaD2dJKrNgxw1YYBPsomjhwr8ODeMe5/epQf7xvn358cCbZLJbninD6u2tDPVRsGuGBVz5IZ1LB3NMv3nxrhw2/dSFuq/s971KKr563APncfBTCzbwJvAl6W+N39VuBWCE7uxh2kiJye5R1prr94iOvDbwQjmRzbnpvgx8+O8cDeMf7yX58CYKArzS+ev4K3XrCSn984QEe6cc8P3PajfaRbErzvja+pdSgLUotX+gXgjWbWQdDV8xZAQ3ZElqgV3W0v+yAYPprjR3sO84OnR/m3XcP807b9tLYkuHbTCm64bDXXbloRXHTYIEYmc9z16H5uuHQ1A13zjMmtM7Xo4/+xmX0DeBQoAT8lbNmLyNK3alkb77l8De+5fA3FcoVH9o2z9YlD3PPYAb7z+DC9HSnefslZXH/xEFec01d3yxbOlCuW+a9f3Q7Azb+wvsbRLFxNxvG/WhrHL7L0lcoVfrjnMN989CW2Pj5MvlShvzPNdReuZMtFq7hqw0BdnRx2d37/H3fw7Z0H+ML7LmPLRUO1DukV6mYcv4jIXFqSQXfPtZtWMJUvHe8KunvnAf7xkRfp7Uix5aJV/MolZ3HFOX01Hyr6ue/v4ds7D/BHv7SpLpP+ySjxi0jd6WxtOX5eIFcs88NnDnP3zgP8844D3PGTF+lpa+GqDcH1Am/eOMCa3pNfkLbY7t55gP9579O863Wr+e1rzo217sWgxC8ida0tleS6C1dy3YUrmS6UuW/3CPftHuX+Z4JvBACvXbOMd166mrdfMsSKnujmwM8Vy3z2u7v50gP7uPw1vXyyAS7Wmov6+EWkIbk7e0ezfO/JEb698wCPH5gkYfCGc/q59vxBrtm0go0ruhYtMf/0hQn+8M6dPDs6xa+/YS1/ev0FdT9FxXx9/Er8IrIk7BnJ8M87DrD18UPsPpQBYPXydt6wvo/Xnb2cS8/u5fyh7ld1grhUrvDA3jG++eh+7t55gFU9bXz6PZfw5o21n4BtIZT4RaRpHDgyzX27R/nB08GCS4ezwVoFbakEl63t5cr1/Vx5bj8Xr1n2shlGyxXnubEpnjgwyfbnJ7jnsYMczuZZ1p7iXZet5iPXnUd326mnyK4XSvwi0pTcnf0T0+x48QiPvjDBw8+O8+TByePPt6WCNTM6W1sYmcwzXSwDweyj154/yA2vW8O15w825BTUGs4pIk3JzDi7r4Oz+zr4ldeeBcDEVIEf7xtj93CWqUKw+FI2V2Kgq5ULz+rhwqEeNqzoaqgriF8NJX4RaTq9nWm2XDTElotqHUltLM2PMxERmZcSv4hIk1HiFxFpMkr8IiJNRolfRKTJKPGLiDQZJX4RkSajxC8i0mQaYsoGMzsKPDPHU8uAoycpO9X96s8B4PBphDZX/Qt5fnb5yR4r7lPHdarnTyfuucrijHshZVHGfaqYFxrjfHHOd39mWRRxL/Q9spBYZ96v1/f2a9z9lTPKuXvd34BbF1o+s+xU92f83LaYcb3auE/2WHHXJu55ymKLeyFlUcZ9qphPN+6FvkeiirvZcsl8t0bp6rn7VZTf/Sruz3fchTrV/guN+2SPFff89S30+dOJe77f5XScTtwLKYsy7oXsezpxN8p7ZHZZo8S9oDgaoqsnama2zeeYwa7eKe54Ke54NWLcjRJzo7T4o3ZrrQM4TYo7Xoo7Xo0Yd0PErBa/iEiTUYtfRKTJKPGLiDSZJZf4zew2Mxsxs12nse/lZvYzM9tjZn9nZjbjud8zs91m9riZ/dXiRh1N3Gb252b2kpntCG/XN0LcM57/qJm5mQ0sXsTHjx3F6/1xM3ssfK23mtlZDRDzZ8zsqTDub5nZ8sWMOcK4fy38X6yY2aKeTD2TeOc53k1m9kx4u2lG+Unf/5E6nTGn9XwDrgYuA3adxr4/Aa4EDPg34JfD8muBfwdaw8crGiTuPwc+2mivd/jc2cB3geeBgUaIG+iZsc2HgC80QMxvA1rC+58GPt0gr/UFwCbgPmBzPcQbxrJuVlkf8Gz4sze833uy3y2O25Jr8bv7/cD4zDIzO9fMvmNm283sh2Z2/uz9zGyI4B/3IQ/+Kv8A/Gr49H8DPuXu+bCOkQaJO3IRxv03wB8DkYw+iCJud5+csWnnYsceUcxb3b0UbvowsGYxY44w7ifdffdix3om8c7jl4B73X3c3SeAe4Ettf6/XXKJfx63Ar/n7pcDHwX+9xzbrAb2z3i8PywDOA94s5n92Mx+YGavjzTaE840boDfDb/G32ZmvdGF+jJnFLeZvQN4yd13Rh3oLGf8epvZJ8zsReDXgY9FGGvVYrxHqj5A0PKMw2LGHYeFxDuX1cCLMx5Xf4ea/m5LfrF1M+sC3gTcOaMLrXWuTecoq7bYWgi+pr0ReD3wT2a2PvykjsQixf1/gI+Hjz8O/DXBP3dkzjRuM+sAbiHogojNIr3euPstwC1m9qfA7wJ/tsihnghkkWIOj3ULUAK+tpgxzmUx447DyeI1s/cDvx+WbQD+1cwKwD53v4H5f4ea/m5LPvETfKs54u6Xziw0sySwPXz4bYIkOfNr7hrgQHh/P/DNMNH/xMwqBJMxjdZz3O5+aMZ+/xe4J8J4q8407nOBc4Cd4T/ZGuBRM7vC3YfrOO7Zvg78CxEmfhYp5vCE49uBt0TZmJlhsV/rqM0ZL4C7fxn4MoCZ3Qf8hrs/N2OT/cA1Mx6vITgXsJ9a/m5xnUyI8wasY8aJGeBB4NfC+wa8dp79HiFo1VdPtlwflv8W8Bfh/fMIvrpZA8Q9NGObPwD+sRFe71nbPEcEJ3cjer03ztjm94BvNEDMW4AngMEoXuOo3yNEcHL3dONl/pO7+wh6DHrD+30L+d0i/XvEVVFsvxDcARwEigSfqh8kaEF+B9gZvsk/Ns++m4FdwF7g85y4sjkNfDV87lHgFxsk7v8H/Ax4jKAFNdQIcc/a5jmiGdUTxet9V1j+GMFkWasbIOY9BA2ZHeFtUUciRRj3DeGx8sAh4Lu1jpc5En9Y/oHwdd4DvP/VvP+jumnKBhGRJtMso3pERCSkxC8i0mSU+EVEmowSv4hIk1HiFxFpMkr80pDMLBtzfV80swsX6VhlC2bw3GVmd59qRkwzW25mv70YdYuAVuCSBmVmWXfvWsTjtfiJycoiNTN2M/sK8LS7f+Ik268D7nH3i+KIT5Y+tfhlyTCzQTO7y8weCW9XheVXmNmDZvbT8OemsPw3zOxOM7sb2Gpm15jZfWb2DQvmqP9adY70sHxzeD8bTsa208weNrOVYfm54eNHzOwvFvit5CFOTE7XZWbfM7NHLZin/Z3hNp8Czg2/JXwm3PaPwnoeM7P/sYgvozQBJX5ZSv4W+Bt3fz3wbuCLYflTwNXu/jqCGTP/csY+VwI3ufsvho9fB3wYuBBYD1w1Rz2dwMPu/lrgfuA3Z9T/t2H9p5x3JZyb5i0EV1UD5IAb3P0ygjUg/jr84PkTYK+7X+ruf2RmbwM2AlcAlwKXm9nVp6pPpKoZJmmT5vFW4MIZMyj2mFk3sAz4ipltJJgBMTVjn3vdfebc6z9x9/0AZraDYM6WH82qp8CJCe+2A9eF96/kxJzqXwc+O0+c7TOOvZ1gjnYI5mz5yzCJVwi+CaycY/+3hbefho+7CD4I7p+nPpGXUeKXpSQBXOnu0zMLzexzwH+4+w1hf/l9M56emnWM/Iz7Zeb+Hyn6iZNj821zMtPufqmZLeuW9ZcAAAEcSURBVCP4APkd4O8I5vAfBC5396KZPQe0zbG/AZ90979/lfWKAOrqkaVlK8Ec+ACYWXUa3WXAS+H934iw/ocJupgA3nuqjd39KMESjR81sxRBnCNh0r8WeE24aQbonrHrd4EPhPPEY2arzWzFIv0O0gSU+KVRdZjZ/hm3jxAk0c3hCc8nCKbTBvgr4JNm9gCQjDCmDwMfMbOfAEPA0VPt4O4/JZjx8b0Ei6BsNrNtBK3/p8JtxoAHwuGfn3H3rQRdSQ+Z2c+Ab/DyDwaRk9JwTpFFEq4eNu3ubmbvBW5093eeaj+RuKmPX2TxXA58PhyJc4SIl7kUOV1q8YuINBn18YuINBklfhGRJqPELyLSZJT4RUSajBK/iEiT+f80dvVOeXT4gAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-2\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>6.693191</td>\n",
       "      <td>5.698183</td>\n",
       "      <td>0.846413</td>\n",
       "      <td>0.153587</td>\n",
       "      <td>298.324890</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>5.101120</td>\n",
       "      <td>4.844213</td>\n",
       "      <td>0.795101</td>\n",
       "      <td>0.204899</td>\n",
       "      <td>127.003166</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, lr*10, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned1_sp15_multifit')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned1_enc_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.561766</td>\n",
       "      <td>4.421312</td>\n",
       "      <td>0.758318</td>\n",
       "      <td>0.241682</td>\n",
       "      <td>83.205368</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.144639</td>\n",
       "      <td>4.002900</td>\n",
       "      <td>0.724405</td>\n",
       "      <td>0.275595</td>\n",
       "      <td>54.756721</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.871001</td>\n",
       "      <td>3.759140</td>\n",
       "      <td>0.704319</td>\n",
       "      <td>0.295681</td>\n",
       "      <td>42.911499</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.739063</td>\n",
       "      <td>3.641269</td>\n",
       "      <td>0.692415</td>\n",
       "      <td>0.307585</td>\n",
       "      <td>38.140186</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.635000</td>\n",
       "      <td>3.572180</td>\n",
       "      <td>0.687088</td>\n",
       "      <td>0.312912</td>\n",
       "      <td>35.594131</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.557508</td>\n",
       "      <td>3.531695</td>\n",
       "      <td>0.681380</td>\n",
       "      <td>0.318620</td>\n",
       "      <td>34.181847</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.481839</td>\n",
       "      <td>3.488505</td>\n",
       "      <td>0.677717</td>\n",
       "      <td>0.322283</td>\n",
       "      <td>32.736965</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.427126</td>\n",
       "      <td>3.444504</td>\n",
       "      <td>0.672375</td>\n",
       "      <td>0.327625</td>\n",
       "      <td>31.327742</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.353791</td>\n",
       "      <td>3.407004</td>\n",
       "      <td>0.667628</td>\n",
       "      <td>0.332372</td>\n",
       "      <td>30.174704</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.247763</td>\n",
       "      <td>3.370681</td>\n",
       "      <td>0.663049</td>\n",
       "      <td>0.336951</td>\n",
       "      <td>29.098331</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.161210</td>\n",
       "      <td>3.338184</td>\n",
       "      <td>0.661294</td>\n",
       "      <td>0.338706</td>\n",
       "      <td>28.167883</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.050054</td>\n",
       "      <td>3.319427</td>\n",
       "      <td>0.656364</td>\n",
       "      <td>0.343635</td>\n",
       "      <td>27.644518</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.923504</td>\n",
       "      <td>3.309195</td>\n",
       "      <td>0.653984</td>\n",
       "      <td>0.346016</td>\n",
       "      <td>27.363083</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.853530</td>\n",
       "      <td>3.309102</td>\n",
       "      <td>0.652259</td>\n",
       "      <td>0.347741</td>\n",
       "      <td>27.360546</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.805080</td>\n",
       "      <td>3.311273</td>\n",
       "      <td>0.652579</td>\n",
       "      <td>0.347421</td>\n",
       "      <td>27.420031</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5hV1aH38e+azjSmMCAwwIAgIDJ0xGAIIgICUXMlEcUYS+S9lhvLG28QS9TY4k1yY4uRxJa82II1FixIsVAc+gAibYChTW8w9Zz1/rHPNJgGTNln5vd5nvOc3Wctn+OPddZZe21jrUVERNwroK0LICIiDVNQi4i4nIJaRMTlFNQiIi6noBYRcbmglrhodGycPevMfi1xaRGRdmnt2rVZ1tqEuva1SFDHdetJSkpKS1xaRKRdMsbsrW9fi3R9eDU0W0Sk2bRIUBeXeVrisiIiHVKTuj6MMWlAIeABKqy1oxs6vqCkHGstxpjTL6GISAd3Mn3UF1hrs5p68OfbMrjo7G6nUCQR6WjKy8tJT0+npKSkrYvS4sLCwkhMTCQ4OLjJ57TIj4kAN/4jhbkT+jF/+uCW+hMi0k6kp6cTFRVFUlJSu/4mbq0lOzub9PR0+vbt2+TzmtpHbYFPjTFrjTFz6zrAGDPXGJNijKka7rFgxe4mF0REOq6SkhLi4+PbdUgDGGOIj48/6W8OTQ3q8dbakcDFwC3GmAnHH2CtXWCtHW2tHR0ZWt1QzywsPakCiUjH1N5DutKp1LNJQW2tPeh7zwDeAcY2dHzfLhG8fN0YAMY88jmaSlVE5NQ1GtTGmAhjTFTlMjAFSG3svIkDu1YtP7d812kUUUSkZeXl5fGXv/zlpM+bPn06eXl5LVCi2prSou4GfGWM2QisAT601i5uysXX3jsZgCcWbydp3oenXEgRkZZUX1B7PA3fE/LRRx8RExPTUsWq0mhQW2t3W2uH+V5DrLWPNPXi8ZGhzBqVWLX+8AdbT7GYIiItZ968eezatYvhw4czZswYLrjgAq666iqGDh0KwGWXXcaoUaMYMmQICxYsqDovKSmJrKws0tLSGDx4MDfeeCNDhgxhypQpFBcXN1v5TEv0H48ePdrWnOsjo6CEsY8uAeCK0b34/azkZv+bIuK/tm3bxuDBzlDeB/+9ha0HC5r1+mf3iOa3Px5S7/60tDRmzpxJamoqy5YtY8aMGaSmplYNocvJySEuLo7i4mLGjBnD8uXLiY+PJykpiZSUFIqKiujfvz8pKSkMHz6cn/3sZ1xyySVcffXVjda3kjFmbX03E7bKNKddo8N4++YfAPBGyn6S5n1IVpFGg4iIO40dO7bWOOennnqKYcOGMW7cOPbv38+OHTtOOKdv374MHz4cgFGjRpGWltZs5WmxG16ON7J3LA9dOoT739sCwOiHP+eZq0YwM7lHaxVBRPxAQy3f1hIREVG1vGzZMj7//HNWrlxJeHg4EydOrHMcdGhoaNVyYGBgs3Z9tOqDA645L4kN919UtX7rq+tJSctpzSKIiJwgKiqKwsLCOvfl5+cTGxtLeHg43333HatWrWrl0rXBE15iwkP4963nV63P+utK7ns3Fa/mRhWRNhIfH8/48eM555xzuOuuu2rtmzZtGhUVFSQnJ3Pfffcxbty4Vi9fq/yYWJ9nvtjBHz79vmp9RnJ3nrxiOEGBekKYSEdS149r7Zkrf0ysz62TBvDIT86pWv9w0yH63/Mx5R5vG5ZKRMRd2rzpOufcPnx2xwRG9K4eND7gno9127mIiE+bBzXAgG5RvHPzeHY/Or1q251vbmzDEomIuIcrgrpSQIBh60NTAXhn/QGSH/gEj35kFJEOzlVBDRAeEsRbN50HQEFJBWfO/4gl2460calERNqO64IaYFSfuKoJnQBueCWFK55fyZo9GnMtIh2PK4ManAmd9jw2nYcvc0aFrN6Tw8+eX8nuzKI2LpmICERGRgJw8OBBZs2aVecxEydOpClDlRvj2qAG50kIV4/rUzVPCMCkPy5n3KNL2LC/5eeAFRFpTI8ePVi0aFGL/g1XB3Wlkb1jSXt8RtX64YISLnv26zYskYi0N7/5zW9qzUn9wAMP8OCDD3LhhRcycuRIhg4dynvvvXfCeWlpaZxzjvPNv7i4mNmzZ5OcnMwVV1zRbPN9tNqkTM0h7fEZPPD+Fl7+Jg2Aa19aw8vXNfhUMBHxNx/Pg8Obm/eaZwyFix9v8JDZs2dz++23c/PNNwPw5ptvsnjxYu644w6io6PJyspi3LhxXHLJJfU+9/C5554jPDycTZs2sWnTJkaOHNksxfeLFnVND1wyhDXzLwRg2fZMkuZ9yIebDrVxqUTE340YMYKMjAwOHjzIxo0biY2NpXv37syfP5/k5GQmT57MgQMHOHKk/lFoK1asqJqDOjk5meTk5pl7339a1LuXQ3x/6NyTrtFhLPzlucz5+2oAbnl1HSN6T6JHTKc2LqSInLZGWr4tadasWSxatIjDhw8ze/ZsFi5cSGZmJmvXriU4OJikpKQ6pzitqSWepu4fLeqKUnjnP2HBjyDtKwDG9+/Cryb1JzTIqcIPHv+CXRoRIiKnYfbs2bz++ussWrSIWbNmkZ+fT9euXQkODmbp0qXs3bu3wfMnTJjAwoULAUhNTWXTpk3NUi7/COqgULjmXQiLgVcugZV/AWu5c8pAtj98MVGhzheDC/+4nIuf/JLD+Q3/iyciUpchQ4ZQWFhIz5496d69O3PmzCElJYXRo0ezcOFCBg0a1OD5N910E0VFRSQnJ/PEE08wdmzz/IbWptOcnrSSAqdlvf1DGPpT+PGTEBJBhcdL/3s+rnXonsemt8hXEBFpfprm1MXTnJ60sGi44v/BpPtg8yJ4YQrk7CYoMIC0x2ew5p4Lqw79w6fb27CgIiLNx7+CGiAgACb8Gq5eBPnpsGAi7PgMgK5RYWx6YAoAzy7d1exPMhYRaQv+F9SV+k+Gucugc29Y+FNY/gR4vUSHBfP+reMBmP7Ulzz20bY2LaaINE1HmYP+VOrpv0ENENcXbvjU6a9e+gi8MQdK8klOjOHn4/oA8PyK3Vz85Jds3J9HaYWnjQssInUJCwsjOzu73Ye1tZbs7GzCwsJO6jz/+jGxPtbCmgXwyXyI6QOzF0LXwby2Zh93v137DqcnZw/n0uE9W69sItKo8vJy0tPTGx2j3B6EhYWRmJhIcHBwre0N/ZjYPoK6UtrX8K9roewoXPYsDPkJqQfymfn0V7UOmz99ELNG9SIuIqT1yygiUoeOE9QABQfhzWsg/VsYfxtMup9v0vKIDgtmwYrdvL/xYNWhG++fQufw4AYuJiLSOjpWUINzJ+PieZDyIvT9Ecx6CSLiAcgoKGHso0tqHf63a0Zz0dnd2qKkIiJAexpH3VRBoTDzf+HSZ2HfKufW84PrAegaHcaex6bTKTiw6vAb/5GiHxpFxLXaZ1BXGnE13PCJs/zCVFjv3INvjGHb76aR9vgMzu0bB8DAexe3+1+cRcQ/te+gBugxwhlv3XscvHczfHAnVJRV7X7+56Oqlvve/RGrdme3fhlFRBrQ/oMaIKILXP228+Niygvw8gwocOawjgkPIfXBqVWHzl6wilsWrmurkoqInKDJQW2MCTTGrDfGfNCSBWoxgUFw0UPw05fhyBZ4fgLs/QaAyNAg0h6fwX9N6g/Ah5sPkTTvQ9buzW3DAouIOE6mRX0b4P/3Yw/5Cdy4BEKj4JUfw+rnnRtmgP87ZSBbarSuL3/uG4rL9COjiLStJgW1MSYRmAH8vWWL00q6Doa5S2HAFPj4v2HRdZC1E4CI0CA++tUPSYx1nhYz+P7FZBS2/7ulRMS9mjSO2hizCHgMiAJ+ba2dWccxc4G5AL179x7V2JMQXMHrha/+CMv/BzxlMGgGjL8deo2huMzD4PsXn3DK//lRP+6+uOPMmysireO0xlEbY2YCGdbatQ0dZ61dYK0dba0dnZCQcIpFbWUBATDhLrgj1Zk6Ne0reGEyvHgxnfZ8yu5Hpp1wyvPLd/Pq6n1tUFgR6agabVEbYx4Dfg5UAGFANPC2tfbq+s5p8zsTT1VpEaz/J6x8FvL3Q5eBMP5XHOnzY/6wJI2UvbnsyTpadXjKvZPpEhmKtVZPkxGR09Jst5AbYyZST9dHTX4b1JU85bDlXfj6STiyGaK6w7ibYNS1PPhZOi99nXbCKUnx4aRlHyMuIoR1913U+mUWEb/W8W4hP12BwZD8U/jPL53x113Ogs/uh/89h9+Gvs6uecMZ0Tum1ilp2ccAyDlaRtK8D/nFi2s4Vlahux1F5LS1z0mZWsLB9fD1U7D1XTCBkHwFfzo6hejeQ7l+fF8e+mAr7244QN6x8lqn3TfzbG44v28bFVpE/EXHmz2vJeXsgVV/gXX/hIpiOGuac8dj7/PAGPKOlfHR5sPMf8d5YMFZ3SL59I4ftXGhRcTtFNQt4Wg2fPs354aZ4hxIHOME9sDpEBCIx2v558o0Hvj3Vn44oAv/vOHcti6xiLiY+qhbQkQ8TJwHd2yB6X+Aogx442p4diykvESgp5RLh/ckMjSIL3dkcdvrzjSrRaUVLN2ewdNLdvDNriwKSsob+UMi0tGpRd1cPBWw7X1npMihDRDRFcbOpeLMixj4zD48BDZ4+sq7JxEVFsyzS3cyJimWSYNOfJBBSbmHsOCGryMi/kldH63JWtizwgnsXc6TZDxB4awpTWKd7c867wDWeQeQS3SDl7n2B0n8ZtogVu/J5vsjhXy5I4svd2RV7X997jjG9Ytv0aqISOtRULeVvH2wbzWkr8G7fw3m8GaM9U3yFNcPEsdSkDCC2R952G57NdrqPt6ex6brRhuRdkJB7RZlx5xhfulrYP+3zvvRTGdXQCeCe4/G9BrLi3sTePr7mBNa3b84rw/GGNbvy2Vjej49Yzrx9bxJbVETEWlmDQV1UGsXpkMLCYek8c4LnG6S3DRI/5aQ/Wuc4P7qz1xvPVwfBuUx/SjpNpKoAeOxiWMwXQdDQCAl5R4G3beYA3nFFJaUExWmJ6mLtGdqUbtN2VGn1b1/DaR/67wf8/VNh0RBz5GQOIYtnkTuXlrIfptAnonig//6IUN6dObd9Qe4/Y0NAAw6I4oAY3j+56PoFRcOwNaDBXi8lsHdo9h6qICeMZ3IKCzlrG5RvL/xAJMGdaNzJwW/SGtT14c/sxZy91R3lexf4zyhxlY/0KDQdiLdJpAf1pPUYzHst13ZZ7uy3yaQbhMoIfSk/uSuR6cTGKC+b5HWpK4Pf2aM88NjXD8YdoWzrewo5Oxh5dp1pG7ZREzpAWLLDtG7ZB9zAtfSyZTVukSGjWG/TagK7/22q/PyJtB/wECW78ipdfw/VqZx3Xjd9i7iFmpRtwMer+X2NzaQc7SU30wdSHJsudP3nbvXec9LIyd9BwH5e+lcnoGx3uqTA4KhcyLe2CRKIxN5Zm0puUTxyFUTMOHxEB4H4fHQKQ6CQppUntyjZRzML2ZIj84A7M4s4ozOYYSHqF0gUh91fUg1T7kz13buXsjbWx3olcvHsus9tTQwgqDILnjCYjDh8QRHdoHweEqCO5NLFLFdziAosgs/fmErOTaKPCIppTrc37rpPEb1iWv5Oor4IXV9SLXA4OqulDqUlxYz4/H3CCjJJdYUEkshsaaIWAqJqygkpqyIOAqJMWnEsZkYU0S0KaZ7jWt8XKNL/KgNJZcocm0keS90xpM8kMDIBKelHtEFwrvUeI+HsBinu0dEqiiopZbg0E7Mnz2Ja1/6Fmp82frVhQN4aMmOus+hghiKiDWFxJlCYijimUt7E1CSy55dexjcuYLS/Xvx5Bzh4Kal9Aw5SkDFsboLEBDkdLVUvmqFefX63uIwAqO6ktijJwTqYyztm7o+pE5er+VQQQnBAYau0WEn7M8uKuXzbUfILCzl5W/S+PWUgSTGhjOmbyzlHktkaO3wLK3wMPBe52HBXSJD+PKO8+BYFmFluZjibGc2wmNZcDTL956NPZpJTuYhAktyiDFHTygDgNcavGGdCYrqWt0qj0g4LtwTqgM/PF7BLq6kPmpxhQqPl2tf+pavdmadsK9LZAhZRWVsfWgqb61N5773ttTaH0QFsRQRZwqIM4XEU0CcKSDeFNIloJA554Q7YX800wn6YznU+kpQU6fY6iCv2WKPSDixFd8pFoJObnijyKlQUItrFJVWcM5vP2ny8REhgVx/fl/uvOgsSiu8pKTlMqRHNLERIby+Zh/z3nYe0LDlwalE1GzFez1QnOsEd1UrPaveZXssG1NfsAeHO4Hd1Fd4nPMe3Ol0/lNJB6OgFlcprfDg8dqq4XrXvrSGVbuzKSmvHjb45OzhXDq8Z6PX+mZXFlf9bTWXj0zkjz8b1vCxO7Mo83j519p0/mNET254pfozGoCXGF+LPZ5C4k0+caaQh6YkElCSC8V5zgMiinOrX8dywNvAfOJBYXUEeUz1clhM3dtCoyFAU8V3NApq8Rv5xeUYA9FNnL/EWkvfuz+qWv/ud9Oq5uy21pJZWMqOjCLm/H31KZXn3VvGM7xXTN07rYXyY7XDuzLAj99WnFdjOQcqSur/oyYAwjofF+Y1Av34gK95TPCJvyeIf1BQS7u2cX8elz77ddX65SMTObtHNL/7YGu954QEBfCT4T05kFfMHRedxag+sbX25x4tY8TvPgNO75Z6j9eSc7SMmPBgDPCvtem8unof784dSWBpPpTk1Q7ykrwTw73mMSV5UPOGpeMFhjohHxbte+/stNArl49/1doXDSGRGh7ZRjSOWtq1Yb1i+P7hi7nof5ezN/sYb61L5611tY8Z1SeW2ycP4Pz+XZo0h3dsRAj9EiLYnXmUd9cf4PJRiSdVpnX7cnnhyz18uPlQnfvfTc1xrhndvc799fJ6oayw/jAvyfe9CqqX89OrlxtqyQOYQCewjw/3kEintR4c7nTpBHeq572RYwJD9A/BKVCLWtqVnRlF3PvuZlbtzmHuhH7Mnz74lK9V4fHS/56PAfj2nskEBxqCAwMoq/ASGxFCztEysotKuervq8ksLAVg8uCufL4to9FrTx96Bn+ZM+qUy3bKKkprh3hpfo1wPy7gS2suF0FFMZSXOO/eilMsgKkR3OHOiJrAYGcqg8Ag33uwM56+zu3BEBBY/76a66YF+/mt9X2zsc7ySb1T53Yz6R51fYiciueX7+Kxj787rWvsfORiAgMM17y4hlsv6M/n247wwld7+OT2CQzoFtVMJW1lnorq4C4/5rTUy4sbfz9+W0WJM62Bt8L3Xu5c21tez3pF9XvNc075Hw73MA8WKKhFTkW5x0vyA59SXO6p95ik+HD+PHsEl/n6yR/9yVBmj+lFQD392tlFpfzof5YRGxHMirsuOKXHqX29M4s5f1/NjOTuXDKsB0N6RLNyVzY/Hd3rpK/VLlh7YtjXN9yyOf6WCajuwjEGMKf3box+TBQ5HcVlHiyWAGNIzy0mo7CEtWm5/PKH/egUcmpPhX926U7+55PtACz79USSukTU2r/9cCG948LrvP6rq/cx/53NdV73tgsHcMdFZ51SmaRtKahFXMbjtZw5/6M69w3rFcPG/XkA3DtjMJMHd+Pj1MNMHdKNSX9cXuvYWy/ozzNLd1atj+oTy1s3/aDlCi4tRkEt4kLWWua/k8pra/ad9Lmv/vJcftC/S61r/emz73n6i50sv2sifeIjGjhb3KihoNbtTyJtxBjDY/8xlK0PTeW1G8dx24UDuG58EuEhgSy/ayKv3TiuzvO2PzytVkhXXuvykc4Qwn+u3Nus5SwqreBoqf//WOfP1KIWaUeuXLCKlbudhz9cObY3j/3H0Kp95R4ve7OPcUbnsFqzG24/XMjUP69o9Nppj89o/gJLFd3wItJBXHNen6qgfm3NPjIKSlixI5Nyz+k3yLKLSomP1EyCbUFBLdKOXDy0O+/c/APW7cvjdx9sZcl3jd98A9AtOpT50wdzfv8ueLyWV9fs48Yf9iMiNIjUA/nMfPor3ttwkOvP10OP24K6PkTaqV2ZRVz1t1UcKSjl0zsmkBjbifCQIIpKKyir8BIeEkhIYEC9470rWWuZ9deV7M4sYvl/X9DkCbPqu9apjBvvCE5r1IcxJgxYAYTitMAXWWt/29A5CmqR9mXt3lwuf+4bwLnTMiiw4XEIecfKuGvRJm69oD/DesVQXOZh6AOfUOG1DEvszMb0fAACDOx45NQnvWpPTrePuhSYZK0tMsYEA18ZYz621q5q1lKKiGuN6hNbNY/J9a+k8PCl59ArrtMJrWOv19Kvxvjwz7YeOeFalSEN4LXw740HuWxE43OPd2Qn1fVhjAkHvgJustbWO8GvWtQi7Y+1lsl/Ws6uzOrnVwYGGDzepmXIU1eO4MJBXbn8uW+YNKgrwYEBPLlkB/ERIaTcO7nDd4mc9qgPY0wgsBboDzxbV0gbY+YCcwF69+596qUVEVcyxvDerecz86kvSct2niJfV0if378L/7h+LAEBhpJyD6XlXjqHV/drL759QtWy11qe/mIni9amd9x5SprgZFvUMcA7wH9Za1PrO04tapH2728rdnNOz87syznKrFG9CAwwVHi8jfZf11Tu8TLioc8oKq3grqkDuXnimVUta2stZR4voUGnNp+Kv2nWW8iNMb8Fjlpr/1DfMQpqEWmqmhNUDTojirIKL7uzqrtXhiV25p+/PJecojKueXEN+3KOERnqjF6pS3Cg4bk5o5g0qGujI1rc5HRHfSQA5dbaPGNMJ+BT4PfW2g/qO0dBLSIno2ZYN5d+XSL44tcTm/WaLel0gzoZeAUIxJkb5E1r7UMNnaOgFpGTVVrhYcHy3XQKCeSyET3pEhlKWYWXZ5bu5KklOwD42ehE5pzbh49TD7NyVxb/+k9npsBVu7M5XFDCpcN78OzSXVXHh4cEkvrAVL9oWWv2PBHpULKKShn98OdV68MSO/PMVSPpFRde67h1+3J589v93HHRWcSGh2CxZBWVMf7xLwCYOqQbA8+IJiEqlP4JkWw+kMfmAwU8feWIZi+zglpEOpyCknKSH/i01rb50wfx6Een92g1cJ6N+bdrRp/0kMLSCg8hgQF1nqegFpEOa8GKXaQeKOD9jQfr3H9mQkStseEA04acwZ+uGMZzy3bx1tp0DuY3/PT27343jbBgZ3TK0dIK7nxzA2UVXpZuz6zz+F9POYvPt2WwYX8eK++exDvrD3DLBQMU1CLSsaWk5TDrryu55YIzuWvqIJZuz6BnTCfO6hbVYEv3eMVlHgbfv7jOfW/ddB6/em0DB/KKT7p8e38/U0EtItKcyiq8BAUYrliwkm/TcmvtS4ztxH0zz2bF95lcNz6JA3kl9OsSQXxkCIfyS3hu2S5G9o7lwX9vobTCCyioRURalLWW55bv4onF2/nVhQO48xQeMKwHB4iItCBjDDdP7M/NE/u3yPX1zEQREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RoNamNML2PMUmPMNmPMFmPMba1RMBERcQQ14ZgK4P9aa9cZY6KAtcaYz6y1W1u4bCIiQhNa1NbaQ9badb7lQmAb0LOlCyYiIo6T6qM2xiQBI4DVLVEYERE5UZOD2hgTCbwF3G6tLahj/1xjTIoxJiUzM7M5yygi0qE1KaiNMcE4Ib3QWvt2XcdYaxdYa0dba0cnJCQ0ZxlFRDq0poz6MMALwDZr7Z9avkgiIlJTU1rU44GfA5OMMRt8r+ktXC4REfFpdHietfYrwLRCWUREpA66M1FExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlGg1qY8yLxpgMY0xqaxRIRERqa0qL+mVgWguXQ0RE6tFoUFtrVwA5rVAWERGpg/qoRURcrtmC2hgz1xiTYoxJyczMbK7Lioh0eM0W1NbaBdba0dba0QkJCc11WRGRDk9dHyIiLteU4XmvASuBgcaYdGPMDS1fLBERqRTU2AHW2itboyAiIlI3dX2IiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlmhTUxphpxpjtxpidxph5LV0oERGp1mhQG2MCgWeBi4GzgSuNMWe3dMFERMTRlBb1WGCntXa3tbYMeB24tGWLJSIilYKacExPYH+N9XTg3OMPMsbMBeb6VkuNMamnX7w21QXIautCnCbVwT3aQz1Uh5bVp74dTQlqU8c2e8IGaxcACwCMMSnW2tFNLp4LqQ7u0B7qAO2jHqpD22lK10c60KvGeiJwsGWKIyIix2tKUH8LDDDG9DXGhACzgfdbtlgiIlKp0a4Pa22FMeZW4BMgEHjRWrulkdMWNEfh2pjq4A7toQ7QPuqhOrQRY+0J3c0iIuIiujNRRMTlFNQiIi7XrEHt5lvNjTEvGmMyao7vNsbEGWM+M8bs8L3H+rYbY8xTvnpsMsaMrHHOL3zH7zDG/KKV69DLGLPUGLPNGLPFGHObn9YjzBizxhiz0VePB33b+xpjVvvK9Ibvx2uMMaG+9Z2+/Uk1rnW3b/t2Y8zU1qyH7+8HGmPWG2M+8NWKFQ4AAAPISURBVMc6GGPSjDGbjTEbjDEpvm3+9nmKMcYsMsZ85/t/4zx/q0OjrLXN8sL5oXEX0A8IATYCZzfX9ZuhfBOAkUBqjW1PAPN8y/OA3/uWpwMf44whHwes9m2PA3b73mN9y7GtWIfuwEjfchTwPc5t/f5WDwNE+paDgdW+8r0JzPZt/ytwk2/5ZuCvvuXZwBu+5bN9n7NQoK/v8xfYyp+rO4FXgQ98635VByAN6HLcNn/7PL0C/NK3HALE+FsdGq1jM/7HOg/4pMb63cDdbV3B48qYRO2g3g509y13B7b7lp8Hrjz+OOBK4Pka22sd1wb1eQ+4yJ/rAYQD63Duds0Cgo7/POGMODrPtxzkO84c/xmreVwrlT0RWAJMAj7wlcnf6pDGiUHtN58nIBrYg29ghD/WoSmv5uz6qOtW857NeP2W0M1aewjA997Vt72+urimjr6vziNwWqN+Vw9fl8EGIAP4DKclmWetraijTFXl9e3PB+Jp+3r8GfhvwOtbj8f/6mCBT40xa40zDQT41+epH5AJvOTrgvq7MSYC/6pDo5ozqJt0q7mfqK8urqijMSYSeAu43Vpb0NChdWxzRT2stR5r7XCcVulYYHADZXJdPYwxM4EMa+3ampsbKI/r6uAz3lo7Emd2zFuMMRMaONaNdQjC6dJ8zlo7AjiK09VRHzfWoVHNGdT+eKv5EWNMdwDfe4Zve311afM6GmOCcUJ6obX2bd9mv6tHJWttHrAMp78wxhhTeRNWzTJVlde3vzOQQ9vWYzxwiTEmDWdGyUk4LWx/qgPW2oO+9wzgHZx/NP3p85QOpFtrV/vWF+EEtz/VoVHNGdT+eKv5+0Dlr7u/wOnzrdx+je8X4nFAvu/r0yfAFGNMrO9X5Cm+ba3CGGOAF4Bt1to/1djlb/VIMMbE+JY7AZOBbcBSYFY99ais3yzgC+t0JL4PzPaNqOgLDADWtEYdrLV3W2sTrbVJOJ/1L6y1c/ypDsaYCGNMVOUyzucgFT/6PFlrDwP7jTEDfZsuBLb6Ux2apJk79qfjjETYBdzT1h3wx5XtNeAQUI7zr+cNOH2ES4Advvc437EG52EJu4DNwOga17ke2Ol7XdfKdTgf5+vYJmCD7zXdD+uRDKz31SMVuN+3vR9OSO0E/gWE+raH+dZ3+vb3q3Gte3z12w5c3EafrYlUj/rwmzr4yrrR99pS+f+sH36ehgMpvs/TuzijNvyqDo29dAu5iIjL6c5EERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFzu/wMLuWLfYlwRLwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(15, lr, wd=wd, moms=(0.8,0.7), callbacks=[ShowGraph(learn_lm)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned2_sp15_multifit')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned2_enc_sp15_multifit')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save best LM learner and its encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned_sp15_multifit')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned_enc_sp15_multifit')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Fine-tuning \"backward LM\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.75 s, sys: 1.44 s, total: 9.19 s\n",
      "Wall time: 7.71 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = (TextList.from_df(df_trn_val, path, cols=reviews, \n",
    "                            processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1, backwards=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm.save(f'{path}/{lang}_databunch_lm_aws_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 264 ms, sys: 24 ms, total: 288 ms\n",
      "Wall time: 283 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_aws_sp15_multifit_bwd', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 844 ms, sys: 140 ms, total: 984 ms\n",
      "Wall time: 507 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, config=config, pretrained_fnames=lm_fns3_bwd, drop_mult=1., \n",
    "                                  metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZRcZ3nn8e9TVV29d6tb3doty5YleQM7tuxgDMbELIZDDIQwg08WsyQ+WQgJOWSSHOZAAgdCQpZJwpnJeAKYTFgmYEhsgrFsNgdsx5YX2bIl2fIiS2r1om61unqp/Zk/7m25LbpbLalu3SrV73NUp6tu3ar3qVL3r9773lvvNXdHREQaRyLuAkREpLoU/CIiDUbBLyLSYBT8IiINRsEvItJgUnEXsBR9fX2+YcOGuMsQEakrDz/88GF37z9+eV0E/4YNG9i+fXvcZYiI1BUz2zffcg31iIg0GAW/iEiDUfCLiDQYBb+ISINR8IuINBgFv4hIg1Hwi4g0GAW/iEgNGjya5a+27eG5kcmKP7eCX0SkBr04Ns3ff38vB8dnKv7cCn4RkRo0mSsA0NnSVPHnVvCLiNSgTLYIQEdz5WfWUfCLiNSg2eDvalHwi4g0hGM9fgW/iEhjmMwVSCaM1qZkxZ9bwS8iUoMy2SIdzSnMrOLPreAXEalBk9kinREM84CCX0SkJk2EPf4oKPhFRGrQZK5AVwTH8IOCX0SkJmWyxUiO6AEFv4hITZrMaYxfRKShZDTGLyLSWIKjejTGLyLSELKFEvlSWUM9IiKNYjIXTNeg4BcRaRBRzswJCn4RkZozmZ3t8WuMX0SkIWSywUlY1OMXEWkQGY3xi4g0lpeGeuos+M3sC2Y2bGY75yz7pJk9bmaPmdk2M1sTVfsiIvVqdqinHsf4bwWuP27ZZ939le5+KfBt4GMRti8iUpdmD+dsb678SVggwuB393uBseOWTcy52Q54VO2LiNSrTLZIOpWgORVN8EczgLQIM/sU8KvAUeD1i6x3M3AzwPr166tTnIhIDcjkipGcZH1W1XfuuvtH3f0s4MvABxdZ7xZ33+ruW/v7+6tXoIhIzKKcoA3iParnK8C7YmxfRKQmTWYLke3YhSoHv5ltmnPzBmB3NdsXEakHUff4I3tmM/sqcC3QZ2YHgI8DbzWzLUAZ2Af8RlTti4jUq8lckfW9bZE9f2TB7+43zrP481G1JyJypojytIugb+6KiNScTDa6E62Dgl9EpKa4O5O5M/eoHhEROc50vkTZo5unBxT8IiI15dhJWBT8IiKNYTIX7QRtoOAXEakpE7NTMmuMX0SkMUQ9Fz8o+EVEaorG+EVEGozG+EVEGsyxHr/G+EVEGoOCX0SkwWSyRdrTSZIJi6wNBb+ISA2ZzEU7Fz8o+EVEakrUM3OCgl9EpKZM5oqRHsMPCn4RkZoS9dm3QMEvIlJTop6LHxT8IiI1Jeq5+EHBLyJSUzJZjfGLiDSMUtmZzpd0VI+ISKN4aWZOjfGLiDSEzOwEbRrjFxFpDJkqzMUPCn4RkZoxmYt+Ln5Q8IuI1IxMNvq5+CHC4DezL5jZsJntnLPss2a228weN7NvmdmyqNoXEak31ZiSGaLt8d8KXH/csruBi939lcDTwB9H2L6ISF2ZDf6ueh3qcfd7gbHjlm1z92J48wFgXVTti4jUm0YY438/cOdCd5rZzWa23cy2j4yMVLEsEZF4ZLIFkgmjtSkZaTuxBL+ZfRQoAl9eaB13v8Xdt7r71v7+/uoVJyISk8lwZk6z6M6+BRDt9sQ8zOwm4G3Ade7u1W5fRKRWVWOeHqhy8JvZ9cAfAq9z9+lqti0iUusyVZiZE6I9nPOrwP3AFjM7YGYfAD4HdAJ3m9ljZvYPUbUvIlJvqjEXP0TY43f3G+dZ/Pmo2hMRqXeTuSIrOlsib0ff3BURqRHVGuNX8IuI1Ijx6QLdrdEP9Sj4RURqQKFU5uhMgd72dORtKfhFRGrAkek8AMsV/CIijWFsKgj+3vbmyNtS8IuI1ICxydngV49fRKQhjIY9/uUdCn4RkYbw0lCPgl9EpCGMTuUxg542Bb+ISEMYm8qxrLWJZCLamTlBwS8iUhPGpvJVGeYBBb+ISE0YncyzvAqHcoKCX0SkJqjHLyLSYMam8vRW4VBOUPCLiMSuXHaOTOerMl0DKPhFRGI3PlOg7NU5hh8U/CIisRubygEKfhGRhjE6OTszp47qERFpCNWcrgEU/CIisavmBG2g4BcRid1sj78a8/SAgl9EJHZjU3k6W1KkU9WJZAW/iEjMRqeqdww/KPhFRGI3NpWr2o5dUPCLiMRudDJflXPtzoos+M3sC2Y2bGY75yx7t5k9aWZlM9saVdsiIvVk7Awa6rkVuP64ZTuBXwDujbBdEZG64R7M01OtCdpgicFvZhvNrDm8fq2ZfcjMli32GHe/Fxg7btkud99zytWKiJxhJrJFCiWvyR7/bUDJzM4DPg+cA3wlsqoAM7vZzLab2faRkZEomxIRiU21v7ULSw/+srsXgXcC/8PdPwysjq4scPdb3H2ru2/t7++PsikRkdhUe4I2WHrwF8zsRuAm4NvhsqZoShIRaRzVnqANlh787wOuAj7l7s+b2TnAP0dXlohIYzg21FPFnbuppazk7k8BHwIwsx6g090/s9hjzOyrwLVAn5kdAD5OsLP374F+4N/N7DF3f/Oply8iUt+OTdBWxaGeJQW/mf0QuCFc/zFgxMx+5O6/v9Bj3P3GBe761skWKSJyphqbytOWTtLSlKxam0sd6ul29wmCY/C/6O6XA2+IriwRkcYwNpWv6o5dWHrwp8xsNfBfeGnnroiInKZqT9AGSw/+TwB3Ac+6+0Nmdi7wTHRliYg0hmpP0AZL37n7deDrc24/B7wrqqJERBrF2GSeLSu7qtrmUqdsWGdm3wonXRsys9vMbF3UxYmInMncPRjqqeKhnLD0oZ4vArcDa4C1wB3hMhEROUXT+RK5Yrlmd+72u/sX3b0YXm4lOBZfREROURzz9MDSg/+wmf2ymSXDyy8Do1EWJiJypovjy1uw9OB/P8GhnIPAIeAXCaZxEBGRUxTHBG2wxOB39xfd/QZ373f3Fe7+DoIvc4mIyCmKY4I2OL0zcC04XYOIiJxYHBO0wekFv1WsChGRBjQ2lSedStCert48PXB6we8Vq0JEpAEdOppldXcLZtXtRy/6zV0zyzB/wBvQGklFIiINYmB8hjXd1Y/SRYPf3TurVYiISKMZGJ/hqo19VW/3dIZ6RETkFBVKZQYnsqxd1lL1thX8IiIxGJrIUnZYs6z6Qz0KfhGRGAyMZwEFv4hIwxgYnwEU/CIiDePgseDXGL+ISEMYGJ+hp62JtvSSzodVUQp+EZEYDIzPxDLMAwp+EZFYDIxnFfwiIo1kYHyGtQp+EZHGMJEtkMkVz7zgN7MvhCdn3zlnWa+Z3W1mz4Q/e6JqX0SkVsV5KCdE2+O/Fbj+uGV/BHzP3TcB3wtvi4g0lIEYD+WECIPf3e8Fxo5b/HbgS+H1LwHviKp9EZFadTD81u4ZN9SzgJXufggg/LlioRXN7GYz225m20dGRqpWoIhI1AbGZ2hKGn0d1T3l4qya3bnr7re4+1Z339rf3x93OSIiFTMwPsPq7lYSiXhOZFjt4B8ys9UA4c/hKrcvIhK74Mtb8YzvQ/WD/3bgpvD6TcC/Vbl9EZHYxfnlLYj2cM6vAvcDW8zsgJl9APgM8EYzewZ4Y3hbRKRhFI+dgCW+4I9sdiB3v3GBu66Lqk0RkVo3lMlRKvuZ2eMXEZGfFveXt0DBLyJSVbPBH8e5dmcp+EVEqmj2BCyru9XjFxFpCAPjMyxra6K9ufonYJml4BcRqaKB8SxrYuztg4JfRKSq4jzz1iwFv4hIFR0cn4l1xy4o+EVEqmYiWyCTLbK2Rz1+EZGGcCicjllDPSIiDeK5kUkAzu5tj7UOBb+ISJXsHsxgBptWdsRah4JfRKRKnh7KsGF5Oy1NyVjrUPCLiFTJnqEMm2Pu7YOCX0SkKrKFEi8cnmLLqq64S1Hwi4hUw97hScoOW1Z2xl2Kgl9EpBqeHsoAsGWVhnpERBrCnqEM6WSCs5fHeygnKPhFRKpiz2CGjSs6aErGH7vxVyAi0gCeHsywpQaO6AEFv4hI5CayBQaOZtm8Kv4du6DgFxGJ3DOzO3Zr4IgeUPCLiERu9+DsET0KfhGRhvD0YIb2dJK1Mc/KOUvBLyISsT1DGTav6sTM4i4FUPCLiETK3dkzmKmZ8X2IKfjN7HfNbKeZPWlmvxdHDSIi1TAymePIdIHNjRz8ZnYx8OvAlcAlwNvMbFO16xARqYanB4OTr5xfIzt2IZ4e/wXAA+4+7e5F4EfAO2OoQ0QkcnvCQzlr5Rh+iCf4dwLXmNlyM2sD3gqcdfxKZnazmW03s+0jIyNVL1JEpBKeHsywvD1NX0dz3KUcU/Xgd/ddwJ8DdwPfBXYAxXnWu8Xdt7r71v7+/ipXKSJSGbuHMjU1vg8x7dx198+7+2Xufg0wBjwTRx0iIlHKFkrsOjTBRWviP/nKXKk4GjWzFe4+bGbrgV8AroqjDhGRKD2y7wj5YplXn7c87lJeJpbgB24zs+VAAfhtdz8SUx0iIpG579lRkgnjig29cZfyMrEEv7u/No52RUSq6b5nD/PKdd10tjTFXcrL6Ju7IiIRmMwV2XHgKK/eWFvDPKDgFxGJxIPPj1IqO1dv7Iu7lJ+i4BcRicB9e0dJpxJcdnZP3KX8FAW/iEgE7nt2lMvX99DSlIy7lJ+i4BcRqbAjU3meOjTB1TV2GOcsBb+ISIXd/9woAFfV4Pg+KPhFRCruvmcP055O8sp13XGXMi8Fv4hIhd337ChXntNLU7I2I7Y2q4qBu+PucZchInVu8GiW50amuPq82hzmgfimbKiq6XyRe3YNc+cThxibypNOJWhOJUiYMTqVZziTZXgiRzqV4PVbVvDGC1dy7ZZ+OppTTMwUGcpkmZgpcNGablrTtbeHXkRqx/3PHQbgqhr84tasMzr479t7mK89tJ97dg0xnS+xsquZs3vbyWSLjBbLlMpOb3uay9b3sKKzmfHpAt/fPcztOwZoShoJM3LF8rHna04leM15fbzhwpVcsLqLoYksg0ezDByd4dB4lkNHZxgYzzI+nefS9ct43eZ+rt2ygk0rOsgWyhydKXB0pkCp7DQljWTCSKcSdLc20dGcOqkTMbs7pbJTLL/0sy2drNlNS5FGcc+uYZa3p7lgVW3NyDnXGR38P9gzzL3PjPD2S9fy9kvXcOWGXhKJxcO1VHYeefEI3989TKnsrOhsZmVXCy1NSX6y9zD37Brie7uHX/aYdDLBqu4W1ixr4WfP6aW9OcWDz4/x6e/s5tPf2U0qYRTLiw8jpRLGsrY0rekE2UKZbKFEtlBi9mGzVZfdWeyplrU10dfRTF9HcOKH/s5m+jqaWdbWRFMyQVPSSCUSTOeLjE0VODKdJ5Mt0N/RzJplraxZ1kpnS4rxmQJHp4P7D0/mGJrIMZzJcWQqT1s6SWdLE12tKVqakuSLZfLFMrliiZlCmelckal8UL8ZNCUSNKWMZCJBwiBhRsIgnUrQkkrSkk7S2pSkozlFZ0uKjuYUrelkuJ6RTEB3axP9nS2s6Gymtz3NZK7I2FSeI1N5pvIlUkmjKZEglTQ6mlMsa2sK3s+mJMOZLAeOzLB/bJrpfInV3S3B/1d3K8vamk7qA1dkMUdnCtz91BA3XnHWCbMmTlYP49pbt2717du3n/TjJrIFWlJJ0qnK9YLdnT1DGV4cnWZ1dyurultY3p6e9z/50NEZfrRnhH1j03S1NNHdGlySCSiUgp56vlhmfCbPkekC49N5soUyLU0JWpqStDQlSZrheNg2JBOGhcGZMCOVNFKJICCnciUOT+bmXPKMZHJM5n7qPDfHNKcSdLakGJvKL/iBkkwY/R3NrOhqpqctTbZQYiJbZGKmQK5YIp1MkE4Fl9Z0ivZ0kvbm4EOh7E6xVKZYcgplD/elBB+whVKZmUIpuORLTOaKTOaKVPNXMp1MsLwjHVzam1nRGbzOFZ0tdLc2USo7JXfK5eADtzy7L8iMNd0tnL28nbN6W2lOLX0I0N3JFcvhpUTCLPj/TiVIaYutrn3twRf5o28+wb/99tVcctayuMvBzB52963HLz+je/xdEcyIZ2acv6qL85ewGbe6u5X3XLm+4jWcrJl8iaMzBQqlMsVyEMSt6SS97Wna0sGvQKFUZmgiy8EjM0zli3S3poNec2vQc05Wqffi7swUSkznS8fCtlgOhsmGMzlGJnKMTefpaE7R256mpy1Ne3MyfF3Ba5vMFY9tsUzmiqzoamZdTxvrelppT6c4dHQmHKLLMpIJPiRHJ3OMTObYPTjBSCa36FbV8cxgeXszbeGWS0s6Sbns5IqlY1tvhVKZQsnJl4Kto4U0JY3OluB972ptor05iTvHPgxb00m6WlJBJ6Itzcb+dras6uTcvo6KdnDk1HzzkYOc299es4dxzjqjg18CrenkCXdKNyUTYTi2Vamq+ZkZbenUsQ+kWet64KIKtbGqu2XR+0tlZ3Qqx2S2SDIxO9xk4dZWsKVVKjsHx2fYNzrFvtFphiayzORLxz60kgkLhrGaEjSHW53pVIKmZIJ00mgOt+jSqQTuHg7tBVtAEzMFxmcKTMwUmM6XMIIPF4DhTIG9w0UmssH9sx9QqYSxsqvlWH1m0NqUZFlbsJW5rDXNsvYmetvS9LSn6W1L09fZzPL2NP2dzTU5rUC92T82zYMvjPGRN22u+eFDBb/IcZIJY0VnCytOcJrUlV0tXLY+vgm48sUyzx2eZM9ghj2DGQYnsuDgBENSU7ngQ+T5w1OMT48zPl0gX5p/a6O3Pc363jbOXt7G2b1tbOhrZ0NfO+csb6enPV3dF1an/vXRgwC842fWxlzJiSn4RepUOpVY8rAjBMNo0/kSY1N5xqbyL9sXdODINPtGp9n+whHu2DHwsqGuzuYUK7qCgxxWdDZzTl8HF63p4uK13azsaq753m01uDvffPQgP3tOb+xbzUuh4BdpEGZGe3OK9uYUZ/UuHE65Yon9YzO8cHiKF0an2D82zXAmOKrroReO8G87Bo7tc+jrSHPpWT1csaGHrRt6uHht90nt6D5TPLZ/nOcPT/Gbr9sYdylLouAXkZdpTiU5b0UH563omPf+yVyRXYcmePLgUR4/eJRH9h3hnl1DALQ0JbhiQy+vOa+P12zq44JVXTV9WGOlfPORgzSnErzlFaviLmVJFPwiclI6mlNcsaH3ZScQH8nkeHjfER54bpSf7D3Mn925G+6EVV0tXH/xKt5y8Sq2buit2tFh1ZQvlrnj8QHedNGqmju37kIU/CJy2vo7m7n+4lVcf3HQ4x2ayPIfzxxm25ODfOXBF7n1vhfo62jm9Vv6ef35K3jNpr5IDreOw5f/cx/j0wXedVnt79SdpeAXkYpb2dXCL16+jl+8fB2TuSI/2D3MXU8OcteTg3z94QOkEsYVG3r5+UvW8JaLV9XtkUMD4zP85V17uGZzP6/b3B93OUt2Rn9zV0RqS7FU5tH94/xg9zDffXKQ50amSCWMazb3c8Mla3jjhStpb66f/uiv/9N2/uOZEe7+8OsW3WEel4b85q6I1JZUMnFs/8AfvHkLTw5McMeOAW7fMcD3dw/T0pTgDRes5IZL1nDN5v6a/mLZd3cOcvdTQ/zxW86vydBfjHr8IhK7ctnZvu8It+84yL8/fogj0wVam5K8dlMwG+51569geUdz3GUek8kWeONf30tPe5rbP3h1zc6KW1M9fjP7MPBrBF8yfAJ4n7tn46hFROKXSBhXntPLlef08vGfv4j7nx3lnl1D3P3UENueGiJhcOU5vbzl4tVcf/EqVnYtPu1GlNydz9y5m6FMln/4lctrNvQXU/Uev5mtBX4MXOjuM2b2L8B33P3WhR6jHr9IY3J3nhyYYNuTg3xn5yB7hycBuGz9smMfAtUcZimXnU9/Zxf/+OPn+bXXnMN/f9uFVWv7VCzU448r+B8ALgEmgH8F/s7dty30GAW/iADsHc5w5xOD3LlzkKcOTQDwirXdxw4l3dg//5fOKqFYKvOHtz3BbY8c4L2v3sDH3nZhzX85rWaCPyzmd4FPATPANnf/pcXWV/CLyPH2jU5x587gQ2DH/nEANq3o4LoLVnL52T1cetYy+jsrs18gWyjxwa88yj27hvjwGzbzoevOq4s5imom+M2sB7gN+K/AOPB14Bvu/s/HrXczcDPA+vXrL9+3b19V6xSR+jEwPsO2J4MPgYf3HTl2xrt1Pa1cuaGX12zq4+rz+k5638BEtsBX//NFvviTFxjKZPnEDRfxK1dtiOAVRKOWgv/dwPXu/oHw9q8Cr3L331roMerxi8hSZQsldh48yqMvjvPo/iPc/+woR6YLQLBFcPV5fbzq3OW86txelrW9/Itj5bKz/8g0uwczPPj8GP/vof1M5oq8euNyPvhz5/HqjX1xvKRTVktH9bwIvMrM2giGeq4DlOoiUhEtTUm2buhlaziXULnsPHVogh/vPcxP9h7maw8FU0iYwVk9baQSBsE/Dh3NMp0vAcF5Gd76itXc/NpzeUWNn1HrZMU1xv+nBEM9ReBR4NfcPbfQ+urxi0il5Iolduw/yv3PjvLsyGRwHmUAD+YcOn9VJ1tWdbJ5ZWddfYt4PjUz1HMqFPwiIidvoeCvv28eiIjIaVHwi4g0GAW/iEiDUfCLiDQYBb+ISINR8IuINBgFv4hIg1Hwi4g0mLr4ApeZHQWemeeubuDoIstOdH32Zx9w+BRKm6/9pdx//PLFbqvuE9d1ovtPpe75llWz7qUsi7LuE9W81BoXqnOh63OXRVH3Un9HllLr3Ou1+rt9trv/9Fng3b3mL8AtS10+d9mJrs/5ub2SdZ1s3YvdVt3x1L3AsqrVvZRlUdZ9oppPte6l/o5EVXejZclCl3oZ6rnjJJbfcRLXF3repTrR45da92K3VffC7S31/lOpe6HXcipOpe6lLIuy7qU89lTqrpffkeOX1UvdS6qjLoZ6omZm232e+SxqnequLtVdXfVYd73UXC89/qjdEncBp0h1V5fqrq56rLsualaPX0SkwajHLyLSYBT8IiIN5owLfjP7gpkNm9nOU3js5Wb2hJntNbO/MzObc9/vmNkeM3vSzP6islVHU7eZ/YmZHTSzx8LLW+uh7jn3f8TM3MwqfqLTiN7vT5rZ4+F7vc3M1tRBzZ81s91h3d8ys2WVrDnCut8d/i2WzayiO1NPp94Fnu8mM3smvNw0Z/miv/+ROpVjTmv5AlwDXAbsPIXHPghcRXD6zTuBt4TLXw/cAzSHt1fUSd1/Anyk3t7v8L6zgLuAfUBfPdQNdM1Z50PAP9RBzW8CUuH1Pwf+vE7e6wuALcAPga21UG9Yy4bjlvUCz4U/e8LrPYu9tmpczrgev7vfC4zNXWZmG83su2b2sJn9h5mdf/zjzGw1wR/u/R78r/wT8I7w7t8EPuPheYHdfbhO6o5chHX/DfDfgEiOPoiibnefmLNqe6Vrj6jmbe5eDFd9AFhXyZojrHuXu++pdK2nU+8C3gzc7e5j7n4EuBu4Pu6/2zMu+BdwC/A77n458BHgf86zzlrgwJzbB8JlAJuB15rZf5rZj8zsikirfcnp1g3wwXAz/gtm1hNdqS9zWnWb2Q3AQXffEXWhxznt99vMPmVm+4FfAj4WYa2zKvE7Muv9BD3Paqhk3dWwlHrnsxbYP+f27GuI9bXV9ynkl8DMOoBXA1+fM4TWPN+q8yyb7bGlCDbTXgVcAfyLmZ0bflJHokJ1/y/gk+HtTwJ/RfDHHZnTrdvM2oCPEgxBVE2F3m/c/aPAR83sj4EPAh+vcKkvFVKhmsPn+ihQBL5cyRrnU8m6q2Gxes3sfcDvhsvOA75jZnngeXd/Jwu/hlhf2xkf/ARbNePufunchWaWBB4Ob95OEJJzN3PXAQPh9QPAN8Ogf9DMygSTMY3Uct3uPjTncf8H+HaE9c463bo3AucAO8I/snXAI2Z2pbsP1nDdx/sK8O9EGPxUqOZwh+PbgOui7MzMUen3Omrz1gvg7l8EvghgZj8E3uvuL8xZ5QBw7Zzb6wj2BRwgztdWrZ0J1bwAG5izYwa4D3h3eN2ASxZ43EMEvfrZnS1vDZf/BvCJ8Ppmgk03q4O6V89Z58PA1+rh/T5unReIYOduRO/3pjnr/A7wjTqo+XrgKaA/ivc46t8RIti5e6r1svDO3ecJRgx6wuu9S3ltkf5/VKuhqr0g+CpwCCgQfKp+gKAH+V1gR/hL/rEFHrsV2Ak8C3yOl77ZnAb+ObzvEeDn6qTu/ws8ATxO0INaXQ91H7fOC0RzVE8U7/dt4fLHCSbLWlsHNe8l6Mg8Fl4qeiRShHW/M3yuHDAE3BV3vcwT/OHy94fv817gfSfz+x/VRVM2iIg0mEY5qkdEREIKfhGRBqPgFxFpMAp+EZEGo+AXEWkwCn6pS2Y2WeX2/tHMLqzQc5UsmMFzp5ndcaIZMc1smZn9ViXaFgGdgUvqlJlNuntHBZ8v5S9NVhapubWb2ZeAp939U4usvwH4trtfXI365MynHr+cMcys38xuM7OHwsvV4fIrzew+M3s0/LklXP5eM/u6md0BbDOza83sh2b2DQvmqP/y7Bzp4fKt4fXJcDK2HWb2gJmtDJdvDG8/ZGafWOJWyf28NDldh5l9z8wesWCe9reH63wG2BhuJXw2XPcPwnYeN7M/reDbKA1AwS9nkr8F/sbdrwDeBfxjuHw3cI27/wzBjJmfnvOYq4Cb3P3nwts/A/wecCFwLnD1PO20Aw+4+yXAvcCvz2n/b8P2TzjvSjg3zXUE36oGyALvdPfLCM4B8VfhB88fAc+6+6Xu/gdm9iZgE3AlcClwuZldc6L2RGY1wiRt0jjeAFw4ZwbFLjPrBLqBL5nZJoIZEJvmPOZud5879/qD7n4AwMweI5iz5cfHtZPnpQnvHgbeGF6/ipfmVP8K8JcL1Nk657kfJpijHYI5Wz4dhniZYEtg5TyPf1N4eTS83UHwQVLXWZYAAAFYSURBVHDvAu2JvIyCX84kCeAqd5+Zu9DM/h74gbu/Mxwv/+Gcu6eOe47cnOsl5v8bKfhLO8cWWmcxM+5+qZl1E3yA/DbwdwRz+PcDl7t7wcxeAFrmebwBf+bu//sk2xUBNNQjZ5ZtBHPgA2Bms9PodgMHw+vvjbD9BwiGmADec6KV3f0owSkaP2JmTQR1Doeh/3rg7HDVDNA556F3Ae8P54nHzNaa2YoKvQZpAAp+qVdtZnZgzuX3CUJ0a7jD8ymC6bQB/gL4MzP7CZCMsKbfA37fzB4EVgNHT/QAd3+UYMbH9xCcBGWrmW0n6P3vDtcZBX4SHv75WXffRjCUdL+ZPQF8g5d/MIgsSodzilRIePawGXd3M3sPcKO7v/1EjxOpNo3xi1TO5cDnwiNxxon4NJcip0o9fhGRBqMxfhGRBqPgFxFpMAp+EZEGo+AXEWkwCn4RkQbz/wGbaQmygBJZMAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 2e-2\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>7.415000</td>\n",
       "      <td>6.164595</td>\n",
       "      <td>0.891743</td>\n",
       "      <td>0.108257</td>\n",
       "      <td>475.608276</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>5.391895</td>\n",
       "      <td>5.212091</td>\n",
       "      <td>0.816590</td>\n",
       "      <td>0.183410</td>\n",
       "      <td>183.477478</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, lr*10, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned1_sp15_multifit_bwd')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned1_enc_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.839345</td>\n",
       "      <td>4.741192</td>\n",
       "      <td>0.779655</td>\n",
       "      <td>0.220345</td>\n",
       "      <td>114.570618</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.316499</td>\n",
       "      <td>4.165969</td>\n",
       "      <td>0.709234</td>\n",
       "      <td>0.290766</td>\n",
       "      <td>64.455078</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.967860</td>\n",
       "      <td>3.843466</td>\n",
       "      <td>0.675412</td>\n",
       "      <td>0.324588</td>\n",
       "      <td>46.687012</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.772866</td>\n",
       "      <td>3.689861</td>\n",
       "      <td>0.662576</td>\n",
       "      <td>0.337424</td>\n",
       "      <td>40.039295</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.661542</td>\n",
       "      <td>3.609903</td>\n",
       "      <td>0.656151</td>\n",
       "      <td>0.343849</td>\n",
       "      <td>36.962448</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.581761</td>\n",
       "      <td>3.555538</td>\n",
       "      <td>0.648398</td>\n",
       "      <td>0.351603</td>\n",
       "      <td>35.006664</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.539313</td>\n",
       "      <td>3.508788</td>\n",
       "      <td>0.644643</td>\n",
       "      <td>0.355357</td>\n",
       "      <td>33.407742</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.433682</td>\n",
       "      <td>3.468960</td>\n",
       "      <td>0.637591</td>\n",
       "      <td>0.362408</td>\n",
       "      <td>32.103340</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.364891</td>\n",
       "      <td>3.424580</td>\n",
       "      <td>0.633410</td>\n",
       "      <td>0.366590</td>\n",
       "      <td>30.709761</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.294525</td>\n",
       "      <td>3.384222</td>\n",
       "      <td>0.628297</td>\n",
       "      <td>0.371703</td>\n",
       "      <td>29.495037</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.190253</td>\n",
       "      <td>3.355298</td>\n",
       "      <td>0.624359</td>\n",
       "      <td>0.375641</td>\n",
       "      <td>28.654142</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.076259</td>\n",
       "      <td>3.336091</td>\n",
       "      <td>0.620559</td>\n",
       "      <td>0.379441</td>\n",
       "      <td>28.109028</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.981711</td>\n",
       "      <td>3.325022</td>\n",
       "      <td>0.619353</td>\n",
       "      <td>0.380647</td>\n",
       "      <td>27.799616</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.910093</td>\n",
       "      <td>3.322420</td>\n",
       "      <td>0.617079</td>\n",
       "      <td>0.382921</td>\n",
       "      <td>27.727354</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.866521</td>\n",
       "      <td>3.326313</td>\n",
       "      <td>0.617552</td>\n",
       "      <td>0.382448</td>\n",
       "      <td>27.835518</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8dc3yWTfQ8KqBAEXlrBFZFFEpJbFrUot1r3XS2ttr9b2d4Vugr2t6LW9rUur2LqWuhRcUFFBRREFJChL2GQLEMKSsIQkZJ/v7485hASyATPJmeT9fDzmMWefz1eHN4fvfM85xlqLiIi4V0hrFyAiIo1TUIuIuJyCWkTE5RTUIiIup6AWEXG5sEAcNCEp2fbueU4gDi0i0iatXLmywFqbWt+6gAR1h07dyMrKCsShRUTaJGPMjobWBaTrQyOzRUT8JzBBraQWEfGbAJ1RK6lFRPwlIH3UOqMWkVNRWVlJbm4uZWVlrV1KwEVGRtKtWzc8Hk+z9wlMUAfioCLSZuXm5hIXF0d6ejrGmNYuJ2CstRw4cIDc3Fx69OjR7P0C1EetqBaR5isrKyMlJaVNhzSAMYaUlJRT/peDRn2IiCu09ZA+5nTaqVEfIiIup64PEWn3Dh8+zF//+tdT3m/ChAkcPnw4ABXVFZCg9iqnRSSINBTU1dXVje43f/58EhMTA1VWjYCM+qhWUotIEJk6dSpbt25l4MCBeDweYmNj6dy5M6tWrWL9+vVce+217Nq1i7KyMu655x6mTJkCQHp6OllZWRQXFzN+/HguvvhivvjiC7p27cpbb71FVFSUX+pTUIuIq8x4ex3r84749Zh9usTzwFV9G1w/c+ZMsrOzWbVqFZ988gkTJ04kOzu7Zgjds88+S3JyMqWlpVx44YVcf/31pKSk1DnG5s2befnll3nmmWe44YYbmDt3LjfffLNf6g9MUKuPWkSC2NChQ+uMc37sscd44403ANi1axebN28+Kah79OjBwIEDARgyZAg5OTl+qycgQe3VGbWInKbGznxbSkxMTM30J598wocffsjSpUuJjo5m9OjR9Y6DjoiIqJkODQ2ltLTUb/UE5MfEKq/VyA8RCRpxcXEUFRXVu66wsJCkpCSio6PZuHEjy5Yta+HqAnVGbS25h0o5Kzk6EIcXEfGrlJQURo4cSb9+/YiKiqJjx44168aNG8dTTz1FRkYG5513HsOGDWvx+kwgznwjOve2nW/7MzkzJ/r92CLS9mzYsIELLrigtctoMfW11xiz0lqbWd/2AX1mYmW1N5CHFxFpFwIS1N2SfGMHBz24MBCHFxFpVwIS1PGRvvusFpdXMW91XiA+QkSk3QhIUIeGGL6YOgaA/3r5a7blFwfiY0RE2oWA9VF3SYzihsxuAIz546cKaxGR0xTQHxMfmTSAH13aE4Cfvvx1ID9KRKTNCmhQA0wdfz4X9UhmXd4R0qe+S/rUd3n0g02B/lgRkYCKjY0FIC8vj0mTJtW7zejRo8nKyjrjzwp4UAM8f8fQOvNPLNrC1LlrWuKjRUQCqkuXLsyZMyegn9EiQR0VHkrOzIm8f+8lTOzfGYBXVuxi7J8+bYmPFxFp0v3331/nntTTp09nxowZXH755QwePJj+/fvz1ltvnbRfTk4O/fr1A6C0tJTJkyeTkZHB9773Pb/d76NZl5AbY3KAIqAaqGro6pmmnN8pnidvGsyQJdt58J31bNlfzINvr+c3V17Qbp6XJiJNeG8q7F3r32N26g/jZza6yeTJk7n33nv58Y9/DMBrr73G+++/z89+9jPi4+MpKChg2LBhXH311Q3m1d/+9jeio6NZs2YNa9asYfDgwX4p/1TOqC+z1g483ZCu7QcX92Dlr8cC8Ozn2xkwY4Fu4iQirWrQoEHs37+fvLw8Vq9eTVJSEp07d+aXv/wlGRkZjB07lt27d7Nv374Gj7F48eKae1BnZGSQkZHhl9oCclOm5kiJjeDtn1zMVU8s4UhZFT2mzWfhz0bRu2Nca5UkIm7QxJlvIE2aNIk5c+awd+9eJk+ezOzZs8nPz2flypV4PB7S09PrvcVpbYHoHWjuGbUFFhhjVhpjptS3gTFmijEmyxiTlZ+f36yD9u+WwIYHx9XMf+v/FvPPZTuaWZKIiH9NnjyZV155hTlz5jBp0iQKCwtJS0vD4/GwaNEiduxoPJ9GjRrF7NmzAcjOzmbNGv8MmmhuUI+01g4GxgN3G2NGnbiBtXaWtTbTWpuZmpra7AKO/dD4yhTfrQN//WY233t6KQdLKpp9DBERf+jbty9FRUV07dqVzp07c9NNN5GVlUVmZiazZ8/m/PPPb3T/u+66i+LiYjIyMnjkkUcYOnRoo9s31ynf5tQYMx0ottY+2tA2mZmZ9nTGDn6wbi8/fGllzfxfbxrMBGeUiIi0XbrN6Rne5tQYE2OMiTs2DVwBZPuh1pN8u28n/nTDgJr5H8/+ij/M30BhaWUgPk5EJCg0p+ujI7DEGLMa+BJ411r7fqAKum5wN3JmTuTGoWcBMGvxNgbMWMCijfsD9ZEiIq7WZFBba7dZawc4r77W2t+3RGEPXZdRM4QP4I7nV/C7d9a3xEeLSCtoL0N0T6edLXJl4ulKiY1g0/+M4+lbhgDwjyXbGfPoJ/qhUaSNiYyM5MCBA20+rK21HDhwgMjIyFPaLyDPTDzdHxMbs3zbAb436/jTf+f9ZCQZ3RL9+hki0joqKyvJzc1tcoxyWxAZGUm3bt3weDx1ljf2Y2LQBDUFW8g+Es7Di/bw2eYCAK4e0IW/TB6oy89FJOi12sNt/aaqHF66ln4f3spLN/Zm9Hm+cdrzVufx+le7W7k4EZHACo6gDouACY/CvnXwwlU8/90erJl+BQA///dqMv9nIaUV1a1cpIhIYARHUAOcNw6+/yoc2ArPTyS+ooC/TB4IQEFxBRf89n283rb9Q4SItE/BE9QAPcfAzXPhyG54bjzXpFez+P9dVrN64YaG72olIhKsgiuoAdJHwi1vwtGD8NwEzjZ72fz78XRNjOKHL63khS9y2vwQHxFpX4IvqAHOuhBumwcVJfDseDwHN/Pod32Xnj8wbx09ps1v5QJFRPwnOIMaoMtAuP1dsF54bgLDY/aw/sFv16z+eKO6QUSkbQjeoAbo2AfueM83KuSFK4nOX1MzGuQHz2fx5w+/aeUCRUTOXHAHNUCHXnDHfIiIgxevIX7/Vzxyve/xN3/+cDPpU99lW35xKxcpInL6gj+oAZLS4Y73ISYVXvoON3TYzvSr+tSsHvPHTykq061SRSQ4tY2gBkjo6usGSTwbZn+X29O2kjNzIhMzfA8e6D99gcZZi0hQajtBDRDX0fcDY4fe8PJk2PguT35/MD1TYwA455fzGTnz41YuUkTk1LStoAaISYHb3obOA+C1WyF7LvPvuaRm9e7Dpby1SvcHEZHg0faCGiAqCW59E866CObeSUT2q+TMnMhjNw4C4J5XVvH5loJWLlJEpHnaZlCDbxTITXOgxyh48y5Y8Y+a26IC3PT35aRPfZdHP9jUyoWKiDSu7QY1QHg03Pgq9P42vHsfLP0r1wzsysRaTzZ/YtEW7p+zphWLFBFpXNsOagBPJHzvn3DB1fDBNFj8KI9+dwAf3jeK6wd3A+DVrF18skkPzxURd2r7QQ0QFg6TnoP+N8DHvyNqyUP0So3ljzcM4IUfDAXg9udWsKewtJULFRE5WfsIaoDQMPjOUzD4Vlj8v7Dg12Atl56bWnNDp+EPfcza3MJWLlREpK72E9QAIaFw5V9g6A9h6RMw/xfg9XL94K6kxkUAcNUTS6io8rZyoSIix7WvoAYICYHxD8PIe2DF32HeTzHWy4pfjWVkrxQAzv31eyxcr7vviYg7hLV2Aa3CGBg7A8Ki4NOZUF4IEx7lkUkDaq5c/M8XswgPC6GiysuXv7yctPjIVi5aRNqr9ndGfYwxcNk0uOJ/YNN78Nggun71R3KmX8K08ecD1HSBDP3DRxwqqWjNakWkHWu/QX3MiJ/C3V/CueN8PzI+NpAfRizgnbsu5JwOMTWbDfrdQlbvOtyKhYpIe6WgBkjpCd99Dv5zEXTsC+9Ppd8bY/n4iny2/2EcidEeAK558nOmztXFMSLSskxzHwRrjAkFsoDd1torG9s2MzPTZmVl+aG8VmAtbP0YPnwA9q6FTv2xY2fw6sHeTH19bZ1N7/vWudw2Ip2EKE8rFSsibYUxZqW1NrO+dadyRn0PsME/JbmYMdDrcpiyGK57BsoKMf+8jskbf8rCyfF1Nv3Twm+4+e/LKausbqViRaQ9aNYZtTGmG/AC8HvgvjZ9Rn2iqnLIehY+fQRKD1LV5zoKh9/P5spUJs9aVmfTczvGctuIdC47L41O8ZGEhJhWKlpEgk1jZ9TNDeo5wENAHPCL+oLaGDMFmAJw9tlnD9mxY8cZFe06ZYXwxeOw9EmoroTMH5A34CeMeHxtg7vcO7Y391zeG2MU2CLSuDPq+jDGXAnst9aubGw7a+0sa22mtTYzNTX1NEt1scgEGPNr+K+vYdDNsOLvdHlhODnjsnnltn48d/uFJ+3y5w8388cFehK6iJyZJs+ojTEPAbcAVUAkEA+8bq29uaF92lTXR0MKNsNHD8KGeRCTBpf+Nwy5HUI9eL2Wxz/ewv996AvpqwZ04XHnoQUiIvU5466PWgcaTQNdH7W1i6A+ZtcKWPhb2PkFJJ8Dl/8W+lwLxpC9u5ArH18CQO+0WO4fdz5j+3Rs5YJFxI38NepD6nPWhXDHfPj+axAWCf++HZ4ZA9sX069rAit+NRaAzfuLufPFLNKnvsvImR9TXF7VunWLSNA4pTPq5mpXZ9S1eathzavw8e/hSC70vBz6f5evQ/txy5y8esN59QNXcLCkgt2HSukQF875neLrObCItHV+6/pornYb1MdUlsGXs+CLx6Ak37csuSdV3S/m69AM3j7SkxfXHK1314QoD10SoxjZM4WC4nIiPaHcNiKd8zvFafSISBumoG4tXi/sXwfbF8P2z2DH51B+BIDKlPP5577ufOHty3LvBRwhpomDQa+0WOb/1yV4Qo1CW6SNUVC7RXUV7FkN2z/1hffOZVBVCiYEOmWQHTGQg2nD2BbVnyc+30NaXCRlldVsKyip93Brp19BXKQuXxdpCxTUblVVDrlZkPOZL7h3fQneSggJg66Z0GMU9LgEug1lZV4pP3t1FTsP1u0yyZk5sZWKFxF/UlAHi4qjsGuZ01WyGPK+BuuF0Ag4+yInuC9lX+wFXPTwYgCiPKHMv+cSeji3ZLXWNtktsja3kI7xEXoYgoiLKKiDVVkh7Fh6PLj3OZerh8dSldqHeTvD2WXT2OlNY4dNY6ftSD4JhIeFUe489GDWLUPYXlDCQ+9trPcjfnHFuTy64Bsm9O/E4zcOJlT3JxFpFQrqtqLkgK+bJOczyN+EPbQdCndjOP7/sMx62GV9wb3LprHTprHDdmSnM19OeJMfs/2hCfqxUqSFNRbU7fOZicEqJgX6Xut7AQagqgIKd2EPbodD23n3o89JrdrDxYmFhB7+DE913T5tG9cZk5ROdUJ3/rq6iq1VaSR3683bOyPIJwEw/GPJdu685JwWb56I1E9n1G2ZtXD0ABzcDodyar2c+SN5UOts3Hqi2VKeRIFN4AAJTBwxAG90KnlVcZx1Vg+ITcXGpFIWnkJUVFStjzneL/6v5Tv55Rt17yj4+dQxdNZtX0Uapa4PqV9lGRzeWSfEc3M2sSdvJx0opFNoEVG2/gtzCollvzeBfJtAAQkU2ATybSIFxJN/bNomcIB4qpx/uD1/x4WMPi+t5donEkTU9SH180RC6rm+l6MbEFZYxrCHPgIgknI6mEJSKfS9m0I6ONO++cP0ZxsdQo4QZ0rr/ZiDNpYCm0DFv5Ogdw+ITqn16uC8Jx9fFh7je9JOE6q9luzdhfTtEk9YqG5bI22XglpO0ikhki9/dTkX/eEjymwET//0Om5/bgVfF5UD8MytmcRFhrHjQAn7K6oZdW4qKw6U0C/NQ+jRAlJsIZTsh+J9UJxPcsl+8rdupbhgLyV5G4mpKvR1ydj6H2FmQyPwRiUTGtOhToDbqGQ+32N5dd1RDhLHIRvHQRvHuAsvYPp1mc0Kd5FgpK4PaRH7jpRx0R98Z+nTxp/PQ+9t4K6LUpn/ZTbJFJFkikg2RSRzhGRTTBJF9EmspLq4gDjvEVJCikikuMHj29BwTGQiRCVCQ+9RSfWv80Qp5KXVqY9aXOH7zyzji60HTnv/UKpJoMQX6hTx1PXpmKMFPPPBVySYEu7MTCKsohBKD0PZ4ePvZUeo/aPpyQcObyDck3xn9FHJzvsJ8+GxCnjxG/VRiys8duMg/m/hN8xevhOA0BDDmz8eSf9uCVRVewkLDaGorJK4SA8VVV4yZnxAWaWXD++7lAXr99KvSwKjzj35MW9rv1nOki0FzFwG62Z8myhPKBXVXiI9ob4NvNW+m2E5wb09dzf79u3FlB7mk9WbSagq4aLkENI8pZQXHqRk5zckUELniDI8lUUNNyjE03iQ1/celQSh+mMnp0Zn1BL0isur6PfABwE59roHLiemughKD8LRg428H6o7761s+KARCRCVULcr5qRumaSTpyPidQbfhqnrQ9q8orJK+k9fcMr7zb1rBO+syeO5z3NIiPJQWFrJrFuGADDlJd/znFc/cAUJUadwl0JroaK48SA/1jVTeqjudGMBb0Ia73ePSoKIOF+XTHisb/RMeIyzLOb4spDQU/7vJIGnoJZ26WBJBat2HeLC9GRW5Bwk73AZqXERxEWGMaJnhyb3H/PoJ2wrKGFI9yTm3jWCoxVVLFi3j/H9OwEQEXZy4OUXlfNe9h6qvZaLe3XgpWU7eHHpDgCGdE9i5Y5DgK+LJibihC4Qa6HyaP0B3tR02WHfDbyawxN9PMTDnRCPiK0V5rG1ljkvT5TvFRbhe+RcWASE1Z6P9A33DIvUXwSnSUEtchqstfSYNj8gx75/3PncNbqn/w7o9UJFEZQX+87mK4qhosSZL/Gtqyhxlh2bLj7+Xl5r+th8A8MnmxQSdjy8a0K9VpDXDvewCN/2oR5fn39ImK8P/8TpUGe+Zttjy8NOmPYc38aEAMa514LxdRvVLDMNv9fZhvq3sV7nZZ2XF7AnLD9xGcen69nW9BytHxNFTpUxhs/++zIueWTRGR3n1uHd+fkV5/Hz11YzslcKM95ez8Pvb2REzxQGnJXon2JDQqgOjyc0MsE/x7PWd7/0mpA/CtXlvqtZq8p862reS0+YL6tnu1rT5UW+R9Qdm6+u8nX5VFf6fvg9Nn26f1G0QTqjFmmGsspqyqu8J/VVV1Z7+WZfEV0SokiKqXtnwmqvrfe2sU8u2sL/frCJjG4JvHX3SIwxHCqpYNO+IlJiwumYEEl8PU/usdayatdhZr63ke8M6orXwoPvrKOs8niXx23DuzPjmn5+anUrsxa8VU6AVzqBXjvUqxpZX1XrzNWe8O6tZ1ntd5reJiQU39l1iPM68Ww85Ph8nWUhvjP8k5YbTI9L1PUh4ib/XLaDX7+Z3eD6KzM6c9uIdM5NiyM+KozXsnZx/9y1DW5f28bfjTs+NFGChsZRi7jMjUPPZvE3+SxYv69m2TmpMWzL9z0f8501e3hnzZ4G98/snoTXWvYUljH6vDQSoz3065LA3f/6ij/M38CDbeWsWgAFtUirCA0xzLq13pMnCorLufrxJeQVltVZPveu4QzpntzgMau9lk7xkby4dAe3Du9Or7S4ZtXi9VqKyqooLK0kv7iMronRdEqI5ImPN7NyxyF+e1VfuiRG1jvKRVqGuj5EXG7ZtgN0TYzirOToJrdduvUANz6zDIDfXNmH7w89m037ikiK9lBUVkXOgRLG9e1EWGgIb63azetf7ebTb/JPOk6n+Ej2Hjn+F0VsRBirfvst3aUwgDQ8T6Qdaar/+1TUDuzpV/Xh9pE9/HJcOZn6qEXakZuHdadTfCR3vtj0ydKgsxO5ZVh3rhrQBY9ztjxnZS6/+Pdq5vxoOJnpyZRVVnPJI4uY/vZ6RvbqQO+OzetSEf/RGbVIO3D4aAWJ0U0/2Lgha3MLueqJJQD87tp+3DKsO3sLy0iLi6h5xJq1lq92Huazzfkkx4Qz9oKOdEn0PbKtrLIaT2gIJRVVRIaFEh4Wwsa9R+iWFE3siVdotlPq+hCRM/bQ/A08vXhbvetuGdadl5btOK3jLvjZKM7VWXqjQd3kLwPGmEhjzJfGmNXGmHXGmBn+L1FE3G7ahAv4150X1buudkiP79fplI77j8+2n1Fd7UFz/s1RDoyx1hYbYzzAEmPMe9baZQGuTURcZkSvDiydNobYiDDinKsn1+UVct+rq/njDQPo17XhS9i9XnvSk+inz1vH81/kcPOw7vTvduqXvxeWVlJaUU2nhMhT3jeYnFLXhzEmGlgC3GWtXd7Qdur6EJHmOHy0gmEPfVRzGfxzt1/IZeenYa1l5Y5DJEaHU+X1Mu7Pn9XZL7N7ElnOnQiP6ZUWS3xkGBMzulBV7WVsn470TI1tsbacqTPuozbGhAIrgV7Ak9ba++vZZgowBeDss88esmPH6fVXiUj78vPXVjP3q9zT3v+c1BhyD5ZSUX3ybV7f/snFp3Wm3hr89mOiMSYReAP4qbW2wYGaOqMWkeay1vLJN/m8t3YPr2WdHNhdE6P4zZUXMK5fZ6q9lqVbD7Dz4FEu6d2h5iIgr9fy9po8jpRW8pu31nFxrw4s2VJAUrSHD++7lOSYcCqqvTVXV9bXDVNbWWX1SfdLsdZiAviEHb+O+jDGPACUWGsfbWgbBbWInK7Kai+b9hbRt0v8GQXj+9l7+NE/v2pyu84Jkfzo0p48MG/dSesuTE9iRc6hk5b/7pq+TB56ds3Yc384o6A2xqQCldbaw8aYKGAB8LC19p2G9lFQi0hrs9Zy32ureePr3QH7jDfvHsm2/GLG9+tMRFgIz3+Rw6JN+7lteDp7Cks5dLSSJz7eQkW1l8vOS+Wh6zLq/PBZeLSSovJKuiZGERISckZBnQG8AITiG873mrX2wcb2UVCLiNtYa6ms9uWdJ9SQX1xOWlwk015fyysrdvL0zUOI9IQSERbChenJ5BWW8vt3NzCiVweuGdiF+EgPJeVVvP71bn5zhpfoXzeoK8u3H2T34dKaZTsevlIXvIiI+IvXa5k8axlf5hxk+DkpLN12oGbd0PRk9hWV0a9rAu+t3cO9Y8/l7st6sXzbAb7/97qD5aLDQ6mo8lLltQpqEZFAKi6vIr+onPSU6Eb71dfnHSE6PJRITyipcRF1ngCkmzKJiARQbERYs+5Z0qdL/GkdXzeXFRFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLNRnUxpizjDGLjDEbjDHrjDH3tERhIiLiE9aMbaqAn1trvzLGxAErjTELrbXrA1ybiIjQjDNqa+0ea+1XznQRsAHoGujCRETE55T6qI0x6cAgYHk966YYY7KMMVn5+fn+qU5ERJof1MaYWGAucK+19siJ6621s6y1mdbazNTUVH/WKCLSrjUrqI0xHnwhPdta+3pgSxIRkdqaM+rDAP8ANlhr/xT4kkREpLbmnFGPBG4BxhhjVjmvCQGuS0REHE0Oz7PWLgFMC9QiIiL10JWJIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXK7JoDbGPGuM2W+MyW6JgkREpK7mnFE/D4wLcB0iItKAJoPaWrsYONgCtYiISD381kdtjJlijMkyxmTl5+f767AiIu2e34LaWjvLWptprc1MTU3112FFRNo9jfoQEXE5BbWIiMs1Z3jey8BS4DxjTK4x5j8CX5aIiBwT1tQG1tobW6IQERGpn7o+RERcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5ZoV1MaYccaYTcaYLcaYqYEuSkREjmsyqI0xocCTwHigD3CjMaZPoAsTERGf5pxRDwW2WGu3WWsrgFeAawJbloiIHBPWjG26ArtqzecCF524kTFmCjDFmS03xmSfeXmtqgNQ0NpFnCG1wT3aQjvUhsDq3tCK5gS1qWeZPWmBtbOAWQDGmCxrbWazy3MhtcEd2kIboG20Q21oPc3p+sgFzqo13w3IC0w5IiJyouYE9QqgtzGmhzEmHJgMzAtsWSIickyTXR/W2ipjzE+AD4BQ4Flr7bomdpvlj+JamdrgDm2hDdA22qE2tBJj7UndzSIi4iK6MlFExOUU1CIiLufXoHbzpebGmGeNMftrj+82xiQbYxYaYzY770nOcmOMecxpxxpjzOBa+9zmbL/ZGHNbC7fhLGPMImPMBmPMOmPMPUHajkhjzJfGmNVOO2Y4y3sYY5Y7Nb3q/HiNMSbCmd/irE+vdaxpzvJNxphvt2Q7nM8PNcZ8bYx5JxjbYIzJMcasNcasMsZkOcuC7fuUaIyZY4zZ6PzZGB5sbWiStdYvL3w/NG4FzgHCgdVAH38d3w/1jQIGA9m1lj0CTHWmpwIPO9MTgPfwjSEfBix3licD25z3JGc6qQXb0BkY7EzHAd/gu6w/2NphgFhn2gMsd+p7DZjsLH8KuMuZ/jHwlDM9GXjVme7jfBjXaFwAAAM4SURBVM8igB7O9y+0hb9X9wH/At5x5oOqDUAO0OGEZcH2fXoBuNOZDgcSg60NTbbRj/+xhgMf1JqfBkxr7QaeUGM6dYN6E9DZme4MbHKmnwZuPHE74Ebg6VrL62zXCu15C/hWMLcDiAa+wne1awEQduL3Cd+Io+HOdJiznTnxO1Z7uxaqvRvwETAGeMepKdjakMPJQR003ycgHtiOMzAiGNvQnJc/uz7qu9S8qx+PHwgdrbV7AJz3NGd5Q21xTRudfzoPwnc2GnTtcLoMVgH7gYX4ziQPW2ur6qmppl5nfSGQQuu348/AfwNeZz6F4GuDBRYYY1Ya320gILi+T+cA+cBzThfU340xMQRXG5rkz6Bu1qXmQaKhtriijcaYWGAucK+19khjm9azzBXtsNZWW2sH4jsrHQpc0EhNrmuHMeZKYL+1dmXtxY3U47o2OEZaawfjuzvm3caYUY1s68Y2hOHr0vybtXYQUIKvq6MhbmxDk/wZ1MF4qfk+Y0xnAOd9v7O8oba0ehuNMR58IT3bWvu6szjo2nGMtfYw8Am+/sJEY8yxi7Bq11RTr7M+AThI67ZjJHC1MSYH3x0lx+A7ww6mNmCtzXPe9wNv4PtLM5i+T7lArrV2uTM/B19wB1MbmuTPoA7GS83nAcd+3b0NX5/vseW3Or8QDwMKnX8+fQBcYYxJcn5FvsJZ1iKMMQb4B7DBWvunWquCrR2pxphEZzoKGAtsABYBkxpox7H2TQI+tr6OxHnAZGdERQ+gN/BlS7TBWjvNWtvNWpuO77v+sbX2pmBqgzEmxhgTd2wa3/cgmyD6Pllr9wK7jDHnOYsuB9YHUxuaxc8d+xPwjUTYCvyqtTvgT6jtZWAPUInvb8//wNdH+BGw2XlPdrY1+B6WsBVYC2TWOs4PgC3O644WbsPF+P45tgZY5bwmBGE7MoCvnXZkA791lp+DL6S2AP8GIpzlkc78Fmf9ObWO9SunfZuA8a303RrN8VEfQdMGp9bVzmvdsT+zQfh9GghkOd+nN/GN2giqNjT10iXkIiIupysTRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXG5/w9Lss6zMbb9RQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(15, lr, wd=wd, moms=(0.8,0.7), callbacks=[ShowGraph(learn_lm)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned2_sp15_multifit_bwd')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned2_enc_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Save best LM learner and its encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned_sp15_multifit_bwd')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned_enc_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning \"forward Classifier\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # \"error\", \"ignore\", \"always\", \"default\", \"module\" or \"on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 248 ms, sys: 16 ms, total: 264 ms\n",
      "Wall time: 261 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_aws_sp15_multifit', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# # keep reviews with a label not nan\n",
    "# df2_trn_val = df_trn_val[df_trn_val[label].notnull()].copy()\n",
    "# print(len(df2_trn_val))\n",
    "# df2_trn_val.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8.63 s, sys: 1.47 s, total: 10.1 s\n",
      "Wall time: 8.81 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = (TextList.from_df(df_trn_val, path, vocab=data_lm.vocab, cols=reviews, \n",
    "                              processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_from_df(cols=label)\n",
    "    .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 828 ms, sys: 36 ms, total: 864 ms\n",
      "Wall time: 865 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas.save(f'{lang}_textlist_class_sp15_multifit')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Get weights to penalize loss function of the majority class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.05 s, sys: 12 ms, total: 1.06 s\n",
      "Wall time: 1.06 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_multifit', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(33993, 3777, 37770)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_trn = len(data_clas.train_ds.x)\n",
    "num_val = len(data_clas.valid_ds.x)\n",
    "num_trn, num_val, num_trn+num_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([10126, 23867]), array([1144, 2633]))"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_LabelCounts = np.unique(data_clas.train_ds.y.items, return_counts=True)[1]\n",
    "val_LabelCounts = np.unique(data_clas.valid_ds.y.items, return_counts=True)[1]\n",
    "trn_LabelCounts, val_LabelCounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.7021151413526314, 0.2978848586473686],\n",
       " [0.6971141117288854, 0.3028858882711146])"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_weights = [1 - count/num_trn for count in trn_LabelCounts]\n",
    "val_weights = [1 - count/num_val for count in val_LabelCounts]\n",
    "trn_weights, val_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Training (Loss = FlattenedLoss of weighted CrossEntropyLoss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.08 s, sys: 64 ms, total: 1.15 s\n",
      "Wall time: 1.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_multifit', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, pretrained=False, drop_mult=1.0, \n",
    "                                  metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15_multifit');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "loss_weights = torch.FloatTensor(trn_weights).cuda()\n",
    "learn_c.loss_func = partial(F.cross_entropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "functools.partial(<function cross_entropy at 0x7f14b8914f80>, weight=tensor([0.7021, 0.2979], device='cuda:0'))"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEJCAYAAAC+I6F6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU5fX48c+ZyUaABEISCFlYww5GCCCubFrEBW1dwC5qbd3qUtvaahfrz6W1ta2trbZfaovFDfeKW3EDVAQl7DuEgCRsCUsgELKf3x8zwSFMkiGZO5PlvF+veTFz73PvnIdJcua5z3JFVTHGGGOayhXuAIwxxrRulkiMMcY0iyUSY4wxzWKJxBhjTLNYIjHGGNMslkiMMcY0i6OJRESmiMgmEckVkXv87M8QkfkiskJEVovIVO/23iJyTERWeh//8DlmlIis8Z7zcRERJ+tgjDGmYeLUPBIRcQObgfOBAmApMENV1/uUmQmsUNW/i8gQ4B1V7S0ivYG3VHWYn/N+AdwJLAHeAR5X1XcdqYQxxphGRTh47jFArqrmAYjIHGAasN6njAJx3ufxwK6GTigiKUCcqi72vp4NXAY0mEgSExO1d+/eTaiCMca0X8uWLdunqkmNlXMykaQC+T6vC4CxdcrcD7wnIrcDHYHJPvv6iMgK4DDwS1X9xHvOgjrnTPX35iJyI3AjQEZGBjk5OU2viTHGtEMi8mUg5ZzsI/HXd1H3OtoM4GlVTQOmAs+IiAvYDWSo6unAj4DnRSQuwHN6NqrOVNVsVc1OSmo0oRpjjGkiJ1skBUC6z+s0Tr50dQMwBUBVF4tIDJCoqoVAuXf7MhHZCgzwnjOtkXMaY4wJISdbJEuBTBHpIyJRwHRgbp0yO4BJACIyGIgBikQkydtZj4j0BTKBPFXdDZSIyBne0VrfAd5wsA7GGGMa4ViLRFWrROQ2YB7gBv6tqutE5AEgR1XnAj8G/ikid+G5RHWdqqqInAs8ICJVQDVws6oe8J76FuBpoAOeTnYbsWWMMWHk2PDfliQ7O1uts90YY06NiCxT1ezGytnMdmOMMc1iicQYY0yzWCIJoYKDpRyrqA53GAEpLCnj1WUF1NS0/UufxpjmsUQSIuVV1Uz9yyf86o214Q4lIL94fS0/fnkV97y2mmpLJsaYBlgiCZHP8w5wuKyKuSt3UVRSHu5wGrSm4BDvr9/LkJQ4Xsop4O6XV1kyMcbUyxJJiMzfVEikW6ioruG5zwNadSBs/vzBZuJiIphz0xn8+PwBvLZiJz98cSVV1TXhDs0Y0wI5ObPd+Ji/sZCz+icC8OySHdwyvh/REe4wR3WyVfnFfLixkJ9cMIC4mEhun5RJZISLR97dyNHyKu6aPIBhqXHY6v3GmFrWImmGYxXVVAbwLX3bvqNs31/KxEHJXH9WH/YdKeft1btDEOGpe+yDzXSJjeS6s/oc33bzef349SVD+GzrPi7526dc9PinzF68nUPHKsMXqDGmxbBE0kSqyvSZi5n4xwWs33W4wbLzNxYCMGFgMudmJtIvqSOzFm2npU0GXb7jIAs2FXHjuX3pFH1iY/X6s/rw+c8n8+BlwxCB+95Yx/l/Wkjh4bIwRWuMaSkskTTRqoJDrCo4xN5D5Xz974t4Y+XOesvO31RI/+ROpCfEIiJcd1Yf1uw8xPIdB0MYceMee38zCR2juHZcb7/74ztE8u0zevH2Hefw8s3jOFxWyQ9fXBlQR/wT83O5+K+fcPMzy3j47fXMXrydrUVHglsBY0xYWCJpoheX7qBDpJt5d53L8NR47pyzkofeWn9Sh/TR8io+zzvAhIFfLWX/jZGpdI6J4N+Ltoc46vot+/IAn2zZx03n9qVjdONdZ6N7J/DApcP4bOt+npyf22DZncXH+PMHmyktr2ZLYQmzF3/JfW+s4+LHP+W9dXuCVQVjTJhYImmC0ooq3ly1m6nDU+iT2JHnvncG147rxVOfbuOHL6484ZLVotx9VFTXMGFQ8vFtsVERTB+dzv/W7mFX8bFwVOEk//x4G11iI/n2uF4BH3NldhrTsnry2Aeb+WLbgXrLPeFNNM9+bywf/ng8Gx6Ywsd3T2BAj87c9Owynvok74T/s/wDpfxh3iZLMsa0EpZImuDt1bs5Ul7F1aM9t1uJinDx/6YN4ycXDOCt1bt57vMdx8vO31REp+gIsnslnHCO74zrjaryh/c2hb2vZGfxMd5bv4fpozOIjQp8IJ+I8PDlw8lIiOXOOSs4eLTipDIFB0t5OSefq0en07NLBwBcLiGjWyxzvn8GU4b24KG3N3DfG+v4LHcfN87O4bxH5/O3+bnc/sIKNu5puP/JGBN+lkia4KWcfPomdmR0764nbL91fH/OHZDEA2+tZ8Puw6gqCzYVck5mIlERJ/5XpyfEctuE/ry2fCcvLs0nnJ5d4pnX8q0zMk752E7REfztmpHsP1LBD19cSUXViZf2nlywFUG4dXz/k47tEOXmiWtGctN5fXlmyZdc89TnLN1+gJvP68ebt51NXIdIfvDcco6WVzWtYsaYkLBEcoq2Fh1h6faDXDU6/aS5FC6X8KerTiO+QyS3Pb+c5TsOsvtQGRMGJvs9152TB3BOZiL3zV3H2p2HQhH+Scoqq3nhix1cMKQHaV1jm3SOYanx/L9pQ1m4uYgbn8mhrNKznpi/1khdLpdw74WDeeKakTx6xQgW3zuJn04ZxPC0eP5ydRZ5+45y3xvrmlw/Y4zzLJGcopeW5uN2CV8fmep3f2Kn6ON/AG+cvQyA8QP93zPe7RL+Mv10unWM4uZnl3GoNPTzMuau3EVxaSXXntm7WeeZMSaDR74+nIWbi7hu1hccKa/iifne1siEfo0ef9GIFK7MTicm8qtJmmf2T+T2iZm8uryAV5YVNCs+Y4xzLJGcgsrqGl5dXsCkQckkd46pt9yZ/RO5fUJ/9h+tYFhqHMlx9ZdN6BjFE98cyd7DZfzopZUhXW1XVZn12XYGdu/MGX0TGj+gEdPHZPDnq7NYuv0g02cu5uWcfKaPSScl3n9rJBB3TsrkjL4J/Oq/a8ktLGl2jMaY4LNEcgo+2ljIviMVxzvZG3LHpEyuGJXGDWf3abTsyIyu/PKiIXy4sZAH3lofsmSydPtBNuw+zHVn9Q7akifTslJ58psj2bznCC4RbhnfeGukIbWttg5Rbn726pqwD0wwxpzM1to6Ba8v30ly52jOG+D/UpWvCLeLP1x5WsDn/s64Xmzff5RZi7ZzpLyKR74+nAh38PL80fIqXs7JZ2hqPFnpXYh0u/jPZ9uJ7xDJZVn+L9M11deG9mDOTWdwqLSyWa2RWt3jYrhnyiB++upq5q7axbQgx2uMaR5HE4mITAH+AriBp1T1kTr7M4D/AF28Ze5R1XdE5HzgESAKqADuVtWPvMcsAFKA2gkYF6hqoZP1qLVxz2HG9EkI6h/4WiLCfRcPIS4mkr98uIUjZVX8ZUYW0RFuqmuU5TsO8snmIg6WVlJWWc2xymoUuOncvoxI69Lo+Z9ckMsT87cC0DHKzZg+CXy8ZR/fO7sPHaKCv3jkyIyujRc6BVeMSmP2ku088u5Gzh/S/ZSGKRtjnOXYb6OIuIEngPOBAmCpiMxV1fU+xX4JvKSqfxeRIcA7QG9gH3CJqu4SkWHAPMD3a+g3VTXHqdj9qa5RdhYfY+rwFMfeQ0S46/wBxHeI5IG31nPdv5eS2rUDH20s5MDRCtwuIS4mgg6RbmKi3BSXVvL+ur38+tIhXDMmo97LU4eOVTL7sy+ZPLg7V4xKZVHufhbl7iMmwsW3zgh8AmI4uVzCry8ZypX/WMz/LczjrvMHhDskY4yXk1/rxgC5qpoHICJzgGmAbyJRIM77PB7YBaCqK3zKrANiRCRaVcN2R6g9h8uorFbSE5o2RPZUfPfsPnSOieCe19YQu8vNxEHJnD+kO+cNSKJzTOTxcgePeuZu/OL1tSzbfpCHLx/ut3Xx7JIvKSmv4oeTMxmWGs+UYZ5kWFOjuFytZzn40b0TuHhECv9YuJWrRqeTWs+QYmNMaDmZSFIB35l2BcDYOmXuB94TkduBjsBkP+f5BrCiThKZJSLVwKvAQ+qnB1ZEbgRuBMjIOPWJdnXt2F8KQEYIEgnAldnpXDC0B7FRbiLruZTWtWMUs64bzd/m5/LYB5tZv/sws64ffUK/RGlFFf/6dBsTBiYxLDX+hONbUxKpde/Uwby/fi+/fWcDf7tmJAAVVTV8uf8oGd1iW+Q9Xoxp65xMJP7+StX9gz8DeFpV/ygi44BnRGSYqtYAiMhQ4HfABT7HfFNVd4pIZzyJ5NvA7JPeSHUmMBMgOzu72UN98g96Ekl6EyftNUV8h8hGy7hcwh2TMslK78Ktzy3nun8v5aWbxx0/9oUv8jlwtILbJp48s7w1Su3SgZvO68fjH24BlpNXdJQthSVUViszxqTz26+PCHeIxrQ7Tg7/LQB8x8mm4b105eMG4CUAVV0MxACJACKSBrwOfEdVt9YeoKo7vf+WAM/juYTmuPwDpbhdQkqX+ueEhNO5A5L4v2+PIm/fEb4/2zO7vLyqmn9+nMfYPgmM6tX8eSItxc3n9aVvYkeW5O0nsXM0N5zdl6nDezBnaT4bdtvaXMaEmpMtkqVApoj0AXYC04Fr6pTZAUwCnhaRwXgSSZGIdAHeBu5V1UW1hUUkAuiiqvtEJBK4GPjAwTocl3+glJT4mHovM7UEZ/VP5I9XZXHHCyu468WVnJ2ZyJ7DZTx6Zdv6lh4bFcGHPz7vhMEFxaUVLMrdz2/e2cDs746xWwEbE0KO/VVU1SrgNjwjrjbgGZ21TkQeEJFLvcV+DHxfRFYBLwDXefs7bgP6A78SkZXeRzIQDcwTkdXASjwJ6p9O1cHXjgOlIesfaY5LT+vJLy8azLtr9/DrN9YxIi2es733im9L6iaKLrFR3DEpk0+27GPB5qIwRWVM++ToYHxVfQfPkF7fbff5PF8PnOXnuIeAh+o57ahgxhioHQeOMXmw/8UXW5rvndOXwpJyZn6cx+0TM9vNt/Nvn9GLZxZv5zdvb+Cc/omOzPcxxpzMftMCcKyimn1HykMy9DdY7r1wEJ/8dALnD+ke7lBCJirCxT0XDmJL4RFezPlqwOCh0kreW7eHkrLQL4ppTHtg04MDcHzEVitKJCLSquINlq8N7cGY3gk89v5myitr+GDDXj7fdoDqGmVUr648c8MYmxVvTJBZiyQA+Qdqh/7aBLiWTkT4xUWD2XekggfeWk9hSTk3ntuX+y4ewoodB7n1ueVUVtc0fiJjTMDsq1kAdhwI7WRE0zynpXfh1VvG0TU2ir5JnY5v7xDl5t7X1vCTl1fx2FVZrXJCpjEtkSWSAOw4UErHKDcJHaPCHYoJkL95MzPGZHDgaAWPzttEfIdIrh6dzt7DZew+VMahY5V864xexMU0PgnUGHMiSyQByD9wjPSE2HYz+qktu3V8Pw4ereCpT7cxe/GXJ+yrrFLunJwZpsiMab0skQQg/0ApGd3sslZbICL8fOpgsnsnUKNKj/gYUuJjuPvl1by4dAe3TeyP2y55GXNKrLO9EapK/sHSkK6xZZzlcglThvVg6vAURmZ0JSW+A98cm8GuQ2Us3BySW9sY06ZYImnE/qMVlFZUk5FgI7basslDupPYKZrnP98R7lCMaXUskTTi+NBfG7HVpkW6XVyVncZHGwvZVXys8QOMMcdZImmEDf1tP2aMyUCBF5fmn7D9f2v3MO63H/Lpln3hCcyYFs4SSSNqWyRp1kfS5qUnxHJOZhIv5eRT5Z20uHjrfu54YQV7Dpdx63PL2LbvaJijNKblsUTSiPwDx0jqHO33Fram7blmTAa7D5WxYFMRa3ce4vuzc+jVLZa5PzibCLeLG/6zlEPHbM0uY3xZImlEa1k+3gTHpMHJJHeO5okFuVw3aylxMRHMvmEMw9PiefKbI9mxv5TbX1hxvMVijLFE0ijP0F8bsdVeRLpdXD06nRU7iqmuqWH2DWNJifd8/mf07caDlw3j481F/PbdjWGO1JiWwyYkNqCyuoZdxcfIOD013KGYEPr2Gb3YsPswt03MpH9ypxP2zRiTwaY9Jfzr021MHZ7CqF5dwxSlMS2HtUgasLu4jBqFNLu01a4kx8Xw1LWjyUrv4nf/3V8bSMcoN3O+sDknxoAlkgbZ0F/jT8foCC4e0ZO3Vu+2m2UZg8OJRESmiMgmEckVkXv87M8QkfkiskJEVovIVJ9993qP2yQiXwv0nMFkicTU5+ox6RyrrOat1bvDHYoxYedYIhERN/AEcCEwBJghIkPqFPsl8JKqng5MB570HjvE+3ooMAV4UkTcAZ4zaPIPlhLpFrrHxTj1FqaVOj29C5nJnU6avGhMe+Rki2QMkKuqeapaAcwBptUpo0Cc93k8sMv7fBowR1XLVXUbkOs9XyDnDJodB0pJ6xprq8Gak4gIV49OZ2V+MRv3HA53OMaElZOJJBXw/bpW4N3m637gWyJSALwD3N7IsYGcEwARuVFEckQkp6ioqEkVGJISxwVDuzfpWNP2fX1kGpFusVaJafecTCT+vsZrndczgKdVNQ2YCjwjIq4Gjg3knJ6NqjNVNVtVs5OSkk4h7K/8YEJ/7r1wcJOONW1fQscoLhjSg9dX7KS8qjrc4RgTNk4mkgIg3ed1Gl9duqp1A/ASgKouBmKAxAaODeScxoTM1aPTKS6t5L11e8MdijFh42QiWQpkikgfEYnC03k+t06ZHcAkABEZjCeRFHnLTReRaBHpA2QCXwR4TmNC5uz+iaR26cBLOXZ5y7RfjiUSVa0CbgPmARvwjM5aJyIPiMil3mI/Br4vIquAF4Dr1GMdnpbKeuB/wA9Utbq+czpVB2Ma43IJV2an8cmWfezYXxrucIwJC1H128XQpmRnZ2tOTk64wzBt1J5DZZz9u4+49sze/Opix0ajGxNyIrJMVbMbK2cz241pph7xMVw4PIWXluZztLwq3OEYE3KWSIwJguvO7E1JeRWvLi8IdyjGhJwlEmOCYGRGF05Li+fpRdupqWn7l4uN8WWJxJggEBGuP6sPefuO8vGWpk2ANaa1skRiTJBMHZ5CcudoZi3aHu5QjAkpSyTGBElUhItvndGLhZuLyC08Eu5wjAkZSyTGBNGMMRlEuV3MXrw93KEYEzKWSIwJoqTO0VxyWk9eWVbAtn1Hwx2OMSFhicSYILtjUn9iIt18+1+fs+dQWbjDMcZxlkiMCbJe3Try9PWjOXi0gu/8+3OKSyvCHZIxjrJEYowDRqR14Z/XZrN9fynXzVpqM95Nm2aJxBiHnNkvkb/OOJ3VBcXc/OwyKqtrwh2SMY6wRGKMg742tAePfH0En2zZx8Nvbwh3OMY4IiLcARjT1l01Op1Ne0v416fbGJ4azzdGpYU7JGOCylokxoTAvRcO4sx+3bj39TWsLigOdzjGBJUlEmNCIMLt4m/XjCSpUzQ3PbOMopLycIdkTNBYIjEmRBI6RjHzO6M4WFrBbc8vt1WCTZthicSYEBraM577LxnK59sO8MGGveEOx5igcDSRiMgUEdkkIrkico+f/Y+JyErvY7OIFHu3T/DZvlJEykTkMu++p0Vkm8++LCfrYEywXTEqjV7dYvnrR7m0h1tdm7bPsUQiIm7gCeBCYAgwQ0ROuKG1qt6lqlmqmgX8FXjNu32+z/aJQCnwns+hd9fuV9WVTtXBGCdEuF38YHx/1uw8xIJNdu8S0/o52SIZA+Sqap6qVgBzgGkNlJ8BvOBn+xXAu6pa6kCMxoTF5SNTSe3Sgcc/2mKtEtPqOZlIUoF8n9cF3m0nEZFeQB/gIz+7p3NygnlYRFZ7L41FByNYY0Ip0u3ilvH9WLGjmEW5+8MdjjHN4mQiET/b6vvqNR14RVWrTziBSAowHJjns/leYBAwGkgAfub3zUVuFJEcEckpKrLLB6bluTI7jR5xMTz+4ZZwh2JMsziZSAqAdJ/XacCuesr6a3UAXAW8rqqVtRtUdbd6lAOz8FxCO4mqzlTVbFXNTkpKalIFjHFSdISbm8/ryxfbD7Akz1olpvVyMpEsBTJFpI+IROFJFnPrFhKRgUBXYLGfc5zUb+JtpSAiAlwGrA1y3MaEzPQxGSR2irZWiWnVHEskqloF3IbnstQG4CVVXSciD4jIpT5FZwBztE6Po4j0xtOiWVjn1M+JyBpgDZAIPORMDYxxXkykp1Xy2db91ioxrZa0hxEj2dnZmpOTE+4wjPGrrLKa8x6dT0ZCLC/dNA5PY9uY8BORZaqa3Vg5m9luTJjFRLq5bWImS7cfZOFmGxhiWh9LJMa0AFdnp5PWtQN/fG+zzSsxrY4lEmNagKgIF3dOymTNzkPMW2drcJnWxRKJMS3E5aen0jepI396fxPVtjKwaUUskRjTQkS4Xdw1eQCb9x7hzVX1TbkypuWxRGJMC3LR8BQG9ejMnz/YTFV1TbjDMSYglkiMaUFcLuGu8wewfX8pb662VolpHSyRGNPCnD+4O4N6dOavH+VaX4lpFSyRGNPCuFzCHZMyySs6yttrdoc7HGMaZYnEmBZoytAeZCZ34q8fbrF7u5sWzxKJMS2QyyXcPimTLYVH+N+6PeEOx5gGWSIxpoW6aHgKfZM68ri1SkwLF1AiEZF+tXciFJHxInKHiHRxNjRj2je3S7h9Yn827inh/Q022920XIG2SF4FqkWkP/AvPLfFfd6xqIwxAFwyoie9u8Xy5PzccIdiTL0CTSQ13vuLXA78WVXvAlKcC8sYA57Z7ted2ZtVBYfYsPtwuMMxxq9AE0mliMwArgXe8m6LdCYkY4yvaVmpRLqFV5YVhDsUY/wKNJFcD4wDHlbVbSLSB3jWubCMMbW6doxi8uDu/HfFTipt2RTTAgWUSFR1vareoaoviEhXoLOqPuJwbMYYryuz09h/tIL5GwvDHYoxJwl01NYCEYkTkQRgFTBLRP7kbGjGmFrnZiaR1Dmal+3ylmmBAr20Fa+qh4GvA7NUdRQwubGDRGSKiGwSkVwRucfP/sdEZKX3sVlEin32Vfvsm+uzvY+IfC4iW0TkRRGJCrAOxrRaEW4XXz89lfkbC9l3pDzc4RhzgkATSYSIpABX8VVne4NExA08AVwIDAFmiMgQ3zKqepeqZqlqFvBX4DWf3cdq96nqpT7bfwc8pqqZwEHghgDrYEyrdsWoNKpqlP+u2BnuUIw5QaCJ5AFgHrBVVZeKSF9gSyPHjAFyVTVPVSuAOcC0BsrPAF5o6IQiIsBE4BXvpv8AlwUQvzGtXmb3zpyW3oVXlhXYfd1NixJoZ/vLqjpCVW/xvs5T1W80clgqkO/zusC77SQi0gvPJMePfDbHiEiOiCwRkdpk0Q0o9s5paeycN3qPzykqKmokVGNahytHpbFxTwnrdtmcEtNyBNrZniYir4tIoYjsFZFXRSStscP8bKvva9R04BVVrfbZlqGq2cA1wJ9FpN+pnFNVZ6pqtqpmJyUlNRKqMa3DJSN6EhXhYs7SHeEOxZjjAr20NQuYC/TE0wJ407utIQVAus/rNKC+W75Np85lLVXd5f03D1gAnA7sA7qISEQA5zSmzYmPjeSyrJ48u2QHzyzeHu5wjAECTyRJqjpLVau8j6eBxr7mLwUyvaOsovAki7l1C4nIQKArsNhnW1efRSITgbOA9eq5MDwfuMJb9FrgjQDrYEyb8MC0YUwenMyv3ljHkwtsDS4TfoEmkn0i8i0RcXsf3wL2N3SAtx/jNjyd9BuAl1R1nYg8ICK+o7BmAHP0xN7DwUCOiKzCkzgeUdX13n0/A34kIrl4+kz+FWAdjGkTYiLd/P1bo7j0tJ78/n+b+P3/NlrnuwkrCeQHUEQygL/hWSZFgc+AO1S1VVyozc7O1pycnHCHYUxQVdcov/zvGl74Ip9LTuvJj84fQJ/EjieUyT9QyvNf7GBM7wQmDEoOU6SmtRKRZd6+6gZFNFYAwJswfFsRiMgPgT83LTxjTHO5XcJvLh9OcucY/r5wK2+t3sXUYSncMr4f1TXKzE/yeHfNbmoUlmTst0RiHBNQi8TvgSI7VDUjyPE4wlokpq0rKinn34u28eziLykp94yO7xwTwTVjMyg6XM5ba3az9v6vERVhN0U1gQtqi6S+92jGscaYIErqHM3PpgzilvH9eGlpPpFuF98YlUan6AjeWr2L11bsZOOew4xIsxubtlab95bwvf/k8OQ3RzIsNT7c4ZygOV9PrHfPmBYmLiaS753Tl2vP7E2naM/3xKx0T/JYlV/c0KGmBVNVfv3GOnYcKOW5z1te13SDiURESkTksJ9HCZ45JcaYFi61SwcSO0WzwhJJq/Xu2j0szttPYqdo3l69i/Kq6sYPCqEGE4mqdlbVOD+PzqranMtixpgQERGy0uNZaYmkVTpWUc3Db29gcEocv79iOIfLqliwqWUt+2Q9b8a0A1npXcgrOsqhY5XhDsX4+GD9XopLKxos8/eFW9lZfIz7LxnCuZlJJHaKanErQFsiMaYdyErvCsDqAmuVtBS7io/xvdk5/OD55dTU+O9yzj9Qyj8WbuXS03oytm83ItwuLh7Rkw83FraoLwWWSIxpB4aneUb5rNxhiaSlqB38sCh3P/9etM1vmYfeXo9bhHunDjq+7fLTU6moquF/a3eHJM5AWCIxph2I7xBJv6SOrLIWSYuxquAQkW5hwsAkfv+/TWzc89WtAWpqlEfe3ci8dXu5bWJ/UuI7HN83Ii2ePokd+e+KlrNerSUSY9qJ09K7sDK/2NblaiFWFxQzqEccf7jyNOI6RHLnCyspq6ymrLKa219YwT8WbuWbYzO46dy+JxwnIlyWlcqSbfvZfehYmKI/kSUSY9qJ09O7sO9IBTuLW8Yfn/aspkZZU3CIEWnxdOsUzaNXjGDT3hLun7uOa/65hLfX7ObnUwfx0GXDiHCf/Gd6WlZPVGHuypbRKrFEYkw7UdvhbsOAw2/b/qOUlFdxmnelgQmDkvnOuF7MWZrPul2HefKbI7nx3H547i5+st6JHTk9owuvt5DRWzYXxJh2YmCPzkRFuFi5o5iLR9h84nCqHT03Iv2rpU7uvXAwkW4XF8/p8nIAABu4SURBVI9I4fSMro2e4/LTU7nvjXWM/c0HRLhcRLiFuJhI/nVtNslxMY7F7o8lEmPaiagIF8N6xlmHewuwKv8QHSLd9E/qdHxbhyg3v7p4SMDn+PrINL7cX8qRsiqqapTK6hreXL2L5z7fwV3nD3Ai7HpZIjGmHclK78rzX3xJZXUNkX6uvZvQWF1QzLDUOL/9H4HqFB1xUuI5dKySF5fmc/vE/s0696mynyRj2pHT0uMpq6xh896ScIfSblVW17BulzMrMV8zNoM9h8v4aGNh0M/dEEskxrQjp1uHe9ht3ltCeVUNI9KCvxT8pEHJdI+L5vkvQrtCsKOJRESmiMgmEckVkXv87H9MRFZ6H5tFpNi7PUtEFovIOhFZLSJX+xzztIhs8zkuy8k6GNOWpCd0oFvHKBbl7gt3KO3W6oJDAMdHbAVThNvF1dnpLNxcRP6B0qCfvz6OJRIRcQNPABcCQ4AZInLCBT1VvUtVs1Q1C/gr8Jp3VynwHVUdCkwB/iwivv/rd9cep6ornaqDMW2NiPCNUWnMW7eXgoOh+0NjvrK6oJj4DpH06hbryPmvHpOBAC8uzXfk/P442SIZA+Sqap6qVgBzgGkNlJ8BvACgqptVdYv3+S6gEEhyMFZj2o3rzuyNALMWbQ93KO3SqnzPRMT65og0V2qXDkwYmMyLOflUVtc48h51OZlIUgHflFjg3XYSEekF9AE+8rNvDBAFbPXZ/LD3ktdjIhJdzzlvFJEcEckpKmpZa/cbE049u3Tg4hEpzPliR4taQbY9KKusZtPeEkf6R3xdMzaDopJyPli/19H3qeVkIvGXbutb5Gc68IqqnnDbLxFJAZ4BrlfV2tR6LzAIGA0kAD/zd0JVnamq2aqanZRkjRljfH3vnL4crahmTog7Zdu7dbsOU12jjozY8jV+YDI942NC1unuZCIpANJ9XqcB9S0MMx3vZa1aIhIHvA38UlWX1G5X1d3qUQ7MwnMJzRhzCoalxnNW/27MWrSdiqrQXP4wX81od6Kj3ZfbJVw9OoNPtuzjy/1HHX0vcDaRLAUyRaSPiEThSRZz6xYSkYFAV2Cxz7Yo4HVgtqq+XKd8ivdfAS4D1jpWA2PasO+f05c9h8t4a3XLWPivPVhdcIjkztH0iHd+CZOrR6fz4/MH0Dkm0vH3ciyRqGoVcBswD9gAvKSq60TkARG51KfoDGCOnri29VXAucB1fob5Picia4A1QCLwkFN1MKYtO29AEgO6d2Lmx3m2tHyIrCoodvyyVq0e8THcPimThI5Rjr+Xo0ukqOo7wDt1tt1X5/X9fo57Fni2nnNODGKIxrRbIsL3zunLT19Zzae5+zgn0/oSnVRWWU1e0VEuPa3tLZhpM9uNacemZfWkW8co5oRwzkF7VVRSDkBPn7sdthWWSIxpx6Ij3Ewe3J2PNxVZp7vDCkvKAEiO8ztjoVWzRGJMOzdpcDIl5VUs3X4g3KG0aXsPe1okyZ1De6+QULBEYkw7d3ZmItERLt4P0eS19qrwsKdF0t1aJMaYtiY2KoKz+ify4ca9NnrLQYUl5US4hK6xzo+iCjVLJMYYJg/uTv6BY2wpPBLuUNqsvYfLSeocjcvlzBpb4WSJxBjDpMHJAHZ5y0GFJWUhv5d6qFgiMcbQPS6GEWnxfLjBEolTCg+Xk9y57fWPgCUSY4zXpEHdWZFfzL4j5eEOpU0qLClrkx3tYInEGOM1aXAyqoT8ft/tQXlVNQdLK9vk0F+wRGKM8RraM46U+JiQ3cOiPamd1W4tEmNMmyYiTBqczCdb9lFWWd34ASZghSVtdzIiWCIxxviYPLg7xyqrWZy3P9yhtCm1kxGTrLPdGNPWjevXjQ6RbhZYP0lQFR6/tGUtEmNMGxcd4Sa7d1drkQTZ3sNluF1CtxDcGyQcLJEYY04wrl83Nu89YsOAg6jwcDlJndrmrHawRGKMqWNc324ALLFWSdAUlpS3yeXja1kiMcacYHhqPJ2iI/hsqyWSYNl7uKzNjtgCSyTGmDoi3C5G9+7KEkskQVNkLZKmE5EpIrJJRHJF5B4/+x8TkZXex2YRKfbZd62IbPE+rvXZPkpE1njP+biItM2LjsaE0bh+3cjbd5S93mGrpukqqmrYf7Siza6zBQ4mEhFxA08AFwJDgBkiMsS3jKrepapZqpoF/BV4zXtsAvBrYCwwBvi1iHT1HvZ34EYg0/uY4lQdjGmvxvVNBGCxtUqarXbQQlsd+gvOtkjGALmqmqeqFcAcYFoD5WcAL3iffw14X1UPqOpB4H1gioikAHGqulg9d+CZDVzmXBWMaZ+G9IwjLibCEkkQ1LbqrEXSNKlAvs/rAu+2k4hIL6AP8FEjx6Z6nwdyzhtFJEdEcoqKippUAWPaK7dLGNOnm80nCYK2PhkRnE0k/vou6ruP53TgFVWtXeCnvmMDPqeqzlTVbFXNTkpKajRYY8yJxvXrxo4DpewsPhbuUELmifm53Pb8cuZvKqS6Jji3Hf5qna222yKJcPDcBUC6z+s0YFc9ZacDP6hz7Pg6xy7wbk8L8JzGmGaonU+yeOt+rhiV1kjp1m/Zlwd4dN4mIt3CW6t30yMuhitGpfGtM3rRI77prYnCw2W4BLp1aruJxMkWyVIgU0T6iEgUnmQxt24hERkIdAUW+2yeB1wgIl29newXAPNUdTdQIiJneEdrfQd4w8E6GNNuDerRma6xke2in6SiqoZ7X1tDapcOLP3FZP7+zZEMSunMkwtyuejxT1hTcKjJ5y48XE5ip2jcbXRWOziYSFS1CrgNT1LYALykqutE5AERudSn6AxgjrfzvPbYA8CDeJLRUuAB7zaAW4CngFxgK/CuU3Uwpj1zuYSxfbqxJG8/Pr+ebdLMj7eyee8RHrxsKF1io7hweApPXz+G9+46l5hIN9NnLubTLfuadO69JWVteg4JODyPRFXfUdUBqtpPVR/2brtPVef6lLlfVU+aY6Kq/1bV/t7HLJ/tOao6zHvO27St/4QbE0bj+nVjZ/Ex8g+03X6SvKIjPP5RLhcNT2HioO4n7Ouf3JnXbj2T9IRYrn/6C95afepX0gsPl9O9Dc9qB5vZboxpwLh+nn6Sz7Y27dt4S6eq/OL1tURHuPj1JUP8lukeF8OLN40jK70Lt7+wgleWFfgtV59Ca5EYY9qzzOROdI+LZuHmtjmE/tXlO1mct597LxxMcgPDc+M7RPLMDWMZ3TuBB99aT0lZZUDnr6qundVuLRJjTDslIkwY6Ln9bkVVTbjDCbr/fLadoT3jmD46vdGyMZFufnXREA4dq2TWou0BnX/fkQpUsRaJMaZ9mzAomSPlVeRsP9B44VaksKSMNTsPMXV4SsD3CRmeFs/5Q7rzz0/yOHSs8VZJ7ax26yMxxrRrZ/dPJMrt4qM2dvvdhZs8l+vGDzy1Cct3TR5ASVkV//p0W6Nlj09GtBaJMaY96xgdwdi+CXy0qW0lkgWbi0juHM2QlLhTOm5IzzimDu/Bvz/dRnFpRYNlj7dI2vDyKGCJxBgTgAkDk8krOsqX+4+GO5SgqKqu4ePNRYwfmERT7kRx56QBHK2o4p+f5DVYrrCkHBHa7L3aa1kiMcY0auKgZIA2c3lr+Y5iSsqqmDAwuUnHD+zRmYtH9GTWou0cOFp/q6SopIxuHaOJcLftP7Vtu3bGmKDondiRvkkd20wiWbCpkAiXcFZmYpPPceekTMoqq3n8wy31ltl7uJzubbx/BCyRGGMCNHFgMp/nHeBoeVW4Q2m2+ZuKGNWrK3ExkU0+R//kTlwzNoP/LN5e73pkhSVlbXrV31qWSIwxAZk4KJmK6hoW5bbuWe57DpWxYfdhJgxq2mUtXz+fOpje3Try45dWnjQcuLpG2XOorM13tIMlEmNMgLJ7J9ApOoL5rXz01sLNnvhPddivP7FRETx2dRZ7S8q57421x7eXlFXy/dk57DtSwaheXRs4Q9tgicQYE5CoCBfnZCYyf2NRq14NeP7GIlLiYxjYvXNQzpeV3oU7J2XyxspdzF21i/wDpVzx98Us3FzEg5cN48rsxmfNt3ZO3tjKGNPGTBiUzLtr97B+92GG9owPdzinrKKqhk9z93HJaT2bNOy3PreO78f8TYX88vU1RLpdVFbX8J/rx3B2MzrzWxNrkRhjAjZhYDIi8P76veEOpUmWfXmQI+VVQbms5SvC7eLPV2dRXaPEdYjk9R+c1W6SCFiLxBhzCpI6RzO2TwJzV+3izkmZQf1W77TK6hqe/fxLIt3CWf2D/0e+V7eOfPjj8cR1iCA2qn39abUWiTHmlEzLSiWv6Cjrdh0OdygB27G/lCv+sZi3V+/me+f0pVO0M3/oe8THtLskApZIjDGn6MJhPYh0C3NXnfrdAuuqrK7h3TW7HV2i/o2VO5n6+CfkFR3hiWtG8rMpgxx7r/bK0UQiIlNEZJOI5IrISbfT9Za5SkTWi8g6EXneu22CiKz0eZSJyGXefU+LyDaffVlO1sEYc6IusVGcNyCJuSt3UVPTvNFbry0v4JbnlvPTV1Y1+1z+PPLuRu6cs5KBPTrz7p3ncNGIlKC/h3Gwj0RE3MATwPlAAbBUROaq6nqfMpnAvcBZqnpQRJIBVHU+kOUtkwDkAu/5nP5uVX3FqdiNMQ27NCuVDzYU8sX2A5zRt1uTz7Modz9ul/DflbtIjovh51MHBy3Gpz7J4x8Lt3LN2AweuHRom1/vKpyc/J8dA+Sqap6qVgBzgGl1ynwfeEJVDwKoqr+ZTlcA76pqqYOxGmNOweTByXSIdPu9vFUdYMtCVfls634uHpHCd8b1YubHeTzVyGq6gZq7ahcPvb2BqcN78OC0YZZEHObk/24qkO/zusC7zdcAYICILBKRJSIyxc95pgMv1Nn2sIisFpHHRMTvQjYicqOI5IhITlFR27zftDHhEhsVwQVDu/NOnf6N99fvZcT985j58dZGz7F57xH2HSnnrH6J/PqSoUwd3oOH3t7AGyt3Niu2z7bu4ycvrWJM7wT+dFUW7gDvfmiazslE4u/Tq/tVJQLIBMYDM4CnRKTL8ROIpADDgXk+x9wLDAJGAwnAz/y9uarOVNVsVc1OSgrumHFjDEzL6klxaSWf5nq+qM3fWMitzy1DRPjNOxsbvYNg7ZpdZ/bvhtsl/OmqLMb2SeAnL69iZ/GxJsW0Mr+Ym2Yvo3diLP/8TjYxke4mncecGicTSQHguzZAGlC3HVwAvKGqlaq6DdiEJ7HUugp4XVWPr4amqrvVoxyYhecSmjEmxM7un0SX2EjeWLmLjzcXcdOzyxjYozML7x7PhcN68OBb65m9eHu9x3+2dR+9usWS1jUWgJhIN498YwSV1cr76/acUixlldU8Om8j3/j7Z3SOieDp68cQH9v0lX3NqXEykSwFMkWkj4hE4blENbdOmf8CEwBEJBHPpS7fi6QzqHNZy9tKQTwzoS4D1mKMCbmoCBdTh6cwb90evj87h76JHXnmu2Pp1imax2eczvlDunPfG+t4/vMdJx1bVV3D53kHOLPfiRMD+3jve/LhKdz3ZNmXB7no8U94Yv5WLj89lXfvPJeeXTo0u34mcI4lElWtAm7Dc1lqA/CSqq4TkQdE5FJvsXnAfhFZD8zHMxprP4CI9MbTollY59TPicgaYA2QCDzkVB2MMQ279LSelFXWkJEQy3PfG0tX7y1lI90u/nbN6UwclMzPX19z0tLza3YeoqS8irP6nzzia9Igz31PjjRy3xNV5ckFuVzxj88oq6zhP98dwx+uPM1aImEgrXkVz0BlZ2drTk5OuMMwps1RVd5du4exfRLo1unkcS9lldVM+uNCEjtH899bzzy+pMoT83N5dN4mlv1y8knHLd66nxn/XMI/vjWSKcP8z/tQVR7530b+b2Eel5zWk99+fbhjs9XbMxFZpqrZjZWzMXHGmCYTEaYOT/GbRMDT73H7xP6syi8+4Ta9i3L3MahHZ7/HZffuSlxMBB9s8H95q7pG+fnra/m/hXl8+4xe/OXqLEsiYWaJxBjjqG+MSiMjIZY/vreZmhqlrLKanC8P1rtwYqTbxXkDk5m/sfCk2e6V1TX88MWVvPDFDm4d348Hpg3FZcN7w84SiTHGUZFuFz+cnMn63YeZt24Py748SEVVjd/+kVqTByez/2gFKwuKT9j+6LxNvLlqFz+bMoifThnUqlYfbssskRhjHDctK5V+SR350/ub+WTLPiJcwpg+9SeS8wYk4XYJH/lc3lq/6zD/+nQbM8akc8v4fqEI2wTIEokxxnFul/DDyQPYUniEpz/bxmnpXRrs1+gSG8WoXl2PDwP29IusoUuHSFu9twWyRGKMCYmLhqcwqEdnyiprOKtf4ws9ThqUzIbdh9lZfIznv9jByvxifnnxYLrERoUgWnMqLJEYY0LC5RLu/tpARDz3fm/MpMHdAXjxix38/t2NnNW/G5dl1V2uz7QENmbOGBMykwZ3Z9kvzyehY+Otin5JHenVLZbHP8olKsLFg9OGWed6C2UtEmNMSAWSRMAzR2XSIE+r5Afj+9M3qZOTYZlmsBaJMabFuvbMXrhdcPP4vuEOxTTAEokxpsXq1a0jv7hoSLjDMI2wS1vGGGOaxRKJMcaYZrFEYowxplkskRhjjGkWSyTGGGOaxRKJMcaYZrFEYowxplkskRhjjGmWdnHPdhEpAr6sszkeONTINt/XjT1PBPY1MUR/sQS6v6GYG3td+zwUdWioTDA+C99t4fgsTrUOvq9D/fPUUJmW8lmEuw6+z1vqZxGK3+1eqprUaClVbZcPYGZj23xfN/YcyAlmLIHubyjmQOsUijo0VCYYn0WdbSH/LE61Dg3Ebp9FC6hDa/gsQvW7HcijPV/aejOAbW+e4vNgxhLo/oZibuz1m/WUaYpAzlFfmWB8FsGoQyDnCVYdfF+H+uepoTIt5bMIdx0CjaExTtYjVL/bjWoXl7ZCQURyVDU73HE0R1uoA7SNerSFOkDbqIfVoXHtuUUSbDPDHUAQtIU6QNuoR1uoA7SNelgdGmEtEmOMMc1iLRJjjDHNYonEGGNMs1gi8UNE/i0ihSKytgnHjhKRNSKSKyKPi89NpkXkdhHZJCLrROT3wY36pDiCXgcRuV9EdorISu9javAjPykWRz4L7/6fiIiKSGLwIvYbhxOfxYMistr7ObwnIj2DH/kJcThRh0dFZKO3Hq+LSJfgR35SLE7U40rv73SNiDjWod2c2Os537UissX7uNZne4O/N345Oba4tT6Ac4GRwNomHPsFMA4Q4F3gQu/2CcAHQLT3dXIrrMP9wE9a+2fh3ZcOzMMzUTWxtdUBiPMpcwfwj1ZYhwuACO/z3wG/a40/T8BgYCCwAMhuabF74+pdZ1sCkOf9t6v3edeG6tnQw1okfqjqx8AB320i0k9E/iciy0TkExEZVPc4EUnB8wu+WD2fyGzgMu/uW4BHVLXc+x6FrbAOIedgPR4Dfgo4PtrEiTqo6mGfoh1xuB4O1eE9Va3yFl0CpDlZB+97OlGPDaq6qaXGXo+vAe+r6gFVPQi8D0xp6u+/JZLAzQRuV9VRwE+AJ/2USQUKfF4XeLcBDADOEZHPRWShiIx2NFr/mlsHgNu8lyL+LSJdnQu1Qc2qh4hcCuxU1VVOB9qAZn8WIvKwiOQD3wTuczDW+gTj56nWd/F8+w2HYNYj1AKJ3Z9UIN/ndW19mlTPiADftF0TkU7AmcDLPpcLo/0V9bOt9ptiBJ4m5BnAaOAlEenrzfqOC1Id/g486H39IPBHPH8AQqa59RCRWOAXeC6rhEWQPgtU9RfAL0TkXuA24NdBDrVewaqD91y/AKqA54IZYyCCWY9Qayh2EbkeuNO7rT/wjohUANtU9XLqr0+T6mmJJDAuoFhVs3w3iogbWOZ9ORfPH1rf5nkasMv7vAB4zZs4vhCRGjwLqRU5GbiPZtdBVff6HPdP4C0nA65Hc+vRD+gDrPL+8qUBy0VkjKrucTj2WsH4efL1PPA2IUwkBKkO3k7ei4FJofpSVUewP4tQ8hs7gKrOAmYBiMgC4DpV3e5TpAAY7/M6DU9fSgFNqadTHUOt/QH0xqdTC/gMuNL7XIDT6jluKZ5WR21H1VTv9puBB7zPB+BpVkorq0OKT5m7gDmt8bOoU2Y7Dne2O/RZZPqUuR14pRXWYQqwHkgKxc+R0z9PONzZ3tTYqb+zfRueqyRdvc8TAqmn37hC+QG2lgfwArAbqMSToW/A8y32f8Aq7w//ffUcmw2sBbYCf+Or1QOigGe9+5YDE1thHZ4B1gCr8XxLS3GyDk7Vo06Z7Tg/asuJz+JV7/bVeBbmS22FdcjF84Vqpffh6MgzB+txufdc5cBeYF5Lih0/icS7/bvezyAXuP5Ufm/qPmyJFGOMMc1io7aMMcY0iyUSY4wxzWKJxBhjTLNYIjHGGNMslkiMMcY0iyUS0y6JyJEQv99TIjIkSOeqFs+qv2tF5M3GVs0VkS4icmsw3tsYf2z4r2mXROSIqnYK4vki9KsFCB3lG7uI/AfYrKoPN1C+N/CWqg4LRXym/bEWiTFeIpIkIq+KyFLv4yzv9jEi8pmIrPD+O9C7/ToReVlE3gTeE5HxIrJARF4Rz302nqu9l4N3e7b3+RHvgourRGSJiHT3bu/nfb1URB4IsNW0mK8Wo+wkIh+KyHLx3E9imrfMI0A/byvmUW/Zu73vs1pE/l8Q/xtNO2SJxJiv/AV4TFVHA98AnvJu3wicq6qn41ll9zc+x4wDrlXVid7XpwM/BIYAfYGz/LxPR2CJqp4GfAx83+f9/+J9/0bXN/KuBzUJzyoDAGXA5ao6Es/9b/7oTWT3AFtVNUtV7xaRC4BMYAyQBYwSkXMbez9j6mOLNhrzlcnAEJ+VVONEpDMQD/xHRDLxrIQa6XPM+6rqe4+IL1S1AEBEVuJZG+nTOu9TwVcLXi4Dzvc+H8dX9354HvhDPXF28Dn3Mjz3kgDP2ki/8SaFGjwtle5+jr/A+1jhfd0JT2L5uJ73M6ZBlkiM+YoLGKeqx3w3ishfgfmqerm3v2GBz+6jdc5R7vO8Gv+/Y5X6VedkfWUackxVs0QkHk9C+gHwOJ77kiQBo1S1UkS2AzF+jhfgt6r6f6f4vsb4ZZe2jPnKe3ju6wGAiNQuzx0P7PQ+v87B91+C55IawPTGCqvqITy32f2JiETiibPQm0QmAL28RUuAzj6HzgO+672fBSKSKiLJQaqDaYcskZj2KlZECnweP8LzRznb2wG9Hs/S/wC/B34rIosAt4Mx/RD4kYh8AaQAhxo7QFVX4Fn5dTqeG0Nli0gOntbJRm+Z/cAi73DhR1X1PTyXzhaLyBrgFU5MNMacEhv+a0wL4b174zFVVRGZDsxQ1WmNHWdMuFkfiTEtxyjgb96RVsWE+DbGxjSVtUiMMcY0i/WRGGOMaRZLJMYYY5rFEokxxphmsURijDGmWSyRGGOMaZb/D/eVrlWnF/XlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 3e-3\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.326372</td>\n",
       "      <td>0.253366</td>\n",
       "      <td>0.913953</td>\n",
       "      <td>0.903402</td>\n",
       "      <td>00:23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.310734</td>\n",
       "      <td>0.223429</td>\n",
       "      <td>0.896479</td>\n",
       "      <td>0.858856</td>\n",
       "      <td>00:23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas1_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.337300</td>\n",
       "      <td>0.209216</td>\n",
       "      <td>0.923220</td>\n",
       "      <td>0.919562</td>\n",
       "      <td>00:23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.312671</td>\n",
       "      <td>0.207063</td>\n",
       "      <td>0.915012</td>\n",
       "      <td>0.895087</td>\n",
       "      <td>00:23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas1_sp15_multifit');\n",
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas2_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.336217</td>\n",
       "      <td>0.211217</td>\n",
       "      <td>0.921631</td>\n",
       "      <td>0.912532</td>\n",
       "      <td>00:28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.280237</td>\n",
       "      <td>0.195936</td>\n",
       "      <td>0.919513</td>\n",
       "      <td>0.906492</td>\n",
       "      <td>00:27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas2_sp15_multifit');\n",
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(2, slice(lr/(2.6**4),lr), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas3_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.287673</td>\n",
       "      <td>0.190608</td>\n",
       "      <td>0.918983</td>\n",
       "      <td>0.901841</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.277706</td>\n",
       "      <td>0.189318</td>\n",
       "      <td>0.921631</td>\n",
       "      <td>0.905540</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas3_sp15_multifit');\n",
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas4_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.312985</td>\n",
       "      <td>0.193669</td>\n",
       "      <td>0.927985</td>\n",
       "      <td>0.916088</td>\n",
       "      <td>00:40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas4_sp15_multifit');\n",
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas5_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas5_sp15_multifit')\n",
    "learn_c.save(f'{lang}clas_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas_sp15_multifit');\n",
    "learn_c.to_fp32().export(f'{lang}_classifier_sp15_multifit')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.41 s, sys: 252 ms, total: 1.66 s\n",
      "Wall time: 1.65 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_multifit', bs=bs, num_workers=1)\n",
    "\n",
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3\n",
    "\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, pretrained=False, drop_mult=1.0, \n",
    "                                  metrics=[accuracy,f1])\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15_multifit');\n",
    "\n",
    "learn_c.load(f'{lang}clas_sp15_multifit');\n",
    "\n",
    "# put weight on cpu\n",
    "loss_weights = torch.FloatTensor(trn_weights).cpu()\n",
    "learn_c.loss_func = partial(F.cross_entropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAAEGCAYAAAAJw7AFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXzklEQVR4nO3deZxcZZ3v8c8PspCQBBKWRGAAlSTsBAIMIIQlsrshKptcFxRFVJBlYIRhG73DICoqwgAXMI4OAiKjApfFXDYVGAiERYmgLAphDUsW2ZL87h/nNFSadKe7k+pqnv68X696pc45T536Vbq6v3Wees5zIjORJKkUy7W6AEmSliWDTZJUFINNklQUg02SVBSDTZJUFINNklQUg01ahiJiSET8OiJejojLl2I/B0XE9cuytlaJiB0i4k+trkP9R3gem/qjiDgQOApYH5gDTAe+mZm/Xcr9Hgx8BdguM+cvdaF9XEQkMDYz/9zqWqQ2HrGp34mIo4CzgP8NjAbWBs4BPrwMdr8O8FB/CLWuiIgBra5B/Y/Bpn4lIlYCTgMOz8xfZOa8zHwjM3+dmcfWbQZHxFkRMbO+nRURg+ttO0XEExFxdEQ8GxFPRcRn6m2nAicB+0XE3Ig4JCJOiYifNDz/uhGRbX/wI+LTEfFIRMyJiEcj4qCG9b9teNx2EXFn3cV5Z0Rs17Dtpoj414j4Xb2f6yNi1Q5ef1v9/9RQ/0ciYq+IeCgiXoiIrze03zoibouIl+q2Z0fEoHrbLXWze+vXu1/D/o+LiKeBi9vW1Y95b/0cW9TLa0TE8xGx01L9YKUGBpv6m22BFYArO2lzArANMAHYDNgaOLFh+xhgJWBN4BDghxExMjNPpjoKvDQzh2XmhZ0VEhErAt8H9szM4cB2VF2i7duNAq6u264CfAe4OiJWaWh2IPAZYHVgEHBMJ089hur/YE2qIL4A+CQwEdgBOCki3lO3XQB8DViV6v9uMvAlgMycVLfZrH69lzbsfxTV0euhjU+cmX8BjgN+GhFDgYuBH2XmTZ3UK3WLwab+ZhXg+SV0FR4EnJaZz2bmc8CpwMEN29+ot7+RmdcAc4HxPaxnIbBxRAzJzKcy8w+LabM38HBm/mdmzs/MS4AZwAcb2lycmQ9l5ivAZVSh3JE3qL5PfAP4GVVofS8z59TP/wdgU4DMnJaZt9fP+xhwHrBjF17TyZn5Wl3PIjLzAuBh4A7gXVQfJKRlxmBTfzMLWHUJ3/2sATzesPx4ve7NfbQLxr8Dw7pbSGbOA/YDvgg8FRFXR8T6XainraY1G5af7kY9szJzQX2/LXieadj+StvjI2JcRFwVEU9HxGyqI9LFdnM2eC4zX11CmwuAjYEfZOZrS2grdYvBpv7mNuBV4COdtJlJ1Y3WZu16XU/MA4Y2LI9p3JiZ12XmrlRHLjOo/uAvqZ62mp7sYU3dcS5VXWMzcwTwdSCW8JhOh1pHxDCqwTsXAqfUXa3SMmOwqV/JzJepvlf6YT1oYmhEDIyIPSPijLrZJcCJEbFaPQjjJOAnHe1zCaYDkyJi7Xrgyj+3bYiI0RHxofq7tteoujQXLGYf1wDjIuLAiBgQEfsBGwJX9bCm7hgOzAbm1keTh7Xb/gzwnrc9qnPfA6Zl5ueovjv8j6WuUmpgsKnfyczvUJ3DdiLwHPA34MvAf9dNvgHcBdwH3A/cXa/ryXPdAFxa72sai4bRcsDRVEdkL1B9d/WlxexjFvCBuu0s4J+AD2Tm8z2pqZuOoRqYMofqaPLSdttPAabUoyY/saSdRcSHgT2oul+h+jls0TYaVFoWPEFbklQUj9i0iIi4qD6/6YGGdaMi4oaIeLj+d2S9fqV6+qh7I+IPbedz1dvOqNc9GBHfj4glfS8j9WkR8VhE3B8R0yPirob1X4mIP9Xv9zM624d6h8Gm9n5E1VXU6HhgamaOBabWywCHA3/MzM2AnYBvR8Sg+uTh91ENGd8Y2IolDxGX3gl2zswJmbklQETsTDVjzaaZuRFwZkurE2CwqZ3MvIXq+55GHwam1Pen8NaIwgSG10djw+rHza/Xr0B1ovBgYCCLDieXSnEYcHrbKQuZ+WyL6xFNDLZ66qAHI+KC+hD9+qhmPn9vRFwbEdMi4ta283bq9bfX0wWdFhFzm1Wbum10Zj4FUP+7er3+bGADqsEP9wNHZObCzLwNuBF4qr5dl5kP9n7Z0jKVwPX13662GVXGATtExB0RcXNEbNXC+lRr9hHbWOCH9SH6S8C+wPnAVzJzItWIq3Pqtt+jmv1gK3p+zpB61+5Uw9nXoJrp4uyIGBER61EF3lpUJxHvEhGTOt6N9I7wvszcAtgTOLx+Tw8ARlJNwXYscJnfJ7de00ZFRsS6wA319zJExHFUXVInAI3XZhqcmRtExCyqI4P5ETECmJmZb5s9of6kdCjAoBWGTlx17e6eQqMlmf/G67zw5OOsvu5YAJ559CFW/Yd3s/yAgSyY/wbP/+1RRr97HLOefIxhI1dj8NAVAXj+b48yYtXRvP7KPDKT4atUB3ZzZj0LEQwftVrLXlOJxgwb3OoS+q2ZM2ey/PLLMWfOHEaPHsPw4cMBeOCB+xk/fn0GDhzY4gr7h7vvnvZ8Zr7tD0uzLynROFXOAqpLhLyUmZ3NY9epzDyf6qiPNcdvkoef09lctuqJF59+gh+feCht/7f/97zTGTpiJDse8AVuvuQ8XpnzEnscehy/POskho1clcmf+ipzX3yes7/4EQ496xL+cvdt3HXNZXzq9Ashkx/98yFs99FPscG2k1v8yspy1I7rtbqEfmPevHksXLiQ4cOHM2/ePPbeY1e+fuJJPP7YYzz11ExOOuU0Hn7oIfbcfTJ3TJuOB229Y8jAaD/VHND8YGtvNvBoRHw8My+vD9k3zcx7gdupuiovBfbv5bpUu/SbR/LIvf/D319+kX/ff3smf+oIdtz/C1zyjSOYdu3lrLT6GhzwL98HYOdPHs4V3zqO739ub5Jkj88fy4orjWLjSXvwyPTb+MHn9waCcVtNMtT0jvbsM8+w38f2AWD+gvnst/+B7Lb7Hrz++ut84XOfZeKEjRk0cBD/56Iphlof0OyuyKsyc+N6+RiqkXNTqOafexdV1+TPMvO0iBhLNW1RUE2zc2hmrrmYXb/JIzb1Zx6xqb8bMjCmtZ160ahpR2z1JS42blhuPL+j/XlSUE3ouk1mZkTsTzWlkSRJ3dKXLts+kWpUXVCNoPxsi+uRJL0D9Zlgy8xbqa5WLElSjznziCSpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAabJKkoBpskqSgGmySpKAM62hARvwayo+2Z+aGmVCRJ0lLoMNiAM3utCkmSlpEOgy0zb+7NQiRJWhY6O2IDICLGAv8GbAis0LY+M9/TxLokSeqRrgweuRg4F5gP7Az8GPjPZhYlSVJPdSXYhmTmVCAy8/HMPAXYpbllSZLUM0vsigRejYjlgIcj4svAk8DqzS1LkqSe6coR25HAUOCrwETgYOBTzSxKkqSeWuIRW2beWd+dC3ymueVIkrR0ujIq8kYWc6J2Zvo9mySpz+nKd2zHNNxfAdiXaoSkJEl9Tle6Iqe1W/W7iPDkbUlSn9SVrshRDYvLUQ0gGdO0irphtRUH8bmt12l1GVJLjNzqy60uQeqTutIVOY3qO7ag6oJ8FDikmUVJktRTXQm2DTLz1cYVETG4SfVIkrRUunIe2+8Xs+62ZV2IJEnLQmfXYxsDrAkMiYjNqboiAUZQnbAtSVKf01lX5O7Ap4G1gG/zVrDNBr7e3LIkSeqZzq7HNgWYEhH7ZuYVvViTJEk91pXv2CZGxMptCxExMiK+0cSaJEnqsa4E256Z+VLbQma+COzVvJIkSeq5rgTb8o3D+yNiCOBwf0lSn9SV89h+AkyNiIvr5c8AU5pXkiRJPdeVuSLPiIj7gPdTjYy8FnAeK0lSn9SVrkiAp4GFVDP7TwYebFpFkiQthc5O0B4H7A8cAMwCLgUiM3fupdokSeq2zroiZwC3Ah/MzD8DRMTXeqUqSZJ6qLOuyH2puiBvjIgLImIyb80+IklSn9RhsGXmlZm5H7A+cBPwNWB0RJwbEbv1Un2SJHXLEgePZOa8zPxpZn6Aat7I6cDxTa9MkqQe6OqoSAAy84XMPC8zd2lWQZIkLY1uBZskSX2dwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwabFevKJv7HP3ruy/ZabMGnrzTj/nB8A8Ksrf86krTdjzEqDmX73tEUe84cH7mOvyTswaevN2HGbzXn11VdbUbrUY2uNXplrz/8q91xxItN+fgKHH7DTItuPPHgyr9xzNqusvCIA49YdzU1TjualO77LkQdPfrPd2HVW5/afHf/m7Zlbv8WXD1x0X2qeAa0uQH3TgAEDOPWbZ7DphM2ZO2cOu076R3bcZTLrb7gRF/30Mo494vBF2s+fP5/DP/9pfnj+xWy0yWa8MGsWAwcObFH1Us/MX7CQ47/zC6bPeIJhQwfz+/86jql3zGDGI0+z1uiV2WWb9fnrUy+82f7Fl+dx9L9fzgd33myR/Tz8+LNss//pACy3XPCX677Jr268t1dfS3/mEZsWa/SYd7HphM0BGDZ8OGPHr8/TM2cybvwGrDd2/Nva3zT1BjbcaBM22qT6BR+1yiosv/zyvVqztLSefn4202c8AcDcv7/GjEefZo3VVgbgjGP25YTv/TeZ+Wb7516cy7Q//pU35i/ocJ87bz2eR594jr8+9WJzi9ebmhpsEbFuRMyIiCkRcV9E/DwihkbE5Ii4JyLuj4iLImJw3f70iPhj3fbMZtamrvvr44/xwH33ssWWW3fY5i9/fpiIYL+P7M37d9ias8/yx6d3trXfNYoJ49fizgceY+8dN2Hmsy9x/0NPdns/H999IpddO23JDbXM9MYR23jg/MzcFJgNHAX8CNgvMzeh6g49LCJGAfsAG9Vtv9ELtWkJ5s2dyyEH78e/nn4mw0eM6LDdggXzueP233POhVP41XU3cc2vf8ktN/2/XqxUWnZWHDKIS878HMeeeQXzFyzguEN257Rzr+72fgYOWJ69d9yEX9xwTxOqVEei8bB6me88Yl3glsxcu17eBfgXYPnMnFSvmwwcDnwCmAbcBVwNXJWZry9mn4cCh9aL44E/Ne0FKID1qD6QPNNu23hgDjCzXh4JrAQ8Vi+/C1i4mMdJfdqgQYNi6tSp6/3mN7+Zfeqppz6z1VZbDbn22mvHvfLKKwsBRo8ePei55557fcstt3x25syZzwB8+9vfXmPu3LkLTj755EXe7wcddNDKhx122Grbb7/9w614Lf3AOpm5WvuVvRFsN2fmOvVyh8GWmR+tuyQnA/sDa2XmLk0rTp2KiACmAC9k5pGL2X4TsHpmblgvjwSmAtsDrwPXAt/NzO5/zJVa5833PfC2933tMWDLiLg2M7es150CzAXa98H/DLgOuHiZV6oO9UZX5NoRsW19/wDgN8C6EbFeve5g4OaIGAaslJnXUL2hJvRCberY+6h+NrtExPT6tldE7BMRTwDbAmMj4jqAzHwR+A5wJzAduNtQ0zvQm+97qvfxdGCvTtqPAZ6g+orlxPp+W5/9UGBX4BfNKlaL1xtHbNcAtwDbAQ9TvWm2pfpkM4DqD+FhwCjgl8AKVJ+azszMKU0rTkstIu5q+MQq9Su+//uu3jiPbWFmfrHduqnA5u3WPQV0POxOfdH5rS5AaiHf/31UbxyxXZWZGzftSSRJatDUYJMkqbc584gkqSgGmySpKAabJKkoBpu6JSLmRMTsdre/RcSVEfGeVtcnNVNEnBERIyJiYERMjYjnI+KTra5LizLY1F3fAY4F1gTWAo4BLqCaYeGiFtYl9YbdMnM28AGqk7HHUf0+qA8x2NRde2TmeZk5JzNnZ+b5wF6ZeSnVfJFSydouMrgXcElmvtBZY7WGwabuWhgRn4iI5erbJxq2ee6ISvfriJgBbAlMjYjVAC8V38d4Hpu6pf4e7XtU06IlcDvwNeBJYGJm/raF5UlNV0/4PTszF0TEUGBEZj7d6rr0FoNNkrooIgZSzW07qV51M/AfmflG66pSe3ZFqlsiYlw9GuyBennTiDix1XVJveRcYCJwTn3bol6nPsQjNnVLRNxMNQrsvMzcvF73gPOBqj+IiHszc7MlrVNrecSm7hqamf/Tbt38llQi9b4FEfHetoX6O+cFLaxHi9Ebl61RWZ6vf7ETICI+RnXJIak/OBa4MSIeqZfXBT7TunK0OHZFqlvqT6jnU1049kXgUeCgzHy8pYVJvSAiVgCOBibXq24AvpuZDvnvQww2dUtEDAY+RvVJdRQwG8jMPK2VdUm9ISIuo3rP/7RedQAwMjM/3rqq1J5dkequXwIvAXcDM1tci9TbxrcbKHJjRNzbsmq0WAabumutzNyj1UVILXJPRGyTmbcDRMQ/Ar9rcU1qx65IdUtEnA/8IDPvb3UtUm+LiAeB8cBf61VrAw8CC6m65DdtVW16i8GmbomIPwLrUQ0aeQ0I/IVWPxER63S23UFUfYPBpm7p6BfbX2hJfYXBJkkqijOPSJKKYrBJkopisElNFhELImJ6RDwQEZfX1/Dq6b52ioir6vsfiojjO2m7ckR8qQfPcUpEHNPTGqVWM9ik5nslMyfUV0B4Hfhi48aodPt3MTN/lZmnd9JkZaDbwSa90xlsUu+6FVgvItaNiAcj4hyqWVz+ISJ2i4jbIuLu+shuGEBE7BERMyLit8BH23YUEZ+OiLPr+6Mj4sqIuLe+bQecDry3Plr8Vt3u2Ii4MyLui4hTG/Z1QkT8KSJ+Q3WelvSOZbBJvSQiBgB7Am0nt48Hflxf124ecCLw/szcArgLOKqedPcC4IPADsCYDnb/feDmerqnLYA/AMcDf6mPFo+NiN2AscDWwARgYkRMioiJwP7A5lTBudUyfulSr3JKLan5hkTE9Pr+rcCFwBrA421TMwHbABsCv4sIgEHAbcD6wKOZ+TBARPwEOHQxz7EL8L8AMnMB8HJEjGzXZrf6dk+9PIwq6IYDV2bm3+vn+NVSvVqpxQw2qfleycwJjSvq8JrXuAq4ITMPaNduAvW175aBAP4tM89r9xxHLsPnkFrOrkipb7gdeF9ErAcQEUMjYhwwA3h3w1WbD+jg8VOBw+rHLh8RI4A5VEdjba4DPtvw3d2aEbE6cAuwT0QMiYjhVN2e0juWwSb1AZn5HPBp4JKIuI8q6NavL2B5KHB1PXiko6nLjgB2joj7gWnARpk5i6pr84GI+FZmXg/8F3Bb3e7nwPDMvBu4FJgOXEHVXSq9YzmlliSpKB6xSZKKYrBJkopisEmSimKwSZKKYrBJkopisEmSimKwSZKKYrBJkory/wHKQHmV1JmzVQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds,y,losses = learn_c.get_preds(with_loss=True)\n",
    "predictions = np.argmax(preds, axis = 1)\n",
    "\n",
    "interp = ClassificationInterpretation(learn_c, preds, y, losses)\n",
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1088   56]\n",
      " [ 216 2417]]\n",
      "accuracy global: 0.9279851734180566\n",
      "accuracy on negative reviews: 95.1048951048951\n",
      "accuracy on positive reviews: 91.79642992783896\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(np.array(y), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1])/(cm[0,0]+cm[0,1]+cm[1,0]+cm[1,1])}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on negative reviews: {cm[0,0]/(cm[0,0]+cm[0,1])*100}') \n",
    "print(f'accuracy on positive reviews: {cm[1,1]/(cm[1,0]+cm[1,1])*100}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁o ▁xxmaj ▁produto ▁xxmaj ▁relógio ▁xxmaj ▁invi cta ▁xxmaj ▁pro ▁xxmaj ▁di ver ▁ - ▁2 19 5 4 , ▁xxup ▁veio ▁xxup ▁trocado ▁xxup ▁pelo ▁xxup ▁invi cta ▁xxup ▁pro ▁xxup ▁di ver ▁2 400 1 , ▁xxup ▁tudo ▁xxup ▁bem ▁xxup ▁produto ▁xxup ▁similar , ▁xxup ▁mas ▁o ▁xxup ▁produto ▁xxmaj ▁relógio ▁xxmaj ▁invi cta ▁xxmaj ▁ s pe ci al t y ▁xxup ▁2 14 39</td>\n",
       "      <td>neg</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁pedido : ▁02-66 2 11 99 05 ▁xxmaj ▁alça ▁xxmaj ▁dupla ▁xxmaj ▁x - do u b le ▁xxmaj ▁couro ▁xxmaj ▁ le gi timo ▁xxmaj ▁camera s ▁xxmaj ▁d s l r ▁xxmaj ▁do u b le ▁xxmaj ▁ s t ra p ▁xxup ▁informo ▁xxup ▁que ▁o ▁xxup ▁produto ▁xxup ▁não ▁xxup ▁foi ▁xxup ▁entregue , ▁xxup ▁porém ▁xxup ▁no ▁xxup ▁site ▁xxup ▁está ▁xxup ▁sendo</td>\n",
       "      <td>neg</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxup ▁ola ! ▁xxup ▁bom ▁xxup ▁dia ▁ . ▁o ▁xxup ▁produto ▁xxup ▁foi ▁xxup ▁cobrado ▁xxup ▁um ▁xxup ▁alto ▁xxup ▁frete ▁e ▁xxup ▁não ▁xxup ▁foi ▁xxup ▁entregue ▁xxup ▁no ▁xxup ▁meu ▁xxup ▁endereço , tive ▁xxup ▁que ▁xxup ▁retirar ▁xxup ▁na ▁xxup ▁agência ▁xxup ▁do ▁xxup ▁correios ▁xxup ▁ , pois ▁xxup ▁vcs ▁xxup ▁deveria ▁xxup ▁devolver ▁o ▁xxup ▁valor ▁xxup ▁do ▁xxup ▁frete ▁xxup ▁ ,</td>\n",
       "      <td>pos</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxup ▁desde ▁o ▁xxup ▁dia ▁23 - 04 - 18 ▁xxup ▁era ▁xxup ▁pra ▁xxup ▁ser ▁xxup ▁entregue ▁xxup ▁eles ▁xxup ▁nao ▁xxup ▁com pr ira o ▁a ▁xxup ▁entraga r ▁xxup ▁do ▁xxup ▁produto ▁xxup ▁to ▁xxup ▁mt ▁xxup ▁triste ▁xxup ▁com ▁xxup ▁essa ▁xxup ▁empresa ▁25 - 04 - 18 ▁xxup ▁as ▁17 , 20 ▁xxup ▁nao ▁xxup ▁chegou ▁xxup ▁aguardo ▁xxup ▁contato ▁xxup ▁te l ▁</td>\n",
       "      <td>neg</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxup ▁foi ▁xxup ▁feito ▁o ▁xxup ▁pedido ▁xxup ▁de ▁2 ▁xxup ▁( u nd ) ▁xxup ▁do ▁xxup ▁produto , ▁xxup ▁chegou ▁xxup ▁apenas ▁xxup ▁1 ( u nd ), ▁e ▁xxup ▁mesmo ▁xxup ▁assim ▁xxup ▁fora ▁xxup ▁do ▁xxup ▁prazo ▁xxup ▁de ▁xxup ▁entrega , ▁xxup ▁já ▁xxup ▁enviei ▁xxup ▁vários ▁xxup ▁emails ▁xxup ▁para ▁xxup ▁resolve ▁o ▁xxup ▁problema ▁e ▁xxup ▁ate ▁xxup ▁agora ▁xxup ▁não ▁xxup</td>\n",
       "      <td>neg</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions some random sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "neg tensor([0.9429, 0.0571])\n"
     ]
    }
   ],
   "source": [
    "# Get the prediction neg/pos\n",
    "review = 'Eu não gosto deste produto'\n",
    "pred = learn_c.predict(review)\n",
    "print(pred[0], pred[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.043\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁xxbos</span> <span title=\"0.131\" style=\"background-color: rgba(238, 236, 244, 0.5);\">▁xxmaj</span> <span title=\"0.307\" style=\"background-color: rgba(204, 205, 228, 0.5);\">▁eu</span> <span title=\"0.567\" style=\"background-color: rgba(141, 138, 192, 0.5);\">▁não</span> <span title=\"1.000\" style=\"background-color: rgba(63, 0, 125, 0.5);\">▁gosto</span> <span title=\"0.420\" style=\"background-color: rgba(177, 176, 212, 0.5);\">▁deste</span> <span title=\"0.197\" style=\"background-color: rgba(227, 226, 239, 0.5);\">▁produto</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The darker the word-shading in the below example, the more it contributes to the classification. \n",
    "txt_ci = TextClassificationInterpretation.from_learner(learn_c)\n",
    "test_text = \"Eu não gosto deste produto\"\n",
    "txt_ci.show_intrinsic_attention(test_text,cmap=cm.Purples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.0432, 0.1314, 0.3068, 0.5668, 1.0000, 0.4200, 0.1972],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt_ci.intrinsic_attention(test_text)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Text</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁tudo ▁dentro ▁do ▁prazo , ▁muito ▁bem ▁embalado ▁e ▁delicado ... ▁xxmaj ▁ótimo .</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "      <td>6.29</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁super ▁recomendo , ▁entrega ▁rápida ▁como ▁sempre .</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "      <td>5.13</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁comprei ▁na ▁comodidade ▁de ▁internet ▁porém ▁não ▁entregaram ▁na ▁minha ▁casa . tive ▁que ▁buscar ▁no ▁correio , odiei ▁isso , não ▁recomendo ▁pq ▁não ▁tive ▁a ▁comodidade ▁de ▁receber ▁em ▁casa . apesar ▁de ▁pagar ▁frete . &lt;unk&gt;</td>\n",
       "      <td>pos</td>\n",
       "      <td>neg</td>\n",
       "      <td>4.74</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁cancelei ▁o ▁produto ▁e ▁não ▁recebi ▁o ▁voucher ▁de ▁reembolso .</td>\n",
       "      <td>pos</td>\n",
       "      <td>neg</td>\n",
       "      <td>4.68</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁adoro ▁a ▁lannister !</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "      <td>4.55</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# tabulation showing the first k texts in top_losses along with their prediction, actual,loss, and probability of actual class.\n",
    "# max_len is the maximum number of tokens displayed. If max_len=None, it will display all tokens.\n",
    "txt_ci.show_top_losses(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Fine-tuning \"backward Classifier\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # \"error\", \"ignore\", \"always\", \"default\", \"module\" or \"on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 196 ms, sys: 68 ms, total: 264 ms\n",
      "Wall time: 262 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_aws_sp15_multifit_bwd', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# # keep reviews with a label not nan\n",
    "# df2_trn_val = df_trn_val[df_trn_val[label].notnull()].copy()\n",
    "# print(len(df2_trn_val))\n",
    "# df2_trn_val.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8.22 s, sys: 344 ms, total: 8.56 s\n",
      "Wall time: 6.45 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = (TextList.from_df(df_trn_val, path, vocab=data_lm.vocab, cols=reviews, \n",
    "                              processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_from_df(cols=label)\n",
    "    .databunch(bs=bs, num_workers=1, backwards=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.27 s, sys: 60 ms, total: 1.33 s\n",
      "Wall time: 864 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas.save(f'{lang}_textlist_class_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Get weights to penalize loss function of the majority class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.14 s, sys: 24 ms, total: 1.16 s\n",
      "Wall time: 1.07 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_multifit_bwd', bs=bs, num_workers=1, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(33993, 3777, 37770)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_trn = len(data_clas.train_ds.x)\n",
    "num_val = len(data_clas.valid_ds.x)\n",
    "num_trn, num_val, num_trn+num_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([10126, 23867]), array([1144, 2633]))"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_LabelCounts = np.unique(data_clas.train_ds.y.items, return_counts=True)[1]\n",
    "val_LabelCounts = np.unique(data_clas.valid_ds.y.items, return_counts=True)[1]\n",
    "trn_LabelCounts, val_LabelCounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.7021151413526314, 0.2978848586473686],\n",
       " [0.6971141117288854, 0.3028858882711146])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_weights = [1 - count/num_trn for count in trn_LabelCounts]\n",
    "val_weights = [1 - count/num_val for count in val_LabelCounts]\n",
    "trn_weights, val_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Training (Loss = FlattenedLoss of weighted CrossEntropyLoss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.29 s, sys: 36 ms, total: 1.32 s\n",
      "Wall time: 1.15 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_multifit_bwd', bs=bs, num_workers=1, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, drop_mult=1.0, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15_multifit_bwd');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "loss_weights = torch.FloatTensor(trn_weights).cuda()\n",
    "learn_c.loss_func = partial(F.cross_entropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "functools.partial(<function cross_entropy at 0x7f87fbb73170>, weight=tensor([0.7021, 0.2979], device='cuda:0'))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3zU9f3A8dc7e5A9GEnIYO9gwhLFVRFxFxeOqmjRtlrraNVatT9brbWttg7qqlq1ioi2VUEcFVAEhACyh0kYCUlIQhbZ4z6/P+4Cl3AJAe5yd8n7+XjcI3ffde9Lcve+zxZjDEoppVR7Pu4OQCmllGfSBKGUUsohTRBKKaUc0gShlFLKIU0QSimlHPJzdwDOEhsba1JSUtwdhlJKeZV169aVGmPiHO3rMQkiJSWFrKwsd4ehlFJeRUT2drRPq5iUUko5pAlCKaWUQ5oglFJKOeTSBCEiM0Rkp4hki8j9DvYni8j/RGSTiCwTkUS7fTeIyPe22w2ujFMppdTRXJYgRMQXeB44HxgJzBaRke0O+zPwhjFmLPAo8AfbudHAI8AkYCLwiIhEuSpWpZRSR3NlCWIikG2MyTXGNALzgUvaHTMS+J/t/lK7/ecBnxtjyowx5cDnwAwXxqqUUqodVyaIBCDP7nG+bZu9jcAs2/3LgDARieniuUoppVzIlQlCHGxrP7f4vcAZIrIBOAPYDzR38VxEZK6IZIlIVklJycnGq5RSXmfhunzeWbPPJdd2ZYLIB5LsHicCBfYHGGMKjDE/NMaMBx60bavsyrm2Y18yxmQaYzLj4hwOBFRKqR5tQVYe/96w3yXXdmWCWAsMEZFUEQkArgY+tD9ARGJFpDWGB4BXbfc/BaaLSJStcXq6bZtSSik7BRV1JEQGu+TaLksQxphm4HasH+zbgQXGmK0i8qiIXGw77Exgp4jsAvoCj9nOLQN+hzXJrAUetW1TSill02IxFFXWMyAyyCXXd+lcTMaYxcDidtsetru/EFjYwbmvcqREoZRSqp3iQ/U0WwwDvK0EoZRSyrUKKuoANEEopZRqa39FPYD3tUEopZRyrdYSRP8I17RBaIJQSikvVVBRR3iQH2FB/i65viYIpZTyUgUVdS5rfwBNEEop5bX2V9S7rP0BNEEopZTXKqioIyFKE4RSSik71Q3NVNY1aRWTUkqptgpdPAYCNEEopZRX2m9LEAkummYDNEEopZRXKrANktMShFJKqTb2V9Ti6yPEh2kJQimllJ2Cinr6hQfh6+NofTXn0AShlFJeaL8L14FopQlCKaW8kHUUteuql0AThFJKeZ0jCwVpCUIppZSdkkMNLl0oqJUmCKWU8jJHxkBoglBKKWXH1SvJtdIEoZRSXuZIgtBGaqWUUnYKKuoIc+FCQa00QSillJdx9ToQrTRBKKWUl3H1SnKtNEEopZSX6Y5R1KAJQimlvEp3LBTUShOEUkp5kcJu6sEEmiCUUsqrdNcgOdAEoZRSXqU7FgpqpQlCKaW8SEFFnW2hoECXP5cmCKWU8iIFlXXEhwXi5+v6j29NEEop5UUOVNXTN9z1DdSgCUIppbxKUWU9/SM0QSillGrnQFWDliCUUkq1Vd3QTHVDM/20BKGUUspeUaW1i2s/LUEopZSyd6DKmiC0ikkppVQbh0sQWsWklFLKXlGVVjEppZRyoKiynvAgP4IDfLvl+VyaIERkhojsFJFsEbnfwf6BIrJURDaIyCYRmWnbniIidSLyne32givjVEopb1BUVd9t1UsAfq66sIj4As8D5wL5wFoR+dAYs83usN8AC4wxfxeRkcBiIMW2L8cYk+6q+JRSytt05yhqcG0JYiKQbYzJNcY0AvOBS9odY4Bw2/0IoMCF8SillFcrqqzvtvYHcG2CSADy7B7n27bZ+y1wnYjkYy093GG3L9VW9bRcRE53YZxKKeXxmlsslFY3dNs0G+DaBCEOtpl2j2cDrxtjEoGZwJsi4gMUAgONMeOBu4G3RSS83bmIyFwRyRKRrJKSEieHr5RSnqOkugGLgb49JEHkA0l2jxM5ugrpZmABgDFmFRAExBpjGowxB23b1wE5wND2T2CMeckYk2mMyYyLi3PBS1BKKc/Q3aOowbUJYi0wRERSRSQAuBr4sN0x+4BzAERkBNYEUSIicbZGbkQkDRgC5LowVqWU8mjdPYoaXNiLyRjTLCK3A58CvsCrxpitIvIokGWM+RC4B3hZRO7CWv10ozHGiMg04FERaQZagNuMMWWuilUppTxdd4+iBhcmCABjzGKsjc/22x62u78NmOrgvPeB910Zm1JKeZPCqnr8fYXokIBue04dSa2UUl7gQGU98WFB+Pg46v/jGpoglFLKC3T3KGrQBKGUUl7hQFVDt/ZgAk0QSinl8YwxFFV27zQboAmixzDG8MaqPWwvrHJ3KEopJ6uqb6auqYV+EYHd+ryaIE7S80uzufXNLBqaW9waxzfZB3n4v1u57pVvySurdWssSinnah0D0S8iuFufVxPESTDG8NbqvXy69QC//mALxrSfSaT74nj6i13EhwXS1GLh5n+upaq+yS2xKKWczx2jqEETxDEt2VLI9KeXU1HbeNS+7OJqCivrGZ0Qzvvr85m3LMcNEcLX35eybm85d5wzhBeuyyC3pIbb395Ac4vFLfEopZyru1eSa6UJohPZxYe4e8FGdh2o5rOtB47av3yXdYLAv1+bwWXjE/jTpzv5aGP3zljeWnoYEBHElZmJnDo4lt9fOpqvdpXwfx9tc1upRinlPAdsJYj4cG2D8AjVDc3c+uY6QgJ86RcexCdbCo865qvvS0mLCyUpOoQnZo1hQkoU97y3kfX7yrstzuW7Stiwr4KfnT2YQD/rMoRXTxzIrdPSeHP1Xj7ZUtRtsSilXKOwqp6oEH+C/LtnqdFWmiAcMMZw//ub2F1awzOzx3PRuP6syC5tU69f39TCt7kHmTbEOotsoJ8vL16fSUxoAH/5bKdL4npxeQ5zXl/L2j1lh+P86xffkxAZzBUZSW2O/dWM4aTFhTJvWbaWIpTycgfc0MUVNEE49PrKPXy8qZBfnjecUwfFMmN0P5paDEt3FB8+Zs3uMhqaLZwx9Mg049GhAZw3qh/r9pbT2Ozc+n9jDK99s4cvdxRzxQuruPaV1fx9eQ7f5VXws7MGE+DX9k/p6yPccloaW/ZXsTpX5zlUypu5YxQ1aII4yt6DNTy2aDvnjuzLbWekATA+KYq+4YF8svlIdc1Xu0oI8PVhUlp0m/MnpUZT32RhS0GlU+PKLa2hqKqehy8cyW8uGMHOokM8uWQnCZHBXJ6R6PCcH56SQExoAK98rTOlK+XNDlR171KjrVw6m6s3+i6vgmaL4d7pwxCxTorl4yOcN6ofC7LyqG1sJiTAj6++L2FCahQhAW1/hRNSrQljze4yThkY5bS4VmaXAnDOiHiSY0K5dlIy76/PZ0T/8KNKD62C/H350ZQUnv5iF9nFhxgcH+a0eJRS3aOx2UJpdaNWMXmCnOJqfH2ElNiQNttnjOpHfZOF5TtLKKysY9eB6sPtD/Zi+wQyKC6UNbudW63zTfZBEiKDGRhtjSs4wJfrJieTkdx5Erp+SjKBfj688vVup8ajlOoexYe6fx2IVpog2skprSEpKvhwj6BWE1OjiQrx55MtRXy9y/ptftpQx8ucTkyNYe2eMloszmkcbrEYVuUeZOrgmMOlmq6KDg3gisxEPli///A/mlLKexwZRa0Jwu1yiqsZFNfnqO1+vj5MH9mPL3cU8/n2A8SHBTK8n+Mqm0mp0Ryqb2Zn0SGnxLStoIrKuiamDo49ofNvPi2NJouFN1ftdUo8SqnuU1TZAHT/IDnQBNFGi8Wwu7SGQfFHJwiAGWP6Ud3QzOfbDnD6kLgOv81PPNwOcdApcX2TYy2xTBkUc0Lnp8aGMn1kX95cvZeahmanxKSU6h7uWIu6lSYIOwUVdTQ0WxgUF+pw/6mDYggLtDZKTxva8bf5AZHBJEQGs2aPc9ohvskuZWjfPsSHnfg/yE/OHExVXRO/WrhJx0Uo5UXKaxsRgYhg/25/bk0QdrJLqgEcVjGBdTDcOSPiEYHTHTRQ25uUGs2a3WUn/WHc0NzC2j1lnDroxKqXWqUnRXLfjOEs2lzotjmjlFLHr7y2kchgf3y7canRVpog7OQUd54gAO47fziv/CiT6NDOFw6fmBpNaXUjuaU1JxXThn0V1DdZTrj9wd7caWlckj6AP3+2k/9tP3puKaWU5ymvaSIqpPPPG1fRBGEnp6SG6NAAojr58O8fEcw5I/oe81oT7cZDnIyV2aX4yJHrnQwR4Y+zxjJqQDh3zv+ObFtCVEp5rvLaxk4/k1xJE4SdnJLqDtsfjldqbCixfQJPOkF8k3OQMYmRTqt/DPK3zhkV5O/Dj9/IorCyzinXVUq5RllNI1Eh3d/+AJog2sgtcdzF9USIyOF2CHu1jc1Yujg+orqhmY15FUw9wd5LHUmIDObF6zMoOdTArHkryS52TndcpZTzVdS6r4pJp9qwqahtpLS60WkJAqzVQos2F5JXVkteWS2vr9zDF9sPEBrgx4gB4YwaEE56UiQXjOmPn+/RuXrN7oM0W4xT2h/ay0iO5t1bJ3Pja2uZ9fdVvHpjJhnJJ1+NpZRyHmOMW6uYNEHY5JRYG5PTnFTFBDAhxfqBe8nz3xwuJs6ZmkpDs3Uyv3fW7OO1b/bw2dYDPH1Veps5larqm5i3NIcgf59jTqdxokYNiOCDn5zKj15dwzUvf8tvLhhBYnQIQX6+BPn7MLRvGKGB+i+ilLvUNbXQ0GzREoS75R6ji+uJGN4vjMHxfQjy9+GB84dz0bgBbRb8aLEY/rEil8cX76C+qYXnrz2FIH9fDlTVc8Ora8gpqeYvV6a7dJGQpOgQFt42hTmvr+Wh/25tsy8lJoT/3n6aW/pfK6WgvNa6Bo272iA0QdjklNQQ4OtDYlSw067p4yN8cfcZHe739RHmThtEcIAfD/1nC7f8M4v7zx/OrW+uo6K2kVdvnHDM8RbOENMnkPd/cirfF1dT19RCfVML+8vreOCDzfzyvY28eH3Gcc8BpZQ6eeU1jQBaxeRuOSXVpMSGOGwLcLXrJycT7O/LrxZu5MJnVxDbJ4D5c6cwJjGi22Lw8/VhRP/wNtsq65r4/aLtvPL1bn48Le3w9vqmFhauy2diajRD++oU4kq5SnmtLUFoFZN75ZRUM8yNH3aXZyQSEuDLO2v28ftLR5Mc47y2kBN182mprNtbzhNLdjAuKZKJqdF8ueMAj3y4lbyyOvx9hZ+cOZifnTXoqNlvlVInr8xWgogO1Somt2lqsbDvYC0zR/d3axwzx/Rn5hj3xmBPRHjy8rHseO4bbn97PeOSIvl82wEGxYXy0vUZfLKliGf+9z2fbC7kiVljXdaYrlRvVWFrg4jUkdTus/dgLc0Ww6B4939r9zRhQf7Mu/YUquqbWPF9KffNGM4nd05j+qh+PH1VOq/dOIGahmYuf2Elf/p0B00tzl2LW6nerLUEEemmjiJagsBavQTO7cHUk4zoH87Hd5xGn0D/oxYtOWt4PJ/dfQaPfrSV55fmsCrnIH+7ejxJtpXvquqbWLypkN2lNZw7si8ZyVHa4K1UF1XUNhIe5OeWtlHQBAEcSRBpmiA61Nl61n0C/Xjy8nGcNiSOX3+wmZnPfM2d5wxhU34ln24toqHZgo/Ai1/lkhQdzGXpCVyRmXQ4iSilHCurbTrmxKCupAkCyCmuoW94IH10UNhJuXjcANITI7lj/gZ+v2g7EcH+XJGZyKxTEhnSN4xPtxTxn+/289zSbF7+ejd/vTqd80b1c3fYSnmsitpGt7U/gCYIoHWSPi09OMPAmBDeu3UK2wurGN4/rE3vplkZiczKSGR/RR0//dd6bntrHb8+fwS3nJ7aptpp78EaIkMCdICe6vXKahrdspJcq16fIIwx5JRUc2l6grtD6TEC/HwYlxTZ4f6EyGDenTuZexZs5LHF28ktreb6ySks2VrE4s2FZBdXE+zvyw9PSeCmqSmdVm8p1ZNV1DYxrJ/7/v97fYIorW7kUH2z06b5Vl0T5O/Ls7PHkxobynNLs3lnTR4i1pX4rp00ku2FVby3Lp9/fbuP04fEcv/5wxk1oPsGDirlCaxzuGkVk9uEB/vx3m1TnDrFhuoaHx/h3vOGkZEcRUFlHeeO7Ntm3e37ZgznnTX7eH3lHi6bt5KHLhzJdZMGai8o1SvUN7VQ19Ti+Y3UIjIIyDfGNIjImcBY4A1jTIUrg+sOgX6+h2ddVe5x1vB4h9tj+gRy+9lDmD1xIHcv2MhD/9nC6tyD/OGHYwgP0vYJ1bMdGSTnvv/1rnaufR9oEZHBwD+AVODtY50kIjNEZKeIZIvI/Q72DxSRpSKyQUQ2ichMu30P2M7bKSLndTFO1QPF9AnktRsncN+M4SzZUsRFz64gv7zW3WEp5VKHp9lwYxVTVxOExRjTDFwG/NUYcxfQ6ZwQIuILPA+cD4wEZovIyHaH/QZYYIwZD1wNzLOdO9L2eBQwA5hnu57qpXx8hJ+cOYh3506m9FAD972/CWO6tjKfUt6owjZRnzu7uXY1QTSJyGzgBuBj27ZjlXsmAtnGmFxjTCMwH7ik3TEGaJ1CNAIosN2/BJhvjGkwxuwGsm3XU71cZko0D8wcwTfZB5m/Ns/d4SjlMmW1rRP1eX6CuAmYAjxmjNktIqnAW8c4JwGwfwfn27bZ+y1wnYjkA4uBO47jXERkrohkiUhWSUlJF1+K8nbXTBzIlLQYHlu0nYKKOneHo5RLuHuxIOhigjDGbDPG/NwY846IRAFhxpgnjnGao64m7esEZgOvG2MSgZnAmyLi08VzMca8ZIzJNMZkxsW5fmEd5Rl8fIQ/zhpLi8XwwAebtapJ9UitiwV5fBWTiCwTkXARiQY2Aq+JyFPHOC0fSLJ7nMiRKqRWNwMLAIwxq4AgILaL56pebGBMCPfNGMbyXSW8v37/CV+nsdmiCUZ5pPLaRvoE+rVZq767dXUcRIQxpkpEbgFeM8Y8IiKbjnHOWmCIrTpqP9ZG52vaHbMPOAd4XURGYE0QJcCHwNu2JDQAGAKs6WKsqpf40ZQUFm0u5Df/2czrK3cTGuBHWJAfcWFBnD4kltOGxHbYHba+qYV/rNjNvKXZXHZKAr+/dEw3R69U58prGoly00JBrbqaIPxEpD9wJfBgV04wxjSLyO3Ap4Av8KoxZquIPApkGWM+BO4BXhaRu7BWId1orF/ntorIAmAb0Az8zBjTclyvTPV4Pj7C01el89cvvqesppHqhmYKKur5dncZ76zZh5+PkJEcxZRBMQztG8bg+D4kx4Tw6dYD/PGTHeyvqCM1NpS3Vu/jvFH9umX9b6W6qry2ya2jqAGkK8VrEbkCeAj4xhjzExFJA/5kjJnl6gC7KjMz02RlZbk7DOUBmlssbMirYOmOYpbuLGFHURWt/+YiYIx1jYuHLhzBKQOjmPnM1zQ0Wfj0rmk6o6/yGBc/t4KokAD+Oce1HThFZJ0xJtPhvp5S/6oJQnWkrrGFnJJqsoutt9TYUC4dn4Cvj7UvxLq9ZVz+wiqunTRQq5qUxzj9yS/JGBjFX68e79Ln6SxBdHWqjUTgWWAq1qqgFcCdxph8p0WplIsEB/gyOiGC0QmOJ/vLSI5mztRU/rFiNzPH9OfUQbHdHKFSRyuvaSLKjWMgoOvjIF7D2nA8AOt4hI9s25TqEe6dPoyUmBDue38TNQ3N7g5H9XKNzRaqG5rd3gbR1QQRZ4x5zRjTbLu9DmiLnuoxggN8+eOsseSV1fH7RdvcHY7q5Vqn2fCWEkSpiFwnIr6223XAQVcGplR3m5QWw21nDOKdNXl8vEmH3Sj38YRR1ND1BDEHaxfXIqAQuBzr9BtK9Sj3TB9KelIkD7y/mbwynTFWuUd5awnCG6qYjDH7jDEXG2PijDHxxphLgR+6ODalup2/rw/Pzrb2GrnjnQ00tVjcHJHqjVqn2fCKBNGBu50WhVIeJCk6hCdmjeW7vAqeXLKD1bkHeX5pNje/vpYrX1hFbkm1u0NUPdzhKiYvGUntiK77qHqsC8b2Z0V2Ei9/vZuXv94NwKC4UMpqGrnm5W9ZcOsUBsaEuDlK1VN5ShXTySSInjHCTqkOPHLRKIb2DSMlJpTxAyOJDAlge2EVs19ezTWvrGbBrVMYEKlrmSvnK69pJNjflyB/966T1mkVk4gcEpEqB7dDWMdEKNVjBfn7ctPUVM4aHn94yuUR/cN5c84kKmubuObl1RRX1bs5StUTldU2unWhoFadJghjTJgxJtzBLcwYo5PWqF5pTGIEr8+ZSMmhBq56aTWb8ivcHZLqYSpqm4h0cxdXOLlGaqV6rYzkKP45ZyJ1jS1cNm8lT32+i8Zm7fGknKOsxgtKEEqpjmWmRPPpL6ZxybgBPPO/77ls3jds2V/p7rBUD1BR2+jWleRaaYJQ6iREhPjz1FXpvHh9Bgeq6rnw2RVc/NwK3ly1h0pbV0WljldZTSPRWsWkVM9w3qh+fHH3GTx84Ugamy089N+tTHj8C175OtfdoSkv09xioaq+WUsQSvUkkSEBzDktlU/uPJ2P7ziNyWkxPPHJDrKLdWCd6rqKOmvJU9sglOqBRITRCRE8deU4QgJ8+e2HW+kpC3Mp12udyVV7MSnVg8X2CeSe6cNYkV3Kki1F7g5HeYmyGi1BKNUrXDtpIMP7hfG7j7dR19ji7nCUFyisrAMgPizIzZFoglDKpfx8ffjdpaMpqKzn+aXZ7g5HeYGckhpEINkD5vrSBKGUi01Iieay8Qm89FUuX39foiUJ1anckmqSokLcPg8TnNxkfUqpLnrg/OEs3VnM9f9Yg4/A4Pg+jEuM5L7zhxPbJ9Dd4SkPkltSQ1pcqLvDADRBKNUt4sOD+PKeM8naU8aW/ZVsKaji3xv2Exrox28vHuXu8JSHsFgMuaXVTE6LcXcogFYxKdVtokMDmD6qH3dPH8arN05g5pj+fLA+X6uc1GGFVfXUN1kYFO8ZJQhNEEq5yeyJA6mqb2bx5kJ3h6I8ROtqhWmxfdwciZUmCKXcZHJaNKmxobyzZp+7Q1EeIsc26n6Qh7RBaIJQyk1EhNkTk8jaW86uA4fcHY7yALmlNYQF+hEX5hkdFzRBKOVGl2ckEeDro6UIBRzpwSQi7g4F0AShlFtFhwZw3uh+vL8un/ombazu7XJKqkmL84z2B9AEoZTbzZ6YpI3VitrGZgor6z2m/QE0QSjldlPSYkiJCTmuaqY9pTXkl9e6MCrV3XJLagC0BKGUOsLaWD2QtXvK2VrQ+ZKl2cWHuOOdDZz1l2XMfWNdN0WoukNOaxdXLUEopexdNSGJiGB//rB4h8O1Iwor67hz/gbOffor/rf9AKMHRLC9qIrqhmY3RKtcIdc2SV9KjCYIpZSdyJAA7jxnCCuyS/lyR3GbffVNLdz02lo+23qAudPS+PpXZ3HP9KEYA5vyK9wUsXK23NIaEqOCPWKSvlaaIJTyENdPSSYtNpTHFm+nqcVyePtji7azo+gQ8647hQfOH0FMn0DSkyIB+C5PE0RPkVNc7TEjqFtpglDKQ/j7+vDgBSPILanhrdV7AViypZA3V+/lx6enctaw+MPHRoYEkBobynf7NEH0BBaLYXdpDYM8qIEaNEEo5VHOHh7PaYNj+esX37O1oJJfLdzE2MQIfnne8KOOTU+KZENeha533QMUVdVT19TiUQ3UoAlCKY8iIvzmwhEcqm/ih/NW0mIxPHP1eAL8jn6rpidFUnKogYLKejdEqpzJE3swgYsThIjMEJGdIpItIvc72P+0iHxnu+0SkQq7fS12+z50ZZxKeZLh/cK5asJAGpotPP7DMaTEOv7QONwOodVMXq91DMRgD6tictmCQSLiCzwPnAvkA2tF5ENjzLbWY4wxd9kdfwcw3u4SdcaYdFfFp5Qne+SikVyekUhGclSHx4zoH06Anw/f5ZVzwdj+3Ridcrbckmr6eNAkfa1cWYKYCGQbY3KNMY3AfOCSTo6fDbzjwniU8hpB/r6dJgeAAD8fRg0I155MPUCOh03S18qVCSIByLN7nG/bdhQRSQZSgS/tNgeJSJaIrBaRSzs4b67tmKySkhJnxa2U10hPimTz/so23WKV98ktqfa4Hkzg2gThKBV21N3iamChMcZ+OsuBxphM4BrgryIy6KiLGfOSMSbTGJMZFxd38hEr5WXSkyKpb7Kws0jXk/BW1Q3NFFTWk9ZBW5M7uTJB5ANJdo8TgYIOjr2adtVLxpgC289cYBlt2yeUUsD4JGs1lFYzeSeLxfDAB5sBmJQW4+ZojubKBLEWGCIiqSISgDUJHNUbSUSGAVHAKrttUSISaLsfC0wFtrU/V6neLik6mOjQAE0QXuovn+/ko40F3DdjOBNTo90dzlFc1ovJGNMsIrcDnwK+wKvGmK0i8iiQZYxpTRazgfmm7WifEcCLImLBmsSesO/9pJSyEhHSkyI1QXih+Wv28fzSHGZPTOK2M9LcHY5DLksQAMaYxcDidtsebvf4tw7OWwmMcWVsSvUU45MiWbqzmKr6JsKD/N0djuqCr3aV8OB/tjBtaByPXjLa43ovtdKR1Ep5ufSBkdaZXfM6X0tCeYamFgs/n7+BIfF9eP6a8fj7eu7HsOdGppTqkrGJ1hHVG/aVuzkS1RVbC6qoqG3ijrOHEObhJT5NEEp5uYhgf8YlRTJvWY6ua+0F1u4uA2BCSucDIT2BJgileoCXr89gRP8wfvqv9Tz12U4sFp3h1VOt3VNGckwI8eFB7g7lmDRBKNUDxIcH8c7cyVyRkcgzX2Zz21vrqNHlSD2OMYasveVkJntel1ZHNEEo1UME+vny5OVjeeSikXyx/QB//mynu0NS7eSU1FBW08jEVM+vXgJNEEr1KCLCTVNTuWjcABZm5VPbqKUIT5K1x9r+kJmiJQillJtcPzmZQw3N/GdDR7PbKHdYs6eMmNAAj5x3yRFNEEr1QBnJUYzoH84bq/bokqQeJGtPOZkpUR47MK49TRBK9UAiwvWTk9lRdIj1Oj7CIxyoqmdfWS0TvKR6CYc0sU4AABOpSURBVDRBKNVjXTp+AGGBfryxaq+7Q1FYu7cCmiCUUu4XEuDHrIxEFm8upLS6wd3h9HpZe8oJ9vdl5IBwd4fSZZoglOrBrpucTFOL4d21ecc+WLnUmt1lnJIc6dFzL7XnPZEqpY7b4Pg+TB0cw79W76VFR1e7THFVPQUVdR3ur6pvYkdRldcMkGvl0um+lVLud/3kZG57az3LdxVz9vC+7g6nR7BYDF/uKGZFdikrskvJLq7Gz0f446yxzMpIPOr49XvLsRg8clGgzmgJQqke7uzhfQkN8OWL7cXuDqXHeOnrXG55I4t31+YxIDKYX8+0rgh3z3sbmbcs+6iuxVl7yvH1sS7u5E20BKFUDxfg58Opg2NZvrMEY4zX9MH3VLWNzbz0VS6nD4nllRsyCfTzBeDGU1O5972NPLlkJ0WV9fzmgpFsK6xiZU4pH6zPZ/SAcEIDvesj17uiVUqdkDOHxfH5tgNkF1czpG+Yu8Pxam9/u4+ymkZ+8YMhh5MDWBPxX69Kp294IC9/vZv5a/JobLEAMLRvH+ZOG+SukE+YJgileoEzhsYBsHxXiSaIk1Df1MLLX+cyOS2aDAcNzj4+woMXjGRo3zA25VcyMTWayWkxxIUFuiHak6cJQqleIDEqhMHxfVi2s4RbTk9zdzhea+G6fA5UNfDUlemdHndFZhJXZCZ1U1Suo43USvUSZw6NY83uMl0n4gQ1tVj4+7Icxg+M5NRBMe4Op1toglCqlzhzWDyNLRZW5Rx0dyhe6b/fFbC/oo7bzxrcaxr6NUEo1UtMSI0iJMCX5btK3B2K12mxGOYtzWZE/3DOHh7v7nC6jSYIpXqJQD9fTh0Uw7JdxToF+HH6alcJuaU1/OysQb2m9ACaIJTqVc4YGkdeWR25pTXuDsWrLNlSRJ9AP84d2btGomuCUKoXOXOYtXpk+U6tZuqq5hYLn28/wNnD49uMe+gNNEEo1YskRYeQFhfKMm2H6LKsveWU1TRy3qh+7g6l22mCUKqXOXNoPKtzD1LX2OLuULzCki1FBPj5cOawOHeH0u00QSjVy5wzIp7GZgtf7tDJ+47FGMNnW4uYNiTO6+ZRcgZNEEr1MpPTYugfEcR763QRoWPZvL+Sgsp6zhvVuxqnW2mCUKqX8fURZp2SyFe7SiiqrHd3OB5tyZYifH2EH4zQBKGU6iUuz0jEYuD99fnuDsWjLdlaxOS0aKJCA9wdiltoglCqF0qJDWViajTvZeXpoLkOZBcfIrekplf2XmqlCUKpXurKzCT2HKwla2/5cZ1XVd9Ek22dg/aMMXy54wCH6pucEaJbLdlSBMD0kZoglFK9zMwx/QgN8GXB2q43VhdW1nHGk0v51cJNDvd/urWIOa9ncdGzK9iyv9JZobrFkq1FpCdF0i8iyN2huI0mCKV6qZAAPy4cO4BFmwu7NAV4i8Vw17vfUV7bxIcbrTObtvfaN3voGx5IfZOFH85byRur9nhlFdaiTYVs2V/F+aN7b+kBNEEo1atdOSGR2sYWFm0uPOaxL36Vw+rcMn7xgyEYY3hj1Z42+7cXVvHt7jLmTE1l8Z2nM3VwDA//dys//dd6GpsdV0k5U3FVPf/ekE9zB9VfXfXljgPcOX8DE1Ki+NGUFOcE56U0QSjVi50yMIq0uFDey+q8mmljXgVPfbaLC8b0585zhjBjdD/mr8mjtvFIyeONVXsI8vfhqglJRIcG8I8bJnDv9KF8sqWIxV1IQCfKYjG8uXov5/xlOXe9u5E73tlwwglpZU4pt721nhH9w/nHjRMIDuhdcy+1pwlCqV5MRLgyM4m1e8rZWuC4zaCmoZk7528gPiyQxy8bg4hw09RUKuua+GD9fgAqa5v494b9XJqeQGSItUuoj4/w0zMHkxobelRpw1l2FFUx64WVPPSfLYxNiuDnZw/mky1FzH0zi/qm45tKZP2+cm75ZxbJ0SH8c85EwoP8XRKzN3FpghCRGSKyU0SyReR+B/ufFpHvbLddIlJht+8GEfnedrvBlXEq1ZvNnjiQsEA/5i3Ncbj/94u2sa+slqevSicixPqhmZkcxZiECF77ZjcWi2FBVh71TRZuODWlzbk+PsK1kwayfl+F0xutv9xxgAufWcHeg7U8fdU43rp5EndPH8bjl41h+a4SbnxtDdVdXF61uKqem19fS1xYIG/dMonoXjruoT2XTS4iIr7A88C5QD6wVkQ+NMZsaz3GGHOX3fF3AONt96OBR4BMwADrbOceX388pdQxRQT7c/2UZP6+PIfs4moGx/c5vG/9vnLeWZPH3GlpTEo7sg6ztRSRwt0LNrL8+xLeWL2HianRjOgfftT1r8hI4s+f7eSt1Xt5YtZYp8S8ZX8lt7+9geH9w3hzzqQ2A9mumTSQ0EBf7l6wkXOfWk54kD+NLRYamy2kD4zkqSvHtZm22xjDLxduoq6phYU/OZW+4b2311J7rixBTASyjTG5xphGYD5wSSfHzwbesd0/D/jcGFNmSwqfAzNcGKtSvdrNp6US6OfD35cdKUW0WAyP/HcrfcMD+fk5Q44654Kx/YkLC+S+hZvIK6vjxnalh1YRIf5cmp7Af77bT2XtyY+P2F9Rx5zX1xIVEsCrN0xwOMr5kvQEXvlRJqMGRJASG8LohAjSB0ayaFMhdy/YiMVypGfVW9/uY/muEn49cwSD4vocda3ezJXTEyYA9i1f+cAkRweKSDKQCnzZybkJDs6bC8wFGDhw4MlHrFQvFdMnkNkTB/LGqr384gdDSIoOYUFWHpv3V/K3q9Pp42Am00A/X66blMzTX+yif0QQ0ztZbe26ycnMX5vHe+vyuOX0tBOOs6q+iTmvraWuqYW3bplEfCff9s8aHs9Z7daPHpeYw+OLd9AvPIiHLhxJbkk1jy/azrShcVw/OfmE4+qpXFmCcLRwa0cdoq8GFhpjWluVunSuMeYlY0ymMSYzLq73zdWulDPNnZaGj1i7s1bUNvLkkh1MTI3m4nEDOjzn2snW9os5U1Px8+3442R0QgSnDIzkX9/ua/Pt/Xg0tVj46VvrySmp5sXrMhjaN+y4r/Hj09O4aWoK/1ixmxeW53D3go0E+Pnwp8vH9qq1prvKlSWIfCDJ7nEiUNDBsVcDP2t37pntzl3mxNiUUu30jwjm8oxEFmTlU17bRGVdE/938ahOPzhj+wSy6tfnENqF7qDXT0nmrnc3siK7lGlDj/8L3WOLtrMiu5Q/XT6WUwfHHvf5YG07eeiCkRRXNfDEJzsAeO6a8dru0AFXJoi1wBARSQX2Y00C17Q/SESGAVHAKrvNnwKPi0iU7fF04AEXxqqUAm47YxDvrs1j0aZCbpiS7LDRuT1H1U+OzBzTn99/vJ3nlmazu7SGkkMNlBxqID48kFvPGNTpdRZk5fH6yj3cfFoqV2QmdXhcV/j4CH+5chxNLRYGRAZz4diOS0i9ncsShDGmWURux/ph7wu8aozZKiKPAlnGmA9th84G5hu78fjGmDIR+R3WJAPwqDGmzFWxKqWskmNCuTIziaU7i7n73GFOvXagny/XThrIM19ms2Z3GT5ibfsorW7gvax8fnvxSM4b1e+oEsuGfeX85t9bOG1wLA+cP9wpsQT5+/LSjzKdcq2eTLxxnhRHMjMzTVZWlrvDUMrrWSyG+uYWQgKc//2xqcXCntIaokIDiAoJwNdH2LCvnF//ewvbC6s4Z3g8Pz1rEIlRIcTaksdFz64g0N+HD392Wq9dl8GVRGSdMcZhttQEoZRyu+YWC699s4enPt9FnW0EtJ+PEOjngwE++OmpDO937Ooudfw6SxC9bxVupZTH8fP14cfT0rhk/AA251vXgS6qrKO4qoHLxidocnATTRBKKY8RHxbEOSO0R5Gn0Mn6lFJKOaQJQimllEOaIJRSSjmkCUIppZRDmiCUUko5pAlCKaWUQ5oglFJKOaQJQimllEM9ZqoNESkB9jrYFQG0Xwy3/Tb7x47ut/6MBUpPMERHcXRlv8bfdtuJvoZjxd/ZMZ3F2/7xse5r/Md/zLH+hzp6Pc6Mv7P4jrXf09/DycYYx/OvG2N69A146Vjb7B87um/3M8uZcXRlv8Z/1LYTeg3Hiv94XsPxxu+Mv4HG3/G2jl6PM+Pvymvw9vewo1tvqGL6qAvbPjrGfUfXcEYcXdmv8XdP/J0d01m87R935f6J0Pg73tbR63Fm/F25Rk94D7TRY6qYuoOIZJkOZj30Bt4eP3j/a9D43UvjPz69oQThTC+5O4CT5O3xg/e/Bo3fvTT+46AlCKWUUg5pCUIppZRDmiCUUko51GsThIi8KiLFIrLlBM7NEJHNIpItIs+I3SrrInKHiOwUka0i8qRzo24Tg9PjF5Hfish+EfnOdpvp/MgPx+CS379t/70iYkQk1nkRO4zDFX+D34nIJtvv/zMRGeD8yA/H4Ir4/yQiO2yv4d8iEun8yA/H4Ir4r7C9dy0i4pLG4JOJu4Pr3SAi39tuN9ht7/R90iUn2qfW22/ANOAUYMsJnLsGmAII8Alwvm37WcAXQKDtcbyXxf9b4F5v/f3b9iUBn2IdNBnrba8BCLc75ufAC14W/3TAz3b/j8AfvSz+EcAwYBmQ6Ulx22JKabctGsi1/Yyy3Y/q7DUez63XliCMMV8BZfbbRGSQiCwRkXUi8rWIDG9/noj0x/omXmWsf4U3gEttu38CPGGMabA9R7GXxd9tXBj/08CvAJf3vnDFazDGVNkdGooLX4eL4v/MGNNsO3Q1kOhl8W83xux0VcwnE3cHzgM+N8aUGWPKgc+BGc56n/faBNGBl4A7jDEZwL3APAfHJAD5do/zbdsAhgKni8i3IrJcRCa4NNqjnWz8ALfbqgdeFZEo14Xq0EnFLyIXA/uNMRtdHWgnTvpvICKPiUgecC3wsAtjdcQZ/0Ot5mD95tqdnBl/d+pK3I4kAHl2j1tfi1Neo9/xntBTiUgf4FTgPbuqukBHhzrY1votzw9rMW8yMAFYICJptgzuUk6K/+/A72yPfwf8Beub3OVONn4RCQEexFrF4RZO+htgjHkQeFBEHgBuBx5xcqgOOSt+27UeBJqBfzkzxs44M/7u1FncInITcKdt22BgsYg0AruNMZfR8WtxymvUBHGED1BhjEm33ygivsA628MPsX6I2hebE4EC2/184ANbQlgjIhask2uVuDJwm5OO3xhzwO68l4GPXRlwOycb/yAgFdhoe5MlAutFZKIxpsjFsbdyxv+QvbeBRXRTgsBJ8dsaSi8EzumOL0d2nP377y4O4wYwxrwGvAYgIsuAG40xe+wOyQfOtHuciLWtIh9nvEZXNMJ4yw1Iwa6hCFgJXGG7L8C4Ds5bi7WU0Nr4M9O2/TbgUdv9oViLfuJF8fe3O+YuYL43/f7bHbMHFzdSu+hvMMTumDuAhV4W/wxgGxDn6t+9K/+HcGEj9YnGTceN1Lux1lxE2e5Hd+U1dinO7vgjeuINeAcoBJqwZtubsX4DXQJstP2TP9zBuZnAFiAHeI4jI9IDgLds+9YDZ3tZ/G8Cm4FNWL9p9fem+NsdswfX92Jyxd/gfdv2TVgnV0vwsvizsX4x+s52c2UvLFfEf5ntWg3AAeBTT4kbBwnCtn2O7feeDdx0PO+TY910qg2llFIOaS8mpZRSDmmCUEop5ZAmCKWUUg5pglBKKeWQJgillFIOaYJQPZqIVHfz870iIiOddK0Wsc7qukVEPjrWzKgiEikiP3XGcysFuqKc6uFEpNoY08eJ1/MzRyajcyn72EXkn8AuY8xjnRyfAnxsjBndHfGpnk9LEKrXEZE4EXlfRNbablNt2yeKyEoR2WD7Ocy2/UYReU9EPgI+E5EzRWSZiCwU69oH/2qda9+2PdN2v9o28d5GEVktIn1t2wfZHq8VkUe7WMpZxZFJCfuIyP9EZL1Y5/u/xHbME8AgW6njT7Zjf2l7nk0i8n9O/DWqXkAThOqN/gY8bYyZAMwCXrFt3wFMM8aMxzqL6uN250wBbjDGnG17PB74BTASSAOmOnieUGC1MWYc8BXwY7vn/5vt+Y85P45tLqFzsI5uB6gHLjPGnIJ1DZK/2BLU/UCOMSbdGPNLEZkODAEmAulAhohMO9bzKdVKJ+tTvdEPgJF2M2eGi0gYEAH8U0SGYJ350t/unM+NMfZz+K8xxuQDiMh3WOfWWdHueRo5MuHhOuBc2/0pHJmb/23gzx3EGWx37XVY5/oH69w6j9s+7C1YSxZ9HZw/3XbbYHvcB2vC+KqD51OqDU0QqjfyAaYYY+rsN4rIs8BSY8xltvr8ZXa7a9pdo8HufguO30tN5kgjX0fHdKbOGJMuIhFYE83PgGewrhMRB2QYY5pEZA8Q5OB8Af5gjHnxOJ9XKUCrmFTv9BnWdRYAEJHWaZYjgP22+ze68PlXY63aArj6WAcbYyqxLj96r4j4Y42z2JYczgKSbYceAsLsTv0UmGNbbwARSRCReCe9BtULaIJQPV2IiOTb3e7G+mGbaWu43YZ1mnaAJ4E/iMg3gK8LY/oFcLeIrAH6A5XHOsEYswHrTJ9XY12EJ1NEsrCWJnbYjjkIfGPrFvsnY8xnWKuwVonIZmAhbROIUp3Sbq5KdTPb6nd1xhgjIlcDs40xlxzrPKW6m7ZBKNX9MoDnbD2PKuimZV2VOl5aglBKKeWQtkEopZRySBOEUkophzRBKKWUckgThFJKKYc0QSillHLo/wHai3zsJM0WyAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 4e-3\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.412701</td>\n",
       "      <td>0.310972</td>\n",
       "      <td>0.883241</td>\n",
       "      <td>0.851683</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.403772</td>\n",
       "      <td>0.260574</td>\n",
       "      <td>0.873709</td>\n",
       "      <td>0.810236</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas1_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.381172</td>\n",
       "      <td>0.216032</td>\n",
       "      <td>0.911835</td>\n",
       "      <td>0.887068</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.378184</td>\n",
       "      <td>0.218601</td>\n",
       "      <td>0.905216</td>\n",
       "      <td>0.877285</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas1_sp15_multifit_bwd');\n",
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(2, slice(lr/(2.6**4),lr), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas2_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.348173</td>\n",
       "      <td>0.215961</td>\n",
       "      <td>0.919778</td>\n",
       "      <td>0.904313</td>\n",
       "      <td>00:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.320736</td>\n",
       "      <td>0.200614</td>\n",
       "      <td>0.912364</td>\n",
       "      <td>0.887739</td>\n",
       "      <td>00:24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas2_sp15_multifit_bwd');\n",
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas3_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.308242</td>\n",
       "      <td>0.198684</td>\n",
       "      <td>0.909452</td>\n",
       "      <td>0.886407</td>\n",
       "      <td>00:32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas3_sp15_multifit_bwd');\n",
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.322763</td>\n",
       "      <td>0.205428</td>\n",
       "      <td>0.907334</td>\n",
       "      <td>0.879158</td>\n",
       "      <td>00:39</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas3_sp15_multifit_bwd');\n",
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.310434</td>\n",
       "      <td>0.199419</td>\n",
       "      <td>0.910776</td>\n",
       "      <td>0.885801</td>\n",
       "      <td>00:33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas3_sp15_multifit_bwd');\n",
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas4_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas4_sp15_multifit_bwd')\n",
    "learn_c.save(f'{lang}clas_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas_sp15_multifit_bwd');\n",
    "learn_c.to_fp32().export(f'{lang}_classifier_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.48 s, sys: 404 ms, total: 3.88 s\n",
      "Wall time: 2.43 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_multifit_bwd', bs=bs, num_workers=1, backwards=True)\n",
    "\n",
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3\n",
    "\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, drop_mult=1.0, \n",
    "                                  metrics=[accuracy,f1])\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15_multifit_bwd');\n",
    "\n",
    "learn_c.load(f'{lang}clas_sp15_multifit_bwd');\n",
    "\n",
    "# put weight on cpu\n",
    "loss_weights = torch.FloatTensor(trn_weights).cpu()\n",
    "learn_c.loss_func = partial(F.cross_entropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAAEGCAYAAAAJw7AFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAX+klEQVR4nO3deZhcVZnH8e+bkNUkwx42w5YEUJZAgAEURBjZRBxxZFFxEJ2wiSKrIiAgOiqIuKGAgjggxB0h4oCRTQWEYGQZgqAxEghrhCwskuSdP+5tLJqk091JdXVOfz/P00/6nnvq1ltJd351T517bmQmkiSVol+rC5AkaXky2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdik5SgihkTENRHxXET8cBmO876IuH551tYqEbFzRDzY6jrUd4TXsakvioj3AscBmwJzganAZzPzN8t43EOAY4CdMnPBMhfay0VEAmMy8+FW1yK18YxNfU5EHAecD3wOGAmMAi4A3rkcDr8+8Ke+EGqdERErtboG9T0Gm/qUiPgX4Czg6Mz8SWbOz8yXM/OazDyx7jMoIs6PiMfqr/MjYlC9b9eImBkRx0fEkxExKyI+WO87EzgdODAi5kXEhyLijIi4vOH5N4iIbPsPPyIOjYi/RMTciJgeEe9raP9Nw+N2iog76yHOOyNip4Z9N0XEZyLit/Vxro+I1Zfw+tvqP6mh/n+PiH0i4k8RMTsiTmnov31E3BYRz9Z9vx4RA+t9t9Td/li/3gMbjn9yRDwOXNrWVj9m4/o5tqm314mIpyNi12X6h5UaGGzqa3YEBgM/7aDPp4AdgHHAVsD2wKkN+9cC/gVYF/gQ8I2IWCUzP011FjgxM4dl5nc6KiQiXgd8Fdg7M4cDO1ENibbvtyowqe67GnAeMCkiVmvo9l7gg8CawEDghA6eei2qv4N1qYL4YuD9wHhgZ+D0iNio7rsQ+DiwOtXf3e7AUQCZuUvdZ6v69U5sOP6qVGevExqfODP/DJwMXBERQ4FLge9m5k0d1Ct1icGmvmY14OmlDBW+DzgrM5/MzKeAM4FDGva/XO9/OTN/AcwDNulmPYuAzSNiSGbOysz7F9Pn7cBDmfk/mbkgM68EpgHvaOhzaWb+KTNfAH5AFcpL8jLV54kvA1dRhdZXMnNu/fz3A1sCZOaUzLy9ft6/AhcCb+nEa/p0Zr5U1/MqmXkx8BBwB7A21RsJabkx2NTXPAOsvpTPftYBZjRsz6jbXjlGu2B8HhjW1UIycz5wIHAEMCsiJkXEpp2op62mdRu2H+9CPc9k5sL6+7bgeaJh/wttj4+IsRFxbUQ8HhFzqM5IFzvM2eCpzHxxKX0uBjYHvpaZLy2lr9QlBpv6mtuAF4F/76DPY1TDaG1G1W3dMR8Y2rC9VuPOzPzfzHwb1ZnLNKr/8JdWT1tNj3azpq74JlVdYzJzBHAKEEt5TIdTrSNiGNXkne8AZ9RDrdJyY7CpT8nM56g+V/pGPWliaEQMiIi9I+KLdbcrgVMjYo16EsbpwOVLOuZSTAV2iYhR9cSVT7btiIiREbFf/VnbS1RDmgsXc4xfAGMj4r0RsVJEHAi8Abi2mzV1xXBgDjCvPps8st3+J4CNXvOojn0FmJKZH6b67PBby1yl1MBgU5+TmedRXcN2KvAU8AjwEeBndZezgbuAe4B7gbvrtu481w3AxPpYU3h1GPUDjqc6I5tN9dnVUYs5xjPAvnXfZ4CTgH0z8+nu1NRFJ1BNTJlLdTY5sd3+M4DL6lmTByztYBHxTmAvquFXqP4dtmmbDSotD16gLUkqimdsepWIuKS+vum+hrb3RMT9EbEoIrZt1/+TEfFwRDwYEXvWba+PiBsj4oH6cR/r6dchNUNE9I+IP0TEtfX2FfXP/n31786AVtcog02v9V2qoaJG9wH7A7c0NkbEG4CDgDfWj7kgIvoDC4DjM3MzquvBjq77Siu6jwEPNGxfQbUs2xbAEODDrShKr2aw6VUy8xaqz3sa2x7IzMUtYvtO4Kr6eqXpwMPA9vX1WHfXj51L9R/Buot5vLTCiIj1qK4p/HZbW2b+ImvA74H1WlWf/qlpwVYvHfRARFxcD0ddH9XK5xtHxC8jYkpE3Np23U7dfnu9XNBZETGvWbVpuVmXauJFm5m0C7CI2ADYmupiXGlFdj7VxJ1F7XfUQ5CHAL/s6aL0Ws0+YxsDfCMz3wg8C7wbuAg4JjPHU824uqDu+xWq1Q+2o/vXDKlnLe56pldmI9XXK/0YODYz5/RYVdJyFhH7Ak9m5pQldLkAuCUzb+3BsrQETZsVWb9TvyEzx9TbJwMDqJbPaRzWGpSZm0XEM8DIzFwQESOAxzLzNasnRMQE6vXnBgweOn619TZsSv192YKX/8Gzs/7G6qNGv6r9mZnTGbH6WgwYPASAebOfAmDYqmsAMPvRvzJs1TUZOGQomcnfH5vBoKHDeN0qS1uoQt2xzojBrS6hz3j00UeZPfsZIoJFixaxcOFCVl55FTbccENmzXqM559/no022piIpV27ruXp7runPJ2Za7Rvb/YtJRqXyllIdYuQZzOzo3XsOpSZF1Gd9bH2mM3zsK/+ZNkq1Gs8+8RMfnDGEbT/u7385EPY/UMnsfbYLQB4asZDXP3F4/nAl65i3jNP8P1TDmXCtyYR/fpxzZdOZtTm2/K2w10GsFlOe9vYVpfQJ91y802cf965/OTqa7n0O9/msu9ewnXXT2bIkCGtLq3PGTIg2i81BzQ/2NqbA0yPiPdk5g+jenuzZWb+EbidaqhyItVMO7XAz75wHDPu+T0vzPk7XztkF3Z+/zEMGb4y13/zMzz/3GwmnnE4IzfajIPP/g5rrD+GzXbem4sO34d+/fuz55Gn069/fx65/y7u+/XVrLHBWL79keoWZ7v+53GM3m5pa+dKK5Zjjj6CUeuvz65v3hGAd75rf0459fQWV6VmD0Vem5mb19snUC2sehnV+nNrUw1NXpWZZ0XEGKpli4JqmZ0JmdnhTDrP2NSXecamvm7IgJiSmdu2b2/aGVt9i4vNG7bPbdjd/jopqBZ03SEzMyIOolrSSJKkLulNt20fD3y9Hp58FjisxfVIklZAvSbY6mmyW7W6DknSis2VRyRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRVlpSTsi4hogl7Q/M/drSkWSJC2DJQYbcG6PVSFJ0nKyxGDLzJt7shBJkpaHjs7YAIiIMcB/A28ABre1Z+ZGTaxLkqRu6czkkUuBbwILgLcC3wP+p5lFSZLUXZ0JtiGZORmIzJyRmWcAuzW3LEmSumepQ5HAixHRD3goIj4CPAqs2dyyJEnqns6csR0LDAU+CowHDgH+s5lFSZLUXUs9Y8vMO+tv5wEfbG45kiQtm87MiryRxVyonZl+ziZJ6nU68xnbCQ3fDwbeTTVDUpKkXqczQ5FT2jX9NiK8eFuS1Ct1Zihy1YbNflQTSNZqWkVdsMbrBnH49qNaXYbUEqts95FWlyD1Sp0ZipxC9RlbUA1BTgc+1MyiJEnqrs4E22aZ+WJjQ0QMalI9kiQtk85cx/a7xbTdtrwLkSRpeejofmxrAesCQyJia6qhSIARVBdsS5LU63Q0FLkncCiwHvAl/hlsc4BTmluWJEnd09H92C4DLouId2fmj3uwJkmSuq0zn7GNj4iV2zYiYpWIOLuJNUmS1G2dCba9M/PZto3M/DuwT/NKkiSp+zoTbP0bp/dHxBDA6f6SpF6pM9exXQ5MjohL6+0PApc1ryRJkrqvM2tFfjEi7gH+jWpm5C+B9ZtdmCRJ3dGZoUiAx4FFVCv77w480LSKJElaBh1doD0WOAg4GHgGmAhEZr61h2qTJKnLOhqKnAbcCrwjMx8GiIiP90hVkiR1U0dDke+mGoK8MSIujojd+efqI5Ik9UpLDLbM/GlmHghsCtwEfBwYGRHfjIg9eqg+SZK6ZKmTRzJzfmZekZn7Uq0bORX4RNMrkySpGzo7KxKAzJydmRdm5m7NKkiSpGXRpWCTJKm3M9gkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2CRJRTHYJElFWanVBah3euzRR/j4UR/mqSefoF+/frz3A4dx2OEf4f/uu4dTTjiG5+fPZ73Xr89XLryU4cNH8MjfZrD7TuPYePRYALYevz2f+9LXWvwqpK5Zb+TKfPszH2DkaiNYlMklP/4t37jyJk4/6u3s+5YtWZTJU7PnMuHTlzPrqefYd9ctOP3IfVmUyYKFizjpnB/xu6l/AeCzH3sne+28Of0i+PUd0zj+iz9q8avrOww2LVb//itx6lmfZ4uttmbe3Lnsu/tOvHnX3Tn52CP51JmfZ4c37czEKy7jwq9/mRM++WkA1t9gI6676Y4WVy5134KFi/jEeT9h6rSZDBs6iN99/2Qm3zGNL182mbMumATAUQe/hU9O2JuPfvYqbrzjQa696V4ANh+zDpd/4TDG7X82O2y1ITuO24jtDvgcAL++9Dh2Hj+GW6c81LLX1pc4FKnFGrnW2myx1dYADBs+nNFjN+WJWY/xl4cf4l93ejMAO++6G9dd87NWliktV48/PYep02YCMO/5l5g2/XHWWWNl5s5/8ZU+Q4cMIjMBmP/CP15pf92QQdTNZMKggQMYOGAlBg1ciZVW6s+Ts+f03Avp45p6xhYRGwC/BO4Atgb+BHwA2BE4t37+O4EjM/OliPg8sB+wALg+M09oZn3qnEf+NoP7753KuPHbMXazN3DDddeyxz7vYNLVP2HWozMb+v2Vvd+6A8OHDeeEUz7N9ju+uYVVS8tm1NqrMm6T9bjzvr8CcMbR7+B9+27Pc/NeYK8JX32l335v3ZKzjtmPNVYdzv4f/RYAd9wznVvueojpN3yWIPjWxFt4cPoTrXgZfVJPnLFtAlyUmVsCc4DjgO8CB2bmFlThdmRErAq8C3hj3ffsHqhNSzF/3jyOOPRgTv/sOQwfPoJzvnoh37vkQt6+207MnzePAQMHArDmyLW4beqfuO7G2zntM1/go4cfyty5vkPViul1QwZy5bkf5sRzf/zK2doZ37iGMXufxlXX3cURB+7ySt+f33gP4/Y/mwOOu4jTj3o7ABu9fnU22XAko/c8lY33/BS7bj+WN22zcUteS18UbafUTTl4dcZ2S2aOqrd3A04D+mfmLnXb7sDRwAHAFOAuYBJwbWb+YzHHnABMqDc3AR5s2gtQAKOp3pAs7u3mWsAqwAOL2bcJ8AjwfNOqk5pg4MCBMXny5NG/+tWv5px55pmv+bkfM2bMwEmTJo0ZO3bsE8DTjftmzpy5xbbbbvvAUUcdtdrgwYP7nXTSSbMAzjnnnLVffPHFRaeddpqnbcvX+pm5RvvGngi2mzNz/Xp7icGWmftHxCBgd+AgYL3M3K1pxalDERHAZcDszDy2oX3NzHwyIvoBTwInZeYlEbFG3XdhRGwE3ApskZmzW/ICpO555eceOLahfQzQNvPjGOAtEbFBZh4E/BlIYBvgGmA9qjfq/wXsVR/zl8D59X41WU/MihwVETtm5m3AwcCvgMMjYnRmPgwcAtwcEcOAoZn5i4i4HXi4B2rTkr2J6t/m3oiYWredAoyJiKPr7ZeBS+vvdwHOiogFwELgCENNK6BXfu6Bxp/7D1GNQiwCZgBHAFcD76aaN/Ay8AJwIFXI/QjYrT5OUgWbodZDeuKM7RfALcBOVO94DmExk0eAVal+UAZTvcM5NzMva1pxWmYRcVdmbtvqOqRW8Oe/9+qJM7ZFmXlEu7bJVLMkG80Ctu+BerT8XNTqAqQW8ue/l+qJM7ZrM3Pzpj2JJEkNmhpskiT1NFcekSQVxWCTJBXFYJMkFcVgU5dExNyImNPu65GI+Gl9YbZUrIj4YkSMiIgBETE5Ip6OiPe3ui69msGmrjoPOBFYl2qFhROAi4GrgEtaWJfUE/bIzDnAvsBMYCzV74N6EYNNXbVXZl6YmXMzc05mXgTsk5kTqdaNlEo2oP5zH+BKV9fpnQw2ddWiiDggIvrVXwc07PPaEZXumoiYBmwLTK7XSH1xKY9RD/M6NnVJ/TnaV6iWRUvgduDjwKPA+Mz8TQvLk5ouIlYB5tQLfg8FRmTm462uS/9ksElSJ0XEAKq1bdtuyHYz8K3MfLl1Vak9hyLVJRExtp4Ndl+9vWVEnNrquqQe8k1gPHBB/bVN3aZexDM2dUlE3Ew1C+zCzNy6brvP9UDVF0TEHzNzq6W1qbU8Y1NXDc3M37drW9CSSqSetzAiNm7bqD9zXtjCerQYPXHbGpXl6foXOwEi4j+objkk9QUnAjdGxF/q7Q2AD7auHC2OQ5Hqkvod6kVUN479OzAdeF9mzmhpYVIPiIjBwPHA7nXTDcCXM9Mp/72IwaYuiYhBwH9QvVNdFZgDZGae1cq6pJ4QET+g+pm/om46GFglM9/TuqrUnkOR6qqrgWeBu4HHWlyL1NM2aTdR5MaI+GPLqtFiGWzqqvUyc69WFyG1yB8iYofMvB0gIv4V+G2La1I7DkWqSyLiIuBrmXlvq2uRelpEPABsAvytbhoFPAAsohqS37JVtemfDDZ1SUT8HzCaatLIS0DgL7T6iIhYv6P9TqLqHQw2dcmSfrH9hZbUWxhskqSiuPKIJKkoBpskqSgGm9RkEbEwIqZGxH0R8cP6Hl7dPdauEXFt/f1+EfGJDvquHBFHdeM5zoiIE7pbo9RqBpvUfC9k5rj6Dgj/AI5o3BmVLv8uZubPM/PzHXRZGehysEkrOoNN6lm3AqMjYoOIeCAiLqBaxeX1EbFHRNwWEXfXZ3bDACJir4iYFhG/AfZvO1BEHBoRX6+/HxkRP42IP9ZfOwGfBzauzxbPqfudGBF3RsQ9EXFmw7E+FREPRsSvqK7TklZYBpvUQyJiJWBvoO3i9k2A79X3tZsPnAr8W2ZuA9wFHFcvunsx8A5gZ2CtJRz+q8DN9XJP2wD3A58A/lyfLZ4YEXsAY4DtgXHA+IjYJSLGAwcBW1MF53bL+aVLPcoltaTmGxIRU+vvbwW+A6wDzGhbmgnYAXgD8NuIABgI3AZsCkzPzIcAIuJyYMJinmM34AMAmbkQeC4iVmnXZ4/66w/19jCqoBsO/DQzn6+f4+fL9GqlFjPYpOZ7ITPHNTbU4TW/sQm4ITMPbtdvHPW975aDAP47My9s9xzHLsfnkFrOoUipd7gdeFNEjAaIiKERMRaYBmzYcNfmg5fw+MnAkfVj+0fECGAu1dlYm/8FDmv47G7diFgTuAV4V0QMiYjhVMOe0grLYJN6gcx8CjgUuDIi7qEKuk3rG1hOACbVk0eWtHTZx4C3RsS9wBTgjZn5DNXQ5n0RcU5mXg98H7it7vcjYHhm3g1MBKYCP6YaLpVWWC6pJUkqimdskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKL8P/53nVMJ5K3qAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds,y,losses = learn_c.get_preds(with_loss=True)\n",
    "predictions = np.argmax(preds, axis = 1)\n",
    "\n",
    "interp = ClassificationInterpretation(learn_c, preds, y, losses)\n",
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1102   42]\n",
      " [ 295 2338]]\n",
      "accuracy global: 0.910775747948107\n",
      "accuracy on negative reviews: 96.32867132867133\n",
      "accuracy on positive reviews: 88.79605013292822\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(np.array(y), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1])/(cm[0,0]+cm[0,1]+cm[1,0]+cm[1,1])}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on negative reviews: {cm[0,0]/(cm[0,0]+cm[0,1])*100}') \n",
    "print(f'accuracy on positive reviews: {cm[1,1]/(cm[1,0]+cm[1,1])*100}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>. ▁providências ▁xxup ▁ate ▁xxup ▁peço ▁xxup , ▁chegou ▁xxup ▁não ▁xxup b y s 39 14 ▁2 ▁xxup y t al ci pe s ▁ ▁xxmaj cta ▁invi ▁xxmaj ▁relógio ▁xxmaj ▁produto ▁xxup ▁o ▁mas ▁xxup , ▁similar ▁xxup ▁produto ▁xxup ▁bem ▁xxup ▁tudo ▁xxup , 1 400 ▁2 ver ▁di ▁xxup ▁pro ▁xxup cta ▁invi ▁xxup ▁pelo ▁xxup ▁trocado ▁xxup ▁veio ▁xxup , 4 5 19 ▁2 -</td>\n",
       "      <td>neg</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>. ▁esclarecimentos ▁xxup ▁peço ▁xxup . ▁foi ▁xxup ▁que ▁xxup ▁informado ▁xxup ▁sendo ▁xxup ▁está ▁xxup ▁site ▁xxup ▁no ▁xxup ▁porém ▁xxup , ▁entregue ▁xxup ▁foi ▁xxup ▁não ▁xxup ▁produto ▁xxup ▁o ▁que ▁xxup ▁informo ▁xxup p ra t s ▁ ▁xxmaj le b u ▁do ▁xxmaj r l s ▁d ▁xxmaj s ▁camera ▁xxmaj timo gi le ▁ ▁xxmaj ▁couro ▁xxmaj le b u do - ▁x ▁xxmaj ▁dupla</td>\n",
       "      <td>neg</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>. ▁att ▁xxup , ▁ ▁retorno ▁xxup ▁o aguardo , ▁ ▁xxup ▁frete ▁xxup ▁do ▁xxup ▁valor ▁xxup ▁o ▁devolver ▁xxup ▁deveria ▁xxup ▁vcs ▁xxup pois , ▁ ▁xxup ▁correios ▁xxup ▁do ▁xxup ▁agência ▁xxup ▁na ▁xxup ▁retirar ▁xxup ▁que ▁xxup tive , ▁endereço ▁xxup ▁meu ▁xxup ▁no ▁xxup ▁entregue ▁xxup ▁foi ▁xxup ▁não ▁xxup ▁e ▁frete ▁xxup ▁alto ▁xxup ▁um ▁xxup ▁cobrado ▁xxup ▁foi ▁xxup ▁produto ▁xxup ▁o .</td>\n",
       "      <td>pos</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2 24 09 7 2 99 - 71 ▁ l ▁te ▁xxup ▁contato ▁xxup ▁aguardo ▁xxup ▁chegou ▁xxup ▁nao ▁xxup 20 , ▁17 ▁as ▁xxup 18 - 04 - ▁25 ▁empresa ▁xxup ▁essa ▁xxup ▁com ▁xxup ▁triste ▁xxup ▁mt ▁xxup ▁to ▁xxup ▁produto ▁xxup ▁do ▁xxup r ▁entraga ▁xxup ▁a o ira pr ▁com ▁xxup ▁nao ▁xxup ▁eles ▁xxup ▁entregue ▁xxup ▁ser ▁xxup ▁pra ▁xxup ▁era ▁xxup 18 - 04</td>\n",
       "      <td>neg</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>! . ▁resposta ▁xxup ▁nenhuma ▁xxup ▁obtive ▁xxup ▁não ▁xxup ▁agora ▁xxup ▁ate ▁xxup ▁e ▁problema ▁xxup ▁o ▁resolve ▁xxup ▁para ▁xxup ▁emails ▁xxup ▁vários ▁xxup ▁enviei ▁xxup ▁já ▁xxup , ▁entrega ▁xxup ▁de ▁xxup ▁prazo ▁xxup ▁do ▁xxup ▁fora ▁xxup ▁assim ▁xxup ▁mesmo ▁xxup ▁e ), nd u ( ▁1 ▁xxup ▁apenas ▁xxup ▁chegou ▁xxup , ▁produto ▁xxup ▁do ▁xxup ) nd u ▁( ▁xxup ▁2 ▁de ▁xxup ▁pedido</td>\n",
       "      <td>neg</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Predictions some random sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "neg tensor([0.9755, 0.0245])\n"
     ]
    }
   ],
   "source": [
    "# Get the prediction neg/pos\n",
    "review = 'Eu não gosto deste produto'\n",
    "pred = learn_c.predict(review)\n",
    "print(pred[0], pred[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style=\"font-family: monospace;\"><span title=\"0.339\" style=\"background-color: rgba(197, 197, 224, 0.5);\">▁produto</span> <span title=\"0.668\" style=\"background-color: rgba(119, 108, 177, 0.5);\">▁deste</span> <span title=\"1.000\" style=\"background-color: rgba(63, 0, 125, 0.5);\">▁gosto</span> <span title=\"0.719\" style=\"background-color: rgba(111, 91, 168, 0.5);\">▁não</span> <span title=\"0.138\" style=\"background-color: rgba(236, 235, 244, 0.5);\">▁eu</span> <span title=\"0.039\" style=\"background-color: rgba(247, 246, 250, 0.5);\">▁xxmaj</span> <span title=\"0.015\" style=\"background-color: rgba(250, 249, 252, 0.5);\">▁xxbos</span></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The darker the word-shading in the below example, the more it contributes to the classification. \n",
    "txt_ci = TextClassificationInterpretation.from_learner(learn_c)\n",
    "test_text = \"Eu não gosto deste produto\"\n",
    "txt_ci.show_intrinsic_attention(test_text,cmap=cm.Purples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.3385, 0.6681, 1.0000, 0.7190, 0.1381, 0.0393, 0.0155],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt_ci.intrinsic_attention(test_text)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Text</th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Actual</th>\n",
       "      <th>Loss</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁comprei ▁na ▁comodidade ▁de ▁internet ▁porém ▁não ▁entregaram ▁na ▁minha ▁casa . tive ▁que ▁buscar ▁no ▁correio , odiei ▁isso , não ▁recomendo ▁pq ▁não ▁tive ▁a ▁comodidade ▁de ▁receber ▁em ▁casa . apesar ▁de ▁pagar ▁frete . &lt;unk&gt;</td>\n",
       "      <td>pos</td>\n",
       "      <td>neg</td>\n",
       "      <td>5.04</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁super ▁recomendo , ▁entrega ▁rápida ▁como ▁sempre .</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "      <td>4.96</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁demora ▁na ▁entrega , ▁o ▁produto ▁veio ▁arranhado ▁em ▁algumas ▁partes ▁e ▁a ▁película ▁mesmo ▁propria ▁para ▁o ▁celular , ▁e ▁me nho r ▁do ▁que ▁a ▁tela .</td>\n",
       "      <td>pos</td>\n",
       "      <td>neg</td>\n",
       "      <td>4.08</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁tudo ▁dentro ▁do ▁prazo , ▁muito ▁bem ▁embalado ▁e ▁delicado ... ▁xxmaj ▁ótimo .</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "      <td>3.99</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁adoro ▁a ▁lannister !</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "      <td>3.82</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# tabulation showing the first k texts in top_losses along with their prediction, actual,loss, and probability of actual class.\n",
    "# max_len is the maximum number of tokens displayed. If max_len=None, it will display all tokens.\n",
    "txt_ci.show_top_losses(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_multifit', bs=bs, num_workers=1)\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, drop_mult=1.0, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load(f'{lang}clas_sp15_multifit', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9280), tensor(0.9467))"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds,targs = learn_c.get_preds(ordered=True)\n",
    "accuracy(preds,targs),f1(preds,targs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas_bwd = load_data(path, f'{lang}_textlist_class_sp15_multifit_bwd', bs=bs, num_workers=1, backwards=True)\n",
    "learn_c_bwd = text_classifier_learner(data_clas_bwd, AWD_LSTM, config=config, drop_mult=1.0, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c_bwd.load(f'{lang}clas_sp15_multifit_bwd', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9108), tensor(0.9328))"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_b,targs_b = learn_c_bwd.get_preds(ordered=True)\n",
    "accuracy(preds_b,targs_b),f1(preds_b,targs_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_avg = (preds+preds_b)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9230), tensor(0.9426))"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(preds_avg,targs_b),f1(preds_avg,targs_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1096   48]\n",
      " [ 243 2390]]\n",
      "accuracy global: 0.9229547259729944\n",
      "accuracy on negative reviews: 95.8041958041958\n",
      "accuracy on positive reviews: 90.77098366881884\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "predictions = np.argmax(preds_avg, axis = 1)\n",
    "cm = confusion_matrix(np.array(targs_b), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1])/(cm[0,0]+cm[0,1]+cm[1,0]+cm[1,1])}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on negative reviews: {cm[0,0]/(cm[0,0]+cm[0,1])*100}') \n",
    "print(f'accuracy on positive reviews: {cm[1,1]/(cm[1,0]+cm[1,1])*100}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}