Text classification with BERT in PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text classification with BERT in PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2018 was an exciting year for Natural Language Processing. One of the most promising evolutions was the breakthrough of transfer learning. Models like Elmo Embeddings, ULMFit and BERT allow us to pre-train a neural network on a large collection of unlabelled texts. Thanks to an auxiliary task such as language modelling, these models are able to learn a lot about the syntax, semantics and morphology of a language. This knowledge can be put to good use: because they already know so much about language use, these models need much less labelled data to reach state-of-the-art performance on other tasks, such as text classification, sequence labelling or question answering. \n",
    "\n",
    "One of the most popular models is [BERT](https://arxiv.org/abs/1810.04805), developed by researchers at Google. BERT stands for Bidirectional Encoder Representations from Transformers. It uses the Transformer architecture to pretrain bidirectional \"language models\". By adding just one task-specific output layer, it is possible to use such a pre-trained BERT model on a variety of NLP tasks. In this notebook, we're going to investigate its performance on a sentiment analysis task, where the task is to predict whether a review is positive or negative. Unfortunately, we can't share the data, but you can easily plug in your own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first get our data. Our corpus is a simple newline-delimited json file with a list of product reviews from Amazon. It's a subset of the huge [Amazon review corpus](https://nijianmo.github.io/amazon/index.html) that has been so popular in sentiment analysis. Each of the documents in our file is a dictionary with a \"title\", \"body\" and \"rating\". We'll try to build a model that can predict the rating from the title and the body."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "CORPUS_PATH = \"data/sentiment_analysis/review_corpus_en.ndjson\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we split up the data into a train, development and test portion. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: 2430\n",
      "Dev size: 270\n",
      "Test size: 300\n"
     ]
    }
   ],
   "source": [
    "import ndjson\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "with open(CORPUS_PATH) as i:\n",
    "    data = ndjson.load(i)\n",
    "    \n",
    "texts = [\" \".join([doc[\"title\"], doc[\"body\"]]) for doc in data]\n",
    "labels = [doc[\"rating\"] for doc in data]\n",
    "    \n",
    "rest_texts, test_texts, rest_labels, test_labels = train_test_split(texts, labels, test_size=0.1, random_state=1)\n",
    "train_texts, dev_texts, train_labels, dev_labels = train_test_split(rest_texts, rest_labels, test_size=0.1, random_state=1)\n",
    "\n",
    "print(\"Train size:\", len(train_texts))\n",
    "print(\"Dev size:\", len(dev_texts))\n",
    "print(\"Test size:\", len(test_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to determine the number of labels in our data. We'll map each of these labels to an index. In our sentiment analysis example, there are three labels: positive (all 4-star and 5-star reviews), mixed (all 3-star reviews) and negative (all 2-star and 1-star reviews)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'pos': 0, 'neg': 1, 'mixed': 2}\n"
     ]
    }
   ],
   "source": [
    "target_names = list(set(labels))\n",
    "label2idx = {label: idx for idx, label in enumerate(target_names)}\n",
    "print(label2idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline\n",
    "\n",
    "Let's train a baseline model for our task. In this way we have something to compare BERT's performance to. As our baseline, we choose a simple Logistic Regression classifier from Scikit-learn. We use grid search to find the optimal settings for its hyperparameter _C_. At the end of this process, we find that our best baseline classifier obtains an accuracy of 62.67%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 8 candidates, totalling 40 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  40 out of  40 | elapsed:   33.9s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline accuracy: 0.6266666666666667\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('lr', LogisticRegression(multi_class=\"ovr\", solver=\"lbfgs\"))\n",
    "])\n",
    "\n",
    "parameters = {'lr__C': [0.1, 0.5, 1, 2, 5, 10, 100, 1000]}\n",
    "\n",
    "best_classifier = GridSearchCV(pipeline, parameters, cv=5, verbose=1)\n",
    "best_classifier.fit(train_texts, train_labels)\n",
    "best_predictions = best_classifier.predict(test_texts)\n",
    "\n",
    "baseline_accuracy = np.mean(best_predictions == test_labels)\n",
    "print(\"Baseline accuracy:\", baseline_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we move to BERT. The team at [HuggingFace](https://github.com/huggingface) has developed a great Python library, [transformers](https://github.com/huggingface/transformers), with implementations of an impressive number of transfer-learning models in PyTorch and Tensorflow. It makes finetuning these models pretty easy. Let's first install this library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: transformers in /opt/anaconda3/lib/python3.7/site-packages (2.1.1)\n",
      "Requirement already satisfied: requests in /opt/anaconda3/lib/python3.7/site-packages (from transformers) (2.21.0)\n",
      "Requirement already satisfied: boto3 in /opt/anaconda3/lib/python3.7/site-packages (from transformers) (1.9.82)\n",
      "Requirement already satisfied: tqdm in /opt/anaconda3/lib/python3.7/site-packages (from transformers) (4.28.1)\n",
      "Requirement already satisfied: regex in /opt/anaconda3/lib/python3.7/site-packages (from transformers) (2018.1.10)\n",
      "Requirement already satisfied: sacremoses in /opt/anaconda3/lib/python3.7/site-packages (from transformers) (0.0.35)\n",
      "Requirement already satisfied: sentencepiece in /opt/anaconda3/lib/python3.7/site-packages (from transformers) (0.1.82)\n",
      "Requirement already satisfied: numpy in /opt/anaconda3/lib/python3.7/site-packages (from transformers) (1.17.3)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/anaconda3/lib/python3.7/site-packages (from requests->transformers) (3.0.4)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/anaconda3/lib/python3.7/site-packages (from requests->transformers) (1.24.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/lib/python3.7/site-packages (from requests->transformers) (2019.6.16)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /opt/anaconda3/lib/python3.7/site-packages (from requests->transformers) (2.8)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/anaconda3/lib/python3.7/site-packages (from boto3->transformers) (0.9.3)\n",
      "Requirement already satisfied: botocore<1.13.0,>=1.12.82 in /opt/anaconda3/lib/python3.7/site-packages (from boto3->transformers) (1.12.93)\n",
      "Requirement already satisfied: s3transfer<0.2.0,>=0.1.10 in /opt/anaconda3/lib/python3.7/site-packages (from boto3->transformers) (0.1.13)\n",
      "Requirement already satisfied: joblib in /opt/anaconda3/lib/python3.7/site-packages (from sacremoses->transformers) (0.13.1)\n",
      "Requirement already satisfied: click in /opt/anaconda3/lib/python3.7/site-packages (from sacremoses->transformers) (7.0)\n",
      "Requirement already satisfied: six in /opt/anaconda3/lib/python3.7/site-packages (from sacremoses->transformers) (1.11.0)\n",
      "Requirement already satisfied: docutils>=0.10 in /opt/anaconda3/lib/python3.7/site-packages (from botocore<1.13.0,>=1.12.82->boto3->transformers) (0.14)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= \"2.7\" in /opt/anaconda3/lib/python3.7/site-packages (from botocore<1.13.0,>=1.12.82->boto3->transformers) (2.7.5)\n",
      "\u001b[33mWARNING: You are using pip version 19.2.1, however version 19.3.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You really need a GPU to finetune BERT. Still, to make sure this code runs on any machine we'll let PyTorch determine whether a GPU is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Google has made available a range of BERT models for us to experiment with. For English, there is a choice between three models: `bert-large-uncased` is the largest model that will likely give the best results. Its smaller siblings are `bert-base-uncased` and `bert-base-cased`, which are more practical to work with. For Chinese there is `bert-base-chinese`, and for the other languages we have `bert-base-multilingual-uncased` and `bert-base-multilingual-cased`. \n",
    "\n",
    "Uncased means that the training text has been lowercased and accents have been stripped. This is usually better, unless you know that case information is important for your task, such as with Named Entity Recognition. \n",
    "\n",
    "In our example, we're going to investigate sentiment analysis on English. We'll therefore use the English BERT-base model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "BERT_MODEL = \"bert-base-uncased\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each model comes with its own tokenizer. This tokenizer splits texts into [word pieces](https://github.com/google/sentencepiece). In addition, we'll tell the tokenizer it should lowercase the text, as we're going to work with the uncased model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "I1101 11:57:58.966336 139947573684032 file_utils.py:39] PyTorch version 1.1.0 available.\n",
      "I1101 11:57:59.004704 139947573684032 modeling_xlnet.py:194] Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex .\n",
      "I1101 11:57:59.558555 139947573684032 tokenization_utils.py:374] loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/yves/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n"
     ]
    }
   ],
   "source": [
    "from transformers.tokenization_bert import BertTokenizer\n",
    "\n",
    "tokenizer = BertTokenizer.from_pretrained(BERT_MODEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A full BERT model consists of a common, pretrained core, and an extension on top that depends on the particular NLP task. After all, the output of a sequence classification model, where we have just one prediction for every sequence, looks very different from the output of a sequence labelling or question answering model. As we're looking at sentiment classification, we're going to use the pretrained BERT model with a final layer for sequence classification on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1101 11:58:00.115528 139947573684032 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /home/yves/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.bf3b9ea126d8c0001ee8a1e8b92229871d06d36d8808208cc2449280da87785c\n",
      "I1101 11:58:00.117981 139947573684032 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "I1101 11:58:00.627755 139947573684032 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n",
      "I1101 11:58:02.789199 139947573684032 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1101 11:58:02.789839 139947573684032 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "BertForSequenceClassification(\n",
       "  (bert): BertModel(\n",
       "    (embeddings): BertEmbeddings(\n",
       "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (token_type_embeddings): Embedding(2, 768)\n",
       "      (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1)\n",
       "    )\n",
       "    (encoder): BertEncoder(\n",
       "      (layer): ModuleList(\n",
       "        (0): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (1): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (2): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (3): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (4): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (5): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (6): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (7): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (8): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (9): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (10): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (11): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (pooler): BertPooler(\n",
       "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (activation): Tanh()\n",
       "    )\n",
       "  )\n",
       "  (dropout): Dropout(p=0.1)\n",
       "  (classifier): Linear(in_features=768, out_features=3, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers.modeling_bert import BertForSequenceClassification\n",
    "\n",
    "model = BertForSequenceClassification.from_pretrained(BERT_MODEL, num_labels = len(label2idx))\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to prepare our data for BERT. We'll present every document as a BertInputItem object, which contains all the information BERT needs: \n",
    "\n",
    "- a list of input ids. Take a look at the logging output to see what this means. Every text has been split up into subword units, which are shared between all the languages in the multilingual model. When a word appears frequently enough in a combined corpus of all languages, it is kept intact. If it is less frequent, it is split up into subword units that do occur frequently enough across all languages. This allows our model to process every text as a sequence of strings from a finite vocabulary of limited size. Note also the first `[CLS]` token. This token is added at the beginning of every document. The vector at the output of this token will be used by the BERT model for its sequence classification tasks: it serves as the input of the final, task-specific part of the neural network.\n",
    "- the input mask: the input mask tells the model which parts of the input it should look at and which parts it should ignore. In our example, we have made sure that every text has a length of 100 tokens. This means that some texts will be cut off after 100 tokens, while others will have to be padded with extra tokens. In this latter case, these padding tokens will receive a mask value of 0, which means BERT should not take them into account for its classification task. \n",
    "- the segment_ids: some NLP task take several sequences as input. This is the case for question answering, natural language inference, etc. In this case, the segment ids tell BERT which sequence every token belongs to. In a text classification task like ours, however, there's only one segment, so all the input tokens receive segment id 0.\n",
    "- the label id: the id of the label for this document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1101 11:58:05.087090 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.122189 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (613 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.158490 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1380 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.165140 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (593 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.186419 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (605 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.210195 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (915 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.228912 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1237 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.266400 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1126 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.305600 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2160 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.314717 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (723 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.328258 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (904 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.336314 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (611 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.363429 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (545 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.382237 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (622 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.408720 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (680 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.455323 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (732 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.504949 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2399 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.515107 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (524 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.540899 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (925 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.545575 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (524 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.551510 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (629 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.567061 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (626 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.576939 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (696 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.582462 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (626 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.625283 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2749 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.634007 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (529 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.644608 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (542 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.683824 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2444 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.716797 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (569 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.740634 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (522 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.785679 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (843 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.796592 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (558 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.801804 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (613 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.871086 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (950 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1101 11:58:05.876364 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (585 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.923251 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (3767 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.935417 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1023 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.947354 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (632 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:05.961196 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (694 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.004495 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (964 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.018391 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1181 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.028790 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (529 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.039705 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (605 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.056825 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1468 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.080770 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1891 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.140810 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1355 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.163167 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (635 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.215455 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1299 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.261300 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (597 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.266278 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (560 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.275294 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1250 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.289369 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (543 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.309119 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1488 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.324135 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (651 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.344534 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2328 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.349823 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (586 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.361002 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1224 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.374011 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (647 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.391790 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (746 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.398928 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (552 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.409263 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1359 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.438881 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (741 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.514507 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (578 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.533681 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (575 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.547131 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (789 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.573996 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1041 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.581871 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (684 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.592659 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (862 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1101 11:58:06.602647 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1059 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.663275 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1471 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.691465 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (719 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.715592 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1570 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.751939 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (517 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.787316 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (562 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.812062 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (605 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.845813 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (572 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.861477 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (569 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:06.954937 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (556 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.010853 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (523 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.036399 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (577 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.045826 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (789 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.068399 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2067 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.096223 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1099 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.105217 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (562 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.129151 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (840 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.142798 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (822 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.196090 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (663 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.204911 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (679 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.225714 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1134 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.233810 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (631 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.242085 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (558 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.262124 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1265 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.269417 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (638 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.303914 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (549 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.333623 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1078 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.356777 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (527 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.407447 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (961 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.424862 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (654 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.449445 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (638 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.458180 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (606 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.482141 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1064 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.502715 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (619 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1101 11:58:07.509202 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (521 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.528077 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (513 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.542267 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (608 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.554512 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (640 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.594978 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (689 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.616530 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (961 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.647893 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (597 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.720068 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (639 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.746891 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (818 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.758056 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1004 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.764601 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (584 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.770482 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (693 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.781407 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1306 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.791976 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (576 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.850841 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (520 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.890401 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (849 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.912446 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (895 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.951932 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1056 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:07.982511 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (3065 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.019877 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.069903 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (775 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.075320 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (569 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.120177 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1041 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.137918 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (565 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.201707 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (724 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.207045 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (617 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.255190 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (533 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.298442 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1660 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.347996 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (626 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.386141 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (576 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.406882 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.428283 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (608 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.476621 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (642 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.510965 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (951 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1101 11:58:08.533667 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1599 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.562301 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1142 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.575947 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (923 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.609409 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1170 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.671119 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (531 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.725161 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (688 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.743610 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2223 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.776786 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (699 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.812823 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (699 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.823746 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (951 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.842881 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1155 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.879555 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (553 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.919370 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1487 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.928882 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (893 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.938069 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (605 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:08.980506 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (525 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.017772 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1447 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.063471 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (589 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.118908 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1289 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.165007 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1078 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.172655 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (663 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.255758 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (708 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.260909 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (533 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.303549 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (656 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.337419 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (555 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.354185 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (839 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.360134 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (549 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.377962 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (821 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1101 11:58:09.397569 139947573684032 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (682 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import numpy as np\n",
    "\n",
    "logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s -   %(message)s',\n",
    "                    datefmt = '%m/%d/%Y %H:%M:%S',\n",
    "                    level = logging.INFO)\n",
    "logger = logging.getLogger(__name__)\n",
    "\n",
    "MAX_SEQ_LENGTH=100\n",
    "\n",
    "class BertInputItem(object):\n",
    "    \"\"\"An item with all the necessary attributes for finetuning BERT.\"\"\"\n",
    "\n",
    "    def __init__(self, text, input_ids, input_mask, segment_ids, label_id):\n",
    "        self.text = text\n",
    "        self.input_ids = input_ids\n",
    "        self.input_mask = input_mask\n",
    "        self.segment_ids = segment_ids\n",
    "        self.label_id = label_id\n",
    "        \n",
    "\n",
    "def convert_examples_to_inputs(example_texts, example_labels, label2idx, max_seq_length, tokenizer, verbose=0):\n",
    "    \"\"\"Loads a data file into a list of `InputBatch`s.\"\"\"\n",
    "    \n",
    "    input_items = []\n",
    "    examples = zip(example_texts, example_labels)\n",
    "    for (ex_index, (text, label)) in enumerate(examples):\n",
    "\n",
    "        # Create a list of token ids\n",
    "        input_ids = tokenizer.encode(f\"[CLS] {text} [SEP]\")\n",
    "        if len(input_ids) > max_seq_length:\n",
    "            input_ids = input_ids[:max_seq_length]\n",
    "\n",
    "        # All our tokens are in the first input segment (id 0).\n",
    "        segment_ids = [0] * len(input_ids)\n",
    "\n",
    "        # The mask has 1 for real tokens and 0 for padding tokens. Only real\n",
    "        # tokens are attended to.\n",
    "        input_mask = [1] * len(input_ids)\n",
    "\n",
    "        # Zero-pad up to the sequence length.\n",
    "        padding = [0] * (max_seq_length - len(input_ids))\n",
    "        input_ids += padding\n",
    "        input_mask += padding\n",
    "        segment_ids += padding\n",
    "\n",
    "        assert len(input_ids) == max_seq_length\n",
    "        assert len(input_mask) == max_seq_length\n",
    "        assert len(segment_ids) == max_seq_length\n",
    "\n",
    "        label_id = label2idx[label]\n",
    "\n",
    "        input_items.append(\n",
    "            BertInputItem(text=text,\n",
    "                          input_ids=input_ids,\n",
    "                          input_mask=input_mask,\n",
    "                          segment_ids=segment_ids,\n",
    "                          label_id=label_id))\n",
    "\n",
    "        \n",
    "    return input_items\n",
    "\n",
    "train_features = convert_examples_to_inputs(train_texts, train_labels, label2idx, MAX_SEQ_LENGTH, tokenizer, verbose=0)\n",
    "dev_features = convert_examples_to_inputs(dev_texts, dev_labels, label2idx, MAX_SEQ_LENGTH, tokenizer)\n",
    "test_features = convert_examples_to_inputs(test_texts, test_labels, label2idx, MAX_SEQ_LENGTH, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we're going to initialize a data loader for our training, development and testing data. This data loader puts all our data in tensors and will allow us to iterate over them during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import TensorDataset, DataLoader, SequentialSampler\n",
    "\n",
    "def get_data_loader(features, max_seq_length, batch_size, shuffle=True): \n",
    "\n",
    "    all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)\n",
    "    all_input_mask = torch.tensor([f.input_mask for f in features], dtype=torch.long)\n",
    "    all_segment_ids = torch.tensor([f.segment_ids for f in features], dtype=torch.long)\n",
    "    all_label_ids = torch.tensor([f.label_id for f in features], dtype=torch.long)\n",
    "    data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids)\n",
    "\n",
    "    dataloader = DataLoader(data, shuffle=shuffle, batch_size=batch_size)\n",
    "    return dataloader\n",
    "\n",
    "BATCH_SIZE = 16\n",
    "\n",
    "train_dataloader = get_data_loader(train_features, MAX_SEQ_LENGTH, BATCH_SIZE, shuffle=True)\n",
    "dev_dataloader = get_data_loader(dev_features, MAX_SEQ_LENGTH, BATCH_SIZE, shuffle=False)\n",
    "test_dataloader = get_data_loader(test_features, MAX_SEQ_LENGTH, BATCH_SIZE, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to write our evaluation method. This method takes as input a model and a data loader with the data we would like to evaluate on. For each batch, it computes the output of the model and the loss. We use this output to compute the obtained precision, recall and F-score. During training, we will print the simple numbers. When we evaluate on the test set, we will output a full classification report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, dataloader):\n",
    "    model.eval()\n",
    "    \n",
    "    eval_loss = 0\n",
    "    nb_eval_steps = 0\n",
    "    predicted_labels, correct_labels = [], []\n",
    "\n",
    "    for step, batch in enumerate(tqdm(dataloader, desc=\"Evaluation iteration\")):\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        input_ids, input_mask, segment_ids, label_ids = batch\n",
    "\n",
    "        with torch.no_grad():\n",
    "            tmp_eval_loss, logits = model(input_ids, attention_mask=input_mask,\n",
    "                                          token_type_ids=segment_ids, labels=label_ids)\n",
    "\n",
    "        outputs = np.argmax(logits.to('cpu'), axis=1)\n",
    "        label_ids = label_ids.to('cpu').numpy()\n",
    "        \n",
    "        predicted_labels += list(outputs)\n",
    "        correct_labels += list(label_ids)\n",
    "        \n",
    "        eval_loss += tmp_eval_loss.mean().item()\n",
    "        nb_eval_steps += 1\n",
    "\n",
    "    eval_loss = eval_loss / nb_eval_steps\n",
    "    \n",
    "    correct_labels = np.array(correct_labels)\n",
    "    predicted_labels = np.array(predicted_labels)\n",
    "        \n",
    "    return eval_loss, correct_labels, predicted_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to start training. We're going to use the AdamW optimizer with a base learning rate of 5e-5, and train for a maximum of 100 epochs. Here are some additional things to note: \n",
    "\n",
    "- Gradient Accumulation allows us to keep our batches small enough to fit into the memory of our GPU, while getting the advantages of using larger batch sizes. In practice, it means we sum the gradients of several batches, before we perform a step of gradient descent. \n",
    "- We use the WarmupLinearScheduler to vary our learning rate during the training process. First, we're going to start with a small learning rate, which increases linearly during the warmup stage. Afterwards it slowly decreases again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.optimization import AdamW, WarmupLinearSchedule\n",
    "\n",
    "GRADIENT_ACCUMULATION_STEPS = 1\n",
    "NUM_TRAIN_EPOCHS = 20\n",
    "LEARNING_RATE = 5e-5\n",
    "WARMUP_PROPORTION = 0.1\n",
    "MAX_GRAD_NORM = 5\n",
    "\n",
    "num_train_steps = int(len(train_dataloader.dataset) / BATCH_SIZE / GRADIENT_ACCUMULATION_STEPS * NUM_TRAIN_EPOCHS)\n",
    "num_warmup_steps = int(WARMUP_PROPORTION * num_train_steps)\n",
    "\n",
    "param_optimizer = list(model.named_parameters())\n",
    "no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']\n",
    "optimizer_grouped_parameters = [\n",
    "    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},\n",
    "    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n",
    "    ]\n",
    "\n",
    "optimizer = AdamW(optimizer_grouped_parameters, lr=LEARNING_RATE, correct_bias=False)\n",
    "scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_train_steps)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're finally ready to train our model. At each epoch, we're going to train it on our training data and evaluate it on the development data. We keep a history of the loss, and stop training when the loss on the development set doesn't improve for a certain number of steps (we call this number our `patience`). Whenever the development loss of our model improves, we save it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d92cc226eb8425aa09d7bd1b47da8f0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=152, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4c7f5873fe34fbb8713b28004fb1101",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=17, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: []\n",
      "Dev loss: 0.760436054538278\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:   5%|▌         | 1/20 [00:34<10:47, 34.10s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8b00efa370a549df8287646602e2f74e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=152, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4ed4913144a4c39be0675c57d87389d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=17, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  10%|█         | 2/20 [01:07<10:11, 33.97s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.760436054538278]\n",
      "Dev loss: 0.7900203220984515\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c52eaff1fa043b4a54bac7d6b4a8b1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=152, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1dbbaf99b15245f0a28214e7213cd7b1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=17, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.760436054538278, 0.7900203220984515]\n",
      "Dev loss: 0.7881268595947939\n",
      "No improvement on development set. Finish training.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import os\n",
    "from tqdm import trange\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "from sklearn.metrics import classification_report, precision_recall_fscore_support\n",
    "\n",
    "OUTPUT_DIR = \"/tmp/\"\n",
    "MODEL_FILE_NAME = \"pytorch_model.bin\"\n",
    "PATIENCE = 2\n",
    "\n",
    "loss_history = []\n",
    "no_improvement = 0\n",
    "for _ in trange(int(NUM_TRAIN_EPOCHS), desc=\"Epoch\"):\n",
    "    model.train()\n",
    "    tr_loss = 0\n",
    "    nb_tr_examples, nb_tr_steps = 0, 0\n",
    "    for step, batch in enumerate(tqdm(train_dataloader, desc=\"Training iteration\")):\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        input_ids, input_mask, segment_ids, label_ids = batch\n",
    "\n",
    "        outputs = model(input_ids, attention_mask=input_mask, token_type_ids=segment_ids, labels=label_ids)\n",
    "        loss = outputs[0]\n",
    "\n",
    "        if GRADIENT_ACCUMULATION_STEPS > 1:\n",
    "            loss = loss / GRADIENT_ACCUMULATION_STEPS\n",
    "\n",
    "        loss.backward()\n",
    "        tr_loss += loss.item()\n",
    "\n",
    "        if (step + 1) % GRADIENT_ACCUMULATION_STEPS == 0:\n",
    "            torch.nn.utils.clip_grad_norm_(model.parameters(), MAX_GRAD_NORM)  \n",
    "            \n",
    "            optimizer.step()\n",
    "            optimizer.zero_grad()\n",
    "            scheduler.step()\n",
    "            \n",
    "    dev_loss, _, _ = evaluate(model, dev_dataloader)\n",
    "    \n",
    "    print(\"Loss history:\", loss_history)\n",
    "    print(\"Dev loss:\", dev_loss)\n",
    "    \n",
    "    if len(loss_history) == 0 or dev_loss < min(loss_history):\n",
    "        no_improvement = 0\n",
    "        model_to_save = model.module if hasattr(model, 'module') else model\n",
    "        output_model_file = os.path.join(OUTPUT_DIR, MODEL_FILE_NAME)\n",
    "        torch.save(model_to_save.state_dict(), output_model_file)\n",
    "    else:\n",
    "        no_improvement += 1\n",
    "    \n",
    "    if no_improvement >= PATIENCE: \n",
    "        print(\"No improvement on development set. Finish training.\")\n",
    "        break\n",
    "        \n",
    "    \n",
    "    loss_history.append(dev_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now evaluate the model on some documents it has never seen. We'll load our best model and have it predict the labels for all documents in our data. We'll compute its precision, recall and F-score for the training, development and test set and print a full classification report for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1101 11:59:51.784697 139947573684032 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /home/yves/.cache/torch/transformers/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.bf3b9ea126d8c0001ee8a1e8b92229871d06d36d8808208cc2449280da87785c\n",
      "I1101 11:59:51.786259 139947573684032 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "I1101 11:59:52.310831 139947573684032 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ab2e8185c30a4209bc4a540c59e74d03",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=152, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ac6d81f84d545b2af51fc9e77474203",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=17, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "71ecc74b0a1f44119fbe72b1834fab7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=19, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training performance: (0.7563786008230453, 0.7563786008230453, 0.7563786008230453, None)\n",
      "Development performance: (0.662962962962963, 0.662962962962963, 0.662962962962963, None)\n",
      "Test performance: (0.7, 0.7, 0.7, None)\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        pos       0.76      0.72      0.74        95\n",
      "        neg       0.73      0.71      0.72        93\n",
      "      mixed       0.63      0.68      0.65       112\n",
      "\n",
      "avg / total       0.70      0.70      0.70       300\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_state_dict = torch.load(os.path.join(OUTPUT_DIR, MODEL_FILE_NAME), map_location=lambda storage, loc: storage)\n",
    "model = BertForSequenceClassification.from_pretrained(BERT_MODEL, state_dict=model_state_dict, num_labels = len(target_names))\n",
    "model.to(device)\n",
    "\n",
    "model.eval()\n",
    "\n",
    "_, train_correct, train_predicted = evaluate(model, train_dataloader)\n",
    "_, dev_correct, dev_predicted = evaluate(model, dev_dataloader)\n",
    "_, test_correct, test_predicted = evaluate(model, test_dataloader)\n",
    "\n",
    "print(\"Training performance:\", precision_recall_fscore_support(train_correct, train_predicted, average=\"micro\"))\n",
    "print(\"Development performance:\", precision_recall_fscore_support(dev_correct, dev_predicted, average=\"micro\"))\n",
    "print(\"Test performance:\", precision_recall_fscore_support(test_correct, test_predicted, average=\"micro\"))\n",
    "\n",
    "bert_accuracy = np.mean(test_predicted == test_correct)\n",
    "\n",
    "print(classification_report(test_correct, test_predicted, target_names=target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that BERT obtains an accuracy of around 70% on the test data. This is around 8% more than our initial baseline classifier. This confirms that BERT's transfer learning helps us achieve significantly higher accuracies for small datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f46ec397128>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa8AAAEcCAYAAABwNTvaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFRZJREFUeJzt3X+w3XV95/HnqzdhUwLFAFenEDBpN7VEjCJXQN1aXGUnjEqQli5oHQhKhtW03bqWoqvYiXTGuqud0WZW0i7+WEYjhdKJbZSFRYurUBMsK4UQzGBsLrvd3gYKZTWSwHv/yEk4XG9yDzcnOflwno+ZO/l+Pt9Pvt/3zeTmlc/nfM73pKqQJKklPzXoAiRJeq4ML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzZg3qxscff3wtWLBgULeXJB2G7r777n+sqtHpxg0svBYsWMDGjRsHdXtJ0mEoyQ96GeeyoSSpOYaXJKk5hpckqTkDe81Lklq2c+dOxsfH2bFjx6BLadKcOXOYP38+s2fPntHvN7wkaQbGx8c5+uijWbBgAUkGXU5Tqort27czPj7OwoULZ3SNnpYNkyxNsjnJliRXTXH+D5Pc0/l6MMk/zagaSWrEjh07OO644wyuGUjCcccdd0Cz1mlnXklGgNXAOcA4sCHJuqq6f8+YqvrtrvG/AZw244okqREG18wd6J9dLzOvM4AtVfVQVT0JrAWW7Wf8xcAXD6gqSZL2o5fXvE4EtnW1x4EzpxqY5MXAQuD2Ay9Nktqx4Kq/7Ov1tn70TX293vNNvzdsXATcWFVPTXUyyQpgBcDJJ5/c51sfev3+y6qZ8YdcOnh27drFrFmH396+XpYNHwZO6mrP7/RN5SL2s2RYVWuqaqyqxkZHp310lSRpP84//3xOP/10XvrSl7JmzRoAvvrVr/LKV76Sl7/85bzhDW8A4IknnmD58uW87GUvY8mSJdx0000AHHXUUXuvdeONN3LppZcCcOmll3LFFVdw5plncuWVV/Ltb3+bV7/61Zx22mm85jWvYfPmzQA89dRTvO997+PUU09lyZIlfOpTn+L222/n/PPP33vdW2+9lbe+9a19/957idMNwKIkC9kdWhcBb5s8KMkvAvOAO/taoSRpStdddx3HHnssP/rRj3jVq17FsmXLuPzyy7njjjtYuHAhjzzyCAAf+chHOOaYY7j33nsBePTRR6e99vj4ON/61rcYGRnh8ccf5xvf+AazZs3itttu4wMf+AA33XQTa9asYevWrdxzzz3MmjWLRx55hHnz5vHud7+biYkJRkdH+cxnPsNll13W9+992vCqql1JVgK3ACPAdVV1X5JVwMaqWtcZehGwtqqq71VKkn7CJz/5SW6++WYAtm3bxpo1a3jd6163971Txx57LAC33XYba9eu3fv75s2bN+21L7zwQkZGRgB47LHHuOSSS/je975HEnbu3Ln3uldcccXeZcU993vHO97B9ddfz/Lly7nzzjv5/Oc/36fv+Bk9LWRW1Xpg/aS+qye1f69/ZUmS9ufrX/86t912G3feeSdHHnkkZ599Nq94xSt44IEHer5G93b1ye+5mjt37t7jD33oQ7z+9a/n5ptvZuvWrZx99tn7ve7y5ct5y1vewpw5c7jwwgsPymtmPttQkhr02GOPMW/ePI488kgeeOAB7rrrLnbs2MEdd9zB97//fYC9y4bnnHMOq1ev3vt79ywbvuhFL2LTpk08/fTTe2dw+7rXiSeeCMBnP/vZvf3nnHMO1157Lbt27XrW/U444QROOOEErrnmGpYvX96/b7rL4beFRJIadKh3vS5dupRPf/rTnHLKKbzkJS/hrLPOYnR0lDVr1nDBBRfw9NNP88IXvpBbb72VD37wg7znPe/h1FNPZWRkhA9/+MNccMEFfPSjH+XNb34zo6OjjI2N8cQTT0x5ryuvvJJLLrmEa665hje96Znv813vehcPPvggS5YsYfbs2Vx++eWsXLkSgLe//e1MTExwyimnHJTvP4N6iWpsbKxa/zBKt8ofHtwqr0HYtGnTQfuH+flg5cqVnHbaabzzne/c55ip/gyT3F1VY9Nd35mXJKmvTj/9dObOncvHP/7xg3YPw0uS1Fd33333Qb+HGzYkaYZ8Z9DMHeifneElSTMwZ84ctm/fboDNwJ7P85ozZ86Mr+GyoSTNwPz58xkfH2diYmLQpTRpzycpz5ThJUkzMHv27Bl/CrAOnMuGkqTmGF6SpOYYXpKk5hhekqTmGF6SpOYYXpKk5hhekqTm+D4vSQfMT1g4PAzTJyw485IkNcfwkiQ1x/CSJDXH8JIkNaen8EqyNMnmJFuSXLWPMb+W5P4k9yX5Qn/LlCTpGdPuNkwyAqwGzgHGgQ1J1lXV/V1jFgHvB15bVY8meeHBKliSpF5mXmcAW6rqoap6ElgLLJs05nJgdVU9ClBV/9DfMiVJekYv4XUisK2rPd7p6/YLwC8k+WaSu5IsnepCSVYk2Zhkox/gJkmaqX5t2JgFLALOBi4G/jjJCyYPqqo1VTVWVWOjo6N9urUkadj0El4PAyd1ted3+rqNA+uqamdVfR94kN1hJklS3/USXhuARUkWJjkCuAhYN2nMn7N71kWS49m9jPhQH+uUJGmvacOrqnYBK4FbgE3ADVV1X5JVSc7rDLsF2J7kfuBrwO9U1faDVbQkabj19GDeqloPrJ/Ud3XXcQHv7XxJknRQ+YQNSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnN6Cq8kS5NsTrIlyVVTnL80yUSSezpf7+p/qZIk7TZrugFJRoDVwDnAOLAhybqqun/S0C9V1cqDUKMkSc/Sy8zrDGBLVT1UVU8Ca4FlB7csSZL2rZfwOhHY1tUe7/RN9itJvpvkxiQn9aU6SZKm0K8NG18GFlTVEuBW4HNTDUqyIsnGJBsnJib6dGtJ0rDpJbweBrpnUvM7fXtV1faq+nGn+SfA6VNdqKrWVNVYVY2Njo7OpF5JknoKrw3AoiQLkxwBXASs6x6Q5Ge7mucBm/pXoiRJzzbtbsOq2pVkJXALMAJcV1X3JVkFbKyqdcBvJjkP2AU8Alx6EGuWJA25acMLoKrWA+sn9V3ddfx+4P39LU2SpKn5hA1JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzDC9JUnMML0lScwwvSVJzegqvJEuTbE6yJclV+xn3K0kqyVj/SpQk6dmmDa8kI8Bq4FxgMXBxksVTjDsa+C3gr/tdpCRJ3XqZeZ0BbKmqh6rqSWAtsGyKcR8B/gDY0cf6JEn6Cb2E14nAtq72eKdvrySvBE6qqr/c34WSrEiyMcnGiYmJ51ysJEnQhw0bSX4K+ATwH6YbW1VrqmqsqsZGR0cP9NaSpCHVS3g9DJzU1Z7f6dvjaOBU4OtJtgJnAevctCFJOlh6Ca8NwKIkC5McAVwErNtzsqoeq6rjq2pBVS0A7gLOq6qNB6ViSdLQmza8qmoXsBK4BdgE3FBV9yVZleS8g12gJEmTzeplUFWtB9ZP6rt6H2PPPvCyJEnaN5+wIUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWpOT+GVZGmSzUm2JLlqivNXJLk3yT1J/meSxf0vVZKk3aYNryQjwGrgXGAxcPEU4fSFqnpZVb0C+Bjwib5XKklSRy8zrzOALVX1UFU9CawFlnUPqKrHu5pzgepfiZIkPdusHsacCGzrao8DZ04elOQ9wHuBI4B/3ZfqJEmaQt82bFTV6qr6eeB3gQ9ONSbJiiQbk2ycmJjo160lSUOml/B6GDipqz2/07cva4HzpzpRVWuqaqyqxkZHR3uvUpKkLr2E1wZgUZKFSY4ALgLWdQ9Isqir+Sbge/0rUZKkZ5v2Na+q2pVkJXALMAJcV1X3JVkFbKyqdcDKJG8EdgKPApcczKIlScOtlw0bVNV6YP2kvqu7jn+rz3VJkrRPPmFDktQcw0uS1BzDS5LUHMNLktQcw0uS1BzDS5LUHMNLktQcw0uS1BzDS5LUHMNLktQcw0uS1BzDS5LUHMNLktQcw0uS1BzDS5LUHMNLktQcw0uS1BzDS5LUHMNLktQcw0uS1BzDS5LUnJ7CK8nSJJuTbEly1RTn35vk/iTfTfI/kry4/6VKkrTbtOGVZARYDZwLLAYuTrJ40rC/AcaqaglwI/CxfhcqSdIevcy8zgC2VNVDVfUksBZY1j2gqr5WVT/sNO8C5ve3TEmSntFLeJ0IbOtqj3f69uWdwFcOpChJkvZnVj8vluTXgTHgl/dxfgWwAuDkk0/u560lSUOkl5nXw8BJXe35nb5nSfJG4D8C51XVj6e6UFWtqaqxqhobHR2dSb2SJPUUXhuARUkWJjkCuAhY1z0gyWnAtewOrn/of5mSJD1j2vCqql3ASuAWYBNwQ1Xdl2RVkvM6w/4TcBTwp0nuSbJuH5eTJOmA9fSaV1WtB9ZP6ru66/iNfa5LkqR98gkbkqTmGF6SpOYYXpKk5hhekqTmGF6SpOYYXpKk5hhekqTmGF6SpOYYXpKk5hhekqTmGF6SpOYYXpKk5hhekqTmGF6SpOYYXpKk5hhekqTmGF6SpOYYXpKk5hhekqTmGF6SpOYYXpKk5vQUXkmWJtmcZEuSq6Y4/7ok30myK8mv9r9MSZKeMW14JRkBVgPnAouBi5MsnjTs74BLgS/0u0BJkiab1cOYM4AtVfUQQJK1wDLg/j0Dqmpr59zTB6FGSZKepZdlwxOBbV3t8U7fc5ZkRZKNSTZOTEzM5BKSJB3aDRtVtaaqxqpqbHR09FDeWpL0PNJLeD0MnNTVnt/pkyRpIHoJrw3AoiQLkxwBXASsO7hlSZK0b9OGV1XtAlYCtwCbgBuq6r4kq5KcB5DkVUnGgQuBa5PcdzCLliQNt152G1JV64H1k/qu7jrewO7lREmSDjqfsCFJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqjuElSWqO4SVJao7hJUlqTk/hlWRpks1JtiS5aorz/yLJlzrn/zrJgn4XKknSHtOGV5IRYDVwLrAYuDjJ4knD3gk8WlX/EvhD4A/6XagkSXv0MvM6A9hSVQ9V1ZPAWmDZpDHLgM91jm8E3pAk/StTkqRn9BJeJwLbutrjnb4px1TVLuAx4Lh+FChJ0mSzDuXNkqwAVnSaTyTZfCjvrykdD/zjoIs4EHGRWv3hz8Lh4cW9DOolvB4GTupqz+/0TTVmPMks4Bhg++QLVdUaYE0vhenQSLKxqsYGXYc0aP4stKWXZcMNwKIkC5McAVwErJs0Zh1wSef4V4Hbq6r6V6YkSc+YduZVVbuSrARuAUaA66rqviSrgI1VtQ74r8B/S7IFeITdASdJ0kERJ0jDLcmKznKuNNT8WWiL4SVJao6Ph5IkNcfwkiQ1x/AaIp23MUhS8wyv4fLtQRcgHW6S/KskyzvHo0kWDromTc/wGi4+b1LqkuTDwO8C7+90zQauH1xF6pXLSMNlNMl793Wyqj5xKIuRDgNvBU4DvgNQVf87ydGDLUm9MLyGywhwFM7ApD2erKpKUgBJ5g66IPXG8Bou/6eqVg26COkwckOSa4EXJLkcuAz44wHXpB74JuUhkuRvquq0QdchHU6SnAP8G3avSNxSVbcOuCT1wPAaIkmOrapHpuh/AfCeqvr9AZQlSc+Zuw2Hy9wka5L8RZJ3JZmb5OPAg8ALB12cdKgluSDJ95I8luTxJP+c5PFB16XpOfMaIkm+BvwVcCewtPN1D/DbVfX3g6xNGoTOJ2G8pao2DboWPTeG1xBJ8r+q6uVd7XHg5Kp6eoBlSQOT5JtV9dpB16Hnzt2GQybJPJ7ZKr8dOCZJAKZ6PUx6ntuY5EvAnwM/3tNZVX82uJLUC2deQyTJVuBppn6fV1XVzx3aiqTBSvKZKbqrqi475MXoOTG8JEnNcdlwiCT59aq6vnP82qr6Zte5lVX1R4OrTjp0klxZVR9L8ingJ/4HX1W/OYCy9BwYXsPlvTzz0NFPAa/sOncZYHhpWOzZXbhxoFVoxgyv4ZJ9HE/Vlp63qurLnV8/N+haNDOG13CpfRxP1Zaet5J8mf38na+q8w5hOZoBN2wMkSQ/BLawe5b1851jOu2fqyqfqK2hkOSX93e+qv7qUNWimTG8hkiSF+/vfFX94FDVIh0ukvw0u9+sv3nQtah3hteQS3I8sL38i6AhlOQtwH8GjqiqhUleAaxy2fDw54N5h0iSs5J8PcmfJTktyd8Cfwv83yRLB12fNAC/B5wB/BNAVd0DLBxkQeqNGzaGyx8BHwCOAW4Hzq2qu5L8IvBF4KuDLE4agJ1V9VjnCWl7uArRAGdew2VWVf33qvpT4O+r6i6AqnpgwHVJg3JfkrcBI0kWdd60/K1BF6XpGV7Dpfvp8T+adM7/bWoY/QbwUnY/lPeLwOPAvx9oReqJGzaGSJKngP/H7q3xPw38cM8pYE5VzR5UbdKgJRkB5laVH0bZAGdeQ6SqRqrqZ6rq6Kqa1Tne0za4NHSSfCHJzySZC9wL3J/kdwZdl6ZneEkaZos7M63zga+we6fhOwZbknpheEkaZrOTzGZ3eK2rqp34+m8TDC9Jw+xaYCswF7ij8xQaX/NqgBs2JKlLkllVtWvQdWj/fJOypKGW5E3s3i4/p6t71YDKUY9cNpQ0tJJ8Gvi37H6/V4ALgf0+wFqHB5cNJQ2tJN+tqiVdvx4FfKWqfmnQtWn/nHlJGmZ7njTzwyQnADuBnx1gPeqRr3lJGmZ/keQFwMeAuzt9fzLAetQjlw0lDa3OB1H+O+CX2P3+rm8A/6Wqdgy0ME3L8JI0tJLcAPwzcH2n623AMVX1a4OrSr0wvCQNrST3V9Xi6fp0+HHDhqRh9p0kZ+1pJDkT2DjAetQjN2xIGjpJ7mX3a1yzgW8l+btO+8WAH87aAJcNJQ2dzjMM96mqfnCoatHMGF6SpOb4mpckqTmGlySpOYaXJKk5hpckqTmGlySpOf8fczaqcAuB93MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 504x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df = pd.DataFrame({\"accuracy\": {\"baseline\": baseline_accuracy, \"BERT\": bert_accuracy}})\n",
    "plt.rcParams['figure.figsize'] = (7,4)\n",
    "df.plot(kind=\"bar\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}