Working BiLSTM CRF but needs explanations

rguthrie3 · rguthrie3 · commit 3e85a4f2107a · 2017-03-06T01:32:36.000-05:00
diff --git a/Deep Learning for Natural Language Processing with Pytorch.ipynb b/Deep Learning for Natural Language Processing with Pytorch.ipynb
@@ -13,18 +13,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 2,
    "metadata": {
     "collapsed": false
    },
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "<torch._C.Generator at 0x7f8c23c1a678>"
+       "<torch._C.Generator at 0x7f3085495af8>"
       ]
      },
-     "execution_count": 1,
+     "execution_count": 2,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -665,15 +665,14 @@
     "\n",
     "All network components should inherit from nn.Module and override the forward() method.  That is about it, as far as the boilerplate is concerned.  Inheriting from nn.Module provides functionality to your component.  For example, it makes it keep track of its trainable parameters, you can swap it between CPU and GPU with the .cuda() or .cpu() functions, etc.\n",
     "\n",
-    "Let's write an annotated example of a network that takes in a sparse bag-of-words representation and outputs a probability distribution over two labels: \"English\" and \"Spanish\".\n",
-    "\n",
-    "Note: This is just for demonstration, so that we can build Pytorch components in later sections and you will know what is going on.  Handing in a sparse bag-of-words representation is not how you would actually want to do things.  There are better ways to do text classification.  I made up this model to be extremely simple and to not use word embeddings (which we don't introduce until the next section)."
+    "Let's write an annotated example of a network that takes in a sparse bag-of-words representation and outputs a probability distribution over two labels: \"English\" and \"Spanish\".  This model is just logistic regression."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "### Example: Logistic Regression Bag-of-Words classifier\n",
     "Our model will map a sparse BOW representation to log probabilities over labels.  We assign each word in the vocab an index.  For example, say our entire vocab is two words \"hello\" and \"world\", with indices 0 and 1 respectively.\n",
     "The BoW vector for the sentence \"hello hello hello hello\" is\n",
     "$$ \\left[ 4, 0 \\right] $$\n",
@@ -691,7 +690,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 89,
+   "execution_count": 68,
    "metadata": {
     "collapsed": false
    },
@@ -728,7 +727,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 90,
+   "execution_count": 69,
    "metadata": {
     "collapsed": false
    },
@@ -760,14 +759,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 91,
+   "execution_count": 70,
    "metadata": {
     "collapsed": true
    },
    "outputs": [],
    "source": [
     "def make_bow_vector(sentence, word_to_ix):\n",
-    "    vec = torch.Tensor([0] * len(word_to_ix))\n",
+    "    vec = torch.zeros(len(word_to_ix))\n",
     "    for word in sentence:\n",
     "        vec[word_to_ix[word]] += 1\n",
     "    return vec.view(1, -1)\n",
@@ -778,7 +777,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 92,
+   "execution_count": 71,
    "metadata": {
     "collapsed": false
    },
@@ -790,21 +789,21 @@
       "Parameter containing:\n",
       "\n",
       "Columns 0 to 9 \n",
-      " 0.0058  0.1147  0.1744 -0.1844  0.0339  0.1503  0.1582  0.0160 -0.1422 -0.0204\n",
-      " 0.1503  0.0746  0.0485  0.0580  0.0984 -0.0573 -0.0593  0.1032 -0.0902 -0.0563\n",
+      "-0.0650  0.0114  0.0729  0.0944  0.1127 -0.0445 -0.0844 -0.0186 -0.0314  0.1862\n",
+      " 0.1257 -0.1513  0.0712 -0.0120 -0.0240 -0.0998 -0.0409  0.0621  0.1474 -0.1760\n",
       "\n",
       "Columns 10 to 19 \n",
-      "-0.1415  0.1538  0.1206 -0.0480 -0.0401  0.0151 -0.1313  0.0597  0.1677 -0.0544\n",
-      " 0.1553  0.0992 -0.0282  0.1496  0.1823 -0.1915  0.0641 -0.0007  0.0477 -0.1672\n",
+      " 0.1500  0.0017  0.1036 -0.0334 -0.0844 -0.0557  0.0878 -0.0663 -0.1855  0.0172\n",
+      "-0.1025 -0.1243 -0.0416  0.0733 -0.1398 -0.0462  0.0676  0.1198 -0.1272 -0.1517\n",
       "\n",
       "Columns 20 to 25 \n",
-      "-0.0597  0.0279  0.0984  0.0541  0.0886 -0.1466\n",
-      "-0.1511  0.1126  0.1763 -0.1710 -0.0196 -0.0568\n",
+      " 0.0248 -0.1245 -0.1800 -0.1680 -0.1467 -0.0838\n",
+      " 0.1960 -0.1035  0.1822 -0.0159 -0.1695  0.1666\n",
       "[torch.FloatTensor of size 2x26]\n",
       "\n",
       "Parameter containing:\n",
-      " 0.0307\n",
-      " 0.1733\n",
+      "-0.1683\n",
+      "-0.0214\n",
       "[torch.FloatTensor of size 2]\n",
       "\n"
      ]
@@ -871,7 +870,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "So lets train!  To do this, we pass instances through to get log probabilities, compute a loss function, compute the gradient of the loss function, and then update the parameters with a gradient step.  Loss functions are provided by Torch in the nn package.  nn.NLLLoss() is the negative log likelihood loss we want.  It also defines optimization functions in torch.optim.  Here, we will just use SGD."
+    "So lets train!  To do this, we pass instances through to get log probabilities, compute a loss function, compute the gradient of the loss function, and then update the parameters with a gradient step.  Loss functions are provided by Torch in the nn package.  nn.NLLLoss() is the negative log likelihood loss we want.  It also defines optimization functions in torch.optim.  Here, we will just use SGD.\n",
+    "\n",
+    "Note that the *input* to NLLLoss is a vector of log probabilities, and a target label.  It doesn't compute the log probabilities for us.  This is why the last layer of our network is log softmax.\n",
+    "The loss function nn.CrossEntropyLoss() is the same as NLLLoss(), except it does the log softmax for you."
    ]
   },
   {
@@ -1193,10 +1195,9 @@
     "        \n",
     "    def forward(self, inputs):\n",
     "        embeds = self.embeddings(inputs).view((1, -1))\n",
-    "        log_probs = F.log_softmax(\n",
-    "                        self.linear2(\n",
-    "                            F.relu(\n",
-    "                                self.linear1(embeds))))\n",
+    "        out = F.relu(self.linear1(embeds))\n",
+    "        out = self.linear2(out)\n",
+    "        log_probs = F.log_softmax(out)\n",
     "        return log_probs"
    ]
   },
@@ -1706,7 +1707,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# 8. Making Dynamic Decisions: Structure Prediction (Coming Soon!)"
+    "# 8. Advanced: Making Dynamic Decisions and the Bi-LSTM CRF (WIP)"
    ]
   },
   {
@@ -1742,11 +1743,156 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "For this section, we will see a full, complicated example of a Bi-LSTM Conditional Random Field for named-entity recognition.  Familiarity with CRF's is assumed.  Although this name sounds scary, all the model is is a CRF but where an LSTM provides the features.\n",
+    "For this section, we will see a full, complicated example of a Bi-LSTM Conditional Random Field for named-entity recognition.  Familiarity with CRF's is assumed.  Although this name sounds scary, all the model is is a CRF but where an LSTM provides the features.  This is an advanced model though, far more complicated than any earlier model in this tutorial.  If you want to skip it, that is fine.\n",
+    "\n",
+    "TODO explain BiLSTM CRF Here"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 67,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(Variable containing:\n",
+       "  1.8765\n",
+       " [torch.FloatTensor of size 1], [2, 1, 2])"
+      ]
+     },
+     "execution_count": 67,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Work in progress.  Needs extensive commenting but it runs.\n",
+    "\n",
+    "\n",
+    "def to_scalar(var):\n",
+    "    return var.view(-1).data.tolist()[0]\n",
     "\n",
-    "Let $\\textbf{y}$ be a tag sequence, and $\\textbf{w}$ a sequence of words.  Recall that the CRF wants to compute\n",
-    "$$ P(\\textbf{y} | \\textbf{w}) = \\frac{ \\exp{ ( \\sum_i f(y_{i-1}, y_i, i, \\textbf{w}) \\cdot \\theta ) }}\n",
-    "{\\sum_{\\textbf{y'}} \\exp{ ( \\sum_j f(y'_{j-1}, y'_j, j, \\textbf{w} \\cdot \\theta } )  } $$"
+    "def argmax(vec):\n",
+    "    _, idx = torch.max(vec, 1)\n",
+    "    return to_scalar(idx)\n",
+    "\n",
+    "def log_sum_exp(vec):\n",
+    "    max_score = vec[0][argmax(vec)]\n",
+    "    max_score_broadcast = max_score.expand(vec.size()[1])\n",
+    "    return max_score + torch.log(torch.sum(torch.exp(vec - max_score_broadcast)))\n",
+    "    \n",
+    "\n",
+    "class BiLSTM_CRF(nn.Module):\n",
+    "    \n",
+    "    def __init__(self, vocab_size, tag_to_ix, embedding_dim, hidden_dim):\n",
+    "        super(BiLSTM_CRF, self).__init__()\n",
+    "        self.embedding_dim = embedding_dim\n",
+    "        self.hidden_dim = hidden_dim\n",
+    "        self.vocab_size = vocab_size\n",
+    "        self.tag_to_ix = tag_to_ix\n",
+    "        self.tagset_size = len(tag_to_ix)\n",
+    "        \n",
+    "        self.word_embeds = nn.Embedding(vocab_size, embedding_dim)\n",
+    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim/2, num_layers=1, bidirectional=True)\n",
+    "        self.hidden2tag = nn.Linear(hidden_dim, self.tagset_size)\n",
+    "        self.transitions = nn.Parameter(torch.randn(self.tagset_size, self.tagset_size))\n",
+    "        \n",
+    "        self.hidden = self.init_hidden()\n",
+    "        \n",
+    "    def init_hidden(self):\n",
+    "        return ( autograd.Variable( torch.randn(2, 1, self.hidden_dim)),\n",
+    "                 autograd.Variable( torch.randn(2, 1, self.hidden_dim)) )\n",
+    "    \n",
+    "    \n",
+    "    def _forward_alg(self, feats):\n",
+    "        init_alphas = torch.Tensor(1, self.tagset_size).fill_(-10000.)\n",
+    "        init_alphas[0][self.tag_to_ix[START_TAG]] = 0.\n",
+    "        \n",
+    "        forward_var = autograd.Variable(init_alphas)\n",
+    "        \n",
+    "        for feat in feats:\n",
+    "            alphas_t = []\n",
+    "            for next_tag in xrange(self.tagset_size):\n",
+    "                emit_score = feat[next_tag].expand(self.tagset_size)\n",
+    "                trans_score = self.transitions[next_tag]\n",
+    "                next_tag_var = forward_var + trans_score + emit_score\n",
+    "                alphas_t.append(log_sum_exp(next_tag_var))\n",
+    "            forward_var = torch.cat(alphas_t).view(1, -1)\n",
+    "        terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]\n",
+    "        alpha = log_sum_exp(terminal_var)\n",
+    "        return alpha\n",
+    "        \n",
+    "    def _get_lstm_features(self, sentence):\n",
+    "        embeds = self.word_embeds(sentence).view(len(sentence), 1, -1)\n",
+    "        lstm_out, self.hidden = self.lstm(embeds)\n",
+    "        lstm_out = lstm_out.view(len(sentence), self.hidden_dim)\n",
+    "        lstm_feats = self.hidden2tag(lstm_out)\n",
+    "        return lstm_feats\n",
+    "        \n",
+    "    def _score_sentence(self, feats, tags):\n",
+    "        score = autograd.Variable( torch.Tensor([0]) )\n",
+    "        tags = [self.tag_to_ix[START_TAG]] + tags\n",
+    "        for i, feat in enumerate(feats):\n",
+    "            score = score + self.transitions[tags[i+1], tags[i]] + feat[i+1]\n",
+    "        score = score + self.transitions[self.tag_to_ix[STOP_TAG]][tags[-1]]\n",
+    "        return score\n",
+    "    \n",
+    "    def _viterbi_decode(self, feats):\n",
+    "        backpointers = []\n",
+    "        init_vvars = torch.Tensor(1, self.tagset_size).fill_(-10000.)\n",
+    "        init_vvars[0][self.tag_to_ix[START_TAG]] = 0\n",
+    "        forward_var = autograd.Variable(init_vvars)\n",
+    "        for feat in feats:\n",
+    "            bptrs_t = []\n",
+    "            viterbivars_t = []\n",
+    "            \n",
+    "            for next_tag in range(self.tagset_size):\n",
+    "                next_tag_var = forward_var + self.transitions[next_tag]\n",
+    "                best_tag_id = argmax(next_tag_var)\n",
+    "                bptrs_t.append(best_tag_id)\n",
+    "                viterbivars_t.append(next_tag_var[0][best_tag_id])\n",
+    "            forward_var = (torch.cat(viterbivars_t) + feat).view(1, -1)\n",
+    "            backpointers.append(bptrs_t)\n",
+    "        \n",
+    "        terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]\n",
+    "        best_tag_id = argmax(terminal_var)\n",
+    "        path_score = terminal_var[0][best_tag_id]\n",
+    "        \n",
+    "        best_path = [best_tag_id]\n",
+    "        for bptrs_t in reversed(backpointers):\n",
+    "            best_tag_id = bptrs_t[best_tag_id]\n",
+    "            best_path.append(best_tag_id)\n",
+    "        start = best_path.pop()\n",
+    "        assert start == self.tag_to_ix[START_TAG]\n",
+    "        best_path.reverse()\n",
+    "        return best_path, path_score\n",
+    "        \n",
+    "    def log_likelihood(self, sentence, tags):\n",
+    "        feats = self._get_lstm_features(sentence)\n",
+    "        forward_score = self._forward_alg(feats)\n",
+    "        gold_score = self._score_sentence(feats, tags)\n",
+    "        return gold_score - forward_score\n",
+    "        \n",
+    "    def forward(self, sentence):\n",
+    "        embeds = self.word_embeds(sentence).view(len(sentence), 1, -1)\n",
+    "        lstm_out, self.hidden = self.lstm(embeds)\n",
+    "        lstm_out = lstm_out.view(len(sentence), self.hidden_dim)\n",
+    "        lstm_feats = self.hidden2tag(lstm_out)\n",
+    "        \n",
+    "        tag_seq, score = self._viterbi_decode(lstm_feats)\n",
+    "        return score, tag_seq\n",
+    "\n",
+    "START_TAG = \"<START>\"\n",
+    "STOP_TAG = \"<STOP>\"\n",
+    "tag_to_ix = { START_TAG: 0, STOP_TAG: 1, \"NN\": 2, \"V\": 3 }\n",
+    "word_to_ix = {\"hello\": 1, \"word\": 0}\n",
+    "sentence = \"hello hello word\".split()\n",
+    "idxs = autograd.Variable( torch.LongTensor(map(lambda w: word_to_ix[w], sentence) ))\n",
+    "model = BiLSTM_CRF(2, tag_to_ix, 4, 6)\n",
+    "model(idxs)\n"
    ]
   },
   {
