3-Word-Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we have a vocabulary of words that we are working with. We want to represent each word as a vector, so that we can use it in a network. One way we could do this is by using **one-hot encoding**. Each word is represented as a vector of all zeros, except for the index corresponding to that word, which is represented as a 1. This means that each sentence is represented by a matrix with dimensions `<nb_words, nb_vocab>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary:\n",
      "{'brown': 0, 'lazy': 1, 'jumped': 2, 'over': 3, 'fox': 4, 'dog': 5, 'the': 6}\n",
      "\n",
      "One-hot encoded:\n",
      "[[ 0.  0.  0.  0.  0.  0.  1.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  1.  0.  0.]\n",
      " [ 0.  0.  1.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  1.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  1.]\n",
      " [ 0.  1.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  1.  0.]]\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "np.random.seed(42)\n",
    "\n",
    "sentence = ['the', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog']\n",
    "vocabulary = dict([(word, i) for i, word in enumerate(set(sentence))])\n",
    "print('Vocabulary:')\n",
    "print(vocabulary, end='\\n\\n')\n",
    "\n",
    "def one_hot_encode(sentence):\n",
    "    ohe = np.zeros((len(sentence), len(vocabulary)))\n",
    "    for i, word in enumerate(sentence):\n",
    "        ohe[i, vocabulary[word]] = 1\n",
    "    return ohe\n",
    "\n",
    "print('One-hot encoded:')\n",
    "print(one_hot_encode(sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The obvious problem is that this scales horribly! If we have 10000 words in our vocabulary, then we are representing each word with 9999 zeros and a single 1! On top of that, the vectors don't tell us anything about how the words are related. A better way is to use **word embeddings**.\n",
    "\n",
    "There are different ways of generating word embeddings. In general, word embeddings are just a vector representation of words which hopefully conveys something about their meaning, for example:\n",
    "\n",
    "`king - man + woman = queen`\n",
    "\n",
    "Two popular methods for generating word embeddings are **word2vec** and **GloVe**. [Elsewhere](http://benjaminbolte.com/blog/2016/keras-gensim-embeddings.html) I have explained how to initialize these models using Gensim and pop the weights into Keras, which is a good idea for a lot of language models. For now, let's try generating some word embeddings using Keras that will help us solve a very simple task.\n",
    "\n",
    "A task that I routinely encounter is having to remember whether or not my friends are red or green. I really wish I had a smarter way to do this, instead of chunking through mounds of data. Fortunately, just such a smarter way exists, and it is what I wanted to write about anyway! Let's see what it is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sam is red -> [0]\n",
      "hannah not red -> [1]\n",
      "hannah is green -> [1]\n",
      "bob is green -> [1]\n",
      "bob not red -> [1]\n",
      "sam not green -> [0]\n",
      "sarah is red -> [0]\n",
      "sarah not green -> [0]\n"
     ]
    }
   ],
   "source": [
    "import itertools\n",
    "import numpy as np\n",
    "\n",
    "sentences = '''\n",
    "sam is red\n",
    "hannah not red\n",
    "hannah is green\n",
    "bob is green\n",
    "bob not red\n",
    "sam not green\n",
    "sarah is red\n",
    "sarah not green'''.strip().split('\\n')\n",
    "is_green = np.asarray([[0, 1, 1, 1, 1, 0, 0, 0]], dtype='int32').T\n",
    "\n",
    "for s, g in zip(sentences, is_green):\n",
    "    print(s, '->', g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's **tokenize** the sentences to get individual words, then generate our vocabulary and convert our sentences from words to indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary:\n",
      "{'sarah': 0, 'sam': 1, 'hannah': 2, 'is': 3, 'green': 4, 'not': 5, 'bob': 6, 'red': 7}\n",
      "\n",
      "Sentences:\n",
      "[[1 3 7]\n",
      " [2 5 7]\n",
      " [2 3 4]\n",
      " [6 3 4]\n",
      " [6 5 7]\n",
      " [1 5 4]\n",
      " [0 3 7]\n",
      " [0 5 4]]\n"
     ]
    }
   ],
   "source": [
    "tokenize = lambda x: x.strip().lower().split(' ')\n",
    "sentences_tokenized = [tokenize(sentence) for sentence in sentences]\n",
    "words = set(itertools.chain(*sentences_tokenized))\n",
    "\n",
    "word2idx = dict((v, i) for i, v in enumerate(words))\n",
    "idx2word = list(words)\n",
    "print('Vocabulary:')\n",
    "print(word2idx, end='\\n\\n')\n",
    "\n",
    "to_idx = lambda x: [word2idx[word] for word in x] # convert a list of words to a list of indices\n",
    "sentences_idx = [to_idx(sentence) for sentence in sentences_tokenized]\n",
    "sentences_array = np.asarray(sentences_idx, dtype='int32')\n",
    "print('Sentences:')\n",
    "print(sentences_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's some parameters for our model. I chose to embed each word into a two-dimensional vector space, because that is easily visualized. For a vocabulary with around 20,000 words, it is common to use embeddings of around 100 to 200 dimensions, although this varies greatly by application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 words per sentence, 8 in vocabulary, 2 dimensions for embedding\n"
     ]
    }
   ],
   "source": [
    "sentence_maxlen = 3\n",
    "n_words = len(words)\n",
    "n_embed_dims = 2\n",
    "print('%d words per sentence, %d in vocabulary, %d dimensions for embedding' % (sentence_maxlen, n_words, n_embed_dims))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Now let's put together our model in Keras. It takes as inputs a list of indices as integers, then embeds them in a `n_embed_dims`-dimensional space, and then feeds them to a neural network which predicts what color they refer to. Don't worry about the network too much, just understand that the word embedding vectors are being trained on the data so that they provide a representation which is more useful for solving our problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Input, Embedding, merge, Flatten, Reshape, Lambda\n",
    "import keras.backend as K\n",
    "from keras.models import Model\n",
    "\n",
    "input_sentence = Input(shape=(sentence_maxlen,), dtype='int32')\n",
    "input_embedding = Embedding(n_words, n_embed_dims)(input_sentence)\n",
    "avepool = Lambda(lambda x: K.mean(x, axis=1, keepdims=True), output_shape=lambda x: (x[0], 1))\n",
    "color_prediction = avepool(Reshape((sentence_maxlen * n_embed_dims,))(input_embedding))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, let's fit the model. In Keras, we can easily specify that we want to use binary crossentropy for our loss function; no need to look up it's formula on Wikipedia!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sarah: [-1.22824633 -1.17062831]\n",
      "sam: [-1.19280672 -1.20614004]\n",
      "hannah: [ 1.80140889  1.8014065 ]\n",
      "is: [ 0.56213224  0.64294231]\n",
      "green: [ 0.59127456  0.60197026]\n",
      "not: [ 0.55490071  0.64983344]\n",
      "bob: [ 1.83830953  1.77629972]\n",
      "red: [ 0.59618628  0.59634411]\n"
     ]
    }
   ],
   "source": [
    "predict_green = Model(input=[input_sentence], output=[color_prediction])\n",
    "predict_green.compile(optimizer='sgd', loss='binary_crossentropy')\n",
    "\n",
    "predict_green.fit([sentences_array], [is_green], nb_epoch=5000, verbose=0)\n",
    "embeddings = predict_green.layers[1].W.get_value()\n",
    "\n",
    "for i in range(n_words):\n",
    "    print('{}: {}'.format(idx2word[i], embeddings[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Awesome! Let's plot our results to see what kind of embeddings we learned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf8AAAFdCAYAAAANJWRbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xt4VPWdx/HPL8koECaJmIBAgYTaSAhyySg1RqtUCmws\noNgFA1vlKqzcNpaCLdiKD17achEQAR9WuVRjLc/DLu6qsAUBuQQkQdhF2MpNWJVEioSAXELmt38E\npgQwIcDMhPm9X88zzzNz5nfO73uOks+cc37nHGOtFQAAcEdUuAsAAAChRfgDAOAYwh8AAMcQ/gAA\nOIbwBwDAMYQ/AACOIfwBAHBMTLgLOJ8x5mZJXSXtk3QyvNUAAHBdqSMpWdIya+3fqmpYq8JfFcH/\nZriLAADgOtZP0ltVNaht4b9Pkv74xz8qLS3tmi00NzdX06ZNu2bLQ9XY3qHF9g4dtnVosb1rZseO\nHfqnf/on6WyWVqW2hf9JSUpLS1NGRsY1W2h8fPw1XR6qxvYOLbZ36LCtQ4vtfcWqPW3OgD8AABxD\n+AMA4BjCHwAAxzgR/jk5OeEuwSls79Bie4cO2zq02N7BY6y14a4hwBiTIamgoKCAQR4AANRAYWGh\nfD6fJPmstYVVtXVizx8AAPwd4Q8AgGMIfwAAHEP4AwDgGMIfAADHEP4AADiG8AcAwDGEPwAAjiH8\nAQBwDOEPAIBjCH8AABxD+AMA4BjCHwAAxxD+AAA4hvAHAMAxhD8AAI4h/AEAcAzhDwCAYwh/AAAc\nQ/gDAOAYwh8AAMcQ/gAAOIbwBwDAMYQ/AACOIfwBAHAM4Q8AjujUqZOeeuqpcJdRrdWrVysqKkpH\njx4NdykRi/AHANQ6xphwlxDRCH8AcNDVHgVYsGCBbrrppmtYEUKJ8AcAh/j9fo0bN05r167V3Llz\nNXHixMB306ZNU9u2bVW/fn01b95cw4cP1/HjxwPfnwv85cuX61e/+pWOHDmif/iHf1BRUVGgzYAB\nA/Twww9rypQpatKkiRITEzVixAiVl5cH2rz55pu68847FRcXp8aNG6tfv376+uuvL6p18+bNuvPO\nOxUbG6usrCx99tlnQdoq7iH8AcAhCxYsUP369eXz+XTvvffqueee04oVKyRJ0dHRmjlzprZv366F\nCxfqww8/1Lhx4yrN/+2332rKlCkaNmyY4uLitH//fo0ZM6ZSmw8//FB79uzRqlWrtHDhQs2fP1/z\n588PfF9WVqZJkyZp27Zt+vd//3d9/vnnGjBgQKVlWGs1YcIETZs2TQUFBYqJidHAgQODs1EcRPgD\ngEPatm2rZ555RnXr1lVqaqoSExP14IMPKikpSYcOHdJ9992nFi1aqH379kpKStKrr76q2NhYZWdn\nq6ioSGfOnNHcuXPVokULRUdH695771VeXp7q1q2rbt266fjx42rQoIFeeeUVpaamKjs7Ww8++GDg\nB4Yk9e/fX127dlVycrI6duyol19+We+//76+/fbbQBtjjF544QXdc889atWqlZ5++mmtX79ep0+f\nDsdmiziEPwA4pG3btoH38+fPl9frVffu3TVjxgz94Q9/UOvWrfW9731PSUlJ+uijjyRJq1atkrVW\nU6dOVd26dZWcnCxJOn78uFauXClrrdavX68jR45o9erVSk9PrzRgr3HjxiouLg58LigoUI8ePdSi\nRQvFxcXp/vvvlyTt37+/Uq233357pWVIqrQcXDnCHwAc4vF4Au+bN2+u22+/XV6vV3fffbfKyspU\nXFys6dOnq7y8XL/+9a9ljNFtt92mN998U998802lZZ05c0ZDhw6VJHXo0EELFixQcXGxjh07Vqmd\nMUZ+v19SxWmDbt26KSEhQW+99ZY2b96sJUuWSNJFe/Xn13rux8S55eDqEP4A4Ki77ror8L6goEDG\nGJWWlsrj8cjj8ejGG28MfN+gQQM1bty40sC9mJgYff/73w98vu2223TDDTeotLT0O/vcuXOnDh8+\nrBdffFFZWVlKTU2tNGAQoUH4AwB06623qry8XOXl5SoqKpLf79fcuXMrtbHWXjRfTa/Hb968uW64\n4QbNmDFDe/fu1dKlSzVp0qSL2l2qr0tNw5Uh/AHAERcGdX5+fuB927Zt9eMf/1jGGI0ePVpnzpzR\nkCFDAt//7W9/08GDBxUdHR2YdubMGe3atSvw+X//9391+vRpeb3e76whMTFR8+fP1+LFi5Wenq7f\n//73mjJlSrW1ftc0XJmYcBcAAAiNlStXVvp84MABdevWTU888YTy8vK0ceNGzZ49W4MHD9bDDz+s\nxYsXa82aNdq7d6+efvpppaSkaPv27YH5Y2JitHjxYm3YsEGFhYUaOXKksrKy9OGHH1bqZ9q0aZU+\n9+nTR3369Kk07fzTCffdd1+lz5LUrl27i6bhygV1z98Y8ytjzCZjzFFjTJExZokxJjWYfQIAqmeM\n0WOPPaYTJ06oY8eOGjlypHJzczV48GBJFVcC+Hw+de/eXVlZWYqKitJ//ud/Vtrzj42N1bhx49S3\nb1/dc8898nq9evvtt8O1SqgBE8xzKMaY9yTlSdqsiqMML0pqIynNWnviEu0zJBUUFBQoIyMjaHUB\nABBpCgsL5fP5JMlnrS2sqm1QD/tba7PP/2yM6S+pWJJP0tpg9g0AAC4t1AP+EiRZSYdD3C8AADgr\nZOFvKoZpvixprbX201D1CwAAKgvlaP9XJbWWlFVdw9zcXMXHx1ealpOTo5ycnCCVBgDA9SMvL095\neXmVppWUlFz2/EEd8BfoxJhXJHWXdK+1dn8V7RjwBwDAFag1A/6kQPD3lHRfVcEPAABCI6jhb4x5\nVVKOpB6SjhtjGp39qsRaezKYfQMAgEsL9oC/YZLiJK2S9OV5r95B7hcAAHyHYF/nz7MDAACoZQhn\nAAAcQ/gDAK4pHr1b+xH+AICrVlpaqlGjRiklJUXNmjVTSkqKRo0apdLS0nCXhkvgkb4AgKtSWlqq\nzMxM7dixQ36/PzB91qxZWrFihfLz8+X1esNYIS7Enj8A4KqMHz/+ouCXJL/fr08//VRNmzblKEAt\nQ/gDAK7Ku+++e1Hwn6+0tFSzZs1SZmYmPwBqCcIfAHDFrLUqKyurtp3f79eOHTs0YcKEEFSF6hD+\nAIArZoyRx+O5rLZ+v19Lly4NckW4HIQ/AOCqdO/eXVFRlxcnZWVlXApYCxD+AICr8vzzzystLe2y\nfgB4PB4ZY0JQFapC+AMArorX69WGDRs0YsSIKi/pi4qKUo8ePUJYGb4L4Q8AuGper1fTp0/XF198\nofT09IuOAkRFRSktLU2TJk0KU4U4H+EPALhmzj8KkJycrKZNmyo5OVkjRozQhg0buNlPLcEd/gAA\n19S5owDTp0+XtZZz/LUQe/4AgKAh+Gsnwh8AAMcQ/gAAOIbwBwDAMYQ/AACOIfwBAHAM4Q8AgGMI\nfwAAHEP4AwDgGMIfAADHEP4AADiG8AcAwDGEPwAAjiH8AQBwDOEPAIBjCH8AABxD+AMA4BjCHwAA\nxxD+AAA4hvAHAMAxhD8AAI4h/AEAcAzhDwCAYwh/AAAcQ/gDAOAYwh8AAMcQ/gAAOIbwBwDAMYQ/\nAACOIfwBAHAM4Q8AgGMIfwAAHEP4AwDgGMIfAADHEP4AADiG8AcAwDGEPwAAjglq+Btj7jXGLDXG\nfGGM8RtjegSzPwAAUL1g7/nHSvpE0nBJNsh9AbiOTJw4UR06dAh3GYCTYoK5cGvtB5I+kCRjjAlm\nXwCuP/xZAMKDc/4ArkpZWVm4SwBQQ4Q/gBrp1KmTRo4cqdzcXCUlJalbt24qKSnR4MGD1bBhQ8XH\nx6tz587atm1bpfleeukl3XLLLYqPj9fgwYN18uTJMK0BgKAe9r9Subm5io+PrzQtJydHOTk5YaoI\nwPkWLlyof/7nf9b69eslSf/4j/+o2NhYLVu2THFxcZo7d646d+6sv/71r0pISNA777yjiRMnavbs\n2crKytLChQs1Y8YMff/73w/zmgDXp7y8POXl5VWaVlJSctnzG2tDMw7PGOOX9JC1dmkVbTIkFRQU\nFCgjIyMkdQGomU6dOuno0aMqKCiQJK1bt04//elPVVxcLI/HE2j3gx/8QOPGjdPgwYOVlZUln8+n\nGTNmBL7PzMzUqVOnVFhYGPJ1ACJRYWGhfD6fJPmstVX+w+KwP4Aau+OOOwLvt27dqtLSUjVo0EBe\nrzfw2rdvn/bs2SNJ2rFjhzp27FhpGZmZmSGtGcDfBfWwvzEmVtKtks4N6W1pjGkn6bC19kAw+wYQ\nPLGxsYH3x44dU5MmTbR69WpdeCQxISEh8J6R/UDtEexz/ndI+lAV1/hbSVPOTl8gaWCQ+wYQAhkZ\nGTp48KCio6PVvHnzS7ZJS0tTfn6++vXrF5iWn58fqhIBXCDY1/mvFqcWgIjWuXNnZWZm6qGHHtLv\nfvc7paam6osvvtB7772nXr16KSMjQ6NHj9aAAQPk8/mUlZWlP/7xj9q+fTsD/oAwqZWj/QHUXpc6\nfP/ee+9p/PjxGjhwoL7++mvdcsst+tGPfqRGjRpJknr37q09e/Zo3LhxOnnypB555BE9+eSTWrZs\nWajLB6AQjva/HIz2BwDgyjDaHwAAfCfCHwAAxxD+AAA4hvAHAMAxhD8AAI4h/AEAcAzhDwCAYwh/\nAAAcQ/gDAOAYwh9ArWCtveipgACCg/AHEDalpaUaNmyY4uLi5PF45PF4FBcXp2HDhqm0tDTc5QER\niwf7AAiL0tJS/fCHP9SOHTsumj537lytXr1amzZtktfrDVOFQORizx9AWIwfP/6i4D/fzp07NWHC\nhBBWBLiD8AcQFu+++261bWbMmKF27drpyy+/DEFFgDsIfwAhZ61VWVnZZbXdtm2bWrZsyQ8A4Boi\n/AGEnDFGHo/nstufOnVK2dnZQawIcAvhDyAsunfvLmPMZbffvn17EKsB3EL4AwiL559/Xq1atbrs\n9n6/X36/P4gVAe4g/AGEhdfr1caNGzVs2LDLupzPGKOoKP5kAdcC/5IAhI3X69Xs2bN19OhRtW3b\ntsq2bdq0CVFVQOQj/AHUCu+//75uvPHGS35344036r333gtxRUDkIvwB1ApNmjTRnj171K5dO8XE\nxCgqKkoxMTFq166d9uzZoyZNmoS7RCBicHtfANXq1KmTOnTooKlTpwa1nyZNmuiTTz6RVDHAj3P8\nQHAQ/gCqtWTJkhpdl3++iRMn6t/+7d+0ZcuWGs1H8APBQ/gDqFZCQsJVzV+T6/kBBB8/rQFUqVOn\nTvre976nO++8UzfffLPi4+N18803q27durrlllv005/+VD179pTX61V8fLz69Omj4uJiSdKCBQs0\nceJEbd26VVFRUYqOjtbChQvDvEYACH8A1SoqKpLH49Hrr7+uY8eO6ZtvvtHrr7+uZcuWaevWrTpy\n5Ig++ugj/eUvf9Hu3bv16KOPSpL69OmjX/ziF0pPT1dRUZG++uor9enTJ8xrA4DD/gCqFRsbq7vu\nukt+v19xcXG69dZb9d///d9KSkpSUVGRNm7cGBiNv2jRIqWnp6ugoEA+n09169ZVTEyMkpKSwrwW\nAM5hzx9AterXry9J6tKli5o2barCwkK99NJL6tGjh7xer/r166ennnpKkpSdna06deroX/7lX5SQ\nkBB4dO///d//qU+fPrrpppuUmJiohx56SJ9//nmlfubNm6fWrVurbt26at26tWbPnh347vPPP1dU\nVJSWLFmiH//4x4qNjVX79u2Vn58foq0ARA7CH0C1zg3YO3cEwOPxKDk5WXXq1NE333yjwsLCSu1P\nnjypFi1aaMuWLbrvvvtkrVXXrl0VHx+vdevWad26dfJ6verWrZvOnDkjSXrzzTf17LPP6sUXX9TO\nnTv1wgsv6De/+Y0WLVpUadkTJkzQ2LFjtXXrVqWmpqpv377c8x+oIQ77A7hsx44d06JFi9S+fXul\np6erV69e6t69u06dOhVoc/r0aVlr9Ytf/EIpKSlq1KiRDh8+rNjYWL322muBdv/6r/+qm266SatW\nrVLnzp317LPPasqUKerZs6ckqUWLFtq+fbvmzJmjn//854H5fvnLX6pbt26SKi4jbNOmjXbt2qXU\n1NQQbQXg+kf4A7hs8+fPV1lZmWJiYnTs2DEdOHBAkhQdHa3i4mJt2rRJhw4dUsuWLdWhQwdJUnJy\nsg4ePKjy8vLA6YNzRxJOnTql3bt36+6779bu3bs1aNAgDR48ONBfeXn5RZcZ3n777YH3jRs3lrVW\nxcXFhD9QA4Q/gCqdf42+1+uVtVYff/yxNm3apM8++0xz5szRuHHj9Oc//1n/8R//IY/Ho0GDBgXm\neeSRR/TMM89o3759OnHihH73u9+pV69ege+TkpJ07NgxSRXn/Dt27Fip/+jo6Eqfz7/Z0LnaOOwP\n1AzhD6BKK1euDLw/duyYnnjiCb399tt6+OGHJUlHjx7VmDFjNHz4cE2dOlUpKSmBPXxJuuGGG/T0\n00/r6aef1r59+yp9d47X61XTpk0rXSZ4KdwsCLg2GPAH4LLVr19fjz/+uMaMGaNVq1Zp+/btGjRo\nkKKjo6sM5n79+ikxMVE9e/bU2rVrtW/fPq1atUqjR4/Wl19+KUmBwX4zZ87UZ599pv/5n//R/Pnz\n9fLLLweWY60N+joCLiD8AdTItGnTdPfdd6t79+7q0qWL7rnnHrVq1Up16tSRdOm987p162rNmjVq\n3ry5HnnkEbVu3VpDhgzRqVOnFBcXJ0kaNGiQ5s2bpzfeeENt27bV/fffrwULFiglJSWwnEstm6MB\nQM2Z2vRL2hiTIamgoKBAGRkZ4S4HwGX49ttv1bRpU02dOlUDBgwIdzmAswoLC+Xz+STJZ60trKot\n5/wB1Mgnn3yinTt3qmPHjjpy5Iiee+45GWMCl+gBqP0IfwA1NnnyZP31r3/VDTfcIJ/Pp7Vr16pB\ngwbhLgvAZSL8AdRI+/bttXnz5nCXAeAqMOAPAADHEP4AADiG8AcAwDGEPwAAjiH8AQBwDOEPAIBj\nCH8AABxD+AMA4JiQhL8xZrgxZq8x5oQxJt8Yc2co+gUAABcLevgbY/pImiLpt5I6SNoqaZkxJjHY\nfQMAgIuFYs8/V9Jca+1Ca+1OScMkfStpYAj6BgAAFwhq+BtjPJJ8klacm2YrniH8F0mZwewbAABc\nWrD3/BMlRUsqumB6kaRbgtw3AAC4hHCN9jeSbJj6BgDAacF+pO8hSeWSGl0wvaEuPhoQkJubq/j4\n+ErTcnJylJOTc80LBADgepOXl6e8vLxK00pKSi57flNxCj54jDH5kjZaa0ef/Wwk7Zc0w1r7hwva\nZkgqKCgoUEZGRlDrAgAgkhQWFsrn80mSz1pbWFXbYO/5S9JUSQuMMQWSNqli9H89SfND0DcAALhA\n0MPfWvvO2Wv6n1PF4f9PJHW11n4d7L4BAMDFQrHnL2vtq5JeDUVfAACgatzbHwAAxxD+AAA4hvAH\nAMAxhD8AAI4h/AEAcAzhDwCAYwh/AAAcQ/gDAOAYwh8AAMcQ/gAAOIbwBwDAMYQ/AACOIfwBAHAM\n4Q8AgGMIfwAAHEP4AwDgGMIfAADHEP4AADiG8AcAwDGEPwAAjiH8AQBwDOEPAIBjCH8AABxD+AMA\n4BjCHwAAxxD+AAA4hvAHAMAxhD8AAI4h/AEAcAzhDwCAYwh/AAAcQ/gDAOAYwh8AAMcQ/gAAOIbw\nBwDAMYQ/AACOIfwBAHAM4Q8AgGMIfwAAHEP4AwDgGMIfAADHEP4AADiG8AcAwDGEPwAAjiH8rwOL\nFy9W27ZtVa9ePSUmJqpLly46ceKENm/erC5duigpKUkJCQm6//77tWXLlkrzRkVF6bXXXlP37t0V\nGxur1q1bKz8/X7t371anTp1Uv359ZWVlae/evWFaOwBAqBH+tdzBgwfVt29fDR48WDt37tTq1avV\nq1cvWWtVWlqq/v37a926ddq4caNSU1OVnZ2t48ePV1rGpEmT1L9/f23dulVpaWnq27evhg0bpvHj\nx6ugoEDWWo0YMSJMawgACLWYcBeAqn311VcqLy/Xww8/rGbNmkmS0tPTJUmdOnWq1HbOnDn605/+\npNWrVys7OzswfeDAgXrkkUckSWPHjlVmZqZ++9vfqnPnzpKk0aNHa+DAgaFYHQBALcCefy3Xrl07\nPfDAA2rTpo169+6tefPm6ciRI5Kk4uJiDRkyRKmpqUpISFB8fLyOHz+u/fv3V1rG7bffHnjfqFEj\nSVKbNm0qTTt58qSOHTsWgjUCAIQb4V/LRUVFafny5frggw+Unp6umTNnqlWrVtq3b58ee+wxbdu2\nTTNnztSGDRu0detWNWjQQKdPn660DI/HE3hvjPnOaX6/PwRrBAAIN8L/OnHuUP2WLVvk8Xi0ZMkS\nrV+/XqNGjVLXrl2VlpYmj8ejQ4cOVbusc2EPAHAT5/xruU2bNmnFihXq0qWLGjZsqPz8fB06dEit\nW7dWamqqFi1aJJ/Pp5KSEo0dO1b16tWrdpnW2suaBgCITEELf2PMryU9KKm9pFPW2gbB6iuSxcXF\nac2aNZo+fbqOHj2qFi1aaOrUqeratasaNWqkoUOHKiMjQ82bN9cLL7ygMWPGVJr/Unv5lzsNABCZ\nTLD2+Iwxv5V0RFIzSQMvJ/yNMRmSCgoKCpSRkRGUugAAiESFhYXy+XyS5LPWFlbVNmh7/tbaiZJk\njHk8WH0AAICaY8AfAACOIfwBAHBMjcLfGPOiMcZfxavcGJMarGIBAMDVq+k5/8mS3qimzZ4rrCUg\nNzdX8fHxlabl5OQoJyfnahcNAMB1Ly8vT3l5eZWmlZSUXPb8QRvtH+igYsDfNEb7AwAQPLVitL8x\nppmkBpJaSIo2xrQ7+9Uua+3x754TAAAEUzDv8PecpMfO+3zuV0gnSWuC2C8AAKhC0Eb7W2sHWGuj\nL/Ei+AEACCMu9XMc9/QHAPcQ/g4qLS3VqFGjlJKSombNmiklJUWjRo1SaWlpuEsDAIQAT/VzTGlp\nqTIzM7Vjxw75/f7A9FmzZmnlypXasGGDvF5vGCsEAAQbe/6OGT9+/EXBL0l+v1+ffvqpJkyYEKbK\nAAChQvg75t13370o+M+x1mrWrFmcAgCACEf4O8Raq7KysirblJeXa9asWcrMzOQHAABEKMLfIcYY\neTyeatv5/X7t2LGDUwAAEKEIf8d0795dUVHV/2f3+/1aunRpCCoCAIQa4e+Y559/XmlpaTLGVNu2\nrKyM+wAAQAQi/B3j9Xq1YcMGjRw5UtHR0VW29Xg8l/UjAQBwfSH8HeT1ejV9+nQ9+eST33kKICoq\nSj169AhxZQCAUCD8HXbuFMCFPwCioqKUlpamSZMmhakyAEAwEf4OO3cKYMSIEUpOTlbTpk2VnJys\nESNGcKc/AIhghL/jzp0C2Lt3r37yk5+oQ4cOmj59OsEPABGM8AcAwDGEvwPOnDkT7hIAALUI4V9L\nLV68WG3btlW9evWUmJioLl266MSJE9q8ebO6dOmipKQkJSQk6P7779eWLVsqzRsVFaU5c+aoZ8+e\nql+/vl544QX5/X4NHjxYLVu2VL169dSqVSvNmDHjkn1PmTJFTZo0UWJiokaMGKHy8vJQrDIAIER4\npG8tdPDgQfXt21eTJ0/WQw89pNLSUn300Uey1qq0tFT9+/fXK6+8ImutpkyZouzsbO3atUuxsbGB\nZUycOFEvvfSSpk+frpiYGPn9fjVr1kyLFy/WzTffrPXr1+uJJ55QkyZN9LOf/Sww38qVK9W4cWOt\nWrVKu3btUu/evdWhQwcNGjQoHJsCABAEpjbdwc0YkyGpoKCgQBkZGeEuJ2y2bNmiO+64Q/v27VOz\nZs2qbOv3+3XTTTcpLy9P2dnZkir2/J966ilNnjy5ynlHjhypoqIivfPOO5KkAQMGaPXq1dq9e3fg\n5j59+vRRdHS03nrrrWuwZgCAYCksLJTP55Mkn7W2sKq2HPavhdq1a6cHHnhAbdq0Ue/evTVv3jwd\nOXJEklRcXKwhQ4YoNTVVCQkJio+P1/Hjx7V///5Kyzj7P0Als2bN0h133KGGDRvK6/Xqtddeu2i+\n9PT0Snf1a9y4sYqLi4OwlgCAcCH8a6GoqCgtX75cH3zwgdLT0zVz5ky1atVK+/bt02OPPaZt27Zp\n5syZ2rBhg7Zu3aoGDRro9OnTlZZx/ikASXr77bf1y1/+UkOGDNF//dd/aevWrRowYMBF81341D9j\njPx+f3BWFAAQFpzzr8UyMzOVmZmpZ555Ri1atNCSJUu0fv16zZ49W127dpUkHThwQIcOHap2WevX\nr1dWVpaGDh0amLZ79+6g1Q4AqL0I/1po06ZNWrFihbp06aKGDRsqPz9fhw4dUuvWrZWamqpFixbJ\n5/OppKREY8eOVb169apd5g9+8AMtWrRIy5cvV0pKihYtWqSPP/5YLVu2DMEaAQBqEw7710JxcXFa\ns2aNHnzwQd122236zW9+o6lTp6pr166aN2+evvnmG2VkZOjxxx/X6NGj1bBhw0rzX+pJfEOHDlWv\nXr306KOP6q677tLhw4c1fPjwUK0SAKAWYbQ/AAARgNH+AADgOxH+AAA4hvAHAMAxhD8AAI4h/AEA\ncAzhDwCAYwh/AAAcQ/gDAOAYwh8AAMcQ/gAAOIbwBwDAMYQ/AACOIfwBAHAM4Q8AgGMIfwAAHEP4\nAwDgGMIfAADHEP4AADiG8AcAwDGEPwAAjiH8AQBwDOEPAIBjCH8AABxD+AMA4BjCHwAAxxD+AAA4\nJmjhb4xpYYyZZ4zZY4z51hjzmTHmWWOMJ1h9AgCA6sUEcdmtJBlJQyTtltRG0jxJ9SSNDWK/AACg\nCkELf2vtMknLzpu0zxgzWdIwEf4AAIRNqM/5J0g6HOI+AQDAeUIW/saYWyWNkDQnVH0CAICL1Tj8\njTEvGmP8VbzKjTGpF8zTVNL7kv5krX39WhUPAABq7krO+U+W9EY1bface2OMaSJppaS11tqhl9NB\nbm6u4uPjK03LyclRTk5ODUsFACDy5OXlKS8vr9K0kpKSy57fWGuvdU1/X3jFHv9KSR9L+rmtpjNj\nTIakgoJkc3qiAAAEUUlEQVSCAmVkZAStLgAAIk1hYaF8Pp8k+ay1hVW1Ddpof2NMY0mrJO1Txej+\nhsYYSZK1tihY/QIAgKoF8zr/LpJann0dODvNSLKSooPYLwAAqELQRvtbaxdYa6MveEVZawl+AADC\niHv7AwDgGCfC/8IRkQgutndosb1Dh20dWmzv4CH8cc2xvUOL7R06bOvQYnsHjxPhDwAA/o7wBwDA\nMYQ/AACOCeZ1/leijiTt2LHjmi60pKREhYVV3uwI1xDbO7TY3qHDtg4ttnfNnJeddaprG9Tb+9aU\nMaavpDfDXQcAANexftbat6pqUNvC/2ZJXVVxS+CT4a0GAIDrSh1JyZKWWWv/VlXDWhX+AAAg+Bjw\nBwCAYwh/AAAcQ/gDAOAYwh8AAMcQ/gAAOMaZ8DfGtDDGzDPG7DHGfGuM+cwY86wxxhPu2iKVMebX\nxph1xpjjxpjD4a4n0hhjhhtj9hpjThhj8o0xd4a7pkhkjLnXGLPUGPOFMcZvjOkR7poimTHmV8aY\nTcaYo8aYImPMEmNMarjrijTOhL+kVpKMpCGSWkvKlTRM0vPhLCrCeSS9I2l2uAuJNMaYPpKmSPqt\npA6StkpaZoxJDGthkSlW0ieShkvi2ujgu1fSTEk/lNRZFX9Hlhtj6oa1qgjj9HX+xpgxkoZZa28N\ndy2RzBjzuKRp1toG4a4lUhhj8iVttNaOPvvZSDogaYa19vdhLS6CGWP8kh6y1i4Ndy2uOPuDtljS\nj6y1a8NdT6Rwac//UhIkcTga15Wzp6p8klacm2YrfsX/RVJmuOoCgiRBFUdc+Ft9DTkb/saYWyWN\nkDQn3LUANZQoKVpS0QXTiyTdEvpygOA4e0TrZUlrrbWfhrueSHLdh78x5sWzg3C+61V+4WARY0xT\nSe9L+pO19vXwVH59upLtjZAx4pw0Isurqhij9Wi4C4k0te2RvldisqQ3qmmz59wbY0wTSStV8Uty\naDALi1A12t4IikOSyiU1umB6Q118NAC4LhljXpGULelea+1X4a4n0lz34X/2yUVVPr3onLN7/Csl\nfSxpYDDrilQ12d4IDmttmTGmQNIDkpZKgcOjD0iaEc7agGvhbPD3lHSftXZ/uOuJRNd9+F8uY0xj\nSatU8bjgsZIaVvy9lKy17C0FgTGmmaQGklpIijbGtDv71S5r7fHwVRYRpkpacPZHwCZVXLpaT9L8\ncBYViYwxsZJuVcVpFUlqefb/5cPW2gPhqywyGWNelZQjqYek48aYc0e4Sqy1POr9GnHmUr+zl5td\neH7fqGKgdHQYSop4xpg3JD12ia86WWvXhLqeSGOMeVIVP2QbqeI69JHW2s3hrSryGGPuk/ShLh5P\nscBayxHEa+zs5ZSXCqYB1tqFoa4nUjkT/gAAoMJ1P9ofAADUDOEPAIBjCH8AABxD+AMA4BjCHwAA\nxxD+AAA4hvAHAMAxhD8AAI4h/AEAcAzhDwCAYwh/AAAc8/8+aRu22CVBTAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10be04850>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "x1, x2 = embeddings[:, 0], embeddings[:, 1]\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.gca()\n",
    "offsets_x = [random.randint(0, 1) * 0.6 - 0.3 for _ in range(n_words)]\n",
    "offsets_y = [random.randint(0, 1) * 0.6 - 0.3 for _ in range(n_words)]\n",
    "for i, ox, oy in zip(range(n_words), offsets_x, offsets_y):\n",
    "    ax.plot(x1[i], x2[i], 'o', color='black')\n",
    "    ax.annotate(idx2word[i], xy=[x1[i], x2[i]], xytext=[x1[i] + ox, x2[i] + oy])\n",
    "\n",
    "ax.set_xlim(min(x1)-1, max(x1)+1)\n",
    "ax.set_ylim(min(x2)-1, max(x2)+1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Bob and Hannah (who were both green) are together in one corner, and Sarah and Sam (who were both red) are together in another corner, with the other parts floating in the middle. The network didn't learn anything unique about \"not\" or \"is\" or \"green\" or \"red\" because it those are higher-order correlations, and it is a simple linear model.\n",
    "\n",
    "In more complex language modeling problems, word embeddings are often useful for providing a starting place for the network, which reduces the parameter search space and hopefully provides a better minimum for the loss function."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}