04_evaluating_word_and_kg_embeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04_evaluating_word_and_kg_embeddings.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "J7q5A_Kb3fgJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Evaluating Word (and Concept) Embeddings\n",
        "\n",
        "In the previous notebooks we have seen how to generate word, knowledge graph and joint (word-concept) embeddings.\n",
        "\n",
        "We also saw that it is easy to explore the resulting embedding spaces using cosine similarity and selecting the *k-nearest neighbours*.\n",
        "\n",
        "In this notebook we look further into how (word) embeddings are evaluated. In particular, we look into the following methods:\n",
        "  - **Visual Exploration**: whereby (a subsection of) the embeddings are displayed\n",
        "  - **Intrinsic Evaluation**: whereby the embeddings are used to perform a token-based task and the results are compared with a gold standard.\n",
        "    + **Word Prediction**: whereby we look into using a test corpus to evaluate the embeddings by defining a word prediction task.\n",
        "  - **Extrinsic Evaluation**: whereby a new model is learned (using the embeddings as inputs) to perform a complex task. \n",
        "  \n",
        "KG embeddings tend to be evaluated using **graph completion** tasks, which we will also discuss briefly."
      ]
    },
    {
      "metadata": {
        "id": "Kkl0BcnA3fgK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Recommended papers in this area\n",
        "\n",
        "[Schnabel, T., Labutov, I., Mimno, D., & Joachims, T. (2015). Evaluation methods for unsupervised word embeddings. In EMNLP (pp. 298–307). Association for Computational Linguistics.](http://anthology.aclweb.org/D/D15/D15-1036.pdf) Provides a good overview of methods and introduces terminology to refer to different types of evaluations.\n",
        "\n",
        "[Baroni, M., Dinu, G., & Kruszewski, G. (2014). Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (pp. 238–247).](http://anthology.aclweb.org/P/P14/P14-1023.pdf) Focuses mostly on *intrinsic* evaluations. Showed that predictive models (like word2vec) produced better results than count models (based on co-occurrence counting).\n",
        "\n",
        "[Levy, O., Goldberg, Y., & Dagan, I. (2015). Improving Distributional Similarity with Lessons Learned from Word Embeddings. Transactions of the Association for Computational Linguistics, 3(0), 211–225.](https://www.transacl.org/ojs/index.php/tacl/article/view/570) Studied how various implementation or optimization 'details' used in predictive models, which were not needed or used in count models affect the performance of the resulting embeddings. Example of such details are: negative sampling, dynamic context windows, subsampling and vector normalization. The paper shows that once such details are taken into account, the difference between count and predictive models is not that large."
      ]
    },
    {
      "metadata": {
        "id": "dNopvQU8-xSR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "09ccb2b2-ca2c-42bb-db49-fab4cba5a0b8"
      },
      "cell_type": "code",
      "source": [
        "%cd /content/tutorial\n",
        "!git pull\n",
        "%cd /content/"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/tutorial\n",
            "remote: Enumerating objects: 9, done.\u001b[K\n",
            "remote: Counting objects: 100% (9/9), done.\u001b[K\n",
            "remote: Compressing objects: 100% (1/1), done.\u001b[K\n",
            "remote: Total 5 (delta 4), reused 5 (delta 4), pack-reused 0\u001b[K\n",
            "Unpacking objects: 100% (5/5), done.\n",
            "From https://github.com/HybridNLP2018/tutorial\n",
            "   c433662..e57213f  master     -> origin/master\n",
            "Updating c433662..e57213f\n",
            "Fast-forward\n",
            " scripts/swivel/wordsim.py | 2 \u001b[32m+\u001b[m\u001b[31m-\u001b[m\n",
            " 1 file changed, 1 insertion(+), 1 deletion(-)\n",
            "/content\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "xDt1O-r398fi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a89a52e1-49d0-46fc-b9bf-1e411d481cd0"
      },
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/HybridNLP2018/tutorial.git"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fatal: destination path 'tutorial' already exists and is not an empty directory.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "FWXfjSCB3fgL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Visual Exploration\n",
        "\n",
        "Use dimensionality reduction algorithms such as t-SNE and PCA to visualize (a subset) of the embedding space to project points to a 2-D or 3-D space.\n",
        "\n",
        "[Embedding Projector](http://projector.tensorflow.org/)\n",
        "\n",
        " - Pros:\n",
        "   - Can give you a sense of whether the model has correctly learned meaningful relations. Especially if you have a small number of pre-categorized words.\n",
        "   - Easy to explore the space\n",
        " - Cons:\n",
        "   - Subjective: neighbourhoods may look good, but are they? There is no gold standard\n",
        "   - Works best for a small subset of the embedding space. But who decides which subset?\n",
        "   - resulting projection can be deceiving: what looks close in 3-D space can be far in 300-D space (and vice-versa)."
      ]
    },
    {
      "metadata": {
        "id": "XEgeeVpA3fgM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Intrinsic Evaluation\n",
        "\n",
        "**Intrinsic** evaluations are those where you can use embeddings to perform relatively simple, word-related tasks.\n",
        "\n",
        "Schnabel et al. distinguish between:\n",
        " - **Absolute intrinsic**: you have a (human annotated) gold standard for a particular task and use the embeddings to make predictions.\n",
        " - **Comparative intrinsic**: you use the embedding space to present predictions to humans, who then rate them. Mostly used when there is no gold standard available.\n",
        " \n",
        "Tasks:\n",
        " - **Relatedness**: How well do embeddings capture human-perceived word similarity? Datasets typically consist of triples: two words and a similarity score (e.g. between 0.0 and 1.0). Several available datasets, although interpretation of 'word similarity' can vary.\n",
        " - **Synonym detection**: Can embeddings select a synonym for a given word and a set of options? Datasts are n-tuples where the first word is the input word and the other `n-1` words are the options. Only one of the options is a synonym.\n",
        " - **Analogy**: Do embeddings encode relations between words? Datasets are 4-tuples: the first two words define the relation, the third word is the source of the query and the fourth word is the solution. Good embeddings should predict an embedding close to the solution word.\n",
        " - **Categorization**: Can embeddings be clustered into hand-annotated categories? Datasets are word-category pairs. Standard clustering algorithms can then be used to generate k-clusters and the purity of the clusters can be computed.\n",
        " - **Selectional preference**: Can embeddings predict whether a noun-verb pair is more likely to represent a verb-subject or a verb-object relation? E.g. people-eat is more likely to be found as a verb-subject."
      ]
    },
    {
      "metadata": {
        "id": "6Ae-1nPU3fgN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Compute Relatedness Score\n",
        "\n",
        "Swivel comes with a `eval.mk` script that downloads and unzips various relatedness and analogy datasets. The script also compiles an `analogy` executable. It assumes you have a unix environment and tools such as `wget`, `tar`, `unzip` and `egrep`, as well as `make` and a `c++` compiler.\n",
        "\n",
        "For convenience, we have included various relatedness datasets as part of this repo in `eval-datastets/relatedness`. We assume you have generated vectors as part of previous notebooks, which we will test here."
      ]
    },
    {
      "metadata": {
        "id": "c53h8V-B3fgO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import os"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GysGW8l83fgS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e73a61ea-4fbf-4125-dac6-89be054ac81e"
      },
      "cell_type": "code",
      "source": [
        "%ls /content/tutorial/datasamples/relatedness/"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "rarewords.ws.tab  simverb3500.ws.tab  ws353sim.ws.tab\n",
            "simlex999.ws.tab  ws353rel.ws.tab\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Nxi6HFq83fgZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%cp /content/umbc/coocs/tlgs_wnscd_5K_ls_f/row_vocab.txt /content/umbc/vec/tlgs_wnscd_5k_ls_f/vocab.txt\n",
        "umbc_5k_vec = '/content/umbc/vec/tlgs_wnscd_5k_ls_f/'\n",
        "umbc_full_vec = '/content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ppa9hX7U3fgg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You can use Swivel's `wordsim.py` to produce metrics for the k-cap embeddings we produced in previous notebooks:"
      ]
    },
    {
      "metadata": {
        "id": "1qEzvSEv3fgg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "14a29f03-7131-41bf-d5a9-322809d0773e"
      },
      "cell_type": "code",
      "source": [
        "!python /content/tutorial/scripts/swivel/wordsim.py --vocab={umbc_5k_vec}vocab.txt \\\n",
        "  --embeddings={umbc_5k_vec}vecs.bin \\\n",
        "  --word_prefix=\"lem_\" \\\n",
        "  /content/tutorial/datasamples/relatedness/*.ws.tab  "
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Opening vector with expected size 5632 from file /content/umbc/vec/tlgs_wnscd_5k_ls_f/vocab.txt\n",
            "vocab size 5632 (unique 5632)\n",
            "read rows\n",
            "65 of 2034 pairs found\n",
            "0.576 /content/tutorial/datasamples/relatedness/rarewords.ws.tab\n",
            "288 of 999 pairs found\n",
            "0.066 /content/tutorial/datasamples/relatedness/simlex999.ws.tab\n",
            "1126 of 3500 pairs found\n",
            "0.073 /content/tutorial/datasamples/relatedness/simverb3500.ws.tab\n",
            "92 of 252 pairs found\n",
            "0.371 /content/tutorial/datasamples/relatedness/ws353rel.ws.tab\n",
            "57 of 203 pairs found\n",
            "0.459 /content/tutorial/datasamples/relatedness/ws353sim.ws.tab\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "9rX9pVwR3fgl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "outputId": "4792fcf9-ef4d-45e0-c156-e3fab738ccf5"
      },
      "cell_type": "code",
      "source": [
        "%ls {umbc_full_vec}vocab.txt\n",
        "!python /content/tutorial/scripts/swivel/wordsim.py --vocab=/content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/vocab.txt \\\n",
        "  --embeddings={umbc_full_vec}vecs.bin \\\n",
        "  --word_prefix=\"lem_\" \\\n",
        "  /content/tutorial/datasamples/relatedness/*.ws.tab"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/vocab.txt\n",
            "Opening vector with expected size 1499136 from file /content/umbc/vec/vecsi_tlgs_wnscd_ls_f_6e_160d/vocab.txt\n",
            "vocab size 1499136 (unique 1499125)\n",
            "read rows\n",
            "1433 of 2034 pairs found\n",
            "0.401 /content/tutorial/datasamples/relatedness/rarewords.ws.tab\n",
            "999 of 999 pairs found\n",
            "0.276 /content/tutorial/datasamples/relatedness/simlex999.ws.tab\n",
            "3494 of 3500 pairs found\n",
            "0.191 /content/tutorial/datasamples/relatedness/simverb3500.ws.tab\n",
            "250 of 252 pairs found\n",
            "0.529 /content/tutorial/datasamples/relatedness/ws353rel.ws.tab\n",
            "202 of 203 pairs found\n",
            "0.649 /content/tutorial/datasamples/relatedness/ws353sim.ws.tab\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VRQU5VWu3fgq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The numbers show that both embedding spaces only have a small coverage of the evaluation datasets. Furthermore, the correlation score achieved is in the range of 0.07 to 0.22, which is very poor, but expected given the size of the corpus. \n",
        "\n",
        "For comparison state-of-the-art results are in the range of 0.65 to 0.8.\n"
      ]
    },
    {
      "metadata": {
        "id": "igU8-ZlQ3fgr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Conclusion for Intrinsic Evaluation\n",
        "\n",
        "Intrinsic evaluations are the most direct way of evaluating (word) embeddings.\n",
        "\n",
        "Pros:\n",
        " - they provide a single objective metric that enables easy comparison between different embeddings\n",
        " - there are several readily available evaluation datasets (for English)\n",
        " - if you have an existing, manually crafted, knowledge graph, you can generate your own evaluation datasets\n",
        " \n",
        "Cons:\n",
        " - evaluation datasets are small and can be biased in terms of word selection and annotation\n",
        " - you need to take coverage into account (besides final metric)\n",
        " - existing datasets only support English words (few datasets in other languages, few compound words, few concepts)\n",
        " - tasks are low level and thus somewhat artificial: people care about document classification, but not about word categories or word similarities."
      ]
    },
    {
      "metadata": {
        "id": "UnJKkA-g3fgs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Word Prediction (plots)\n",
        "\n",
        "This can be seen as a task for intrinsic evaluation, however the task is very close to the original training task used to derive the embeddings in the first place.\n",
        "\n",
        "Recall that *predictive models* (such as `word2vec`), try to minimize the distance between a word embedding and the embeddings of the context words (and that over a whole corpus).\n",
        "\n",
        "![word2vec diagrams](https://github.com/hybridNLP2018/tutorial/blob/master/images/word2vec_diagrams.png?raw=1)\n",
        "\n",
        "This means that, if we have a **test corpus**, we can use the embeddings to try to predict words based on their contexts. Assuming the test corpus and the training corpus contain similar language we should expect better embeddings to produce better predictions on average.\n",
        "\n",
        "A major advantage of this approach is that we do not need human annotation. Also, we can reuse the tokenization pipeline used for training to produce similar tokens as those in our embedding space. E.g. we can use word-sense-disambiguation to generate a test corpus including lemmas and concepts.\n",
        "\n",
        "The algorithm in pseudo-code is: \n",
        "\n",
        "``` python\n",
        "similarities = {}\n",
        "for window in corpus:\n",
        "  focus_word, context_words = window\n",
        "  focus_vector = embedding(focus_word)\n",
        "  context_vector = predict_embedding(context_words, focus_word)\n",
        "  similarities[focus_word].append(cosine_similarity(focus_vector, context_vector))\n",
        "return similarities.values().average()\n",
        "```\n",
        "\n",
        "The result is a single number that tells you how far the prediction embedding was from the actual word embedding over the whole test corpus. When using cosine similarity this should be a number between -1 and 1."
      ]
    },
    {
      "metadata": {
        "id": "C0I0kQH53fgt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Word prediction plots\n",
        "\n",
        "We can also use the intermediate `similarities` dictionary to plot diagrams which can provide further insight. For example, random embeddings result in \n",
        "\n",
        "![Word prediction plot for random embeddings](https://github.com/hybridNLP2018/tutorial/blob/master/images/Avg_cosine_similarities_for_random_words_at_different_winSizes_recentered.PNG?raw=1)\n",
        "\n",
        "The horizontal axis is the rank of the `focus_word` sorted by their frequency in the training corpus. (For example, frequent words such as 'be' and 'the' would be close to the origin, while infrequent words would be towards the end of the axis.\n",
        "\n",
        "The plot shows that, when words have random embeddings, on average the distance between the prediction for each word and the word embedding is close to 0."
      ]
    },
    {
      "metadata": {
        "id": "rUN0Pj5G3fgu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "These plots can be useful for detecting implementation bugs. For example, when we were implementing the `CogitoPrep` utility for counting co-occurrences for lemmas and concepts, we generated the following plot:\n",
        "\n",
        "![Buggy embeddings](https://github.com/hybridNLP2018/tutorial/blob/master/images/correlationbug-avg_token_cosine_similarity_skipgram_10.PNG?raw=1)\n",
        "\n",
        "This showed that we were learning to predict frequent words and some non-frequent words, but that we were not learning most non-frequent words correctly."
      ]
    },
    {
      "metadata": {
        "id": "PgnM4m4n3fgv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "After fixing the bug, we got the following plot:\n",
        "\n",
        "![uncentered](https://github.com/hybridNLP2018/tutorial/blob/master/images/uncentered-avg_token_cosine_similarity_skipgram_4.PNG?raw=1)\n",
        "\n",
        "This shows that now we were able to learn embeddings that improved word prediction across the whole vocabulary. But it also showed that prediction for the most frequent words lagged behind more uncommon words."
      ]
    },
    {
      "metadata": {
        "id": "25M5AnWj3fgw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "After applying some vector normalization techniques to Swivel and re-centering the vectors (we noticed that the centroid of all the vocabulary embeddings was not the origin), we got:\n",
        "\n",
        "![recentered](https://github.com/hybridNLP2018/tutorial/blob/master/images/recentered-es10k-avg_token_cosine_similarity_average_rowcol__skipgram__harmonic__5.PNG?raw=1)\n",
        "\n",
        "This shows better overall prediction."
      ]
    },
    {
      "metadata": {
        "id": "5LsJVHUr3fgx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Conclusion for Word Prediction\n",
        "\n",
        "Pros:\n",
        " - provides a single objective metric\n",
        " - does not require human annotation (although it may requiring pre-processing of the test corpus)\n",
        " - allows to re-use the tokenization steps used during embedding creation.\n",
        " - can be used to generate plots, which can provide insights about implementation or representation issues \n",
        " \n",
        " \n",
        "Cons:\n",
        " - there are no standard test corpora\n",
        " - can be slow to generate the metric for large test corpus. We recommend balancing the size of the test corpus to maximise the vocabulary coverage, while minimising the time required to process the corpus."
      ]
    },
    {
      "metadata": {
        "id": "C-NJtM2y3fgz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Extrinsic Evaluation\n",
        "\n",
        "In Extrinsic Evaluations, we have a more complex task we are interested in (e.g. text classification, text translation, image captioning), whereby we can use embeddings as a way to represent words (or tokens). Assuming we have:\n",
        " - a model architecture and \n",
        " - a corpus for training and evaluation (for which the embeddings provide adequate coverage), \n",
        " \n",
        "we can then train the model using different embeddings and evaluate its overall performance. The idea is that better embeddings will make it easier for the model to learn the overall task.\n"
      ]
    }
  ]
}