hw3_rel_ext.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 3: Relation extraction using distant supervision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Bill MacCartney\"\n",
    "__version__ = \"CS224U, Stanford, Spring 2019\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [Baseline](#Baseline)\n",
    "1. [Homework questions](#Homework-questions)\n",
    "  1. [Different model factory [1 point]](#Different-model-factory-[1-point])\n",
    "  1. [Directional unigram features [2 points]](#Directional-unigram-features-[2-points])\n",
    "  1. [The part-of-speech tags of the \"middle\" words [2 points]](#The-part-of-speech-tags-of-the-\"middle\"-words-[2-points])\n",
    "  1. [Your original system [4 points]](#Your-original-system-[4-points])\n",
    "1. [Bake-off [1 point]](#Bake-off-[1-point])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This homework and associated bake-off are devoted to the developing really effective relation extraction systems using distant supervision. \n",
    "\n",
    "As with the previous assignments, this notebook first establishes a baseline system. The initial homework questions ask you to create additional baselines and suggest areas for innovation, and the final homework question asks you to develop an original system for you to enter into the bake-off."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up\n",
    "\n",
    "See [the first notebook in this unit](rel_ext_01_task.ipynb#Set-up) for set-up instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import rel_ext\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, we unite our corpus and KB into a dataset, and create some splits for experimentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "rel_ext_data_home = os.path.join('data', 'rel_ext_data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = rel_ext.Corpus(os.path.join(rel_ext_data_home, 'corpus.tsv.gz'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "kb = rel_ext.KB(os.path.join(rel_ext_data_home, 'kb.tsv.gz'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = rel_ext.Dataset(corpus, kb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are not wedded to this set-up for splits. The bake-off will be conducted on a previously unseen test-set, so all of the data in `dataset` is fair game:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = dataset.build_splits(\n",
    "    split_names=['tiny', 'train', 'dev'],\n",
    "    split_fracs=[0.01, 0.79, 0.20],\n",
    "    seed=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tiny': Corpus with 3,474 examples; KB with 445 triples,\n",
       " 'train': Corpus with 263,285 examples; KB with 36,191 triples,\n",
       " 'dev': Corpus with 64,937 examples; KB with 9,248 triples,\n",
       " 'all': Corpus with 331,696 examples; KB with 45,884 triples}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_bag_of_words_featurizer(kbt, corpus, feature_counter):\n",
    "    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "        for word in ex.middle.split(' '):\n",
    "            feature_counter[word] += 1\n",
    "    for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):\n",
    "        for word in ex.middle.split(' '):\n",
    "            feature_counter[word] += 1\n",
    "    return feature_counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "featurizers = [simple_bag_of_words_featurizer]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_factory = lambda: LogisticRegression(fit_intercept=True, solver='liblinear')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.906      0.371      0.703        340       5716\n",
      "author                    0.785      0.538      0.719        509       5885\n",
      "capital                   0.667      0.189      0.443         95       5471\n",
      "contains                  0.783      0.594      0.736       3904       9280\n",
      "film_performance          0.775      0.568      0.723        766       6142\n",
      "founders                  0.803      0.387      0.661        380       5756\n",
      "genre                     0.561      0.135      0.344        170       5546\n",
      "has_sibling               0.865      0.257      0.587        499       5875\n",
      "has_spouse                0.865      0.325      0.649        594       5970\n",
      "is_a                      0.669      0.219      0.474        497       5873\n",
      "nationality               0.618      0.183      0.419        301       5677\n",
      "parents                   0.877      0.526      0.774        312       5688\n",
      "place_of_birth            0.613      0.210      0.443        233       5609\n",
      "place_of_death            0.528      0.119      0.314        159       5535\n",
      "profession                0.613      0.198      0.432        247       5623\n",
      "worked_at                 0.685      0.252      0.510        242       5618\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.726      0.317      0.558       9248      95264\n"
     ]
    }
   ],
   "source": [
    "baseline_results = rel_ext.experiment(\n",
    "    splits,\n",
    "    train_split='train',\n",
    "    test_split='dev',\n",
    "    featurizers=featurizers,\n",
    "    model_factory=model_factory,\n",
    "    verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Studying model weights might yield insights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Highest and lowest feature weights for relation adjoins:\n",
      "\n",
      "     2.503 Taluks\n",
      "     2.499 Córdoba\n",
      "     2.473 Valais\n",
      "     ..... .....\n",
      "    -1.490 other\n",
      "    -1.519 century\n",
      "    -2.166 Earth\n",
      "\n",
      "Highest and lowest feature weights for relation author:\n",
      "\n",
      "     2.881 books\n",
      "     2.392 author\n",
      "     2.392 poem\n",
      "     ..... .....\n",
      "    -2.741 Alice\n",
      "    -2.972 1852\n",
      "    -6.996 1865\n",
      "\n",
      "Highest and lowest feature weights for relation capital:\n",
      "\n",
      "     3.090 capital\n",
      "     1.715 posted\n",
      "     1.590 Airport\n",
      "     ..... .....\n",
      "    -1.212 and\n",
      "    -1.234 century\n",
      "    -1.420 Bronx\n",
      "\n",
      "Highest and lowest feature weights for relation contains:\n",
      "\n",
      "     2.819 third-largest\n",
      "     2.198 transferred\n",
      "     2.151 bounded\n",
      "     ..... .....\n",
      "    -3.359 6th\n",
      "    -3.725 Wayne\n",
      "    -6.022 Bronx\n",
      "\n",
      "Highest and lowest feature weights for relation film_performance:\n",
      "\n",
      "     4.016 starring\n",
      "     3.767 alongside\n",
      "     3.693 co-starring\n",
      "     ..... .....\n",
      "    -1.943 Malice\n",
      "    -1.951 Gandolfini\n",
      "    -1.987 Zigzag\n",
      "\n",
      "Highest and lowest feature weights for relation founders:\n",
      "\n",
      "     3.862 co-founder\n",
      "     3.576 founder\n",
      "     3.531 founded\n",
      "     ..... .....\n",
      "    -1.848 music\n",
      "    -1.936 novel\n",
      "    -1.945 band\n",
      "\n",
      "Highest and lowest feature weights for relation genre:\n",
      "\n",
      "     2.518 series\n",
      "     2.456 movie\n",
      "     2.431 game\n",
      "     ..... .....\n",
      "    -1.636 his\n",
      "    -1.897 at\n",
      "    -1.948 follows\n",
      "\n",
      "Highest and lowest feature weights for relation has_sibling:\n",
      "\n",
      "     5.652 brother\n",
      "     4.047 sister\n",
      "     2.932 Marlon\n",
      "     ..... .....\n",
      "    -1.422 friend\n",
      "    -1.622 sultan\n",
      "    -1.999 formed\n",
      "\n",
      "Highest and lowest feature weights for relation has_spouse:\n",
      "\n",
      "     5.447 wife\n",
      "     5.090 husband\n",
      "     4.503 married\n",
      "     ..... .....\n",
      "    -1.992 Straus\n",
      "    -1.992 Isidor\n",
      "    -2.853 friend\n",
      "\n",
      "Highest and lowest feature weights for relation is_a:\n",
      "\n",
      "     2.763 \n",
      "     2.541 genus\n",
      "     2.534 Genus\n",
      "     ..... .....\n",
      "    -1.598 during\n",
      "    -1.634 at\n",
      "    -5.540 characin\n",
      "\n",
      "Highest and lowest feature weights for relation nationality:\n",
      "\n",
      "     2.692 born\n",
      "     1.903 caliph\n",
      "     1.888 President\n",
      "     ..... .....\n",
      "    -1.426 part\n",
      "    -1.587 American\n",
      "    -1.589 state\n",
      "\n",
      "Highest and lowest feature weights for relation parents:\n",
      "\n",
      "     5.157 son\n",
      "     4.824 daughter\n",
      "     4.253 father\n",
      "     ..... .....\n",
      "    -1.993 geologist\n",
      "    -2.111 VIII\n",
      "    -2.189 Jolie\n",
      "\n",
      "Highest and lowest feature weights for relation place_of_birth:\n",
      "\n",
      "     3.864 born\n",
      "     2.983 birthplace\n",
      "     2.640 mayor\n",
      "     ..... .....\n",
      "    -1.378 or\n",
      "    -1.452 and\n",
      "    -2.205 Oldham\n",
      "\n",
      "Highest and lowest feature weights for relation place_of_death:\n",
      "\n",
      "     2.316 died\n",
      "     1.839 prominent\n",
      "     1.804 rebuilt\n",
      "     ..... .....\n",
      "    -1.119 or\n",
      "    -1.203 ”\n",
      "    -1.250 and\n",
      "\n",
      "Highest and lowest feature weights for relation profession:\n",
      "\n",
      "     3.306 \n",
      "     2.574 American\n",
      "     2.365 English\n",
      "     ..... .....\n",
      "    -1.300 at\n",
      "    -1.327 from\n",
      "    -1.977 on\n",
      "\n",
      "Highest and lowest feature weights for relation worked_at:\n",
      "\n",
      "     3.075 CEO\n",
      "     2.849 professor\n",
      "     2.798 employee\n",
      "     ..... .....\n",
      "    -1.305 ”\n",
      "    -1.454 Thiruvananthapuram\n",
      "    -1.650 or\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rel_ext.examine_model_weights(baseline_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework questions\n",
    "\n",
    "Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Different model factory [1 point]\n",
    "\n",
    "The code in `rel_ext` makes it very easy to experiment with other classifier models: one need only redefine the `model_factory` argument. This question asks you to assess a [Support Vector Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html).\n",
    "\n",
    "__To submit:__ A call to `rel_ext.experiment` training on the 'train' part of `splits` and assessing on its `dev` part, with `featurizers` as defined above in this notebook and the `model_factory` set to one based in an `SVC` with `kernel='linear'` and all other arguments left with default values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Directional unigram features [2 points]\n",
    "\n",
    "The current bag-of-words representation makes no distinction between \"forward\" and \"reverse\" examples. But, intuitively, there is big difference between _X and his son Y_ and _Y and his son X_. This question asks you to modify `simple_bag_of_words_featurizer` to capture these differences. \n",
    "\n",
    "__To submit:__\n",
    "\n",
    "1. A feature function `directional_bag_of_words_featurizer` that is just like `simple_bag_of_words_featurizer` except that it distinguishes \"forward\" and \"reverse\". To do this, you just need to mark each word feature for whether it is derived from a subject–object example or from an object–subject example. The precise nature of the mark you add for the two cases doesn't make a difference to the model.\n",
    "\n",
    "2. The macro-average F-score on the `dev` set that you obtain from running `rel_ext.experiment` with `directional_bag_of_words_featurizer` as the only featurizer. (Aside from this, use all the default values for `experiment` as exemplified above in this notebook.)\n",
    "\n",
    "3. `rel_ext.experiment` returns some of the core objects used in the experiment. How many feature names does the `vectorizer` have for the experiment run in the previous step? (Note: we're partly asking you to figure out how to get this value by using the sklearn documentation, so please don't ask how to do it on Piazza!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The part-of-speech tags of the \"middle\" words [2 points]\n",
    "\n",
    "Our corpus distribution contains part-of-speech (POS) tagged versions of the core text spans. Let's begin to explore whether there is information in these sequences, focusing on `middle_POS`.\n",
    "\n",
    "__To submit:__\n",
    "\n",
    "1. A feature function `middle_bigram_pos_tag_featurizer` that is just like `simple_bag_of_words_featurizer` except that it creates a feature for bigram POS sequences. For example, given \n",
    "\n",
    "  `The/DT dog/N napped/V`\n",
    "  \n",
    "   we obtain the list of bigram POS sequences\n",
    "  \n",
    "   `['<s> DT', 'DT N', 'N V', 'V </s>']`. \n",
    "   \n",
    "   Don't forget the start and end tags, to model those environments properly!\n",
    "\n",
    "2. The macro-average F-score on the `dev` set that you obtain from running `rel_ext.experiment` with `middle_bigram_pos_tag_featurizer` as the only featurizer. (Aside from this, use all the default values for `experiment` as exemplified above in this notebook.)\n",
    "\n",
    "Note: To parse `middle_POS`, one splits on whitespace to get the `word/TAG` pairs. Each of these pairs `s` can be parsed with `s.rsplit('/', 1)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your original system [4 points]\n",
    "\n",
    "There are many options, and this could easily grow into a project. Here are a few ideas:\n",
    "\n",
    "- Try out different classifier models, from `sklearn` and elsewhere.\n",
    "- Add a feature that indicates the length of the middle.\n",
    "- Augment the bag-of-words representation to include bigrams or trigrams (not just unigrams).\n",
    "- Introduce features based on the entity mentions themselves. <!-- \\[SPOILER: it helps a lot, maybe 4% in F-score. And combines nicely with the directional features.\\] -->\n",
    "- Experiment with features based on the context outside (rather than between) the two entity mentions — that is, the words before the first mention, or after the second.\n",
    "- Try adding features which capture syntactic information, such as the dependency-path features used by Mintz et al. 2009. The [NLTK](https://www.nltk.org/) toolkit contains a variety of [parsing algorithms](http://www.nltk.org/api/nltk.parse.html) that may help.\n",
    "- The bag-of-words representation does not permit generalization across word categories such as names of people, places, or companies. Can we do better using word embeddings such as [GloVe](https://nlp.stanford.edu/projects/glove/)?\n",
    "- Consider adding features based on WordNet synsets. Here's a little code to get you started with that:\n",
    "  ```\n",
    "  from nltk.corpus import wordnet as wn\n",
    "  dog_compatible_synsets = wn.synsets('dog', pos='n')\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bake-off [1 point]\n",
    "\n",
    "For the bake-off, we will release a test set right after class on April 29. The announcement will go out on Piazza. You will evaluate your custom model from the previous question on these new datasets using the function `rel_ext.bake_off_experiment`. Rules:\n",
    "\n",
    "1. Only one evaluation is permitted.\n",
    "1. No additional system tuning is permitted once the bake-off has started.\n",
    "\n",
    "The bake-off submission will be via a Google form (link to be given in the announcement). It will ask for the following:\n",
    "\n",
    "1. A description of your original system.\n",
    "1. Your macro-averaged F<sub>0.5</sub>-score on the `test` split, as reported by `rel_ext.bake_off_experiment`.\n",
    "\n",
    "People who enter will receive the additional homework point, and people whose systems achieve at least the third-best score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.\n",
    "\n",
    "The bake-off will close at 4:30 pm on May 1. Late entries will be accepted, but they cannot earn the extra 0.5 points. Similarly, you cannot win the bake-off unless your homework is submitted on time."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}