hw_wordrelatedness.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework and bake-off: Word relatedness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2021\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [Development dataset](#Development-dataset)\n",
    "  1. [Vocabulary](#Vocabulary)\n",
    "  1. [Score distribution](#Score-distribution)\n",
    "  1. [Repeated pairs](#Repeated-pairs)\n",
    "1. [Evaluation](#Evaluation)\n",
    "1. [Error analysis](#Error-analysis)\n",
    "1. [Homework questions](#Homework-questions)\n",
    "  1. [PPMI as a baseline [0.5 points]](#PPMI-as-a-baseline-[0.5-points])\n",
    "  1. [Gigaword with LSA at different dimensions [0.5 points]](#Gigaword-with-LSA-at-different-dimensions-[0.5-points])\n",
    "  1. [t-test reweighting [2 points]](#t-test-reweighting-[2-points])\n",
    "  1. [Pooled BERT representations [1 point]](#Pooled-BERT-representations-[1-point])\n",
    "  1. [Learned distance functions [2 points]](#Learned-distance-functions-[2-points])\n",
    "  1. [Your original system [3 points]](#Your-original-system-[3-points])\n",
    "1. [Bake-off [1 point]](#Bake-off-[1-point])\n",
    "1. [Submission Instruction](#Submission-Instruction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "Word similarity and relatedness datasets have long been used to evaluate distributed representations. This notebook provides code for conducting such analyses with a new word relatedness datasets. It consists of word pairs, each with an associated human-annotated relatedness score. \n",
    "\n",
    "The evaluation metric for each dataset is the [Spearman correlation coefficient $\\rho$](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) between the annotated scores and your distances, as is standard in the literature.\n",
    "\n",
    "This homework ([questions at the bottom of this notebook](#Homework-questions)) asks you to write code that uses the count matrices in `data/vsmdata` to create and evaluate some baseline models. The final question asks you to create your own original system for this task, using any data you wish. This accounts for 9 of the 10 points for this assignment.\n",
    "\n",
    "For the associated bake-off, we will distribute a new dataset, and you will evaluate your original system (no additional training or tuning allowed!) on that datasets and submit your predictions. Systems that enter will receive the additional homework point, and systems that achieve the top score will receive an additional 0.5 points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "import csv\n",
    "import itertools\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import random\n",
    "from scipy.stats import spearmanr\n",
    "\n",
    "import vsm\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.fix_random_seeds()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VSM_HOME = os.path.join('data', 'vsmdata')\n",
    "\n",
    "DATA_HOME = os.path.join('data', 'wordrelatedness')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Development dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use development dataset freely, since our bake-off evalutions involve a new test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_df = pd.read_csv(\n",
    "    os.path.join(DATA_HOME, \"cs224u-wordrelatedness-dev.csv\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset consists of word pairs with scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives the number of word pairs in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test set will contain 1500 word pairs with scores of the same type. No word pair in the development set appears in the test set, but some of the individual words are repeated in the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full vocabulary in the dataframe can be extracted as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_vocab = set(dev_df.word1.values) | set(dev_df.word2.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(dev_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The vocabulary for the bake-off test is different – it is partly overlapping with the above. If you want to be sure ahead of time that your system has a representation for every word in the dev and test sets, then you can check against the vocabularies of any of the VSMs in `data/vsmdata` (which all have the same vocabulary). For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "task_index = pd.read_csv(\n",
    "    os.path.join(VSM_HOME, 'yelp_window5-scaled.csv.gz'),\n",
    "    usecols=[0], index_col=0)\n",
    "\n",
    "full_task_vocab = list(task_index.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(full_task_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you can process every one of those words, then you are all set. Alternatively, you can wait to see the test set and make system adjustments to ensure that you can process all those words. This is fine as long as you are not tuning your predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the scores fall in $[0, 1]$, and the dataset skews towards words with low scores, meaning low relatedness:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = dev_df.plot.hist().set_xlabel(\"Relatedness score\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeated pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The development data has some word pairs with multiple distinct scores in it. Here we create a `pd.Series` that contains these word pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "repeats = dev_df.groupby(['word1', 'word2']).apply(lambda x: x.score.var())\n",
    "\n",
    "repeats = repeats[repeats > 0].sort_values(ascending=False)\n",
    "\n",
    "repeats.name = 'score variance'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "repeats.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pd.Series` is sorted with the highest variance items at the top:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "repeats.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is development data, it is up to you how you want to handle these repeats. The test set has no repeated pairs in it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our evaluation function is `vsm.word_relatedness_evaluation`. Its arguments:\n",
    "    \n",
    "1. A relatedness dataset `pd.DataFrame` – e.g., `dev_df` as given above.\n",
    "1. A VSM `pd.DataFrame` – e.g., `giga5` or some transformation thereof, or a GloVe embedding space, or something you have created on your own. The function checks that you can supply a representation for every word in `dev_df` and raises an exception if you can't.\n",
    "1. Optionally a `distfunc` argument, which defaults to `vsm.cosine`.\n",
    "\n",
    "The function returns a tuple:\n",
    "\n",
    "1. A copy of `dev_df` with a new column giving your predictions.\n",
    "1. The Spearman $\\rho$ value (our primary score).\n",
    "\n",
    "Important note: Internally, `vsm.word_relatedness_evaluation` uses `-distfunc(x1, x2)` as its score, where `x1` and `x2` are vector representations of words. This is because the scores in our data are _positive_ relatedness scores, whereas we are assuming that `distfunc` is a _distance_ function.\n",
    "\n",
    "Here's a simple illustration using one of our count matrices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_df = pd.read_csv(\n",
    "    os.path.join(VSM_HOME, \"giga_window5-scaled.csv.gz\"), index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_pred_df, count_rho = vsm.word_relatedness_evaluation(dev_df, count_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_rho"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_pred_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's instructive to compare this against a truly random system, which we can create by simply having a custom distance function that returns a random number in [0, 1] for each example, making no use of the VSM itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_scorer(x1, x2):\n",
    "    \"\"\"`x1` and `x2` are vectors, to conform to the requirements\n",
    "    of `vsm.word_relatedness_evaluation`, but this function just\n",
    "    returns a random number in [0, 1].\"\"\"\n",
    "    return random.random()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_pred_df, random_rho = vsm.word_relatedness_evaluation(\n",
    "    dev_df, count_df, distfunc=random_scorer)\n",
    "\n",
    "random_rho"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a truly baseline system!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error analysis\n",
    "\n",
    "For error analysis, we can look at the words with the largest delta between the gold score and the distance value in our VSM. We do these comparisons based on ranks, just as with our primary metric (Spearman $\\rho$), and we normalize both rankings so that they have a comparable number of levels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def error_analysis(pred_df):\n",
    "    pred_df = pred_df.copy()\n",
    "    pred_df['relatedness_rank'] = _normalized_ranking(pred_df.prediction)\n",
    "    pred_df['score_rank'] = _normalized_ranking(pred_df.score)\n",
    "    pred_df['error'] =  abs(pred_df['relatedness_rank'] - pred_df['score_rank'])\n",
    "    return pred_df.sort_values('error')\n",
    "\n",
    "\n",
    "def _normalized_ranking(series):\n",
    "    ranks = series.rank(method='dense')\n",
    "    return ranks / ranks.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Best predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "error_analysis(count_pred_df).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Worst predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "error_analysis(count_pred_df).tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework questions\n",
    "\n",
    "Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PPMI as a baseline [0.5 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The insight behind PPMI is a recurring theme in word representation learning, so it is a natural baseline for our task. This question asks you to write code for conducting such experiments.\n",
    "\n",
    "Your task: write a function called `run_giga_ppmi_baseline` that does the following:\n",
    "\n",
    "1. Reads the Gigaword count matrix with a window of 20 and a flat scaling function into a `pd.DataFrame`, as is done in the VSM notebooks. The file is `data/vsmdata/giga_window20-flat.csv.gz`, and the VSM notebooks provide examples of the needed code.\n",
    "1. Reweights this count matrix with PPMI.\n",
    "1. Evaluates this reweighted matrix using `vsm.word_relatedness_evaluation` on `dev_df` as defined above, with `distfunc` set to the default of `vsm.cosine`.\n",
    "1. Returns the return value of this call to `vsm.word_relatedness_evaluation`.\n",
    "\n",
    "The goal of this question is to help you get more familiar with the code in `vsm` and the function `vsm.word_relatedness_evaluation`.\n",
    "\n",
    "The function `test_run_giga_ppmi_baseline` can be used to test that you've implemented this specification correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_giga_ppmi_baseline():\n",
    "    pass\n",
    "    ##### YOUR CODE HERE\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_run_giga_ppmi_baseline(func):\n",
    "    \"\"\"`func` should be `run_giga_ppmi_baseline\"\"\"\n",
    "    pred_df, rho = func()\n",
    "    rho = round(rho, 3)\n",
    "    expected = 0.351\n",
    "    assert rho == expected, \\\n",
    "        \"Expected rho of {}; got {}\".format(expected, rho)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_run_giga_ppmi_baseline(run_giga_ppmi_baseline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gigaword with LSA at different dimensions [0.5 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We might expect PPMI and LSA to form a solid pipeline that combines the strengths of PPMI with those of dimensionality reduction. However, LSA has a hyper-parameter $k$ – the dimensionality of the final representations – that will impact performance. This problem asks you to create code that will help you explore this approach.\n",
    "\n",
    "Your task: write a wrapper function `run_ppmi_lsa_pipeline` that does the following:\n",
    "\n",
    "1. Takes as input a count `pd.DataFrame` and an LSA parameter `k`.\n",
    "1. Reweights the count matrix with PPMI.\n",
    "1. Applies LSA with dimensionality `k`.\n",
    "1. Evaluates this reweighted matrix using `vsm.word_relatedness_evaluation` with `dev_df` as defined above. The return value of `run_ppmi_lsa_pipeline` should be the return value of this call to `vsm.word_relatedness_evaluation`.\n",
    "\n",
    "The goal of this question is to help you get a feel for how LSA can contribute to this problem. \n",
    "\n",
    "The  function `test_run_ppmi_lsa_pipeline` will test your function on the count matrix in `data/vsmdata/giga_window20-flat.csv.gz`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_ppmi_lsa_pipeline(count_df, k):\n",
    "    pass\n",
    "    ##### YOUR CODE HERE\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_run_ppmi_lsa_pipeline(func):\n",
    "    \"\"\"`func` should be `run_ppmi_lsa_pipeline`\"\"\"\n",
    "    giga20 = pd.read_csv(\n",
    "        os.path.join(VSM_HOME, \"giga_window20-flat.csv.gz\"), index_col=0)\n",
    "    pred_df, rho = func(giga20, k=10)\n",
    "    rho = round(rho, 3)\n",
    "    expected = 0.319\n",
    "    assert rho == expected,\\\n",
    "        \"Expected rho of {}; got {}\".format(expected, rho)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_run_ppmi_lsa_pipeline(run_ppmi_lsa_pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### t-test reweighting [2 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The t-test statistic can be thought of as a reweighting scheme. For a count matrix $X$, row index $i$, and column index $j$:\n",
    "\n",
    "$$\\textbf{ttest}(X, i, j) = \n",
    "\\frac{\n",
    "    P(X, i, j) - \\big(P(X, i, *)P(X, *, j)\\big)\n",
    "}{\n",
    "\\sqrt{(P(X, i, *)P(X, *, j))}\n",
    "}$$\n",
    "\n",
    "where $P(X, i, j)$ is $X_{ij}$ divided by the total values in $X$, $P(X, i, *)$ is the sum of the values in row $i$ of $X$ divided by the total values in $X$, and $P(X, *, j)$ is the sum of the values in column $j$ of $X$ divided by the total values in $X$.\n",
    "\n",
    "Your task: implement this reweighting scheme. You can use `test_ttest_implementation` below to check that your implementation is correct.  You do not need to use this for any evaluations, though we hope you will be curious enough to do so!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ttest(df):\n",
    "    pass\n",
    "    ##### YOUR CODE HERE\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_ttest_implementation(func):\n",
    "    \"\"\"`func` should be `ttest`\"\"\"\n",
    "    X = pd.DataFrame([\n",
    "        [1.,  4.,  3.,  0.],\n",
    "        [2., 43.,  7., 12.],\n",
    "        [5.,  6., 19.,  0.],\n",
    "        [1., 11.,  1.,  4.]])\n",
    "    actual = np.array([\n",
    "        [ 0.04655, -0.01337,  0.06346, -0.09507],\n",
    "        [-0.11835,  0.13406, -0.20846,  0.10609],\n",
    "        [ 0.16621, -0.23129,  0.38123, -0.18411],\n",
    "        [-0.0231 ,  0.0563 , -0.14549,  0.10394]])\n",
    "    predicted = func(X)\n",
    "    assert np.array_equal(predicted.round(5), actual), \\\n",
    "        \"Your ttest result is\\n{}\".format(predicted.round(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_ttest_implementation(ttest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pooled BERT representations [1 point]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The notebook [vsm_04_contextualreps.ipynb](vsm_04_contextualreps.ipynb) explores methods for deriving static vector representations of words from the contextual representations given by models like BERT and RoBERTa. The methods are due to [Bommasani et al. 2020](https://www.aclweb.org/anthology/2020.acl-main.431). The simplest of these methods involves processing the words as independent texts and pooling the sub-word representations that result, using a function like mean or max.\n",
    "\n",
    "Your task: write a function `evaluate_pooled_bert` that will enable exploration of this approach. The function should do the following:\n",
    "\n",
    "1. Take as its arguments (a) a word relatedness `pd.DataFrame` `rel_df` (e.g., `dev_df`), (b) a `layer` index (see below), and (c) a `pool_func` value (see below).\n",
    "1. Set up a BERT tokenizer and BERT model based on `'bert-base-uncased'`.\n",
    "1. Use `vsm.create_subword_pooling_vsm` to create a VSM (a `pd.DataFrame`) with the user's values for `layer` and `pool_func`.\n",
    "1. Return the return value of `vsm.word_relatedness_evaluation` using this new VSM, evaluated on `rel_df` with `distfunc` set to its default value.\n",
    "\n",
    "The function `vsm.create_subword_pooling_vsm` does the heavy-lifting. Your task is really just to put these pieces together. The result will be the start of a flexible framework for seeing how these methods do on our task. \n",
    "\n",
    "The function `test_evaluate_pooled_bert` can help you obtain the design we are seeking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertModel, BertTokenizer\n",
    "\n",
    "def evaluate_pooled_bert(rel_df, layer, pool_func):\n",
    "    bert_weights_name = 'bert-base-uncased'\n",
    "\n",
    "    # Initialize a BERT tokenizer and BERT model based on\n",
    "    # `bert_weights_name`:\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "\n",
    "    # Get the vocabulary from `rel_df`:\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "\n",
    "    # Use `vsm.create_subword_pooling_vsm` with the user's arguments:\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "    # Return the results of the relatedness evalution:\n",
    "    ##### YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_evaluate_pooled_bert(func):\n",
    "    import torch\n",
    "    rel_df = pd.DataFrame([\n",
    "        {'word1': 'porcupine', 'word2': 'capybara', 'score': 0.6},\n",
    "        {'word1': 'antelope', 'word2': 'springbok', 'score': 0.5},\n",
    "        {'word1': 'llama', 'word2': 'camel', 'score': 0.4},\n",
    "        {'word1': 'movie', 'word2': 'play', 'score': 0.3}])\n",
    "    layer = 2\n",
    "    pool_func = vsm.max_pooling\n",
    "    pred_df, rho = evaluate_pooled_bert(rel_df, layer, pool_func)\n",
    "    rho = round(rho, 2)\n",
    "    expected_rho = 0.40\n",
    "    assert rho == expected_rho, \\\n",
    "        \"Expected rho={}; got rho={}\".format(expected_rho, rho)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_evaluate_pooled_bert(evaluate_pooled_bert)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learned distance functions [2 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The presentation thus far leads one to assume that the `distfunc` argument used in the experiments will be a standard vector distance function like `vsm.cosine` or `vsm.euclidean`. However, the framework itself simply requires that this function map two fixed-dimensional vectors to a real number. This opens up a world of possibilities. This question asks you to dip a toe in these waters.\n",
    "\n",
    "Your task: write a function `run_knn_score_model` for models in this class. The function should:\n",
    "\n",
    "1. Take as its arguments (a) a VSM dataframe `vsm_df`, (b) a relatedness dataset (e.g., `dev_df`), and (c) a `test_size` value between 0.0 and 1.0 that can be passed directly to `train_test_split` (see below).\n",
    "1. Create a feature matrix `X`: each word pair in `dev_df` should be represented by the concatenation of the vectors for word1 and word2 from `vsm_df`.\n",
    "1. Create a score vector `y`, which is just the `score` column in `dev_df`.\n",
    "1. Split the dataset `(X, y)` into train and test portions using [sklearn.model_selection.train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html).\n",
    "1. Train an [sklearn.neighbors.KNeighborsRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html#sklearn.neighbors.KNeighborsRegressor) model on the train split from step 4, with default hyperparameters.\n",
    "1. Return the value of the `score` method of the trained `KNeighborsRegressor` model on the test split from step 4.\n",
    "\n",
    "The functions `test_knn_feature_matrix` and `knn_represent` will help you test the crucial representational aspects of this.\n",
    "\n",
    "Note: if you decide to apply this approach to our task as part of an original system, recall that `vsm.create_subword_pooling_vsm` returns `-d` where `d` is the value computed by `distfunc`, since it assumes that `distfunc` is a distance value of some kind rather than a relatedness/similarity value. Since most regression models will return positive scores for positive associations, you will probably want to undo this by having your `distfunc` return the negative of its value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "\n",
    "def run_knn_score_model(vsm_df, dev_df, test_size=0.20):\n",
    "    pass\n",
    "\n",
    "    # Complete `knn_feature_matrix` for this step.\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "\n",
    "    # Get the values of the 'score' column in `dev_df`\n",
    "    # and store them in a list or array `y`.\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "\n",
    "    # Use `train_test_split` to split (X, y) into train and\n",
    "    # test protions, with `test_size` as the test size.\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "\n",
    "    # Instantiate a `KNeighborsRegressor` with default arguments:\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "    # Fit the model on the training data:\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "\n",
    "    # Return the value of `score` for your model on the test split\n",
    "    # you created above:\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "\n",
    "def knn_feature_matrix(vsm_df, rel_df):\n",
    "    pass\n",
    "    # Complete `knn_represent` and use it to create a feature\n",
    "    # matrix `np.array`:\n",
    "    ##### YOUR CODE HERE\n",
    "\n",
    "\n",
    "def knn_represent(word1, word2, vsm_df):\n",
    "    pass\n",
    "    # Use `vsm_df` to get vectors for `word1` and `word2`\n",
    "    # and concatenate them into a single vector:\n",
    "    ##### YOUR CODE HERE\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_knn_feature_matrix(func):\n",
    "    rel_df = pd.DataFrame([\n",
    "        {'word1': 'w1', 'word2': 'w2', 'score': 0.1},\n",
    "        {'word1': 'w1', 'word2': 'w3', 'score': 0.2}])\n",
    "    vsm_df = pd.DataFrame([\n",
    "        [1, 2, 3.],\n",
    "        [4, 5, 6.],\n",
    "        [7, 8, 9.]], index=['w1', 'w2', 'w3'])\n",
    "    expected = np.array([\n",
    "        [1, 2, 3, 4, 5, 6.],\n",
    "        [1, 2, 3, 7, 8, 9.]])\n",
    "    result = func(vsm_df, rel_df)\n",
    "    assert np.array_equal(result, expected), \\\n",
    "        \"Your `knn_feature_matrix` returns: {}\\nWe expect: {}\".format(\n",
    "        result, expected)\n",
    "\n",
    "def test_knn_represent(func):\n",
    "    vsm_df = pd.DataFrame([\n",
    "        [1, 2, 3.],\n",
    "        [4, 5, 6.],\n",
    "        [7, 8, 9.]], index=['w1', 'w2', 'w3'])\n",
    "    result = func('w1', 'w3', vsm_df)\n",
    "    expected = np.array([1, 2, 3, 7, 8, 9.])\n",
    "    assert np.array_equal(result, expected), \\\n",
    "        \"Your `knn_represent` returns: {}\\nWe expect: {}\".format(\n",
    "        result, expected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_knn_represent(knn_represent)\n",
    "    test_knn_feature_matrix(knn_feature_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your original system [3 points]\n",
    "\n",
    "This question asks you to design your own model. You can of course include steps made above (ideally, the above questions informed your system design!), but your model should not be literally identical to any of the above models. Other ideas: retrofitting, autoencoders, GloVe, subword modeling, ... \n",
    "\n",
    "Requirements:\n",
    "\n",
    "1. Your system must work with `vsm.word_relatedness_evaluation`. You are free to specify the VSM and the value of `distfunc`.\n",
    "\n",
    "1. Your code must be self-contained, so that we can work with your model directly in your homework submission notebook. If your model depends on external data or other resources, please submit a ZIP archive containing these resources along with your submission.\n",
    "\n",
    "In the cell below, please provide a brief technical description of your original system, so that the teaching team can gain an understanding of what it does. This will help us to understand your code and analyze all the submissions to identify patterns and strategies. We also ask that you report the best score your system got during development, just to help us understand how systems performed overall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PLEASE MAKE SURE TO INCLUDE THE FOLLOWING BETWEEN THE START AND STOP COMMENTS:\n",
    "#   1) Textual description of your system.\n",
    "#   2) The code for your original system.\n",
    "#   3) The score achieved by your system in place of MY_NUMBER.\n",
    "#        With no other changes to that line.\n",
    "#        You should report your score as a decimal value <=1.0\n",
    "# PLEASE MAKE SURE NOT TO DELETE OR EDIT THE START AND STOP COMMENTS\n",
    "\n",
    "# NOTE: MODULES, CODE AND DATASETS REQUIRED FOR YOUR ORIGINAL SYSTEM\n",
    "# SHOULD BE ADDED BELOW THE 'IS_GRADESCOPE_ENV' CHECK CONDITION. DOING\n",
    "# SO ABOVE THE CHECK MAY CAUSE THE AUTOGRADER TO FAIL.\n",
    "\n",
    "# START COMMENT: Enter your system description in this cell.\n",
    "# My peak score was: MY_NUMBER\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    pass\n",
    "\n",
    "# STOP COMMENT: Please do not remove this comment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bake-off [1 point]\n",
    "\n",
    "For the bake-off, you simply need to evaluate your original system on the file \n",
    "\n",
    "`vsmdata/cs224u-wordrelatedness-bakeoff-test-unlabeled.csv`\n",
    "\n",
    "This contains only word pairs (no scores), so `vsm.word_relatedness_evaluation` will simply make predictions without doing any scoring. Use that function to make predictions with your original system, store the resulting `pred_df` to a file, and then upload the file as your bake-off submission.\n",
    "\n",
    "The following function should be used to conduct this evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_bakeoff_submission(\n",
    "        vsm_df,\n",
    "        distfunc,\n",
    "        output_filename=\"cs224u-wordrelatedness-bakeoff-entry.csv\"):\n",
    "\n",
    "    test_df = pd.read_csv(\n",
    "        os.path.join(DATA_HOME, \"cs224u-wordrelatedness-test-unlabeled.csv\"))\n",
    "\n",
    "    pred_df, _ = vsm.word_relatedness_evaluation(test_df, vsm_df, distfunc=distfunc)\n",
    "\n",
    "    pred_df.to_csv(output_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, if `count_df` were the VSM for my system, and I wanted my distance function to be `vsm.euclidean`, I would do"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_bakeoff_submission(count_df, vsm.euclidean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates a file `cs224u-wordrelatedness-bakeoff-entry.csv` in the current directory. That file should be uploaded as-is. Please do not change its name.\n",
    "\n",
    "Only one upload per team is permitted, and you should do no tuning of your system based on what you see in `pred_df` – you should not study that file in anyway, beyond perhaps checking that it contains what you expected it to contain. The upload function will do some additional checking to ensure that your file is well-formed.\n",
    "\n",
    "People who enter will receive the additional homework point, and people whose systems achieve the top score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.\n",
    "\n",
    "Late entries will be accepted, but they cannot earn the extra 0.5 points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission Instruction\n",
    "\n",
    "Submit the following files to gradescope submission\n",
    "\n",
    "- Please do not change the file name as described below\n",
    "- `hw_wordrelatedness.ipynb` (this notebook)\n",
    "- `cs224u-wordrelatedness-bakeoff-entry.csv` (bake-off output)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}