material/notebooks/Session_1.3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we provided code that enables you to train Word Embeddings with the `glove` method.  \n",
    "\n",
    "Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation.  \n",
    "\n",
    "This notebook creates three different Word Embeddings, one trained on aggressive comments, one trained on non-aggressive comments and one trained on both types of comments.  \n",
    "\n",
    "You could either:\n",
    "\n",
    "* Run the notebook, retrieve the embeddings and test them for biases with the code from Session1.1 and Session 1.2\n",
    "* Modify the notebook to get different Word Embeddings. You could change the dimensionality or other training parameters, you could also change the data on which the embeddings are trained.\n",
    "* In any case, you should have a look at your embeddings. In the last cell of this notebook, each embedding is saved to two .tsv files. Please upload a pair of these embeddings to http://projector.tensorflow.org/ to visualize the embedding. Can you pinpoint directions of bias?\n",
    "\n",
    "\n",
    "(If you are using Google Colab and you would like to use the embeddings for the other sessions, you have to download the embeddings and upload them to the other sessions. (Or save them to your google drive))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Glove"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/millawell/bias-ml-dh.git#subdirectory=material/notebooks/bias_ml_dh_utils\n",
    "!pip install --upgrade tqdm\n",
    "!git clone https://github.com/stanfordnlp/GloVe.git\n",
    "%cd GloVe\n",
    "!make\n",
    "%cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the aggressive wikipedia comments data set\n",
    "Wikipedia released a corpus of comments on their talk pages that they had annotated by crowd workers.\n",
    "\n",
    "The data is released here:  \n",
    "https://meta.wikimedia.org/wiki/Research:Detox/Data_Release  \n",
    "\n",
    "We have already prepared a portion of the data set, namely the aggression comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bias_ml_dh_utils as utils\n",
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "import pickle\n",
    "\n",
    "data_identifier = \"agression_comments_wikipedia\"\n",
    "data_path = utils.download_dataset(data_identifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agression_data = pd.read_pickle(data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comments = []\n",
    "labels = []\n",
    "\n",
    "def regex_filter(comment):\n",
    "    return comment.replace(\"NEWLINE_TOKEN\", \"\\t\")\n",
    "\n",
    "for rev_id, rev in tqdm(agression_data.groupby(\"rev_id\")):\n",
    "    comments.append(regex_filter(rev.iloc[0].comment))\n",
    "    labels.append(rev.aggression.sum()/len(rev) >.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split    \n",
    "train_embeddings, X_other, embeddings_labels, y_other  = train_test_split(comments, labels, random_state=123)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_other, y_other, random_state=456)\n",
    "\n",
    "with open(\"data/wikipedia_toxic_classification_data\", \"wb\") as fout:\n",
    "    pickle.dump((X_other, y_other), fout)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.lang.en import English\n",
    "nlp = English()\n",
    "tokenizer = nlp.Defaults.create_tokenizer(nlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/all_comments_plain.txt\", \"w\") as all_fout:\n",
    "    with open(\"data/aggressive_comments_plain.txt\", \"w\") as agg_fout:\n",
    "        with open(\"data/non_aggressive_comments_plain.txt\", \"w\") as non_agg_fout:\n",
    "            for comment, label in tqdm(\n",
    "                zip(tokenizer.pipe(train_embeddings), embeddings_labels),\n",
    "                total=len(train_embeddings),\n",
    "                desc=\"storing comments\"\n",
    "            ):\n",
    "                out_line = \" \".join([t.text.replace(\"\\n\", \"\\t\") for t in comment])\n",
    "                all_fout.write(\"{}\\n\".format(out_line))\n",
    "                if label == 1:\n",
    "                    agg_fout.write(\"{}\\n\".format(out_line))\n",
    "                else:\n",
    "                    non_agg_fout.write(\"{}\\n\".format(out_line))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls GloVe/build/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!GloVe/build/vocab_count -max-vocab 25000 -min-count 10 < data/all_comments_plain.txt > data/vocab.txt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create cooccurrences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!GloVe/build/cooccur -vocab-file data/vocab.txt < data/non_aggressive_comments_plain.txt > data/non_aggressive_comments_cooccurrences.bin\n",
    "!GloVe/build/cooccur -vocab-file data/vocab.txt < data/aggressive_comments_plain.txt > data/aggressive_comments_cooccurrences.bin\n",
    "!GloVe/build/cooccur -vocab-file data/vocab.txt < data/all_comments_plain.txt > data/all_comments_cooccurrences.bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!GloVe/build/shuffle -verbose 2 -memory 4 < data/non_aggressive_comments_cooccurrences.bin > data/non_aggressive_comments_cooccurrences_shuffled.bin\n",
    "!GloVe/build/shuffle -verbose 2 -memory 4 < data/aggressive_comments_cooccurrences.bin > data/aggressive_comments_cooccurrences_shuffled.bin\n",
    "!GloVe/build/shuffle -verbose 2 -memory 4 < data/all_comments_cooccurrences.bin > data/all_comments_cooccurrences_shuffled.bin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Glove"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!GloVe/build/glove -input-file data/non_aggressive_comments_cooccurrences_shuffled.bin -vocab-file data/vocab.txt -save-file data/non_aggressive_comments_vec -verbose 2 -vector-size 32 -threads 8 -alpha 0.75 -x-max 100.0 -eta 0.05 -binary 0 -model 2\n",
    "!GloVe/build/glove -input-file data/aggressive_comments_cooccurrences_shuffled.bin -vocab-file data/vocab.txt -save-file data/aggressive_comments_vec -verbose 2 -vector-size 32 -threads 8 -alpha 0.75 -x-max 100.0 -eta 0.05 -binary 0 -model 2\n",
    "!GloVe/build/glove -input-file data/all_comments_cooccurrences_shuffled.bin -vocab-file data/vocab.txt -save-file data/all_comments_vec -verbose 2 -vector-size 32 -threads 8 -alpha 0.75 -x-max 100.0 -eta 0.05 -binary 0 -model 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bisect import bisect_left\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "with open(\"data/vocab.txt\") as fin:\n",
    "    vocab,_ = zip(*map(lambda x: x.split(\" \"), fin))\n",
    "vocab = sorted(list(vocab) + [\"<unk>\"])\n",
    "\n",
    "def save_for_tf_projector(embedding, vocab, outdir, identifier):\n",
    "    out_path_data = os.path.join(outdir, \"{}_data.tsv\".format(identifier))\n",
    "    out_path_meta = os.path.join(outdir, \"{}_meta.tsv\".format(identifier))\n",
    "\n",
    "    with open(out_path_data, \"w\") as fout:\n",
    "        for row in embedding:\n",
    "            fout.write(\"{}\\n\".format(\"\\t\".join(map(str, row.tolist()))))\n",
    "    \n",
    "    with open(out_path_meta, \"w\") as fout:\n",
    "        for word in vocab:\n",
    "            fout.write(\"{}\\n\".format(word.encode(\"utf-8\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def index(a, x):\n",
    "    'Locate the leftmost value exactly equal to x'\n",
    "    i = bisect_left(a, x)\n",
    "    if i != len(a) and a[i] == x:\n",
    "        return i\n",
    "    raise ValueError\n",
    "\n",
    "def load_embedding(path_, vocab):\n",
    "    dim = 32\n",
    "    mat = np.empty((len(vocab), dim))\n",
    "    \n",
    "    with open(path_) as fin:\n",
    "        for row in tqdm(fin):\n",
    "            splitted = row.replace(\"\\n\", \"\").split(\" \")\n",
    "            key, vec = splitted[0], splitted[1:]\n",
    "            mat[index(vocab, key)] = vec\n",
    "            \n",
    "    return mat\n",
    "            \n",
    "all_vec = load_embedding(\"data/all_comments_vec.txt\", vocab)\n",
    "aggressive_vec = load_embedding(\"data/aggressive_comments_vec.txt\", vocab)\n",
    "non_aggressive_vec = load_embedding(\"data/non_aggressive_comments_vec.txt\", vocab)\n",
    "\n",
    "save_for_tf_projector(all_vec, vocab, \"data\", \"all_vec\")\n",
    "save_for_tf_projector(aggressive_vec, vocab, \"data\", \"agg_vec\")\n",
    "save_for_tf_projector(non_aggressive_vec, vocab, \"data\", \"noagg_vec\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "wh4",
   "language": "python",
   "name": "wh4"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}