examples/colab/text_classification_with_tf_hub_on_kaggle.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "N6ZDpd9XzFeN"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "KUu4vOt5zI9d"
      },
      "outputs": [],
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ok9PfyoQ2rH_"
      },
      "source": [
        "# How to solve a problem on Kaggle with TF-Hub\n",
        "\n",
        "Note: You can run this notebook [live in Colab](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/text_classification_with_tf_hub_on_kaggle.ipynb) with zero setup.\n",
        "\n",
        "TF-Hub is a platform to share machine learning expertise packaged in reusable resources, notably pre-trained **modules**. In this tutorial, we will use a TF-Hub text embedding module to train a simple sentiment classifier with a reasonable baseline accuracy. We will then submit the predictions to Kaggle.\n",
        "\n",
        "For more detailed tutorial on text classification with TF-Hub and further steps for improving the accuracy, take a look at [Text classification with TF-Hub](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/text_classification_with_tf_hub.ipynb)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "_8N3Hx2dyUC-"
      },
      "outputs": [],
      "source": [
        "# Install the latest TensorFlow version.\n",
        "!pip install --quiet --upgrade --pre tensorflow\n",
        "# Install TF-Hub.\n",
        "!pip install tensorflow-hub\n",
        "# Install API to download the data directly from Kaggle.\n",
        "!pip install kaggle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TvdcyyNft4kT"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "v7hy0bhngTUp"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "import zipfile\n",
        "\n",
        "from sklearn import model_selection"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JvgBdeMsuu_3"
      },
      "source": [
        "Since this tutorial will be using a dataset from Kaggle, it requires [creating an API Token](https://github.com/Kaggle/kaggle-api) for your Kaggle account, and uploading it to the Colab environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "nI7C-Zc4urOH"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from google.colab import files\n",
        "\n",
        "# Upload the API token.\n",
        "def get_kaggle_credentials():\n",
        "  token_dir = \"/content/.kaggle\"\n",
        "  token_file = os.path.join(token_dir, \"kaggle.json\")\n",
        "  if not os.path.isdir(token_dir):\n",
        "    os.mkdir(token_dir)\n",
        "  if not os.path.isfile(token_file):\n",
        "    uploaded = files.upload()\n",
        "    with open(token_file, \"w\") as f:\n",
        "      f.write(uploaded[\"kaggle.json\"])\n",
        "    os.chmod(token_file, 600)\n",
        "\n",
        "get_kaggle_credentials()\n",
        "# Note: Only import kaggle after adding the credentials.\n",
        "import kaggle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6OPyVxHuiTEE"
      },
      "source": [
        "# Getting started\n",
        "\n",
        "## Data\n",
        "We will try to solve the [Sentiment Analysis on Movie Reviews](https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews/data) task from Kaggle. The dataset consists of syntactic subphrases of the Rotten Tomatoes movie reviews. The task is to label the phrases as **negative** or **positive** on the scale from 1 to 5."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "rKzc-fOGV72G"
      },
      "outputs": [],
      "source": [
        "SENTIMENT_LABELS = [\n",
        "    \"negative\", \"somewhat negative\", \"neutral\", \"somewhat positive\", \"positive\"\n",
        "]\n",
        "\n",
        "# Add a column with readable values representing the sentiment.\n",
        "def add_readable_labels_column(df, sentiment_value_column):\n",
        "  df[\"SentimentLabel\"] = df[sentiment_value_column].replace(\n",
        "      range(5), SENTIMENT_LABELS)\n",
        "\n",
        "\n",
        "# Download data from Kaggle and create a DataFrame.\n",
        "def load_data_from_zip(competition, file):\n",
        "  with zipfile.ZipFile(os.path.join(competition, file), \"r\") as zip_ref:\n",
        "    unzipped_file = zip_ref.namelist()[0]\n",
        "    zip_ref.extractall(competition)\n",
        "    return pd.read_csv(\n",
        "        os.path.join(competition, unzipped_file), sep=\"\\t\", index_col=0)\n",
        "\n",
        "\n",
        "# The data does not come with a validation set so we'll create one from the\n",
        "# training set.\n",
        "def get_data(competition, train_file, test_file, validation_set_ratio=0.1):\n",
        "  kaggle.api.competitionDownloadFiles(competition, competition)\n",
        "  train_df = load_data_from_zip(competition, train_file)\n",
        "  test_df = load_data_from_zip(competition, test_file)\n",
        "\n",
        "  # Add a human readable label.\n",
        "  add_readable_labels_column(train_df, \"Sentiment\")\n",
        "\n",
        "  # We split by sentence ids, because we don't want to have phrases belonging\n",
        "  # to the same sentence in both training and validation set.\n",
        "  train_indices, validation_indices = model_selection.train_test_split(\n",
        "      np.unique(train_df[\"SentenceId\"]),\n",
        "      test_size=validation_set_ratio,\n",
        "      random_state=0)\n",
        "\n",
        "  validation_df = train_df[train_df[\"SentenceId\"].isin(validation_indices)]\n",
        "  train_df = train_df[train_df[\"SentenceId\"].isin(train_indices)]\n",
        "  print(\"Split the training data into %d training and %d validation examples.\" %\n",
        "        (len(train_df), len(validation_df)))\n",
        "\n",
        "  return train_df, validation_df, test_df\n",
        "\n",
        "\n",
        "train_df, validation_df, test_df = get_data(\n",
        "    \"sentiment-analysis-on-movie-reviews\", \"train.tsv.zip\", \"test.tsv.zip\")\n",
        "train_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YPuHgx3BWBOg"
      },
      "source": [
        "## Training an Estimator\n",
        "\n",
        "We will use a premade [DNN Classifier](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier) along with [text_embedding_column](https://github.com/tensorflow/hub/blob/master/docs/api_docs/python/hub/text_embedding_column.md) that applies a TF-Hub module on the given text feature and returns the embedding vectors.\n",
        "\n",
        "*Note: We could model this task also as a regression, see [Text classification with TF-Hub](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/text_classification_with_tf_hub.ipynb).*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "23U30yEkVq4w"
      },
      "outputs": [],
      "source": [
        "# Training input on the whole training set with no limit on training epochs.\n",
        "train_input_fn = tf.estimator.inputs.pandas_input_fn(\n",
        "    train_df, train_df[\"Sentiment\"], num_epochs=None, shuffle=True)\n",
        "\n",
        "# Prediction on the whole training set.\n",
        "predict_train_input_fn = tf.estimator.inputs.pandas_input_fn(\n",
        "    train_df, train_df[\"Sentiment\"], shuffle=False)\n",
        "# Prediction on the validation set.\n",
        "predict_validation_input_fn = tf.estimator.inputs.pandas_input_fn(\n",
        "    validation_df, validation_df[\"Sentiment\"], shuffle=False)\n",
        "# Prediction on the test set.\n",
        "predict_test_input_fn = tf.estimator.inputs.pandas_input_fn(\n",
        "    test_df, shuffle=False)\n",
        "\n",
        "embedded_text_feature_column = hub.text_embedding_column(\n",
        "    key=\"Phrase\", \n",
        "    module_spec=\"https://tfhub.dev/google/nnlm-en-dim128/1\")\n",
        "\n",
        "estimator = tf.estimator.DNNClassifier(\n",
        "    hidden_units=[500, 100],\n",
        "    feature_columns=[embedded_text_feature_column],\n",
        "    n_classes=5,\n",
        "    optimizer=tf.train.AdagradOptimizer(learning_rate=0.003))\n",
        "\n",
        "estimator.train(input_fn=train_input_fn, steps=10000);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "s8j7YTRSe7Pj"
      },
      "source": [
        "# Prediction\n",
        "\n",
        "Run predictions for the validation set and training set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "zbLg5LzGwAfC"
      },
      "outputs": [],
      "source": [
        "train_eval_result = estimator.evaluate(input_fn=predict_train_input_fn)\n",
        "validation_eval_result = estimator.evaluate(input_fn=predict_validation_input_fn)\n",
        "\n",
        "print \"Training set accuracy: {accuracy}\".format(**train_eval_result)\n",
        "print \"Validation set accuracy: {accuracy}\".format(**validation_eval_result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DR2IsTF5vuAX"
      },
      "source": [
        "## Confusion matrix\n",
        "\n",
        "Another very interesting statistic, especially for multiclass problems, is the [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix). The confusion matrix allows visualization of the proportion of correctly and incorrectly labelled examples. We can easily see how much our classifier is biased and whether the distribution of labels makes sense. Ideally the largest fraction of predictions should be distributed along the diagonal."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "nT71CtArpsKz"
      },
      "outputs": [],
      "source": [
        "def get_predictions(estimator, input_fn):\n",
        "  return [x[\"class_ids\"][0] for x in estimator.predict(input_fn=input_fn)]\n",
        "\n",
        "# Create a confusion matrix on training data.\n",
        "with tf.Graph().as_default():\n",
        "  cm = tf.confusion_matrix(train_df[\"Sentiment\"],\n",
        "                           get_predictions(estimator, predict_train_input_fn))\n",
        "  with tf.Session() as session:\n",
        "    cm_out = session.run(cm)\n",
        "\n",
        "# Normalize the confusion matrix so that each row sums to 1.\n",
        "cm_out = cm_out.astype(float) / cm_out.sum(axis=1)[:, np.newaxis]\n",
        "\n",
        "sns.heatmap(\n",
        "    cm_out,\n",
        "    annot=True,\n",
        "    xticklabels=SENTIMENT_LABELS,\n",
        "    yticklabels=SENTIMENT_LABELS)\n",
        "plt.xlabel(\"Predicted\")\n",
        "plt.ylabel(\"True\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Pic7o2m04weY"
      },
      "source": [
        "We can easily submit the predictions back to Kaggle.\n",
        "\n",
        "```\n",
        "test_df[\"Predictions\"] = get_predictions(estimator, predict_test_input_fn)\n",
        "test_df.to_csv(\n",
        "    tf.gfile.GFile(\"predictions.csv\", \"w\"),\n",
        "    columns=[\"Predictions\"],\n",
        "    header=[\"Sentiment\"])\n",
        "kaggle.api.competitionSubmit(\"predictions.csv\", \"Submitted from Colab\",\n",
        "                             \"sentiment-analysis-on-movie-reviews\")\n",
        "```\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "N6ZDpd9XzFeN"
      ],
      "default_view": {},
      "name": "How to solve a problem on Kaggle with TF-Hub",
      "provenance": [],
      "version": "0.3.2",
      "views": {}
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}