examples/colab/action_recognition_with_tf_hub.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Action Recognition on the UCF101 Dataset",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": [
        "x8Q7Un821X1A"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "x8Q7Un821X1A",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "metadata": {
        "id": "1W4rIAFt1Ui3",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cDq0CIKc1vO_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# TF-Hub Action Recognition Model\n",
        "\n",
        "<table align=\"left\"><td>\n",
        "  <a target=\"_blank\"  href=\"https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/action_recognition_with_tf_hub.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab\n",
        "  </a>\n",
        "</td><td>\n",
        "  <a target=\"_blank\"  href=\"https://github.com/tensorflow/hub/blob/master/examples/colab/action_recognition_with_tf_hub.ipynb\">\n",
        "    <img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "</td></table>"
      ]
    },
    {
      "metadata": {
        "id": "h6W3FhoP3TxC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This Colab demonstrates use of action recognition from video data using the\n",
        "[tfhub.dev/deepmind/i3d-kinetics-400/1](https://tfhub.dev/deepmind/i3d-kinetics-400/1) module.\n",
        "\n",
        "The underlying model is described in the paper \"[Quo Vadis, Action Recognition? A New\n",
        "Model and the Kinetics Dataset](https://arxiv.org/abs/1705.07750)\" by Joao\n",
        "Carreira and Andrew Zisserman. The paper was posted on arXiv in May 2017, and\n",
        "was published as a CVPR 2017 conference paper.\n",
        "The source code is publicly available on\n",
        "[github](https://github.com/deepmind/kinetics-i3d).\n",
        "\n",
        "\"Quo Vadis\" introduced a new architecture for video classification, the Inflated\n",
        "3D Convnet or I3D. This architecture achieved state-of-the-art results on the UCF101\n",
        "and HMDB51 datasets from fine-tuning these models. I3D models pre-trained on Kinetics\n",
        "also placed first in the CVPR 2017 [Charades challenge](http://vuchallenge.org/charades.html).\n",
        "\n",
        "The original module was trained on the [kinetics-400 dateset](https://deepmind.com/research/open-source/open-source-datasets/kinetics/)\n",
        "and knows about 400 diferrent actions.\n",
        "Labels for these actions can be found in the\n",
        "[label map file](https://github.com/deepmind/kinetics-i3d/blob/master/data/label_map.txt).\n",
        "\n",
        "In this Colab we will use it recognize activites in videos from a UCF101 dataset."
      ]
    },
    {
      "metadata": {
        "id": "R_0xc2jyNGRp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Setting up the environment"
      ]
    },
    {
      "metadata": {
        "id": "mOHMWsFnITdi",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Install the necessary python packages.\n",
        "!pip install -q \"tensorflow>=1.7\" \"tensorflow-hub\" \"imageio\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "USf0UvkYIlKo",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "cellView": "form"
      },
      "cell_type": "code",
      "source": [
        "#@title Import the necessary modules\n",
        "# TensorFlow and TF-Hub modules.\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "\n",
        "# Some modules to help with reading the UCF101 dataset.\n",
        "import random\n",
        "import re\n",
        "import os\n",
        "import tempfile\n",
        "import cv2\n",
        "import numpy as np\n",
        "\n",
        "# Some modules to display an animation using imageio.\n",
        "import imageio\n",
        "from IPython import display\n",
        "\n",
        "from urllib import request  # requires python3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IuMMS3TGdws7",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "cellView": "form"
      },
      "cell_type": "code",
      "source": [
        "#@title Helper functions for the UCF101 dataset\n",
        "\n",
        "# Utilities to fetch videos from UCF101 dataset\n",
        "UCF_ROOT = \"http://crcv.ucf.edu/THUMOS14/UCF101/UCF101/\"\n",
        "_VIDEO_LIST = None\n",
        "_CACHE_DIR = tempfile.mkdtemp()\n",
        "\n",
        "def list_ucf_videos():\n",
        "  \"\"\"Lists videos available in UCF101 dataset.\"\"\"\n",
        "  global _VIDEO_LIST\n",
        "  if not _VIDEO_LIST:\n",
        "    index = request.urlopen(UCF_ROOT).read().decode(\"utf-8\")\n",
        "    videos = re.findall(\"(v_[\\w_]+\\.avi)\", index)\n",
        "    _VIDEO_LIST = sorted(set(videos))\n",
        "  return list(_VIDEO_LIST)\n",
        "\n",
        "def fetch_ucf_video(video):\n",
        "  \"\"\"Fetchs a video and cache into local filesystem.\"\"\"\n",
        "  cache_path = os.path.join(_CACHE_DIR, video)\n",
        "  if not os.path.exists(cache_path):\n",
        "    urlpath = request.urljoin(UCF_ROOT, video)\n",
        "    print(\"Fetching %s => %s\" % (urlpath, cache_path))\n",
        "    data = request.urlopen(urlpath).read()\n",
        "    open(cache_path, \"wb\").write(data)\n",
        "  return cache_path\n",
        "\n",
        "# Utilities to open video files using CV2\n",
        "def crop_center_square(frame):\n",
        "  y, x = frame.shape[0:2]\n",
        "  min_dim = min(y, x)\n",
        "  start_x = (x // 2) - (min_dim // 2)\n",
        "  start_y = (y // 2) - (min_dim // 2)\n",
        "  return frame[start_y:start_y+min_dim,start_x:start_x+min_dim]\n",
        "\n",
        "def load_video(path, max_frames=0, resize=(224, 224)):\n",
        "  cap = cv2.VideoCapture(path)\n",
        "  frames = []\n",
        "  try:\n",
        "    while True:\n",
        "      ret, frame = cap.read()\n",
        "      if not ret:\n",
        "        break\n",
        "      frame = crop_center_square(frame)\n",
        "      frame = cv2.resize(frame, resize)\n",
        "      frame = frame[:, :, [2, 1, 0]]\n",
        "      frames.append(frame)\n",
        "      \n",
        "      if len(frames) == max_frames:\n",
        "        break\n",
        "  finally:\n",
        "    cap.release()\n",
        "  return np.array(frames) / 255.0\n",
        "\n",
        "def animate(images):\n",
        "  converted_images = np.clip(images * 255, 0, 255).astype(np.uint8)\n",
        "  imageio.mimsave('./animation.gif', converted_images, fps=25)\n",
        "  with open('./animation.gif','rb') as f:\n",
        "      display.display(display.Image(data=f.read(), height=300))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pIKTs-KneUfz",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "cellView": "form"
      },
      "cell_type": "code",
      "source": [
        "#@title Get the kinetics-400 labels\n",
        "# Get the kinetics-400 action labels from the GitHub repository.\n",
        "KINETICS_URL = \"https://raw.githubusercontent.com/deepmind/kinetics-i3d/master/data/label_map.txt\"\n",
        "with request.urlopen(KINETICS_URL) as obj:\n",
        "  labels = [line.decode(\"utf-8\").strip() for line in obj.readlines()]\n",
        "print(\"Found %d labels.\" % len(labels))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GBvmjVICIp3W",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Using the UCF101 dataset"
      ]
    },
    {
      "metadata": {
        "id": "V-QcxdhLIfi2",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Get the list of videos in the dataset.\n",
        "ucf_videos = list_ucf_videos()\n",
        "  \n",
        "categories = {}\n",
        "for video in ucf_videos:\n",
        "  category = video[2:-12]\n",
        "  if category not in categories:\n",
        "    categories[category] = []\n",
        "  categories[category].append(video)\n",
        "print(\"Found %d videos in %d categories.\" % (len(ucf_videos), len(categories)))\n",
        "\n",
        "for category, sequences in categories.items():\n",
        "  summary = \", \".join(sequences[:2])\n",
        "  print(\"%-20s %4d videos (%s, ...)\" % (category, len(sequences), summary))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "c0ZvVDruN2nU",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Get a sample cricket video.\n",
        "sample_video = load_video(fetch_ucf_video(\"v_CricketShot_g04_c02.avi\"))\n",
        "\n",
        "print(\"sample_video is a numpy array of shape %s.\" % str(sample_video.shape))\n",
        "animate(sample_video)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Yjk_SlQUhq3v",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Run the i3d model on the video and print the top 5 actions.\n",
        "\n",
        "# First add an empty dimension to the sample video as the model takes as input\n",
        "# a batch of videos.\n",
        "model_input = np.expand_dims(sample_video, axis=0)\n",
        "\n",
        "# Create the i3d model and get the action probabilities.\n",
        "with tf.Graph().as_default():\n",
        "  i3d = hub.Module(\"https://tfhub.dev/deepmind/i3d-kinetics-400/1\")\n",
        "  input_placeholder = tf.placeholder(shape=(None, None, 224, 224, 3), dtype=tf.float32)\n",
        "  logits = i3d(input_placeholder)\n",
        "  probabilities = tf.nn.softmax(logits)\n",
        "  with tf.train.MonitoredSession() as session:\n",
        "    [ps] = session.run(probabilities,\n",
        "                       feed_dict={input_placeholder: model_input})\n",
        "\n",
        "print(\"Top 5 actions:\")\n",
        "for i in np.argsort(ps)[::-1][:5]:\n",
        "  print(\"%-22s %.2f%%\" % (labels[i], ps[i] * 100))"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}