seminar10/MLatFIAN_2021_seminar10_Intro_to_CNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "MLatFIAN-2021-seminar10-Intro-to-CNN.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyNG2itnhAB08pIGP7WNYDBl",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/HSE-LAMBDA/MLatFIAN2021/blob/main/seminar10/MLatFIAN_2021_seminar10_Intro_to_CNN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-9YRgxPNxoR5"
      },
      "source": [
        "# Convolutions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bjAm7BDByib1"
      },
      "source": [
        "!wget https://github.com/HSE-LAMBDA/MLatFIAN2021/raw/main/seminar10/img.npy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_3w6ZXzJt8KL"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PLX7yImcCwF7"
      },
      "source": [
        "## Demonstration: convolving to extract features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C6FoDIUQyv0I"
      },
      "source": [
        "Let's check out the image we have:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BOatDoQmBC9r"
      },
      "source": [
        "img = np.load(\"img.npy\")\n",
        "\n",
        "plt.figure(dpi=150)\n",
        "plt.imshow(img);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RDSkiUcsyyWT"
      },
      "source": [
        "At first, we'll experiment with `tf.nn.conv2d` - the function that performs 2d image convolution.\n",
        "\n",
        "*Note:* this function is designed to work in the context of a neural network (i.e. where input and output come in batches and have multiple channels), so the function expects 4D tensors rather than 2D. We'll write a short wrapper to work with 2D images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gqxS91NrzmCU"
      },
      "source": [
        "def convolve(img, kernel):\n",
        "  return tf.nn.conv2d(\n",
        "      img[None,...,None],\n",
        "      kernel[...,None,None], strides=1, padding='VALID'\n",
        "    ).numpy().squeeze()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUNG0CawzszI"
      },
      "source": [
        "Let's try some simple kernels extracting horizontal and vertical edges:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nZpMpzar2OQ8"
      },
      "source": [
        "kernel_ver_edge = tf.convert_to_tensor(\n",
        "    [[ 1., -1.],\n",
        "     [ 1., -1.]]\n",
        ")\n",
        "kernel_hor_edge = tf.convert_to_tensor(\n",
        "    [[ 1.,  1.],\n",
        "     [-1., -1.]]\n",
        ")\n",
        "\n",
        "vertical_edges = convolve(img, kernel_ver_edge)\n",
        "horizontal_edges = convolve(img, kernel_hor_edge)\n",
        "\n",
        "plt.figure(figsize=(4, 5), dpi=150)\n",
        "plt.subplot(2, 1, 1)\n",
        "plt.imshow(vertical_edges);\n",
        "plt.colorbar()\n",
        "plt.subplot(2, 1, 2)\n",
        "plt.imshow(horizontal_edges);\n",
        "plt.colorbar();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CPKISCxqz1GS"
      },
      "source": [
        "We can combine the result, e.g. like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hq8nOprW3_6g"
      },
      "source": [
        "edges = (vertical_edges**2 + horizontal_edges**2)**0.5\n",
        "plt.figure(dpi=150)\n",
        "plt.imshow(edges)\n",
        "plt.colorbar();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3HliLWlz6_Z"
      },
      "source": [
        "Another example, blurring kernel:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7AIGeOsi_BAl"
      },
      "source": [
        "def blur_n_times(img, n=5):\n",
        "    assert n >= 1\n",
        "    kernel_blur = tf.convert_to_tensor([[1.,  4.,  7.,  4., 1.],\n",
        "                                        [4., 16., 26., 16., 4.],\n",
        "                                        [7., 26., 41., 26., 7.],\n",
        "                                        [4., 16., 26., 16., 4.],\n",
        "                                        [1.,  4.,  7.,  4., 1.]]) / 273\n",
        "\n",
        "    img_blurred = convolve(img, kernel_blur)\n",
        "\n",
        "    for _ in range(n - 1):\n",
        "        img_blurred = convolve(img_blurred, kernel_blur)\n",
        "    return img_blurred\n",
        "\n",
        "edges_blurred = blur_n_times(edges, n=5)\n",
        "plt.imshow(edges_blurred);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X770Mh2A0Wqb"
      },
      "source": [
        "Let's pick up a small patch out of this image:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0X-vXBPF4EW0"
      },
      "source": [
        "edges_blurred = blur_n_times(edges, n=5)\n",
        "edges_subset = edges_blurred[210:243, 246:282]\n",
        "plt.imshow(edges_subset);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ro9iN5up0aHg"
      },
      "source": [
        "What do you think will happen if we use this patch as a kernel when running convolution on the edges image?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZvZ_X8ES702i"
      },
      "source": [
        "plt.figure(figsize=(12, 3.5), dpi=100)\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.imshow(edges_blurred)\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.imshow(convolve(edges_blurred, edges_subset))\n",
        "plt.colorbar();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lL9bNU1s0kCj"
      },
      "source": [
        "Note how this kernel highlighted the location of that shape on the input!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8vIYgMUDDDtl"
      },
      "source": [
        "## Convolutional layer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ry0hAdre05d4"
      },
      "source": [
        "Keras has predefined convolutional layers that make use of the convolution function described above.\n",
        "\n",
        "Note that in the context of deep learning the convolutional kernel is **trainable**, i.e. the network tries to find the best kernel to extract useful features."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ShVolbP9AP6O"
      },
      "source": [
        "# Let's build a layer that takes an image with a single channel and outputs \n",
        "# two-channel feature representation:\n",
        "conv_layer = tf.keras.layers.Conv2D(\n",
        "    filters=2, kernel_size=2)\n",
        "conv_layer.build(input_shape=(None, None, 1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NpCn2kLb1hGI"
      },
      "source": [
        "Note that the kernel is initialized randomly (for optimization):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Elq467gqB-vU"
      },
      "source": [
        "conv_layer.kernel"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ILl2tmYR1nTE"
      },
      "source": [
        "but we can set it to e.g. our edge detecting kernel values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ri17z9UDCJRl"
      },
      "source": [
        "conv_layer.kernel[..., 0, 0].assign(kernel_hor_edge)\n",
        "conv_layer.kernel[..., 0, 1].assign(kernel_ver_edge)\n",
        "conv_layer.kernel"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XJfbMQte1uNV"
      },
      "source": [
        "And now the layer performs exactly the same edge-detecting operation:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9MgArqZ7CWss"
      },
      "source": [
        "# Note how we add the batch and channel dimensions here\n",
        "result = conv_layer(img[None,...,None].astype('float32')).numpy().squeeze()\n",
        "\n",
        "plt.figure(figsize=(10, 4), dpi=100)\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.imshow(result[...,0])\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.imshow(result[...,1]);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ofjxSPFFDtc9"
      },
      "source": [
        "## Ridiculously impractical example: trying to learn the kernels from the 1st demo"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k1_JL4qf3X9y"
      },
      "source": [
        "Let's make a keras model that make a similar transformation to the one we did above (i.e. edge detection + blur). We'll try to learn corresponding kernels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q_RhBYbaEK2B"
      },
      "source": [
        "model = tf.keras.Sequential(\n",
        "    [\n",
        "      # a block to \"reproduce\" edge detection:\n",
        "      tf.keras.layers.Conv2D(filters=2, kernel_size=2, activation='elu'),\n",
        " \n",
        "      tf.keras.layers.Conv2D(filters=100, kernel_size=1, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=1, kernel_size=1, activation='elu'),\n",
        "\n",
        "      # a block to \"reproduce\" blurring\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation='elu'),\n",
        "      tf.keras.layers.Conv2D(filters=1, kernel_size=3, activation='elu'),\n",
        "    ]\n",
        ")\n",
        "model.build(input_shape=(None, None, None, 1))\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B85zCOvh4Jnr"
      },
      "source": [
        "Note: we have quite a lot of parameters and just a single image - we'll probably overfit heavily..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XKZqE80NFMcp"
      },
      "source": [
        "from tqdm import trange\n",
        "\n",
        "opt = tf.optimizers.Adam()\n",
        "\n",
        "loss_values = []\n",
        "for _ in trange(500):\n",
        "  with tf.GradientTape() as t:\n",
        "    prediction = model(img[None,...,None].astype('float32'))\n",
        "    loss = tf.reduce_mean((prediction - edges_blurred[None,...,None])**2)\n",
        "  grads = t.gradient(loss, model.trainable_variables)\n",
        "  opt.apply_gradients(zip(grads, model.trainable_variables))\n",
        "  loss_values.append(loss.numpy())\n",
        "\n",
        "plt.plot(loss_values);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p4Eg2NzB4aDi"
      },
      "source": [
        "Let's have a look on what the result of our model's transformation is:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jabJeGT-GwFm"
      },
      "source": [
        "plt.imshow(model(img[None,...,None].astype('float32')).numpy().squeeze());"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "heSFA7kB4obd"
      },
      "source": [
        "Try checking the following things:\n",
        " - Do the first layers indeed extract the edges?\n",
        " - What the intermediate representations of our model look like? (e.g. take the input and only apply a subset of layers from our model to it)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IJrOLZB75emu"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pJ14SRxNY0TS"
      },
      "source": [
        "## Deep CNN to classify images"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ydn6W26TZAGU"
      },
      "source": [
        "Let's work with the dataset from the previous seminar"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eYSbcV2mY3iU"
      },
      "source": [
        "import tensorflow_datasets as tfds\n",
        "\n",
        "data_train = tfds.load(name=\"fashion_mnist\", split=\"train\").prefetch(60000).cache()\n",
        "data_test  = tfds.load(name=\"fashion_mnist\", split=\"test\" ).prefetch(10000).cache()\n",
        "\n",
        "# Array for decoding the categories\n",
        "label_names = np.array(['T-shirt/top',\n",
        "                        'Trouser',\n",
        "                        'Pullover',\n",
        "                        'Dress',\n",
        "                        'Coat',\n",
        "                        'Sandal',\n",
        "                        'Shirt',\n",
        "                        'Sneaker',\n",
        "                        'Bag',\n",
        "                        'Ankle boot'])\n",
        "\n",
        "# Get a single data batch of 25 images\n",
        "sample_data = next(iter(data_train.batch(25)))\n",
        "sample_images = sample_data['image']\n",
        "sample_labels = sample_data['label']\n",
        "\n",
        "# Plot the images in a 5x5 grid\n",
        "plt.figure(figsize=(5, 5))\n",
        "plt.imshow(\n",
        "    sample_images.numpy().reshape(5, 5, 28, 28).transpose((0, 2, 1, 3)).reshape(140, 140),\n",
        "    cmap='gray'\n",
        ")\n",
        "# Print corresponding labels\n",
        "print(label_names[sample_labels.numpy().reshape(5, 5)])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PDXRTaPMaItA"
      },
      "source": [
        "Fill the gaps below to build a convolutional neural network and classify the images.\n",
        "\n",
        "Some hints for classes:\n",
        " - `tf.keras.layers.Conv2D` - convolutional layer\n",
        " - `tf.keras.layers.MaxPool2D` - maxpool layer\n",
        " - `tf.keras.layers.BatchNormalization` - batchnorm layer\n",
        " - `tf.keras.layers.Dropout` - dropout layer\n",
        " - `tf.keras.layers.Reshape` - reshaping layer (to convert the image-like representation to a vector-like representation deep down in the network\n",
        "\n",
        "Try to follow the general deep convolutional architecture:\n",
        " - combine convolutions with maxpoolings to reduce the spacial size of the representation\n",
        " - increase the number of filters as you go deeper\n",
        " - when the spacial size of your representation is small enough (1-2 pixels), convert (reshape) it to a vector and then use fully connected layers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZZHHccyZ0CTG"
      },
      "source": [
        "print(data_train)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ODDRCjUoZ27M"
      },
      "source": [
        "def build_model(use_batchnorm=False, dropout_rate=0.):\n",
        "  \"\"\"\n",
        "  Fill in the layers below.\n",
        "\n",
        "  If use_batchnorm is True, add a batchnorm layer to **every** convolution and\n",
        "  dense layer (except for the output one).\n",
        "  If dropout_rate > 0, add a dropout layer with `rate=dropout_rate` to **every**\n",
        "  convolution and dense layer (except for the output one).\n",
        "  \"\"\"\n",
        "  layers = []\n",
        "\n",
        "  layers.append(<YOUR CODE>)\n",
        "  if use_batchnorm: layers.append(<YOUR CODE>)\n",
        "  <YOUR CODE>\n",
        "\n",
        "  model = tf.keras.Sequential(layers)\n",
        "  model.compile(optimizer='adam',\n",
        "                loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "                metrics=[tf.metrics.SparseCategoricalAccuracy()])\n",
        "  return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_SS-xhvnaak9"
      },
      "source": [
        "configs = [\n",
        "  dict(use_batchnorm=False, dropout_rate=0),\n",
        "  dict(use_batchnorm=False, dropout_rate=0.01),\n",
        "  dict(use_batchnorm=False, dropout_rate=0.05),\n",
        "  dict(use_batchnorm=False, dropout_rate=0.5),\n",
        "  dict(use_batchnorm=True, dropout_rate=0),\n",
        "  dict(use_batchnorm=True, dropout_rate=0.01),\n",
        "  dict(use_batchnorm=True, dropout_rate=0.05),\n",
        "  dict(use_batchnorm=True, dropout_rate=0.5),\n",
        "]\n",
        "\n",
        "models = {str(config) : build_model(**config) for config in configs}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2qRMzQ1jaqer"
      },
      "source": [
        "batch_size = 512\n",
        "\n",
        "def preprocess(x):\n",
        "  return (tf.cast(x['image'], 'float32') / 255., x['label'])\n",
        "\n",
        "for config, model in models.items():\n",
        "  print(\"Working on model:\", config)\n",
        "  model.fit(x=data_train.map(preprocess).shuffle(60000).batch(batch_size), epochs=10,\n",
        "            validation_data=data_test.map(preprocess).batch(4096))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hIWeiejxavhc"
      },
      "source": [
        "plt.figure(figsize=(6, 8), dpi=100)\n",
        "color_cycle = iter(plt.rcParams['axes.prop_cycle'])\n",
        "\n",
        "colors = {}\n",
        "\n",
        "lines = []\n",
        "labels = []\n",
        "for config, model in models.items():\n",
        "  config = eval(config)\n",
        "  if config['dropout_rate'] not in colors:\n",
        "    colors[config['dropout_rate']] = next(color_cycle)\n",
        "\n",
        "  color = colors[config['dropout_rate']]['color']\n",
        "\n",
        "  style = '-' if config['use_batchnorm'] else '--'\n",
        "  line, = plt.plot(model.history.history['val_sparse_categorical_accuracy'], style,\n",
        "                   c=color)\n",
        "  \n",
        "  if config['use_batchnorm']:\n",
        "    lines.append(line)\n",
        "    labels.append(f\"do_rate = {config['dropout_rate']}\")\n",
        "\n",
        "plt.xlabel(\"epoch\")\n",
        "plt.ylabel(\"test accuracy\");\n",
        "\n",
        "from matplotlib.lines import Line2D\n",
        "\n",
        "lines += [Line2D([0], [0], linestyle='-', color='w'),\n",
        "          Line2D([0], [0], linestyle='-', color='k'),\n",
        "          Line2D([0], [0], linestyle='--', color='k')]\n",
        "labels += ['', 'batchnorm on', 'batchnorm off']\n",
        "plt.legend(lines, labels);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ozDHqzJF6Cqo"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}