SIT744_prac_4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "SIT744_prac_4.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZeHe2I9Ix2iM",
        "colab_type": "text"
      },
      "source": [
        "# SIT744 Practical 4: Second look at TensorFlow and Keras\n",
        "\n",
        "\n",
        "*Dr Wei Luo*\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l--dd-g0DOiD"
      },
      "source": [
        "<div class=\"alert alert-info\">\n",
        "We suggest that you run this notebook using Google Colab.\n",
        "</div>\n",
        "\n",
        "## Pre-practical readings\n",
        "\n",
        "- [TensorFlow tensors](https://www.tensorflow.org/guide/tensor)\n",
        "- [TensorFlow Variables](https://www.tensorflow.org/guide/variable)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6qqyuRZ3GmPj",
        "colab_type": "text"
      },
      "source": [
        "## Task 1. Low-level tensor manipulation in TensorFlow\n",
        "\n",
        "TensorFlow APIs cover three deep-learning components:\n",
        "\n",
        "- Tensors, including variables for keeping layer weights\n",
        "- Tensor operations, including tensor multiplication,  addition, and the activation functions\n",
        "- Backpropagation, implemented through the GradientTape object\n",
        "\n",
        "In this task, we will learn how to use these TensorFlow components. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TJVIrhwYGwCg",
        "colab_type": "text"
      },
      "source": [
        "### Task 1.1 Tensors and tensor operations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Za_ox3Xeit20",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "#### tf.Tensor vs tensors\n",
        "\n",
        "We mentioned that tensors (with lower-case t) are like NumPy arrays. However, [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) (with upper-case T) is different. A tf.Tensor defines a function (or computation) that, when called, produces a tensor. Such a function is called an **Operator** in the TensorFlow nomenclature.\n",
        "\n",
        "Therefore, a TensorFlow program forms a computation graph that chains a collection of tf.Tensor's together. You can think that the program  itself contains no data. When we invoke the program, data (tensors) are generated or passed in and flow through operators in the program. But how can we store the model parameters and train the model parameters?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3x8D8V-2ixES",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "#### Constant tensors and Variables\n",
        "\n",
        "Assuming that we separate data from computation. Then `tf.constant` and `tf.Variable` are the two main `tf.Tensor` that directly deal with data. Data in `tf.constant` is immutable and data in `tf.Variable` can be changed.\n",
        "\n",
        "Constant tensors return the same data at every invocation of the computation graph.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tq3b2gtaRFxe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "a = tf.constant([[1, 2],\n",
        "                 [3, 4]])\n",
        "print(a)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1sUVaVzpWy94",
        "colab_type": "text"
      },
      "source": [
        "As you can see that `tf.constant` returns a tensor. There are some other functions that return constant tensors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jkhiymqTWbAl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x = tf.ones(shape=(2, 2))\n",
        "print(x)\n",
        "x = tf.zeros(shape=(2, 2))\n",
        "print(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PqERjp2DYL1z",
        "colab_type": "text"
      },
      "source": [
        "A particularly important class of functions are those used for generating random initial weights in a network. They produced constant tensors as the initial values for variables.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q-MAXnakY_-5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## Normal initialiser\n",
        "w_init = tf.random_normal_initializer(\n",
        "    mean=0.0, stddev=0.05, seed=None\n",
        ")\n",
        "\n",
        "initial_value=w_init(shape=(2, 2),\n",
        "                     dtype='float32')\n",
        "\n",
        "print(initial_value)\n",
        "\n",
        "\n",
        "## Uniform initialiser\n",
        "w_init = tf.random_uniform_initializer(\n",
        "    minval = -0.05, maxval = 0.05, seed=None\n",
        ")\n",
        "\n",
        "initial_value=w_init(shape=(2, 2),\n",
        "                     dtype='float32')\n",
        "\n",
        "print(initial_value)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AyEx_1Lyf09n",
        "colab_type": "text"
      },
      "source": [
        "**exercise** Try to assign a new value to a constant tensor. Can you do that?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4daBVDTUgES4",
        "colab_type": "text"
      },
      "source": [
        "In comparison to constants, variables can be assigned a different value. They are required to represent trainable network/layer weights"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oO5-Vy5Cgk98",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "w = tf.Variable(initial_value=initial_value,\n",
        "                         trainable=True)\n",
        "print(w)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HnsodVqShn0H",
        "colab_type": "text"
      },
      "source": [
        "You can use the `assign` function to change the value in a variable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gILckVTthCgY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(w[0,0])\n",
        "\n",
        "\n",
        "w[0,0].assign(0)\n",
        "print(w[0,0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IOXmlZ0ai_uo",
        "colab_type": "text"
      },
      "source": [
        "### Task 1.2 Math operations in TensorFlow\n",
        "\n",
        "The transformation of tensors are achieved by matrix multiplications, additions, reshaping, and activation functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2st7tlXj3_K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "a = tf.ones((2, 2))\n",
        "print(f'a: {a}')\n",
        "b = tf.square(a)\n",
        "print(f'b: {b}')\n",
        "c = tf.sqrt(a)\n",
        "print(f'c: {c}')\n",
        "d = b + c\n",
        "print(f'd = b + c: {d}')\n",
        "e = tf.matmul(a, b)\n",
        "print(f'e = tf.matmul(a, b): {e}')\n",
        "e *= d\n",
        "print(f'e *= d: {e}')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EtjaWP2KlwIG",
        "colab_type": "text"
      },
      "source": [
        "### Task 1.3 Performing differentiation with tensor operations\n",
        "\n",
        "Tensor operations come with the ability to perform automatic differentiation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JztuHNjymED5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x = tf.Variable(initial_value= tf.ones((2, 2)))\n",
        "\n",
        "with tf.GradientTape() as tape:\n",
        "   y = tf.square(x)\n",
        "dy_dx = tape.gradient(y, x)\n",
        "print(dy_dx)\n",
        "\n",
        "with tf.GradientTape() as paper:\n",
        "   z = tf.sqrt(x)\n",
        "\n",
        "dz_dx = paper.gradient(z, [x])\n",
        "\n",
        "\n",
        "print(dz_dx)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l1vdti4sm_ET",
        "colab_type": "text"
      },
      "source": [
        "**exercise** Modify the code above to compute the gradient of the exponential function $y=e^{x}$.\n",
        "\n",
        "**question** Can you call `tape.gradient` twice?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NVBPzC3pCFeH",
        "colab_type": "text"
      },
      "source": [
        "## Task 2  Reimplementing a Keras model in TensorFlow\n",
        "\n",
        "Last week, we used a Keras model to run through the MNIST example. In this practical, we learn how to reimplement the model without using Keras. This will deepen your understanding of some key concepts.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VFn0BndZDORM",
        "colab_type": "text"
      },
      "source": [
        "### Task 2.1 A simple Dense class\n",
        "\n",
        "We know that a dense layer is essentially performing an affine transformation followed by an activation function.\n",
        "\n",
        "> output = activation(dot(W, input) + b)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZgkoyWBOY3Y5",
        "colab_type": "text"
      },
      "source": [
        "We can define a class for network layers.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_OiUXe6qDtpA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class NaiveDense:\n",
        "\n",
        "    def __init__(self, units, input_dim, activation):\n",
        "        self.activation = activation\n",
        "\n",
        "        W_init = tf.random_normal_initializer()\n",
        "        self.W = tf.Variable(initial_value=W_init(shape=(input_dim, units),\n",
        "                                              dtype='float32'),\n",
        "                         trainable=True)\n",
        "\n",
        "        b_init = tf.zeros_initializer()\n",
        "        self.b = tf.Variable(initial_value=b_init(shape=(units,),\n",
        "                                              dtype='float32'),\n",
        "                         trainable=True)\n",
        "        \n",
        "    def __call__(self, inputs):\n",
        "        return self.activation(tf.matmul(inputs, self.W) + self.b)\n",
        "\n",
        "    @property\n",
        "    def weights(self):\n",
        "        return [self.W, self.b]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CwVnV45HZNGA",
        "colab_type": "text"
      },
      "source": [
        "Let's try to pass a tensor through the layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EcZK6Ik91hYM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "relu_layer = NaiveDense(units=10, input_dim=2, activation = tf.nn.relu)\n",
        "\n",
        "x = tf.ones((2, 2))\n",
        "y = relu_layer(x)\n",
        "print(y)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mUc4le2VRI_r",
        "colab_type": "text"
      },
      "source": [
        "You can compare this with the original implementation from Keras. The difference in values is due to random initialisation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y-P3mu82Q-Fu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "keras_layer = tf.keras.layers.Dense(units=10, activation = tf.nn.relu)\n",
        "y = keras_layer(x)\n",
        "print(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UmTP7AnDMTx3",
        "colab_type": "text"
      },
      "source": [
        "**question** Do you see negative values in the output? Why? Are we using the 10 output units effectively?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vFIXRAjmedgo",
        "colab_type": "text"
      },
      "source": [
        "### Task 2.2 A simple Sequential class\n",
        "\n",
        "Once we have defined some layers, we can chain them together. Let's define a class similar to the Sequential Model in Keras.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pA_l3xFOZ2Ra",
        "colab": {}
      },
      "source": [
        "class NaiveSequential:\n",
        "\n",
        "    def __init__(self, layers):\n",
        "        self.layers = layers\n",
        "\n",
        "    def __call__(self, inputs):\n",
        "        x = inputs\n",
        "        for layer in self.layers:\n",
        "           x = layer(x)\n",
        "        return x\n",
        "\n",
        "    @property\n",
        "    def weights(self):\n",
        "       weights = []\n",
        "       for layer in self.layers:\n",
        "           weights += layer.weights\n",
        "       return weights"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WJBlvhAAZ3KY",
        "colab_type": "text"
      },
      "source": [
        "It takes a list of layers and returns a model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qJfzpqFSYEAa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = NaiveSequential([\n",
        "    NaiveDense(units=512, input_dim=28 * 28, activation=tf.nn.relu),\n",
        "    NaiveDense(units=10, input_dim=512,  activation=tf.nn.softmax)\n",
        "])\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gtYRMuhUazKf",
        "colab_type": "text"
      },
      "source": [
        "Let's try to feed the model two identical \"images\".\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oVcWreS4aPTu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model(tf.ones((2, 28 * 28)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "La67OKRHGtH0"
      },
      "source": [
        "### Task 2.3 A batch generator\n",
        "\n",
        "To run stochastic gradient-descent, we need to feed the model with mini-batches of the input data. Later on, we will learn how to build TensorFlow input pipelines with `tf.data`. Here we will create a simple iterator for retrieving training batches."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m-3G9JI3b0is",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class BatchGenerator:\n",
        "\n",
        "    def __init__(self, images, labels, batch_size=128):\n",
        "        self.index = 0\n",
        "        self.images = images\n",
        "        self.labels = labels\n",
        "        self.batch_size = batch_size\n",
        "\n",
        "    def next(self):\n",
        "        images = self.images[self.index : self.index + self.batch_size]\n",
        "        labels = self.labels[self.index : self.index + self.batch_size]\n",
        "        self.index += self.batch_size\n",
        "        return images, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZS9WrR5icG4G",
        "colab_type": "text"
      },
      "source": [
        "Let's try it on the MNIST data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pAKNUkujZwTG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.datasets import mnist\n",
        "\n",
        "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
        "\n",
        "train_images = train_images.reshape((60000, 28 * 28))\n",
        "train_images = train_images.astype('float32') / 255\n",
        "test_images = test_images.reshape((10000, 28 * 28))\n",
        "test_images = test_images.astype('float32') / 255\n",
        "\n",
        "batch_generator = BatchGenerator(train_images, train_labels)\n",
        "x, y = batch_generator.next()\n",
        "print(f'x: {x}')\n",
        "print(f'y: {y}')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jJil0L1MyzIy",
        "colab_type": "text"
      },
      "source": [
        "## Task 3 Training the model\n",
        "\n",
        "As mentioned in the lecture. training a neural network involves a loop with the following steps:\n",
        "\n",
        "1. Compute the predictions of the examples in the batch\n",
        "2. Compute the loss value for these predictions given the actual labels\n",
        "3. Compute the gradient of the loss with regard to the model’s weights\n",
        "4. Move the weights by a small amount in the direction opposite to the gradient\n",
        "\n",
        "These four steps comprise one **training step**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5hVfO79afq98",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learning_rate = 1e-3\n",
        "\n",
        "def update_weights(gradients, weights):\n",
        "    for g, w in zip(gradients, model.weights):\n",
        "        w.assign_sub(g * learning_rate)\n",
        "\n",
        "def one_training_step(model, images_batch, labels_batch):\n",
        "    with tf.GradientTape() as tape:\n",
        "      predictions = model(images_batch) ## 1. Compute the predictions of the examples in the batch \n",
        "      per_sample_losses = tf.keras.losses.sparse_categorical_crossentropy(\n",
        "          labels_batch, predictions) \n",
        "      average_loss = tf.reduce_mean(per_sample_losses) ## 2. Compute the loss value for these predictions given the actual labels\n",
        "    gradients = tape.gradient(average_loss, model.weights) ## 3. Compute the gradient of the loss with regard to the model’s weights\n",
        "    update_weights(gradients, model.weights) ## 4. Move the weights by a small amount in the direction opposite to the gradient\n",
        "    return average_loss"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "98d3KS-5hS9B",
        "colab_type": "text"
      },
      "source": [
        "Let's test-run it with a training batch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jBSJibnBf6Cw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "one_training_step(model, x, y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fkr_elRrhrbA",
        "colab_type": "text"
      },
      "source": [
        "Knowing that it is working, we can add a for-loop. Actually, we will use two nested for-loops: the outer for-loop to keep track of the number of epochs and the inner for-loop to iterate through the whole training data (multiple mini-batches). \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DFr4pexgh88Z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def fit(model, images, labels, epochs, batch_size=128):\n",
        "    for epoch_counter in range(epochs):\n",
        "      print('Epoch %d' % epoch_counter)\n",
        "      batch_generator = BatchGenerator(images, labels)\n",
        "      for batch_counter in range(len(images) // batch_size):\n",
        "          images_batch, labels_batch = batch_generator.next()\n",
        "          loss = one_training_step(model, images_batch, labels_batch)\n",
        "          if batch_counter % 100 == 0:\n",
        "              print('loss at batch %d: %.2f' % (batch_counter, loss))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tWxT0QKVbOEs",
        "colab_type": "text"
      },
      "source": [
        "Now we are ready to train the model. Specify 5 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KMZJ9kVMiBM5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "fit(model, train_images, train_labels, epochs=50, batch_size=2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ZH-m1ySjAAe",
        "colab_type": "text"
      },
      "source": [
        "**exercise** \n",
        "\n",
        "1. Write a program to evaluate the accuracy of the model and evaluate the accuracy on both training and test datasets.\n",
        "\n",
        "2. Modify the code above so that you collect the gradients at each layer and each epoch.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cbd7zbd1NW42",
        "colab_type": "text"
      },
      "source": [
        "## Additional resources\n",
        "\n",
        "- [Tensorflow, The Confusing Parts (1)](https://jacobbuckman.com/2018-06-25-tensorflow-the-confusing-parts-1/). Don't worry if some parts are still \"confusing\" after reading this."
      ]
    }
  ]
}