01_part3_Keras.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2020_01_part3_Keras.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MatchLab-Imperial/deep-learning-course/blob/master/2020_01_part3_Keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bb1Zl0tVSKCN",
        "colab_type": "text"
      },
      "source": [
        "# **Starting with Keras**\n",
        "\n",
        "Keras is a high-level Deep Learning API  written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. In other words, Keras is especially good for beginners, it is modular, minimalist and it is easy to get a neural network up and running in no time. \n",
        "\n",
        "Based on [this tutorial](https://elitedatascience.com/keras-tutorial-deep-learning-in-python). The following code is just a brief introduction to Keras, refer to [Keras documentation](https://keras.io/) to learn more about it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rF1rW5C1dS4R",
        "colab_type": "text"
      },
      "source": [
        "## Importing Libraries and Modules\n",
        "\n",
        "Google Colab already has everything you need installed. We highly recommend to use Colab for this tutorial, however, if you want to use your our environment, you will need to set up a Python environment and install Keras on it. \n",
        "\n",
        "You can find multiple guides that will help you with the task of installing Keras on your machine. Here are some examples for [Windows](http://inmachineswetrust.com/posts/deep-learning-setup/) or [Ubuntu](https://medium.com/@naomi.fridman/install-conda-tensorflow-gpu-and-keras-on-ubuntu-18-04-1b403e740e25)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wzb418Yih5SN",
        "colab_type": "text"
      },
      "source": [
        "We will import numpy and set a seed for the pseudorandom number generator. This seed is important for reproducibility, since different initializations might lead to varying results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VTilKf6uRhML",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "np.random.seed(123)  # for reproducibility"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ob8EYZ89h96k",
        "colab_type": "text"
      },
      "source": [
        "Now we are ready to start importing some of Keras' modules.\n",
        "\n",
        "First, we will import the `Sequential` module from Keras. A `Sequential` module is a stack of neural network layers, and it will be used when creating our first feed-forward network.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tE570PePip7f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.models import Sequential"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IC1aRifsj4Eu",
        "colab_type": "text"
      },
      "source": [
        "As mentioned before, Keras is a wrapper that allows us to use either Theano or TensorFlow as backend. It is easy to switch between them if required. However, newer versions of Keras will focus on the TensorFlow backend, dropping support for the rest of backends. In this tutorial, we will use the TensorFlow backend. Same as with the numpy pseudorandom number generator, we also fix Keras pseudorandom number generator for reproducibility."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "81_cpwH9UhRn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow import set_random_seed\n",
        "set_random_seed(123)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zCje2VIWFIV",
        "colab_type": "text"
      },
      "source": [
        "Next, we will import two of the core layers used to build a network on Keras: [`Dense`](https://keras.io/layers/core/#dense) and [`Activation`](https://keras.io/layers/core/#activation). In short, `Dense` defines a fully-connected network where all the neurons are connected to all of the elements of the input data and `Activation` defines a non-linear function applied to the network. We will explain more about the available layers in the next tutorials."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ua1JaDKTj5XO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import Dense, Activation"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vbm3UwLeKdDv",
        "colab_type": "text"
      },
      "source": [
        "Also, we import some utils, that will help us later with visualisation and transforming the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yUVkzIz3KtXt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.utils import np_utils\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZBQn3bnIkpOE",
        "colab_type": "text"
      },
      "source": [
        "## Loading datasets from Keras\n",
        "\n",
        "Besides the modules to build our neural networks, Keras library also contains [standard datasets](https://keras.io/datasets/). The fact that those datasets are inside Keras is handy and can save us a lot of time when learning how to create our first neural network. \n",
        "\n",
        "To show this, we import the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) dataset, which contains binary images of handwritten images, and load it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "92F5A9XEIiVQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.datasets import mnist\n",
        "\n",
        "# the data, shuffled and split between train and test sets\n",
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nuHAmcCaImS8",
        "colab_type": "text"
      },
      "source": [
        "Now we have on `X_train` and `X_test` variables the training/test partitions provided by Keras,  and on `y_train` and `y_test` their corresponding labels.\n",
        "\n",
        "We can now look into the shape of the data, and visualise some data examples:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l5o9qsfjJdNv",
        "colab_type": "code",
        "outputId": "95f30ccc-a932-4290-9baa-5c653cc08ed8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 322
        }
      },
      "source": [
        "print('Image shape: {0}'.format(X_train.shape[1:]))\n",
        "print('Total number of training samples: {0}'.format(X_train.shape[0]))\n",
        "print('Total number of test samples: {0}'.format(X_test.shape[0]))\n",
        "\n",
        "N=5\n",
        "start_val = 0 # pick an element for the code to plot the following N**2 values\n",
        "fig, axes = plt.subplots(N,N)\n",
        "items = list(range(0, 10))\n",
        "for row in range(N):\n",
        "  for col in range(N):\n",
        "    idx = start_val+row+N*col\n",
        "    axes[row,col].imshow(X_train[idx], cmap='gray')\n",
        "    fig.subplots_adjust(hspace=0.5)\n",
        "    y_target = int(y_train[idx])\n",
        "    target = str(items[y_target])\n",
        "    axes[row,col].set_title(target)\n",
        "    axes[row,col].set_xticks([])\n",
        "    axes[row,col].set_yticks([])"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Image shape: (28, 28)\n",
            "Total number of training samples: 60000\n",
            "Total number of test samples: 10000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUkAAAD7CAYAAADn/2ZiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAIABJREFUeJztnXlcFVX/xz+HXWRRlABlS3mAFJOU\nlNREE9ey9ElTHpfoV2qWWpZmlJqKpZZr5PJoaepjqLlCruWWgZq5JImICwIpgiCbiMDM/f7+uN7J\n68Ury7137nLer9d5wZ2Ze89nPnPmO2fOOXOGERE4HA6HUz1WcgvgcDgcY4YHSQ6Hw9ECD5IcDoej\nBR4kORwORws8SHI4HI4WeJDkcDgcLfAgyeFwOFrQe5BkjB1mjN1jjN25ny7qO09jhzFmzxj7jjGW\nyRgrZYydZYz1lVuX3DDG/scYy2GMlTDG0hljb8mtSU4eOGdUSWSMxcmty1hgjP3rfmz5nz7zMVRN\nchwROd1PQQbK05ixAZANIAKAK4CpADYzxvxl1GQMzAHgT0QuAF4GMJsx1l5mTbLxwDnjBMATQDmA\nH2WWZUwsBXBS35nw220ZIKIyIppBRNeISEFEPwHIAGCxAQEAiOg8EVWoPt5PLWWUZEy8CiAPwFG5\nhRgDjLGhAIoAHNB3XoYKknMYY/mMsSTGWDcD5WkyMMY8AAQCOC+3FrlhjC1jjN0FkAYgB8BumSUZ\nC68DWEf8OWIwxlwAzALwgSHyM0SQnAKgBYDmAFYCSGSM8drBfRhjtgA2AFhLRGly65EbInoHgDOA\n5wFsA1Ch/RvmD2PMD8qmmbVyazESYgF8R0R/GyIzvQdJIjpBRKVEVEFEawEkAein73xNAcaYFYD1\nACoBjJNZjtFARCIR/QbAG8BYufUYASMA/EZEGXILkRvGWCiASACLDJWnjaEyegACwGTI16hgjDEA\n3wHwANCPiKpklmSM2IC3SQLASABz5RZhJHQD4A8gS3kKwQmANWOsFRG100eGeq1JMsYaMcZ6M8Yc\nGGM2jLFhALoC2KvPfE2E5QCeAtCfiMrlFiM3jLEnGGNDGWNOjDFrxlhvAFEwQMO8McMY6wRlUxXv\n1VayEsoLZ+j9tALALgC99ZWhvmuStgBmAwgGIELZGD+AiNL1nK9Rc7+NaQyU7W03718RAWAMEW2Q\nTZi8EJS31iugvHhnAnifiBJkVSU/rwPYRkSlcgsxBojoLoC7qs+MsTsA7hHRLX3lyXhnGYfD4Twa\nPk6Sw+FwtMCDJIfD4WiBB0kOh8PRAg+SHA6HowUeJDkcDkcLtRoCxBiTtSuciIxuEDr3RBPuSfVw\nXzQxBU94TZLD4XC0wIMkh8PhaIEHSQ6Hw9GCHBNccDgatGrVCi+99BJGjx6NkydP4syZMwCAxYsX\nAwAqKyvllMcxAQIDAwEAK1aswLBhw5CTk6OT3zVIkLS2toarq6v0edy4cXB0dAQABAUF4d1338X8\n+fMRFRWFe/fuYe7cuZg5c6YhpBkFzs7OcHJywosvvggAcHd3x8KFC1FRYf5TKY4ZMwYAMH/+fDg5\nOQEAWrZsiaFDhwIATp5Uzs5/6NAheQRyDIazszMAwMnJCcXFxbh79+5jvqFOv37KGRi7du2Kt956\nC3PmzIEgCPUXRkQ1TvhnSv1HJl9fXwoICKCRI0fSyJEjaeXKlbR582YSRbHalJmZSVu2bCFRFKmk\npISOHj1K3bp1q/a3a6PVUKkmnjwq+fv70/Lly+ns2bMkCIJa+vrrr2v0G3Lvf309cXNzIzc3N8rN\nzaXqKCwspMLCQurVq1eNf1Pu/ddHWdFFknv/H+dJbGwsxcbGkiAINHHixFrvX5cuXahLly7SORQQ\nEKATT3RakwwNDcXBgwfVao2PQqFQAACmTp2KO3fuYMOGDcjJyUFhYSEuXjTvFyoGBwfj/fffx7Bh\nw9CgQQMwxpCdnY3SUuVEL0899RRee+01LFu2DGlp5j1Z+e3btwEAn332GRYsWABHR0dkZWXB19cX\nANCoUSMAQJ8+fbB//37ZdBoLH374Iezs7PDUU08BAIYNGwYASEtLQ+vWreWUplM+++wzXL16FTt3\n7qzxdzw9PfUjRpdXQjc3N7p06dIja43Jycm0e/duKi8vp+LiYiouLjarK6G25OrqSitWrKAVK1ZQ\nYWGhWq0xLS2N/Pz8KCAggAICAqTlXbp0MWtPHk5nz54lIqKUlBR6mBYtWph0OamPLxEREfTuu+/S\n5s2bqaqqSuOuQxAEqqyspNTUVJPz5UF9D9YkBUGg27dvU1hYWI08cnJyouTkZEpOTpa+HxMTY3w1\nydu3b2Py5Ml46aWXpIb3r7/+GgBw9uxZ9OzZE2VlZWjdujXee+89XWZt9AwcOBBvvaX5GukrV66g\nZ8+eyM7ORkBAgAzKjIfZs2fj008/RWhoqMY6Ozs7GRQZHi8vL8THx6NFixbSMldXVzRs2BCMMZw6\ndQrt2mlOwG1lZYWGDRsaUqrOuXbtmtpnFxcXzJw5E8OHD0dhYaHW7wYEBKBDhw76EaaPK6GLiwsx\nxogxRitXriRRFCkqKqpOV1HUMuoba+1g165dalf+y5cvU3x8PPn6+krb9O/fn/r372+xNUkA5Onp\nSefOnaOH2bJli0mXk5r4EhkZSRkZGdXWFAVBoKCgIGrSpAkFBQVR9+7dqXv37nTt2jVp/Z49e0zO\nlwf1WVtbk7W1NU2bNk1tv996663HHvNmzZpReno6paen67xNko+T5HA4HC3oZQhQSUmJ9H9xcTEA\nYNSoUdi0aZPUYWNpjBo1CqNHjwYA7N+/H5cvX0ZeXp7aNh4eHnJIMxqGDRuGtm3bIiQkRGPdb7/9\nJoMiw/LRRx/Bx8dHbVlFRQWmTJmC48ePSx2aBQUFUnOVt7c3AOWt6ogRIwwrWMeIoghA2UQ3bNgw\nqfnp3Xffxfbt21FQUPDI7z7xxBNqTRS6RO/jJGfMmIH27dsjIiICkZGRFttDeePGDcyYMUPrNs89\n95xhxBgRwcHBAIDt27cjICAANjbVF8mEBPN+1U2vXr0QHh4ufc7KygIAjBgxAklJSRrbq4Kjip07\ndyI/P1+/Ig1EcXExkpKSpCDZpk0b+Pj4aARJOzs7aZzt4MGD9aZH70GyrKwMo0aNwunTp7Fq1Soc\nOnQIf/zxB5YuXQoAqnYJi2TChAlqje1t2rSR/k9OTsaxY8fkkGVQVENZnnzyyUcGSACYOHEixo8f\nbyhZBufDDz+UHrBITk6WHqZ4OEA2btwYffr0QdeuXaVlycnJ2L17t+HEGoBjx47h9ddflz4/99xz\nOHv2LACgU6dO6NSpE5ycnDB16tRqv3/hwoXHdvbUGEM1yA8cOJCKioqk4UBTpkyhKVOmkJeXl0k3\nyNfGA0dHR3J0dKSwsDBKTEyUvCAitaFS2dnZ1LJlS4vwRJUmTJhA5eXl9CjMvePm1VdfpVOnTtHB\ngwfJ09PzkdvFxMSodWr8+eefWrc3dl+06V2/fv0jhxNWd948nN58802deKL3wv9gCgkJof3796vt\nyLJly6h58+ZmeZBVydbWljp06EDZ2dmUnZ1NgiBQaWkpZWdn0+bNm6mkpESt4Ofk5NDkyZPJzs7O\nbD2pLvXt25eioqIoKiqKRowYQcXFxaTC3INkTVL//v2pvLycBEGge/fu0b1792js2LEm7Ys2vaGh\noY/s6RcEgURR1Lp+1apVphckAVCjRo1oxIgRajv6888/m+VBBkB2dnb08ssvqx28adOmUefOnQlQ\nDsCv7rFEQRBoyJAhZG9vb3ae1CQxxmjmzJmk4vLly+Tn52ey5UQXvjwYFEaPHk2jR4+u1ffl3v/a\nevK4IHnx4kVKS0ujBQsW0PTp080nSKpSRUUFVVRUkCiKVFFR8cjntU35INva2tKcOXOkg5aYmEiJ\niYnUqFEjAkDu7u508uRJEkWRysvLaebMmTRz5kzaunWr9J29e/dS9+7dKTQ0VEqm7ElNk729PT3I\nhQsXyNvb22SDQX19+eKLL4jon9vLoKAgCgoKsqggmZeXR3l5efTHH39ojLuuLqDqKkjycZIcDoej\nDUPWEJ5++mmaNWsW7dmzR61d8syZM2RlZWVWV0Jra2uaO3cuCYJAxcXFNHbsWGrcuDE1btyYAFBY\nWBgdP35cena7e/fu0nddXFyoT58+tH79eiouLpaujBkZGZSRkWGyntQmffXVV/QgkyZNMukaU318\nsbOzk84ZQRDo3XffJSsrqxqdM8buiza9Li4utHr1ajp8+DAtX76cQkJCKCQkpEa1TkEQKDMzUzrf\n6uOJQQ5yUFAQxcXF0fXr1zV6oCorK2n37t1md5DHjh1LgiBQSUkJDR06lNzc3Khv377Ut29f2rx5\nM5WWlpIgCDR9+nTy8fF55D5HRUVJt+mqCTBM1RMA1KRJE0pISND6mKqXl5dapw2R5U5w4ejoSKNH\nj5ba79evXy8119Q2yb3/uvKkpkFSEATy8PAw3iDp6elJnp6eNHHiRLpy5YpGcDxx4gSdOHGCXn75\nZbM8yDk5OSQIApWVldGpU6coLS1N4wBOnTqVrK2t61U45N7/2paTDRs2EBFRWloaRUREqAX+9u3b\nU1RUFJ05c4ZUzJ8/n+bPn08ODg4m7Ultzx9nZ2dydnamjRs3SuVlwoQJta49GrsvugqS/v7+aqNH\nVOm7774jGxsb4wqSHh4e1L17dzp//jydP3++2unSBg4caPa3C2fOnNEIigkJCZSQkECTJk2igIAA\nrQfPHD0BQOHh4ZScnEwqVE0Iu3btopKSEmm5QqGg1NRUatiwITVs2NDkPXmcLw+n4OBgCg4OVuvJ\ntbSyUtvUsWNH6tixI12/fl3tvNNWfmqkUVc75ObmRj/++OMj55M8evQoDRgwgBo0aGARB9nZ2ZlG\njBhBixYtopiYGPLw8CA7O7sajX00V09UacGCBTR27FjSRkFBgVl5UhNfHgyQq1atolWrVpEgCJSa\nmlrj4U+m5osuzwVVCgsLo5s3b0pBMiIiQt4g2bFjR9qyZQtlZWVVGxxLS0vp888/r3VtwJIPsiV4\nYm9vT5MnT5bSDz/8QCqKioqoXbt2ZuVJbcrKhg0b1GpCtRkwbmq+6Ou8iIqKotOnT1NsbGy9a5Ls\nvtAawRjT2Hju3LmYPHmy9Dk1NRU//fST9AKeBQsWoKioqMZ5aIOImE5+SIdU54kh4Z5oYoyeADXz\npXXr1pg7dy769u0LAFi5ciWWLFmik1eaGKMvplBW+DhJDofD0UK9a5KGhF8JNeGeaGKMngA182Xe\nvHn48MMPkZmZCUD5mlRdvRjPGH0xhbLCg2Q94Z5owj2pnpr40qNHD+zbtw+vvvoqANTqbYGPwxh9\nMYWywoNkPeGeaMI9qR7uiyam4EltJ93NB5BZNzn1xk+mfB8H90QT7kn1cF80MXpPalWTrAuMMTcA\n3wHoBaUhMUT0g14zNXIYY+MARANoAyCeiKJlFWQkMMYOAwgHINxfdJ2IguRTJC+MMXsAywBEAnAD\ncAXK82ePrMJkxtAxRe+vbwCwFEAlAA8AoQB2Mcb+JKLzBsjbWLkBYDaA3gAayKzF2BhHRN/KLcJI\nsAGQDSACQBaAfgA2M8baENE1OYXJjEFjil6HADHGGgJ4FcA0IrpDRL8BSABg2q91qydEtI2IdgB4\n9OvfOBYPEZUR0QwiukZECiL6CUAGgPZya5MLOWKKvsdJBgIQiCj9gWV/Amit53w5psscxlg+YyyJ\nMdZNbjHGBGPMA8pzypLvwgweU/QdJJ0AlDy0rBiAs57z5ZgmUwC0ANAcwEoAiYyxlvJKMg4YY7YA\nNgBYS0RpcuuREYPHFH0HyTsAXB5a5gKgVM/5ckwQIjpBRKVEVEFEawEkQdkOZ9EwxqwArIeyHW6c\nzHLkxuAxRd9BMh2ADWPsXw8sawvLvl3g1BwCYHRj+wwJY4xB2ZPrAeBVIqqSWZLcGDym6DVIElEZ\ngG0AZjHGGjLGOgN4BcqrosXCGLNhjDkAsAZgzRhzYIwZYqSB0cIYa8QY663ygjE2DEBXAHvl1iYz\nywE8BaA/EZXLLUZu5Igphpjg4h0oh7nkAYgHMNbCh/8AwFQA5QA+BjD8/v9TZVUkP7ZQDou6BeXY\nt/EABjzUQG9RMMb8AIyBcpjLTcbYnftpmMzS5MagMUXvg8k5HA7HlOFTpXE4HI4WeJDkcDgcLfAg\nyeFwOFrgQZLD4XC0UKthJ6Yw95uh4Z5owj2pHu6LJqbgCa9JcjgcjhZ4kORwOBwt8CDJ4XA4WuBB\nksPhcLRg8CDZvn17rFmzBqIoQhRFrFmzBu3atTO0DNlZsmQJiAgpKSnw8zPW149wOKZDt27d0K1b\nN4iiCCJCRESEbn6YiGqcoJyVpc4pNDSUbt++TYIgqKWCgoIafb82Wg2V6uKDv78/FRQUkCiKJAgC\n9e7du86eyr3/+ign9U1y778ufRkxYgStX7+eUlJSqLKykiorK+nXX38lV1dXcnV1NXlfdHXMo6Oj\n6dSpU3Tq1CmqqqoiURTp1KlT9N5775GNjU29PDHYDnXo0IGys7OlwFBYWEiFhYWUm5tLgiBQeHg4\n2dnZWcRBbtiwIW3fvp0HST0lufe/vr40bdqUduzYQTt27CBRFKmgoIASExMpMTGRSktLSaFQUGpq\nKqWmppq8L7o43tHR0XTgwAGqqqqSkiiK0v9+fn7GHSQdHR2pS5cudO3aNRIEQQoMv//+O/3+++80\naNAgaXlMTIzFHOTFixfzIFmH5OfnR8HBwTRz5kzKy8uT0po1a4zak9r48scff1B+fj7l5+fTnDlz\nyM3NTVoXHBxMJSUl0l3Y9OnTLbKsNGrUiLp160YZGRlUWlqqFiBTUlLo/PnzphMk169fr3ZrrQoM\nqqS6CoiiSPHx8RZxkBs1akSHDh3iQbIWKTIykpYtW0aFhYWkUChIFEW1dOHCBaP2pKa+9OzZUzoX\nHnU+zJo1ixQKBSkUCsrIyLC4sjJgwADatWuXRo1RlUaOHEnR0dE6C5J6m+i1fXvlC91efPFFKCdX\nBo4cOYLExETMnz8fN27cAACcOXMGhYWFeOGFF6TtzB1HR0f4+vpKn5999lmkpaUhM1Oud7QbJ99+\n+y3atGkDQOmRitLSUmzYsAEnT55EfHw8AODevXuyaNQ1NjY2uHz5MjZu3PjIbbZs2YKpU5XTjzo4\nOMDFxQUlJQ+/9sX8GD58OABg7dq10jIrK82+Z1UcqW5dndBH1Fd10DzYSZOYmEhOTk704osvUkxM\nDLm7u5O7u7v0HVEUqbS0lNq1a2f2V0IANG3aNLVa9bhx43hN8n5q0qQJrVy5khQKhXTbefLkSXr1\n1VcpKCiIfH19Tc6Tmvri4OBAjo6OWrcJCgqSapIKhYLefvttsy8rw4cPl8pCVVUVlZaW0uXLlyk3\nN1eqMZaWllJpaSkNHDhQpzVJPk6Sw+FwtKHrqB8YGEgbNmyQ2opyc3Pp7NmzNGjQIK3fU9WqNmzY\nYJZXQm37zGuS6knVqbVkyRJycnIiJycnk/dEF76okq2tLaWkpFBKSgopFApasGCByfpSE90DBgzQ\naHv85ZdfCIBajXH8+PE0fvx4jeVG1SZpb2+P+fPno1+/figtVb7hceTIkfjjjz/QoEGDGv3Gg211\n5o6VlRUUCoXcMowCR0dHTJkyBSNGjMD777+PQ4cOYd++fWbT1qhLqqqqIAiC3DIMQnR0NBYvXgzg\nn3bnEydOYMKECdI2f/75J9auXYvly5dLy7Zs2YJRo0ahQ4cO9dag0yD5zDPPoF8/5WuSX3nlFQDK\nzhpO9SgUCtXV1OKZOnUqpkyZgs2bN2P//v08OGrB3t4eDg4O0mdVhcQcmTZtGho2bAgA+OKLLwAA\nc+bMkdb/9ttv2LNnD3Jzc9W+d+fOHVRUVOhEg06D5MKFC8EYw5EjR2odHHmtyrKJiYkBESE+Pp4H\nyMfg7++PoKAg6fPevf+8dbdp06Zo27YtnnvuOQDAjz/+iIsXLxpcoy4IDQ2Fs7MzrKysYG1tXe02\nly9ffuT3GWM66eHWWZB86aWXEBoaCiJCQkJCrb+vqlWdPXtWV5I4JsTvv/+OsLAwfPPNNygvL8fP\nP/8stySjw97eHt7e3ujUqZPa8hUrVuDUqVNo164d3Nzc4OPjI9UuAwICEB0dLYPa+hESEoKtW7ei\ncePGdao8OTk5wc7OTjcVL101sg4ePJgEQaAbN26Ql5dXjRuh7e3tac6cOSSKIu3fv19rI73cjcy6\nbownIqmDa/PmzRbVcdOxY0e1x1Dd3NxoxowZJIoiFRcXU3BwcH18ld2DuvrSoEED8vPzo4EDB9LA\ngQNp3rx5dPLkSTp58qTUUfNwqqqqooyMDMrIyKDPPvuMwsLCyN/fn/z9/Y3el0f5cPDgQbWOmtqW\nAV123OjsIKuCZE2fALC3tyd7e3uKjY0lQRAoMzPzsU+eyH1A61rwH5UefvqoVatW1KpVK5MPCNr0\nenl50alTp+jWrVs0fPhwtXVNmzaVLhqdOnWyqCDZoEEDmjdvHqWmplYbCIuKiig7O5sqKyvVlq9c\nuVLr2GJj96UmQXLixIm1Ov7BwcGUl5dHVVVVdOnSJfLw8KiXJ3ycJIfD4WhB548l1qQ9MjQ0FJMn\nTwYADBkyBDt37sSrr76qaylGz4oVKzBmzBjp8+jRowEA77//vlyS9M7p06fh4uKCKVOm4H//+5/a\nuvfeew8A8Msvv+Cvv/6SQ55s7NixAz179kRFRQV27dqFjIwMAMDOnTtRUVGBa9eu4e+//0ZaWhoC\nAwNx9epVAMAHH3yAO3fuyCld7xQUFNRou+DgYABKz5o0aYK8vDwMGjRIo+e71ujqduG1114jURQp\nMzNTa1V44sSJdPv2bem2at26dSZ9G1VT7dWl8ePHq91uL168mBYvXmzyt5ba9MbExFBZWZnG7eTF\nixelCRtqevtoSp48zhcioqtXr1JoaGi1621sbGjevHlUWlpKN2/epHbt2tXaJ7n3vzaeqCaAUSVt\n++Xk5ETbtm1T2/7SpUsUFBSkE090dpBVbZIVFRX09ddfU2hoKIWGhpKPjw8NHjyYEhISKDMzk0RR\npIyMDGmWk/DwcLM8yDVN6enp0oFV0bJlS7P2ZNKkSbRhwwbKzc2VUl5eHiUmJlJQUBBZW1tbXJBU\nKBR0+vTpaieIdXBwoMTERFIoFFReXk7dunUzG18epbVHjx7Sc9pVVVV06NAhOnjwIB08eJDee+89\n6tOnDx08eJAOHTpESUlJas9ux8TE1ChA1tQTdl9ojdD2jtzBgwdLM7IAkKq4JSUl+Ne//iUtP3bs\nGA4dOoTp06fXOF8VZIbvDd6+fTv69++v+i0AQGBgIK5cuVKj75ujJ/XFGD0BtPuiuo3+/vvv0aRJ\nE/z5558AgKtXr2Ly5MkICgrCyZMnMXbs2DoPkzNGX7R5EhERga1bt8LV1fWR46hVy48cOYJ169YB\ngPS3JtTIE11dCb29venYsWNq80Y+2Hubm5tLS5YsMbsaQn32BwD17dtXwzNzr0nqO8m9/3X1JTY2\nlsrLy6miokIt/fjjj9SnTx+z9OVxmps3b04zZ86sdt7Iqqoqun79Om3cuLHWr7KojSc63SEvLy+a\nMWOGRpBcsGABBQQEWORBflzy8/OjlJQUHiR5kLRIX2qq/fXXX5cm9VDNPj5y5Ejq3Lmz3j3R2e22\nISATu10wBNwTTYzRE4D7Uh2m4AkfJ8nhcDha4EGSw+FwtMCDJIfD4WiBB0kOh8PRQm0fS8wHUNdX\n+tkDaA2gEEBGHb7vV8d89U1dPXkSgDMAawBVAG7e/63awD3RxFg9Aermix0AXwBOUPbIFgLIqkPe\nxupLXTxxgNITRwACgL8BFNUh7xp5Uqve7frAGNsPoAGATCIabpBMjRjGWGsAl4mogjEWDOAwgBeJ\n6JS8yuSDe6IJY2w3gDwAbwNoBOBnAKuI6GtZhckEY8wGQCqAFQCWAIgAkAjgGSJK10eeBrndZowN\nhTLSHzBEfqYAEZ0nItX88qpxWy1llCQ73JNqeRLAZiK6R0Q3AeyF8o7MUgkG0AzAIiISiegggCQA\nI/SVod6DJGPMBcAsAB/oOy9TgzG2jDF2F0AagBwAu2WWJDvcEw0WAxjKGHNkjDUH0BfKQMn5BwYg\nRF8/boiaZCyA74jobwPkZVIQ0TtQtsE9D2AbAN28uciE4Z5o8CuUNccSKNve/gCwQ1ZF8nIRyuaH\nyYwxW8ZYLyhvuR31laFegyRjLBRAJIBF+szHlLl/y/AbAG8AY+XWYwxwT5QwxqygrDVuA9AQQFMA\njQHMk1OXnBBRFYABAF6EsmPvQwCbobyA6AWdT7r7EN0A+APIuj/DjRMAa8ZYKyJqp+e8TQ0b8Pa3\nh7F0T9yg7MX95n5bbQVjbA2A2QA+klWZjBDROShrjwAAxlgygLX6yk/ft9sroSzkoffTCgC7APTW\nc75GDWPsCcbYUMaYE2PMmjHWG0AULLhji3uiCRHlQzlcbixjzIYx1gjA6wDOyatMXhhjTzPGHO63\n004C4AXge33lp9cgSUR3ieimKgG4A+AeEd3SZ74mAEF5G/k3lOPe5gN4n4hq/y5e84F7Uj3/BtAH\nwC0Al6EcPzpRVkXyMwLKTr08AD0A9HxgVITOMdg4SQ6HwzFF+GOJHA6HowUeJDkcDkcLPEhyOByO\nFniQ5HA4HC3wIMnhcDhaqNVgclN4H4Wh4Z5owj2pHu6LJqbgCa9JcjgcjhZ4kORwOBwt8CDJ4XA4\nWuBBksPhcLSg71mAakyPHj2wYcMGRERE4OLFi3LL0SuBgYGwtbVF165dsWzZMigUimq327lzJ4YO\nHYrKykoDK9QdTk5OAIAhQ4bg3r17aN++PZydnTFs2DAcPnwY169f1/jOzZs3sXPnTvzxxx+GlmuS\nNGzYEIcPH0azZs3QuXNnXLt2TW5JZkW9g2TXrl3RpEkTbN++vV6/8+yzz+LkyZP1lWO0tG7dGtHR\n0QCAwYMHw8rKCs2aNYNCocCjnp9/+eWXsWLFCrz//vsoKSkxoFrdMX36dADApEmTNNb16dPnkd+L\niYlBamoq4uPjAQDx8fH85Ad3YZFyAAAgAElEQVTQrFkzuLu7S58LCwvRvXt3tG/fHhcvXkRBQYGM\n6uTBz88PEydOxDvvvAMAsLGxwcaNG/Gf//xHJ79f7yDZrVs3/Otf/6pXkLSyssKTTz4JPz8/3J93\n0uyYM2cO+vXrV+vvjRw5Et999x2SkpL0oEr//Pvf/9ZYVlBQgHPnNGf7unjxIoKCgtCoUSM888wz\nCAkJweeffw4AOHfunEUGyZCQEEyYMAF+fsoX+wUGBsLX11daP3fuXLRq1QqMMVy/fh12dnZySZWF\nN954A4sXL8alS5cwZswYAICPjw8+++wzzJo1C2lpafXPhIhqnPDPy5mkdPnyZVq/fr3G8tqk5s2b\nk0KhoHXr1mndrjZaDZVquo/jx48nQRCkdOPGDYqNjaXPP/+cYmNjKTY2lg4cOEAHDhxQ204QBOrc\nubPJetKyZUtq2bIl9e3bV/rfy8tLq1fOzs507do1UigUUvrvf/9b4/Ik9/7Xt6w8mCZMmECiKErp\n7t27tHbtWsrOzqbs7GxpuUKhoOHDh5ucL3WNGXZ2dhQTE0Pl5eX0+eefU6NGjaR17dq1I1EUqXnz\n5jopK/XeoatXr9Y7SO7Zs4cUCgVNmzbNbA+yjY0N+fj4SMnT01NjGxcXF3JxcaGsrCwpQG7ZsoXs\n7e3N0pNHpaioKCk4lpeXU3l5OYWFhVlUkJwxYwbNmDGDysrKSBRFWr16Nc2bN4/c3d0JAIWGhlJo\naCjl5uaSKIqUm5tLDg4OJudLXcvImDFjSBRFGj9+vMa6jz/+mHJycnRWVup8u/30008DADw8POr6\nExKurq4AgJ9//rnev2WsCIKA7Oxsrdv07q2csL1x48bSsr///hsVFZbxLiw7Ozt8/fXXGDlypLTs\nueeeAwCcPXtWLlmy0LBhQwBAgwYNkJmZiU8//RQ5OTkAgICAAHzyyScAAHd3d5SVlWHGjBm4d++e\nbHoNiZubG2JjY7FlyxYsX75cWq5qknjrrbd0mh8fAsThcDjaqGvV+OOPP6aPP/6YFApFvW63PTw8\nKCcnhxQKBfn4+FjM7cLDaejQodW2Sbq4uFiEJ927d6fVq1dLt9kVFRU0ZswYcnBweOxtpCl4Ultf\nOnbsSB07dqSUlBQSRZHWrl1LDRs2pGbNmlFCQoLUFpmfn08TJ040WV9q44mNjQ3Z2NjQhQsXKCUl\nhdzc3NTWHz16lI4ePUoKhYK++uor+W+3g4KCpP/Pnz9f15/B/Pnz4eHhgfT0dJSWltb5d0yVYcOG\n4eOPP0ZAQABsbW3V1p09exZVVVUyKTMMHTp0AADs378f1tbW0nIiQlZWFkRRlEuarKiaF44fP45W\nrVrhhRdeQM+ePbFo0SK13u2ZM2ciLi5OLpkGZdCgQQCUPfwvvPACbt++La2LiopCeHg4AODOnTuY\nP3++zvLVyWDy2oxvdHFxQZ8+fTB8+HAAQK9evQAAsbGxKCoq0oUco8Tf3x8jRowAAERGRkrLu3Tp\norqiSpSUlODjjz/G7t27UV5eblCdhua1114DALUACSjbJ3ft2iUNKE9MTMT27dvx119/GVyjHKja\noVXjY5s1a4atW7eCMQYiwnfffQcA2LFjh2waDc3rr78OQDlULDk5WVru6emJxYsXw8pK2XoYFxeH\n3NxcneWrkyDp5uam9rlt27bSeMfIyEh4e3vDzs4Ow4YNg5WVFcrLy3HixAkAysJgY2ODU6dO6UKK\nURISEoKEhAS1GoA2jh49ipUrV+pZlXGwbds2AMBTTz2FZ599Fk2bNlVbHxYWJv397LPPsHjxYnz5\n5ZfIy8szuFY5yMzM1Fi2e/duqab0uM5Ac0LVsTl9+nTpDsvFxQVbt25F06ZNsWLFCgDAvHnzdJtx\nXdsPli1bRsuWLSNRFKmgoIDOnDkjJdW4LYVCQZWVlVRUVETJycm0aNEiGjZsGHl7e5OtrS3Z2tpS\nbm4uVVZWmnWbSkhICGVkZKiNd1MlIqp2ed++fc3ak+qSr68vtWvXjvr27UurVq1SK0cPpkOHDpGV\nlZVJeVJbX6ytrcna2pp+/PFHtX1PTEysk7fG6ktNtffo0UPyICQkhABQ7969pfG06enp5OrqSq6u\nrjr3pM41SdUjQJmZmejUqZPauqysLOk24MKFCzh+/LjG90ePHg1AOYTh6tWrdZVhEvz111/o1q2b\n1MSwb98+jeEab775JsaPHy+HPKMhKysLWVlZAIA9e/bg8OHDkieqtksAiIiIwKRJk/Dll1/KotMQ\nbNy4EYDyiaX7wQQA1P63JHJzc6VzZvPmzXB2doa7uzsqKirAGMPSpUtRXFysn8wNUUOoLm3atIk2\nbdpECoWC5s2bZ/ZXwsclV1dXtV5tS6xJVpdUPZqHDx9Wq1GtXLnSpDypqS/NmjWjTz/9VNpPURTp\n5MmT9O2335IoinT8+HGLrEkCoJEjR9LIkSPpyJEjFB8fTy+//DJlZWXRgQMHaj0Cojae8HGSHA6H\nowWjCJL1nUHIHFA1SnPUEQQBgiBodOylp6fLpEi/9OjRA7NmzZI+T506FS+88AKOHDkCAEhNTZVL\nmuysW7cO69atQ0REBP7zn/+gR48e8PDwwJQpU/T6tJHRzCdpjtja2qJXr144ePCg1qE8b7zxBpYs\nWWJAZfLj5eWFUaNGSbO0bN68udrtVEOD2rZtKy0TBKHadm5Tp1u3bvj6668BKKfJA4BffvkFnp6e\n0pRzljgTUnVERERg3Lhx+Pzzz/U+76jsQZIxhsDAQLMr9F26dMGnn36Knj174sknn6x2qIabmxv6\n9euHhQsXwtHRUVpeXl5u1s/henp6Yu/evWjTpo3ac+oP4+HhgQ8++AAA8MILL0jLL1y4gN9++03v\nOg1Nz5494erqiiNHjuCnn34CoLzQvvTSS3B1dQVjDLdu3ZJZpXHwww8/4MaNGwbpvJM9SBKRNAjU\nnPjmm28QEhICAPjoo4+qfZqoZ8+eaNeunaoBG4cPHwYALF++HIcOHTKYVkOzePFitGnTBgDw5JNP\nAlAOEFbVths0aICPPvoIH3zwAZydnaXvMcZQWlqKCRMmGF60AVBNwExE0tNXAwYMwJIlS1BYWIhv\nv/1WbUIHSyUsLAxNmzbFhAkTcOfOHb3nJ3uQBJQzvXz//fdyy9AbY8eO1bo+Ly8PiYmJeO+99wDA\nrGuRAHDgwAHpSZvTp08DAM6cOSMN4XB1dcUzzzyj8b3S0lIMHDhQap8zN5544gkAwK1bt6QZsZ5/\n/nkAyiaZxMRE2bQZAw4ODgCAlStX4vr161i/fr1B8pU9SJrrTOTR0dEYP3689CjVw1y5cgV3796V\nnq6xlMftAOWUeBs3bsTQoUOlZdUFRUDZ/ggoa59bt26VntQyRy5cuABA+Yyy6ry4ffs2li5dil9+\n+UVOaUbBG2+8AUDZPt22bVuUlZUZJF+mutWr0caM1Xzjx6B638vq1auxatUqaep1bRCR0UVUbZ7Y\n29sjOjoas2fPRuPGjaUB9j///DN27tyJmzdv1jt/U/NEhb29PQYOHCi1Naanp0udFarOnIMHD0r/\n12Y+SWP0BHi8L40bN8aoUaMwbdo0qTMiISEBixYt0kn+xuhLbWKKqme/oqICzz77rHQBrQ818cT8\nGgM5HA5Hh8hWk6wLpn4l1AfcE02M0ROA+1IdtfFEdec1c+ZMnXVg1cQTHiTrCfdEE+5J9XBfNDEF\nT/jtNofD4Wihtr3b+QA0J7gzDH4y5fs4uCeacE+qh/uiifF7YoBZPsYB+ANABYDv5Z51xBgS90Sr\nN0MBXABQBuAKgOfl1iR34p5o+GHQ88cQ4yRvAJgNoDeABgbIzxTgnlQDY6wngHkAhgD4HYCXvIrk\nh3tSLQY9f/QeJIloGwAwxsIAeOs7P1OAe/JIZgKYRUSqB/mvyynGSOCePIShzx/eccMxChhj1gDC\nALgzxi4zxv5mjH3DGLPYmjb3xDjgQZJjLHgAsAUwCMDzAEIBPANgqpyiZIZ7YgTwIMkxFlQTbsYR\nUQ4R5QNYCKCfjJrkhntiBPAgyTEKiKgQwN9QvntEWiyTHKOAe2Ic6D1IMsZsGGMOAKwBWDPGHBhj\nss8+JCfck0eyBsB4xtgTjLHGACYC+ElmTXLDPXkIg58/BhjTNAOabymbIfdYK5nHeXFPqvfFFsAy\nAEUAbgL4GoCD3Lq4J8aVDH3+1OrZbQ6Hw7E0eJskh8PhaIEHSQ6Hw9ECD5IcDoejBR4kORwORws8\nSHI4HI4WajW2yBRmETY03BNNuCfVw33RxBQ84TVJjtESGBiIq1ev4urVq8jMlGteVo6lw4MkxyiJ\ni4vDb7/9Bl9fX/j6+uLMmTNySzIaWrRogU2bNmHTpk2orKxEcHCw3JLMGh4kORwORwv8eWGOUeHh\n4YFt27YhPDwcRIS//voLAPDmm2/KrMw46NSpE/bu3Ytbt24BAJYuXYrc3FyZVZk5tXxm8uHnJeud\npk6dSqIoEhFRRESE1m3lfmZUl57Y2tpSREQEJSUl1cs/ufdfl54EBgZSQkICCYJAoijS5MmTqXv3\n7tS9e3eT90QX58+LL75I5eXltGjRInJ0dCRHR0eT90UfMSUmJoYUCgXNnTtXJ57ItkPR0dEUHR1N\npaWlVFVVRaIoUteuXS3mIDdt2pQUCgXduHGDPD096+yj3PuvS0/Cw8NJEAQpSEZFRZmNJ/U9fwIC\nAqisrIz27t1LVlZWZuOLLmOKs7MzOTs7040bN0gURbp37x69+eab9fZEttttPz8/AICDg4NcEowC\nT09PeHp64ubNm3JLkZXAwED88MMPYEw5IuPf//43du7cKbMq48DBwQHffvstUlJS8Nprr0GhUMgt\nyeiwsbHB2LFjASibbAAgNzcXx44dq/+PyxH1IyMjKT8/n/Lz86mqqopSUlLIz8+PHBwcLOZKqKpJ\nEhGFhobW2Uu5919XnsTGxpIgCJSYmEjNmzevV/mSe/91XVa++uorKi8vJ29vb7PzRVcxpUuXLiSK\nolrq27evTjwx+A516dKFsrOzqaqqSkojR460uIOsCpIKhYLCw8MttuAnJydTcnIy3b17ly5fvkwB\nAQH1LmNy778uy4q9vT3l5OTQnj17zNIXXcQUf39/On36tFqA3L9/PzVo0EAnnhh8h1atWqW2MwcO\nHLDIg/xgkBw3bpxFFvxXXnlFKgeCINCcOXPqXVsyVk/qWlamTZtGpaWl1K5dO7P0RRcxJSUlRS2m\nFBYWUs+ePXXmCR8nyeFwOFowaMdN06ZN8X//939QKBQoKioCAMyePduQEowGQRBQXFwMV1dXtGzZ\nUm45BqdRo0Z4/vnn1ZYVFhbi77//Vlv23nvvAQB8fHwAAJMmTTKMQCOhV69eSEpKwunTp+WWYrS0\nbt1aVSsFACxbtgw///yzzn7fYEHS398fW7dulT7HxcUBAA4dOmQoCUZFUVERjh49ipdeekluKbIg\niiLat28PKyvlzYxCocCvv/4qrZ84cSIAYPz48QD+GQ3x4YcfwtvbG9evXzewYsPSpUsXAEB4eDja\ntGmjtq5bt27SYPLz588bXJsxsXDhQjDGQEQ4cOAAACA2Nla3mRiq/eDtt9+WxkPu3buXXF1dydXV\n1aLbVBISEkihUNCiRYssrp3ppZdeksZDiqJIGRkZUqdNaGgobdu2TRozKQgClZSUUGpqKomiSMeO\nHSM/Pz+T8qS2ZWXFihW0YsUKOnfuHNnb2xOgHFtcUFBACoWCysvLqby8nN59912zLyuPSkuXLqXr\n16+TQqGgM2fOkLu7O7m7u+v8/DHIDg0YMICKioqoqqqKjhw5Qh4eHhYVEB4XJNetW2dRQdLZ2ZnG\njx9PgiBQdnY2ZWdn06xZswhQPnWzYcMGEkWRcnNzaf369bR+/XoKDQ2liIgIEkWRUlNTzT5IVlZW\nUmVlJQ0aNIgAkJ2dHWVkZNCgQYPI2dmZhgwZQkOGDKG7d+9Snz59zLasPCp16NCBrl+/TqIokkKh\noLFjx+rt/NH7Dvn7+6v1PK1Zs8aiAkJNgmRRUZFFedK3b1+phjh9+nSaPn06ASAPDw/pscSioiKK\ni4sje3t7sre3p5CQELpw4YK03NQ8qU1Zad26tTTyYcCAAQQon0Zavny5xrZxcXF05MgRsy0rj0or\nVqyQPDp//jw1a9bMdIPk8uXL1cZEBgUFWVRA0JYmTpxokUFyypQpUpB8cHlSUpK0XPUcf3h4uNrj\nivPnzzdJT2pTVnr06CEFgODgYAKUte8mTZpobNuqVSsSRdFsy0p16f3335ea7kRRrHOArKkneuu4\nCQ0NBaDsnVOxc+dOXLx4UV9ZmhxZWVkAAFtbW/j5+VnMxLKNGjUCY0ztscPQ0FD4+/uDMYYPP/wQ\nR44ckR5VBCAtX7x4sVyyZUHVQVVaWlrt+odHA5gzqhEOb775JqysrCCKIlatWoUbN27oNV8+TpLD\n4XC0oLea5P79+wEAjRs3BgAcP34c0dHR+srOJBEEAYCylmRvby+zGsPywO2WhEKhABHh6aefRlZW\nFhwcHJCRkQEAeP7551FcXCyHVIPDGJMm+ngcERERj6xlmhMBAQFISEgAAAQFBQEAFi1ahClTpug/\nc321H6jaC1RtkXWd9gq1bD8whTaVB1NqaiopFApatmyZxXjyYBujqs3x7bffpqKiIrWp0nJzc6lv\n3741mqjA2D2pTVl5sE3S2dm52m1sbW3J1taWdu3a9diOLGP2paaevPjiixoTWHTu3NkgMUUvNck1\na9ZIg4RVJCcn6yMrk2f//v1o3rw5PvjgA7mlGIyqqircvXsXjo6OSEpKAgDVCSNRWlqKzZs3Y8+e\nPXJIlJXU1FTk5OQAAIYPH47ly5errbe1tZWW+fv74/XXXze4RkPj5uam9vnw4cNITU01SN46D5Kh\noaGIjIyU5ryrrKzkU8w/BiJCZWWl3DIMxqlTpxAVFYUPPvgA3bp1U1u3du1apKSk4MyZMzhy5Ig8\nAmUmJycHX3zxBQBgwYIFAIANGzagRYsWaNu2LT755BPcu3cPgLJjND8/XzathuLhp2iWL1+OwsJC\nw2Su66pxt27d1LrnL1++XO8qMWpRNTbW24VHpcWLF5NCoaCBAwdyT8y4nNTVl3fffZfu3r0rnU9F\nRUU0a9YssrOzIzs7O5P3pSa6W7duTbdu3ZI8mD59Ot1/X7dBygp/EZjMvPbaa6ioqMCFCxfklsIx\nQpYuXYqlS5fKLUNWwsPD4ezsLH2uqKhQBViDoPMhQGlpabz9sRb8+uuvSE9PR3l5udxSOByj5Lvv\nvkNubi5u376N27dvY9++fQbNn4+T5HA4HC2w2lRb77cDyAYR1WzwmAHhnmjCPake7osmpuAJr0ly\nOByOFniQ5HA4HC3Utnc7H4BcszD4yZTv4+CeaMI9qR7uiyZG70mt2iTrAmPsKQBLAbQHcAvAZCLa\nrtdMTQTG2L8ApADYQkTD5dYjJ4yxcQCiAbQBEE9E0bIKMhL4+aOJocuKXm+3GWM2AHYC+AmAG4DR\nAP7HGAvUZ74mxFIAJ+UWYSTcADAbwGq5hRgL/Px5JAYtK/pukwwG0AzAIiISiegggCQAI/Scr9HD\nGBsKoAjAAbm1GANEtI2IdgAokFuLEcHPn2owdFmRo+OGAQiRIV+jgTHmAmAWAMuZ1YKjKyz+/DE0\n+g6SFwHkAZjMGLNljPUCEAHAUc/5GjuxAL4jIsuZVppTF/j5YwToNUgSURWAAQBeBHATwIcANgOw\n2ODAGAsFEAlgkdxaOMYNP3+MA71PcEFE56C8+gEAGGPJANbqO18jphsAfwBZ92efdgJgzRhrRUTt\nZNTFMUL4+SM/eg+SjLGnAaRDWWt9B4AXgO/1na8RsxLAxgc+T4IyaI6VRY2RcL8n1waANZQXDQcA\nAhEJ8iqTF37+aGLosmKIjpsRAHKgbFvpAaAnEVUYIF+jhIjuEtFNVQJwB8A9IroltzaZmQqgHMDH\nAIbf/3+qrIqMA37+aGLQsqL3weQcDodjyvBntzkcDkcLPEhyOByOFniQ5HA4HC3wIMnhcDhaqNUQ\nIFOYRdjQcE804Z5UD/dFE1PwhNckORwORws8SHI4HJMmPj4eGRkZ6NixIzp27Kjz3+dBksPhcLSg\n98cSOfXnwIEDYIzhhRdekFsKh2N0+Pn5wd/fH//73/8AAK1atUJVVZXOfp/XJI2cRYsWoVOnTrh0\n6ZLcUvQKYwwbN25ESUkJvL295ZbDMRF8fHwQFhYGAAgICEBAQABsbHRb9+NB0oiZO3cu3n77bVRV\nVeHAAfOewLxBgwbo3LkznJyc0KdPH7nlcEwEV1dX2NraAgB27NiBHTt2oKJCt4+2GyRIduzYEXFx\ncTh37hxEUYQoipg4cSImTpyIIUOGIC4uTi8NrqZOeHg4bG1tceLECWzevFluOXrl7t27Um3Z3d1d\nZjXGh5ubG8aOHYvs7GzpHBJFETExMXJLkw0bGxu1/f/hhx/www8/QKFQ6DYjIqpxAkC1TUOGDKGb\nN2+SIAgkiiIdOHCAzp07R4IgSEkURdq4ceNjf6s2Wg2V6uIJAOratSvt27eP3Nzcql0fFRVF+fn5\ndPHiRWrbtq1FePLqq6+SQqGgdevW1en7xuxJfXwJDw+nY8eOkSiKaueNKq1Zs8ZkfanPcY6Li6MH\nGTx4MA0ePFjnZUUvO2RjY0Ph4eEUHh5OJSUlJAgCHTx4kLp37062trbk5OREu3fvpt27d0tBctKk\nSRZ1kNPS0kgURerSpUu161NSUoiIaODAgRbjiY+PDykUCrp37x55eXnxIAlQ06ZNKSUlhQRBoJs3\nb9KKFSuob9++FB8fT/Hx8SQIAl24cIHs7OxM0pe6HuNRo0bR3bt36UFMKkhGR0erXen27NlDLi4u\n0vrhw4errc/MzCR3d3eLOsinT58mQRAoMjJSY11oaCiVlJSQKIoUFRVlMZ74+PgQEZFCoaAxY8bw\nIAlQUlISCYJAu3fvVlseEBBAAQEBdOvWLSotLdV6t2HMvtTFkzfeeIMqKiqIiOjUqVOkQl9Bknfc\ncDgcjhZ0Pk4yNjYWn3zyieoqgWXLlmHq1KkoKSmRtvn000/VvjNhwgTcumU5E3PHxsaiTZs2uHDh\nAv78809pecOGDQEAU6ZMgaOjI44fP44tW7bIJVMWVOXGzs5OZiXGQXl5OQBg586dj9ympKQE+fn5\nhpJkMJycnNC2bVsEBgaiY8eOeO211wAAjRs3BqCMG7t378bly5f1qkOnQXL69On45JNPUFlZiX37\n9gFQnvCqA+3g4IBevXrB19cX91+ChdmzZ2stAOaEj48PAGDUqFEQBAHjxo1TuzgsXLgQADB48GDc\nuHEDnTt3lkUnx3hgjIExhsLCQjg4OKBly5aIjo5G+/btAQA3b95EVFQUrl+/LrNS3ePt7Y3Vq1cj\nMDAQAFBcXAwAWLVqFb788ktcu3bNMGNqddV+0KhRI6kXe8eOHRrrAwIC6MSJE1I75KZNm2jTpk3U\nsGFDk25rqqn2kJAQunTpEl26dIkEQaBFixaprZ80aRJVVFRQRUUFCYJQ4zY5ufdfV+1MwD8dNwqF\ngsaPH8/bJAHpnDp+/Lja+TNo0CAaNGiQyfvyOM329vYUEhJCISEh5OvrS76+vmrr/f39SUV0dDRF\nR0fr3BOd7dATTzwhHUBfX1964okn6IknnqApU6ZQUlISFRcXS0MYqqqqqH///tS/f3+zP8g2NjYU\nHR1NoihKB1MURTp+/DjFxMSQvb09eXl50YkTJ6iyspIqKytp9erVZu2JtiBJRDxIPpDOnz9PZWVl\n0igQQRCopKSEWrVqRa1atTJ5X+p6jFWpSZMmlJOTQ0RE27dvp+3bt+vcE53dbldWVuLWrVtwd3dH\nRkaGygCJGzduoKSkBF5eXsjPz0diYqKusjZqhg4dim+//RZEJA1yvXz5MsLCwhAWFoZXXnkFzZs3\nh5eXl3Tr/X//939ySpaVh8uNpdO6dWuEh4fD29sbmzZtAgBs27YNqampMiszDgoKCpCRkQFPT08c\nOnRIL3norHe7qKgIAwYMQGFhIQDgypUruHLlChYuXIinn34aHTp0wJUrVwAAGzdu1PZTZsOQIUOw\nZs0aVFVV4datW4iMjERkZCSGDBmCI0eOAADCwsLQrFkzAEDTpk3RtGlTZGdno2XLlnJKl51z587J\nLcFoOH78ONLS0qTPX3zxhYxqjJecnBzk5OTo/Hd12nFz4sSJRz5S1rVrV0REREChUODq1au6zNZo\nGTNmDLKysjB79mysWbNGbd348ePx3//+F88995y0TNWZdejQIemCYqlY+v4/TJs2bWBlZaX7R+7M\nBCJCXl6eXn6bj5PkcDgcLRhsPskGDRpAoVCAiCzmdnvnzp3Ytm0bsrOzNdY1bdoUISEhAICoqCj8\n9ddf0rq///7bYBo5pkF5eTkUCgUOHz6MyspKueXIQkBAAADlZB+AclKU27dvY+HChfjyyy+lu1h3\nd3c4Ojpi9uzZ+PHHH5GQkFC/jA3ZE6XqnavJI4jVJbl74nTliaurK8XFxZEoipSenl4vT+Xef12W\nkweHALVs2dKsPKmPL8HBwZSYmEg3b96s9bAfY/flcZrt7OwoODiYFi5cSGVlZVRWVkYqKioqqKCg\ngB4mLy+PioqKiIjoq6++qrcnBqtJ9u7d21BZGT3vvPMOxo4di7y8PD7b+CPo168f4uLi5JYhO66u\nrti3bx+aN2+OKVOmWNQTWB4eHliyZAmGDBmisS4nJwdEhPPnz6s9tfYwa9eurbcOgwXJFi1aGCor\no8XPzw8A8NZbb4GIsHLlSn5r/QC5ubk4f/48WrduLbcUo+HLL79E8+bNER8fjwULFsgtx6D85z//\nUQuQP/30EwBgwYIFSEpK0ukrGrRiiNsFQPnECZFyILWl3m6np6dTeno6CYJA33//fb1us83Fk4fT\nyZMnSaFQUEJCgll5Uh7wG7sAAAG1SURBVFtfIiMjKTIyksrKyqi0tJReeeUViysr/v7+dO7cOdq7\ndy+98cYbOjlf6uKJwWqSf/31Fy5duoQWLVqgZcuWFjWhhQrVMKDY2FiLeV69tpw9exbt27eHk5OT\n3FJkw9/fXxo4DgAjR460yPJy7do1PP3003LLgMFqCMA/80weOHCAP1ZlwCuhKXni7+9Px44do7ff\nftusPKmpLw0aNKC4uDi1OQ54WZH3/OHjJDkcDkcL7H40r9nGjNV842pwcXHB5s2bERkZiW3btgEA\n3njjDZSVldXo+0TE6pO/PqivJ/WFe6KJMXoC1MyXsWPH4ptvvkFycjIAIDIyUmdv/zNGX0yirBi6\nauzi4qJ2O1GbW265bw1M9XaBe2Ic6XG6O3ToQNnZ2TRz5kzy9vYmb29vs/fFFMqKQWuS9YX4lVAD\n7okmxugJwH2pDlPwpLa92/kAMusmp974yZTv4+CeaMI9qR7uiyZG70mtapIcDodjafDebQ6Hw9EC\nD5IcDoejBR4kORwORws8SHI4HI4WeJDkcDgcLfAgyeFwOFrgQZLD4XC0wIMkh8PhaIEHSQ6Hw9HC\n/wNZ0uieVAjGXwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 25 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WoNhnQtRNhHu",
        "colab_type": "text"
      },
      "source": [
        "Therefore, we have 60000 training samples and 10000 test samples, where each image has a shape of $28\\times 28$ pixels. Visualizing the data before starting to deploy any algorithms is always a good idea: it is a quick sanity check that can prevent easily avoidable mistakes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DohVlsvsN9np",
        "colab_type": "text"
      },
      "source": [
        "## Preprocessing Keras datasets\n",
        "\n",
        "In this first example, we will train a simple model to classify the digits on MNIST dataset. As will be introduced in the lectures, convolutional layers are usually employed to classify images. However, for the moment we use only `Dense` layers, so we have to reshape the images to have a single dimension. Therefore, the same network could be used for any 1D data. We will see on following weeks that when using Convolutional Neural Networks (CNNs) we must define a 3D shape for images (depth, width, height). CNNs usually have higher performance compared to models with only `Dense` layers when applied to 2D data.\n",
        "\n",
        "Let's flatten the data now."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aa6-T9T9PABQ",
        "colab_type": "code",
        "outputId": "e4cec3c0-5760-4fc9-93d6-200722e4c84a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "X_train_flatten = X_train.reshape(X_train.shape[0], X_train.shape[1] * X_train.shape[2])\n",
        "X_test_flatten = X_test.reshape(X_test.shape[0], X_test.shape[1] * X_test.shape[2])\n",
        "\n",
        "print('New X_train shape: {0}'.format(X_train_flatten.shape))"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "New X_train shape: (60000, 784)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aVLnq7URVwxH",
        "colab_type": "text"
      },
      "source": [
        "A standard practise is to normalise the dataset to the range $[0, 1]$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jbOITHGSV9WS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X_train_flatten = X_train_flatten.astype('float32')\n",
        "X_test_flatten = X_test_flatten.astype('float32')\n",
        "X_train_flatten /= 255\n",
        "X_test_flatten /= 255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZTwrNtOWNBe",
        "colab_type": "text"
      },
      "source": [
        "To train the model we also process the labels. The `y_train` and `y_test` labels are represented as a single array with the class values as integer. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hZtCSmMos8uo",
        "colab_type": "text"
      },
      "source": [
        "For the loss we will use for training, we need to transform those values to a [one-hot encoded vector](https://en.wikipedia.org/wiki/One-hot)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z-KifrbWXreU",
        "colab_type": "code",
        "outputId": "8d4b862b-8fa5-46be-c9a2-d10dd478629b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "Y_train_class = np_utils.to_categorical(y_train, 10)\n",
        "Y_test_class = np_utils.to_categorical(y_test, 10)\n",
        "\n",
        "print('New Y_train shape: {0}'.format(Y_train_class.shape))"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "New Y_train shape: (60000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jokxf-U8thD8",
        "colab_type": "text"
      },
      "source": [
        "To see the difference in representation, let's print the first element of `y_train` and `Y_train_class`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0oYrx0kStdfy",
        "colab_type": "code",
        "outputId": "efb838da-2a0e-4732-e241-2e48cc378913",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "print(y_train[0], Y_train_class[0])"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(5, array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32))\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OcgAG0OZLZ3W",
        "colab_type": "text"
      },
      "source": [
        "## Model Architecture\n",
        "\n",
        "Defining model architectures is a critical step when trying to solve a machine learning problem. In this tutorial, we will understand how models can be built in Keras, starting from a simpler model and then using a more complex example. Here is a list of [implemented models for specific datasets](https://github.com/keras-team/keras/tree/master/examples) which can be a useful reference. \n",
        "\n",
        "First, we need to declare a `Sequential` module. As mentioned, a `Sequential` module is a linear stack of neural network layers. We, therefore, can create a mode  by passing a list of layer instances to the `Sequential` module, "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9_sJo3MbS1Lz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = Sequential([\n",
        "    Dense(10, input_shape=(784,)),\n",
        "    Activation('softmax'),\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEVpTbQ9TX_u",
        "colab_type": "text"
      },
      "source": [
        "Keras also allows us to simply add layers via the `.add()` method:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w-i9u2fbTZCk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = Sequential()\n",
        "model.add(Dense(10, input_dim=784))\n",
        "model.add(Activation('softmax'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xGsQ9NTBBHFC"
      },
      "source": [
        "We now have a simple model ready. However, the model needs to know what input shape is going to receive. For this reason, the first layer in a Sequential model (and only the first, because the following layers can do automatic shape inference) needs to receive information about its input shape. As seen previously, after the flatten operation our images have a shape of 784. If we were using directly images, the input shape would have been (28, 28, 1), which corresponds to the (width, height, depth) of each digit image. The formatting (width, height, depth) is called `channels_last`, which is used by default in Keras, however Keras also accepts the format (depth, width, height), which is called `channels_first`. You can either pass the argument `data_format` when defining a layer or setting your preferred data format as default by using the code given [here](https://keras.rstudio.com/reference/k_image_data_format.html). \n",
        "\n",
        "We now can check the model shape output:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h5CZJUssUW1b",
        "colab_type": "code",
        "outputId": "3d6d1fe8-f5a3-47a9-ef2b-516a088f338e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "print(model.output_shape)"
      ],
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(None, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kpz5BNJ8UnuP",
        "colab_type": "text"
      },
      "source": [
        "The dense layer used in our model is a fully connected layer, where the first parameter is the output size of the layer. The final layer has an output size of 10, corresponding to the 10 classes of digits. Keras automatically handles the connections between layers. In this case, the layer has $784\\cdot 10+ 10=7850$ parameters, where the first term refers to the connections between input data and neurons and the second term to the output bias.\n",
        "\n",
        "We can see how our model looks like by using `.summary()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a37n1PQzU3BX",
        "colab_type": "code",
        "outputId": "5a39f960-7153-43b3-b6f2-2e606ffd2d90",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 215
        }
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_15 (Dense)             (None, 10)                7850      \n",
            "_________________________________________________________________\n",
            "activation_4 (Activation)    (None, 10)                0         \n",
            "=================================================================\n",
            "Total params: 7,850\n",
            "Trainable params: 7,850\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TPQEWWDuwCzk",
        "colab_type": "text"
      },
      "source": [
        "The number of parameters given by Keras corresponds to the number we computed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q_zb2cNDVaJi",
        "colab_type": "text"
      },
      "source": [
        "## Compilation\n",
        "\n",
        "Before starting to train our model, we need to configure the learning process, which is done via the compile method. We need to define three important parameters here:\n",
        "\n",
        "*   The loss function that the model will try to minimize.\n",
        "*   The optimiser used to minimise the loss function and update the weights.\n",
        "*   The list of metrics you want the model to compute.\n",
        "\n",
        "In our example, as we are doing classification, we will define the categorical_crossentropy as the loss function and we will compute the accuracy metric. You can refer to Keras documentation to learn more about [losses](https://keras.io/losses/) or [optimisers](https://keras.io/optimizers/). There are many other parameters (learning rate, decay factor, weight normalisation...) than could be manually set. We will explore in more detail all of these parameters in the following tutorials. Now, we will use the default values for them. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kHasLrmvWtSf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iqImvb8kWznJ",
        "colab_type": "text"
      },
      "source": [
        "## Training\n",
        "\n",
        "To train the model, we call `.fit()` after compiling the model. We have to define the batch size and the number of total epochs the model is going to train. We are finally ready to train our classifier! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "trVlIdWNXUwd",
        "colab_type": "code",
        "outputId": "36243793-5fb3-4ba2-bcf4-02cae0ce8177",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 377
        }
      },
      "source": [
        "history = model.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=10)"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "60000/60000 [==============================] - 5s 85us/step - loss: 0.4693 - acc: 0.8793\n",
            "Epoch 2/10\n",
            "60000/60000 [==============================] - 5s 80us/step - loss: 0.3042 - acc: 0.9152\n",
            "Epoch 3/10\n",
            "60000/60000 [==============================] - 5s 80us/step - loss: 0.2836 - acc: 0.9210\n",
            "Epoch 4/10\n",
            "60000/60000 [==============================] - 5s 80us/step - loss: 0.2728 - acc: 0.9235\n",
            "Epoch 5/10\n",
            "60000/60000 [==============================] - 5s 80us/step - loss: 0.2664 - acc: 0.9260\n",
            "Epoch 6/10\n",
            "60000/60000 [==============================] - 5s 79us/step - loss: 0.2620 - acc: 0.9268\n",
            "Epoch 7/10\n",
            "60000/60000 [==============================] - 5s 79us/step - loss: 0.2580 - acc: 0.9286\n",
            "Epoch 8/10\n",
            "60000/60000 [==============================] - 5s 80us/step - loss: 0.2550 - acc: 0.9294\n",
            "Epoch 9/10\n",
            "60000/60000 [==============================] - 5s 79us/step - loss: 0.2528 - acc: 0.9308\n",
            "Epoch 10/10\n",
            "60000/60000 [==============================] - 5s 78us/step - loss: 0.2506 - acc: 0.9308\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yj8MJvUnXzHj",
        "colab_type": "text"
      },
      "source": [
        "Great, we have trained our first model! \n",
        "\n",
        "Now, imagine we need to train the model until it converges, and we do not know how many epochs it will take. Then, we could use callbacks to define early-stopping rules. In addition, we can use those [callbacks](https://keras.io/callbacks/) to save the model's weights or log history along the training process. An example of using the early stopping callback will be given later in this tutorial.\n",
        "\n",
        "\n",
        "Most of the times, the training curves give important information about the training process. In order to visualise them, we can use the information saved on the training history:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OGy405W2boBB",
        "colab_type": "code",
        "outputId": "6ad13767-79e9-4b9f-8fff-f8ff5e4724cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        }
      },
      "source": [
        "fig, axes = plt.subplots(2,1)\n",
        "axes[0].plot(history.history['acc'])\n",
        "axes[0].set_title('Model Accuracy')\n",
        "axes[0].set_ylabel('Accuracy')\n",
        "axes[0].set_xlabel('Epoch')\n",
        "\n",
        "fig.subplots_adjust(hspace=0.5)\n",
        "\n",
        "axes[1].plot(history.history['loss'], 'g')\n",
        "axes[1].set_title('Model Loss')\n",
        "axes[1].set_ylabel('Cross Entropy Loss')\n",
        "axes[1].set_xlabel('Epoch')"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5,0,'Epoch')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAIABJREFUeJzt3Xd4HNXZ9/HvT12WreKCm2zLLYBp\nxohikwcSQ0wNhDQghfImQPKGagglAQI8kIQOIYQ8BEgIkPASCHkgITGEGsAUG1NsTDHGxnIBy0Vu\n6rrfP2Ykr2SVlVarkbT357rm2pkzM7v3LnhunXNmzpGZ4ZxzzrUnLeoAnHPO9X6eLJxzznXIk4Vz\nzrkOebJwzjnXIU8WzjnnOuTJwjnnXIc8WbiUJqlEkknKiOPYUyS92BNxOdfbeLJwfYakZZJqJA1t\nUb4gvOCXRBNZs1gGStoi6Z9Rx+Jcd/Jk4fqaj4ETGzck7QEMiC6cHXwNqAa+JGlET35wPLUj57rK\nk4Xra+4DTorZPhn4Y+wBkgok/VHSWknLJV0qKS3cly7pBknlkpYCR7Vy7t2SVktaKelqSemdiO9k\n4LfA28B3Wrz3GEl/DeNaJ+nXMftOk7RY0mZJ70qaFpabpEkxx/1B0tXh+hcklUm6SNIa4PeSiiT9\nPfyMDeF6ccz5gyX9XtKqcP/fwvKFkr4cc1xm+Bvt3Ynv7voxTxaur3kFyJe0a3gRPwG4v8UxtwEF\nwATgYILkcmq47zTgaGBvoBT4eotz/wDUAZPCY2YB348nMEnjgC8AD4TLSTH70oG/A8uBEmA08GC4\n7xvAFeHx+cAxwLp4PhMYAQwGxgGnE/yb/n24PRaoBH4dc/x9BDWx3YCdgJvD8j/SPLkdCaw2swVx\nxuH6OzPzxZc+sQDLgEOBS4FfAIcDTwEZgBFchNOBGmBKzHlnAM+F688AP4jZNys8NwMYTtCElBuz\n/0Tg2XD9FODFduK7FHgzXB8N1AN7h9vTgbVARivnzQHOaeM9DZgUs/0H4Opw/Qvhd81pJ6apwIZw\nfSTQABS1ctwoYDOQH24/DFwY9X9zX3rP4m2cri+6D3gBGE+LJihgKJBJ8Bd8o+UEF28ILoorWuxr\nNC48d7WkxrK0Fse35yTgdwBmtlLS8wTNUguAMcByM6tr5bwxwEdxfkZLa82sqnFD0gCC2sLhQFFY\nPCis2YwB1pvZhpZvYmarJL0EfE3So8ARwDldjMn1Q94M5focM1tO0NF9JPDXFrvLgVqCC3+jscDK\ncH01wUUzdl+jFQQ1i6FmVhgu+Wa2W0cxSZoBTAYukbQm7EPYH/hW2PG8AhjbRif0CmBiG2+9jeYd\n+C07zVsOG30+sDOwv5nlAwc1hhh+zmBJhW181r0ETVHfAOaa2co2jnMpyJOF66u+B8w0s62xhWZW\nDzwEXCNpUNiPMJvt/RoPAWdLKpZUBFwcc+5q4EngRkn5ktIkTZR0cBzxnEzQJDaFoOlnKrA7kEvw\nV/prBInql5LyJOVIOjA89y7gAkn7KDApjBvgTYKEky7pcII+mPYMIuin2ChpMPCzFt/vn8Bvwo7w\nTEkHxZz7N2AaQY2iZY3NpThPFq5PMrOPzGxeG7vPArYCS4EXgT8B94T7fkfQR/AW8AY71kxOArKA\nd4ENBG33I9uLRVIO8E3gNjNbE7N8TNBkdnKYxL5M0HH+CVAGHB9+l78A14Rxbia4aA8O3/6c8LyN\nwLfDfe25hSBBlRPcDPCvFvu/S1Dzeg/4DDi3cYeZVQKPEDTvtfxdXIqTmU9+5JwLSLoc+JyZfafD\ng11K8Q5u5xwQPINB0Lz33ahjcb2PN0M555B0GkEH+D/N7IWo43G9jzdDOeec65DXLJxzznWo3/RZ\nDB061EpKSqIOwznn+pT58+eXm9mwjo7rN8mipKSEefPaupPSOedcayQt7/gob4ZyzjkXh35Ts3DO\nuWQwM6rrGthUWcumqloqKuvYVFXLpspaKmvqg2OaHR+zHu5pXrbjwW2e39H+8HXYoGyO2WtUZ75W\np3mycM71a2ZGZW09myrr2FxVG17ot1/wN1XVNSWCpvKqOjbHlNXUN0T9Ndo1dUyhJwvnXOoxM2rq\nG6iqaaCytp6q2noqw6Wqtp7KmvqmC/nmqjYu+DGJoK6h/UcEsjPSyM/NJD8ng/zcTApzMxk7eAD5\nORkMyskkPzeD/JzMZsfk52QyICudxgGKRdNIxWwftJjtpc3Kdjw2Zjcxox43lauN8xGkp8WenRye\nLJxzcatvsKYLd1XThbuBqrrgAt68vJ6quobgtdkFv6HFMeFrWN6YFDrzCFhuZnqzC/qQvCxKhuS1\nuMjveNEflJPJoJwMcjI7MxliavJk4VwKq6ypZ/22GtZvqWHd1mo2bKth3ZYa1m/dcVm3tYaKytou\nfU52Rhq5WenkZKQHr5np5GSmkZuZTkFuJjlN+4KynHBpXG+tPPaCn5Xh9+okmycL5/oJM2NTZV1w\n8d9avf2iHyaDxgt+bEKorK1v9b3S00TRgCyG5GVRlJfJriPzGZyXRVFeFnlZ6U0X/pysxgt6i4t5\nWJ6bmU52RhppPdBM4pLLk4VzvUTLdvrK2u1NO5U19WysDC/4W8IL/tbtSWD9tho2bK1ps20+JzON\nIXnZDM7LYnBeFhOHDWxaHxImgSFN29kMysnwC7xrxpOFc3FoaLAd2uUrd7io1zWVVcVe6GvrqYpZ\nb2zD31ZTv8OxHfTDNikI2+WL8rIYO2QAe48tbHbBj12G5GWTm+Vt8i4xnixcSquuq+ezTdWsrqhi\ndUUlayqqWF1RFb5WsrqiiorKWqrrOn/rZJqCjtfcrObNMjmZ6RTlZTGqMNyO2dfYnp/bop0+NzOd\nwgFBk1DRgCwy072N3vUsTxau36qqrd9+8d8UXPhXb9y+vaaiivItNTucNyg7gxEFOYwoyGHnEYMo\nystq/WK+w8U9JiFkpZGVntbsFkjn+jJPFq5P2lZT11QDWLUxrBFsqoqpGVSyYduOd+4UDshkRH4O\nIwty2GN0ISPDpDAyXIbn5zAoJzOCb+Rc7+bJwvU6W6vrWF1RycqNwUV/e7PQ9uahTVV1O5w3JC+L\nEQU5jC7MYZ9xhYwsyG1KDI01hQFZ/r+8c13h/3Jcj6pvMNZurmblxkpWxSwrNwY1hFUVlWxsUSOQ\nYOjAbEYW5DBuyAAOmDCYEQW5TbWBkQW57JSf7Q9WOZdEnixct9pSXRde/GOTQVXT9pqKqh1u78zP\nyWBUYS6jC3PZZ1wRowpzGVWYw6jCICHsNCjHH7pyLmKeLFzc6uob+GxzdUwyqIqpGQSvLZuHMtLE\niILgwr9vyeCmJNCYHEYWeB+Bc31BUpOFpMOBW4F04C4z+2WL/eOAe4BhwHrgO2ZWJmkqcAeQD9QD\n15jZ/0tmrC5QvqWat8s2bm8WiqkdrNlURX2LWkHhgExGFeRSXDSA/ccPbkoEjclg2KDsHhnkzDmX\nXElLFpLSgduBLwFlwOuSHjOzd2MOuwH4o5ndK2km8Avgu8A24CQz+1DSKGC+pDlmtjFZ8aaqLdV1\nvLp0HS8tWcfLH5Xz3prNTfsy0sTIwhxGFeS2SAQ5jA7X87K9cupcKkjmv/T9gCVmthRA0oPAsUBs\nspgCzA7XnwX+BmBmHzQeYGarJH1GUPvwZJGg6rp63li+kZc/KuelJeW8VVZBfYORlZHGviVF/Piw\nndlv/GDGDh7A0IFeK3DOBZKZLEYDK2K2y4D9WxzzFvBVgqaq44BBkoaY2brGAyTtB2QBH7X8AEmn\nA6cDjB07tluD7y/qG4xFqyqaag6vL1tPVW0DaYI9iwv5wcETOHDiUKaNK/K7iZxzbYq6DeEC4NeS\nTgFeAFYS9FEAIGkkcB9wspntMN6Cmd0J3AlQWlraidHv+y8zY2n5Vl5eUs5LS9Yxd+m6pmGlJ+80\nkBP2HcuMiUPYf8IQCnK9Y9k5F59kJouVwJiY7eKwrImZrSKoWSBpIPC1xn4JSfnAP4CfmtkrSYyz\nz1tTUcVLS8p56aNyXl6yjjWbqgAYXZjLrCnDOXDSUGZMHMJO+TkRR+qc66s6TBaSzgLuN7MNnXzv\n14HJksYTJIkTgG+1eO+hwPqw1nAJwZ1RSMoCHiXo/H64k5/b723cVsMrYaf0Sx+Vs3TtVgCKBmQy\nY+JQZkwawoEThzJuyAAfm8g51y3iqVkMJ7iT6Q2Ci/kcs44nPDSzOklnAnMIbp29x8wWSboKmGdm\njwFfAH4hyQiaoX4Unv5N4CBgSNhEBXCKmb0Z/1frPypr6nl92fqmmsPCVRWYwYCsdPYbP5gT9x3L\njElD2HVEvs9B4JxLCsVx3UfBn6ezgFOBUuAh4G4z26HTOSqlpaU2b968qMPoFrX1DbxdtjGoOSwp\nZ8EnG6mpbyAzXew9piioOUwayl7Fhf5ks3MuIZLmm1lpR8fF1WdhZiZpDbAGqAOKgIclPWVmFyYW\nqoMgQfz5tU947v21vPbxerZU1yHBlJH5nHJgCTMmDmG/8YN9IDznXCTi6bM4BzgJKAfuAn5sZrWS\n0oAPAU8W3eA3z37Ezf/+gPFD8zh26igOnDSU6ROGUJSXFXVozjkXV81iMPBVM1seW2hmDZKOTk5Y\nqWVZ+VZuf24JR+85kl9/a1rU4Tjn3A7iafD+J8G4TUBwS6uk/QHMbHGyAksVZsZl/7uQrPQ0Ljt6\nStThOOdcq+JJFncAW2K2t4Rlrhv8453V/OfDci6Y9TmG+3MQzrleKp5kodhbZcNnIryXtRtsrqrl\nqsffZffR+Xx3eknU4TjnXJviSRZLJZ0tKTNczgGWJjuwVHDjkx+wdks113xlDx+wzznXq8WTLH4A\nzCB4CrtxMMDTkxlUKli4soI/zl3Gd/Yfx15jCqMOxznn2tVhc5KZfUYwVIfrJvUNxk8ffYfBedlc\ncNjOUYfjnHMdiuc5ixzge8BuQFMPrJn9nyTG1a/96bVPeKusgluOn+ojvzrn+oR4mqHuA0YAhwHP\nE4weu7ndM1yb1m6u5rp/vceMiUM4duqoqMNxzrm4xJMsJpnZZcBWM7sXOIodJzFycfr5E4uprm3g\nv7+yu48I65zrM+JJFrXh60ZJuwMFwE7JC6n/enlJOY8uWMkZB09g4rCBUYfjnHNxi+d5iTslFQGX\nAo8BA4HLkhpVP1RdV8+l/7uQsYMH8KMvToo6HOec65R2k0U4WOCmcOKjF4AJPRJVP/S7F5aydO1W\n/nDqvj7XtXOuz2m3GSp8WttHlU3QJ+u2cdszSzhyjxF8YWdvwXPO9T3x9Fn8W9IFksZIGty4JD2y\nfsLMuPyxhWSkicuP3i3qcJxzrkvi6bM4Pnz9UUyZ4U1ScfnXwjU89/5aLjt6CiMKfKBA51zfFM8T\n3ON7IpD+aEt1HVc+/i5TRuZz8vRxUYfjnHNdFs8T3Ce1Vm5mf+z+cPqXm5/6gE83V/Gb70wjI93n\nynbO9V3xNEPtG7OeAxwCvAF4smjHu6s28YeXl3HCvmOZNrYo6nCccy4h8TRDnRW7LakQeDBpEfUD\nDQ3GT//2DoW5mVx0uA8U6Jzr+7rSNrIViKsfQ9Lhkt6XtETSxa3sHyfpaUlvS3pOUnHMvpMlfRgu\nJ3chzsg8+PoKFnyykZ8cuSuFA7KiDsc55xIWT5/F4wR3P0GQXKYAD8VxXjpwO/AlgnkwXpf0mJm9\nG3PYDcAfzexeSTOBXwDfDW/N/RlQGn72/PDcDfF/tWiUb6nm2n+9x/7jB/PVaaOjDsc557pFPH0W\nN8Ss1wHLzawsjvP2A5aY2VIASQ8CxwKxyWIKMDtcfxb4W7h+GPCUma0Pz30KOBz4cxyfG6lfPPEe\n22rquOY4HyjQOdd/xNMM9Qnwqpk9b2YvAesklcRx3mhgRcx2WVgW6y3gq+H6ccAgSUPiPBdJp0ua\nJ2ne2rVr4wgpuV5Zuo5H3ijjtP+awKSdBkUdjnPOdZt4ksVfgIaY7fqwrDtcABwsaQFwMMHUrfXx\nnmxmd5pZqZmVDhs2rJtC6pqaugYu/dtCiotyOWvm5Ehjcc657hZPM1SGmdU0bphZjaR4em1XAmNi\ntovDsiZmtoqwZiFpIPA1M9soaSXwhRbnPhfHZ0bmrheXsuSzLdxzSim5WT5QoHOuf4mnZrFW0jGN\nG5KOBcrjOO91YLKk8WFyOYFgiPMmkoaGI9sCXALcE67PAWZJKgqHR58VlvVKK9Zv41dPf8hhuw1n\n5i7Dow7HOee6XTw1ix8AD0j6dbhdBrT6VHcsM6uTdCbBRT4duMfMFkm6CphnZo8R1B5+IckIhkD/\nUXjuekn/TZBwAK5q7OzubcyMKx5bRJrEz77sAwU65/onmVnHR9HUTISZbUlqRF1UWlpq8+bN6/HP\nnbNoDWfcN5+fHrkrpx3kYys65/oWSfPNrLSj4zpshpL0c0mFZrbFzLaETUNXd0+YfdvW6jqufGwR\nu4wYxCkHlkQdjnPOJU08fRZHmNnGxo3wwbgjkxdS33Hr0x+yqqKKq7+yO5k+UKBzrh+L5wqXLim7\ncUNSLpDdzvEp4b01m7j7xY85vnQMpSU+F5Rzrn+Lp4P7AeBpSb8HBJwC3JvMoHq7hgbj0kcXkp+T\nwcVH7BJ1OM45l3TxjDp7raS3gEMJxmmaA6T0TD4Pzy9j3vINXPf1PSnK84ECnXP9X7wN7Z8SJIpv\nADOBxUmLqJdbv7WGn/9zMfuWFPH1acUdn+Ccc/1AmzULSZ8DTgyXcuD/Edxq+8Ueiq1X+uU/F7Ol\nqo6rv7IHaWk+UKBzLjW01wz1HvAf4GgzWwIg6bweiaqXen3Zeh6aV8YZB09g5xE+UKBzLnW01wz1\nVWA18Kyk30k6hKCDOyXV1jdw6aMLGV2YyzmH+ECBzrnU0mayMLO/mdkJwC4Ec02cC+wk6Q5Js3oq\nwN7inhc/5v1PN3PFMbsxICuem8icc67/6LCD28y2mtmfzOzLBKO/LgAuSnpkvcjKjZXc8u8POXTX\n4Xxpig8U6JxLPZ167NjMNoRzSBySrIB6oysfWwTAFcdMiTgS55yLho9R0YF/v/spT777KWcfMpni\nogFRh+Occ5HwZNGObTV1/OyxRXxu+EC+/1/jow7HOeci4z217bjtmSWs3FjJQ2dM94ECnXMpza+A\nbfjg08387oWlfH2fYvYb7wMFOudSmyeLVpgZl/5tIQNzMrjEBwp0zjlPFq155I2VvPbxei4+fBeG\nDEz50didc86TRUsbttbw8ycWM21sId8sHRN1OM451yt4smjhujnvUVFZyzXH+UCBzjnXyJNFjPnL\nN/Dn11bwfw4sYdeR+VGH45xzvUZSk4WkwyW9L2mJpItb2T9W0rOSFkh6W9KRYXmmpHslvSNpsaRL\nkhknQF19Az999B1GFuRw7qGfS/bHOedcn5K0ZCEpHbgdOAKYApwoqeV4GZcCD5nZ3sAJwG/C8m8A\n2Wa2B7APcIakkmTFCvCHl5fx3prN/OzLU8jL9sdPnHMuVjJrFvsBS8xsqZnVAA8Cx7Y4xoDG9p4C\nYFVMeZ6kDCAXqAE2JSvQ1RWV3PzUB8zcZScO221Esj7GOef6rGQmi9HAipjtsrAs1hXAdySVAU8A\nZ4XlDwNbCebT+AS4wczWt/wASadLmidp3tq1a7sc6FWPv0u9GVcesxuSd2o751xLUXdwnwj8wcyK\ngSOB+ySlEdRK6oFRwHjgfEkTWp4cjoBbamalw4YN61IAH63dwpPvfspZMyczZrAPFOicc61JZuP8\nSiD2QYXisCzW94DDAcxsrqQcYCjwLeBfZlYLfCbpJaAUWNrdQU4cNpAnzv4vxg/N6+63ds65fiOZ\nNYvXgcmSxkvKIujAfqzFMZ8AhwBI2hXIAdaG5TPD8jzgAII5wZNi5xGDyMqIupLlnHO9V9KukGZW\nB5wJzAEWE9z1tEjSVZKOCQ87HzhN0lvAn4FTzMwI7qIaKGkRQdL5vZm9naxYnXPOtU/Btbnvk7QW\nWJ7AWwwFyrspnL7Of4vm/Pdozn+P7frDbzHOzDrs9O03ySJRkuaZWWnUcfQG/ls0579Hc/57bJdK\nv4U31DvnnOuQJwvnnHMd8mSx3Z1RB9CL+G/RnP8ezfnvsV3K/BbeZ+Gcc65DXrNwzjnXIU8Wzjnn\nOpTyyaKjOTdSiaQx4fwi70paJOmcqGOKmqT0cL6Vv0cdS9QkFUp6WNJ74Twz06OOKUqSzgv/nSyU\n9OdwuKJ+K6WTRZxzbqSSOuB8M5tCMMTKj1L89wA4h2AEAge3EozZtguwFyn8u0gaDZwNlJrZ7kA6\nwZBG/VZKJwvim3MjZZjZajN7I1zfTHAxaDmsfMqQVAwcBdwVdSxRk1QAHATcDWBmNWa2MdqoIpcB\n5Ibz7gxg+3w8/VKqJ4t45txISeHMhHsDr0YbSaRuAS4EGqIOpBcYTzDI5+/DZrm7wkE+U5KZrQRu\nIBj0dDVQYWZPRhtVcqV6snCtkDQQeAQ418ySNkNhbybpaOAzM5sfdSy9RAYwDbgjnAZ5K5CyfXyS\nighaIcYTzLuTJ+k70UaVXKmeLOKZcyOlSMokSBQPmNlfo44nQgcCx0haRtA8OVPS/dGGFKkyoMzM\nGmuaDxMkj1R1KPCxma0N5935KzAj4piSKtWTRTxzbqQMBXPK3g0sNrOboo4nSmZ2iZkVm1kJwf8X\nz5hZv/7LsT1mtgZYIWnnsOgQ4N0IQ4raJ8ABkgaE/24OoZ93+Cdzprxez8zqJDXOuZEO3GNmiyIO\nK0oHAt8F3pH0Zlj2EzN7IsKYXO9xFvBA+IfVUuDUiOOJjJm9Kulh4A2CuwgX0M+H/vDhPpxzznUo\n1ZuhnHPOxcGThXPOuQ55snDOOdchTxbOOec65MnCOedchzxZONcJkuolvRmzdNtTzJJKJC3srvdz\nrjul9HMWznVBpZlNjToI53qa1yyc6waSlkm6TtI7kl6TNCksL5H0jKS3JT0taWxYPlzSo5LeCpfG\noSLSJf0unCfhSUm5kX0p52J4snCuc3JbNEMdH7Ovwsz2AH5NMGItwG3AvWa2J/AA8Kuw/FfA82a2\nF8EYS40jB0wGbjez3YCNwNeS/H2ci4s/we1cJ0jaYmYDWylfBsw0s6XhYIxrzGyIpHJgpJnVhuWr\nzWyopLVAsZlVx7xHCfCUmU0Oty8CMs3s6uR/M+fa5zUL57qPtbHeGdUx6/V4v6LrJTxZONd9jo95\nnRuuv8z26Ta/DfwnXH8a+CE0zfNd0FNBOtcV/leLc52TGzMibxawK5AZbhdJepugdnBiWHYWwexy\nPwcGAXuG5ecAd0r6HkEN4ocEM6451yt5n4VLCWGfwihglJmVx5QvAKYC481sWSffswT4mCBZLAFK\nY9+7xbGnAN83s8+39z5mVteZGJzrKd4M5VLJx2z/ix9JewADogvHub7Dk4VLJfcBJ8Vsnwz8MfYA\nSQWS/ihpraTlki6VlBbuS5d0g6RySUuBoxrPC2fUq5V0t6TVklZKulpSeiIBS8qWdIukVeFyi6Ts\ncN9QSX+XtFHSekn/iYn1ojCGzZLel3RIInE458nCpZJXgHxJu4YX8ROAlvNq3wYUABOAgwmSS+OM\ncKcBRwN7A6XA11uc+weCWdMmhcfMAr6fYMw/BQ4gaCrbC9gPuDTcdz7B3NjDgOHATwALpz49E9jX\nzAYBhwHLEozDpThPFi7VNNYuvkQwZ/LKxh0xCeQSM9sc9mHcSDDVLMA3gVvMbIWZrQd+EXPucOBI\n4Fwz22pmnwE3s/1OqK76NnCVmX1mZmuBK2PiqQVGAuPMrNbM/mNBJ2Q9kA1MkZRpZsvM7KME43Ap\nzpOFSzX3Ad8CTqFFExQwlKCzenlM2XJgdLg+CljRYl+jceG5q8NmoY3A/wA7JRjvqFbiGRWuX0/Q\nsf6kpKWNgxqa2RLgXOAK4DNJD0oahXMJ8GThUoqZLSfo6D4S+GuL3eUEf62Piykby/bax2pgTIt9\njVYQ3DI71MwKwyU/HLYjEataiWdV+F02m9n5ZjYBOAaY3dg3YWZ/Cu+8GkfwgOC1CcbhUpwnC5eK\nvkcwNMfW2EIzqwceAq6RNEjSOGA22/s1HgLOllQsqQi4OObc1cCTwI2S8iWlSZoo6eBOxJUtKSdm\nSQP+DFwqaZikocDljfFIOlrSJEkCKgianxok7SxpZtgRXgVUAg2d/I2ca8aThUs5ZvaRmc1rY/dZ\nwFZgKfAi8CfgnnDf74A5wFvAG+xYMzmJ4EG9d4ENwMMEfQrx2kJwYW9cZgJXA/OAt4F3ws9tHCtq\nMvDv8Ly5wG/M7FmC/opfEtSU1hA0hV3SiTic24E/lOecc65DXrNwzjnXIU8WzjnnOpRQsghnBsuX\nlBnOArZW0ne6KzjnnHO9Q6I1i1lmtongqdZlBE+u/jjRoJxzzvUuiQ5R3nj+UcBfzKwiuIuv5w0d\nOtRKSkoi+WznnOur5s+fX25mwzo6LtFk8XdJ7xHc5vdDScMI7uvucSUlJcyb19bdkM4551ojaXnH\nRyXYDGVmFwMzCMbxryW4P/3YRN4zChsqN9Bg/sySc861JdEO7m8AtWZWL+lSgidL+9QYNB+u+5BJ\nt03ivrfuizoU55zrtRLt4L7MzDZL+jxwKHA3cEfiYfWciYMnMmnwJH7yzE/YWrO14xOccy4FJZos\n6sPXo4A7zewfBMMd9BlpSuPmw25m1eZVXPfSdVGH45xzvVKiyWKlpP8BjgeeCAcu63MP+s0YM4Pj\ndzue61++nrJNZVGH45xzvU6iF/ZvEgysdpiZbQQG00efs7j20GtpsAYuedrHW3POuZYSvRtqG/AR\ncJikM4GdzOzJbomsh40rHMfs6bO5/+37eW3la1GH45xzvUqid0OdAzxAMATyTsD9ks7qjsCicMnn\nL2F43nDOm3MePhqvc85tl2gz1PeA/c3scjO7nGBi+dMSDysag7IHcfXMq3l5xcs8tOihqMNxzrle\nI9FkIbbfEUW4Hs14H93k1Kmnstfwvbjo3xdRVRfJw+jOOdfrJJosfg+8KukKSVcAr7B9VrE+KT0t\nnZsOu4nlFcu5ee7NUYfjnHO9QqId3DcBpwLrw+VUM+vzV9iZ42dyzM7H8PMXf86aLWuiDsc55yKX\n8DMRZvaGmf0qXBZI+qQ7AotwkVkIAAATs0lEQVTa9V+6nqq6Ki575rKoQ3HOucgl4wG6Pt1n0ehz\nQz7Hmfueyd0L7uatNW9FHY5zzkUqGcmi39xzevnBl1OUW8TsJ2f7rbTOuZTWpfksJM1uaxcwsOvh\n9C5FuUVc+YUrOeufZ/H4B49zzM7HRB2Sc85Foqs1i0FtLAOBW7sntN7hjH3OYJehu3DBkxdQU18T\ndTjOOReJLtUszOzK7g6kt8pMz+TGWTdy1J+O4jev/4ZzDzg36pCcc67H9bkRYqNwxKQjmDVxFlc+\nfyXrtq2LOhznnOtxniziIIkbZ93IpupNXPHcFVGH45xzPS7RgQTTuyuQ3m73nXbn9Gmnc8e8O1i8\ndnHU4TjnXI9KtGbxoaTrJU3plmh6uau+eBV5WXlc8NQFUYfinHM9KtFksRfwAXCXpFcknS4pvxvi\n6pWG5Q3j0v+6lCc+fIInP+qT03Y451yXJDo21GYz+52ZzQAuAn4GrJZ0r6RJ3RJhL3P2/mczoWgC\ns+fMpq6hLupwnHOuRyTcZyHpGEmPArcANwITgMeBJ9o573BJ70taIunido77miSTVJpInN0pOyOb\n6w69jkVrF3HXG3dFHY5zzvWIhPssgGOB681sbzO7ycw+NbOHgX+1dkLYKX47cAQwBTixtT4PSYOA\nc4BXE4yx2311169y0LiDuPzZy6moqog6HOecS7pEk8WeZvY9M3u55Q4zO7uNc/YDlpjZUjOrAR4k\nSDgt/TdwLdDrZiCSxE2zbqJ8WznX/OeaqMNxzrmkSzRZ7CTpcUnlkj6T9L+SJnRwzmhgRcx2WVjW\nRNI0YIyZ/aO9Nwo71OdJmrd27doufYGu2mfUPpw89WRuffVWlm5Y2qOf7ZxzPS3RZPEn4CFgBDAK\n+Avw50TeUFIacBNwfkfHmtmdZlZqZqXDhg1L5GO75JqZ15CZlsmFT13Y45/tnHM9KdFkMcDM7jOz\nunC5H8jp4JyVwJiY7eKwrNEgYHfgOUnLgAOAx3pTJ3ejUYNGcdGBF/HI4kd4YfkLUYfjnHNJk2iy\n+KekiyWVSBon6ULgCUmDJQ1u45zXgcmSxkvKAk4AHmvcaWYVZjbUzErMrIRgXu9jzGxegrEmxfkz\nzqc4v5jZc2bTYA1Rh+Occ0mRaLL4JnAG8CzwHPBDgov/fKDVi7uZ1QFnAnOAxcBDZrZI0lWS+tyE\nEQMyB/DLQ37J/NXzue+t+6IOxznnkkL9ZQa40tJSmzcvmspHgzUw/e7plG0q4/0z32dgVr+Z/8k5\n189Jmm9mHTbzJ/pQXqaksyU9HC5nSspM5D37ojSlcfNhN7Nq8yque+m6qMNxzrlul2gz1B3APsBv\nwmWfsCzlzBgzg+N3O54bXr6BFRUrOj7BOef6kESTxb5mdrKZPRMupwL7dkdgfdG1h15LgzVwydOX\nRB2Kc851q0STRb2kiY0b4QN59Qm+Z581rnAcs6fP5oF3HuC1la9FHY5zznWbRJPFj4FnJT0n6Xng\nGeJ4mK4/u+TzlzA8bzjnzTmP/nLzgHPOdTlZhE9aVwKTgbOBs4CdzezZboqtTxqUPYirZ17Nyyte\n5qFFD0UdjnPOdYsuJwszawBuN7NqM3s7XKq7MbY+69Spp7LX8L246N8XUVXX68ZBdM65Tku0Gerp\ncM4JdUs0/UR6Wjo3HXYTyyuWc/Pcm6MOxznnEpZosjiDYPDAakmbJG2WtKkb4urzZo6fybE7H8vP\nX/w5a7asiToc55xLSKLTqg4yszQzyzKz/HC7387B3VnXf+l6quuqueyZy6IOxTnnEpLoE9xPx1OW\nqiYPmcyZ+53J3Qvu5q01b0UdjnPOdVmXkoWknHBU2aGSihpHmZVUQouJjFLdZQddxuDcwcx+crbf\nSuuc67O6WrM4g2Bk2V3C18blf4Ffd09o/UNRbhFXfOEKnvn4GR7/4PGow3HOuS5JaNRZSWeZ2W3d\nGE+XRTnqbEdq62vZ87d7Ut9Qz8L/u5Cs9KyoQ3LOOaCHRp01s9skzZD0LUknNS6JvGd/lJmeyY2z\nbuTD9R9y+2u3Rx2Oc851WqId3PcBNwCfJxhAcF+g101/2hscMekIZk2cxVUvXMW6beuiDsc55zol\nI8HzS4Ep5j23HZLEjbNuZK/f7sUVz13BbUf2itY755yLS6IP5S0ERnRHIKlg95125/Rpp3PHvDtY\nvHZx1OE451zcEk0WQ4F3Jc2R9Fjj0h2B9VdXffEq8rLyuOCpC6IOxTnn4pZoM9QV3RFEKhmWN4xL\n/+tSLvz3hTz50ZPMmjgr6pCcc65DXX0obxcAM3seeMXMnm9cAB95tgNn7382E4omMHvObOoa6qIO\nxznnOtTVZqg/xazPbbHvN118z5SRnZHNdYdex6K1i7jrjbuiDsc55zrU1WShNtZb23at+OquX+Wg\ncQdx+bOXU1FVEXU4zjnXrq4mC2tjvbVt1wpJ3HzYzZRvK+ea/1wTdTjOOdeurnZwF0v6FUEtonGd\ncNsHEozTtJHTOHnqydz66q38oPQHTCiaEHVIzjnXqq7WLH5MMHDgvJj1xu0Luye01HDNzGvITMvk\nwqf8Z3PO9V5dqlmY2b3dHUiqGjVoFBcdeBGXP3c5Lyx/gYPGHRR1SM45t4NEH8rrEkmHS3pf0hJJ\nF7ey/weS3pH0pqQXJU2JIs6ecv6M8ynOL+a8Oef5FKzOuV4poSHKu/SBUjrwAfAloAx4HTjRzN6N\nOSbfzDaF68cA/9fMDm/vfXvzEOXxeHDhg5z4yIkAlBSWcEDxAUwvns4BxQcwdcRUH9bcOZcU8Q5R\nnugT3F2xH7DEzJYCSHoQOBZoShaNiSKURwrcYXXC7icwefBkXlj+AnPL5vLiJy/y4MIHAcjJyGGf\nkfs0SyCj8/0+Audcz0l08qPrgKuBSuBfwJ7AeWZ2fzvnfB043My+H25/F9jfzM5scdyPgNlAFjDT\nzD5s5b1OB04HGDt27D7Lly/v8nfpjco2lfFK2Su8UvYKc8vmMn/VfKrrgwfkx+SPYfqY6Rww+gCm\nj5nO3iP2JjsjO+KInXN9Tbw1i0STxZtmNlXSccDRBBf3F8xsr3bOiStZxBz/LeAwMzu5vVj6ejNU\nPGrqa3hzzZvMXTGXV1a+wtwVc1leESTIrPQspo2c1pQ8phdPpzi/GMmfkXTOta2nmqEazz8K+IuZ\nVcRxcVoJjInZLg7L2vIgcEeXI+xHstKz2G/0fuw3ej/O4RwAVm9e3VTzeKXsFX47/7fc8uotQHCn\nVWOz1fTi6UwbOY3czNwov4Jzro9KNFn8XdJ7BM1QP5Q0DKjq4JzXgcmSxhMkiROAb8UeIGlyTLPT\nUcAOTVAuMHLQSI7b9TiO2/U4IJjv+61P32qWQB5Z/AgAmWmZTB0xdXsCGTOdcQXjvPbhnOtQwndD\nSRoMVJhZvaQBQL6ZtXv/p6QjgVuAdOAeM7tG0lXAPDN7TNKtwKFALbABONPMFrX3nqnQDNVVn275\nlFdXvsrcFXOZWzaX11e9zrbabQCMGDiiWcd56ahSBmQOiDhi51xP6ak+i28A/zKzzZIuBaYBV5vZ\nG11+0y7yZBG/uoY63vn0nabax9yyuSxZvwSAdKWz2067UVJYwpj8MYzJH8PYgrGMKQjWRw0aRWZ6\nZsTfwDnXXXoqWbxtZntK+jzBXVHXA5eb2f5dftMu8mSRmPJt5U13Xi1Ys4AVFSv4pOITKqqbj4ib\npjRGDhzJmIIwieRvf20sGzZgmDdtOddH9FSyWGBme0v6BfCOmf2psazLb9pFniySY3P1ZlZsChLH\niooV29c3rWhKKI238zbKTs+mOL+4WY2kZULJz86P6Bs552L11N1QKyX9D8HT2NdKyiaiIURccgzK\nHsSUYVOYMqz1EVfMjPJt5W0mlGc/fpZVm1dRb/XNzsvPzm+WRFomluL8Yn9uxLleJNFk8U3gcOAG\nM9soaSTBKLQuRUhiWN4whuUNY9rIaa0eU9dQx+rNq5vVRlZs2p5U5q2ax9pta3c4b0juEIpyixic\nO5iinOA1dr1xX8uynIycZH9t51JOQsnCzLZJ+gg4TNJhwH/M7MnuCc31FxlpGUGtoWBM8ydsYlTW\nVlK2qaxZQlmzZQ0bqjawvnI9G6o28NGGj9hQuYENVRtosIY2Py83I7f1RNJBkinILiA9LT1Jv4Jz\nfVtCyULSOcBpwF/Dovsl3WlmtyUcmUspuZm5TB4ymclDJnd4bIM1sKl6ExsqtyeS9ZXrg/UWZRuq\nNvDxxo+Zv3o+Gyo3sLV2a5vvK0RBTsEOiaQgu4CBWQPJy8pjYNbAuJbcjFzv5Hf9SqLNUN8jGKpj\nK4Cka4G5gCcLlzRpSqMwp5DCnELGF43v1Lk19TVxJZnG9eUVy9lUvYktNVvYWrMVi3NMS6GOE0zm\njmXtHZ+Xmee3LbvIJJosBMT2XNaHZc71SlnpWQwfOJzhA4d3+twGa6CytjJIHLVb2VKzpdPLum3r\nWL5xebP3qKmviTuGzLRM8rLyyMvMa/s1M68p8bR7XIvXrPQsrw25NiWaLH4PvCrp0XD7K8DdCb6n\nc71SmtKCC2tWXre+b019DVtr2k8+m2s2s7VmK1trt25/jVlfX7meFZtWNDumsq6yU3GkK70pcbSX\nbHIzcsnJyCEnI4fczO3rORk5zfZ1tN/7h/qWRDu4b5L0HPD5sOhUM1uQcFTOpZCs9CyycrMoyi3q\n1vdtsAa21W5rlkAaazQ7JJ52ElFFVQWrNq9q2q6qq6KytpLahtqE4stIy+hUcmm5nZWe1eUlOyN7\nh7LMtEyvWbWjy8kinPFukZntAvT48B7OufalKa2pvyMZ6hvqqa6vbkoeVXVVTUtlXYvtjva3cnxF\nVQWf1n3a6v7ONN11RmZaZkJJqLuX7PQdk1pWelYktbIuJ4tw4MD3JY01s0+6MyjnXO+XnpbOgLQB\nwcCTPTzyfYM1UFtfS019TatLdX11m/u6a9lWu43q+uoO40iGNKU1Sx77jtqXJ779RFI+q1GifRZF\nwCJJrwFN9ySa2TEJvq9zzrUpTWlkZ2T3+qf8zYy6hrqkJ64xBW08wNSNEk0Wl3VLFM451w9JIjM9\nk8z0TPLo3hsjelqXkoWkScBwM3u+RfnngdXdEZhzzrneo6uD/t0CbGqlvCLc55xzrh/p0hDlkl43\ns33b2PeOme2RcGSdj2ktsDyBtxgKlHdTOH2d/xbN+e/RnP8e2/WH32KcmQ3r6KCu9lkUtrOvh++L\nCMTzZdsjaV48Y7qnAv8tmvPfozn/PbZLpd+iq81Q8ySd1rJQ0veB+YmF5Jxzrrfpas3iXOBRSd9m\ne3IoBbKA47ojMOecc71Hl5KFmX0KzJD0RWD3sPgfZvZMt0XW8+6MOoBexH+L5vz3aM5/j+1S5rdI\naA5u55xzqcHny3bOOdchTxbOOec6lPLJQtLh4YCISyRdHHU8UZI0RtKzkt6VtCicNjelSUqXtEDS\n36OOJWqSCiU9LOk9SYslTY86pihJOi/8d7JQ0p8l5UQdUzKldLIIh1m/HTgCmAKcKGlKtFFFqg44\n38ymAAcAP0rx3wPgHGBx1EH0ErcC/wqnJdiLFP5dJI0GzgZKzWx3IB04IdqokiulkwWwH7DEzJaa\nWQ3wIHBsxDFFxsxWm9kb4fpmgovB6Gijio6kYuAo4K6oY4mapALgIMKZMM2sxsw2RhtV5DKAXEkZ\nwABgVcTxJFWqJ4vRwIqY7TJS+OIYS1IJsDfwarSRROoW4EKgIepAeoHxwFrg92Gz3F2S+vYwqgkw\ns5XADcAnBIOnVpjZk9FGlVypnixcKyQNBB4BzjWz1gaM7PckHQ18ZmY+IkEgA5gG3GFmexPMX5Oy\nfXySighaIcYDo4A8Sd+JNqrkSvVksRKInTWkOCxLWZIyCRLFA2b216jjidCBwDGSlhE0T86UdH+0\nIUWqDCgzs8aa5sMEySNVHQp8bGZrzawW+CswI+KYkirVk8XrwGRJ4yVlEXRQPRZxTJFRMFv93cBi\nM7sp6niiZGaXmFmxmZUQ/H/xjJn1678c22Nma4AVknYOiw4B3o0wpKh9AhwgaUD47+YQ+nmHf6Iz\n5fVpZlYn6UxgDsHdDPeY2aKIw4rSgcB3gXckvRmW/cTMkju5r+srzgIeCP+wWgqcGnE8kTGzVyU9\nDLxBcBfhAvr50B8+3IdzzrkOpXozlHPOuTh4snDOOdchTxbOOec65MnCOedchzxZOOec65AnC+c6\nQVK9pDdjlm57illSiaSF3fV+znWnlH7OwrkuqDSzqVEH4VxP85qFc91A0jJJ10l6R9JrkiaF5SWS\nnpH0tqSnJY0Ny4dLelTSW+HSOFREuqTfhfMkPCkpN7Iv5VwMTxbOdU5ui2ao42P2VZjZHsCvCUas\nBbgNuNfM9gQeAH4Vlv8KeN7M9iIYY6lx5IDJwO1mthuwEfhakr+Pc3HxJ7id6wRJW8xsYCvly4CZ\nZrY0HIxxjZkNkVQOjDSz2rB8tZkNlbQWKDaz6pj3KAGeMrPJ4fZFQKaZXZ38b+Zc+7xm4Vz3sTbW\nO6M6Zr0e71d0vYQnC+e6z/Exr3PD9ZfZPt3mt4H/hOtPAz+Epnm+C3oqSOe6wv9qca5zcmNG5IVg\nTurG22eLJL1NUDs4MSw7i2B2uR8TzDTXOFLrOcCdkr5HUIP4IcGMa871St5n4Vw3CPssSs2sPOpY\nnEsGb4ZyzjnXIa9ZOOec65DXLJxzznXIk4VzzrkOebJwzjnXIU8WzjnnOuTJwjnnXIf+P11s1FRK\nWpj7AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xMgl2FYaSA9",
        "colab_type": "text"
      },
      "source": [
        "## Evaluating Model\n",
        "\n",
        "Finally, we can evaluate our model on the test data by using the method `.evaluate()`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dac7-bRJab05",
        "colab_type": "code",
        "outputId": "651da636-8ec2-4dfe-d653-a9a991b790eb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "source": [
        "score = model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "('Test loss:', 0.26236837894022463)\n",
            "('Test accuracy:', 0.9283)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jzdmn3rCgo8l",
        "colab_type": "text"
      },
      "source": [
        "## Training A Simple Multi-layer Perceptron\n",
        "\n",
        "Previously, we showed a simple network mapping directly from $784$ to $10$. Here we show how to combine different sequential layers. We now define a more complex model with two extra layers to see how that affects the final test accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WRRqajaThGgf",
        "colab_type": "code",
        "outputId": "da60a0fa-04b5-4d98-e6d5-f81a6a1f5d09",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 215
        }
      },
      "source": [
        "model2 = Sequential()\n",
        "model2.add(Dense(100, activation='relu', input_shape=(784,)))\n",
        "model2.add(Dense(10, activation='softmax'))\n",
        "\n",
        "model2.summary()"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_16 (Dense)             (None, 100)               78500     \n",
            "_________________________________________________________________\n",
            "dense_17 (Dense)             (None, 10)                1010      \n",
            "=================================================================\n",
            "Total params: 79,510\n",
            "Trainable params: 79,510\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK4s-S-ihT2v",
        "colab_type": "text"
      },
      "source": [
        "We have increased the number of parameters from 7850 to 79510 by adding this layer. Remember that the number of units given to the layer in Keras defines the dimensionality of the output space, that is why we need the last layer to have the same number of digit classes. \n",
        "\n",
        "In addition, we use ReLU (Rectified Linear Unit) activation function in the first dense layer. You can learn more about it [here](https://keras.io/activations/#relu).\n",
        "\n",
        "As done above, we can train our new model and visualise the model curves:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ni6ov50rkltw",
        "colab_type": "code",
        "outputId": "074b44ed-5b92-4a12-8578-a971097a211c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 673
        }
      },
      "source": [
        "model2.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "history2 = model2.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=10)\n",
        "\n",
        "fig, axes = plt.subplots(2,1)\n",
        "axes[0].plot(history2.history['acc'])\n",
        "axes[0].set_title('Model Accuracy')\n",
        "axes[0].set_ylabel('Accuracy')\n",
        "axes[0].set_xlabel('Epoch')\n",
        "\n",
        "fig.subplots_adjust(hspace=0.5)\n",
        "\n",
        "axes[1].plot(history2.history['loss'], 'g')\n",
        "axes[1].set_title('Model Loss')\n",
        "axes[1].set_ylabel('Cross Entropy Loss')\n",
        "axes[1].set_xlabel('Epoch')"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "60000/60000 [==============================] - 6s 96us/step - loss: 0.2722 - acc: 0.9215\n",
            "Epoch 2/10\n",
            "60000/60000 [==============================] - 5s 90us/step - loss: 0.1243 - acc: 0.9636\n",
            "Epoch 3/10\n",
            "60000/60000 [==============================] - 5s 89us/step - loss: 0.0870 - acc: 0.9736\n",
            "Epoch 4/10\n",
            "60000/60000 [==============================] - 5s 89us/step - loss: 0.0655 - acc: 0.9791\n",
            "Epoch 5/10\n",
            "60000/60000 [==============================] - 5s 89us/step - loss: 0.0507 - acc: 0.9843\n",
            "Epoch 6/10\n",
            "60000/60000 [==============================] - 5s 90us/step - loss: 0.0423 - acc: 0.9871\n",
            "Epoch 7/10\n",
            "60000/60000 [==============================] - 5s 91us/step - loss: 0.0338 - acc: 0.9894\n",
            "Epoch 8/10\n",
            "60000/60000 [==============================] - 5s 89us/step - loss: 0.0279 - acc: 0.9913\n",
            "Epoch 9/10\n",
            "60000/60000 [==============================] - 5s 90us/step - loss: 0.0234 - acc: 0.9932\n",
            "Epoch 10/10\n",
            "60000/60000 [==============================] - 5s 91us/step - loss: 0.0201 - acc: 0.9936\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5,0,'Epoch')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAIABJREFUeJzt3Xl8FuW9///XOysJSSAQlrCGTRBR\nVHC3Yt2K2mqttoJbtXav27G21R7b0+PXLqfa1lo9/opWW5dKLVXrcbdqaxe1griwiQgiWyDsWSDr\n5/fHTMKdkJCb3LkzSe7P8/GYR2auWe5PBjKfe65r5rpkZjjnnHP7khZ1AM4557o/TxbOOefa5cnC\nOedcuzxZOOeca5cnC+ecc+3yZOGcc65dnixcSpNUIskkZcSx7aWS/tEVcTnX3XiycD2GpA8l1Ugq\nalG+MLzgl0QTWbNY8iRVSHom6lic60yeLFxPswqY3bgg6WAgN7pw9nIuUA2cKmloV35wPHdHznWU\nJwvX0zwAXBKz/Hng/tgNJPWTdL+kMkmrJd0oKS1cly7pVkmbJa0Ezmxl399I2iBpnaSbJaXvR3yf\nB/4/4B3gohbHHinp0TCuLZLuiFn3JUlLJZVLWiLp8LDcJI2P2e63km4O50+UtFbSdySVAvdJKpT0\nZPgZ28L5ETH7D5B0n6T14frHw/JFkj4Vs11meI4O24/f3fVinixcT/MaUCDpwPAiPgt4sMU2vwL6\nAWOBGQTJ5bJw3ZeATwKHAdOB81rs+1ugDhgfbnMa8MV4ApM0GjgReCicLolZlw48CawGSoDhwNxw\n3WeBH4TbFwBnAVvi+UxgKDAAGA18meBv+r5weRSwC7gjZvsHCO7EDgIGA78Iy++neXI7A9hgZgvj\njMP1dmbmk089YgI+BE4BbgR+DMwEXgAyACO4CKcDNcDkmP2+Avw1nH8J+GrMutPCfTOAIQRVSDkx\n62cDL4fzlwL/2Ed8NwJvhfPDgXrgsHD5GKAMyGhlv+eAq9s4pgHjY5Z/C9wczp8Y/q599hHTocC2\ncL4YaAAKW9luGFAOFITL84BvR/1v7lP3mbyO0/VEDwCvAGNoUQUFFAGZBN/gG60muHhDcFFc02Jd\no9HhvhskNZaltdh+Xy4B7gYws3WS/kZQLbUQGAmsNrO6VvYbCXwQ52e0VGZmuxsXJOUS3C3MBArD\n4vzwzmYksNXMtrU8iJmtl/RP4FxJjwGnA1d3MCbXC3k1lOtxzGw1QUP3GcCjLVZvBmoJLvyNRgHr\nwvkNBBfN2HWN1hDcWRSZWf9wKjCzg9qLSdKxwATgBkmlYRvCUcAFYcPzGmBUG43Qa4BxbRy6iuYN\n+C0bzVt2G/1NYCJwlJkVACc0hhh+zgBJ/dv4rN8RVEV9FnjVzNa1sZ1LQZ4sXE91OXCSmVXGFppZ\nPfAI8ENJ+WE7wrXsadd4BLhK0ghJhcD1MftuAJ4HfiapQFKapHGSZsQRz+cJqsQmE1T9HApMAXII\nvqX/myBR/URSX0l9JB0X7nsPcJ2kaQqMD+MGeIsg4aRLmknQBrMv+QTtFNslDQD+q8Xv9wzwv2FD\neKakE2L2fRw4nOCOouUdm0txnixcj2RmH5jZ/DZWXwlUAiuBfwC/B+4N191N0EbwNvAme9+ZXAJk\nAUuAbQR198X7ikVSH+BzwK/MrDRmWkVQZfb5MIl9iqDh/CNgLXB++Lv8EfhhGGc5wUV7QHj4q8P9\ntgMXhuv25TaCBLWZ4GGAZ1usv5jgzmsZsAm4pnGFme0C/kRQvdfyvLgUJzMf/Mg5F5D0feAAM7uo\n3Y1dSvEGbuccELyDQVC9d3HUsbjux6uhnHNI+hJBA/gzZvZK1PG47seroZxzzrXL7yycc861q9e0\nWRQVFVlJSUnUYTjnXI+yYMGCzWY2qL3tek2yKCkpYf78tp6kdM451xpJq9vfyquhnHPOxaHX3Fk4\n51xXMzPqG4y6BqMhnG+azGhogLqGBhoaoN6M+oYG6htotk3sPg0WHqth38dttn29UZSfzVlThyX1\nd/Vk4ZxLWfUNRvnuWnbuqmPHrlp27q5lZ9PPOnburg3Kd9Wyc3dds3U7dtWyq7Y+6l8BgKkj+3uy\ncM65tjQ0GOXVdXtd4HfuCi/yu1tf13jxr6hurRPgPdIEBTmZFPTJpF9OJgU5GYzLz6OgTzCfm5VB\nRppITxfpEulpIk0iIz34mZ4WTrHzaXvWZaSJtL3WQ3paGukSaWns2a7FMWL3yUxPfouCJwvnXGRi\nv9nv3N38ol7eyoW+vMVFv7y6jvZeFcvvkxFe3DMp6JPBqAG5TQmgICcjJhEE6wvC+X45mfTNSiem\nu/qU5snCOddhNXUNwQU85sLe8iLfcn1sWXvf7AHyszOCC354gR/Wvw8T++QHF/iY8oKcjOZ3AX0y\nyeuTQXqaX+w7gycL51KcmVFVU8/2XbVsq6xhe1Ut23fVsK2qlu2VNUF5VVheVdPswr+7tmGfx04T\n5Md8gy/ok8nogc2/2ef3aX7Rz++T4Rf7bsiThXO9SE1dA9urapou/NuqatkRXvi3VdWwI/y5Lbzw\nBwmglpr6ti/6fbPS6Z+bRf/cTPrnZjK0X5+map387MZqm4zwQh+TGLwap1fxZOFcN1Vb38CWihrK\nyqvZXFHN1vBb/vaqmqYLfuPFv/Fbf2VN20/nZKWn0T83k8Lwwj+mqC+FuVn0C8sKczPplxP8LOyb\nRf+cTPrlZpKdkd6Fv7XrrjxZONeFausb2FoZJICyiuqmRLC5vIayimo2h8tlFdVsr6pt9RhpYs83\n/ZxMhhb0YdLQgjARZDatK2y6GwgSQE6mf8t3HefJwrkE1YUJYFPjhb68ms0VNTHze8q3tZEA8rIz\nKMrLoigvm/GD8zh67EAG5WdTlJcdlOdnM7BvFv1zs8jPziDN6/FdF/Nk4Vwb6uob+HBLJeu27465\nAwi+9cfeDWyrqmn18c3crPSmC/7YojyOHDOAorzsmCSQzeBwPifLq3pc9+bJwqW8hgZjzbYq3ist\n5/1NFbxXWs7yjeWsLKvcq+E3J7MxAWQxemAu00sKgwt/fjaD8rIZlJ/FoLw+FOVnkZvlf16u9/D/\nzS5lmBnrd+xm+cZylpeWs3xjBcs3lvP+pvJmj4AO75/DxKH5zJg4iIlD8hk1ILfpjqBvtv/JuNSU\n1P/5kmYCvwTSgXvM7Cct1o8G7gUGAVuBi8xsbbjup8CZBD3jvgBcbT6sn4uDmVFWUc3y0iAZNE7v\nb6ygPOYlsCEF2RwwJJ8LjxrNAUPyOGBIPhOG5JPnCcG5vSTtr0JSOnAncCqwFnhD0hNmtiRms1uB\n+83sd5JOAn4MXCzpWOA44JBwu38AM4C/Jite1zNtq6yJSQgVvLexnPc3ljdrSC7MzWTi0HzOOXw4\nBwzJD6c8+udmRRi5cz1LMr9CHQmsMLOVAJLmAmcDscliMnBtOP8y8Hg4b0AfIAsQkAlsTGKsrpsr\n313L+5sqWF5aHiaEIDGUlVc3bZOfncEBQ/OZOaWYA4bkMTG8UyjKy/JHRp1LUDKTxXBgTczyWuCo\nFtu8DXyGoKrqHCBf0kAze1XSy8AGgmRxh5ktbfkBkr4MfBlg1KhRnf8buC5XXVfPB5sqWVa6s6mh\nefnGCtZt39W0TU5mOhOG5DHjgEFhQshj4tB8hhb08aTgXJJEXTl7HXCHpEuBV4B1QL2k8cCBwIhw\nuxckfczM/h67s5nNAeYATJ8+3dszehAzY8OO3bxXWs7S0p0s21DOstKdrCyrpK4h+KfMSk9j3OA8\nppcUcsGQUUwMq5BGFOb4ewbOdbFkJot1wMiY5RFhWRMzW09wZ4GkPOBcM9su6UvAa2ZWEa57BjgG\naJYsXM9QWV3HexvLea+0nGUbdrI0/Llz957G5uH9cziwOJ/TJg9l4tB8DizOp2RgXzK6oJ9+51z7\nkpks3gAmSBpDkCRmARfEbiCpCNhqZg3ADQRPRgF8BHxJ0o8JqqFmALclMVbXCeobjI+2VrFsw06W\nlQZ3CstKy1m9pappm7zsDCYOzedTU4cxqbiASUPzmTg0n4I+mRFG7pxrT9KShZnVSboCeI7g0dl7\nzWyxpJuA+Wb2BHAi8GNJRlAN9Y1w93nAScC7BI3dz5rZ/yUrVrf/tlXWNCWEoCopeHehcZjJNEFJ\nUV+mDOvHeYePaEoMw/t7FZJzPZHae3VB0pXAg2a2rWtC6pjp06fb/Pnzow6j16mpa2Dl5gqWbQja\nFoKqpHJKd+5u2mZA3ywmDc1n0tACJhXnc+DQAiYMyaNPpndh4Vx3J2mBmU1vb7t47iyGELwj8SZB\nNdFz/nJc72RmLF6/k3+u2Myy0nKWbtjJB2UV1NbvaXAePziPY8cNZFLxnuQwKC/bn0JyrpdrN1mY\n2Y2SvgecBlxG8PTSI8BvzOyDZAfokquhwVi4ZhvPLirl2cWlrNkaPKI6rF8fJhUX8PFJg5k0NJ8D\niwsYU9S3SwaGd851P3G1WZiZSSoFSoE6oBCYJ+kFM/t2MgN0na+uvoF/r9rKM4tKeW5xKZvKq8lM\nF8ePL+LKj0/gpAMHU5SXHXWYzrlupN1kIelq4BJgM3AP8C0zq5WUBrwPeLLoAarr6vnXii08s2gD\nLyzZyLaqWvpkpnHiAYM5/eChfHzSYH8iyTnXpnjuLAYAnzGz1bGFZtYg6ZPJCct1hqqaOl5ZXsYz\ni0p5aekmyqvryM/O4KQDB3P6lKHMOGCwj6PgnItLPMniGYIeYQGQVAAcaGavt9YFh4vWzt21vLR0\nE88uKuWvyzexu7aBwtxMzji4mJlThnLs+IE+prJzbr/FkyzuAg6PWa5opcxFaGtlDS8sKeXZRaX8\nc8UWauobGJyfzeemj2TmQUM5cswAfxPaOZeQeJKFYh+VDaufou5TKuVt3Lmb5xaX8sy7pby+agsN\nBiMKc/j8saOZOWUoh40s9JffnHOdJp6L/kpJVxHcTQB8HViZvJBcW9ZsreLZRaU8s2gDb360HYBx\ng/ry9RPHM3PKUA4aVuDvOzjnkiKeZPFV4HbgRoKuN14k7BbcJd+KTeVhgihl8fqdABw0rIBvnnoA\npx88lPGD8yOO0DmXCuJ5KW8TQSeArgs0vkXd+JLcik0VABw+qj/fPWMSMw8qZtTA3IijdM6lmnje\ns+gDXA4cRDB6HQBm9oUkxpVyqmrquP3FFTz17nrWbN1FmuCoMQO5+OjRfOKgoQzt16f9gzjnXJLE\nUw31ALAM+ARwE3Ah4I/MdrIbH1/EYwvXMeOAQXzjxPGcOnkIA/0taudcNxHP85Tjzex7QKWZ/Q44\nk72HR22VpJmS3pO0QtL1rawfLelFSe9I+qukETHrRkl6XtJSSUsklcT3K/U8j765lkffXMdVJ03g\nt5cdyawjR3micM51K/Eki9rw53ZJU4B+wOD2dpKUDtwJnA5MBmZLmtxis1uB+83sEIK7lh/HrLsf\nuMXMDgSOBDbFEWuPs7KsghsfX8SRJQO48qTxUYfjnHOtiidZzJFUSPA01BPAEuB/4tjvSGCFma00\nsxpgLnB2i20mAy+F8y83rg+TSoaZvQBgZhVmVkUvU11Xz5UPLyQrI43bZh3qL84557qtfV6dws4C\nd5rZNjN7xczGmtlgM/t1HMceDqyJWV4blsV6m3AMbuAcIF/SQOAAgjuZRyUtlHRLeKfSMr4vS5ov\naX5ZWVkcIXUvP3lmGYvX7+SW86YyrH9O1OE451yb9pkswrGxk9mr7HXADEkLCcbZXgfUEzS8fyxc\nfwQwFri0lfjmmNl0M5s+aNCgJIbZ+f6yZCP3/fNDLj22hFMnD4k6HOec26d46j3+Iuk6SSMlDWic\n4thvHTAyZnlEWNbEzNab2WfM7DDgP8Oy7QR3IW+FVVh1wOP0or6oNuzYxbfmvc3k4gKuP31S1OE4\n51y74nl09vzw5zdiyozg2/6+vAFMkDSGIEnMAi6I3UBSEbA1vIO5gWDY1sZ9+0saZGZlwElArxhg\nu77BuGbuW1TXNfCrCw7zcaqdcz1CPG9wj+nIgc2sTtIVwHNAOnCvmS2WdBMw38yeAE4EfizJgFcI\nE5KZ1Uu6DnhRQWdHC4C7OxJHd3PHSyt4fdVWfvbZqYwblBd1OM45FxfFdCjb+gbSJa2Vm9n9SYmo\ng6ZPn27z53fvm4/XV25h9t2vcfahw/nF+YdGHY5zziFpgZlNb2+7eKqhjoiZ7wOcDLxJ8B6Ei9O2\nyhqunvsWowbk8v8+PSXqcJxzbr/EUw11ZeyypP4E70y4OJkZ35r3Dlsqq3ns68eRl+3DgTjnepaO\nvAVWCXSoHSNV/e5fH/KXpRu5/vQDmTK8X9ThOOfcfoun19n/I3j6CYLkMhl4JJlB9SaL1u3gR08v\n4+RJg/nCcSVRh+Occx0ST33IrTHzdcBqM1ubpHh6lcrqOq56eCGFfTO55bNTfRQ751yPFU+y+AjY\nYGa7ASTlSCoxsw+TGlkv8P0/L2bVlkp+/8WjGdA3K+pwnHOuw+Jps/gj0BCzXB+WuX14bOFa/vTm\nWq48aQLHjBsYdTjOOZeQeJJFRthrLADhvH9N3odVmyu58bGg2/GrvNtx51wvEE+yKJN0VuOCpLOB\nzckLqWcLuh1/k4x073bcOdd7xNNm8VXgIUl3hMtrgVbf6nbwP8+8x6J1O5lz8TTvdtw512vE81Le\nB8DRkvLC5YqkR9VDvbh0I/f+cxWXHlvCaQcNjToc55zrNO3WkUj6kaT+4Wh1FZIKJd3cFcH1JKU7\ndnPdH73bcedc7xRPhfrp4RgTAJjZNuCM5IXU89Q3GFfPXejdjjvneq14kkW6pOzGBUk5QPY+tm8i\naaak9yStkHR9K+tHS3pR0juS/ippRIv1BZLWxrSXdEt3vhx0O37T2VO823HnXK8UT7J4iGBcicsl\nfRF4AfhdezuFY2bfCZxO0EXIbEmTW2x2K3C/mR0C3AT8uMX6/0cwzkW39e9VW7ntL8s557DhnHt4\nyyHGnXOud2g3WZjZ/wA3AwcCEwkGMxodx7GPBFaEQ6PWEPRUe3aLbSYDL4XzL8eulzQNGAI8H8dn\nRSLodnxhU7fj3p2Hc663ivclgI0EnQl+lmCI06Vx7DMcWBOzvDYsi/U28Jlw/hwgX9JASWnAz4Dr\n9vUBkr4sab6k+WVlZXGE1HnMjG//6R02V1Tzq9mHe7fjzrlerc1kIekASf8laRnwK4I+omRmHzez\nzmpDuA6YIWkhMINgrO564OvA0+11WGhmc8xsuplNHzRoUCeFFJ/7X13NC0uCbscPHuHdjjvnerd9\nfR1eBvwd+KSZrQCQ9B/7cex1wMiY5RFhWRMzW094ZxG+x3GumW2XdAzwMUlfB/KALEkVZrZXI3kU\nFq/fwQ+fWspJ3u24cy5F7CtZfAaYBbws6VmCNof9qZR/A5ggaQxBkpgFXBC7gaQiYKuZNQA3APcC\nmNmFMdtcCkzvLomisrqOKxu7HT/vEG+ncM6lhDaroczscTObBUwiaHy+Bhgs6S5Jp7V3YDOrA64g\naBBfCjxiZosl3RTT19SJwHuSlhM0Zv8wod+mC/zXE4tZtbmS284/jIF5cT1B7JxzPZ7MrP2tGjeW\nCgkauc83s5OTFlUHTJ8+3ebPn5/Uz3h84Tqu+cNbXHXSeK49bWJSP8s557qCpAVmNr297farS1Qz\n2xY2KnerRNEVPtxcyX8+9i5HlBRy1ckTog7HOee6lPefHYfqunquCLsd/+Wsw7zbcedcyvGXA+Lw\n02e923HnXGrzr8jteGnZRn7zj1V8/pjR3u24cy5lebLYh6Db8Xc4sLiAG844MOpwnHMuMp4s2lDf\nYFzzh4Xsrq3nDu923DmX4rzNog3/+/IKXlu5lVvOO8S7HXfOpTy/s2jFv1dt5Rd/Wc6nDx3GedNG\ntL+Dc871cp4sWtheVcM1YbfjN59zsHfn4ZxzeDVUM2bGt+e9Q1lFNY9+7Tjvdtw550J+ZxHjgddW\n8/ySjXxn5iTvdtw552J4sggtWb+Tm8Nuxy8/fkzU4TjnXLfiyQKoqqnjioffpH+OdzvunHOtSWqy\nkDRT0nuSVkjaazwKSaMlvSjpHUl/lTQiLD9U0quSFofrzk9mnP/157Db8VmHerfjzjnXiqQlC0np\nwJ3A6cBkYLakyS02uxW438wOAW4CfhyWVwGXmNlBwEzgNkn9kxHnB2UVPLZwHVd+fDzHjitKxkc4\n51yPl8zHfY4EVpjZSgBJc4GzgSUx20wGrg3nXwYeBzCz5Y0bmNl6SZuAQcD2zg5y3KA8/u/K45kw\n2F+8c865tiSzGmo4sCZmeW1YFuttwjG4gXOAfEkDYzeQdCSQBXzQ8gMkfVnSfEnzy8rKOhzogcUF\n3u24c87tQ9RXyOuAGZIWAjMIxuqub1wpqRh4ALgsHKe7mXAgpulmNn3QoEFdFbNzzqWcZFZDrQNG\nxiyPCMuamNl6wjsLSXnAuWa2PVwuAJ4C/tPMXmvvwxYsWLBZ0uoE4i0CNiewf2/i56I5Px/N+fnY\nozeci9HxbJTMZPEGMEHSGIIkMQu4IHYDSUXA1vCu4Qbg3rA8C3iMoPF7XjwfZmYJ3VpImh/POLSp\nwM9Fc34+mvPzsUcqnYukVUOZWR1wBfAcsBR4xMwWS7pJ0lnhZicC70laDgwBfhiWfw44AbhU0lvh\ndGiyYnXOObdvMrOoY+gWUukbQnv8XDTn56M5Px97pNK5iLqBuzuZE3UA3Yifi+b8fDTn52OPlDkX\nfmfhnHOuXX5n4Zxzrl2eLJxzzrUr5ZNFe50dphJJIyW9LGlJ2Inj1VHHFDVJ6ZIWSnoy6liiJqm/\npHmSlklaKumYqGOKkqT/CP9OFkl6WFKfqGNKppROFnF2dphK6oBvmtlk4GjgGyl+PgCuJnj028Ev\ngWfNbBIwlRQ+L5KGA1cB081sCpBO8C5Zr5XSyYKYzg7NrAZo7OwwJZnZBjN7M5wvJ7gYtOzPK2WE\nXeafCdwTdSxRk9SP4N2n3wCYWU1jbwspLAPIkZQB5ALrI44nqVI9WcTT2WFKklQCHAa8Hm0kkboN\n+DawV79kKWgMUAbcF1bL3SOpb9RBRcXM1hEMsfARsAHYYWbPRxtVcqV6snCtCPvp+hNwjZntjDqe\nKEj6JLDJzBZEHUs3kQEcDtxlZocBlUDKtvFJKiSohRgDDAP6Sroo2qiSK9WTRbudHaYaSZkEieIh\nM3s06ngidBxwlqQPCaonT5L0YLQhRWotsNbMGu805xEkj1R1CrDKzMrMrBZ4FDg24piSKtWTRVNn\nh2HnhbOAJyKOKTIKBh//DbDUzH4edTxRMrMbzGyEmZUQ/L94ycx69TfHfTGzUmCNpIlh0ck0H8gs\n1XwEHC0pN/y7OZle3uCfzF5nuz0zq5PU2NlhOnCvmS2OOKwoHQdcDLwr6a2w7Ltm9nSEMbnu40rg\nofCL1UrgsojjiYyZvS5pHvAmwVOEC+nlXX94dx/OOefalerVUM455+LgycI551y7PFk455xrlycL\n55xz7fJk4Zxzrl2eLJzbD5LqY8aFf6szeyqWVCJpUWcdz7nOlNLvWTjXAbvM7NCog3Cuq/mdhXOd\nQNKHkn4q6V1J/5Y0PiwvkfSSpHckvShpVFg+RNJjkt4Op8auItIl3R2Ok/C8pJzIfinnYniycG7/\n5LSohjo/Zt0OMzsYuIOgx1qAXwG/M7NDgIeA28Py24G/mdlUgj6WGnsOmADcaWYHAduBc5P8+zgX\nF3+D27n9IKnCzPJaKf8QOMnMVoadMZaa2UBJm4FiM6sNyzeYWZGkMmCEmVXHHKMEeMHMJoTL3wEy\nzezm5P9mzu2b31k413msjfn9UR0zX4+3K7puwpOFc53n/Jifr4bz/2LPcJsXAn8P518EvgZN43z3\n66ognesI/9bi3P7JiemRNws4EMgMlwslvUNwdzA7LLuSYHS5HwH5wCFh+dXAHEmXE9xBfI1gxDXn\nuiVvs3ApIWxTGAYMM7PNMeULgUOBMWb24X4eswRYRZAsVgDTY4/dYttLgS+a2fH7Oo6Z1e1PDM51\nFa+GcqlkFXu+8SPpYCA3unCc6zk8WbhU8gBwSczy54H7YzeQ1E/S/ZLKJK2WdKOktHBduqRbJW2W\ntBI4s3G/cES9Wkm/kbRB0jpJN0tKTyRgSdmSbpO0Ppxuk5QdriuS9KSk7ZK2Svp7TKzfCWMol/Se\npJMTicM5TxYulbwGFEg6MLyIzwJajqv9K6AfMBaYQZBcGkeE+xLwSeAwYDpwXot9f0swatr4cJvT\ngC8mGPN/AkcTVJVNBY4EbgzXfZNgbOxBwBDgu4CFQ59eARxhZvnAJ4APE4zDpThPFi7VNN5dnEow\nZvK6xhUxCeQGMysP2zB+RjDULMDngNvMbI2ZbQV+HLPvEOAM4BozqzSzTcAv2PMkVEddCNxkZpvM\nrAz475h4aoFiYLSZ1ZrZ3y1ohKwHsoHJkjLN7EMz+yDBOFyK82ThUs0DwAXApbSoggKKCBqrV8eU\nrQaGh/PDgDUt1jUaHe67IawW2g78GhicYLzDWolnWDh/C0HD+vOSVjZ2amhmK4BrgB8AmyTNlTQM\n5xLgycKlFDNbTdDQfQbwaIvVmwm+rY+OKRvFnruPDcDIFusarSF4ZLbIzPqHU0HYbUci1rcSz/rw\ndyk3s2+a2VjgLODaxrYJM/t9+OTVaIIXBP8nwThcivNk4VLR5QRdc1TGFppZPfAI8ENJ+ZJGA9ey\np13jEeAqSSMkFQLXx+y7AXge+JmkAklpksZJmrEfcWVL6hMzpQEPAzdKGiSpCPh+YzySPilpvCQB\nOwiqnxokTZR0UtgQvhvYBTTs5zlyrhlPFi7lmNkHZja/jdVXApXASuAfwO+Be8N1dwPPAW8Db7L3\nncklBC/qLQG2AfMI2hTiVUFwYW+cTgJuBuYD7wDvhp/b2FfUBOAv4X6vAv9rZi8TtFf8hOBOqZSg\nKuyG/YjDub34S3nOOefa5XcWzjnn2uXJwjnnXLsSShbhyGAFkjLDUcDKJF3UWcE555zrHhK9szjN\nzHYSvNX6IcGbq99KNCjnnHPdS6JdlDfufybwRzPbETzF1/WKioqspKQkks92zrmeasGCBZvNbFB7\n2yWaLJ6UtIzgMb+vSRpE8Fw8eAB5AAAWrUlEQVR3lyspKWH+/LaehnTOOdcaSavb3yrBaigzux44\nlqAf/1qC59PPTuSYUdhctRl/hNg559qWaAP3Z4FaM6uXdCPBm6U9qg+a5VuWM/GOifxm4W+iDsU5\n57qtRBu4v2dm5ZKOB04BfgPclXhYXWf8gPEcXnw4Vz1zFUvKlkQdjnPOdUuJJov68OeZwBwze4qg\nu4MeI01p3P/p+8nLymPWvFnsqt0VdUjOOdftJJos1kn6NXA+8HTYcVmPe9GvOL+Y+8+5n3c3vcu3\nXvAnf51zrqVEL+yfI+hY7RNmth0YQA99z2Lm+Jl885hvcucbd/L4ssejDsc557qVRJ+GqgI+AD4h\n6QpgsJk93ymRReBHJ/+IacXT+MKfv8CaHWva38E551JEok9DXQ08RNAF8mDgQUlXdkZgUchKz2Lu\neXOpbajlwkcvpL6hvv2dnHMuBSRaDXU5cJSZfd/Mvk8wsPyXEg8rOuMHjOeuM+/i7x/9nZtfubn9\nHZxzLgUkmizEnieiCOej6e+jE110yEVcfMjF3PTKTbyy+pWow3HOucglmizuA16X9ANJPwBeY8+o\nYj3anWfcydjCsVz46IVs3bU16nCccy5SiTZw/xy4DNgaTpeZ2S86I7Co5WfnM/fcuWys2MjlT1zu\n3YE451Jawu9EmNmbZnZ7OC2U9FFnBNYdTBs2jZ+c8hMeX/Y4d83vUS+mO+dcp0rGC3Q9vs0i1jVH\nX8Pp40/n2ueu5d2N70YdjnPORSIZyaJX1dekKY3ffvq3FOYUcv6886mqrYo6JOec63IdGs9C0rVt\nrQLyOh5O9zS472AeOOcBTnvgNK559hrmfGpO1CE551yX6uidRX4bUx7wy84JrXs5ZewpfOe473D3\nm3fzx8V/jDoc55zrUuotT/lMnz7dkj1SXm19LR+772Ms27yMt776FiX9S5L6ec45l2ySFpjZ9Pa2\n63E9xEYpMz2Th899GMO44E8XUFtfG3VIzjnXJTxZ7KcxhWOY88k5vLr2Vf77b/8ddTjOOdclEu1I\nML2zAulJzp9yPl849Av86O8/4qVVL0UdjnPOJV2idxbvS7pF0uT92UnSTEnvSVoh6fpW1l8raYmk\ndyS9KGl0gnF2uttPv52JRRO56NGLKKssizoc55xLqkSTxVRgOXCPpNckfVlSwb52CO9G7gROByYD\ns1tJNguB6WZ2CDAP+GmCcXa6vll9mXvuXLbs2sJlf77MuwNxzvVqifYNVW5md5vZscB3gP8CNkj6\nnaTxbex2JLDCzFaaWQ0wFzi7xXFfDgdWgqBzwhGJxJksU4dO5dZTb+Wp95/i9tdvjzoc55xLmoTb\nLCSdJekx4DbgZ8BY4P+Ap9vYbTgQOwzd2rCsLZcDz7Tx+V+WNF/S/LKyaKqCrjjyCj51wKf49l++\nzcINCyOJwTnnki3hNguCu4JbzOwwM/u5mW00s3nAs4kGJ+kiYDpwS2vrzWyOmU03s+mDBg1K9OM6\nRBL3nn0vRblFzPrTLCpqKiKJwznnkinRZHGImV1uZv9qucLMrmpjn3XAyJjlEWFZM5JOAf4TOMvM\nqhOMM6mKcot46DMP8f6W97nymR47qqxzzrUp0WQxWNL/SdosaZOkP0sa284+bwATJI2RlAXMAp6I\n3UDSYcCvCRLFpgRj7BInlpzIjSfcyG/f+i2/f/f3UYfjnHOdKtFk8XvgEWAoMAz4I/DwvnYwszrg\nCuA5YCnwiJktlnSTpLPCzW4h6Gfqj5LekvREG4frVr4/4/scN/I4vvrkV/lg6wdRh+Occ50mob6h\nJL0TPt4aW/a2mU1NOLL91BV9Q8Vj9fbVHPrrQ5kwYAL/+MI/yErPijok55xrU1f1DfWMpOsllUga\nLenbwNOSBkgakOCxe6TR/Udzz6fu4Y31b3DjSzdGHY5zznWKDo1nEeNz4c+vtCifRTAIUnvtF73S\nuZPP5SvTvsIt/7qFU8aewmnjTos6JOecS4h3UZ4ku2p3ccTdR7C5ajNvf/VthuQNiTok55zbS5dU\nQ0nKlHSVpHnhdIWkzESO2VvkZOYw97y57KjewSWPX0KDNUQdknPOdViibRZ3AdOA/w2naWGZA6YM\nnsIvPvELnv/geX7+6s+jDsc55zos0TaLI1o8+fSSpLcTPGav8pVpX+EvK//CDS/ewIzRMzhi+BFR\nh+Scc/st0TuLeknjGhfCF/LqEzxmryKJuz91N8V5xcz+02x2Vu+MOiTnnNtviSaLbwEvS/qrpL8B\nLwHfTDys3qUwp5Dfn/t7Vm1fxdef+rp3Z+6c63E6nCwkpQG7gAnAVcCVwEQze7mTYutVjh91PD+Y\n8QMeevch7n/7/qjDcc65/dLhZGFmDcCdZlZtZu+EU7fu8C9q3/3Yd5kxegbfePobLN+yPOpwnHMu\nbolWQ70o6VxJ6pRoern0tHQe/MyDZGdkM2veLKrrPLc653qGRJPFVwg6D6yWtFNSuSRvwd2HEQUj\nuO/s+1hYupDr/7LX8OPOOdctJTqsar6ZpZlZlpkVhMv7HIPbwVkTz+KKI67gttdv46nlT0UdjnPO\ntSvRN7hfjKfM7e2W025h6pCpXPrnS1lfvj7qcJxzbp86lCwk9Ql7lS2SVNjYy6ykEvY9nrYL9cno\nw9zz5lJVW8XFj11MfYO/nuKc6746emfxFWABMCn82Tj9Gbijc0Lr/SYVTeL2mbfz0qqX+Ok/fxp1\nOM4516YOJQsz+6WZjQGuM7OxZjYmnKaamSeL/fCFw77A+Qedz/de/h6vrnk16nCcc65VCXdRLulY\noISYfqbMrMvfOutuXZTvjx27d3Dorw/FzLjrzLs4ZewpZKZ7573OueTrqi7KHwBuBY4Hjgindj/U\nNdevTz/+cN4fqKip4Izfn0Hxz4r56pNf5W8f/s27NnfOdQuJjsG9FJhs3aCzo558Z9Gouq6a5z94\nnocXPcyf3/szVbVVDMsfxvkHnc+sKbM4YtgR+PuPzrnOFO+dRaLJ4o/AVWa2ocMH6SS9IVnEqqyp\n5MnlT/Lwood5ZsUz1NTXMLZwLLMOmsXsg2czZfCUqEN0zvUCXZUsXgYOBf4NNPVdYWZndfigHdTb\nkkWs7bu389jSx5i7eC4vrnyReqtnyuApzDpoFrOmzGLcgHHtH8Q551rRVcliRmvlZva3Dh+0g3pz\nsoi1qXIT85bM4+FFD/OPj/4BwBHDjmD2lNl87qDPMbzAX3NxzsUvqclC0iQzWxbOZ8f2NivpaDN7\nrZ39ZwK/BNKBe8zsJy3WnwDcBhwCzDKzee3FlCrJItaaHWv4w+I/8PCih3lzw5sIccLoE5g1ZRbn\nTT6PotyiqEN0znVzyU4Wb5rZ4S3nW1tuZd90YDlwKrAWeAOYbWZLYrYpAQqA64AnPFm0b/mW5cxd\nNJeHFz3Mss3LyEjL4NSxpzJryiw+PenTFGR7l13Oub0l+9FZtTHf2nJLRwIrzGylmdUAc4GzYzcw\nsw/N7B3AnxuN0wEDD+D7M77Pkq8v4a2vvMV1x1zH0s1L+fzjn2fwLYM595FzmbdkHrtqd0UdqnOu\nB8pof5NWWRvzrS23NBxYE7O8FjiqI0FI+jLwZYBRo0Z15BC9jiSmDp3K1KFT+dHJP+K1ta8xd9Fc\n/rD4Dzy69FHysvL49KRPM3vKbE4de6q//Oeci0tHk8UISbcT3EU0zhMud1kLq5nNAeZAUA3VVZ/b\nU0jimJHHcMzIY/j5J37OXz/8K3MXzeVPS//Eg+88yICcAZx34HnMmjKLE0afQHpaetQhO+e6qY4m\ni2/FzLdsKGiv4WAdMDJmeURY5pIoPS2dk8eezMljT+bOM+9sevnvoXcfYs6bcyjOK+ZzB32O2VNm\nc+TwI/3lP+dcMwn3DbXfHyhlEDRwn0yQJN4ALjCzxa1s+1vgSW/gTp6q2iqeXP4kcxfN5an3n6Km\nvoYRBSM4avhRTCuexrRh05hWPI2BuQOjDtU5lwRd8p5FR0k6g+DR2HTgXjP7oaSbgPlm9oSkI4DH\ngEJgN1BqZgft65ieLBK3Y/cOHl/2OE+9/xQLNixg5baVTetG9xvdlDgOLz6cacXTGNR3UITROuc6\nQ7dOFsngyaLzbdu1jYWlC1mwfgELNgTTiq0rmtaPLBjZlEAa70IG9x0cYcTOuf3lycIlxfbd21m4\nYWFT8liwfgHvb32/af2IghHNksfhxYczNG9ohBE75/alq7r7+ClwM7ALeJbgjev/MLMHO3zQDvJk\nEZ2d1Tv3SiDLtyzHwqeoh+UPa5ZAphVPozi/OOKonXPQdcniLTM7VNI5wCeBa4FXzGxqhw/aQZ4s\nupfy6vK9qrDe2/xeUwIpziveqwprWP6wiKN2LvXEmyw6+uhsy/3PBP5oZjv8kUsHkJ+dzwmjT+CE\n0Sc0lZVXl/P2xrebJZCn33+6aYCnoXlDmxrPpxVPY+rQqYwsGOnvfzjXDSSaLJ6UtIygGuprkgYR\nPL3k3F7ys/M5ftTxHD/q+KayyppK3ip9q1kV1rMrnm1KIJlpmZT0L2H8gPGMKxzHuAHjmn6OLRxL\nn4w+Uf06zqWUzhiDewCww8zqJeUCBWZW2inR7Qevhuo9KmsqeXvj2yzetJgPtn0QTFuDnzurdzbb\ndkTBiCB5xCSS8QPGM27AOPr36R/Rb+Bcz9El1VCSPgs8GyaKG4HDCRq8uzxZuN6jb1Zfjh15LMeO\nPLZZuZmxZdcWVmxd0ZQ8GhPJ0yueprSi+X+7ATkD9k4i4XJxXrG/pe7cfki0gfsdMztE0vEESeIW\n4Ptm1qGOARPhdxauoqaCldtW7kkkWz9gxbYgsXy04yPqrb5p25yMHMYWjmXcgHGMLxzfrHprdL/R\n3sGiSxld1cDd+Nd3JjDHzJ6SdHOCx3SuQ/Ky8jhkyCEcMuSQvdbV1teyesfqpkSyYuuKpoTywgcv\nsKtuT9ft6UpndP/RTdVbIwpGUJRbtNc0MHcgGWmJ/gk51zMk+j99naRfEwxk9D+Ssun4GBnOJU1m\neibjB4xn/IDxe60zMzZUbAjuRBqTSJhIHlnyCFt3bW3zuIV9CltNJG1N/fv0J03+J+J6nkSroXKB\nmcC7Zva+pGLgYDN7vrMCjJdXQ7lkqa6rZsuuLZRVlrG5avPe067my2WVZVTXV7d6rDSlMTBnYFPy\nGNR3EEU5+04weVl53r7ikqZLqqHMrErSB8AnJH0C+HsUicK5ZMrOyGZY/rC4Xxo0M6pqq5onkKrW\nE817m9/jn1X/ZHPV5mZtKrGy0rOaJY9BuYNaTSqx5dkZ2Z15CpxL+Gmoq4EvAY+GRQ9KmmNmv0o4\nMud6KEn0zepL36y+jO4/Oq59zIwd1Ttav3MJ71Ya72AWli5kc9XmfVaP5WXl7VdyGZAzwF9+dPuU\n8NNQwDFmVhku9wVeNbO9WxiTzKuhXKqpa6hj666t7SaX2KmipqLVYwlRmFPYZmJpTC4DcwfSv09/\nCrILKMguoG9mX68i6+G66mkoseeJKMJ5/5/jXBfISMtgcN/B+9Ut/K7aXWzZtaXdBLNq+yreWP8G\nZZVl1DbUtnk8oabEkciUn5XvdzbdXKLJ4j7gdUmPhcufBn6T4DGdc0mSk5nDiMwRjCgYEdf2ZkZ5\nTXmzpLKzeuc+p+27t/PRjo+alstryuP6rL6ZffcrwfTN7EtuZm5Q5Rczn5uZS9/Mvv6uTCdLtIH7\n55L+CjR29nOZmS1MOCrnXLcg7blzGFs4tkPHaLAGKmoq2k0yrU2bKjc1ze+o3tHUZ1g8MtMyW00m\nrc5n9m2WaOLZL9WSUYeThaR0YLGZTQLe7LyQnHO9SZrSmhJOIsyMXXW7mpJHZU0llbWVVNVW7TVf\nVVtFZW1l8/lwfUVNBRsrNzbbr7Kmss2n0dqSkZZBbmYufTL6tDtlp2d37nYZ2V3+QmiHPy3sD+o9\nSaPM7KPODMo551qSRG5mLrmZuUkZfbGmvqbVxNPefHVdNbvrd7O7rvm0s3onZZVle5U3To1ju3RU\nutKbksdRI47iqQue6qQz0bpEU1MhsFjSv4HKxkIzOyvB4zrnXJfKSs8iKz2rS3orNjNqG2pbTSLV\nddVtJpi2ppH9RiY95kSTxfc6JQrnnEshkpqSU6LVc12lQ8lC0nhgiJn9rUX58cCGzgjMOedc99HR\nHs1uA3a2Ur4jXOecc64X6dAb3JLeMLMj2lj3rpkdnHBk+x9TGbA6gUMUAZs7KZyezs9Fc34+mvPz\nsUdvOBejzWxQext1tM1iXy1AOR08ZkLi+WX3RdL8eF55TwV+Lprz89Gcn489UulcdLQaar6kL7Us\nlPRFYEFiITnnnOtuOnpncQ3wmKQL2ZMcpgNZwDmdEZhzzrnuo0PJwsw2AsdK+jgwJSx+ysxe6rTI\nut6cqAPoRvxcNOfnozk/H3ukzLlIqIty55xzqcEHA3bOOdcuTxbOOefalfLJQtLMsEPEFZKujzqe\nKEkaKellSUskLQ6HzU1pktIlLZT0ZNSxRE1Sf0nzJC2TtFTSMVHHFCVJ/xH+nSyS9LCkPlHHlEwp\nnSzCbtbvBE4HJgOzJU2ONqpI1QHfNLPJwNHAN1L8fABcDSyNOohu4pfAs+GwBFNJ4fMiaThwFTDd\nzKYA6cCsaKNKrpROFsCRwAozW2lmNcBc4OyIY4qMmW0wszfD+XKCi8HwaKOKjqQRwJnAPVHHEjVJ\n/YATCEfCNLMaM9sebVSRywByJGUAucD6iONJqlRPFsOBNTHLa0nhi2MsSSXAYcDr0UYSqduAbwPx\nD8/We40ByoD7wmq5eyT1jTqoqJjZOuBW4COCzlN3mNnz0UaVXKmeLFwrJOUBfwKuMbPWOozs9SR9\nEthkZt4jQSADOBy4y8wOIxi/JmXb+CQVEtRCjAGGAX0lXRRtVMmV6sliHRA7asiIsCxlScokSBQP\nmdmjUccToeOAsyR9SFA9eZKkB6MNKVJrgbVm1ninOY8geaSqU4BVZlZmZrXAo8CxEceUVKmeLN4A\nJkgaIymLoIHqiYhjiowkEdRJLzWzn0cdT5TM7AYzG2FmJQT/L14ys179zXFfzKwUWCNpYlh0MrAk\nwpCi9hFwtKTc8O/mZHp5g3/XjvjdzZhZnaQrgOcInma418wWRxxWlI4DLgbelfRWWPZdM3s6wphc\n93El8FD4xWolcFnE8UTGzF6XNA94k+ApwoX08q4/vLsP55xz7Ur1aijnnHNx8GThnHOuXZ4snHPO\ntcuThXPOuXZ5snDOOdcuTxbO7QdJ9ZLeipk67S1mSSWSFnXW8ZzrTCn9noVzHbDLzA6NOgjnuprf\nWTjXCSR9KOmnkt6V9G9J48PyEkkvSXpH0ouSRoXlQyQ9JuntcGrsKiJd0t3hOAnPS8qJ7JdyLoYn\nC+f2T06LaqjzY9btMLODgTsIeqwF+BXwOzM7BHgIuD0svx34m5lNJehjqbHngAnAnWZ2ELAdODfJ\nv49zcfE3uJ3bD5IqzCyvlfIPgZPMbGXYGWOpmQ2UtBkoNrPasHyDmRVJKgNGmFl1zDFKgBfMbEK4\n/B0g08xuTv5v5ty++Z2Fc53H2pjfH9Ux8/V4u6LrJjxZONd5zo/5+Wo4/y/2DLd5IfD3cP5F4GvQ\nNM53v64K0rmO8G8tzu2fnJgeeSEYk7rx8dlCSe8Q3B3MDsuuJBhd7lsEI8019tR6NTBH0uUEdxBf\nIxhxzbluydssnOsEYZvFdDPbHHUsziWDV0M555xrl99ZOOeca5ffWTjnnGuXJwvnnHPt8mThnHOu\nXZ4snHPOtcuThXPOuXb9//EPdbu4x0AgAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LDcMSVr_l6C3",
        "colab_type": "text"
      },
      "source": [
        "We have now trained the two models and we can see how these extra layers improve the results in the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnk6_wuemCXO",
        "colab_type": "code",
        "outputId": "8b2a1bc9-7eb6-4036-97e9-c3ac6a4b04c5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        }
      },
      "source": [
        "print('Old model:')\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])\n",
        "\n",
        "print('New model:')\n",
        "score2 = model2.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "print('Test loss:', score2[0])\n",
        "print('Test accuracy:', score2[1])"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Old model:\n",
            "('Test loss:', 0.26236837894022463)\n",
            "('Test accuracy:', 0.9283)\n",
            "New model:\n",
            "('Test loss:', 0.08035141767178139)\n",
            "('Test accuracy:', 0.9783)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a-DHj7P7n1Eh",
        "colab_type": "text"
      },
      "source": [
        "We have improved accuracy results on test, however, there is still margin for parameter tuning. Try to improve upon this model by adding more layers and changing the dimensionality in each layer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9XyUx6ThaT47",
        "colab_type": "text"
      },
      "source": [
        "## Saving/Loading Model\n",
        "\n",
        "Some networks will require long training times (hours, days or even weeks), it is essential to know how to save models for using them afterwards without the need of starting the training each time. \n",
        "\n",
        "We can save and load the trained model in different ways.\n",
        "\n",
        "The first way is to save everyting into a single HDF5 file, which will contain:\n",
        "\n",
        "*   the architecture of the model, allowing to re-create the model\n",
        "*   the weights of the model\n",
        "*   the training configuration (loss, optimiser)\n",
        "*   the state of the optimiser, allowing to resume training exactly where you left off.\n",
        "\n",
        "[Here](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) you can find more information about the process.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0KJp2Pj2aTOc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.models import load_model\n",
        "\n",
        "model.save('my_model.h5')  # creates a HDF5 file 'my_model.h5'\n",
        "del model  # deletes the existing model\n",
        "\n",
        "# Load a compiled model identical to the previous one\n",
        "model = load_model('my_model.h5') "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vD5h5qsUblji",
        "colab_type": "text"
      },
      "source": [
        "Another way is to save the architecture and the weights independently.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dJXTYfTDcaWs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.save_weights('my_model_weights.h5') # creates a HDF5 file 'my_model_weights.h5'\n",
        "\n",
        "with open('my_model_architecture.json', 'w') as f:\n",
        "    f.write(model.to_json()) # creates a json file 'my_model_architecture.json'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wPgCrD4Leony",
        "colab_type": "text"
      },
      "source": [
        "If we want to load the model, without having to define it, we can directly use `model_from_json()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gXr0L8nqeipI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.models import model_from_json\n",
        "\n",
        "# Model reconstruction from JSON file\n",
        "with open('my_model_architecture.json', 'r') as f:\n",
        "    model = model_from_json(f.read())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NzmL07T_dh7Z",
        "colab_type": "text"
      },
      "source": [
        "If we have a model defined, we can then load the weights by using the method `load_weights(file_path)`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yBahatkweEZx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.load_weights('my_model_weights.h5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nwXWMk1vdm48",
        "colab_type": "text"
      },
      "source": [
        "If we modify the model and the new model shares some layers with the saved one, we could load the weights into a different architecture by using the layer name. This situation is common when you do fine-tuning or transfer-learning, among others."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KAGF3VbPdroU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.load_weights('my_model_weights.h5', by_name=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-6v_S7oxfUQC",
        "colab_type": "text"
      },
      "source": [
        "## Obtaining an output of an intermediate layer\n",
        "\n",
        "Sometimes we need to check how features of intermediate layers look like. This can be used for extracting features, but also for debugging purposes. That is why, we need to see how we can extract information of a trained architecture.\n",
        "\n",
        "The easiest way is to define a new model that will have as final output the given output of an specific layer. We first define the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dZa6sDFLgXoC",
        "colab_type": "code",
        "outputId": "5e1fba08-8d1a-4603-b1a3-37a85deae0c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 251
        }
      },
      "source": [
        "model = Sequential()\n",
        "model.add(Dense(64, activation='relu', input_shape=(784,), name='first_dense'))\n",
        "model.add(Dense(128, activation='relu', name='second_dense'))\n",
        "model.add(Dense(10, activation='softmax', name='final_dense'))\n",
        "\n",
        "model.summary()\n",
        "\n"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "first_dense (Dense)          (None, 64)                50240     \n",
            "_________________________________________________________________\n",
            "second_dense (Dense)         (None, 128)               8320      \n",
            "_________________________________________________________________\n",
            "final_dense (Dense)          (None, 10)                1290      \n",
            "=================================================================\n",
            "Total params: 59,850\n",
            "Trainable params: 59,850\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hoXvhyLzhKU2",
        "colab_type": "text"
      },
      "source": [
        "Now, we specify the name of the layer in which we are interested to obtain the output. In our example, it is the second dense layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Zu29bschSps",
        "colab_type": "code",
        "outputId": "d9d107b8-ae4e-42f0-c575-09ccaa10c5f3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "source": [
        "from keras.models import Model\n",
        "\n",
        "layer_name = 'second_dense'\n",
        "intermediate_layer_model = Model(inputs=model.input,\n",
        "                                 outputs=model.get_layer(layer_name).output)\n",
        "intermediate_output = intermediate_layer_model.predict(X_test_flatten)\n",
        "final_output = model.predict(X_test_flatten)\n",
        "\n",
        "print('Second dense layer output shape: {0}'.format(intermediate_output.shape))\n",
        "print('Final output shape: {0}'.format(final_output.shape))"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Second dense layer output shape: (10000, 128)\n",
            "Final output shape: (10000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFo0jHRqiJv2",
        "colab_type": "text"
      },
      "source": [
        "Alternatively, we could build a Keras function that will return the output of a certain layer given a certain input, for example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "msvYp1c9iNEX",
        "colab_type": "code",
        "outputId": "761ae7a1-f60b-4d49-e565-17b0f837465a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "source": [
        "from keras import backend as K\n",
        "\n",
        "get_second_dense_layer_output = K.function([model.layers[0].input],\n",
        "                                  [model.layers[2].output])\n",
        "\n",
        "intermediate_output = get_second_dense_layer_output([X_test_flatten])[0]\n",
        "final_output = model.predict(X_test_flatten)\n",
        "\n",
        "print('Second dense layer output shape: {0}'.format(intermediate_output.shape))\n",
        "print('Final output shape: {0}'.format(final_output.shape))"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Second dense layer output shape: (10000, 10)\n",
            "Final output shape: (10000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iV22i-GFj5np",
        "colab_type": "text"
      },
      "source": [
        "## Using a Callback Function\n",
        "\n",
        "We mentioned callback functions in this tutorial already. We now will show an example of using one of the available callbacks, the `EarlyStopping` callback. The complete list of available callbacks can be found in the documentation [here](https://keras.io/callbacks/).\n",
        "\n",
        "Controlling and managing callbacks is essential when training big models. Sometimes we want the network to train until it converges, but we are not able to determine the number of epochs it will take, therefore we need some mechanism to stop it. In those cases, the `EarlyStopping` callback is useful. Let's see how to stop the training when the validation loss is not decreasing anymore:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GNShEzC-kdKN",
        "colab_type": "code",
        "outputId": "3a7af909-26c1-4e5f-d620-2b19aa25ba16",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 377
        }
      },
      "source": [
        "from keras.callbacks import EarlyStopping\n",
        "\n",
        "# Define the metric that will determine the earlyStopping\n",
        "# patience defines the number of epochs we wait before stopping training \n",
        "# when the monitored metric does not improve\n",
        "early_stopping = EarlyStopping(monitor='val_loss', patience=2)\n",
        "\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(X_train_flatten, Y_train_class, validation_split=0.2, \n",
        "            epochs=100, batch_size=32, callbacks=[early_stopping])"
      ],
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 48000 samples, validate on 12000 samples\n",
            "Epoch 1/100\n",
            "48000/48000 [==============================] - 6s 119us/step - loss: 0.2918 - acc: 0.9149 - val_loss: 0.1580 - val_acc: 0.9513\n",
            "Epoch 2/100\n",
            "48000/48000 [==============================] - 5s 107us/step - loss: 0.1279 - acc: 0.9615 - val_loss: 0.1230 - val_acc: 0.9621\n",
            "Epoch 3/100\n",
            "48000/48000 [==============================] - 5s 108us/step - loss: 0.0922 - acc: 0.9714 - val_loss: 0.1145 - val_acc: 0.9644\n",
            "Epoch 4/100\n",
            "48000/48000 [==============================] - 5s 109us/step - loss: 0.0704 - acc: 0.9778 - val_loss: 0.1068 - val_acc: 0.9703\n",
            "Epoch 5/100\n",
            "48000/48000 [==============================] - 5s 108us/step - loss: 0.0570 - acc: 0.9817 - val_loss: 0.1004 - val_acc: 0.9724\n",
            "Epoch 6/100\n",
            "48000/48000 [==============================] - 5s 107us/step - loss: 0.0482 - acc: 0.9848 - val_loss: 0.1092 - val_acc: 0.9718\n",
            "Epoch 7/100\n",
            "48000/48000 [==============================] - 5s 108us/step - loss: 0.0386 - acc: 0.9876 - val_loss: 0.0964 - val_acc: 0.9735\n",
            "Epoch 8/100\n",
            "48000/48000 [==============================] - 5s 107us/step - loss: 0.0337 - acc: 0.9889 - val_loss: 0.1236 - val_acc: 0.9700\n",
            "Epoch 9/100\n",
            "48000/48000 [==============================] - 5s 105us/step - loss: 0.0290 - acc: 0.9906 - val_loss: 0.1096 - val_acc: 0.9738\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f3187649810>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meMroifllkIN",
        "colab_type": "text"
      },
      "source": [
        "We can combine multiple callbacks when calling the fit module, learn more about it  [here](https://keras.io/callbacks/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3w4AVBmLCDJe",
        "colab_type": "text"
      },
      "source": [
        "## Freezing Keras Layers\n",
        "\n",
        "In some cases, we may want to fix the weights of some layers, meaning we exclude them from training. This is useful in the context of fine-tuning a model, or, for instance, when using fixed embeddings for text inputs. \n",
        "\n",
        "To do so, we can pass a trainable argument (Boolean) to a layer constructor to set a layer to be non-trainable:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q6x8F5hZDLWH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "frozen_layer = Dense(32, trainable=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3s-5Bm8ODNe0",
        "colab_type": "text"
      },
      "source": [
        "Additionally, we can set the trainable property of a layer to `True` or `False` after initialization. For this to take effect, we will need to call `compile()` on our model after modifying the trainable property:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MTnjPeE7DjEs",
        "colab_type": "code",
        "outputId": "fa08ee43-75aa-4d2a-c9a2-52b2da8e5c08",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 791
        }
      },
      "source": [
        "from keras.models import clone_model\n",
        " \n",
        "model = Sequential()\n",
        "model.add(Dense(10, activation='softmax', input_shape=(784,)))\n",
        "\n",
        "# We clone model in frozen_model\n",
        "frozen_model = clone_model(model)\n",
        "frozen_model.layers[0].trainable = False\n",
        "\n",
        "# with this model the weights of the layer will be updated during training\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# in the model below, the weights of `layer` will not be updated during training\n",
        "frozen_model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.summary()\n",
        "frozen_model.summary()\n",
        "\n",
        "score_untrained = model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "score_frozen_untrained = frozen_model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "\n",
        "# Uupdate the weights of `layer`\n",
        "print('Train Models:/n')\n",
        "model.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=5)\n",
        "\n",
        "# Next operation will NOT update the weights of `layer`.\n",
        "# (which will also affect the above model since it uses the same layer instance)\n",
        "frozen_model.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=5)\n",
        "\n",
        "score = model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "score_frozen = frozen_model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "\n",
        "print('Before Training: Trainable Model Test Accuracy:', score_untrained[1])\n",
        "print('After Training:  Trainable Model Test Accuracy:', score[1])\n",
        "\n",
        "print('Before Training: Frozen Model Test Accuracy:', score_frozen_untrained[1])\n",
        "print('After Training:  Frozen Model Test Accuracy:', score_frozen[1])"
      ],
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_19 (Dense)             (None, 10)                7850      \n",
            "=================================================================\n",
            "Total params: 7,850\n",
            "Trainable params: 7,850\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_19 (Dense)             (None, 10)                7850      \n",
            "=================================================================\n",
            "Total params: 7,850\n",
            "Trainable params: 0\n",
            "Non-trainable params: 7,850\n",
            "_________________________________________________________________\n",
            "Train Models:/n\n",
            "Epoch 1/5\n",
            "60000/60000 [==============================] - 5s 87us/step - loss: 0.4699 - acc: 0.8766\n",
            "Epoch 2/5\n",
            "60000/60000 [==============================] - 5s 80us/step - loss: 0.3034 - acc: 0.9154\n",
            "Epoch 3/5\n",
            "60000/60000 [==============================] - 5s 78us/step - loss: 0.2833 - acc: 0.9211\n",
            "Epoch 4/5\n",
            "60000/60000 [==============================] - 5s 79us/step - loss: 0.2722 - acc: 0.9236\n",
            "Epoch 5/5\n",
            "60000/60000 [==============================] - 5s 80us/step - loss: 0.2662 - acc: 0.9258\n",
            "Epoch 1/5\n",
            "60000/60000 [==============================] - 3s 48us/step - loss: 2.4111 - acc: 0.0839\n",
            "Epoch 2/5\n",
            "60000/60000 [==============================] - 3s 45us/step - loss: 2.4111 - acc: 0.0839\n",
            "Epoch 3/5\n",
            "60000/60000 [==============================] - 3s 46us/step - loss: 2.4111 - acc: 0.0839\n",
            "Epoch 4/5\n",
            "60000/60000 [==============================] - 3s 46us/step - loss: 2.4111 - acc: 0.0839\n",
            "Epoch 5/5\n",
            "60000/60000 [==============================] - 3s 46us/step - loss: 2.4111 - acc: 0.0839\n",
            "('Before Training: Trainable Model Test Accuracy:', 0.1466)\n",
            "('After Training:  Trainable Model Test Accuracy:', 0.9261)\n",
            "('Before Training: Frozen Model Test Accuracy:', 0.0905)\n",
            "('After Training:  Frozen Model Test Accuracy:', 0.0905)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mLmeoTdqhmhX",
        "colab_type": "text"
      },
      "source": [
        "We can see in the model description that the frozen model has 0 trainable parameters, while it has 7,850 non-trainable parameters.\n",
        "\n",
        "After training the network, the trainable model improves the accuracy while the frozen one does not change its accuracy."
      ]
    }
  ]
}