09. Neural Networks Advanced - Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise 9 - Advanced Neural Networks\n",
    "==========\n",
    "\n",
    "There are many factors that influence how well a neural network might perform. AI practitioners tend to play around with the structure of the hidden layers, the activation functions used, and the optimisation function.\n",
    "\n",
    "In this exercise we will look at how changing these parameters impacts the accuracy performance of our network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 1\n",
    "------\n",
    "\n",
    "In this exercise we will use the same dog dataset as in exercise 8, building on what we learnt before and trying different parameters for a network to try and improve performance.\n",
    "\n",
    "Let's start by opening up our data set and setting up our train and test sets.\n",
    "\n",
    "#### __Run the code__ below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this!\n",
    "\n",
    "# Here we set a randomisation seed for replicatability.\n",
    "import os\n",
    "os.environ['PYTHONHASHSEED'] = '0'\n",
    "seed = 6\n",
    "import random as rn\n",
    "rn.seed(seed)\n",
    "import numpy as np\n",
    "np.random.seed(seed)\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from keras import backend as K\n",
    "import keras\n",
    "\n",
    "print('keras using %s backend'%keras.backend.backend())\n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "# Sets up the graphing configuration\n",
    "import matplotlib.pyplot as graph\n",
    "%matplotlib inline\n",
    "graph.rcParams['figure.figsize'] = (15,5)\n",
    "graph.rcParams[\"font.family\"] = 'DejaVu Sans'\n",
    "graph.rcParams[\"font.size\"] = '12'\n",
    "graph.rcParams['image.cmap'] = 'rainbow'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this too!\n",
    "# This gets our data ready\n",
    "\n",
    "# Load the data\n",
    "dataset = pd.read_csv('Data/dog_data.csv')\n",
    "\n",
    "# Separate out the features\n",
    "features = dataset.drop(['breed'], axis = 1)\n",
    "\n",
    "# Sets the target one-hot vectors\n",
    "target = OneHotEncoder(sparse = False).fit_transform(np.transpose([dataset['breed']]))\n",
    "\n",
    "# Take the first 4/5 of the data and assign it to training\n",
    "train_X = features.values[:160]\n",
    "train_Y = target[:160]\n",
    "\n",
    "# Take the last 1/5 of the data and assign it to testing\n",
    "test_X = features.values[160:]\n",
    "test_Y = target[160:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 2\n",
    "------\n",
    "\n",
    "The box below contains methods to help us quickly change the structure. Don't edit them - just run the box.\n",
    "\n",
    "The __train_network__ method allows us to change:\n",
    "* the number of layers\n",
    "* the activation functions the layers use\n",
    "* the optimizer of the model\n",
    "* the number of training cycles for the model (__epochs__)\n",
    "\n",
    "The plot_acc and bar_acc just plot our models so we can easily see how well they do.\n",
    "\n",
    "Don't worry about the code - it is simply to make the next steps easier.\n",
    "\n",
    "#### __Run the code__ below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this!\n",
    "# Below are a few helper methods. Do not edit these.\n",
    "\n",
    "def train_network(structure, activation, optimizer, epochs):\n",
    "    \n",
    "    os.environ['PYTHONHASHSEED'] = '0'\n",
    "    rn.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    \n",
    "    # This initialises the model\n",
    "    model = keras.models.Sequential()\n",
    "    \n",
    "    # This is our input + the first hidden layer 1\n",
    "    model.add(keras.layers.Dense(units = structure[1], input_dim = structure[0], activation = activation)) \n",
    "    \n",
    "    # Hidden layer 2, if not ignored (of size 0)\n",
    "    if structure[2] > 0:\n",
    "        model.add(keras.layers.Dense(units = structure[2], activation = activation))\n",
    "        \n",
    "    # Output layer\n",
    "    model.add(keras.layers.Dense(units=structure[-1], activation = \"softmax\"))\n",
    "    \n",
    "    # Compiles the model with parameters\n",
    "    model.compile(loss = 'categorical_crossentropy', optimizer = optimizer, metrics = ['accuracy'])\n",
    "    \n",
    "    # This tells the us training has started, so we know that it's actually running\n",
    "    print('training... ', end = '')\n",
    "\n",
    "    # This trains the network\n",
    "    training_stats = model.fit(train_X, train_Y, batch_size = 1, epochs = epochs, verbose = 0, shuffle = False)\n",
    "    \n",
    "    # Results!\n",
    "    print('train_acc: %0.3f, test_acc: %0.3f' %(training_stats.history['accuracy'][-1], \n",
    "                                                model.evaluate(test_X, test_Y, verbose = 0)[1]))\n",
    "    \n",
    "    # This returns the results and the model for use outside the function\n",
    "    return training_stats, model\n",
    "\n",
    "# Plots our evaluations in a line graph to see how they compare\n",
    "def plot_acc(train_acc, test_acc, title):\n",
    "    # Plots the training and testing accuracy lines\n",
    "    training_accuracy, = graph.plot(train_acc, label = 'Training Accuracy')\n",
    "    testing_accuracy, = graph.plot(test_acc, label = 'Testing Accuracy')\n",
    "    graph.legend(handles = [training_accuracy, testing_accuracy])\n",
    "    \n",
    "    # Plots guide lines along y = 0 and y = 1 to help visualise\n",
    "    xp = np.linspace(0, train_acc.shape[0] - 1, 10 * train_acc.shape[0])\n",
    "    graph.plot(xp, np.full(xp.shape, 1), c = 'k', linestyle = ':', alpha = 0.5)\n",
    "    graph.plot(xp, np.full(xp.shape, 0), c = 'k', linestyle = ':', alpha = 0.5)\n",
    "    \n",
    "    graph.xticks(range(0, train_acc.shape[0]), range(1, train_acc.shape[0] + 1))\n",
    "    graph.ylim(0,1)\n",
    "    graph.title(title)\n",
    "    \n",
    "    graph.show()\n",
    "\n",
    "# Plots our evaluations in a bar chart to see how they compare\n",
    "def bar_acc(train_acc, test_acc, title, xticks):\n",
    "    index = range(1, train_acc.shape[0] + 1)\n",
    "    \n",
    "    # Plots the training and testing accuracy bars\n",
    "    training_accuracy = graph.bar(index, train_acc, 0.4, align = 'center')\n",
    "    testing_accuracy = graph.bar(index, test_acc, 0.4, align = 'edge')\n",
    "    graph.legend((training_accuracy[0], testing_accuracy[0]), ('Training Accuracy', 'Testing Accuracy'))\n",
    "    \n",
    "    graph.xticks(index, xticks)\n",
    "    graph.title(title)\n",
    "    \n",
    "    graph.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 3\n",
    "------\n",
    "\n",
    "Let's first look at how different layer sizes impact performance.\n",
    "\n",
    "Let's look at a network with just one hidden layer. We'll see how it performs with 1 to 10 nodes.\n",
    "\n",
    "### In the cell below replace:\n",
    "#### 1. `<addHidden1>` with `hidden1`\n",
    "#### 2. `<addTrainAcc>` with `train_acc`\n",
    "#### 3. `<addTestAcc>` with `test_acc`\n",
    "#### and then __run the code__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialises empty arrays into which to append new values.\n",
    "train_acc = np.empty((0))\n",
    "test_acc = np.empty((0))\n",
    "\n",
    "for hidden1 in range (1,11):\n",
    "    print('Evaluating model with %i hidden neurons... ' %hidden1, end = '')\n",
    "\n",
    "###\n",
    "# REPLACE <addHidden1> BELOW WITH hidden1\n",
    "###\n",
    "    training_stats, model = train_network(structure = [3, <addHidden1>, <addHidden1>, 3], \n",
    "                                          activation = 'relu', optimizer = 'RMSprop', epochs = 12)\n",
    "###\n",
    "    \n",
    "    train_acc = np.append(train_acc, training_stats.history['accuracy'][-1])\n",
    "    test_acc = np.append(test_acc, model.evaluate(test_X, test_Y, verbose = 0)[1])\n",
    "\n",
    "###\n",
    "# REPLACE <addTrainAcc> WITH train_acc AND <addTestAcc> WITH test_acc\n",
    "###\n",
    "plot_acc(<addTrainAcc>, <addTestAcc>, 'hidden layer size performance comparison')\n",
    "###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, experimenting with different sizes of hidden layers can dramatically improve your results.\n",
    "\n",
    "Step 4\n",
    "------\n",
    "\n",
    "Now we'll look at how different activation functions impact the performance.\n",
    "\n",
    "There's lots we will try, just remember it is common to try both `relu` and `tanh` first.\n",
    "\n",
    "### In the cell below replace:\n",
    "#### 1. `<addActivation>` with `activation`\n",
    "#### 2. `<addActivationFunctions>` with `activation_functions`\n",
    "#### and then __run the code__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "train_acc = np.empty((0))\n",
    "test_acc = np.empty((0))\n",
    "\n",
    "# Makes a list of the activation functions we wish to compare\n",
    "activation_functions = ['elu', 'selu', 'relu', 'tanh', 'sigmoid', \n",
    "                        'hard_sigmoid', 'softplus', 'softsign', 'linear']\n",
    "\n",
    "for activation in activation_functions:\n",
    "    print('Evaluating model with %s hidden layer activation function... ' %activation, end = '')\n",
    "\n",
    "###\n",
    "# REPLACE <addActivation> WITH activation\n",
    "###\n",
    "    training_stats, model = train_network(structure = [3, 4, 2, 3],\n",
    "                                          activation = <addActivation>, optimizer = 'RMSprop', epochs = 12)\n",
    "###\n",
    "    \n",
    "    train_acc = np.append(train_acc, training_stats.history['accuracy'][-1])\n",
    "    test_acc = np.append(test_acc, model.evaluate(test_X, test_Y, verbose=0)[1])\n",
    "    \n",
    "###\n",
    "# REPLACE THE <addActivationFunctions> BELOW WITH activation_functions\n",
    "###\n",
    "bar_acc(train_acc, test_acc, 'activation function performance comparison using (4,2) hidden layer', <addActivationFunctions>)\n",
    "###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's quite a lot of variance there. It's always good to quickly test different activation functions first.\n",
    "\n",
    "Next, lets try changing the shape of the hidden layers.\n",
    "\n",
    "#### Replace `<updateHere>`'s with `3` and run the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_acc = np.empty((0))\n",
    "test_acc = np.empty((0))\n",
    "\n",
    "activation_functions = ['elu', 'selu', 'relu', 'tanh', 'sigmoid',\n",
    "                        'hard_sigmoid', 'softplus', 'softsign', 'linear']\n",
    "\n",
    "for activation in activation_functions:\n",
    "    print('Evaluating model with %s hidden layer activation function... ' %activation, end='')\n",
    "    \n",
    "\n",
    "# The value you choose for <updateHere> below will change the size of the hidden layers. Lets try changing them both to 3 for now\n",
    "# (but you can have a play around with different numbers if you want)\n",
    "###\n",
    "# REPLACE THE <updateHere>'s BELOW WITH 3\n",
    "###\n",
    "    training_stats, model = train_network(structure = [3, <updateHere>, <updateHere>, 3], \n",
    "                                          activation = activation, optimizer = 'RMSprop', epochs = 12)\n",
    "###\n",
    "    \n",
    "    train_acc = np.append(train_acc, training_stats.history['accuracy'][-1])\n",
    "    test_acc = np.append(test_acc, model.evaluate(test_X, test_Y, verbose=0)[1])\n",
    "    \n",
    "bar_acc(train_acc, test_acc, 'activation function performance comparison using (3,3) hidden layer', activation_functions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 5\n",
    "-----\n",
    "\n",
    "The __optimisation function__ is the last major parameter of the network architecture. It changes how the network is trained - so it can have a __very large impact on training time and end performance__.\n",
    "\n",
    "Note: this step won't always provide the same results every time it is run. Optimizers such as SGD will give different results.\n",
    "\n",
    "#### Replace `<addOptimizer>` with `optimizer` and run the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_acc = np.empty((0))\n",
    "test_acc = np.empty((0))\n",
    "\n",
    "# This is a list of the optimisation functions for us to compare\n",
    "optimization_functions = ['SGD', 'RMSprop', 'Adagrad', 'Adadelta',\n",
    "                          'Adam', 'Adamax', 'Nadam']\n",
    "\n",
    "for optimizer in optimization_functions:\n",
    "    print('Evaluating model with %s optimizer... ' %optimizer, end='')\n",
    "    \n",
    "    \n",
    "# The <addOptimizer> below is where we specify the optimizer in the code    \n",
    "###\n",
    "# REPLACE THE <addOptimizer> BELOW WITH optimizer\n",
    "###\n",
    "    training_stats, model = train_network(structure = [3, 4, 2, 3],\n",
    "                                          activation = 'relu', optimizer = <addOptimizer>, epochs = 12)\n",
    "###\n",
    "\n",
    "# This is recording our data for the plot\n",
    "    train_acc = np.append(train_acc, training_stats.history['accuracy'][-1])\n",
    "    test_acc = np.append(test_acc, model.evaluate(test_X, test_Y, verbose=0)[1])\n",
    "\n",
    "# And now, the plot!    \n",
    "bar_acc(train_acc, test_acc, 'optimizer performance comparison using (4,2) hidden layer', optimization_functions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 6\n",
    "-------\n",
    "\n",
    "Let's try to combine what we've seen above and try to create a neural network that performs better than what we made in exercise 7, where we used the structure `[3,4,2,3]`, the activation function `relu`, and the optimiser `SGD` (Stochastic Gradient Descent).\n",
    "\n",
    "### In the cell below replace:\n",
    "#### 1. `<layerSize>`'s with numbers of your choice (how many nodes the hidden layers will have)\n",
    "#### 2. `<activationFunction>` with one of the following: `'relu'`, `'softsign'`, `'tanh'`, `'elu'`, `'selu'`, `'softplus'`, `'linear'`\n",
    "#### 3. `<optimiser>` with one of the following: `'SGD'`, `'adam'`, `'RMSprop'`, `'Adagrad'`, `'Adadelta'`, `'Adamax'`, `'Nadam'`\n",
    "#### and then __run the code__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "###\n",
    "# REPLACE THE <layerSize>'s' BELOW WITH PARAMETERS TO TEST A NEW NEURAL NETWORK e.g. 4 and 2\n",
    "###\n",
    "structure = [3, <layerSize>, <layerSize>, 3]\n",
    "###\n",
    "\n",
    "###\n",
    "# REPLACE <activationFunction> WITH ONE OF THE FOLLOWING: 'relu', 'softsign', 'tanh', 'elu', 'selu', 'softplus', 'linear'\n",
    "###\n",
    "activation = <activationFunction>\n",
    "###\n",
    "\n",
    "###\n",
    "# REPLACE <optimiser> WITH ONE OF THE FOLLOWING: 'SGD', 'adam', 'RMSprop', 'Adagrad', 'Adadelta', 'Adamax', 'Nadam'\n",
    "###\n",
    "optimizer = <optimiser>\n",
    "###\n",
    "\n",
    "training_stats, model = train_network(structure, activation, optimizer, epochs = 24)\n",
    "\n",
    "# We can plot our training statistics to see how it developed over time\n",
    "accuracy, = graph.plot(training_stats.history['accuracy'], label = 'Accuracy')\n",
    "training_loss, = graph.plot(training_stats.history['loss'], label = 'Training Loss')\n",
    "graph.legend(handles = [accuracy, training_loss])\n",
    "loss = np.array(training_stats.history['loss'])\n",
    "xp = np.linspace(0, loss.shape[0], 10 * loss.shape[0])\n",
    "graph.plot(xp, np.full(xp.shape, 1), c = 'k', linestyle = ':', alpha = 0.5)\n",
    "graph.plot(xp, np.full(xp.shape, 0), c = 'k', linestyle = ':', alpha = 0.5)\n",
    "graph.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does it look? Were we able to beat the other network? Try out a number of different configurations to see how they perform!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conclusion\n",
    "-------\n",
    "\n",
    "We've compared how different neural network architecture parameters influence accuracy performance, and we've tried to combine them in such a way that we maximise this performance."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}