code/Urban Sound Classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hrxo9uwUMrOg"
      },
      "source": [
        "# Classifying Urban sounds using Deep Learning\n",
        "\n",
        "## 4 Model Refinement "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m9iResXMMrOg"
      },
      "source": [
        "### Load Preprocessed data "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FL8mB_S-MrOh"
      },
      "source": [
        "#### Model refinement\n",
        "\n",
        "In our inital attempt, we were able to achieve a Classification Accuracy score of: \n",
        "\n",
        "* Training data Accuracy:  92.3% \n",
        "* Testing data Accuracy:  87% \n",
        "\n",
        "We will now see if we can improve upon that score using a Convolutional Neural Network (CNN). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ma9_6JxWMrOi"
      },
      "source": [
        "#### Feature Extraction refinement \n",
        "\n",
        "In the prevous feature extraction stage, the MFCC vectors would vary in size for the different audio files (depending on the samples duration). \n",
        "\n",
        "However, CNNs require a fixed size for all inputs. To overcome this we will zero pad the output vectors to make them all the same size. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Onc8ikE5MrOi"
      },
      "source": [
        "import numpy as np\n",
        "max_pad_len = 174\n",
        "\n",
        "def extract_features(file_name):\n",
        "   \n",
        "    try:\n",
        "        audio, sample_rate = librosa.load(file_name, res_type='kaiser_fast') \n",
        "        mfccs = librosa.feature.mfcc(y=audio, sr=sample_rate, n_mfcc=40)\n",
        "        pad_width = max_pad_len - mfccs.shape[1]\n",
        "        mfccs = np.pad(mfccs, pad_width=((0, 0), (0, pad_width)), mode='constant')\n",
        "        \n",
        "    except Exception as e:\n",
        "        print(\"Error encountered while parsing file: \", file_name)\n",
        "        return None \n",
        "     \n",
        "    return mfccs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "id": "URJitBkBMrOm",
        "outputId": "d140b7f8-e9ac-4a64-a7ca-40f0e63d12f7"
      },
      "source": [
        "# Load various imports \n",
        "import pandas as pd\n",
        "import os\n",
        "import librosa\n",
        "\n",
        "# Set the path to the full UrbanSound dataset \n",
        "fulldatasetpath = r\"C:\\Users\\dell\\Desktop\\imagine cup\\xfree\\Urban sound classification\\Udacity-ML-Capstone-master\\UrbanSound8K\\audio\"\n",
        "\n",
        "metadata = pd.read_csv('../UrbanSound Dataset sample/metadata/UrbanSound8K.csv')\n",
        "\n",
        "features = []\n",
        "\n",
        "# Iterate through each sound file and extract the features \n",
        "for index, row in metadata.iterrows():\n",
        "    \n",
        "    file_name = os.path.join(os.path.abspath(fulldatasetpath),'fold'+str(row[\"fold\"])+'/',str(row[\"slice_file_name\"]))\n",
        "    \n",
        "    class_label = row[\"class_name\"]\n",
        "    data = extract_features(file_name)\n",
        "    features.append([data, class_label])\n",
        "\n",
        "\n",
        "# Convert into a Panda dataframe \n",
        "featuresdf = pd.DataFrame(features, columns=['feature','class_label'])\n",
        "\n",
        "print('Finished feature extraction from ', len(featuresdf), ' files') "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Finished feature extraction from  8732  files\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gmsGLd03MrOq"
      },
      "source": [
        "#print(featuresdf)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2qMzM5kyMrOw",
        "outputId": "12e76382-5e40-4c86-b2cd-425baf37e91f"
      },
      "source": [
        "from sklearn.preprocessing import LabelEncoder\n",
        "from keras.utils import to_categorical\n",
        "\n",
        "# Convert features and corresponding classification labels into numpy arrays\n",
        "X = np.array(featuresdf.feature.tolist())\n",
        "y = np.array(featuresdf.class_label.tolist())\n",
        "\n",
        "# Encode the classification labels\n",
        "le = LabelEncoder()\n",
        "yy = to_categorical(le.fit_transform(y)) \n",
        "\n",
        "# split the dataset \n",
        "from sklearn.model_selection import train_test_split \n",
        "\n",
        "x_train, x_test, y_train, y_test = train_test_split(X, yy, test_size=0.2, random_state = 42)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QCX2lVIaMrOy"
      },
      "source": [
        "### Convolutional Neural Network (CNN) model architecture \n",
        "\n",
        "\n",
        "We will modify our model to be a Convolutional Neural Network (CNN) again using Keras and a Tensorflow backend. \n",
        "\n",
        "Again we will use a `sequential` model, starting with a simple model architecture, consisting of four `Conv2D` convolution layers, with our final output layer being a `dense` layer. \n",
        "\n",
        "The convolution layers are designed for feature detection. It works by sliding a filter window over the input and performing a matrix multiplication and storing the result in a feature map. This operation is known as a convolution. \n",
        "\n",
        "\n",
        "The `filter` parameter specifies the number of nodes in each layer. Each layer will increase in size from 16, 32, 64 to 128, while the `kernel_size` parameter specifies the size of the kernel window which in this case is 2 resulting in a 2x2 filter matrix. \n",
        "\n",
        "The first layer will receive the input shape of (40, 174, 1) where 40 is the number of MFCC's 174 is the number of frames taking padding into account and the 1 signifying that the audio is mono. \n",
        "\n",
        "The activation function we will be using for our convolutional layers is `ReLU` which is the same as our previous model. We will use a smaller `Dropout` value of 20% on our convolutional layers. \n",
        "\n",
        "Each convolutional layer has an associated pooling layer of `MaxPooling2D` type with the final convolutional layer having a `GlobalAveragePooling2D` type. The pooling layer is do reduce the dimensionality of the model (by reducing the parameters and subsquent computation requirements) which serves to shorten the training time and reduce overfitting. The Max Pooling type takes the maximum size for each window and the Global Average Pooling type takes the average which is suitable for feeding into our `dense` output layer.  \n",
        "\n",
        "Our output layer will have 10 nodes (num_labels) which matches the number of possible classifications. The activation is for our output layer is `softmax`. Softmax makes the output sum up to 1 so the output can be interpreted as probabilities. The model will then make its prediction based on which option has the highest probability."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-JzFRZ2dMrOz",
        "outputId": "a27b0155-ac8a-4c46-c4b9-496d86c0c79a"
      },
      "source": [
        "import numpy as np\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, Activation, Flatten\n",
        "from keras.layers import Convolution2D, Conv2D, MaxPooling2D, GlobalAveragePooling2D\n",
        "from keras.optimizers import Adam\n",
        "from keras.utils import np_utils\n",
        "from sklearn import metrics \n",
        "\n",
        "num_rows = 40\n",
        "num_columns = 174\n",
        "num_channels = 1\n",
        "\n",
        "x_train = x_train.reshape(x_train.shape[0], num_rows, num_columns, num_channels)\n",
        "x_test = x_test.reshape(x_test.shape[0], num_rows, num_columns, num_channels)\n",
        "\n",
        "num_labels = yy.shape[1]\n",
        "filter_size = 2\n",
        "\n",
        "# Construct model \n",
        "model = Sequential()\n",
        "model.add(Conv2D(filters=16, kernel_size=2, input_shape=(num_rows, num_columns, num_channels), activation='relu'))\n",
        "model.add(MaxPooling2D(pool_size=2))\n",
        "model.add(Dropout(0.2))\n",
        "\n",
        "model.add(Conv2D(filters=32, kernel_size=2, activation='relu'))\n",
        "model.add(MaxPooling2D(pool_size=2))\n",
        "model.add(Dropout(0.2))\n",
        "\n",
        "model.add(Conv2D(filters=64, kernel_size=2, activation='relu'))\n",
        "model.add(MaxPooling2D(pool_size=2))\n",
        "model.add(Dropout(0.2))\n",
        "\n",
        "model.add(Conv2D(filters=128, kernel_size=2, activation='relu'))\n",
        "model.add(MaxPooling2D(pool_size=2))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(GlobalAveragePooling2D())\n",
        "\n",
        "model.add(Dense(num_labels, activation='softmax')) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n",
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n",
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
            "\n",
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.\n",
            "\n",
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
            "\n",
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-q2FxSIPMrO1"
      },
      "source": [
        "### Compiling the model \n",
        "\n",
        "For compiling our model, we will use the same three parameters as the previous model: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e5ok_xp1MrO2",
        "outputId": "4cb87a1a-4d83-4911-8f55-ec9c546fa84b"
      },
      "source": [
        "# Compile the model\n",
        "model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='adam') "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\keras\\optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zc9HcUO1MrO6",
        "outputId": "76f7ec00-5ba7-487a-beec-399e344ebcab"
      },
      "source": [
        "# Display model architecture summary \n",
        "model.summary()\n",
        "\n",
        "# Calculate pre-training accuracy \n",
        "score = model.evaluate(x_test, y_test, verbose=1)\n",
        "accuracy = 100*score[1]\n",
        "\n",
        "print(\"Pre-training accuracy: %.4f%%\" % accuracy)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_1 (Conv2D)            (None, 39, 173, 16)       80        \n",
            "_________________________________________________________________\n",
            "max_pooling2d_1 (MaxPooling2 (None, 19, 86, 16)        0         \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 19, 86, 16)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_2 (Conv2D)            (None, 18, 85, 32)        2080      \n",
            "_________________________________________________________________\n",
            "max_pooling2d_2 (MaxPooling2 (None, 9, 42, 32)         0         \n",
            "_________________________________________________________________\n",
            "dropout_2 (Dropout)          (None, 9, 42, 32)         0         \n",
            "_________________________________________________________________\n",
            "conv2d_3 (Conv2D)            (None, 8, 41, 64)         8256      \n",
            "_________________________________________________________________\n",
            "max_pooling2d_3 (MaxPooling2 (None, 4, 20, 64)         0         \n",
            "_________________________________________________________________\n",
            "dropout_3 (Dropout)          (None, 4, 20, 64)         0         \n",
            "_________________________________________________________________\n",
            "conv2d_4 (Conv2D)            (None, 3, 19, 128)        32896     \n",
            "_________________________________________________________________\n",
            "max_pooling2d_4 (MaxPooling2 (None, 1, 9, 128)         0         \n",
            "_________________________________________________________________\n",
            "dropout_4 (Dropout)          (None, 1, 9, 128)         0         \n",
            "_________________________________________________________________\n",
            "global_average_pooling2d_1 ( (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 10)                1290      \n",
            "=================================================================\n",
            "Total params: 44,602\n",
            "Trainable params: 44,602\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "1747/1747 [==============================] - 12s 7ms/step\n",
            "Pre-training accuracy: 11.5054%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L558aYQ6MrO8"
      },
      "source": [
        "### Training \n",
        "\n",
        "Here we will train the model. As training a CNN can take a sigificant amount of time, we will start with a low number of epochs and a low batch size. If we can see from the output that the model is converging, we will increase both numbers.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NPQ8vNflMrO9",
        "outputId": "7b7d683c-a846-4dac-dae8-d1b0b5d6746f"
      },
      "source": [
        "from keras.callbacks import ModelCheckpoint \n",
        "from datetime import datetime \n",
        "\n",
        "#num_epochs = 1\n",
        "#num_batch_size = 128\n",
        "\n",
        "num_epochs = 0#72\n",
        "num_batch_size = 256\n",
        "\n",
        "checkpointer = ModelCheckpoint(filepath='saved_models/weights.best.basic_cnn.xml',verbose=1, save_best_only=True)\n",
        "start = datetime.now()\n",
        "\n",
        "model.fit(x_train, y_train, batch_size=num_batch_size, epochs=num_epochs, validation_data=(x_test, y_test), callbacks=[checkpointer], verbose=1)\n",
        "\n",
        "\n",
        "duration = datetime.now() - start\n",
        "print(\"Training completed in time: \", duration)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From C:\\Users\\dell\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "Train on 6985 samples, validate on 1747 samples\n",
            "Epoch 1/72\n",
            "6985/6985 [==============================] - 74s 11ms/step - loss: 3.3875 - acc: 0.2116 - val_loss: 2.0256 - val_acc: 0.2868\n",
            "\n",
            "Epoch 00001: val_loss improved from inf to 2.02562, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 2/72\n",
            "6985/6985 [==============================] - 56s 8ms/step - loss: 1.7669 - acc: 0.3656 - val_loss: 1.8064 - val_acc: 0.3812\n",
            "\n",
            "Epoch 00002: val_loss improved from 2.02562 to 1.80637, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 3/72\n",
            "6985/6985 [==============================] - 56s 8ms/step - loss: 1.5530 - acc: 0.4580 - val_loss: 1.6838 - val_acc: 0.4230\n",
            "\n",
            "Epoch 00003: val_loss improved from 1.80637 to 1.68384, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 4/72\n",
            "6985/6985 [==============================] - 53s 8ms/step - loss: 1.4266 - acc: 0.4986 - val_loss: 1.5329 - val_acc: 0.4785\n",
            "\n",
            "Epoch 00004: val_loss improved from 1.68384 to 1.53285, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 5/72\n",
            "6985/6985 [==============================] - 52s 7ms/step - loss: 1.3387 - acc: 0.5319 - val_loss: 1.4106 - val_acc: 0.5318\n",
            "\n",
            "Epoch 00005: val_loss improved from 1.53285 to 1.41057, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 6/72\n",
            "6985/6985 [==============================] - 51s 7ms/step - loss: 1.2596 - acc: 0.5592 - val_loss: 1.3576 - val_acc: 0.5638\n",
            "\n",
            "Epoch 00006: val_loss improved from 1.41057 to 1.35759, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 7/72\n",
            "6985/6985 [==============================] - 52s 7ms/step - loss: 1.2029 - acc: 0.5792 - val_loss: 1.3170 - val_acc: 0.5529\n",
            "\n",
            "Epoch 00007: val_loss improved from 1.35759 to 1.31697, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 8/72\n",
            "6985/6985 [==============================] - 52s 7ms/step - loss: 1.1485 - acc: 0.6069 - val_loss: 1.2409 - val_acc: 0.5993\n",
            "\n",
            "Epoch 00008: val_loss improved from 1.31697 to 1.24089, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 9/72\n",
            "6985/6985 [==============================] - 51s 7ms/step - loss: 1.0887 - acc: 0.6251 - val_loss: 1.1715 - val_acc: 0.6131\n",
            "\n",
            "Epoch 00009: val_loss improved from 1.24089 to 1.17154, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 10/72\n",
            "6985/6985 [==============================] - 50s 7ms/step - loss: 1.0366 - acc: 0.6478 - val_loss: 1.0862 - val_acc: 0.6485\n",
            "\n",
            "Epoch 00010: val_loss improved from 1.17154 to 1.08620, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 11/72\n",
            "6985/6985 [==============================] - 50s 7ms/step - loss: 1.0098 - acc: 0.6487 - val_loss: 1.0507 - val_acc: 0.6560\n",
            "\n",
            "Epoch 00011: val_loss improved from 1.08620 to 1.05068, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 12/72\n",
            "6985/6985 [==============================] - 55s 8ms/step - loss: 0.9765 - acc: 0.6646 - val_loss: 1.0279 - val_acc: 0.6560\n",
            "\n",
            "Epoch 00012: val_loss improved from 1.05068 to 1.02790, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 13/72\n",
            "6985/6985 [==============================] - 61s 9ms/step - loss: 0.9140 - acc: 0.6958 - val_loss: 0.9817 - val_acc: 0.6629\n",
            "\n",
            "Epoch 00013: val_loss improved from 1.02790 to 0.98174, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 14/72\n",
            "6985/6985 [==============================] - 55s 8ms/step - loss: 0.8850 - acc: 0.6938 - val_loss: 0.9406 - val_acc: 0.6903\n",
            "\n",
            "Epoch 00014: val_loss improved from 0.98174 to 0.94062, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 15/72\n",
            "6985/6985 [==============================] - 54s 8ms/step - loss: 0.8614 - acc: 0.7092 - val_loss: 0.9130 - val_acc: 0.6932\n",
            "\n",
            "Epoch 00015: val_loss improved from 0.94062 to 0.91297, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 16/72\n",
            "6985/6985 [==============================] - 51s 7ms/step - loss: 0.8139 - acc: 0.7240 - val_loss: 0.8712 - val_acc: 0.7132\n",
            "\n",
            "Epoch 00016: val_loss improved from 0.91297 to 0.87116, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 17/72\n",
            "6985/6985 [==============================] - 54s 8ms/step - loss: 0.7897 - acc: 0.7293 - val_loss: 0.8424 - val_acc: 0.7178\n",
            "\n",
            "Epoch 00017: val_loss improved from 0.87116 to 0.84238, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 18/72\n",
            "6985/6985 [==============================] - 51s 7ms/step - loss: 0.7455 - acc: 0.7467 - val_loss: 0.8031 - val_acc: 0.7298\n",
            "\n",
            "Epoch 00018: val_loss improved from 0.84238 to 0.80306, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 19/72\n",
            "6985/6985 [==============================] - 50s 7ms/step - loss: 0.7363 - acc: 0.7526 - val_loss: 0.7747 - val_acc: 0.7596\n",
            "\n",
            "Epoch 00019: val_loss improved from 0.80306 to 0.77468, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 20/72\n",
            "6985/6985 [==============================] - 50s 7ms/step - loss: 0.7185 - acc: 0.7550 - val_loss: 0.7687 - val_acc: 0.7521\n",
            "\n",
            "Epoch 00020: val_loss improved from 0.77468 to 0.76871, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 21/72\n",
            "6985/6985 [==============================] - 50s 7ms/step - loss: 0.6943 - acc: 0.7649 - val_loss: 0.7683 - val_acc: 0.7436\n",
            "\n",
            "Epoch 00021: val_loss improved from 0.76871 to 0.76832, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 22/72\n",
            "6985/6985 [==============================] - 51s 7ms/step - loss: 0.6707 - acc: 0.7735 - val_loss: 0.7344 - val_acc: 0.7550\n",
            "\n",
            "Epoch 00022: val_loss improved from 0.76832 to 0.73437, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 23/72\n",
            "6985/6985 [==============================] - 50s 7ms/step - loss: 0.6561 - acc: 0.7764 - val_loss: 0.6651 - val_acc: 0.7991\n",
            "\n",
            "Epoch 00023: val_loss improved from 0.73437 to 0.66508, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 24/72\n",
            "6985/6985 [==============================] - 45s 6ms/step - loss: 0.6373 - acc: 0.7831 - val_loss: 0.6493 - val_acc: 0.7916\n",
            "\n",
            "Epoch 00024: val_loss improved from 0.66508 to 0.64931, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 25/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.5987 - acc: 0.7933 - val_loss: 0.6910 - val_acc: 0.7670\n",
            "\n",
            "Epoch 00025: val_loss did not improve from 0.64931\n",
            "Epoch 26/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.6062 - acc: 0.7898 - val_loss: 0.6193 - val_acc: 0.7974\n",
            "\n",
            "Epoch 00026: val_loss improved from 0.64931 to 0.61931, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 27/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.5883 - acc: 0.8004 - val_loss: 0.6334 - val_acc: 0.7894\n",
            "\n",
            "Epoch 00027: val_loss did not improve from 0.61931\n",
            "Epoch 28/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.5439 - acc: 0.8166 - val_loss: 0.6420 - val_acc: 0.7813\n",
            "\n",
            "Epoch 00028: val_loss did not improve from 0.61931\n",
            "Epoch 29/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.5489 - acc: 0.8129 - val_loss: 0.6206 - val_acc: 0.7997\n",
            "\n",
            "Epoch 00029: val_loss did not improve from 0.61931\n",
            "Epoch 30/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.5596 - acc: 0.8115 - val_loss: 0.5859 - val_acc: 0.8140\n",
            "\n",
            "Epoch 00030: val_loss improved from 0.61931 to 0.58594, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 31/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.5140 - acc: 0.8295 - val_loss: 0.5802 - val_acc: 0.8128\n",
            "\n",
            "Epoch 00031: val_loss improved from 0.58594 to 0.58016, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 32/72\n",
            "6985/6985 [==============================] - 45s 6ms/step - loss: 0.5009 - acc: 0.8289 - val_loss: 0.5396 - val_acc: 0.8283\n",
            "\n",
            "Epoch 00032: val_loss improved from 0.58016 to 0.53955, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 33/72\n",
            "6985/6985 [==============================] - 49s 7ms/step - loss: 0.5078 - acc: 0.8281 - val_loss: 0.5772 - val_acc: 0.8105\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Epoch 00033: val_loss did not improve from 0.53955\n",
            "Epoch 34/72\n",
            "6985/6985 [==============================] - 52s 7ms/step - loss: 0.4959 - acc: 0.8281 - val_loss: 0.5578 - val_acc: 0.8197\n",
            "\n",
            "Epoch 00034: val_loss did not improve from 0.53955\n",
            "Epoch 35/72\n",
            "6985/6985 [==============================] - 44s 6ms/step - loss: 0.4711 - acc: 0.8362 - val_loss: 0.5295 - val_acc: 0.8351\n",
            "\n",
            "Epoch 00035: val_loss improved from 0.53955 to 0.52952, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 36/72\n",
            "6985/6985 [==============================] - 44s 6ms/step - loss: 0.4558 - acc: 0.8464 - val_loss: 0.5167 - val_acc: 0.8392\n",
            "\n",
            "Epoch 00036: val_loss improved from 0.52952 to 0.51665, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 37/72\n",
            "6985/6985 [==============================] - 45s 6ms/step - loss: 0.4521 - acc: 0.8425 - val_loss: 0.5341 - val_acc: 0.8203\n",
            "\n",
            "Epoch 00037: val_loss did not improve from 0.51665\n",
            "Epoch 38/72\n",
            "6985/6985 [==============================] - 49s 7ms/step - loss: 0.4423 - acc: 0.8525 - val_loss: 0.5414 - val_acc: 0.8180\n",
            "\n",
            "Epoch 00038: val_loss did not improve from 0.51665\n",
            "Epoch 39/72\n",
            "6985/6985 [==============================] - 48s 7ms/step - loss: 0.4209 - acc: 0.8577 - val_loss: 0.4927 - val_acc: 0.8409\n",
            "\n",
            "Epoch 00039: val_loss improved from 0.51665 to 0.49274, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 40/72\n",
            "6985/6985 [==============================] - 49s 7ms/step - loss: 0.4103 - acc: 0.8577 - val_loss: 0.4739 - val_acc: 0.8472\n",
            "\n",
            "Epoch 00040: val_loss improved from 0.49274 to 0.47386, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 41/72\n",
            "6985/6985 [==============================] - 46s 7ms/step - loss: 0.4041 - acc: 0.8576 - val_loss: 0.5022 - val_acc: 0.8449\n",
            "\n",
            "Epoch 00041: val_loss did not improve from 0.47386\n",
            "Epoch 42/72\n",
            "6985/6985 [==============================] - 45s 6ms/step - loss: 0.3970 - acc: 0.8650 - val_loss: 0.4892 - val_acc: 0.8460\n",
            "\n",
            "Epoch 00042: val_loss did not improve from 0.47386\n",
            "Epoch 43/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.3894 - acc: 0.8659 - val_loss: 0.4884 - val_acc: 0.8483\n",
            "\n",
            "Epoch 00043: val_loss did not improve from 0.47386\n",
            "Epoch 44/72\n",
            "6985/6985 [==============================] - 43s 6ms/step - loss: 0.3871 - acc: 0.8677 - val_loss: 0.4756 - val_acc: 0.8535\n",
            "\n",
            "Epoch 00044: val_loss did not improve from 0.47386\n",
            "Epoch 45/72\n",
            "6985/6985 [==============================] - 45s 6ms/step - loss: 0.3748 - acc: 0.8724 - val_loss: 0.5050 - val_acc: 0.8374\n",
            "\n",
            "Epoch 00045: val_loss did not improve from 0.47386\n",
            "Epoch 46/72\n",
            "6985/6985 [==============================] - 45s 6ms/step - loss: 0.3736 - acc: 0.8717 - val_loss: 0.4700 - val_acc: 0.8495\n",
            "\n",
            "Epoch 00046: val_loss improved from 0.47386 to 0.47003, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 47/72\n",
            "6985/6985 [==============================] - 45s 6ms/step - loss: 0.3650 - acc: 0.8750 - val_loss: 0.4743 - val_acc: 0.8517\n",
            "\n",
            "Epoch 00047: val_loss did not improve from 0.47003\n",
            "Epoch 48/72\n",
            "6985/6985 [==============================] - 44s 6ms/step - loss: 0.3452 - acc: 0.8830 - val_loss: 0.4833 - val_acc: 0.8512\n",
            "\n",
            "Epoch 00048: val_loss did not improve from 0.47003\n",
            "Epoch 49/72\n",
            "6985/6985 [==============================] - 47s 7ms/step - loss: 0.3463 - acc: 0.8819 - val_loss: 0.4897 - val_acc: 0.8437\n",
            "\n",
            "Epoch 00049: val_loss did not improve from 0.47003\n",
            "Epoch 50/72\n",
            "6985/6985 [==============================] - 42s 6ms/step - loss: 0.3306 - acc: 0.8885 - val_loss: 0.4445 - val_acc: 0.8546\n",
            "\n",
            "Epoch 00050: val_loss improved from 0.47003 to 0.44445, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 51/72\n",
            "6985/6985 [==============================] - 41s 6ms/step - loss: 0.3265 - acc: 0.8858 - val_loss: 0.4319 - val_acc: 0.8620\n",
            "\n",
            "Epoch 00051: val_loss improved from 0.44445 to 0.43190, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 52/72\n",
            "6985/6985 [==============================] - 41s 6ms/step - loss: 0.3393 - acc: 0.8833 - val_loss: 0.4625 - val_acc: 0.8603\n",
            "\n",
            "Epoch 00052: val_loss did not improve from 0.43190\n",
            "Epoch 53/72\n",
            "6985/6985 [==============================] - 41s 6ms/step - loss: 0.3210 - acc: 0.8903 - val_loss: 0.4291 - val_acc: 0.8609\n",
            "\n",
            "Epoch 00053: val_loss improved from 0.43190 to 0.42910, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 54/72\n",
            "6985/6985 [==============================] - 41s 6ms/step - loss: 0.3029 - acc: 0.8948 - val_loss: 0.3966 - val_acc: 0.8701\n",
            "\n",
            "Epoch 00054: val_loss improved from 0.42910 to 0.39662, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 55/72\n",
            "6985/6985 [==============================] - 46s 7ms/step - loss: 0.3122 - acc: 0.8902 - val_loss: 0.4173 - val_acc: 0.8655\n",
            "\n",
            "Epoch 00055: val_loss did not improve from 0.39662\n",
            "Epoch 56/72\n",
            "6985/6985 [==============================] - 43s 6ms/step - loss: 0.3132 - acc: 0.8919 - val_loss: 0.3731 - val_acc: 0.8764\n",
            "\n",
            "Epoch 00056: val_loss improved from 0.39662 to 0.37306, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 57/72\n",
            "6985/6985 [==============================] - 45s 6ms/step - loss: 0.3026 - acc: 0.8965 - val_loss: 0.3629 - val_acc: 0.8804\n",
            "\n",
            "Epoch 00057: val_loss improved from 0.37306 to 0.36286, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 58/72\n",
            "6985/6985 [==============================] - 43s 6ms/step - loss: 0.3040 - acc: 0.8949 - val_loss: 0.3819 - val_acc: 0.8792\n",
            "\n",
            "Epoch 00058: val_loss did not improve from 0.36286\n",
            "Epoch 59/72\n",
            "6985/6985 [==============================] - 43s 6ms/step - loss: 0.2880 - acc: 0.8995 - val_loss: 0.4100 - val_acc: 0.8666\n",
            "\n",
            "Epoch 00059: val_loss did not improve from 0.36286\n",
            "Epoch 60/72\n",
            "6985/6985 [==============================] - 44s 6ms/step - loss: 0.2836 - acc: 0.9031 - val_loss: 0.4232 - val_acc: 0.8655\n",
            "\n",
            "Epoch 00060: val_loss did not improve from 0.36286\n",
            "Epoch 61/72\n",
            "6985/6985 [==============================] - 47s 7ms/step - loss: 0.2859 - acc: 0.9021 - val_loss: 0.4418 - val_acc: 0.8661\n",
            "\n",
            "Epoch 00061: val_loss did not improve from 0.36286\n",
            "Epoch 62/72\n",
            "6985/6985 [==============================] - 46s 7ms/step - loss: 0.2861 - acc: 0.8998 - val_loss: 0.3983 - val_acc: 0.8643\n",
            "\n",
            "Epoch 00062: val_loss did not improve from 0.36286\n",
            "Epoch 63/72\n",
            "6985/6985 [==============================] - 46s 7ms/step - loss: 0.2771 - acc: 0.9059 - val_loss: 0.3812 - val_acc: 0.8809\n",
            "\n",
            "Epoch 00063: val_loss did not improve from 0.36286\n",
            "Epoch 64/72\n",
            "6985/6985 [==============================] - 43s 6ms/step - loss: 0.2490 - acc: 0.9130 - val_loss: 0.3690 - val_acc: 0.8878\n",
            "\n",
            "Epoch 00064: val_loss did not improve from 0.36286\n",
            "Epoch 65/72\n",
            "6985/6985 [==============================] - 48s 7ms/step - loss: 0.2639 - acc: 0.9091 - val_loss: 0.3377 - val_acc: 0.8912\n",
            "\n",
            "Epoch 00065: val_loss improved from 0.36286 to 0.33774, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Epoch 66/72\n",
            "6985/6985 [==============================] - 53s 8ms/step - loss: 0.2508 - acc: 0.9152 - val_loss: 0.3733 - val_acc: 0.8838\n",
            "\n",
            "Epoch 00066: val_loss did not improve from 0.33774\n",
            "Epoch 67/72\n",
            "6985/6985 [==============================] - 50s 7ms/step - loss: 0.2475 - acc: 0.9118 - val_loss: 0.3657 - val_acc: 0.8855\n",
            "\n",
            "Epoch 00067: val_loss did not improve from 0.33774\n",
            "Epoch 68/72\n",
            "6985/6985 [==============================] - 52s 7ms/step - loss: 0.2528 - acc: 0.9097 - val_loss: 0.3680 - val_acc: 0.8764\n",
            "\n",
            "Epoch 00068: val_loss did not improve from 0.33774\n",
            "Epoch 69/72\n",
            "6985/6985 [==============================] - 54s 8ms/step - loss: 0.2429 - acc: 0.9130 - val_loss: 0.3663 - val_acc: 0.8832\n",
            "\n",
            "Epoch 00069: val_loss did not improve from 0.33774\n",
            "Epoch 70/72\n",
            "6985/6985 [==============================] - 51s 7ms/step - loss: 0.2331 - acc: 0.9154 - val_loss: 0.3505 - val_acc: 0.8878\n",
            "\n",
            "Epoch 00070: val_loss did not improve from 0.33774\n",
            "Epoch 71/72\n",
            "6985/6985 [==============================] - 54s 8ms/step - loss: 0.2413 - acc: 0.9131 - val_loss: 0.3484 - val_acc: 0.8918\n",
            "\n",
            "Epoch 00071: val_loss did not improve from 0.33774\n",
            "Epoch 72/72\n",
            "6985/6985 [==============================] - 55s 8ms/step - loss: 0.2449 - acc: 0.9157 - val_loss: 0.3312 - val_acc: 0.8912\n",
            "\n",
            "Epoch 00072: val_loss improved from 0.33774 to 0.33124, saving model to saved_models/weights.best.basic_cnn.xml\n",
            "Training completed in time:  0:57:58.428977\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xLvewjHMrO_"
      },
      "source": [
        "### Test the model \n",
        "\n",
        "Here we will review the accuracy of the model on both the training and test data sets. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "XWaboL3yMrO_",
        "outputId": "e5db45e6-d7d6-4e59-ce8a-402bb48c93f7"
      },
      "source": [
        "# Evaluating the model on the training and testing set\n",
        "score = model.evaluate(x_train, y_train, verbose=0)\n",
        "print(\"Training Accuracy: \", score[1])\n",
        "\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print(\"Testing Accuracy: \", score[1])\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training Accuracy:  0.9590551181102362\n",
            "Testing Accuracy:  0.8912421294328616\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'y_true' is not defined",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-18-ac64777eb7ae>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;31m# Confusion Matrix\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmetrics\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mconfusion_matrix\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m \u001b[0mconfusion_matrix\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_true\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_pred\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m: name 'y_true' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LhLpFKWBMrPC"
      },
      "source": [
        "def out(filename):\n",
        "    prediction_feature = extract_features(file_name) \n",
        "    prediction_feature = prediction_feature.reshape(1, num_rows, num_columns, num_channels)\n",
        "\n",
        "    predicted_vector = model.predict_classes(prediction_feature)\n",
        "    predicted_class = le.inverse_transform(predicted_vector)\n",
        "    return predicted_vector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jMH6UTCSMrPE",
        "outputId": "a800271a-c195-4445-b81d-7e153271acc1"
      },
      "source": [
        "# Confusion Matrix\n",
        "from sklearn.metrics import confusion_matrix\n",
        "y_pred=out(x_test)\n",
        "print(y_pred)\n",
        "print(y_test)\n",
        "\n",
        "confusion_matrix(y_test,y_pred)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1]\n",
            "[[0. 0. 0. ... 0. 1. 0.]\n",
            " [0. 0. 0. ... 0. 0. 1.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " ...\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 1.]\n",
            " [0. 0. 0. ... 0. 1. 0.]]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "ValueError",
          "evalue": "Found input variables with inconsistent numbers of samples: [1747, 1]",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-31-7de6e474d908>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m \u001b[0mconfusion_matrix\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my_pred\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py\u001b[0m in \u001b[0;36mconfusion_matrix\u001b[1;34m(y_true, y_pred, labels, sample_weight)\u001b[0m\n\u001b[0;32m    251\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    252\u001b[0m     \"\"\"\n\u001b[1;32m--> 253\u001b[1;33m     \u001b[0my_type\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_true\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_pred\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_check_targets\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_true\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_pred\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    254\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0my_type\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32min\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;34m\"binary\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"multiclass\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    255\u001b[0m         \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"%s is not supported\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0my_type\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py\u001b[0m in \u001b[0;36m_check_targets\u001b[1;34m(y_true, y_pred)\u001b[0m\n\u001b[0;32m     69\u001b[0m     \u001b[0my_pred\u001b[0m \u001b[1;33m:\u001b[0m \u001b[0marray\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mindicator\u001b[0m \u001b[0mmatrix\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     70\u001b[0m     \"\"\"\n\u001b[1;32m---> 71\u001b[1;33m     \u001b[0mcheck_consistent_length\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_true\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_pred\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     72\u001b[0m     \u001b[0mtype_true\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtype_of_target\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_true\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     73\u001b[0m     \u001b[0mtype_pred\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtype_of_target\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_pred\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py\u001b[0m in \u001b[0;36mcheck_consistent_length\u001b[1;34m(*arrays)\u001b[0m\n\u001b[0;32m    233\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0muniques\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m>\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    234\u001b[0m         raise ValueError(\"Found input variables with inconsistent numbers of\"\n\u001b[1;32m--> 235\u001b[1;33m                          \" samples: %r\" % [int(l) for l in lengths])\n\u001b[0m\u001b[0;32m    236\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    237\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mValueError\u001b[0m: Found input variables with inconsistent numbers of samples: [1747, 1]"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FOizzKoSMrPH"
      },
      "source": [
        "The Training and Testing accuracy scores are both high and an increase on our initial model. Training accuracy has increased by ~6% and Testing accuracy has increased by ~4%. \n",
        "\n",
        "There is a marginal increase in the difference between the Training and Test scores (~6% compared to ~5% previously) though the difference remains low so the model has not suffered from overfitting. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pTpnnaIeMrPH"
      },
      "source": [
        "### Predictions  \n",
        "\n",
        "Here we will modify our previous method for testing the models predictions on a specified audio .wav file. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DIFKc7EwMrPH"
      },
      "source": [
        "def print_prediction(file_name):\n",
        "    prediction_feature = extract_features(file_name) \n",
        "    prediction_feature = prediction_feature.reshape(1, num_rows, num_columns, num_channels)\n",
        "\n",
        "    predicted_vector = model.predict_classes(prediction_feature)\n",
        "    predicted_class = le.inverse_transform(predicted_vector) \n",
        "    print(\"The predicted class is:\", predicted_class[0], '\\n') \n",
        "\n",
        "    predicted_proba_vector = model.predict_proba(prediction_feature) \n",
        "    predicted_proba = predicted_proba_vector[0]\n",
        "    for i in range(len(predicted_proba)): \n",
        "        category = le.inverse_transform(np.array([i]))\n",
        "        print(category[0], \"\\t\\t : \", format(predicted_proba[i], '.32f') )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4gzDFCpMrPL"
      },
      "source": [
        "### Validation \n",
        "\n",
        "#### Test with sample data \n",
        "\n",
        "As before we will verify the predictions using a subsection of the sample audio files we explored in the first notebook. We expect the bulk of these to be classified correctly. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jn2DiV5PMrPL",
        "outputId": "c3ee3ed3-a03a-471f-a795-ebcc323e9a80"
      },
      "source": [
        "# Class: Air Conditioner\n",
        "\n",
        "filename = '../UrbanSound Dataset sample/audio/100852-0-0-0.wav' \n",
        "print_prediction(filename) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The predicted class is: air_conditioner \n",
            "\n",
            "air_conditioner \t\t :  0.99687838554382324218750000000000\n",
            "car_horn \t\t :  0.00000051920119403803255409002304\n",
            "children_playing \t\t :  0.00065934192389249801635742187500\n",
            "dog_bark \t\t :  0.00006011105142533779144287109375\n",
            "drilling \t\t :  0.00134915194939821958541870117188\n",
            "engine_idling \t\t :  0.00001786953180271666496992111206\n",
            "gun_shot \t\t :  0.00003663754978333599865436553955\n",
            "jackhammer \t\t :  0.00095585657982155680656433105469\n",
            "siren \t\t :  0.00001384064671583473682403564453\n",
            "street_music \t\t :  0.00002834899351000785827636718750\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "DBE5Y_lYMrPO",
        "outputId": "dab61677-1d19-4bf8-8c98-05c19a6d1cb1"
      },
      "source": [
        "# Class: Drilling\n",
        "\n",
        "filename = '../UrbanSound Dataset sample/audio/103199-4-0-0.wav'\n",
        "print_prediction(filename) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The predicted class is: drilling \n",
            "\n",
            "air_conditioner \t\t :  0.00000013431598233637487282976508\n",
            "car_horn \t\t :  0.00000025490251687187992502003908\n",
            "children_playing \t\t :  0.00001615423752809874713420867920\n",
            "dog_bark \t\t :  0.00000021904463665123330429196358\n",
            "drilling \t\t :  0.99468487501144409179687500000000\n",
            "engine_idling \t\t :  0.00000016201553876271646004170179\n",
            "gun_shot \t\t :  0.00000000003519966262910401155750\n",
            "jackhammer \t\t :  0.00004046132744406349956989288330\n",
            "siren \t\t :  0.00000000724200077684145071543753\n",
            "street_music \t\t :  0.00525784213095903396606445312500\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kp8V6pxPMrPQ",
        "outputId": "070bcac7-e9e2-410f-d5d5-75a018f737c7"
      },
      "source": [
        "# Class: Street music \n",
        "\n",
        "filename = '../UrbanSound Dataset sample/audio/101848-9-0-0.wav'\n",
        "print_prediction(filename) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The predicted class is: street_music \n",
            "\n",
            "air_conditioner \t\t :  0.00110940611921250820159912109375\n",
            "car_horn \t\t :  0.00011776933388318866491317749023\n",
            "children_playing \t\t :  0.01611172035336494445800781250000\n",
            "dog_bark \t\t :  0.00072734046261757612228393554688\n",
            "drilling \t\t :  0.00000371879946214903611689805984\n",
            "engine_idling \t\t :  0.00000895845732884481549263000488\n",
            "gun_shot \t\t :  0.00000000263216537454979970789282\n",
            "jackhammer \t\t :  0.00000083919400140075595118105412\n",
            "siren \t\t :  0.00168100092560052871704101562500\n",
            "street_music \t\t :  0.98023921251296997070312500000000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NhDzG-qVMrPS",
        "outputId": "6a6266ce-4eea-4893-c5e1-3d7063513fad"
      },
      "source": [
        "# Class: Car Horn \n",
        "\n",
        "filename = '../UrbanSound Dataset sample/audio/100648-1-0-0.wav'\n",
        "print_prediction(filename) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The predicted class is: car_horn \n",
            "\n",
            "air_conditioner \t\t :  0.00447100168094038963317871093750\n",
            "car_horn \t\t :  0.28345683217048645019531250000000\n",
            "children_playing \t\t :  0.00685080140829086303710937500000\n",
            "dog_bark \t\t :  0.15641531348228454589843750000000\n",
            "drilling \t\t :  0.08936390280723571777343750000000\n",
            "engine_idling \t\t :  0.01047550234943628311157226562500\n",
            "gun_shot \t\t :  0.25447627902030944824218750000000\n",
            "jackhammer \t\t :  0.17366543412208557128906250000000\n",
            "siren \t\t :  0.01640361361205577850341796875000\n",
            "street_music \t\t :  0.00442126486450433731079101562500\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6KEfJ_LIMrPV"
      },
      "source": [
        "#### Observations \n",
        "\n",
        "We can see that the model performs well. \n",
        "\n",
        "Interestingly, car horn was again incorrectly classifed but this time as drilling - though the per class confidence shows it was a close decision between car horn with 26% confidence and drilling at 34% confidence.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P7DDFaNwMrPV"
      },
      "source": [
        "### Other audio\n",
        "\n",
        "Again we will further validate our model using a sample of various copyright free sounds that we not part of either our test or training data. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GJDh2r6cMrPW",
        "outputId": "bc36ba53-2131-4ba0-f2c7-b800411d240f"
      },
      "source": [
        "filename = '../Evaluation audio/dog_bark_1.wav'\n",
        "print_prediction(filename) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The predicted class is: dog_bark \n",
            "\n",
            "air_conditioner \t\t :  0.00692389300093054771423339843750\n",
            "car_horn \t\t :  0.01195707358419895172119140625000\n",
            "children_playing \t\t :  0.03886408731341361999511718750000\n",
            "dog_bark \t\t :  0.78876549005508422851562500000000\n",
            "drilling \t\t :  0.06209024786949157714843750000000\n",
            "engine_idling \t\t :  0.00038470054278150200843811035156\n",
            "gun_shot \t\t :  0.08008616417646408081054687500000\n",
            "jackhammer \t\t :  0.00062373012769967317581176757812\n",
            "siren \t\t :  0.00402355147525668144226074218750\n",
            "street_music \t\t :  0.00628101080656051635742187500000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "hpboXUSsMrPY",
        "outputId": "de05a623-1fe5-4973-bedd-568370e6afa6"
      },
      "source": [
        "filename = '../Evaluation audio/drilling_1.wav'\n",
        "\n",
        "print_prediction(filename) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The predicted class is: jackhammer \n",
            "\n",
            "air_conditioner \t\t :  0.44938567280769348144531250000000\n",
            "car_horn \t\t :  0.00038593693170696496963500976562\n",
            "children_playing \t\t :  0.00039813644252717494964599609375\n",
            "dog_bark \t\t :  0.00285975052975118160247802734375\n",
            "drilling \t\t :  0.01796808838844299316406250000000\n",
            "engine_idling \t\t :  0.00871684495359659194946289062500\n",
            "gun_shot \t\t :  0.00003318917515571229159832000732\n",
            "jackhammer \t\t :  0.52008956670761108398437500000000\n",
            "siren \t\t :  0.00014330238627735525369644165039\n",
            "street_music \t\t :  0.00001956611959030851721763610840\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vgNtq8-gMrPa",
        "outputId": "77687abf-db9b-4780-bf0e-bfddd1c359c1"
      },
      "source": [
        "filename = '../Evaluation audio/gun_shot_1.wav'\n",
        "\n",
        "print_prediction(filename) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The predicted class is: gun_shot \n",
            "\n",
            "air_conditioner \t\t :  0.00000004153686106178611225914210\n",
            "car_horn \t\t :  0.00000016333915198174508986994624\n",
            "children_playing \t\t :  0.00000901593011803925037384033203\n",
            "dog_bark \t\t :  0.00041548791341483592987060546875\n",
            "drilling \t\t :  0.00001343048097623977810144424438\n",
            "engine_idling \t\t :  0.00000009433045278228746610693634\n",
            "gun_shot \t\t :  0.99955993890762329101562500000000\n",
            "jackhammer \t\t :  0.00000000109624220812065686914138\n",
            "siren \t\t :  0.00000020992132476749247871339321\n",
            "street_music \t\t :  0.00000149148161199263995513319969\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "POPlHjaeMrPd"
      },
      "source": [
        "#### Observations \n",
        "\n",
        "The performance of our final model is very good and has generalised well, seeming to predict well when tested against new audio data. "
      ]
    }
  ]
}