t81_558_class_04_2_multi_class.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 4: Training for Tabular Data**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "* Part 4.1: Encoding a Feature Vector for Keras Deep Learning [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_1_feature_encode.ipynb)\n",
    "* **Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC** [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_2_multi_class.ipynb)\n",
    "* Part 4.3: Keras Regression for Deep Neural Networks with RMSE [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_3_regression.ipynb)\n",
    "* Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_4_backprop.ipynb)\n",
    "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_5_rmse_logloss.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC\n",
    "\n",
    "* **Binary Classification** - Classification between two possibilities (positive and negative).  Common in medical testing, does the person have the disease (positive) or not (negative).\n",
    "* **Classification** - Classification between more than 2.  The iris dataset (3-way classification).\n",
    "* **Regression** - Numeric prediction.  How many MPG does a car get? (covered in next video)\n",
    "\n",
    "In this class session we will look at some visualizations for all three.\n",
    "\n",
    "\n",
    "It is important to evaluate the level of error in the results produced by a neural network.  In this part we will look at how to evaluate error for both classification and regression neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binary Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Binary classification is used to create a model that classifies between only two classes.  These two classes are often called \"positive\" and \"negative\".  Consider the following program that uses the [wcbreast_wdbc dataset](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/datasets_wcbc.ipynb) to classify if a breast tumor is cancerous (malignant) or not (benign).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>mean_radius</th>\n",
       "      <th>mean_texture</th>\n",
       "      <th>mean_perimeter</th>\n",
       "      <th>mean_area</th>\n",
       "      <th>mean_smoothness</th>\n",
       "      <th>mean_compactness</th>\n",
       "      <th>mean_concavity</th>\n",
       "      <th>mean_concave_points</th>\n",
       "      <th>...</th>\n",
       "      <th>worst_radius</th>\n",
       "      <th>worst_texture</th>\n",
       "      <th>worst_perimeter</th>\n",
       "      <th>worst_area</th>\n",
       "      <th>worst_smoothness</th>\n",
       "      <th>worst_compactness</th>\n",
       "      <th>worst_concavity</th>\n",
       "      <th>worst_concave_points</th>\n",
       "      <th>worst_symmetry</th>\n",
       "      <th>worst_fractal_dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  mean_radius  mean_texture  mean_perimeter  mean_area  \\\n",
       "0    842302         M        17.99         10.38          122.80     1001.0   \n",
       "1    842517         M        20.57         17.77          132.90     1326.0   \n",
       "2  84300903         M        19.69         21.25          130.00     1203.0   \n",
       "3  84348301         M        11.42         20.38           77.58      386.1   \n",
       "4  84358402         M        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   mean_smoothness  mean_compactness  mean_concavity  mean_concave_points  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "   ...  worst_radius  worst_texture  worst_perimeter  worst_area  \\\n",
       "0  ...         25.38          17.33           184.60      2019.0   \n",
       "1  ...         24.99          23.41           158.80      1956.0   \n",
       "2  ...         23.57          25.53           152.50      1709.0   \n",
       "3  ...         14.91          26.50            98.87       567.7   \n",
       "4  ...         22.54          16.67           152.20      1575.0   \n",
       "\n",
       "   worst_smoothness  worst_compactness  worst_concavity  worst_concave_points  \\\n",
       "0            0.1622             0.6656           0.7119                0.2654   \n",
       "1            0.1238             0.1866           0.2416                0.1860   \n",
       "2            0.1444             0.4245           0.4504                0.2430   \n",
       "3            0.2098             0.8663           0.6869                0.2575   \n",
       "4            0.1374             0.2050           0.4000                0.1625   \n",
       "\n",
       "   worst_symmetry  worst_fractal_dimension  \n",
       "0          0.4601                  0.11890  \n",
       "1          0.2750                  0.08902  \n",
       "2          0.3613                  0.08758  \n",
       "3          0.6638                  0.17300  \n",
       "4          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "display(df[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curves\n",
    "\n",
    "ROC curves can be a bit confusing.  However, they are very common.  It is important to know how to read them.  Even their name is confusing.  Do not worry about their name, it comes from electrical engineering (EE).\n",
    "\n",
    "Binary classification is common in medical testing.  Often you want to diagnose if someone has a disease.  This can lead to two types of errors, know as false positives and false negatives:\n",
    "\n",
    "* **False Positive** - Your test (neural network) indicated that the patient had the disease; however, the patient did not have the disease.\n",
    "* **False Negative** - Your test (neural network) indicated that the patient did not have the disease; however, the patient did have the disease.\n",
    "* **True Positive** - Your test (neural network) correctly identified that the patient had the disease.\n",
    "* **True Negative** - Your test (neural network) correctly identified that the patient did not have the disease.\n",
    "\n",
    "Types of errors:\n",
    "\n",
    "![Type of Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_4_errors.png \"Type of Error\")\n",
    "\n",
    "Neural networks classify in terms of probability of it being positive. However, at what probability do you give a positive result?  Is the cutoff 50%? 90%?  Where you set this cutoff is called the threshold.  Anything above the cutoff is positive, anything below is negative.  Setting this cutoff allows the model to be more sensitive or specific:\n",
    "\n",
    "More info on Sensitivity vs Specificity: [Khan Academy](https://www.youtube.com/watch?v=Z5TtopYX1Gc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hU1dbH8e9Op4ReJRSRIkmAGJpSBOkIAtISASlKU6+KFwvX9uL12r2KqBQBUSSYhH6lgwhKJyAtgPQSEOkdUs/7x57QJJCEmdlT1ud58mQYZs755TCs7Nlz9jrKsiyEEEI4n4/pAEII4a2kAAshhCFSgIUQwhApwEIIYYgUYCGEMEQKsBBCGCIF2AmUUm8opRKVUpuVUhuVUvWcsM/Xb/rzSkfvMzuUUum2Y7BJKbVBKVU/G8+54Ixst9jvMKXUyyb2fTuOfD0ppeYqpQrZbr+glNqulIpRSrVXSg21136EpuQ8YMdSSj0EfAY0sSwrWSlVDAiwLOuIg/d7wbKs/I7cR25cn0sp1Qp43bKsxtl9jjMppYYBFyzL+tTZ+86KM19PSqkdQHPLspLsvW2hyQjY8UoDJyzLSgawLOuEZVlHlFK1lFLLlFLrlVILlFKlAZRSS5VSHyml1iqldiqlGtnuD7Pdt9E28qlsu3+mbRuJSqkBtvs+BPLYHhtju++C7XusUqptZjil1HdKqS5KKV+l1CdKqXW27Q90wrEpAJy25civlPrZNireopTqcPODs3qMUqqCbaQ21nYcFiql8tj+rpJSavF1I+77bPe/ct3P+s51+3jDdtyXA1WdcAxyKqvX036l1Me247JWKVUJQClVXCk1zfazrlNKNbDdn18pNcH2+M1Kqc62+/crpYoppUYDFYF5SqmXlFJ9lFJf2R5TUik1w3ZMN2XnXYzIgmVZ8uXALyA/sBHYCYwEGgP+wEqguO0xUcC3tttLgf/abj8KLLbd/hLoYbsdAOSx3S5i+54H2AoUtf35wk05Lti+Pw58f912DtmeOwB403Z/IJAA3OuA45FuOx47gLNALdv9fkAB2+1iwG6uvUO7cLvHABWANCDC9nfxQE/b7TXA47bbQUBeoCXwje25PsBs4GGgFrDF9pgCtu2/bPo1dKfXk+3+/cAbttu9gNm225OBhrbb5YDtttsfAcOv227h67ZT7Ba3+wBf2W7HAYNtt32BgqaPi7t++SEcyrKsC0qpWkAj4BH0i/c/QDiwSCkF+kX853VPm277vh5dXABWAW8opUKA6ZZl7bLd/4JS6nHb7bJAZeDkbSLNA75QSgUCrYFfLcu6rJRqCdRQSnWxPa6gbVv7cv5T39Zly7Ii4Orb6YlKqXB0MXxfKfUwkAGUAUoCR697blaPAdhnWdZG2+31QAWlVDBQxrKsGQCWZV2x7bclugj/bnt8ftvPGgzMsCzrku1x/7Pzz37XbvV6um5u9sfrvn9uu90cCLW9zgAKKKXy2+6Pvm67p3MQoym6yGNZVjr6F6nIBSnATmB7kS4FliqltgDPAYmWZT2UxVOSbd/Tsf0bWZY1WSm1BmgLzLVNEWSg/yM9ZFnWJaXUUvQo73ZZrtge1wo98o61/ZUCnrcsa0GufshcsCxrlW0Oszh6tF8cPSJOVUrt5+8/S4/bPCb5uselo0f1WVHAB5ZljbnhTqUG5/ZncaZbvJ56Z/7V9Q+zffcBHsz85ZPpuoIsDJI5YAdTSlXNnK+1iQC2A8VtI0CUUv5KqbA7bKcisNeyrBHALKAGepR62lZ87wcevO4pqUop/yw2Fwf0RY+i5tvuWwA8k/kcpVQVpVS+nPysOWXL7IsesRcEjtkK6yNA+Vs8JTuPucqyrPNAklKqo21/gUqpvOif9SnbSBClVBmlVAngV6CjUiqPbfT8mH1+UvvJ4vV0wHY76rrvq2y3FwLPX/f8CNvNReiBQOb9hXMQ42fgGdvzfJVSBXPwXHEdKcCOlx/4Xim1TSm1GQgF3ga6AB8ppTah5/Tu9EFGN2CrUmojevpiIrp4+imltgMfAquve/w3wObMD+FushA9F73YsqwU233jgG3ABqXUVmAMjnmHlPnh4Eb0L4LethFdDFDbNqLrhZ4jvll2HnOzJ9HTNJvR8+6lLMtaiJ4bXWXb1lQg2LKsDbZMm9BTNevu5gd1kFu9nobZ/q6w7b4XgZds972APmablVLbgEG2+/9je/xW22vwkRxkeBF4xHbs1tsyiFyQ09CE8AC26ZjalmWdMJ1FZJ+MgIUQwhAZAQshhCEyAhZCCEOkAAshhCFSgIUQwhApwEIIYYgUYCGEMCRHJ9oXK1bMqlChgoOiiJz4448/AKha1RUbdjmXHItr5Fi4pvXr15+wLKv4zffnqABXqFCBhIQE+6USudakSRMAli5dajSHK5BjcY0cC9eklDpwq/tlCkIIIQyRAiyEEIZIARZCCEOkH7AQ4qrU1FSSkpK4cuXKnR8s/iYoKIiQkBD8/bPqBHsjKcBCiKuSkpIIDg6mQoUK0rQ9hyzL4uTJkyQlJXHvvfdm6zkyBSGEuOrKlSsULVpUim8uKKUoWrRojt49SAEWQtxAim/u5fTYSQEWQrgUpRRDhgy5+udPP/2UYcOG2X0/77///g1/rl//ThelsT8pwEIIlxIYGMj06dM5ccKxF/e4uQCvXLnSofu7FSnAQgiX4ufnx4ABA/j888//9nfHjx+nc+fO1KlThzp16rBixYqr97do0YKwsDD69etH+fLlrxbwjh07UqtWLcLCwvjmm28AGDp0KJcvXyYiIoIePXoAkD9/fgCio6OZM2fO1X326dOHqVOnkp6eziuvvEKdOnWoUaMGY8bccFHtXJECLIRwOc899xwxMTGcPXv2hvtffPFFXnrpJdatW8e0adPo168fAO+88w5NmzYlMTGRLl26cPDgwavP+fbbb1m/fj0JCQmMGDGCkydP8uGHH5InTx42btxITMyN162NiooiPj4egJSUFH7++Wfatm3L+PHjKViwIOvWrWPdunWMHTuWffv23dXPKaehCSFuafD8wWw8utGu24woFcHw1sPv+LgCBQrQq1cvRowYQZ48ea7ev3jxYrZt23b1z+fOnePChQssX76cGTNmANC6dWsKFy589TEjRoy4+neHDh1i165dFC1aNMt9t2nThhdffJHk5GTmz5/Pww8/TJ48eVi4cCGbN29m6tSpAJw9e5Zdu3Zl+5SzW5ECLIRwSYMHDyYyMpK+fftevS8jI4PVq1cTFBSUrW0sXbqUxYsXs2rVKvLmzUuTJk3ueJpYUFAQTZo0YcGCBcTFxREdHQ3o83y//PJLWrVqlfsf6iZSgIUQt5SdkaojFSlShG7dujF+/HieeuopAFq2bMmXX37JK6+8AsDGjRuJiIigQYMGxMfH89prr7Fw4UJOnz4N6FFq4cKFyZs3Lzt27GD16tVXt+/v709qauotV61FRUUxbtw4EhIS+O677wBo1aoVo0aNomnTpvj7+7Nz507KlClDvnz5cv0zyhywEMJlDRky5IazIUaMGEFCQgI1atQgNDSU0aNHA/B///d/LFy4kPDwcKZMmUKpUqUIDg6mdevWpKWlUa1aNYYOHcqDDz54dVsDBgygRo0aVz+Eu17Lli1ZtmwZzZs3JyAgAIB+/foRGhpKZGQk4eHhDBw4kLS0tLv6+XJ0WfratWtb0g/YNUjf12vkWFxzt8di+/btVKtWzX6BnCQ5ORlfX1/8/PxYtWoVzzzzDBs32nf+OrtudQyVUusty6p982NlCkII4fYOHjxIt27dyMjIICAggLFjx5qOlC1SgIUQbq9y5cr8/vvvpmPkmMwBCyGEIVKAhRDCECnAQghhiBRgIYQwRAqwEMKl+Pr6EhERQXh4OF27duXSpUs53ka/fv2uLll2hbaTWZECLIRwKZlNcrZu3UpAQMDVxRY5MW7cOEJDQwHXaDuZFSnAQgiX1ahRI3bv3g3AZ599Rnh4OOHh4QwfrpdJX7x4kbZt21KzZk3Cw8OJi4sD9IKUhIQEl2k7mRU5D1gI4ZLS0tKYN28erVu3Zv369UyYMIE1a9ZgWRb16tWjcePG7N27l3vuuedqIb25feWHH37IV199dctVcZltJ9u2bXu17eSoUaNuaDuZnJxMgwYNaNmy5V11PcuKFGAhxK0NHgz2Xs4bEQHDb9/kJ3PECnoE/PTTTzNq1Cgef/zxq41vOnXqxG+//Ubr1q0ZMmQIr732Gu3ataNRo0bZjuLMtpNZkQIshHApmXPA2VGlShU2bNjA3LlzefPNN2nWrBlvv/12tp7rzLaTWZECLIS4tTuMVJ2pUaNG9OnTh6FDh2JZFjNmzOCHH37gyJEjFClShJ49e1KoUCHGjRv3t+e6QtvJrEgBFkK4vMjISPr06UPdunUBfZrZAw88wIIFC3jllVfw8fHB39+fUaNG/e25mW0nIyMj/3b5oZYtW/Lkk0/SoUOHG9pO7t+/n8jISCzLonjx4sycOdMhP5e0o3RT0oLxGjkW13hrO0pXkpN2lHIamhBCGCIFWAghDJECLIQQhkgBFkLcICefC4kb5fTYSQEWQlwVFBTEyZMnpQjngmVZnDx5kqCgoGw/R05DE0JcFRISQlJSEsePHzcdxS0FBQUREhKS7cdLARZCXOXv7++QJbfi1mQKQgghDJECLIQQhkgBFkIIQ6QACyGEIVKAhRDCECnAQghhiBRgIYQwRAqwEEIYIgVYCCEMkQIshBCGSAEWQghDpAALIYQhUoCFEMIQKcBu5vTl08zfPZ/9Z/az59QeYrfGcuDMAenf6u1OnIDZs2HfPti7F6ZNgyNHTKcSdyDtKN1E0rkknpv7HP/743/6jrOgUDwx7QkAqhWrxph2Y2hUvpHBlMLpdu+GgQNhyZJr9ykFXbro27VqwTffQGSkmXzitmQE7OIsy2L8hvGEjQxj0Z5FvNHoDX7u9TMNyzakUblGJPRP4Ms2X3Il7QqNv2vMC/Ne4GLKRdOxhaOlp8Pnn0ONGrB+Pbz7LixbBo0aQcOGsHo1/Pe/ehRcty68+SYkJ5tOLW4iI2AXlp6RTo/pPYhLjKNx+caMbz+e+4rcB8C/ff4NQK17alHrnlr0iejD6z+/zpdrv2TBngUs7b2U0sGlTcYXjnLlCnToAAsXQrt2MHo0lCmj/87HNqaqV09/9e0LL70E770Hc+fCzz9D4cLmsosbyAjYRVmWxcDZA4lLjOO9pu+xpPeSq8X3VvIH5GdEmxEs6bWEw+cO02pSK05fPu3ExMIp0tKge3ddfEeNgv/971rxvZXCheG772DmTEhM1AX70iWnxRW3JwXYRQ1dPJTxv4/nzUZv8nqj1/FR2funeuTeR5gVPYs/Tv5Bux/byXSEJ7EsGDQIZsyAL77Qt5XK3nM7dIDJk/XURJcukJLi2KwiW6QAu6DPV33Oxys/5pnaz/DvR/6d4+c3q9iMHzv/yOqk1XSd0pUMK8MBKYXTvfkmjB8Pb70FL7yQ8+d37qynK+bNg6ee0gVdGCUF2MX8/ufvvLr4VR6//3G+bPMlKrsjnJt0qtaJrx/9mnm75/HF6i/snFI43S+/wPvvQ79+8M47ud9O//7w739DTAxMmmS/fCJXpAC7kOS0ZHrN7EXxvMUZ134cvj6+d7W9gbUG0r5qe/7187/YcWKHnVIKpzt3Tn+YVrmynnrI5S/lq15/HRo0gOefh6Qk+2QUuSIF2IW8s+wdth7bytjHxlIkT5G73p5SijHtxpAvIB+9Z/YmLSPNDimF0738Mhw6pD9My5v37rfn66u3lZqqR9QyFWGMFGAXsSZpDR+t+IinIp6ibZW2dttuqfylGPnoSNYeXsvHKz6223aFk8yfD2PH6iJcv779tlupEnz8MSxYAOPG2W+7IkekALuA9Ix0+v/UnzLBZfis1Wd2335UeBTdwroxbOkw9pzaY/ftCwe5fBkGDIDQ0Lub983KM89As2YwZAj89Zf9ty/uSAqwC5i4aSJbjm3hs1afUTCooEP2MbzVcPx9/XljyRsO2b5wgC+/1FMPX38NQUH2376Pj972pUt6JZ1wOinAhl1KvcRbv7xFvTL16Fyts8P2Uzq4NEMeGkJcYhzrDq9z2H6EnZw8qc96aNsWmjRx3H6qVtWj7DFjYNcux+1H3JIUYMNGrBnB4fOH+bjFx7k+5Sy7Xqn/CsXzFufVxa9K9zRX9/77cP48fPih4/f1f/8HgYH67AjhVFKADTpx6QQfLP+Ax6o8xsPlH3b4/oIDg3m78dss3b+UebvnOXx/Ipf274evvoLevSE83PH7K1lSf8g3dSqsWeP4/YmrpAAb9N6v73Eh5QIfNnfCKMdmQK0BVCpSidcWv0Z6RrrT9ity4K239Pzsv3O+CjLXhgzRhfjVV+W0NCeSAmzIXxf+YlTCKHrV7EVo8VCn7TfAN4D/PPIfth7byowdM5y2X5FNu3bpVWrPPw8hIc7bb3CwLvy//qq/hFNIATbky7VfkpKewtAGQ52+7y6hXahUpBIfr/hY5oJdzX//CwEB8M9/On/fTz0FJUrARx85f99eSgqwAeeTz/P1uq/peH9Hqhar6vT9+/r48vJDL7PuyDqW7l/q9P2LLBw9qleo9e4NpUo5f/958ugmP/PmwebNzt+/F5ICbMDYDWM5c+UMrzV4zViG3hG9KZmvJB+tkNGOyxgxQreJfPllcxmefRby5dOr5ITDSQF2spT0FD5f/TmNyzemXkg9YzmC/IJ4sd6LLNizgI1HNxrLIWzOnYORI3XLyMqVzeUoXFifFxwbCwcOmMvhJaQAO9mPW34k6VyS0dFvpmfqPENwQLD0iHAF33wDZ8/Ca+ZfF7z0ku649pn9l8WLG0kBdiLLsvh01adUL1Gd1pVam45DoaBCDKylL3t04IyMdoxJTYXhw6FpU6hd23QaKFsWevTQTXpOnTKdxqNJAXaiXw/8ytZjWxn84GCHr3rLrufrPQ/AmPVjDCfxYrNmweHDMHiw6STXvPSS7hHx/femk3g0KcBONCphFIWCChEdHm06ylXlCpajXZV2jP99PMlpctlyI0aNgvLl4dFHTSe5pmZNeOghnS1DLmnlKFKAneTohaNM2z6NvhF9yetvh6badvRs7Wc5dvEY07dPNx3F++zYAUuWwMCBulG6K3n2Wb0wZMkS00k8lhRgJxm3YRxpGWkMqj3IdJS/aXFfC+4rfB8jE0aajuJ9Ro8Gf394+mnTSf6uSxcoVkyfnSEcQgqwE6RlpPHN+m9oXrE5VYpWMR3nb3yUD4NqD2L5weVs+WuL6Tje4+JFvfCiSxe9As3VBAXp1XH/+59cO85BpAA7wZydczh07hDP1n7WdJQs9Y3oS6BvIKMSRpmO4j1iY/WpZ888YzpJ1gYO1HPAY8eaTuKRpAA7wciEkZQJLsNjVR8zHSVLRfMWJSo8ih82/8C55HOm43iHkSN1u8mGDU0nyVrFitCmjT5POTXVdBqPIwXYwQ6cOcCiPYvoF9kPPx8/03Fua1CtQVxIuUB8YrzpKJ7v999hwwY9wnSRUxKzNGiQ7lMxd67pJB5HCrCDfb/peyws+kT0MR3ljh4MeZCqRasyYeME01E834QJuutZ9+6mk9xZmza6V/AEeV3YmxRgB8qwMvhu43c0vbcpFQpVMB3njpRS9I3oy8pDK/njxB+m43iu5GTd87djRyhSxHSaO/PzgyefhDlz4Ngx02k8ihRgB/r1wK/sO7OPvhF9TUfJtidrPomP8uH7TbICymF++kkv8e3rPq8L+vaFtDSYNMl0Eo8iBdiBJmycQIHAAnSq1sl0lGy7J/geWldqzcRNE+WSRY4yYQKUKQMtWphOkn2hoVC3rs4uTfztRgqwg5xPPs/UbVOJCotyuZVvd9I3oi+Hzx9m0d5FpqN4niNHYP586NXL9Va+3UnfvrB1K6xfbzqJx5AC7CDxifFcSr3kVtMPmR6r8hhF8hSRD+Mc4Ycf9Hm1ffqYTpJz0dF6cYZ8GGc3UoAdZMLGCVQtWpUHQx40HSXHAv0C6Vm9JzN3zOTUZWlHaDeWpVe+NWwIVVxvReQdFSoEnTrB5Mlw5YrpNB5BCrAD7D29lxWHVtC7Zm+XaTuZU30i+pCSnsKUxCmmo3iOhATdfKd3b9NJcq9PHzhzRp8RIe6aFGAHiNkcA0CPGj0MJ8m9iFIRhBYPZdIW+dTbbiZNgsBA3fvBXTVtqi8YKmdD2IUUYDuzLIuYLTE8XP5hyhUsZzpOriml6FG9B8sPLmf/mf2m47i/tDTd+6FdO/1W3l35+sITT+hVcXK1jLsmBdjO1v+5nj9O/kHP6j1NR7lr3avrVVqTt0w2nMQDLF6sFzH0dP/XBT176qs3T51qOonbkwJsZzGbYwjwDaBLqBu/zbSpUKgCDcs1ZNLmSVhy7ufdiYnRI982bUwnuXsPPAD3369/JnFXpADbUVpGGrGJsbSt3JbCeQqbjmMXPar3YPuJ7XLp+rtx8SLMmAFdu+o5YHenlL5o56+/wsGDptO4NSnAdrRk3xKOXjhKzxoe8DbTpmtoV/x9/InZIqOdXJs1SxdhT5h+yJTZRGiyTE/dDSnAdhSzJYaCgQV5tLILXVzxLhXNW5Q2ldswectkWZqcWzEx+lLvrtz3N6cqVoT69fXZEDI9lWtSgO3kUuolpm+fTpfQLgT5BZmOY1c9q/fkzwt/snT/UtNR3M/x47BggR4x+njYf7eePSExETZvNp3EbXnYK8KcubvmciHlwtUzBzxJuyrtyB+Qn9itsaajuJ9p0yA93T36/uZUly76tLS4ONNJ3JYUYDuJ3RpLyXwlaVy+sekodpfHPw8d7+/ItO3TSElPMR3HvcTG6k5i1aubTmJ/xYtD8+b6Z5RpiFyRAmwH55LPMXvnbLqFdcPXx806XGVTdFg0p6+cZtEe6ZCWbYcP6zMFoqNd/7JDuRUdDfv2wdq1ppO4JSnAdjBrxyyS05OJDo82HcVhWtzXgsJBhYlNlGmIbIuP1yPDqCjTSRynY0d9aaVYeV3khhRgO4hNjKV8wfI8FPKQ6SgOE+AbQOdqnZm5YyaXUy+bjuMeYmMhMtI9O59lV6FC8Oijeh44Xc6SySkpwHfp5KWTLNyzkKiwKLftfJZd0eHRXEi5wNxdcnXcO9q7V78tj/bcd0VXRUfDn3/C8uWmk7gdKcB3afr26aRlpHn09EOmJhWaUDJfSZmGyI7MMwO6dTObwxnatYO8eWUaIhekAN+l2MRYqhStQkSpCNNRHM7Xx5euoV2ZvXM255PPm47j2mJj9UKF8uVNJ3G8fPmgfXvdnCc11XQatyIF+C4cvXCUpfuXesX0Q6Ynqj/BlbQrzPpjlukormv7dr04wRumHzJFR8OJE/Dzz6aTuBUpwHdh6rapZFgZRIV58KfcN3kw5EFCCoQQnxhvOorriovTp525c+P1nGrVCgoU0Gd+iGyTAnwX4hPjCSseRliJMNNRnMZH+dAttBvzd8/nzJUzpuO4HsvSRahxYyhd2nQa5wkKgg4ddNe3FFmsk11SgHPp8LnDLD+43KtGv5miwqNIzUhl1g6ZhvibrVv1FIQnn/ublagofb24RbJYJ7ukAOfS1G1TsbDoFuYFn3LfpM49dahQqAJxidID4G/i43XTnU6dTCdxvhYt9HnBMg2RbVKAcykuMY6aJWtStVhV01GcTilFt9BuLNq7iJOXTpqO4zosS8//Nm0KJUqYTuN8AQHw+OMwc6Zctj6bpADnwsGzB1mVtMorR7+ZuoV1Iy0jjZk7ZpqO4jo2bYJdu7zj3N+sdOsG587BwoWmk7gFKcC5MCVxCoBXF+DI0pHcV/g+mYa4Xlycbs/ojdMPmZo1gyJFpEVlNkkBzoX4bfFElo6kUpFKpqMYo5SiW1g3luxbwvGLx03HMS/z7IfmzaFoUdNpzPH317+A/vc/uCw9Q+5ECnAO7Tu9j7WH19It1HtHv5miwqJIt9KZvn266SjmrV+v+z948/RDpqgouHAB5s0zncTlSQHOocwFCN48/ZCpRskaVClaRaYhQL/l9vfXH0J5uyZNdLN2mYa4IynAORS/LZ66Zepyb+F7TUcxTilFVFgUyw4s4+iFo6bjmJM5/dCyJRQubDqNeX5+0LkzzJ6trwYtsiQFOAd2n9rNhj83eOXii6xEhUWRYWUwbds001HMWbMGDh70zsUXWYmKgkuXYM4c00lcmhTgHMicfuga2tVwEtcRViKM0OKhxG/z4pPv4+P1ObDt25tO4joaNYJSpWRRxh1IAc6BuMQ46petT9mCZU1HcSlRYVH8duA3jpw/YjqK82Vk6CLTpg0ULGg6jevw9dXNiObMgfPSujQrUoCzaceJHWz+a7Oc/XAL3cK6YWFdPT/aq6xcqS++KWc//F1UlF4R99NPppO4LCnA2RSfGI9C0SXUi1oMZtP9xe6nRska3jkNER+vO4E99pjpJK6nfn0oU0amIW5DCnA2xSXG0bBcQ8oUKGM6ikvqFtqNlYdWcujsIdNRnCc9HaZM0RelDA42ncb1+PhA1676fOCzZ02ncUlSgLNh67GtbDu+Tc5+uI2ocH1svKpR+2+/wdGjcvbD7URF6f7As6R16a1IAc6GuK1x+CgfmX64jUpFKlGrdC3vWpQRF6evh9a2rekkrqtePX1dPFmUcUtSgO/AsixiE2Npem9TSuYvaTqOS4sOj2bdkXXsObXHdBTHS03VF6Fs314XYXFrSulR8MKFcFJal95MCvAdbPhzA7tP7SY6zIsusJhLmcuzvWIUvGSJvgilN114M7eioyEtDaZLz5CbSQG+g9itsfj7+PN4NVnjfyflCpajQdkGxG6NNR3F8WJj9Xm/rVqZTuL6IiKgShV9zMQNpADfRoaVQVxiHK0qtaJIniKm47iF6PBothzbQuKxRNNRHCc5WY/mOnWCwEDTaVyfUnoU/Msv8OefptO4FCnAt7Hq0CoOnTsk0w850CW0Cz7Kx7OnIebP11d9kOmH7IuK0k2Lpk41ncSlSAG+jditsQT5BdG+qqzxz65S+UvxSIVHiEuMw7Is03EcIzYWihXT134T2RMaCjVqyDTETaQAZyEtI434bSXORK4AABkaSURBVPG0rdyW4EA5yT4nosKi2HlyJxuPbjQdxf4uXtRXe+jSRbddFNkXHa2Xbh84YDqJy5ACnIWl+5dy7OIxosPlbWZOdarWCT8fPyZvmWw6iv399JNusyjTDzmXuWBFRsFXSQHOQsyWGIIDgmlbWU6yz6mieYvSulJrYhNjybAyTMexr5gYCAnR7RZFzlSsqBdmTPbAX8y5JAX4Fi6nXmbatml0Du1MHv88puO4pR7Ve5B0LolfD/xqOor9nDihP4B74gnd50DkXI8esHkzbN1qOolLkFfRLczZNYfzKefpUb2H6Shuq33V9uTzz0fM5hjTUexnyhS9oKCHvC5yrVs33StYRsGAFOBbitkSc/XTfJE7ef3z8ni1x5m6fSrJacmm49jH5MnXPs0XuVOyJDRvro9lhodNT+WCFOCbnL58mrm75hIdFo2vj6/pOG6tR/UenLlyhnm7PeDy5AcOwPLlevSrlOk07q1HD308V60yncQ4KcA3mbZ9GinpKfSoIW8z71bzis0pka8EMVs8YBoi8y3zE0+YzeEJOnaEPHn0B5peTgrwTWK2xFClaBVqla5lOorb8/PxIyosip/++ImzV9y4Ibdl6WJRvz7ce6/pNO4vOBg6dNBXykhNNZ3GKCnA10k6l8Sy/cvoHt4dJW8z7aJ79e4kpyczfbsbd8LasgUSE6F7d9NJPEf37ro95YIFppMYJQX4OjGbY7CwZPrBjuqVqUelIpX4YfMPpqPk3sSJetWbXHjTflq10su5f3Dj14UdSAG2sSyL7zd9T4OyDahUpJLpOB5DKUWvGr34Zf8vHDjjhktQ09Jg0iR91YvixU2n8RwBAXo+fdYsOH3adBpjpADbJBxJYPuJ7fSq2ct0FI/zZM0nAdxzFLxwIfz1F/TubTqJ5+nVS7f29OKrJksBtpm4aSKBvoFXr+og7KdCoQo0Lt+YiZsmul+HtIkToUgRfeVjYV+1aunzqidONJ3EGCnAQEp6Cj9u/ZGO93ekUFAh03E8Uu+avdl1aherk1abjpJ9Z87AzJn6rbI0Xrc/pfQ7i5UrYdcu02mMkAIMzNk5h5OXT9K7przNdJQuoV3I45eH7zd9bzpK9sXH67fIMv3gOD176r4aXjoKlgIMTNw8kVL5S9Hivhamo3is4MBgOlXrRFxiHFfSrpiOkz0TJ0K1alC7tukknuuee/TS5B9+8MqlyV5fgE9cOsGcnXPoUb0Hfj7SYNuRetfszZkrZ/jpj59MR7mz3bthxQo9+pVzwh2rd2+9NPlXD+qcl01eX4AnbZ5EakaqTD84QdN7mxJSIIRvN35rOsqdTZig3xr37Gk6iefr2BEKFIBv3eB1YWdeXYAty2LchnHULVOX6iWrm47j8Xx9fOkb0ZcFuxdw8OxB03GylpamC3CbNlCmjOk0ni9vXr0ybsoU/cGnF/HqArw6aTWJxxPpH9nfdBSv8dQDTwEw4fcJhpPcxrx5+vLp/eV14TT9+8OVK17XoMerC/C4DePI55+PqLAo01G8RoVCFWhxXwu+3fgt6RnppuPc2rhxUKqUnPvrTJGR8MADMHasbn7kJby2AJ9LPkdsYixPhD8hVz12sv6R/Tl49iCL9i4yHeXvjhyBOXOgTx/w9zedxrv07w+bNsGGDaaTOI3XFuDYrbFcSr1Ev8h+pqN4nfZV21MsbzHGbRhnOsrfffcdpKfD00+bTuJ9nnhC9wkeO9Z0Eqfx2gI8bsM4wkuEU7dMXdNRvE6AbwC9a/Zm1h+zOHbxmOk412RkwPjx8MgjUEkaMjldoULQtatufn/xouk0TuGVBXjT0U2sO7KO/pH9pe+vIf0i+5GWkcb3G11oZdwvv8DevdBP3hUZ078/nD/vNQ16vLIAj1w3kiC/IHrWkHM8Tbm/2P08XP5hRq8fTYblIiugRo6EokXh8cdNJ/FeDRroBj0jR5pO4hReV4DPXDnDpC2T6B7enSJ5ipiO49Weq/Mce0/vZf7u+aajwKFDujft00/reUhhhlLw7LOQkABr15pO43BeV4C/2/gdl1Iv8Vzd50xH8XqP3/84pfOX5qu1X5mOAmPG6DngZ54xnUT06qWvG/eVC7wuHMyrCnCGlcHIdSN5KOQhIktHmo7j9fx9/RlYayDzds9j96nd5oIkJ+tP3h97DCpUMJdDaMHBugjHxcHx46bTOJRXFeBFexax69Qu/lH3H6ajCJsBtQbg5+PHqHWjzIWYOhWOHYPn5F2Ry3juOUhJ0YtiPJhXFeCv131NiXwl6Fyts+kowqZ0cGk6V+vMtxu/5WKKoVOPvv4aqlTRbRGFa6hWDZo2hdGjdW8OD+U1BXjf6X3M3jmbAZEDCPSTqxu4kn/U/Qdnrpxh8pbJzt/5hg2wapUecfl4zX8H9/CPf8DBgzB7tukkDuM1r7gRa0bg6+PLwNoDTUcRN2lQtgERpSIYvma4809J+/xzyJ9frnrhih57DMqV0/9GHsorCvDpy6cZu2Es0eHRhBQIMR1H3EQpxZCHhrDt+DbnnpJ26BDExuqT/wsWdN5+Rfb4+cHgwbpRu4eekuYVBXjM+jFcTL3Iyw+9bDqKyEJUWBQhBUL4ZOUnztvpF1/ozlsvvui8fYqc6ddP/3L89FPTSRzC4wtwcloyX6z5ghYVW1CzVE3TcUQW/H39GVxvMEv3LyXhSILjd3j2LHzzDURFQfnyjt+fyJ3gYBg0CKZN08vEPYzHF+DJWyZz9MJRXqn/iuko4g761+pPgcACfLrSCaOdMWN0z4GX5V2Ry3vhBfD1hc8+M53E7jy6AGdYGXy66lNqlqxJ84pyipGrKxBYgEG1BjFl2xT2nd7nuB2lpOjph2bNdBNw4druuUdfm+/bb+HECdNp7MqjC/DcXXPZdnwbL9d/WbqeuYkX6r2Ar/Lls1UOHO3ExOjG6zL6dR9DhsDly/qcbQ/isQXYsiz+vezfVChUQS455EbKFChDr5q9GLthLEfOH7H/DtLS4L33ICICWrWy//aFY4SFQfv2MHy4nr/3EB5bgOftnse6I+t4s9Gb+PvKpWXcyeuNXifdSuej5R/Zf+MxMbBnDwwbpjtvCffxf/+nr5o8YoTpJHbjkQXYsiyGLR1GhUIV6FWzl+k4IocqFq5I75q9GbN+jH1HwWlp8O67evTbvr39tiucIzJS/7t99pnHjII9sgDL6Nf9OWQULKNf9+dho2CPK8Ay+vUMdh8Fy+jXM3jYKNjjCvCcXXNk9OshMkfBH/z2wd1v7IcfZPTrKTJHwV98YTrJXfOoApyWkcZri1+jUpFKMvr1ABULV+SpiKcYvX40u07uyv2GLl2Ct9+G2rVl9OsJIiP1dfs++QT++st0mrviUQV4wu8T2HZ8Gx81/0hGvx5iWJNhBPoG8q+f/5X7jQwfDklJup+AjH49w4cfwpUr+h2NG/OYAnwh5QJv/fIWDco24PH75aq2nqJ0cGlebfAq07ZPY8XBFTnfwF9/wQcfQIcO0Lix/QMKM6pU0dfvGzsWtm0znSbXPKYAf7LiE/66+Bf/bflfWfXmYYY8NIR7gu9hyMIhWJaVsye/844eKX38sWPCCXPefhvy5YPXXjOdJNc8ogAfPneYT1Z+QlRYFPVC6pmOI+wsX0A+/vPIf1hzeA1Ttk3J/hO3b9cdzwYN0iMm4VmKFYM33tBXzFiyxHSaXPGIAvzq4ldJt9J5v9n7pqMIB+lVsxc1S9bk5YUvcyHlwp2fkNnnN18+PVISnumFF3Q70RdfhNRU02lyzO0L8MI9C5m8ZTL/avgvKhauaDqOcBBfH1++fvRrDp07xLClw+78hB9/hEWLdN+H4sUdnk8YEhSkF2Vs3eqW7SrdugBfTr3Ms3OepXKRygxtONR0HOFgDco1oH9kf4avHs6mo5uyfuDp0/DSS1Cnjv6gRni29u2hY0c937/PgW1MHcCtC/B7v73HntN7GN1uNEF+QabjCCf4sPmHFMlThIGzB5KekX7rB/3rX7pv7JgxupG38HwjRuh/62ef1dNPbsJtC/C249v4eMXH9KrZi6b3NjUdRzhJkTxF+LzV56w5vIYx68f8/QErV+rCO3iwNFv3JmXLwn/+A/Pnw5QcfFBrmFsW4NT0VHrP7E1wYDCftvDMi/WJrHWv3p3mFZvz2uLX2HNqz7W/uHgR+vbV/xnfecdcQGHGP/4BtWrBc8/B0aOm02SLWxbgYUuHkXAkgW/afUPxfPIBi7dRSjG+/Xh8lS89Z/TEwvaW85//hF274PvvIX9+syGF8/n6wsSJcOGC/kXsBlMRbleAfzvwGx8s/4C+EX3pHNrZdBxhSLmC5RjdbjSrk1Zz4MwBPef7zTf6MkOPPGI6njAlNFT3iJg/H0aONJ3mjtyqAJ+9cpYnZzxJxcIV+aK1+3dCEncnOjyanjV68ufJA1h/7NCtJt9913QsYdpzz0GbNvqXsYsvU3abAmxZFv1+6kfSuSQmdZpEcGCw6UjCBXzVYjihJxVWejqnx38NgYGmIwnTlNJXUM6fH6Ki9JSEi3KbAvzB8g+Yum0qHzb/kAdDHjQdR7iIgm/9h4KXLXYXhc5b3iQ13f1WQwkHKFUKJk/WI+A+fVx2PtgtCvDsnbN5c8mb9KjegyEPDTEdR7iKiRN1q8kyZShQviq/7P+FlxfKpeaFTYsWugnTtGl6RaQL8jMd4E52nNhB92ndeaD0A4x9bKx0OhPaunUwYID+wC09nVJK0bpea75Y8wURpSLo+0Bf0wmFK/jnP2HjRnjrLahRw+Ua8rv0CDjpXBKtJ7UmyC+IGVEzyOOfx3Qk4Qp27oS2baF0aYiPv9pk/dOWn9Ls3mYMnD2QebvmGQ4pXIJS+uyYWrXgiSdg1SrTiW7gsgX4+MXjtPihBacun2Jej3mUK1jOdCThCg4ehObN9e0FC3RLQhs/Hz+mdZtGeIlwOsd35rcDvxkKKVxKnjy6ZeU998Cjj8Km2/QRcTKXLMBnr5yldUxr9p/Zz+zus6l1Ty3TkYQr+OsvPa937hwsXHjLHr8Fgwoyv+d8yhUsR7sf27Hhzw0GggqXU6oULF6sz4xo2VIv2HEBLleAT146SatJrdj812amdZvGw+UfNh1JuIKkJGjaVH+fM0ef85uFEvlKsOjJRRQKKkTLH1qy9vBaJwYVLqt8ed2i1LL0ZwcucI6wSxXgQ2cP0WhCIzYe3cjUrlN5tPKjpiMJV7BjBzRoAIcO6eLboMEdn1K2YFmW9FpCwaCCNP2+KQv3LHRCUOHy7r8ffv4Z0tOhUSPjc8IuU4C3Hd9G/W/rc+T8ERY+uZAO93cwHUm4grVroWFDfV23ZcugSZNsP/W+Ivex4qkVVC5ambaT2zJ5y2TH5RTuo3p13TWvSBFo1gzmzjUWxSUK8NRtU3lw3IOkZaSxrM8ymXYQ2vjx8PDDUKAArFiRq/aSpfKXYmnvpTQo24Ae03vw6qJXSctIc0BY4VbuvVe/pqpVg8ceg/ffh4wMp8cwWoBT01P554J/0nVKV0KLh7K231pqlqppMpJwBZcvw9NPQ79++m3imjVQqVKuN1cwqCALei7gmdrP8MnKT2g2sRlHL7hHu0LhQCVK6HdVUVH64p4dOuirqTiRsQKceCyRhhMa8vnqz3m+7vP82vdXyhYsayqOcBXr1kHdunot/5tv6q5WdrimW6BfICPbjmRix4msO7yOiNERTN8+3Q6BhVvLnx9iYuCrr/RpjRER+gwbJ3F6AU5JT+Gdpe/wwJgH2HNqD3Fd4hjRZgQBvgHOjiJcyaVLunvVgw/CqVN6Xu7dd+1+SaEnaz7Jmn5rKB1cms7xnekS30VGw95OKd1B7bff9DnDrVrpfsKnTjl8104rwJZlMWP7DGqMqsGwZcPoGtaV7c9tp1tYN2dFEK4oIwMmTdJ9XP/7X+jfX58e1KaNw3ZZvWR11vZbywfNPmD2ztnc/9X9fLT8Iy6nXnbYPoUbqFdPL1t+/XX44Qd9xsRXX0FKisN26fACbFkWS/Ytof639ekU3wmlFLOfmE1Mpxi5moU3y8jQq5MiI+HJJ/Un0r/8AqNHQ8GCDt+9v68/QxsOZdOgTTQq34ihPw+l8peVGbt+LMlpyQ7fv3BRQUG6cU9CAoSFwfPP6w/qYmIgzf4f3jqsACenJTNx00Qiv4mk2cRmHDp7iHGPjWPLM1toW6Wto3YrXN2lS/qimWFh+tPn8+fhxx/1Cz4Hp5jZS9ViVfnpiZ9Y1mcZZQuWZcDsAZQfXp53l73L8YvHnZ5HuIiICFiyBObNg+Bg6NkTKlbUV9s4c8Zuu7FrAbYsi7WH1/L83OcJ+TyE3jN7k5KewtjHxrLr+V08Hfk0fj4u34BN2FtGhp5fGzAAypSBQYMgXz499bB9O0RHg4/ZMyIfLv8wK59aycKeC4ksHcnbS98m5PMQusR3YeaOmaSkO+5tqHBRSkHr1rBhA8yaBffdB6++ql/DPXroD4jvclR819XwStoVlu1fxtxdc5m9azZ7T+8l0DeQ9lXb0y+yHy0qtpAWkt7o4kU9gpg7V69eO3RIF91Ona6dXuZirwulFC3ua0GL+1qw/fh2RieMJjYxlmnbp1E4qDCtK7WmbeW2tKrUimJ5i915g8Iz+PjoNpbt28Pvv+t3cPHxuuF78eK6wc+jj+oeE4UK5WjTyspBp/jatWtb03+ezsajG1l1aBUrDq1g7eG1JKcnE+QXRNN7m9Lp/k50Ce1CwSDHz+N5sya2t+tLly41mgPQa+v379cvzpUr9Qnu69dDaqouui1aQJcu0LGj/rOdOfJYpGWksXjvYmK3xjJv9zyOXTyGQhFWIoyGZRvSoFwDHij1AFWLVXWJd3cu9brwZMnJenoiPl6PhE+f1oW6Zk29VL5BA71wqFIl8PVFKbXesqzaN28mRwXYL8TPSu+frm/7+FGrdC0alG1As4rNeKTCI9Kv14mc/h8tPR2OHtUj2YMHYc8e3VFq507YskV3KAMICIA6dfQLsEULPdJ18HXanHUsMqwM1h9Zz4I9C1h+cDmrklZxLln/3IG+gYSVCKNq0apULlKZykUrU75geUIKhFCmQBmnnWYpBdiAtDS9WGjhQli+HFav1p91AOTNC+HhqLVrb1mAc/Qru3Cewrzz6DvULFmTB0o/QF7/vPb5AUTupaXpOdb09L9/paXpr9RUfSpNaqruqZCcrL9fvqy/Ll7UFy48f14X0tOn9depU3D8OBw7pr+np9+471KloHJl/QFFzZrXvoKCzBwLB/NRPtQpU4c6ZeoAkJ6Rzrbj29h4dCOb/trE5r82syppFbFbY7G4cWBTNE9RSuQrQYl8JSiatyiFgwpTOKgwBQILEBwYTHBAMPkC8pHHLw95/fMS5BdEoF8ggb6BBPgGEOAbgL+vP34+fle/fJUvvj6++CgffJX+bmGhcK2pHY/n53dt1Av6/9yWLbrv8KZNsHlzlk/N8RREQkLC3cZ1b8eO6cn4TNk5flk95vr7M29f//3m29d9NbE9bWkOomeLvz8ULnztq0SJa18hIVC2rP667z69isgFuNqoLzktmX1n9nHw7EEOnT3EoXOHOHbx2NWvU5dPcfrKaU5fPs3lNDufezwB8gXk48JO170SsDeyyxSEUuo4cMCewXKhGHDCcAZXIcfiGjkW18ixuMZVjkV5y7L+tvAhRwXYFSilEm71m8QbybG4Ro7FNXIsrnH1Y+ES7SiFEMIbSQEWQghD3LEAf2M6gAuRY3GNHItr5Fhc49LHwu3mgIUQwlO44whYCCE8glsXYKXUEKWUpZTy2oX5SqlPlFI7lFKblVIzlFI5W4zuAZRSrZVSfyildiulhprOY4pSqqxS6hel1DalVKJS6kXTmUxSSvkqpX5XSs02nSUrbluAlVJlgZbAQdNZDFsEhFuWVQPYCfzLcB6nUkr5Al8DbYBQ4AmlVKjZVMakAUMsywoFHgSe8+JjAfAisN10iNtx2wIMfA68Cnj1JLZlWQsty8rsibcaCDGZx4C6wG7LsvZalpUCxAIdDGcywrKsPy3L2mC7fR5dfMqYTWWGUioEaAuMM53ldtyyACulOgCHLcvaZDqLi3kKmGc6hJOVAQ5d9+ckvLToXE8pVQF4AFhjNokxw9EDNOdfaz4HzPfPy4JSajFQ6hZ/9QbwOnr6wSvc7lhYljXL9pg30G9BY5yZTbgepVR+YBow2LKsc6bzOJtSqh1wzLKs9UqpJqbz3I7LFmDLsprf6n6lVHXgXmCTrdF7CLBBKVXXsiyPvLxtVscik1KqD9AOaGZ533mFh4Gy1/05xHafV1JK+aOLb4xlWdNN5zGkAdBeKfUoEAQUUEpNsiyrp+Fcf+P25wErpfYDtS3LcoWGG06nlGoNfAY0tizL6y5ippTyQ3/42AxdeNcB3S3LSjQazAClRyTfA6csyxpsOo8rsI2AX7Ysq53pLLfilnPA4gZfAcHAIqXURqXUaNOBnMn2AeQ/gAXoD53ivbH42jQAngSa2l4LG22jQOGi3H4ELIQQ7kpGwEIIYYgUYCGEMEQKsBBCGCIFWAghDJECLIQQhkgBFkIIQ6QACyGEIVKAhRDCkP8HJf2JMLsEgCcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import scipy.stats as stats\n",
    "import math\n",
    "\n",
    "mu1 = -2\n",
    "mu2 = 2\n",
    "variance = 1\n",
    "sigma = math.sqrt(variance)\n",
    "x1 = np.linspace(mu1 - 5*sigma, mu1 + 4*sigma, 100)\n",
    "x2 = np.linspace(mu2 - 5*sigma, mu2 + 4*sigma, 100)\n",
    "plt.plot(x1, stats.norm.pdf(x1, mu1, sigma)/1,color=\"green\")\n",
    "plt.plot(x2, stats.norm.pdf(x2, mu2, sigma)/1,color=\"red\")\n",
    "plt.axvline(x=-2,color=\"black\")\n",
    "plt.axvline(x=0,color=\"black\")\n",
    "plt.axvline(x=+2,color=\"black\")\n",
    "plt.text(-2.7,0.55,\"Sensative\")\n",
    "plt.text(-0.7,0.55,\"Balanced\")\n",
    "plt.text(1.7,0.55,\"Specific\")\n",
    "plt.ylim([0,0.53])\n",
    "plt.xlim([-5,5])\n",
    "plt.legend(['Negative','Positive'])\n",
    "plt.yticks([])\n",
    "#plt.set_yticklabels([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import zscore\n",
    "\n",
    "# Prepare data - apply z-score to ALL x columns\n",
    "# Only do this if you have no categoricals (and are sure you want to use z-score across the board)\n",
    "x_columns = df.columns.drop('diagnosis').drop('id')\n",
    "for col in x_columns:\n",
    "    df[col] = zscore(df[col])\n",
    "\n",
    "# Convert to numpy - Regression\n",
    "x = df[x_columns].values\n",
    "y = df['diagnosis'].map({'M':1,\"B\":0}) # Binary classification, M is 1 and B is 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "# Plot a confusion matrix.\n",
    "# cm is the confusion matrix, names are the names of the classes.\n",
    "def plot_confusion_matrix(cm, names, title='Confusion matrix', cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(names))\n",
    "    plt.xticks(tick_marks, names, rotation=45)\n",
    "    plt.yticks(tick_marks, names)\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    \n",
    "\n",
    "# Plot an ROC. pred - the predictions, y - the expected output.\n",
    "def plot_roc(pred,y):\n",
    "    fpr, tpr, _ = roc_curve(y, pred)\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "\n",
    "    plt.figure()\n",
    "    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operating Characteristic (ROC)')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Chart Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0819 12:50:06.107440 140736216040320 deprecation.py:323] From /Users/jheaton/miniconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 426 samples, validate on 143 samples\n",
      "Epoch 1/1000\n",
      "426/426 - 0s - loss: 1.7419 - accuracy: 0.6291 - val_loss: 1.0554 - val_accuracy: 0.6224\n",
      "Epoch 2/1000\n",
      "426/426 - 0s - loss: 0.8693 - accuracy: 0.6338 - val_loss: 0.6167 - val_accuracy: 0.6364\n",
      "Epoch 3/1000\n",
      "426/426 - 0s - loss: 0.5085 - accuracy: 0.7042 - val_loss: 0.2984 - val_accuracy: 0.8392\n",
      "Epoch 4/1000\n",
      "426/426 - 0s - loss: 0.2888 - accuracy: 0.8803 - val_loss: 0.1485 - val_accuracy: 0.9441\n",
      "Epoch 5/1000\n",
      "426/426 - 0s - loss: 0.1829 - accuracy: 0.9319 - val_loss: 0.1023 - val_accuracy: 0.9580\n",
      "Epoch 6/1000\n",
      "426/426 - 0s - loss: 0.1461 - accuracy: 0.9460 - val_loss: 0.0803 - val_accuracy: 0.9720\n",
      "Epoch 7/1000\n",
      "426/426 - 0s - loss: 0.0905 - accuracy: 0.9648 - val_loss: 0.0656 - val_accuracy: 0.9790\n",
      "Epoch 8/1000\n",
      "426/426 - 0s - loss: 0.0751 - accuracy: 0.9695 - val_loss: 0.0578 - val_accuracy: 0.9790\n",
      "Epoch 9/1000\n",
      "426/426 - 0s - loss: 0.0650 - accuracy: 0.9742 - val_loss: 0.0546 - val_accuracy: 0.9720\n",
      "Epoch 10/1000\n",
      "426/426 - 0s - loss: 0.0565 - accuracy: 0.9859 - val_loss: 0.0537 - val_accuracy: 0.9790\n",
      "Epoch 11/1000\n",
      "426/426 - 0s - loss: 0.0496 - accuracy: 0.9859 - val_loss: 0.0511 - val_accuracy: 0.9860\n",
      "Epoch 12/1000\n",
      "426/426 - 0s - loss: 0.0451 - accuracy: 0.9859 - val_loss: 0.0537 - val_accuracy: 0.9790\n",
      "Epoch 13/1000\n",
      "426/426 - 0s - loss: 0.0398 - accuracy: 0.9859 - val_loss: 0.0576 - val_accuracy: 0.9790\n",
      "Epoch 14/1000\n",
      "426/426 - 0s - loss: 0.0356 - accuracy: 0.9859 - val_loss: 0.0580 - val_accuracy: 0.9790\n",
      "Epoch 15/1000\n",
      "426/426 - 0s - loss: 0.0325 - accuracy: 0.9883 - val_loss: 0.0597 - val_accuracy: 0.9860\n",
      "Epoch 16/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "426/426 - 0s - loss: 0.0293 - accuracy: 0.9883 - val_loss: 0.0551 - val_accuracy: 0.9860\n",
      "Epoch 00016: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a34676898>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(1,activation='linear',kernel_initializer='random_normal'))\n",
    "model.compile(loss='binary_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "    patience=5, verbose=1, mode='auto', restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxN9f/A8dfbOtaytAg1QmbsyyRbUrKkQiiJkpQtIRKlRUmLhITkJ6lUSiVKi/S1VEhj35dUDMoSY2dmvH9/nDPjNs3cuYY7587M+/l43Mfcs7/v554573s+n3M+R1QVY4wxJjU5vA7AGGNMaLNEYYwxxi9LFMYYY/yyRGGMMcYvSxTGGGP8skRhjDHGL0sUWZyIdBSRuV7HEUpE5KiIXO3BdsNFREUkV0ZvOxhEZL2INErHcuneJ0WkmYh8kZ5l00tE8orIJhG5JCO3G0osUWQgEflDRE64B6q/RGSqiBQM5jZV9QNVbRrMbfgSkXoi8j8ROSIisSLypYhUzKjtpxDPAhF50HecqhZU1e1B2t41IjJDRPa7n3+NiPQXkZzB2F56uQmr3PmsQ1UrqeqCNLbzn+R4nvvkcOBln/WriBxz/6d2icio5GUtIreJyDJ3vgMi8oGIlEo2TwkReVtE9rj77iYReU5ECqjqKWAKMDidMWd6ligy3u2qWhCoDtQAnvA4nnRJ6VexiNQF5gKzgCuAMsBq4Odg/IIPtV/mIlIW+AXYCVRR1YuAO4EooNAF3pZnn92rbYvItcBFqro02aRq7v/UDUB74AGfZdoBHwJjgOJAJeAU8JOIFHHnKQosAfIBdVW1ENAEuBgo667qQ6CziOQN0scLbapqrwx6AX8AN/sMjwDm+AznBUYCO4C/gYlAPp/prYBVwGHgN6C5O/4i4G1gD7ALeAHI6U67H/jJff8mMDJZTLOA/u77K4DPgH3A70Afn/mGAp8C09ztP5jC5/sRmJDC+G+A99z3jYAY4Elgv1smHQMpA59lBwF/Ae8DRYCv3JgPuu9LufMPBxKAk8BRYJw7XoFy7vupwHhgDnAE50Bf1ieepsBmIBaYACxM6bO7807z/T5TmB7ubruz+/n2A0N8ptfGOWAdcr/LcUAen+kKPAxsBX53x72Ok5gOA8uB633mz+mW82/uZ1sOlAYWues65pZLe3f+23D2r0PAYqBqsn13ELAG50CbC5/92Y092o3jb2CUO36Hu62j7qsuPvukO08l4HvgH3fZJ1Mpv2eAycnGJX2X7vAnwHj3vQB/Ao8nWyYHsA543h1+AVgL5Ejj/3crcIPXxxEvXp4HkJ1eyf6xSrk75+s+00cDs4GiOL9AvwRecqfVdg9WTdwdvSQQ4U6bCbwFFAAuBZYB3d1pSf+UQEP3oCLucBHgBE6CyOEeSJ4B8gBXA9uBZu68Q4E4oLU7b75kny0/zkH5xhQ+dxdgj/u+ERAPjMJJCjfgHLAqBFAGicu+4i6bDygGtHW3XwiYAXzhs+0FJDuw899EccAt31zAB8B0d1pxnANfG3daX7cMUksUfwFd/Hz/4e62/8+NvRrOQTfSnV4LqONuKxzYCPRLFvf3btkkJs9ObhnkAga4MYS50wbi7GMVcA6a1YBiycvAHa4B7AWuw0kwnXH217w+++4qnESTz2dc4v68BLjXfV8QqJPsM+fy2db9nN0nC+EkxQFAmDt8XSrlNwMY6Oe7jHDX9ajPsAJlUljXc8AS9/1S4LkA/n9n4/PjKTu9PA8gO73cf6yjOL/uFPgBuNidJjgHTN9fs3U5+8vxLWB0Cuu8zD3Y+J55dADmu+99/ykF5xdeQ3f4IeB/7vvrgB3J1v0E8I77fiiwyM9nK+V+pogUpjUH4tz3jXAO9gV8pn8CPB1AGTQCTuMeCFOJozpw0Gd4AWknisk+01oAm9z39yUeTHzKb2fy9flMj8M9y0tleri77VI+45YBd6cyfz9gZrK4b0pjHzuIUxUDzplQq1TmS54o3gSGJZtnM+4vaHfffSCF/TkxUSzCOfgWT+Uzp5YoOgArA/z/+R7okcLnOOzuNwp8xNnk1sAd95/9BegBbHXfb02+3lS2/wHwTCCxZrWXtVFkvNbq1IE2wvnFU9wdfwnOr+LlInJIRA4B37rjwfkl91sK67sKyA3s8VnuLZwzi39RZ2+fjvPPCXAPzs6fuJ4rEtfhrudJnESUaKefz3UQOAOUSGFaCZxqlqR5VfWYz/CfOGc1aZUBwD5VPZk4ICL5ReQtEflTRA7jHLAuPsfG47983h/H+UWMG1PSZ3bLL8bPeg6Q8ucPaHtuQ/hX7oUOh4EXObt/JPrXdyAij4nIRrfh/BBONWTiMqntMym5ChiQ7PsvjVMGKW47ma7ANcAmEflVRG4LcLvnEuNBUm7rqYlThu1xfvAUcMcn7nNp7ZOBfm+FcKrlsh1LFB5R1YU4v2ZHuqP241QDVVLVi93XReo00oHzT1r2v2tiJ84ZRXGf5QqraqVUNv0R0E5ErsL5p/rMZz2/+6zjYlUtpKotfMP283mO4VQ/3JnC5Ltwzp4SFRGRAj7DVwK7AyiDlGIYgFO1cp2qFsapXgPn17/fmAOwB+dMyVmhiPgOp2AeTjVYer0JbALKu5/lSc5+jkRJn0dErgcexynfIqp6MU71ZOIyqe0zKdkJDE/2/edX1Y9S2nZyqrpVVTvg/EB5BfjU/Y7TKv+dONWcgViDk4xS2r6q6ic4++Az7ujNOIn9X/ukiOTA+Z4S98l5wB3ueH8icS7OyHYsUXhrDNBERKqp6hmcuuvRInIpgIiUFJFm7rxvA11EpLGI5HCnRajqHpwrjV4TkcLutLIickNKG1TVlTgH5MnAd6qa+AtpGXBERAaJSD4RySkild0rTQI1GOfKkD4iUkhEiojICzjVR88lm/c5EcnjHuxuA2YEUAYpKYSTXA65V688m2z63wR+IEpuDlBFRFq7V/o8DFzuZ/5ngXoi8qqIXO7GX05EponIxQFsrxBONcpREYkAegYwfzxOQ34uEXkGKOwzfTIwTETKi6OqiBRzpyUvl/8DeojIde68BUTkVhEJ6GotEekkIpe432HiPnXGje0MqX8HXwElRKSfOPcrFBKR61KZ92ucNi1/XgYeEpHL3TPAx4CnROQeEQlzv5fJOOU02l1mlDv8rvsDKnG/GyUiVROHcdqGkl9xlS1YovCQqu4D3uPsL6BBwDZgqVv1MA/n1zKqugynUXg0zq/GhTjVBeDUpecBNuCcnn+K/1PpD4Gb3b+JsSTgHLCr41zxlJhMLjqHz/MT0Ayn8XcPTpVSDaCBqm71mfUvN87dOFVfPVR1U1plkIoxOA3D+3H+ib9NNv11nDOogyIyNtDP4n6e/Ti/RkfgVE9UxLmy51Qq8/+GkxTDgfUiEotzxhaN0y6VlsdwqgOP4By4P05j/u9wPu8WnLI+yb+rh0bhtP/MxUlAb+OUFThtTu+61Ux3qWo0TpvVOJzvZhtOW0KgmuN85qM4ZX63qp5Q1eM4V5/97G6rju9CqnoE5wKN23H2i63AjSltQFVXALF+Egmquhan+nGgO/wxcC/wKM53uMEtg/qqesCd5x+gHk4b0y8icgTnbCPWLQdwvpd31bmnIttJvPrFmAwhzp2801TVXxVOSHKrJmJwLued73U82ZGINAV6qWrrDNxmXpwqp4aqujejthtKQuqGJWNCjVvt9QtO9dZAnPr/bFn9EApUdS7OGVJGbvMUzoUn2ZZVPRnjX12cq3L241SPtFbVE96GZEzGsqonY4wxftkZhTHGGL8yXRtF8eLFNTw83OswjDEmU1m+fPl+VU1XV+mZLlGEh4cTHR3tdRjGGJOpiMif6V3Wqp6MMcb4ZYnCGGOMX5YojDHG+GWJwhhjjF+WKIwxxvhlicIYY4xfQUsUIjJFRPaKyLpUpouIjBWRbSKyRkRqBisWY4wx6RfMM4qpOF0Pp+YWoLz76obz0BZjjDEhJmg33KnqIhEJ9zNLK+A99+EiS0XkYhEp4T6IJ6R8+MsOZq3a5XUYxhhzTlSVXasWsmvVwvNaj5d3Zpfk3w9ZiXHH/SdRiEg3nLMOrrzyygwJztesVbvYsOcwFUsUTntmY4wJAcf272HFx6+xZ+1iLipZ7rzWlSm68FDVScAkgKioKE+6u61YojAfd6/rxaaNMeacqCpRUVEc3r6Z1157jT59+pA7d+50r8/LRLELKO0zXModZ4wxJh0WL15MlSpVKFSoEJMnT6Z48eKULl067QXT4GWimA30FpHpwHVA7Pm2TwSrLcGqnYwxoezAgQMMHjyYyZMn8+yzzzJ06FBq1KhxwdYftEQhIh8BjYDiIhIDPAvkBlDVicDXQAuch5cfB7qc7zaD1ZZQsURhWlUveUHXaYwx50tVee+993jsscc4ePAgAwcOZODAgRd8O8G86qlDGtMVePhCb9faEowx2cWgQYN49dVXqVevHhMnTqRKlSpB2U6maMz2tX3fMdq/tSTFaVZFZIzJ6k6cOMGxY8coXrw4Xbt2pXz58nTt2pUcOYJ3W1ym68LjRFxCqtOsisgYk5V9++23VK5cme7duwNQoUIFHnrooaAmCciEZxT5cue0qiVjTLaye/du+vXrx4wZM6hQoQK9e/fO0O1nukRhjDHZyQ8//MAdd9zB6dOnGTZsGAMHDiRv3rwZGoMlCmOMCUFxcXHkzp2batWq0aJFC1544QXKlTu/O6zTK9O1URhjTFZ2+PBh+vbty/XXX09CQgLFixdn+vTpniUJsERhjDEhQVWZMWMGERERvPHGG0RFRXHq1CmvwwKs6skYYzy3b98+OnfuzDfffEONGjWYNWsW1157rddhJbEzCmOM8VjhwoXZv38/Y8aMYdmyZSGVJMAShTHGeGLRokU0a9aMo0ePkjdvXpYuXUrfvn3JlSv0KnosURhjTAbav38/Xbp04YYbbmDLli388ccfAEG/ae58hG5kxhiThagqU6ZMoUKFCkybNo0nnniC9evXU7lyZa9DS1PoneMYY0wWNW3aNCpWrMjEiROpVKmS1+EEzM4ojDEmSI4fP85TTz1FTEwMIsJnn33GwoULM1WSAEsUxhgTFF9//TWVKlVi+PDhfPnllwAUKVIkpNsiUpP5IjbGmBAWExNDu3btuPXWW8mXLx8LFy6kZ8+eXod1XixRGGPMBTR8+HDmzJnDiy++yKpVq2jYsKHXIZ03cR40l3kUvSpS//lzo9dhGGNMkmXLlpEvXz6qVKnCgQMHiI2N5eqrr/Y6rH8RkeWqGpWeZe2Mwhhj0ik2NpaHH36YOnXqMGTIEACKFSsWcknifFmiMMaYc6SqTJ8+nYiICCZOnMgjjzzCtGnTvA4raOw+CmOMOUfTpk3jvvvuIyoqiq+++opatWp5HVJQWaIwxpgAnDp1iu3btxMZGcldd91FfHw89913Hzlz5vQ6tKCzqidjjEnD/PnzqVatGs2aNePUqVPkzZuXLl26ZIskAZYojDEmVXv37uW+++7jpptuIi4ujkmTJmX486pDgVU9GWNMCrZt20bt2rU5evQoQ4YMYciQIeTLl8/rsDxhicIYY3wcPnyYwoULU7ZsWbp27coDDzxAZGSk12F5yqqejDEGOHbsGIMGDSI8PDypE79XX3012ycJsDMKY4zhyy+/pHfv3uzYsYOuXbuSP39+r0MKKZYojDHZVnx8PHfddRczZ86kUqVK/PjjjzRo0MDrsEKOVT0ZY7KdxD7ucuXKRYkSJXj55ZdZsWKFJYlUWKIwxmQrS5cuJSoqihUrVgAwfvx4Bg0aRJ48eTyOLHRZojDGZAsHDx6kZ8+e1KtXj7///puDBw96HVKmEdREISLNRWSziGwTkcEpTL9SROaLyEoRWSMiLYIZjzEme/r444+JiIhg0qRJ9OvXj40bN9K4cWOvw8o0gtaYLSI5gfFAEyAG+FVEZqvqBp/ZngI+UdU3RaQi8DUQHqyYjDHZ06ZNmwgPD+fbb7+lRo0aXoeT6QTzjKI2sE1Vt6vqaWA60CrZPAoUdt9fBOwOYjzGmGzi5MmTPPfcc0nPqn7yySdZvHixJYl0CmaiKAns9BmOccf5Ggp0EpEYnLOJR1JakYh0E5FoEYmOi4sLRqzGmCxi3rx5VK1alaFDh7Jw4UIAcufOnW068AsGrxuzOwBTVbUU0AJ4X0T+E5OqTlLVKFWNyp07d4YHaYwJfX///TcdO3akSZMmqCpz585l5MiRXoeVJQQzUewCSvsMl3LH+eoKfAKgqkuAMKB4EGMyxmRR33//PZ9++inPPPMMa9eupUmTJl6HlGUE887sX4HyIlIGJ0HcDdyTbJ4dQGNgqohE4iSKfUGMyRiThaxevZqtW7fSrl07OnbsSP369SlTpozXYWU5QTujUNV4oDfwHbAR5+qm9SLyvIi0dGcbADwkIquBj4D7NfGWSWOMScXRo0cZMGAAtWrVYvDgwcTHxyMiliSCRDLbcbnoVZH6z58bvQ7DGOORL774gkceeYSYmBi6devGSy+9RNGiRb0OK+SJyHJVjUrPstYpoDEm01i7di133HEHVapU4eOPP6ZevXpeh5QteH3VkzHG+BUXF8f//vc/AKpUqcKcOXNYvny5JYkMZInCGBOyFi9eTK1atWjSpAnbtm0DoEWLFthl8hnLEoUxJuT8888/dOvWjfr163Po0CE+//xzypUr53VY2Za1URhjQsrJkyepXr06u3fvZsCAAQwdOpSCBQt6HVa2ZonCGBMSYmJiKFWqFGFhYQwbNozq1atTrVo1r8MyWNWTMcZjJ06c4JlnnqFs2bJJnfh17tzZkkQICeiMQkTyAFeq6rYgx2OMyUbmzp1Lr169+O233+jUqRO1a9f2OiSTgjTPKETkVmAt8L07XF1EZgY7MGNM1vbII4/QrFkzcuTIwbx583j//fe57LLLvA7LpCCQM4rngeuA+QCqukpE7PIDY8w5S0hIACBnzpzUqVOH4sWLM2jQIMLCwjyOzPgTSBtFnKoeSjYuc/X7YYzx3IoVK6hbty4TJkwAoGPHjjz77LOWJDKBQBLFRhG5C8ghImVEZDSwNMhxGWOyiCNHjvDoo49y7bXXsmPHDkqUKOF1SOYcBZIoegO1gDPA58ApoG8wgzLGZA1z584lMjKS119/ne7du7Np0ybatWvndVjmHAXSRtFMVQcBgxJHiEgbnKRhjDGpypMnD5deeimfffYZ1113ndfhmHRKs5txEVmhqjWTjVuuqrWCGlkqrJtxY0JXXFwco0aN4vDhwwwfPhyAM2fOkCOH3bLltaB0My4izYDmQEkRGeUzqTBONZQxxiT56aef6NGjB+vXr+fOO+9MShCWJDI/f9/gXmAdcBJY7/OaC9wS/NCMMZnBgQMHePDBB7n++us5cuQIX375JZ988okliCwk1TMKVV0JrBSRD1T1ZAbGZIzJRA4cOMD06dN5/PHHeeaZZyhQoIDXIZkLLJDG7JIiMhyoCCRd8Kyq1wQtKmNMSNu4cSOffPIJzz77LNdccw07duywx5FmYYGcG04F3gEEp8rpE+DjIMZkjAlRx48fZ8iQIVSrVo3XX3+dmJgYAEsSWVwgiSK/qn4HoKq/qepTWBuFMdnOt99+S+XKlXnxxRe555572Lx5M6VKlfI6LJMBAql6OiUiOYDfRKQHsAsoFNywjDGh5OjRo9x7770UK1aM+fPn06hRI69DMhkokDOKR4ECQB+gPvAQ8EAwgzLGeC8hIYFp06aRkJBAwYIFmTdvHqtXr7YkkQ2leUahqr+4b48A9wKISMlgBmWM8dby5cvp3r07y5cvJ1++fLRt29YeJJSN+T2jEJFrRaS1iBR3hyuJyHvAL/6WM8ZkTrGxsfTp04fatWuza9cupk+fTps2bbwOy3gs1UQhIi8BHwAdgW9FZCjOMylWA3ZprDFZUNu2bRk3bhy9evVi06ZNtG/fHhHxOizjMX9VT62Aaqp6QkSKAjuBKqq6PWNCM8ZkhO3bt3PJJZdQqFAhhg8fTo4cObj22mu9DsuEEH9VTydV9QSAqv4DbLEkYUzWcfr0aV588UUqVarECy+8AMB1111nScL8h78ziqtFJLErcQHK+AyjqlZxaUwmtWjRInr06MHGjRtp164dffr08TokE8L8JYq2yYbHBTMQY0zGGD16NP379yc8PJw5c+bQokULr0MyIc5fp4A/ZGQgxpjgOXPmDMeOHaNQoULceuut7Nu3j6eeeor8+fN7HZrJBNJ8cFGosQcXGXNu1q9fT48ePZKeNGeyp/N5cFFQO4wXkeYisllEtonI4FTmuUtENojIehH5MJjxGJOdHD9+nCeeeILq1auzceNGbrvtNjLbD0MTGgLp6wkAEcmrqqfOYf6cwHigCRAD/Cois1V1g8885YEngPqqelBELg08dGNMalauXEmbNm34448/6NKlCyNGjKB48eJeh2UyqTTPKESktoisBba6w9VE5I0A1l0b2Kaq21X1NDAd594MXw8B41X1IICq7j2n6I0x/5J4xnDllVdy5ZVXsnDhQqZMmWJJwpyXQKqexgK3AQcAVHU1cGMAy5XEuUkvUYw7ztc1wDUi8rOILBWR5gGs1xiTTHx8PGPGjKFx48YkJCRQrFgxFi5cSMOGDb0OzWQBgSSKHKr6Z7JxCRdo+7mA8kAjoAPwfyJycfKZRKSbiESLSHRcXNwF2rQxWcOyZcuoXbs2jz76KGFhYRw+fNjrkEwWE0ii2CkitQEVkZwi0g/YEsByu4DSPsOl3HG+YoDZqhqnqr+76y2ffEWqOklVo1Q1Knfu3AFs2pis7+jRozz88MPUqVOHv//+mxkzZjBnzhyKFCnidWgmiwkkUfQE+gNXAn8DddxxafkVKC8iZUQkD3A3MDvZPF/gnE3g9lB7DWDdhBgTgNy5c7NgwQIeeeSRpDusrQM/EwyBXPUUr6p3n+uKVTVeRHoD3wE5gSmqul5EngeiVXW2O62piGzAqc4aqKoHznVbxmQX27Zt4/nnn2f8+PEUKlSI5cuXExYW5nVYJotL84Y7EfkN2Ax8DHyuqkcyIrDU2A13Jjs6deoUI0aMYPjw4eTJk4c5c+Zw/fXXex2WyUSCesOdqpYFXgBqAWtF5AsROeczDGNM+syfP59q1arxzDPP0Lp1azZt2mRJwmSogO7MVtXFqtoHqAkcxnmgkTEmyFSV4cOHExcXx7fffsv06dO54oorvA7LZDNptlGISEGcG+XuBiKBWUC9IMdlTLZ15swZ3n77bZo3b07p0qV5//33ufjii8mXL5/XoZlsKpAzinU4VzqNUNVyqjpAVe2Z2cYEwZo1a2jQoAHdunVj8uTJAJQoUcKShPFUIFc9Xa2qZ4IeiTHZ2NGjR3nuuecYPXo0RYoUYerUqdx3331eh2UM4CdRiMhrqjoA+ExE/nNplD3hzpgLZ+jQobz22ms8+OCDvPzyyxQrVszrkIxJkurlsSJSW1WXiUjjlKZ79WAjuzzWZBU7d+7k2LFjREREsH//fjZt2kSDBg28DstkUUG5PFZVl7lvI1X1B98XTqO2MSYd4uPjGTVqFJGRkXTv3h2A4sWLW5IwISuQxuwHUhjX9UIHYkx2sHTpUqKiohgwYACNGjXi3Xff9TokY9Lkr42iPc4lsWVE5HOfSYWAQ8EOzJisZs6cOdx+++1cccUVfP7557Ru3dr6ZjKZgr+rnpbhPIOiFM6T6hIdAVYGMyhjsgpVZffu3ZQsWZKbb76Z559/nr59+1KoUCGvQzMmYGn29RRqrDHbZBZbtmyhV69ebNmyhQ0bNlCwYEGvQzLZWFAas0Vkofv3oIj84/M6KCL/pDdYY7K6kydPMnToUKpUqUJ0dDRPPPGE3TBnMjV/VU+Jjzu1h+0aE6C//vqLhg0bsnXrVjp06MCoUaO4/PLLvQ7LmPPi7/LYxLuxSwM5VTUBqAt0BwpkQGzGZBqJj+i97LLLaNiwIXPnzuXDDz+0JGGyhEAuj/0C5zGoZYF3cB5V+mFQozImkzhz5gwTJ06kbNmyxMTEICJMnjyZJk2aeB2aMRdMIInijKrGAW2AN1T1UaBkcMMyJvStXr2aevXq0bNnT8qXL590VmFMVhNIoogXkTuBe4Gv3HG5gxeSMaFNVXnssceoVasW27dv5/3332fevHmUKVPG69CMCYpA78y+Eaeb8e0iUgb4KLhhGRO6RISDBw/StWtXNm/eTKdOnezGOZOlBXQfhYjkAsq5g9tUNT6oUflh91EYL/z555/07duXZ555hpo1a3LmzBly5AjoAZHGhISgPjNbRK4HtgFvA1OALSJSPz0bMyaziYuLY8SIEVSsWJHvv/+ezZs3A1iSMNlKIA8uGg20UNUNACISCbwPpCszGZNZLF68mO7du7Nu3TpatWrF2LFjufLKK70Oy5gMF0iiyJOYJABUdaOI5AliTMaEhHnz5hEbG8sXX3xBq1atvA7HGM+k2UYhIlOBk8A0d1RHIL+qdg5uaCmzNgoTLKrK+++/zyWXXMItt9zCqVOniIuLsz6aTJYQ1DYKoAewHXjcfW3HuTvbmCxj06ZN3HTTTXTu3Jl33nkHgLx581qSMIY0qp5EpApQFpipqiMyJiRjMs6JEyd48cUXeeWVVyhQoABvvfUWDz74oNdhGRNS/PUe+yRO9x0dge9FJKUn3RmTqX355Ze88MILtG/fnk2bNtGtWze7osmYZPydUXQEqqrqMRG5BPga5/JYYzK1v/76i1WrVtG8eXPuvPNOwsPDqV27ttdhGROy/P10OqWqxwBUdV8a8xoT8hISEpgwYQIVKlTg3nvv5cSJE4iIJQlj0uDvjOJqn2dlC1DW99nZqtomqJEZcwGtWLGCHj168Ouvv3LzzTczYcIEe5iQMQHylyjaJhseF8xAjAmW33//ndq1a1O8eHE+/PBD7r77buubyZhzkGqiUNUfMjIQYy4kVWXt2rVUrVqVMmXK8M4773D77bdz8cUXex2aMZmOtTuYLOf333/ntttuo0aNGqxZswaAe++915KEMekU1EQhIs1FZLOIbBORwX7maysiKiLWf5RJt9OnT/Pyyy9TqVIlFi5cyMiRI6lYsbkHsTUAABwQSURBVKLXYRmT6QXS1xMAIpJXVU+dw/w5gfFAEyAG+FVEZvv2G+XOVwjoC/wS6LqNSS4hIYF69eqxfPly2rRpw5gxYyhdurTXYRmTJQTSzXhtEVkLbHWHq4nIGwGsuzbOsyu2q+ppYDqQUs9qw4BXcPqTMuacHD58GICcOXPywAMP8OWXX/LZZ59ZkjDmAgqk6mkscBtwAEBVV+M88S4tJYGdPsMxJHvWtojUBEqr6hx/KxKRbiISLSLR9lxiA05j9dSpU7n66quZNWsWAL169eK2227zODJjsp5AEkUOVf0z2biE892wiOQARgED0ppXVSepapSqRuXObY/rzu42bNhAo0aN6NKlCxEREZQtW9brkIzJ0gJJFDtFpDagIpJTRPoBWwJYbhfge/5fyh2XqBBQGVggIn8AdYDZ1qBt/BkxYgTVqlVj3bp1TJ48mUWLFlG5cmWvwzImSwskUfQE+gNXAn/jHNB7BrDcr0B5ESnjPujobmB24kRVjVXV4qoarqrhwFKgpapGn+NnMNlA4nNTLr/8cjp27MimTZvo2rWrdeBnTAZI88FF57VykRbAGCAnMEVVh4vI80C0qs5ONu8C4LG0EoU9uCh72b17N3379uX666+nT58+XodjTKZ1Pg8uSvPyWBH5P+A/2URVu6W1rKp+jdPrrO+4Z1KZt1Fa6zPZR2IHfkOGDCEuLo569ep5HZIx2VYg91HM83kfBtzBv69mMuaCWrVqFQ8++CDLly+nadOmTJgwwRqsjfFQmolCVT/2HRaR94GfghaRyfZiY2PZvXs3H3/8MXfeead14GeMxwK+M9tHGeCyCx2Iyb5UlRkzZrB161aGDBnCDTfcwPbt2wkLC/M6NGMMgd2ZfVBE/nFfh4DvgSeCH5rJDn777TdatGhB+/btmTVrFok3VFqSMCZ0+E0U4pzzVwMucV9FVPVqVf0kI4IzWdepU6cYPnw4lStX5ueff+b1119n8eLF2A2VxoQev1VPqqoi8rWq2h1N5oLauXMnw4YN4/bbb2fMmDGULFky7YWMMZ4I5G6lVSJSI+iRmCxv3759jBvnPCixXLlybNiwgRkzZliSMCbEpZooRCTxbKMGThfhm0VkhYisFJEVGROeyQrOnDnD22+/TUREBP3792fz5s0AXH311R5HZowJhL+qp2VATaBlBsVisqB169bRs2dPfvrpJ66//nomTpxIhQoVvA7LGHMO/CUKAVDV3zIoFpPFnD59mqZNm3L69GmmTJnC/fffb/dEGJMJ+UsUl4hI/9QmquqoIMRjsoD//e9/3HDDDeTJk4dPPvmEiIgIihcv7nVYxph08teYnRMoiNMdeEovY/4lJiaGtm3b0rhxY9577z0AGjRoYEnCmEzO3xnFHlV9PsMiMZlWfHw848aN4+mnnyYhIYGXXnqJjh07eh2WMeYCSbONwpi03HvvvUyfPp1bbrmF8ePHU6ZMGa9DMsZcQKk+j0JEiqrqPxkcT5rseRSh4dChQ+TKlYuCBQvy008/8ddff9G2bVtrrDYmRJ3P8yhSbaMIxSRhvKeqTJ8+ncjISJ5++mnAaYdo166dJQljsih7jqQJ2LZt22jWrBkdOnSgVKlSdOrUyeuQjDEZwBKFCciHH35I5cqV+eWXXxg3bhxLly6lVq1aXodljMkA6XkehclG4uLiyJ07N1FRUbRr144RI0ZwxRVXeB2WMSYDpdqYHaqsMTtj7N27lwEDBnDs2DE+//xzr8MxxpynoDRmm+zpzJkzTJo0iQoVKvDxxx9TqVIlEhISvA7LGOMhq3oySbZv306nTp1YsmQJjRo14s033yQiIsLrsIwxHrNEYZJcdNFFHDp0iHfffZd7773XLnc1xgBW9ZTtzZ49mzZt2pCQkECxYsVYt24d9913nyUJY0wSSxTZ1I4dO2jdujWtWrViy5Yt7NmzB4AcOWyXMMb8mx0Vspn4+HhGjhxJZGQkc+fO5ZVXXmHlypWUKlXK69CMMSHK2iiymYSEBCZPnsxNN93EG2+8QXh4uNchGWNCnJ1RZAMHDx5k0KBBHDlyhLx58/Lzzz8ze/ZsSxLGmIBYosjCVJUPPviAiIgIXnvtNebPnw9AsWLFrLHaGBMwSxRZ1JYtW2jSpAmdOnUiPDyc6OhoWrZs6XVYxphMyNoosqh+/foRHR3NhAkT6NatGzlz5vQ6JGNMJmWJIgv5/vvviYiIoHTp0rz55pvkzZuXyy+/3OuwjDGZXFCrnkSkuYhsFpFtIjI4hen9RWSDiKwRkR9E5KpgxpNV/fXXX9xzzz00bdqUV155BYCrrrrKkoQx5oIIWqIQkZzAeOAWoCLQQUQqJpttJRClqlWBT4ERwYonKzpz5gwTJ04kIiKCzz77jGeffZaRI0d6HZYxJosJ5hlFbWCbqm5X1dPAdKCV7wyqOl9Vj7uDSwG76+scvPTSS/Ts2ZNatWqxZs0ahg4dSlhYmNdhGWOymGC2UZQEdvoMxwDX+Zm/K/BNShNEpBvQDaBgibIXKr5M6ciRI+zfv58yZcrQo0cPypQpQ4cOHexyV2NM0ITE5bEi0gmIAl5NabqqTlLVKFWNyp07d8YGFyJUlZkzZ1KxYkXat2+PqlKsWDHuueceSxLGmKAKZqLYBZT2GS7ljvsXEbkZGAK0VNVTQYwn0/rzzz9p2bIlbdq0oWjRoowdO9aSgzEmwwSz6ulXoLyIlMFJEHcD9/jOICI1gLeA5qq6N4ixZFpLlizh5ptvBmDkyJH07duXXLnsqmZjTMYJ2hmFqsYDvYHvgI3AJ6q6XkSeF5HEW4RfBQoCM0RklYjMDlY8mc3hw4cBqFmzJg888AAbN25kwIABliSMMRlOVNXrGM5J0asi9Z8/N3odRtAcOHCAwYMHM3fuXNavX0/BggW9DskYkwWIyHJVjUrPsiHRmG2cxur33nuPiIgI3nnnHdq3b2/tEMaYkGD1GCEgNjaW1q1bs2DBAurWrcvEiROpWrWq12EZYwxgicJTqoqIULhwYYoXL86kSZPo2rWrPY7UGBNS7Ijkke+++46aNWsSExODiDBjxgweeughSxLGmJBjR6UMtmfPHu6++26aN2/O8ePH2bvXrgo2xoQ2SxQZaPz48URERPDFF1/w3HPPsWbNGmrWrOl1WMYY45e1UWSg5cuXc9111zF+/HjKly/vdTjGGBMQO6MIosOHD9OvXz+WL18OwIQJE/juu+8sSRhjMhVLFEGgqnz66adERkYyduxYFi5cCEBYWJjdG2GMyXQsUVxgv//+O7fddht33nknl156KUuWLKF///5eh2WMMelmieIC++CDD1i0aBGjR4/m119/5brr/D2CwxhjQp/19XQB/Pjjj5w6dYqbb76ZU6dOsW/fPkqVsof1GWNCh/X15JH9+/fzwAMP0LBhQ55//nkA8ubNa0nCGJOl2OWx6aCqTJ06lYEDBxIbG8ugQYN4+umnvQ7LhJi4uDhiYmI4efKk16GYbCQsLIxSpUpxIZ8GaokiHb7++mseeOAB6tevz8SJE6lcubLXIZkQFBMTQ6FChQgPD7er3UyGUFUOHDhATEwMZcqUuWDrtaqnAB0/fpyff/4ZgBYtWjBr1iwWLVpkScKk6uTJkxQrVsyShMkwIkKxYsUu+FmsJYoAfPPNN1SuXJlbbrmFQ4cOISK0bNnSOvAzabIkYTJaMPY5O9L5sWvXLu68805atGhB3rx5+fLLL7n44ou9DssYYzKUJYpU7N27l4oVK/LVV1/xwgsvsHr1am644QavwzLmnOTMmZPq1atTuXJlbr/9dg4dOpQ0bf369dx0001UqFCB8uXLM2zYMHwvl//mm2+IioqiYsWK1KhRgwEDBnjxEfxauXIlXbt29TqMVB04cIAbb7yRggUL0rt371Tn++eff2jSpAnly5enSZMmHDx4EHDaHPr06UO5cuWoWrUqK1asAGDfvn00b948Qz4DWKL4j127dgFw6aWXMmzYMNatW8eQIUPIkyePx5EZc+7y5cvHqlWrWLduHUWLFmX8+PEAnDhxgpYtWzJ48GA2b97M6tWrWbx4MRMmTABg3bp19O7dm2nTprFhwwaio6MpV67cBY0tPj7+vNfx4osv0qdPnwzd5rkICwtj2LBhjBw50u98L7/8Mo0bN2br1q00btyYl19+GXCS9datW9m6dSuTJk2iZ8+eAFxyySWUKFEiqd002OyqJ1dsbCxPPfUUb731FkuXLqVmzZrntAMa489zX65nw+7DF3SdFa8ozLO3Vwp4/rp167JmzRoAPvzwQ+rXr0/Tpk0ByJ8/P+PGjaNRo0Y8/PDDjBgxgiFDhhAREQE4ZyaJBylfR48e5ZFHHiE6OhoR4dlnn6Vt27YULFiQo0ePAvDpp5/y1VdfMXXqVO6//37CwsJYuXIl9evX5/PPP2fVqlVJVbrly5fnp59+IkeOHPTo0YMdO3YAMGbMGOrXr/+vbR85coQ1a9ZQrVo1AJYtW0bfvn05efIk+fLl45133qFChQpMnTqVzz//nKNHj5KQkMDChQt59dVX+eSTTzh16hR33HEHzz33HACtW7dm586dnDx5kr59+9KtW7eAyzclBQoUoEGDBmzbts3vfLNmzWLBggUAdO7cmUaNGvHKK68wa9Ys7rvvPkSEOnXqcOjQIfbs2UOJEiVo3bo1H3zwwX/KJRiyfaJQVWbMmEG/fv3466+/6N27N2XLlvU6LGMuqISEBH744Yekapr169dTq1atf81TtmxZjh49yuHDh1m3bl1AVU3Dhg3joosuYu3atQBJVSb+xMTEsHjxYnLmzElCQgIzZ86kS5cu/PLLL1x11VVcdtll3HPPPTz66KM0aNCAHTt20KxZMzZu/HePDNHR0f+66jAiIoIff/yRXLlyMW/ePJ588kk+++wzAFasWMGaNWsoWrQoc+fOZevWrSxbtgxVpWXLlixatIiGDRsyZcoUihYtyokTJ7j22mtp27YtxYoV+9d2H330UebPn/+fz3X33XczePDgND9/Sv7++29KlCgBwOWXX87ff/8NODUcpUuXTpqvVKlS7Nq1ixIlShAVFcVTTz2Vru2dq2ydKFSVNm3a8MUXX1CzZk1mz55NVFS67nA3xq9z+eV/IZ04cYLq1auza9cuIiMjadKkyQVd/7x585g+fXrScJEiRdJc5s477yRnzpwAtG/fnueff54uXbowffp02rdvn7TeDRs2JC1z+PBhjh49SsGCBZPG7dmzh0suuSRpODY2ls6dO7N161ZEhLi4uKRpTZo0oWjRogDMnTuXuXPnUqNGDcA5K9q6dSsNGzZk7NixzJw5E4CdO3eydevW/ySK0aNHB1Y46SQiAV25dOmll7J79+6gxpIoWyaKuLg4cufOjYjQoEEDbrrpJnr16pW08xqTVSS2URw/fpxmzZoxfvx4+vTpQ8WKFVm0aNG/5t2+fTsFCxakcOHCVKpUieXLlydV65wr3wNd8mv6CxQokPS+bt26bNu2jX379vHFF18k/UI+c+YMS5cuJSwszO9n8133008/zY033sjMmTP5448/aNSoUYrbVFWeeOIJunfv/q/1LViwgHnz5rFkyRLy589Po0aNUrwfIRhnFJdddllSldKePXu49NJLAShZsiQ7d+5Mmi8mJoaSJUsCJFWxZYRs15i9YMECqlatyqxZswAYMGAAjzzyiCUJk6Xlz5+fsWPH8tprrxEfH0/Hjh356aefmDdvHuCcefTp04fHH38cgIEDB/Liiy+yZcsWwDlwT5w48T/rbdKkSVIDOZyterrsssvYuHEjZ86cSfqFnhIR4Y477qB///5ERkYm/Xpv2rQpb7zxRtJ8q1at+s+ykZGR/6r7j42NTTqITp06NdVtNmvWjClTpiS1oezatYu9e/cSGxtLkSJFyJ8/P5s2bWLp0qUpLj969GhWrVr1n1d6kwRAy5YteffddwF49913adWqVdL49957D1Vl6dKlXHTRRUlVVFu2bMmwG36zTaLYt28fnTt35sYbb+TUqVMUKlTI65CMyVA1atSgatWqfPTRR+TLl49Zs2bxwgsvUKFCBapUqcK1116bdAln1apVGTNmDB06dCAyMpLKlSuzffv2/6zzqaee4uDBg1SuXJlq1aol/dJ++eWXue2226hXr17SgS017du3Z9q0aUnVTgBjx44lOjqaqlWrUrFixRSTVEREBLGxsRw5cgSAxx9/nCeeeIIaNWr4vbqpadOm3HPPPdStW5cqVarQrl07jhw5QvPmzYmPjycyMpLBgwdTp06dtAs1AOHh4fTv35+pU6dSqlSppCq1Bx98kOjoaAAGDx7M999/T/ny5Zk3b15S0mnRogVXX3015cqV46GHHkq6Kg1g/vz53HrrrRckxrRki27GP/roIx5++GGOHj3KwIEDGTJkCPnz5w9ShMY4Nm7cSGRkpNdhZGmjR4+mUKFCPPjgg16HkuEaNmzIrFmzUmwXSmnfs27G0xAfH0/lypVZtWoVw4cPtyRhTBbRs2dP8ubN63UYGW7fvn30798/oIsHLoQseUZx7Ngxhg0bxpVXXkmvXr2S7ja1fndMRrIzCuMVO6NIw1dffUWlSpV45ZVXkhriAr3czJgLLbP9EDOZXzD2uSyTKGJiYmjTpg233347BQoUYNGiRYwZM8brsEw2FhYWxoEDByxZmAyT+DwKf5cVp0eWuY9i+/btfPfdd7z00kv079/f+mYynitVqhQxMTHs27fP61BMNpL4hLsLKVO3USxbtowlS5bQt29fwOmpMfldlMYYY0K4jUJEmovIZhHZJiL/uRtFRPKKyMfu9F9EJDyQ9R46dIhevXpRp04dRo0axbFjxwAsSRhjTBAELVGISE5gPHALUBHoICIVk83WFTioquWA0cAraa339PFYIiIieOutt+jTpw9r16791+35xhhjLqxgnlHUBrap6nZVPQ1MB1olm6cV8K77/lOgsaRxedKx/X9RunRpfv31V8aMGUPhwoUveODGGGPOCmZjdklgp89wDHBdavOoaryIxALFgP2+M4lINyCxY/hT0dHR65J3kZxNFSdZWWVjVhZnWVmcZWVxVoX0LpgprnpS1UnAJAARiU5vg0xWY2VxlpXFWVYWZ1lZnCUi0eldNphVT7uA0j7DpdxxKc4jIrmAi4ADQYzJGGPMOQpmovgVKC8iZUQkD3A3MDvZPLOBzu77dsD/NLNdr2uMMVlc0Kqe3DaH3sB3QE5giqquF5HngWhVnQ28DbwvItuAf3CSSVomBSvmTMjK4iwri7OsLM6ysjgr3WWR6W64M8YYk7GyTF9PxhhjgsMShTHGGL9CNlEEq/uPzCiAsugvIhtEZI2I/CAiV3kRZ0ZIqyx85msrIioiWfbSyEDKQkTucveN9SLyYUbHmFEC+B+5UkTmi8hK9/+khRdxBpuITBGRvSKyLpXpIiJj3XJaIyI1A1qxqobcC6fx+zfgaiAPsBqomGyeXsBE9/3dwMdex+1hWdwI5Hff98zOZeHOVwhYBCwForyO28P9ojywEijiDl/qddwelsUkoKf7viLwh9dxB6ksGgI1gXWpTG8BfAMIUAf4JZD1huoZRVC6/8ik0iwLVZ2vqsfdwaU496xkRYHsFwDDcPoNO5mRwWWwQMriIWC8qh4EUNW9GRxjRgmkLBRI7O/nImB3BsaXYVR1Ec4VpKlpBbynjqXAxSJSIq31hmqiSKn7j5KpzaOq8UBi9x9ZTSBl4asrzi+GrCjNsnBPpUur6pyMDMwDgewX1wDXiMjPIrJURJpnWHQZK5CyGAp0EpEY4GvgkYwJLeSc6/EEyCRdeJjAiEgnIAq4wetYvCAiOYBRwP0ehxIqcuFUPzXCOctcJCJVVPWQp1F5owMwVVVfE5G6OPdvVVbVM14HlhmE6hmFdf9xViBlgYjcDAwBWqrqqQyKLaOlVRaFgMrAAhH5A6cOdnYWbdAOZL+IAWarapyq/g5swUkcWU0gZdEV+ARAVZcAYTgdBmY3AR1PkgvVRGHdf5yVZlmISA3gLZwkkVXroSGNslDVWFUtrqrhqhqO017TUlXT3RlaCAvkf+QLnLMJRKQ4TlXU9owMMoMEUhY7gMYAIhKJkyiy4zNqZwP3uVc/1QFiVXVPWguFZNWTBq/7j0wnwLJ4FSgIzHDb83eoakvPgg6SAMsiWwiwLL4DmorIBiABGKiqWe6sO8CyGAD8n4g8itOwfX9W/GEpIh/h/Dgo7rbHPAvkBlDViTjtMy2AbcBxoEtA682CZWWMMeYCCtWqJ2OMMSHCEoUxxhi/LFEYY4zxyxKFMcYYvyxRGGOM8csShQk5IpIgIqt8XuF+5g1PrafMc9zmArf30dVulxcV0rGOHiJyn/v+fhG5wmfaZBGpeIHj/FVEqgewTD8RyX++2zbZlyUKE4pOqGp1n9cfGbTdjqpaDaezyVfPdWFVnaiq77mD9wNX+Ex7UFU3XJAoz8Y5gcDi7AdYojDpZonCZArumcOPIrLCfdVLYZ5KIrLMPQtZIyLl3fGdfMa/JSI509jcIqCcu2xj9xkGa92+/vO641+Ws88AGemOGyoij4lIO5w+tz5wt5nPPROIcs86kg7u7pnHuHTGuQSfDt1E5E0RiRbn2RPPueP64CSs+SIy3x3XVESWuOU4Q0QKprEdk81ZojChKJ9PtdNMd9xeoImq1gTaA2NTWK4H8LqqVsc5UMe43TW0B+q74xOAjmls/3ZgrYiEAVOB9qpaBacng54iUgy4A6ikqlWBF3wXVtVPgWicX/7VVfWEz+TP3GUTtQempzPO5jjddCQaoqpRQFXgBhGpqqpjcbrUvlFVb3S78ngKuNkty2igfxrbMdlcSHbhYbK9E+7B0lduYJxbJ5+A029RckuAISJSCvhcVbeKSGOgFvCr271JPpykk5IPROQE8AdON9QVgN9VdYs7/V3gYWAczrMu3haRr4CvAv1gqrpPRLa7/exsBSKAn931nkuceXC6bfEtp7tEpBvO/3UJnAf0rEm2bB13/M/udvLglJsxqbJEYTKLR4G/gWo4Z8L/eSiRqn4oIr8AtwJfi0h3nCd5vauqTwSwjY6+HQiKSNGUZnL7FqqN08lcO6A3cNM5fJbpwF3AJmCmqqo4R+2A4wSW47RPvAG0EZEywGPAtap6UESm4nR8l5wA36tqh3OI12RzVvVkMouLgD3u8wPuxen87V9E5Gpgu1vdMgunCuYHoJ2IXOrOU1QCf6b4ZiBcRMq5w/cCC906/YtU9WucBFYthWWP4HR7npKZOE8a64CTNDjXON0O7Z4G6ohIBM7T244BsSJyGXBLKrEsBeonfiYRKSAiKZ2dGZPEEoXJLCYAnUVkNU51zbEU5rkLWCciq3CeS/Gee6XRU8BcEVkDfI9TLZMmVT2J07vmDBFZC5wBJuIcdL9y1/cTKdfxTwUmJjZmJ1vvQWAjcJWqLnPHnXOcbtvHazi9wq7GeT72JuBDnOqsRJOAb0Vkvqruw7ki6yN3O0twytOYVFnvscYYY/yyMwpjjDF+WaIwxhjjlyUKY4wxflmiMMYY45clCmOMMX5ZojDGGOOXJQpjjDF+/T+5PxLenoS+ZwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred = model.predict(x_test)\n",
    "plot_roc(pred,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiclass Classification Error Metrics\n",
    "\n",
    "The following sections will examine several metrics for evaluating classification error. The following classification neural network will be used to evaluate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1500 samples, validate on 500 samples\n",
      "Epoch 1/1000\n",
      "1500/1500 - 0s - loss: 1.5661 - accuracy: 0.4653 - val_loss: 1.1527 - val_accuracy: 0.4980\n",
      "Epoch 2/1000\n",
      "1500/1500 - 0s - loss: 1.1592 - accuracy: 0.4760 - val_loss: 1.1132 - val_accuracy: 0.4980\n",
      "Epoch 3/1000\n",
      "1500/1500 - 0s - loss: 1.1098 - accuracy: 0.4827 - val_loss: 1.0536 - val_accuracy: 0.5120\n",
      "Epoch 4/1000\n",
      "1500/1500 - 0s - loss: 0.9767 - accuracy: 0.5740 - val_loss: 0.9175 - val_accuracy: 0.6620\n",
      "Epoch 5/1000\n",
      "1500/1500 - 0s - loss: 0.8409 - accuracy: 0.6653 - val_loss: 0.8311 - val_accuracy: 0.6800\n",
      "Epoch 6/1000\n",
      "1500/1500 - 0s - loss: 0.7969 - accuracy: 0.6733 - val_loss: 0.8034 - val_accuracy: 0.6800\n",
      "Epoch 7/1000\n",
      "1500/1500 - 0s - loss: 0.7657 - accuracy: 0.6853 - val_loss: 0.7992 - val_accuracy: 0.6640\n",
      "Epoch 8/1000\n",
      "1500/1500 - 0s - loss: 0.7487 - accuracy: 0.6920 - val_loss: 0.7890 - val_accuracy: 0.6980\n",
      "Epoch 9/1000\n",
      "1500/1500 - 0s - loss: 0.7407 - accuracy: 0.6987 - val_loss: 0.7860 - val_accuracy: 0.6760\n",
      "Epoch 10/1000\n",
      "1500/1500 - 0s - loss: 0.7155 - accuracy: 0.6987 - val_loss: 0.7697 - val_accuracy: 0.6860\n",
      "Epoch 11/1000\n",
      "1500/1500 - 0s - loss: 0.7265 - accuracy: 0.6867 - val_loss: 0.7623 - val_accuracy: 0.6900\n",
      "Epoch 12/1000\n",
      "1500/1500 - 0s - loss: 0.6955 - accuracy: 0.7053 - val_loss: 0.7550 - val_accuracy: 0.6900\n",
      "Epoch 13/1000\n",
      "1500/1500 - 0s - loss: 0.6999 - accuracy: 0.6867 - val_loss: 0.7461 - val_accuracy: 0.7020\n",
      "Epoch 14/1000\n",
      "1500/1500 - 0s - loss: 0.6947 - accuracy: 0.6913 - val_loss: 0.7478 - val_accuracy: 0.6900\n",
      "Epoch 15/1000\n",
      "1500/1500 - 0s - loss: 0.6940 - accuracy: 0.6933 - val_loss: 0.7545 - val_accuracy: 0.6740\n",
      "Epoch 16/1000\n",
      "1500/1500 - 0s - loss: 0.6762 - accuracy: 0.7027 - val_loss: 0.7404 - val_accuracy: 0.6940\n",
      "Epoch 17/1000\n",
      "1500/1500 - 0s - loss: 0.6731 - accuracy: 0.6967 - val_loss: 0.7364 - val_accuracy: 0.7000\n",
      "Epoch 18/1000\n",
      "1500/1500 - 0s - loss: 0.6575 - accuracy: 0.7100 - val_loss: 0.7423 - val_accuracy: 0.7000\n",
      "Epoch 19/1000\n",
      "1500/1500 - 0s - loss: 0.6598 - accuracy: 0.7133 - val_loss: 0.7700 - val_accuracy: 0.6540\n",
      "Epoch 20/1000\n",
      "1500/1500 - 0s - loss: 0.6604 - accuracy: 0.7173 - val_loss: 0.7398 - val_accuracy: 0.6980\n",
      "Epoch 21/1000\n",
      "1500/1500 - 0s - loss: 0.6573 - accuracy: 0.7193 - val_loss: 0.7393 - val_accuracy: 0.6940\n",
      "Epoch 22/1000\n",
      "1500/1500 - 0s - loss: 0.6460 - accuracy: 0.7160 - val_loss: 0.7284 - val_accuracy: 0.7080\n",
      "Epoch 23/1000\n",
      "1500/1500 - 0s - loss: 0.6418 - accuracy: 0.7267 - val_loss: 0.7331 - val_accuracy: 0.7040\n",
      "Epoch 24/1000\n",
      "1500/1500 - 0s - loss: 0.6407 - accuracy: 0.7240 - val_loss: 0.7428 - val_accuracy: 0.6880\n",
      "Epoch 25/1000\n",
      "1500/1500 - 0s - loss: 0.6397 - accuracy: 0.7200 - val_loss: 0.7909 - val_accuracy: 0.6560\n",
      "Epoch 26/1000\n",
      "1500/1500 - 0s - loss: 0.6269 - accuracy: 0.7360 - val_loss: 0.7293 - val_accuracy: 0.7080\n",
      "Epoch 27/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "1500/1500 - 0s - loss: 0.6280 - accuracy: 0.7300 - val_loss: 0.7296 - val_accuracy: 0.6940\n",
      "Epoch 00027: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a35c42f60>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(y.shape[1],activation='softmax',kernel_initializer='random_normal'))\n",
    "model.compile(loss='categorical_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, \n",
    "                        verbose=1, mode='auto', restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Accuracy\n",
    " \n",
    "Accuracy is the number of rows where the neural network correctly predicted the target class.  Accuracy is only used for classification, not regression.\n",
    "\n",
    "$ accuracy = \\frac{c}{N} $\n",
    "\n",
    "Where $c$ is the number correct and $N$ is the size of the evaluated set (training or validation). Higher accuracy numbers are desired.\n",
    "\n",
    "As we just saw, by default, Keras will return the percent probability for each class. We can change these prediction probabilities into the actual iris predicted with **argmax**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(x_test)\n",
    "pred = np.argmax(pred,axis=1) # raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the actual iris flower predicted, we can calculate the percent accuracy (how many were correctly classified)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score: 0.708\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "y_compare = np.argmax(y_test,axis=1) \n",
    "score = metrics.accuracy_score(y_compare, pred)\n",
    "print(\"Accuracy score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Log Loss\n",
    "\n",
    "Accuracy is like a final exam with no partial credit.  However, neural networks can predict a probability of each of the target classes.  Neural networks will give high probabilities to predictions that are more likely.  Log loss is an error metric that penalizes confidence in wrong answers. Lower log loss values are desired.\n",
    "\n",
    "The following code shows the output of predict_proba:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy array of predictions\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.    , 0.0806, 0.7416, 0.1773, 0.0006, 0.    , 0.    ],\n",
       "       [0.    , 0.7878, 0.2105, 0.    , 0.0017, 0.    , 0.    ],\n",
       "       [0.    , 0.6929, 0.3035, 0.    , 0.0035, 0.0001, 0.    ],\n",
       "       [0.    , 0.2849, 0.7054, 0.0078, 0.0018, 0.    , 0.    ],\n",
       "       [0.    , 0.0523, 0.7445, 0.2029, 0.0002, 0.    , 0.    ]],\n",
       "      dtype=float32)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As percent probability\n",
      "[ 0.0001  8.0554 74.1616 17.7251  0.0571  0.0007  0.    ]\n",
      "Log loss score: 0.7284428394883871\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "\n",
    "# Don't display numpy in scientific notation\n",
    "np.set_printoptions(precision=4)\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "# Generate predictions\n",
    "pred = model.predict(x_test)\n",
    "\n",
    "print(\"Numpy array of predictions\")\n",
    "display(pred[0:5])\n",
    "\n",
    "print(\"As percent probability\")\n",
    "print(pred[0]*100)\n",
    "\n",
    "score = metrics.log_loss(y_test, pred)\n",
    "print(\"Log loss score: {}\".format(score))\n",
    "\n",
    "pred = np.argmax(pred,axis=1) # raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Log loss](https://www.kaggle.com/wiki/LogarithmicLoss) is calculated as follows:\n",
    "\n",
    "$ \\text{log loss} = -\\frac{1}{N}\\sum_{i=1}^N {( {y}_i\\log(\\hat{y}_i) + (1 - {y}_i)\\log(1 - \\hat{y}_i))} $\n",
    "\n",
    "The log function is useful to penalizing wrong answers.  The following code demonstrates the utility of the log function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jheaton/miniconda3/envs/tensorflow/lib/python3.6/site-packages/ipykernel_launcher.py:12: RuntimeWarning: divide by zero encountered in log\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAE0CAYAAAASSJRcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5icdX338c93z+dD9phzQkgIOQCGDSdRNhARWpF6fKgVlUdFsWppq9YWz61tpSfbq31auap9bKtGKh54FBFQArYQQhIOSQghJxKSbLKbTbLn4+z3+WNmw5Bssvfs7Ow9s/N+XddcM3Pfc/jON/fOfvLb3/zG3F0AAAAAJldO2AUAAAAA0xFBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoA0CaM7OXzWztJDzOm83sxwFvu9HMlif7nACQzQjaAJA9virpLwPe9q8lfSWFtQDAtEfQBoAsYGarJVW6+4aAd7lf0hoza0xhWQAwrRG0ASBDmFmhmX3dzA7HTl83s8K4/Z8xs5bYvg+ZmZvZ+bHdN0p6LO62V5nZMTObG7t+sZmdMLOlkuTu/ZI2S3rz1L1CAJheCNoAkDnuknSFpEskXSzpMkmfkyQzu0HSH0haK+l8Sc2n3XelpJ2jV9z9CUnfkPRtMyuW9J+SPu/uL8bdZ0fseQAAE0DQBoDM8TuSvuLure7eJunLkm6N7Xu3pH9z9+3u3ivpS6fdt0pS12nbviSpUtJGSYck/dNp+7ti9wMATABBGwAyxyxJ++Ou749tG933Sty++MuSdEJSefwGdx+S9H8lrZD0N+7up92nXNLJ5EoGgOxF0AaAzHFY0vy46/Ni2ySpRdKcuH1zT7vv85KWxG8ws9mSvijp3yT9Tfx875gLJT2XZM0AkLUI2gCQOb4n6XNmVmdmtZK+oOjcakm6V9JtZnahmZVI+vxp931A0jWjV8zMFB3N/qakDyoa1P80bn+RpEslPZyalwIA0x9BGwAyx59J2qTo6PRWSVti2+TuP5f0D5IelbRb0ugyfgOx/VskdZjZ5bHtn5RUr+gHIF3SbYoG9TfE9t8kab27j46YAwASZGdOyQMAZDozu1DSNkmF7j4c23a9pI+5+28FuP9Tkj7o7ttSWykATF8EbQCYJszsbYpOESmR9G1JI0FCNQAgNZg6AgDTx0cktUraIyki6Y5wywGA7MaINgAAAJACjGgDAAAAKUDQBgAAAFIgL+wCElFbW+sLFiwI7fl7enpUWloa2vNnOvo3cfQuOfQvOfRv4uhdcuhfcuhfcjZv3nzM3euSeYyMCtoLFizQpk2bQnv+9evXq7m5ObTnz3T0b+LoXXLoX3Lo38TRu+TQv+TQv+SY2f5kH4OpIwAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKhBq0zewGM9tpZrvN7LNh1gIAAABMptCCtpnlSvonSTdKWibpt81sWVj1AAAAAJMpzBHtyyTtdve97j4oaZ2km0OsBwBwmsHhEQ2NeNhlAEBGygvxuWdLeiXu+kFJl4dUCwBkhZER18m+IbV3D+hY96DaewbU3j0Yvd4TPW/vHlR77HJn/7Dee2GB3hR24QCQgcw9nJEKM3unpBvc/UOx67dKutzdP37a7W6XdLskNTQ0XLpu3bopr3VUd3e3ysrKQnv+TEf/Jo7eJWc698/d1R+RugZdnQOurqHoeedg9NQVO49ui95urHd9k1RWIFUUmCoKTOWxU0WBaVHpoFbMnJ79S7XpfOxNBfqXHPqXnDVr1mx296ZkHiPMEe1DkubGXZ8T2/Ya7n6PpHskqampyZubm6ekuLGsX79eYT5/pqN/E0fvkpNp/XN3dfYNq627X61dA2obPXW/OvocHXEe1LHuAQ0Mj4z5OGWFeaopK1RNaYFmNxSqtqxANaWFqikrUE1ZoWpLo+c1ZQWqLilQbo6N+TiZ1r90Qu+SQ/+SQ//CF2bQflrSYjNbqGjAvkXSe0KsBwBSqn8ociowt3UNvDZEx7Yfi10ejJwZnvNzTbWxYFxTWqjz68ui1+MC86nLpQUqys8N4VUCAEaFFrTdfdjMPi7pF5JyJX3L3beHVQ8ATMTIiOt476BaOwfU2tX/miAdH6DbugbU1T98xv3NpJrSAtWWFaquvFCL6kpVV16outj1uvJC1ZcXqq6sSBXFeTIbe9QZAJB+whzRlrs/IOmBMGsAgLG4u7oGhtXa2a8jHQM62tmvo139OtrRr6OdA6cut3YNaHiMVTnKCvNOBeYLGyv0xsWFZwTouvLoyHNeLt8dBgDTUahBGwDCMDqF40hnv4529utILDCffrl3MHLGfcuL8tRYUaSGiiJdsajm1OX68kLVV0RHnmvLC1RSwNsrAGQ7fhMAmFb6BiM63NGnlpP9OtzRpyMd/dq8Y0D/tndjNEh39utk79AZ9yvIy4mF5kItn1Wha5fWq6GiUA2xIN1YUaT6ikICNAAgMH5jAMgYA8MRHeno1+GT/Wrp6FNLR+z8ZL8Oxy6PFaIrCqT5dYOaU12ipgXVaigvUkPlqwG6oaJQlcX5zH8GAEwqgjaAtODuause0METfTp4ok+HT/ap5WSfDndEp3O0dPTpWPfgGferKsnXzMpizaos0qp5VZpVVayZlUXRbVXRML3hf36t5uarQ3hVAIBsRtAGMCVGRlytXQM6eKJXh072xQJ1rw6e6NOhE306eLJPg6etB11emKeZVdHQvHxWhWZWFmtmVZFmxc5nVhYxlQMAkLb4DQVgUkRGXEc6+6Oh+TUBOnq55WT/GWtD15QWaE51sZbOLNfaZQ2aXVWsOdXFmlNdoplVRaooyg/p1QAAkDyCNoDAegaGdeB4b/TU3qv9x3t04HifDrT36OCJvjOWuasrL9Sc6mKtnF2pG1fM1OzqaJCeW12sWVXFjEYDAKY1fssBOGV0nvSB9miY3h87H718rHvgNbevKMrT/JpSLZ9dqRtXztTc6pLYiHQ0SPPNhACAbEbQBrKMu6uta0B7j/VoX9zplVigjl872kyaVVmsuTOKdd3Ses2rKdG8GSWaX1Oi+TNKVVnC1A4AAM6GoA1MU539Q9rXFg3Rr4bqbu1r61FPXJguyMvR/Fh4vmpRrebHwvS8mujodGEeo9IAAEwEQRvIYEOREe1v79Hu1rggHQvV8Uvh5Zg0p7pEC2tL1TR/hs6rK9XC2uhpZmWxcnNYPxoAgMlG0AYywGDE9cLhTu1q7dKe1m7tip1ePtbzmg8g1pUXamFtqdZe2HAqSJ9XV6q5M0oYmQYAYIoRtIE00j0wfCpI727t1u7WLu1q7daB9l75w7+WFB2dXlBTqvPry3T9sgYtbijToroyLawtVTnL4QEAkDYI2kAIhiIj2tvWoxePdOrFI116saVTLx3t1qGTfaduk59rOq+2TCtmV+qSqiG96fIVWlxfrgW1jE4DAJAJCNpACrlHvw1xR0s0UO880qUdLZ3a09atoUh0ykd+rmlRXZlWL6jWexrm6fz6Mi2uL9O8GSXKy82RJK1fv17NF80K86UAAIAEEbSBSTI4PKJdrV3afqhTL7R0nhqtPtk7dOo2MyuLdEFjuZovqNeFM8t1QWO5zqstU0FeToiVAwCAVCBoAxPQPxTRziNd2na4Q9sOdWjboU7tPNJ16ivGSwpytaShXDeuaNTSxgpd0FiupY3lqiopCLlyAAAwVQjawDh6Boa1o6UzGqgPR893tXYrElvto7I4XytmV+i21y/Q8tmVWjGrQgtqSpXDknkAAGQ1gjYQJzLi2tXapWcPnNSzr0RPLx3t0ugKejWlBVoxu1LXXVivlbMrtXxWpeZUF8uMUA0AAF6LoI2sdrSzX8+cCtUntPVgx6lvTawsztfFc6t0/bIGrZxTpZWzK9VQUUioBgAAgRC0kTWGIyN6oaVTT798Qpv3H9czB06qpaNfUnTlj2UzK/SOS+fokrlVumRulRbWlhKqAQDAhBG0MW31DAzr2VdO6umXj2vTyye05cAJ9cZGq+dUF2v1ghnRUD2vSstmVqgon7WpAQDA5CFoY9o40TOop/a16+mXT2jTy8e17XCnIiMuM+nCxgq969I5Wr1whprmz1BjZVHY5QIAgGmOoI2M1dU/pI37juuJPe16Yk+7drR0SpIK83L0unlV+ljzIjUtmKHXzatSBV9NDgAAphhBGxmjbzCizftP6Ik9x/TEnnZtPdShyIirMC9HTQuq9anrl+jKRTVaObuKL4ABAAChI2gjbbm7dh7t0mM72/TYS23a9PIJDUZGlJdjunhudMT6ykU1WjWvmvnVAAAg7RC0kVY6+ob0P7uPnQrXRzqjq4IsbSzX+6+ar6vOr9XqBTNUVsihCwAA0htpBaFyd710tFuP7Diq9TtbteXASUVGXOWFebp6ca2aL6jTG5fUaWZlcdilAgAAJISgjSkXGXFt3n9CD20/ood3HNX+9l5J0vJZFfroNefpmiX1et28KuXnMs8aAABkLoI2pkTfYETPtA7rZ//1nH75YquO9wyqIDdHVy6q0YffcJ7etKxBDRUsuQcAAKYPgjZSpn8oovU72/TT5w/rlzta1TcUUXnREV27tF5vWtaga5bUqZxl9wAAwDRF0MakGhwe0X/vbtNPn2vRQy8cVffAsGaUFuhtq2ZrdqRVH/6tNSy9BwAAsgJBG0lzd23cd1w/euaQfr7tiDr6hlRRlKffWNmot1w0S1ctqlFebo7Wr19PyAYAAFmDoI0Je+V4r3645ZDu23JQB473qqQgV29e3qi3XDRTb1hcR6gGAABZjaCNhPQODuvnW4/oB5sP6sm97ZKkqxbV6M61i3XDikaVFHBIAQAASARtBLS7tUv/ueGA7tt8UF0Dw5o3o0R/8KYletvrZmvujJKwywMAAEg7BG2c1eDwiB564Yj+c8N+bdh7XAW5ObpxZaPec9k8XbZwhsws7BIBAADSFkEbZ2jvHtC/P7lf3914QG1dA5pTXaw/umGp3tU0R7VlhWGXBwAAkBEI2jhlT1u3vvnf+3Tf5oMaGB7RtUvrdesV8/XGJXXKzWH0GgAAIBEEbejpl4/rnsf36pEdR5Wfm6N3rJqjD169UOfXl4VdGgAAQMYiaGexJ/e06+uPvKSn9h1XdUm+PnHtYr3vyvlMDwEAAJgEBO0s4+56cm+7vv7ILm3cd1z15YX64k3LdMvqeSouyA27PAAAgGmDoJ1FNu8/rq89uPNUwP7STct0y2XzVJRPwAYAAJhsoQRtM/srSTdJGpS0R9Jt7n4yjFqywb5jPbr7wRf1821HVFdeqC+/dbn+1+q5BGwAAIAUCmtE+2FJf+zuw2b2NUl/LOmPQqpl2mrvHtA//HKXvvPUARXk5ej31y7Rh9+4kG9vBAAAmAKhJC53fyju6gZJ7wyjjukqMuL6zlP79Ve/2KnewYhuWT1Xd65dorpyPuQIAAAwVdJhaPN/S/p+2EVMF88cOKHP/2Sbth3q1OvPr9GX37pc59eXh10WAABA1jF3T80Dmz0iqXGMXXe5+09it7lLUpOkt/tZCjGz2yXdLkkNDQ2Xrlu3LiX1BtHd3a2ysvRcW7p3yHXvS4N67JVhVRaafntpgS5rzE2rr0lP5/6lO3qXHPqXHPo3cfQuOfQvOfQvOWvWrNns7k3JPEbKgva4T2z2AUkfkXSdu/cGuU9TU5Nv2rQppXWdy/r169Xc3Bza85/NYy+16bP3Pa+jnf267fULdefaxSovyg+7rDOka/8yAb1LDv1LDv2bOHqXHPqXHPqXHDNLOmiHterIDZI+I+maoCEbZ+rqH9JXf7ZD655+RefXl+mHH3u9LplbFXZZAAAAUHhztP9RUqGkh2NTGza4+0dDqiUjbd5/Qp/83jNq6ejTR69ZpDvXLma5PgAAgDQS1qoj54fxvNPByIjrG4/v1V8/tFOzqor0gzuu0qp51WGXBQAAgNOkw6ojCKi9e0C/f+9zevylNv3mRTP1F29fqYo0nIsNAAAAgnbG2H64Qx/+9iYd6xnUV9+2Qu+5bF5arSgCAACA1yJoZ4AHtrboD+99TlUl+brvo1dp5ZzKsEsCAADAOAjaaczd9fe/3KWvP7JLq+ZV6V9uvVT15UVhlwUAAIAACNppKjLi+tyPt+l7Gw/oHavm6M/fvkKFeawqAgAAkCkI2mmofyiiO9c9qwe3H9HvrlmkT11/AfOxAQAAMgxBO830DUb0wW8/rSf2tOvzb1mmD169MOySAAAAMAEE7TTSPxQN2Rv2tutv332x3r5qTtglAQAAYIII2mmifyiiD//7Jj25t11/8y5CNgAAQKbLCbsASMOREX38u1v0613H9LV3XETIBgAAmAYI2iFzd33h/u16ZEer/vTm5Xp309ywSwIAAMAkIGiH7J8f26PvPnVAdzQv0q1XLgi7HAAAAEwSgnaIHtjaorsf3KmbL5mlT19/QdjlAAAAYBIRtEOyu7VLn/qv57RqXpXufudFyslhnWwAAIDphKAdgq7+Id3+H5tVUpCnf37vpXzjIwAAwDTE8n5TzN312R9u1f72Xn3nQ5eroaIo7JIAAACQAoxoT7EfPXNIP3u+RX94/RJdcV5N2OUAAAAgRQjaU+jQyT598SfbtXpBtT7yxkVhlwMAAIAUImhPEXfXp+59TiPu+tt3X6JcPvwIAAAwrRG0p8h9Ww7pyb3tuus3l2nujJKwywEAAECKEbSnwMneQf35Azu0al6VblnNNz8CAABkA4L2FPjagzvV0Tekr75tJetlAwAAZAmCdortaOnUuqcP6P1XLtCFMyvCLgcAAABThKCdYnc/+KLKC/P0e9ctDrsUAAAATCGCdgpt2NuuR3e26WNrzldlSX7Y5QAAAGAKEbRTxN1194MvqrGiSB+4akHY5QAAAGCKjRu0zewTZlY9FcVMJxv2HteWAyf1u2sWqSg/N+xyAAAAMMWCjGg3SHrazO41sxvMjGUzAvjnx/aotqxA72piOT8AAIBsNG7QdvfPSVos6ZuSPiBpl5n9uZnxHeJnsf1whx5/qU23vX4ho9kAAABZKtAcbXd3SUdip2FJ1ZJ+YGZ3p7C2jPWvv96nssI8vfeK+WGXAgAAgJDkjXcDM/s9Se+TdEzSv0r6tLsPmVmOpF2SPpPaEjPLiZ5B/Wxri25ZPVeVxaw0AgAAkK3GDdqSZkh6u7vvj9/o7iNm9pbUlJW57ttyUIPDI3rP5fPCLgUAAAAhGjdou/sXz7Fvx+SWk9ncXd/deECr5lVpaSPfAgkAAJDNWEd7Em3af0J723r0nsuZmw0AAJDtCNqT6P5nD6soP0c3rmgMuxQAAACEjKA9SYYjI3pga4uuu7BBpYVBpr4DAABgOiNoT5In97arvWdQN100K+xSAAAAkAYI2pPkp8+1qKwwT80X1IVdCgAAANIAQXsSjIy4fvliq5ovqOObIAEAACCJoD0pth/u1LHuAV27tD7sUgAAAJAmCNqT4NGdrTKTrlnCtBEAAABEEbQnwaM7W3XxnCrVlBWGXQoAAADSBEE7SR29Q3r2lZOMZgMAAOA1Qg3aZvaHZuZmVhtmHcnYtP+43KUrF9WEXQoAAADSSGhB28zmSrpe0oGwapgMG/cdV0Fuji6ZWxV2KQAAAEgjYY5o/52kz0jyEGtI2saXj+uiOZUs6wcAAIDXCCVom9nNkg65+3NhPP9k6R0c1taDHbps4YywSwEAAECaMffUDCib2SOSGsfYdZekP5F0vbt3mNnLkprc/dhZHud2SbdLUkNDw6Xr1q1LSb1BdHd3q6ys7NT1He0Rfe3pfv3+pYW6uC4vtLoyxen9Q3D0Ljn0Lzn0b+LoXXLoX3LoX3LWrFmz2d2bknmMlKVDd1871nYzWylpoaTnzEyS5kjaYmaXufuRMR7nHkn3SFJTU5M3NzenquRxrV+/XvHPv+vxvZJ26L03voGl/QI4vX8Ijt4lh/4lh/5NHL1LDv1LDv0L35QPw7r7VkmnvkJxvBHtdLb1UIdmVRYRsgEAAHAG1tFOwrbDHVoxuzLsMgAAAJCGQg/a7r4gE0ezuweGte9YD0EbAAAAYwo9aGeqFw53yl1aSdAGAADAGAjaE7TzaJckaenM8pArAQAAQDoiaE/QntZulRbkqrGiKOxSAAAAkIYI2hO0p61bi+rLFFuiEAAAAHgNgvYE7Wnt1qI6FoEHAADA2AjaE9AzMKzDHf1aVFcadikAAABIUwTtCdh3rEeSGNEGAADAWRG0J2A0aC+oZUQbAAAAYyNoT8DBE32SpDnVxSFXAgAAgHRF0J6AQyd7VVmcr/Ki/LBLAQAAQJoiaE/AoRN9ml3FaDYAAADOjqA9AYdO9mk200YAAABwDgTtBLk7I9oAAAAYF0E7QR19Q+oZjPBBSAAAAJwTQTtBh0/2S5JmMaINAACAcyBoJ+hY94Akqa68MORKAAAAkM4I2glq64oG7doygjYAAADOjqCdoNER7dqygpArAQAAQDojaCfoWPeAivJzVFaYF3YpAAAASGME7QQd6x5UbVmhzCzsUgAAAJDGCNoJOtY9wPxsAAAAjIugnaC2LoI2AAAAxkfQTlB06ggfhAQAAMC5EbQT4O7q7BtSZUl+2KUAAAAgzRG0EzA0Ig1GRlRZTNAGAADAuRG0E9Az5JJE0AYAAMC4CNoJ6B2OnlcUEbQBAABwbgTtBPQyog0AAICACNoJGJ06UkHQBgAAwDgI2gkYnTrCiDYAAADGQ9BOwOjUkYqivJArAQAAQLojaCeAqSMAAAAIiqCdgL5hV3F+rvJzaRsAAADOjcSYgIGIVFKQG3YZAAAAyAAE7QQMRKRigjYAAAACIGgnYCDijGgDAAAgEIJ2AqIj2qw4AgAAgPERtBMwGHGV5DOiDQAAgPERtBPAHG0AAAAERdBOwEDECdoAAAAIhKCdgMGImDoCAACAQAjaCWDVEQAAAARF0E7AwDCrjgAAACCY0IK2mX3CzF40s+1mdndYdQQ1HBnRsEvFTB0BAABAAKEMz5rZGkk3S7rY3QfMrD6MOhLRNxSRJBUX8EcAAAAAjC+s1HiHpL909wFJcvfWkOoIbHB4RJJUmMeINgAAAMYXVtBeIukNZvaUmT1mZqtDqiOwoYhLkvJzGdEGAADA+MzdU/PAZo9Iahxj112SvirpUUmflLRa0vclnedjFGNmt0u6XZIaGhouXbduXUrqHU9b74g+/XifPrSyQFfPzg+lhkzX3d2tsrKysMvISPQuOfQvOfRv4uhdcuhfcuhfctasWbPZ3ZuSeYyUzdF297Vn22dmd0j6YSxYbzSzEUm1ktrGeJx7JN0jSU1NTd7c3Jyagsexu7VbevwxrVy+TM2XzA6lhky3fv16hfXvl+noXXLoX3Lo38TRu+TQv+TQv/CFNQ/ix5LWSJKZLZFUIOlYSLUEMhSJztEuYOoIAAAAAghrUehvSfqWmW2TNCjp/WNNG0knp4J2HkEbAAAA4wslaLv7oKT3hvHcEzW66ggfhgQAAEAQpMaABiMEbQAAAARHagxodHk/po4AAAAgCFJjQEPDfBgSAAAAwZEaAzo1dSTPQq4EAAAAmYCgHRDL+wEAACARpMaAWHUEAAAAiSA1BjTIOtoAAABIAKkxID4MCQAAgESQGgMaXd4vL5cPQwIAAGB8BO2AhkdiQTuHlgEAAGB8pMaARjwatMnZAAAACILYGFAkNqKda0wdAQAAwPgI2gGdCto5BG0AAACMj6Ad0Ii7TJIxog0AAIAACNoBRUZcDGYDAAAgKIJ2QBF3MZgNAACAoAjaAY0wog0AAIAEELQDiozQLAAAAARHdgxoxBnRBgAAQHAE7YD4MCQAAAASQdAOKMKINgAAABJA0A4o+mFIkjYAAACCIWgHxNQRAAAAJIKgHVAk9s2QAAAAQBAE7YBYRxsAAACJIGgHFHERtAEAABAYQTsgRrQBAACQCIJ2QHwYEgAAAIkgaAcUXUebpA0AAIBgCNoBMXUEAAAAiSBoBxRxp1kAAAAIjOwYUGTExcwRAAAABEXQDshZ3g8AAAAJIGgHNMI3QwIAACABBO2A3MXUEQAAAARG0A7I5WGXAAAAgAxC0E4AA9oAAAAIiqAdkDOgDQAAgAQQtAMiZwMAACARBO0E8GFIAAAABEXQDoohbQAAACSAoB2Qi3W0AQAAEBxBOyA+DAkAAIBEhBK0zewSM9tgZs+a2SYzuyyMOhLFHG0AAAAEFdaI9t2Svuzul0j6Qux6WmNAGwAAAIkIK2i7pIrY5UpJh0OqIzB35mgDAAAguLyQnvdOSb8ws79WNOxfFVIdAAAAQEqYp+hTfmb2iKTGMXbdJek6SY+5+31m9m5Jt7v72rM8zu2Sbo9dvUDSzlTUG1CtpGMhPn+mo38TR++SQ/+SQ/8mjt4lh/4lh/4l5wJ3L0/mAVIWtM/5pGYdkqrc3c3MJHW4e8V49wubmW1y96aw68hU9G/i6F1y6F9y6N/E0bvk0L/k0L/kTEb/wpqjfVjSNbHL10raFVIdAAAAQEqENUf7w5L+3szyJPXr1akhAAAAwLQQStB29/+WdGkYz52ke8IuIMPRv4mjd8mhf8mhfxNH75JD/5JD/5KTdP9CmaMNAAAATHd8BTsAAACQAgRtSWZ2g5ntNLPdZvbZMfYXmtn3Y/ufMrMFcfv+OLZ9p5m9eSrrThcB+vcHZvaCmT1vZr80s/lx+yJm9mzsdP/UVp4eAvTvA2bWFtenD8Xte7+Z7Yqd3j+1lYcvQO/+Lq5vL5nZybh9HHtm3zKzVjPbdpb9Zmb/EOvv82a2Km5fth974/Xud2I922pmT5jZxXH7XoQda5UAAAYsSURBVI5tf9bMNk1d1ekjQP+azawj7mf0C3H7zvlznw0C9O/Tcb3bFnu/mxHbl9XHn5nNNbNHY7lku5n93hi3mbz3PnfP6pOkXEl7JJ0nqUDSc5KWnXabj0n6l9jlWyR9P3Z5Wez2hZIWxh4nN+zXlIb9WyOpJHb5jtH+xa53h/0aMqB/H5D0j2Pcd4akvbHz6tjl6rBfUzr17rTbf0LSt+KuZ/WxF+vBGyWtkrTtLPt/Q9LPJZmkKyQ9Fdue1cdewN5dNdoTSTeO9i52/WVJtWG/hjTvX7Okn46xPaGf++l6Gq9/p932Jkm/irue1cefpJmSVsUul0t6aYzfu5P23seItnSZpN3uvtfdByWtk3Tzabe5WdK3Y5d/IOk6M7PY9nXuPuDu+yTtjj1eNhm3f+7+qLv3xq5ukDRnimtMZ0GOv7N5s6SH3f24u5+Q9LCkG1JUZzpKtHe/Lel7U1JZhnD3xyUdP8dNbpb07x61QVKVmc0Ux964vXP3J2K9kXjfO0OAY+9sknnPnDYS7B/vfXHcvcXdt8Qud0naIWn2aTebtPc+gna0ua/EXT+oMxt+6jbuPiypQ1JNwPtOd4n24IOK/i9xVJGZbTKzDWb2W6koMM0F7d87Yn+++oGZzU3wvtNV4Ncfm660UNKv4jZn+7EXxNl6nO3HXqJOf99zSQ+Z2WaLfvsxxnalmT1nZj83s+WxbRx7CTCzEkWD4H1xmzn+Yiw6Ffh1kp46bdekvfeFtY42spCZvVdSk179siJJmu/uh8zsPEm/MrOt7r4nnArT1v+T9D13HzCzjyj615VrQ64p09wi6QfuHonbxrGHlDOzNYoG7avjNl8dO/bqJT1sZi/GRijxqi2K/ox2m9lvSPqxpMUh15SJbpL0P+4eP/rN8SfJzMoU/Q/Ine7emarnYURbOiRpbtz1ObFtY97Gol+yUympPeB9p7tAPTCztZLukvRWdx8Y3e7uh2LneyWtV/R/ltlk3P65e3tcz/5Vr65Bn+3HXyKv/xad9qdTjr1AztbjbD/2AjGzixT9mb3Z3dtHt8cde62SfqTsm3I4LnfvdPfu2OUHJOWbWa049hJ1rve+rD3+zCxf0ZD9HXf/4Rg3mbT3PoK29LSkxWa20MwKFD0oT1+B4H5Jo58sfaeiHyrw2PZbLLoqyUJF/7e9cYrqThfj9s/MXifpG4qG7Na47dVmVhi7XCvp9ZJemLLK00OQ/s2Mu/pWReeTSdIvJF0f62O1pOtj27JFkJ9dmdlSRT+08mTcNo69YO6X9L7YJ/CvkNTh7i3i2BuXmc2T9ENJt7r7S3HbS82sfPSyor0bc+WIbGZmjbHPQsnMLlM0r7Qr4M89JDOrVPQvyD+J25b1x1/suPqmpB3u/rdnudmkvfdl/dQRdx82s48r2qhcRVcl2G5mX5G0yd3vV/Qf5D/MbLeiHz64JXbf7WZ2r6K/oIcl/e5pf5qe9gL2768klUn6r9j75gF3f6ukCyV9w8xGFH0T/Ut3z6qwE7B/nzSztyp6jB1XdBUSuftxM/tTRX/xSNJXTvvz4LQWsHdS9Od1Xew/x6Oy/tiTJDP7nqKrO9Sa2UFJX5SUL0nu/i+SHlD00/e7JfVKui22L6uPPSlQ776g6Gd5/k/sfW/Y3ZskNUj6UWxbnqTvuvuDU/4CQhagf++UdIeZDUvqk3RL7Gd4zJ/7EF5CqAL0T5LeJukhd++JuyvHX3Rg5VZJW83s2di2P5E0T5r89z6+GRIAAABIAaaOAAAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBYBoys9Vm9ryZFcW+DW67ma0Iuy4AyCZ8YQ0ATFNm9meSiiQVSzro7n8RckkAkFUI2gAwTZlZgaJfFdwv6Sp3j4RcEgBkFaaOAMD0VSOpTFK5oiPbAIApxIg2AExTZna/pHWSFkqa6e4fD7kkAMgqeWEXAACYfGb2PklD7v5dM8uV9ISZXevuvwq7NgDIFoxoAwAAACnAHG0AAAAgBQjaAAAAQAoQtAEAAIAUIGgDAAAAKUDQBgAAAFKAoA0AAACkAEEbAAAASAGCNgAAAJAC/x9OMXC1ykcagQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import figure, show\n",
    "from numpy import arange, sin, pi\n",
    "\n",
    "#t = arange(1e-5, 5.0, 0.00001)\n",
    "#t = arange(1.0, 5.0, 0.00001) # computer scientists\n",
    "t = arange(0.0, 1.0, 0.00001)  # data     scientists\n",
    "\n",
    "fig = figure(1,figsize=(12, 10))\n",
    "\n",
    "ax1 = fig.add_subplot(211)\n",
    "ax1.plot(t, np.log(t))\n",
    "ax1.grid(True)\n",
    "ax1.set_ylim((-8, 1.5))\n",
    "ax1.set_xlim((-0.1, 2))\n",
    "ax1.set_xlabel('x')\n",
    "ax1.set_ylabel('y')\n",
    "ax1.set_title('log(x)')\n",
    "\n",
    "show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[ 21   1   0   0   0   0   0]\n",
      " [  4 193  52   0   0   0   0]\n",
      " [  0  51 134   2   0   0   0]\n",
      " [  0   0   9   5   0   0   0]\n",
      " [  0   8   0   0   0   0   0]\n",
      " [  9   7   0   0   0   1   0]\n",
      " [  3   0   0   0   0   0   0]]\n",
      "Normalized confusion matrix\n",
      "[[0.95 0.05 0.   0.   0.   0.   0.  ]\n",
      " [0.02 0.78 0.21 0.   0.   0.   0.  ]\n",
      " [0.   0.27 0.72 0.01 0.   0.   0.  ]\n",
      " [0.   0.   0.64 0.36 0.   0.   0.  ]\n",
      " [0.   1.   0.   0.   0.   0.   0.  ]\n",
      " [0.53 0.41 0.   0.   0.   0.06 0.  ]\n",
      " [1.   0.   0.   0.   0.   0.   0.  ]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEmCAYAAAB/OxvoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3debwcVZn/8c/3JuxBIATDGgJMQIWfRBJxhBGjIAMMsqggiyiKhqDAKMM4iDggDqMjboMoGgYENwgOogwwIoMLwoCShBCJLBIkP8AICSg7gSTP/FGnoenUvbdu367bVZ3vm1e90l1Vfc5zY3zu6VNnUURgZmb11dftAMzMbHicyM3Mas6J3Mys5pzIzcxqzonczKzmnMjNzGrOidzaImkdSf8l6XFJPxhGOUdK+mknY+sWSW+SdHe347DVjzyOvLdJOgI4CXgV8CQwDzgrIm4cZrlHAScAu0XE8mEHWnGSApgUEfd2OxazVm6R9zBJJwFfAf4VGA9MAL4OHNiB4rcG7lkdkngRkkZ3OwZbjUWEjx48gA2Ap4BDBrhnLbJE/8d0fAVYK12bBjwI/APwCLAYeH+69mngeeCFVMcxwBnAd5vKnggEMDq9Pxq4j+xbwR+AI5vO39j0ud2AW4HH05+7NV37BfAZ4KZUzk+Bcf38bI34P94U/0HAfsA9wGPAqU337wrcDPwl3XsusGa6dkP6WZ5OP++7m8r/J+BPwHca59Jntkt17JLebw4sAaZ1+9+Gj9473CLvXW8E1gauGOCeTwJ/DUwGdiZLZqc1Xd+U7BfCFmTJ+muSNoqI08la+bMiYkxEXDBQIJLWA84B9o2I9cmS9byc+8YCV6d7Nwa+BFwtaeOm244A3g+8ElgTOHmAqjcl+zvYAvhn4HzgPcAU4E3ApyRtk+5dAXwMGEf2d7cn8GGAiNgj3bNz+nlnNZU/luzbyfTmiiNiIVmS/66kdYFvARdHxC8GiNesLU7kvWtjYGkM3PVxJHBmRDwSEUvIWtpHNV1/IV1/ISKuIWuN7tBmPCuBnSStExGLI2JBzj1/B/w+Ir4TEcsj4hLgLuDtTfd8KyLuiYhngcvIfgn15wWy5wEvAJeSJel/j4gnU/2/I/sFRkTMiYhbUr33A98E3lzgZzo9IpaleF4mIs4H7gV+DWxG9ovTrOOcyHvXo8C4QfpuNwcWNb1flM69WEbLL4JngDFDDSQinibrjpgBLJZ0taRXFYinEdMWTe//NIR4Ho2IFel1I9E+3HT92cbnJW0v6SpJf5L0BNk3jnEDlA2wJCKeG+Se84GdgK9GxLJB7jVrixN577oZWEbWL9yfP5J1CzRMSOfa8TSwbtP7TZsvRsS1EfE2spbpXWQJbrB4GjE91GZMQ3EeWVyTIuIVwKmABvnMgEO+JI0he+5wAXBG6joy6zgn8h4VEY+T9Qt/TdJBktaVtIakfSV9Pt12CXCapE0kjUv3f7fNKucBe0iaIGkD4BONC5LGSzow9ZUvI+uiWZlTxjXA9pKOkDRa0ruB1wBXtRnTUKwPPAE8lb4tHNdy/WFg2yGW+e/A7Ij4IFnf/zeGHaVZDifyHhYRXyQbQ34a2YiJB4DjgR+lW/4FmA3MB34LzE3n2qnrOmBWKmsOL0++fSmOP5KN5HgzqyZKIuJRYH+ykTKPko042T8ilrYT0xCdTPYg9UmybwuzWq6fAVws6S+SDh2sMEkHAvvw0s95ErCLpCM7FrFZ4glBZmY15xa5mVnNOZGbmdWcE7mZWc2Vmsgl/UjSHEkLJE0f/BNmZjZUpT7slDQ2Ih6TtA7ZuhlvTiMTmu+ZTprevO66603ZblK7EwfLNXrUYEOKu6OaUZmNnEWL7mfp0qUd/b/CqFdsHbF8lcm6ueLZJddGxD6drH+oyl6x7URJB6fXWwGTyIaVvSgiZgIzAV47eUpc9bP/LTmk9myy/prdDiGX5FRuq7fd3zC142XG8mdZa4dBR5kC8Ny8rw02A7h0pSVySdOAvYA3RsQzkn5BtoCRmVnFCVSfR4hltsg3AP6ckviryFbZMzOrPgE1+rZbZiL/CTBD0p3A3cAtJdZlZtZZbpFDWult37LKNzMrj6BvVLeDKMzbU5mZ5XHXiplZjQl3rZiZ1ZvcIjczqz23yM3Mas4tcjOzOvOEIDOzevOEIDOzHuAWuZlZnQlGeUKQmVl9eRy5mVkPcB+5mVmdedSKmVn9uUXentGjxNj11uh2GLnG7npCt0PI9ZNLz+x2CP16w3Zjux2CWfs61CKXdCGwP/BIROyUzs0CGvtabgj8JSImS5oINJb+BrglImYMVkelErmZWSWoo2utXAScC3y7cSIi3v1SVfoi8HjT/QsjYvJQKnAiNzPL06EWeUTckFraq1aRbbp7KPDW4dRRn958M7OR1GiVD3bAOEmzm47pQ6jlTcDDEfH7pnPbSLpN0i8lvalIIW6Rm5mtYkijVpZGxNQ2KzocuKTp/WJgQkQ8KmkK8CNJO0bEEwMV4kRuZtZKlL7Vm6TRwDuAKY1zaYvMZen1HEkLge2B2QOV5URuZraKERlHvhdwV0Q8+GKt0ibAYxGxQtK2wCTgvsEKch+5mVme4n3kgxSjS4CbgR0kPSjpmHTpMF7erQKwBzBf0jzgP4EZEfHYYHW4RW5mlqdzo1YO7+f80TnnLgcuH2odTuRmZnk8s9PMrMbktVbMzOrPLXIzs3qTE7mZWX1lW3Y6kZuZ1ZeE+pzIzcxqzS1yM7Oaq1MiL218jaSJku4oq3wzszJJKnRUgVvkZmatlI6aKHvE+2hJ35N0p6T/lLRuyfWZmQ2bKNYar0qLvOxEvgPw9Yh4NfAE8OHWGyRNbyzIvnTJkpLDMTMrxon8JQ9ExE3p9XeBv2m9ISJmRsTUiJg6bpNNSg7HzKyYOiXysvvIY5D3ZmaVVJUkXUTZLfIJkt6YXh8B3FhyfWZmw6chHBVQdiK/G/iIpDuBjYDzSq7PzGzYhOjr6yt0VEFpXSsRcT/wqrLKNzMrU526VjyO3MwsT33yuBO5mdkq5Ba5mVntOZGbmdVcnRJ5NR65mplVSCen6Eu6UNIjzYsISjpD0kOS5qVjv6Zrn5B0r6S7Jf1tkXidyM3M8nRuHPlFwD45578cEZPTcQ2ApNcAhwE7ps98XdKowSpwIjcza6XOTdGPiBuAxwrWfCBwaUQsi4g/APcCuw72ISdyM7McQ0jk4xoL/6VjesEqjpc0P3W9bJTObQE80HTPg+ncgPyw08wsxxD27FwaEVOHWPx5wGfI1p/6DPBF4ANDLONFTuRmZjnKHLUSEQ831XM+cFV6+xCwVdOtW6ZzA3LXiplZi6LdKu0me0mbNb09GGiMaLkSOEzSWpK2ASYBvxmsPLfIzcxydKpFLukSYBpZX/qDwOnANEmTybpW7geOBYiIBZIuA34HLAc+EhErBqvDidzMLEenEnlEHJ5z+oIB7j8LOGsodVQqkQsYPaqavT3Xzjqz2yHkOvH7t3U7hH7976lv7XYI/eor/iDLVlc1+idSqURuZlYVdZqi70RuZtbKqx+amdWbgBrlcSdyM7NVqVbPUZzIzcxyuGvFzKzO5K4VM7NaE/UaoupEbmaWwy1yM7Oacx+5mVmduY/czKzesnHk9cnkTuRmZqtof4nabnAiNzPLUaM87kRuZrYKefihmVmtuY/czKwH1CiPO5GbmeWpU4u89O14JL1X0nxJt0v6Ttn1mZl1glTsqIJSW+SSdgROA3aLiKWSxpZZn5lZR9RsY4myW+RvBX4QEUsBIuKx1hskTZc0W9LsJUuXlByOmdngGhtL1KVF3vWdjiNiZkRMjYipm4zbpNvhmJnRmBBU5Bi0JOlCSY9IuqPp3NmS7krdzldI2jCdnyjpWUnz0vGNItGWnch/BhwiaWMAd62YWV10sEV+EbBPy7nrgJ0i4rXAPcAnmq4tjIjJ6ZhRpIJSE3lELADOAn4p6XbgS2XWZ2bWEWlCUJFjMBFxA/BYy7mfRsTy9PYWYMvhhFv68MOIuBi4uOx6zMw6ZYgTgsZJmt30fmZEzBxCdR8AZjW930bSbcATwGkR8avBCvA4cjOzHENI5EsjYmqbdXwSWA58L51aDEyIiEclTQF+JGnHiHhioHKcyM3McpQ9IkXS0cD+wJ4REQARsQxYll7PkbQQ2B6Y3V854ERuZparzHHkkvYBPg68OSKeaTq/CfBYRKyQtC0wCbhvsPKcyM3MWnVwjLikS4BpZH3pDwKnk41SWQu4Lv3CuCWNUNkDOFPSC8BKYEbe/JtWTuRmZi3UwY0lIuLwnNMX9HPv5cDlQ63DidzMLEdVZm0W4URuZpajr0aZ3InczCxHjfK4E7mZWSsJRnmrNzOzeqvTMrb9JnJJrxjog4PNNDIzq7Ma5fEBW+QLgCBbdqCh8T6ACSXGZWbWNSIbglgX/SbyiNhqJAMxM6uSGnWRF1vGVtJhkk5Nr7dMi7mYmfWmgptKVKUffdCHnZLOBdYgmzr6r8AzwDeA15cbWrXsum0198T4+cendTuEfq1YGd0OoV9F1pG21VtFcnQhRUat7BYRu6T1cYmIxyStWXJcZmZdI3pvQtALkvrIHnCStm1bWWpUZmZdVqM8XqiP/Gtki7hsIunTwI3Av5UalZlZl/VUH3lEfFvSHGCvdOqQiLhjoM+YmdVZr87sHAW8QNa9UuqGzWZmVVCfNF4gKac95S4BNifb6fn7kj5RdmBmZt3UU10rwHuB1zW2I5J0FnAb8NkyAzMz65Zs1Eq3oyiuSCJf3HLf6HTOzKw3Vai1XcRAi2Z9maxP/DFggaRr0/u9gVtHJjwzs+6oUR4fsEXeGJmyALi66fwt5YVjZlYNPdEij4jczUHNzHpd3frIi4xa2U7SpZLmS7qncYxEcGZm3dKpUSuSLpT0iKQ7ms6NlXSdpN+nPzdK5yXpHEn3ppy7S5FYi4wJvwj4FtkvqX2By4BZRQo3M6sjCUZJhY4CLgL2aTl3CnB9REwCrk/vIcuxk9IxHTivSAVFEvm6EXEtQEQsjIjTUmVmZj1LKnYMJiJuIBs00uxA4OL0+mLgoKbz347MLcCGkjYbrI4iww+XpUWzFkqaATwErF/gc2ZmtTWEh53jJM1uej8zImYO8pnxEdEYxv0nYHx6vQXwQNN9D6ZzAw75LpLIPwasB5wInAVsAHygwOdeRtIZwFMR8YWhftbMbKQNYdDK0oiY2m49ERGShrV4f5FFs36dXj4JHDWcyszM6kCo7PXIH5a0WUQsTl0nj6TzDwHN22xumc4NaKAJQVeQ1iDPExHvGKzwtE7L+1KQDwBzBvuMmVnXFez/HoYryXLj59KfP246f7ykS4E3AI83dcH0a6AW+bnDiTLt63kYMDnVM5ecRC5pOtnTWbaaMGE4VZqZdUynJgRJugSYRtaX/iBwOlkCv0zSMcAi4NB0+zXAfsC9ZNtqvr9IHQNNCLq+7cgzbwKuaFps68p+6pkJzASYMmVqdTd5NLPVSqfW646Iw/u5tGfOvQF8ZKh1FF2P3MxstSHqNUW/zE0ibgAOkrSOpPWBt5dYl5lZR/Wp2FEFhVvkktaKiGVF74+IuZJmAbeTPez0iolmVgs9t9WbpF2BC8jGj0+QtDPwwYg4YbDPRsRZZGPPzcxqpUZ5vFDXyjnA/sCjABFxO/CWMoMyM+u2Tk3RHwlFulb6ImJRS8f/ipLiMTPrumwZ24pk6QKKJPIHUvdKSBoFnAB4GVsz62lljgTptCKJ/Diy7pUJwMPA/6RzZmY9q0YN8kJrrTxCNkPTzGy1IJW+1kpHFRm1cj45a65ExPRSIjIzq4Aa5fFCXSv/0/R6beBgXr5erplZz6nT8MMiXSsv29ZN0neAG0uLyMysy0SPTQjKsQ0v7WZhZtZ7KjT9vogifeR/5qU+8j6yvedO6f8TZmb1J+qTyQdM5MpmAe3MSztUrEzLLJqZ9axsQlC3oyhuwDHvKWlfExEr0uEkbmarhTqtflhk8tI8Sa8rPRIzswqRVOiogoH27BwdEcuB1wG3SloIPE32rSMiYpcRirESnnu+msvLrLPmqG6HYNZz6ta1MlAf+W+AXYADRigWM7NqqNDKhkUMlMgFEBELRygWM7PK6JUp+ptIOqm/ixHxpRLiMTPrul7qWhkFjIEaDaY0M+sIMapDLXJJOwDNM+S3Bf4Z2BD4ELAknT81Iq5pp46BEvniiDiznULNzOpMdK6PPCLuBiYDpD0dHgKuAN4PfDkivjDcOgbtIzczW+2UN0Z8T2Bhzq5rwzLQOPI9O1aLmVnN9KU1yQc7gHGSZjcdAy3xfRhwSdP74yXNl3ShpI3ajrW/CxHxWLuFmpnVWaNrpeDmy0sjYmrTMTO3TGlNsuHcP0inzgO2I+t2WQx8sd1421n90Mys55Uw/HBfYG5EPAzQ+BNe3MDnqnYLrtP+omZmI2YILfKiDqepW0XSZk3XDgbuaDdWt8jNzFqIzrZyJa0HvA04tun05yVNJlsm/P6Wa0PiRG5m1kp0dEGsiHga2Ljl3FGdKt+J3MwsR53GXzuRm5m1EHRsZudIcCI3M8tRozzuRG5mtqrqbBpRhBO5mVmLTo9aKZsTuZlZjjq1yEv9pSPpPZJ+I2mepG+mlb/MzCpPBY8qKC2RS3o18G5g94iYDKwAjsy5b3pjsZklS5e0XjYzG3nqkc2XO2BPYArZxs0A6wCPtN6UFpiZCTBlytQoMR4zs0LcR/4SARdHxCdKrMPMrBRVaW0XUeYvneuBd0l6JYCksZK2LrE+M7OO6VOxowpKa5FHxO8knQb8VFIf8ALwEWBRWXWamXVC1rVSkSxdQKnDDyNiFi/fdNTMrBZq1LPiceRmZqsScovczKze3CI3M6sx95GbmdXd0Ldx6yoncjOzHE7kZmY154edZmY1Jqoz2acIJ3Izsxx9NepbcSI3M8vRya4VSfcDT5KtArs8IqZKGks2YXIicD9waET8uZ3y67TAl5nZiGh0rXR4rZW3RMTkiJia3p8CXB8Rk8jWpjql3XidyM3MVqHC/w3DgcDF6fXFwEHtFuREbmbWKo0jL3IA4xqb46Rjek6JQbaA4Jym6+MjYnF6/SdgfLvhuo/czCzHENraS5u6S/rzNxHxUFrW+zpJdzVfjIiQ1PbGOpVK5CsDnn1+RbfDyFWjB9hWcxHV3CirThstDFfWR965nzciHkp/PiLpCmBX4GFJm0XEYkmbkbODWlHuWjEzy9GpzZclrSdp/cZrYG/gDuBK4H3ptvcBP2431kq1yM3MKqNzDfLxwBXpG81o4PsR8RNJtwKXSTqGbMOdQ9utwInczCxHp7pWIuI+YOec84+SbVI/bE7kZmY56vREwInczCxPjTK5E7mZWYvsQWZ9MrkTuZlZK28sYWZWfzXK407kZma5apTJncjNzFYx7AWxRpQTuZlZDveRm5nVWNHp91XhRG5mlqNOi4Q5kZuZ5ahRHnciNzPLU6M8PjLL2Eo6UdKdkr43EvWZmQ1L0TVsK5LtR6pF/mFgr4h4cITqMzMbljoNPyy9RS7pG8C2wH9L+ljZ9ZmZDZcY0p6dXVd6izwiZkjaB3hLRCwtuz4zs06oSI4upOtbvUma3th9eunSJd0Ox8wsU6M+8q4n8oiYGRFTI2LquHGbdDscMzOgMUl/8P+qwMMPzcxy9FUjRxfiRG5mlseJ/OUiYuJI1GNm1gl12yGo633kZmaVU3DoYZHhh5K2kvRzSb+TtEDS36fzZ0h6SNK8dOzXbrjuWjEzy9HB9vhy4B8iYq6k9YE5kq5L174cEV8YbgVO5GZmeTqUySNiMbA4vX5S0p3AFp0pPeOuFTOzVRQdfCiAcY25MOmY3m+p0kTgdcCv06njJc2XdKGkjdqN1onczCzHEPrIlzbmwqRjZn55GgNcDnw0Ip4AzgO2AyaTtdi/2G6sTuRmZi06vfihpDXIkvj3IuKHABHxcESsiIiVwPnAru3G60RuZpanQ5lc2VZDFwB3RsSXms5v1nTbwcAd7Ybqh51mZjn6Ore04e7AUcBvJc1L504FDpc0GQjgfuDYditwIjczy9GpNB4RN/ZT3DUdqsKJ3MxsFRVaa7wIJ3Izs1z1yeRO5GZmLRo7BNWFE7mZWY4a5XEncjOzPG6Rt0mCNUdXc2j7qDqtMm+1pjplkB5Wp2VsK5XIzcwqoz553InczCxPjfK4E7mZWSupozM7S+dEbmaWpz553InczCxPjfK4E7mZWZ4a9aw4kZuZrUoefmhmVmd1m6Jfzdk3ZmZWmFvkZmY56tQidyI3M8vhPnIzsxrLJgR1O4rinMjNzPI4kZuZ1Zu7VszMaq5ODzs9/NDMLIcKHoXKkvaRdLekeyWd0ulYncjNzPJ0KJNLGgV8DdgXeA1wuKTXdDLUUhO5pE+l30I3SrpE0sll1mdm1ikq+F8BuwL3RsR9EfE8cClwYCdjLa2PXNLrgXcCOwNrAHOBOTn3TQemp7dPjVmr7+4OhTAOWNqhsjqtqrFVNS6obmxVjQtWn9i27lA5L7pt7pxr111T4wrevrak2U3vZ0bEzKb3WwAPNL1/EHjDcGNsVubDzt2BH0fEc8Bzkv4r76b0A8/MuzYckmZHxNROl9sJVY2tqnFBdWOralzg2IYjIvbpdgxD4T5yM7NyPQRs1fR+y3SuY8pM5DcBb5e0tqQxwP4l1mVmVlW3ApMkbSNpTeAw4MpOVlBa10pE3CrpSmA+8DDwW+DxsurL0fHumg6qamxVjQuqG1tV4wLHVgkRsVzS8cC1wCjgwohY0Mk6FBGdLO/lhUtjIuIpSesCNwDTI2JuaRWama2Gyp7ZOTONl1wbuNhJ3Mys80ptkZvZyJK0bkQ80+04bGR51IpVlqTxUp1WvOguSQcAX00P1Gw14kQ+AiT1pT+dlAqStAVwGtl05kr+vVUpLkkbAycC/wZsKWlsl0OyEdRziVzSOt2OoVVErEwvt5Y0ukoJoEHSq7sdQ4s/ks0Efh3wjir+nQGbA0iqwiqizwPLgdOBLwErB769OyRtWtH/LWutpxJ5GuLzeUmflbRBBeLZTdJh6fUJwA+BC4FTGq30KpB0HHC2pPHdjgWylm5kD2/6yBYZ+ifgwColgPRv7RuSPgd8WNJa3YwnIp4Efka2hsf8iPhLlf6+ACS9FjgTeGfVYqu7KrQkOkLSh4FDgCPI1nXZQtJnIuL3XQxrI+CzknYEtkvxbQ1MAz4n6ZSm1npXpH7VGcABEfFwN2NpiIiQdCRwAnA08H7gLcBoSZdHl5/QSzoIOBQ4ALgCWBARy7oZUzKL7FvMuZL+HBFf7nZADZLeDpxMlnMmpnNd/9+yV1SmVTgckl4B7EI2Y+qdwG3p0jmSJnUrroi4mmxBsIOzt7EQuBH4DjCerLXZbZsDsyJikaQ1uh1Mkx2A70fE7cDHgXuB44FDKtCa2wD4CnAQ8AJwEoCk7bsZVEQsiojryRozx6VvDV2Xvun9E3BsROxO9s10GhX7llVnPZHII+IJ4CPAK4GD04I37wNeDxzVzaf4EXEd8ElgP0nvjojn07eE9YEq9EsvAvaQtENEvAAg6ajU6uymucDuknZMf2dfJZuPMAUY093QuB84GzgmIvaOiOclnQh8sAq/DCPiNuBdwBmSju12PGT9933Axun9BWQt848Bf9utoHpJz3StRMQySc+Qff3+f2RdGNcD/5HWAO5mbD+WdBTZN4RXA/OASWTJqttuAnYDjpZ0E9kvmBOBw7saFfyC7BfxEZJ+BqwDPAWck/qDu2kO8GNgpaRpwASyhsP7Gr8Muy0i5qfYnq1ALH+WdDnwVkmPR8Qdkq4gWzzqMEk/r0jXVG311ISg9MDpo8BeZF0Gh0TE77ob1UtSK/dy4CrgYxFxX5dDAkDSZmQPyQ4gWw/nsxExv7tRgaTNgXekYzlwchXighf/zg5Ix6PA2RHx2+5GVV2StgSOBaaSNWDeBRxFNsT0U6kLzdrUU4kcIH213RRYGREdXSqyEyS9GVgUEfd3O5ZWjS6obn+DaSVpPbJ/q091O5ZWja6UqrTEqyw9y9qNbLOZa4B1gfOBt1XlQXtd9VwiN7Pqk/QW4LNkD0DdGh8mJ3IzG3Gpa2rNiFjU7Vh6gRO5mVnN9cTwQzOz1ZkTuZlZzTmRm5nVnBO5rULSCknzJN0h6Qdpq752y5om6ar0+gBJpwxw74ZpzZyh1nGGpJOLnm+55yJJ7xpCXRMl3THUGM3K5ERueZ6NiMkRsRPZ9OoZzReVGfK/nYi4MiI+N8AtGwJDTuRmqzsnchvMr4C/Si3RuyV9G7gD2ErS3pJuljQ3tdzHAEjaR9JdkuaSzcoknT9a0rnp9XhJV0i6PR27AZ8DtkvfBs5O9/2jpFslzZf06aayPinpHkk3ki2wNSBJH0rl3C7p8pZvGXtJmp3K2z/dP0rS2U11V2HNErNcTuTWL2UbJuwLNKaeTwK+HhE7Ak+TTa/eKyJ2AWYDJ0lam2y23tvJFrjatJ/izwF+GRE7k61cuQA4BViYvg38o6S9U527ApOBKZL2kDSFbKXLycB+ZGuyDOaHEfH6VN+dwDFN1yamOv6ObI3xtdP1xyPi9an8D0napkA9ZiOuZxbNso5aR9K89PpXZKvVbU62tMAt6fxfky3De1NaiXRN4GbgVcAfGuvAS/ou2VK+rd4KvBcgIlYAj0vaqOWevdPRWJZ4DFliXx+4orHJsKQrC/xMO0n6F7LumzHAtU3XLkvrwv9e0n3pZ9gbeG1T//kGqe57CtRlNqKcyC3PsxExuflEStZPN58CrouIw1vue9nnhklkC3h9s6WOj7ZR1kXAQRFxu6SjydbDbmidFRep7hMiojnhI2liG3WblcpdK9auW8jWC/8ryBa2Shsr3AVMlLRduq+/5XCvB45Lnx2lbGu+J8la2w3XAh9o6nvfQtIrgRuAgyStI2l9sm6cwawPLE6LXB3Zcu0QSX0p5m2Bu1PdxzUWxZK0fVq8y6xy3CK3tkTEktSyvUQv7Vd5WkTcI2k6cLWy9eF/xcuTc8PfAzMlHQOsAI6LiJsl3ZSG9/136id/NXBz+kbwFPCeiJgraRZwO/AIcGuBkD8F/BpYkr5FcuMAAABRSURBVP5sjun/A78BXgHMiIjnJP0HWd/5XGWVLyHbEciscrzWiplZzblrxcys5pzIzcxqzonczKzmnMjNzGrOidzMrOacyM3Mas6J3Mys5v4PpBzofkhdBSYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAEmCAYAAACtaxGwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3debwcVZn/8c/3JkCCCYsEQZKwyKKAI1tABReUyA8cCG4ouwxLBAUcERB/ICLKgDLiuKAQFQFRFkE0ahRHGEQYloRVAgMTkEhQIYGwr4Fn/qjT0On07a6b25Wquvf75lUvqrqqz3n63punT58+55QiAjMzq5e+sgMwM7OBc/I2M6shJ28zsxpy8jYzqyEnbzOzGnLyNjOrISfvYUbSiZLOT/trS3pK0oge13G/pMm9LDNHnYdKeii9ntUGUc5Tkt7Qy9jKImm2pO3LjsOK4eTdYylxPSzpNU2PHSTpqhLDaisi/hoRYyLipbJjGQxJywGnAzum1/PI0paVnn9f76LrPUnnSPpKt+siYtOIuGoZhGQlcPIuxgjg04MtRBn/jrpbAxgFzC47kCqQNLLsGKx4TgzFOA04StIq7U5K2lbSTEmPp/9v23TuKkknS7oWeAZ4Q3rsK5L+O32s/5Wk1ST9RNITqYx1m8r4pqQH0rmbJL2znzjWlRSSRkp6eyq7sT0n6f50XZ+kYyXdK+kRSRdLem1TOftKmpvOHdfpByNptKSvp+sfl3SNpNHp3JT0Uf+x9Jo3bnre/ZKOknR7et5FkkZJ2gi4O132mKQrm19Xy8/1oLS/gaQ/pnIWSLqo6bqQtEHaX1nSeZLmp3iPb7yZSto/xf7vkhZK+ouknTu87vslHZ3if1rSDyWtIem3kp6U9AdJqzZd/zNJ/0gxXi1p0/T4VGBv4JjG30JT+Z+TdDvwdPqdvtJ9JWmGpK83lX+hpLM7/a6s4iLCWw834H5gMvBz4CvpsYOAq9L+a4GFwL7ASGDPdLxaOn8V8Fdg03R+ufTYHGB9YGXgTuCeVM9I4DzgR00x7AOsls59FvgHMCqdOxE4P+2vCwQwsuU1LAf8ETglHX8auB6YAKwAnAVckM5tAjwFvCudOx1YBEzu5+dzRno948k+oWybnrcR8DTwvlT/Mek1L9/0c70RWCv9DO8CDmn3Otq9rlTnQWn/AuA4ssbLKOAdTdcFsEHaPw/4JTA2lXkPcGA6tz/wInBweh2HAn8D1OHv4nqyTwnjgYeBm4EtUgxXAl9suv6AVO8KwH8AtzadO4f0t9VS/q3ARGB0899i2l8z1flesuR/HzC27H8v3gaRa8oOYKhtvJq83ww8DqzO4sl7X+DGludcB+yf9q8CTmo5fxVwXNPx14HfNh3v2vyPu01MC4HN0v6JdE/e3wN+DfSl47uAHZrOvz4lrpHACcCFTedeA7xAm+SdkuWzjVhazn0BuLjl2geB7Zt+rvs0nf8acGa719HudbF48j4PmAZMaBNHABuQJeQXgE2azn2i6fe4PzCn6dyK6blrdvi72Lvp+FLge03HhwO/6Oe5q6SyV07H59A+eR/Q7m+x6fjDwAPAApresLzVc3O3SUEi4g6yBHhsy6m1gLktj80la401PNCmyIea9p9tczymcZC6F+5KH7kfI2utj8sTt6RPANsDe0XEy+nhdYDLUnfGY2TJ/CWyVuRazfFGxNNAf18YjiNrZd7b5txiP5dU9wMs/nP5R9P+MzS95gE6BhBwY+qmOaCfWJdj8d9V6+/plXgi4pm02ymmXL9DSSMknZq6qZ4gS8KNmDpp93fT7Fdkb0p3R8Q1Xa61inPyLtYXyT5WN/+D/xtZMmy2Nlkrs2Gpl3pM/dvHAB8FVo2IVcg+ASjnc78M7BYRTzSdegDYOSJWadpGRcSDwN/JPqo3yliRrMumnQXAc2TdP60W+7lIUir3wTbXdvN0+v+KTY+t2diJiH9ExMERsRZZa/q7jX7ullhfZPHfVevvqSh7AbuRfYJbmeyTBLz6O+zv76Pb383JZG+8r5e05yBjtJI5eRcoIuYAFwFHND08A9hI0l7pS6WPkfUb/7pH1Y4l63OeD4yUdAKwUrcnSZoIXAzsFxH3tJw+EzhZ0jrp2tUl7ZbOXQLsIukdkpYHTqKfv6vUmj4bOF3SWqmF+XZJK6S6/1nSDsqG/n0WeB747wG9+qye+WRJdp9UxwE0vWFI2l3ShHS4kCzpvdxSxkspppMljU2v/Ujg/IHGsxTGkr32R8jegP6t5fxDwIDGokt6F/AvwH7Ax4FvSxrf+VlWZU7exTuJrB8YgMjGIO9ClpweIWsl7xIRC3pU3+XA78i+XJtL1tLt9nEaYAeybpBL9OqIk8bQu28C04HfS3qS7Iu3t6bXMxv4FPBTslb4QmBeh3qOAv4MzAQeBb5K1rd+N9kXrd8ma/XuCuwaES/kfN2tDgaOJvsZb8ribwJbAzdIeiq9rk9H+7Hdh5O14u8DrkmvcVmM0DiP7Hf3INmX09e3nP8hsEnqxvpFt8IkrZTKPCwiHoyIP6UyfpQ+4VgNKcI3YzAzqxu3vM3MasjJ28ysQJLOVrZkxh39nJekb0makyZxbZmnXCdvM7NinQPs1OH8zsCGaZtKNs+iq0KTt6RfKJuePTtN6zUzG1Yi4mqyL+f7sxtwXmSuB1aR9Ppu5Ra9gM0BEfGosrUrZkq6NFpWfEtJPUvsfcttpdFLvZpnoTbbqJqjqvo8VsCGublz72fBggU9/ZcwYqV1IhY9m+vaeHb+bLJRXQ3TImLaAKobz+Ijwualx/7e6UlFJ+8jJH0w7U8k+1iwWPJOL3IaQN+Y18cKmx9YcEhL58rff6nsENoavXxPl+I2q53t3jqp52XGomdZ4Y0fzXXtc7ee8VxE9D6ILgpL3soWgZ8MvD0inlG2nvWoouozM+sdwbJbjflBmmYpky0A13Umb5HRrQwsTIn7TcDbCqzLzKx3BEj5tsGbDuyXRp28DXg8Ijp2mUCx3Sa/Aw6RdBfZesuts8TMzKqrRy1vSReQLfY2TtI8sjWPlgOIiDPJlsx4P9kSyM+QLWPQVWHJOyKeJxsCY2ZWM4K+3nyfFBEdFwGLbJr7pwZarm+XZGbWTsWXfXHyNjNrJZblF5ZLxcnbzGwJPfsysjBO3mZm7bjlbWZWQ255m5nVzTKdpLNUnLzNzFo1JulUmJO3mVk7bnmbmdWNYES1F31z8jYza+Vx3mZmNeU+bzOzuvFoEzOzenLLO7/N3zieq684qeww2nrTkdPLDqGtXx3z3rJD6NfG41cqOwSzpeeWt5lZzfTuRguFcfI2M2vHLW8zsxpyy9vMrG482sTMrH5Ez26DVhQnbzOzJbjlbWZWT+7zNjOrIbe8zcxqyC1vM7Oakfu8zczqyS1vM7P6kZO3mVm9ZLewdPI2M6sXCfU5eZuZ1Y5b3mZmNVT15F3YWBhJ60q6o6jyzcyKJCnXVha3vM3MWiltFVb0KPSRkn4i6S5Jl0haseD6zMwGTeRrdZfZ8i46eb8R+G5EbAw8AXyy9QJJUyXNkjRrwfz5BYdjZpbPcE/eD0TEtWn/fOAdrRdExLSImBQRk8atvnrB4ZiZ5VP15F10n3d0OTYzq6RhO9okWVvS29P+XsA1BddnZjZ4GsBWkqKT993ApyTdBawKfK/g+szMBk2Ivr6+XFtZCus2iYj7gTcVVb6ZWZF62W0iaSfgm8AI4AcRcWrL+bWBc4FV0jXHRsSMTmVWe8FaM7Oy9KjbRNII4AxgZ2ATYE9Jm7RcdjxwcURsAewBfLdbuU7eZmat1NPRJtsAcyLivoh4AbgQ2K3lmgBWSvsrA3/rVqhnWJqZtTGAbpNxkmY1HU+LiGlNx+OBB5qO5wFvbSnjROD3kg4HXgNM7lapk7eZWRsDSN4LImLSIKvbEzgnIr6eRuj9WNKbI+Ll/p7g5G1m1qIxPb5HHgQmNh1PSI81OxDYCSAirpM0ChgHPNxfoe7zNjNrp3fjvGcCG0paT9LyZF9ITm+55q/ADgCSNgZGAR3XC3HL28yslXo3VDAiFkk6DLicbBjg2RExW9JJwKyImA58Fvi+pM+QfXm5f0R0nJHu5G1m1kYvx3mnMdszWh47oWn/TmC7gZTp5G1m1obvYWlmVkNVX5jKydvMrEXZy73m4eRtZtaGk7eZWQ05eQ+AgJEjqjn0/Pwj3ll2CG3tfdb1ZYfQr1knvq/sEPrVV/Evo6wCKv4nUqnkbWZWFW55m5nVTQ8n6RTFydvMrIWAiuduJ28zsyWp8t+LOHmbmbXhbhMzs7qRu03MzGpHVH84qZO3mVkbbnmbmdWQ+7zNzOrGfd5mZvWTjfOudvZ28jYzW4KXhDUzq6WK524nbzOzJchDBc3Masd93mZmNVXx3O3kbWbWTtVb3oXftkbSfpJul3SbpB8XXZ+ZWS9I+bayFNrylrQpcDywbUQskPTaIuszM+uJGtyMoeiW93uBn0XEAoCIeLT1AklTJc2SNGv+gvkFh2Nm1l3jZgxVbnmXfrffiJgWEZMiYtLq41YvOxwzMxqTdPJsZSk6eV8J7C5pNQB3m5hZXVS95V1on3dEzJZ0MvBHSS8BtwD7F1mnmdmgeZIORMS5wLlF12Nm1iuepGNmVlNO3mZmNVTx3O3kbWbWjlveZmZ14zvpmJnVj3wzBjOzeqp47nbyNjNrp6/i2bv06fFmZlXUyxmWknaSdLekOZKO7eeaj0q6U9JsST/tVqZb3mZmLSQY0aMZlpJGAGcA7wPmATMlTY+IO5uu2RD4PLBdRCyU9Lpu5brlbWbWRg8XptoGmBMR90XEC8CFwG4t1xwMnBERCwEi4uFuhfbb8pa0UqcnRsQTXUM2M6upAXR5j5M0q+l4WkRMazoeDzzQdDwPeGtLGRtldepaYARwYkT8rlOlnbpNZgNBNs2/oXEcwNqdCjYzqyuRDRfMaUFETBpklSOBDYHtgQnA1ZL+KSIe6/SEtiJi4iCDMTOrrR4uKvgg0JxPJ6THms0DboiIF4G/SLqHLJnP7De+PDVL2kPS/0/7EyRtNZDIzcxqJWd/d84+75nAhpLWk7Q8sAcwveWaX5C1upE0jqwb5b5OhXYdbSLpO8BywLuAfwOeAc4Ets4T9VDxtvVXKzuEtsaNW7HsEPr18z+3Ni6q4yObTSg7BKu4Xg3zjohFkg4DLifrzz473evgJGBWRExP53aUdCfwEnB0RDzSqdw8QwW3jYgtJd2SAnk0vXuYmQ1JoreTdCJiBjCj5bETmvYDODJtueRJ3i9K6iP7kpJ0S7OX81ZgZlZHFZ9gmavP+wzgUmB1SV8CrgG+WmhUZmYlq/oNiLu2vCPiPEk3AZPTQ7tHxB3FhmVmVp5ezrAsSt7p8SOAF8m6Tjwr08yGvGqn7hyJWNJxwAXAWmTjE38q6fNFB2ZmVqbad5sA+wFbRMQzAJJOBm4BTikyMDOzsmSjTcqOorM8yfvvLdeNTI+ZmQ1NJbeq8+i0MNU3yPq4HwVmS7o8He9IhymbZmZDQcVzd8eWd2NEyWzgN02PX19cOGZm1VDblndE/HBZBmJmVhVDos9b0vrAycAmwKjG4xGxUYFxmZmVquot7zxjts8BfkT2ZrQzcDFwUYExmZmVSoIRUq6tLHmS94oRcTlARNwbEceTJXEzsyGrlzcgLkKeoYLPp4Wp7pV0CNki4mOLDcvMrFxV7zbJk7w/A7wGOIKs73tl4ICBViTpROCpiPj3gT7XzGxZq3juzrUw1Q1p90lg32LDMTMrn1BP1/MuQqdJOpeR1vBuJyI+1K3wtC7Kx4GHye6efNNSxGhmtmyV3J+dR6eW93cGU3C6z+UewOapnptpk7wlTQWmAkxc2zekN7NqqG2fd0RcMciy3wlc1rSgVesNNxv1TAOmAWy11aR+W/pmZstS1de+zruet5nZsCGq3/Iu8s3lauADkkZLGgvsWmBdZmY91ad8W1lyt7wlrRARz+e9PiJulnQRcBvZF5ZeidDMamFI3AZN0jbAD8nGd68taTPgoIg4vNtzI+JksrHhZma1UvHcnavb5FvALsAjABFxG/CeIoMyMyvbUJge3xcRc1s6718qKB4zs9JlS8JWu+mdJ3k/kLpOQtII4HDgnmLDMjMr11AYKngoWdfJ2sBDwB/SY2ZmQ1bFG9651jZ5mGympJnZsCDVeG2TBknfp80aJxExtZCIzMwqoOK5O1e3yR+a9kcBHyRbZMrMbMiq+lDBPN0mi93yTNKPgWsKi8jMrGRiCEzSaWM9YI1eB2JmVhklT33PI0+f90Je7fPuAx4Fji0yKDOzsolqZ++OyVvZzJzNyO5bCfByRHjZVjMb0rJJOmVH0VnHcegpUc+IiJfS5sRtZsNC1VcVzDOJ6FZJWxQeiZlZhUjKtZWl0z0sR0bEImALYKake4GnyT5RRERsuYxirIRVtz6s7BDaWjhzUHerM7M26tBt0qnP+0ZgS2DKMorFzKwaerxioKSdgG8CI4AfRMSp/Vz3YeASYOuImNWpzE7JWwARce/ShWtmVl+9mh6fFvQ7A3gfMI+sJ2N6RNzZct1Y4NPADXnK7ZS8V5d0ZH8nI+L0PBWYmdVNj7tNtgHmRMR9AJIuBHYD7my57svAV4Gj8xTa6QvLEcAYYGw/m5nZECVGKN8GjJM0q2lrXfdpPIsvKTIvPfZqbdKWwMSI+E3eCDu1vP8eESflLcjMbKjI7h6f+/IFETFpqeuS+oDTgf0H8ryufd5mZsNOb8dwPwhMbDqewKsTHyHryXgzcFUaergmMF3SlE5fWnZK3jssfaxmZvXWw/W8ZwIbSlqPLGnvAezVOBkRjwPjGseSrgKO6jbapN8+74h4dJABm5nVUqPbpBc3IE7zZQ4DLgfuAi6OiNmSTpK01EOxl2ZVQTOzIa+Xd9KJiBnAjJbHTujn2u3zlOnkbWbWxlC4k46Z2bAihsbd483MhhdR6qJTeTh5m5m1Ue3U7eRtZrYEQWP2ZGU5eZuZtVHx3O3kbWa2pHJvtJCHk7eZWQuPNjEzq6mqt7wLfXORtI+kGyXdKumstCi5mVnlKedWlsKSt6SNgY8B20XE5sBLwN5trpvaWAd3/oL5RYVjZpafanwD4h7YAdiK7JY/AKOBh1sviohpwDSArbaaFAXGY2aWy3Dv8xZwbkR8vsA6zMwKMZz7vK8APiLpdQCSXitpnQLrMzPrmT7l28pSWMs7Iu6UdDzw+3SbnxeBTwFzi6rTzKwXsm6Tare8Cx0qGBEXARcVWYeZWREq3mvicd5mZksSGs4tbzOzunLL28ysZoZ9n7eZWS3lvLlwmZy8zczacPI2M6shf2FpZlYzotwJOHk4eZuZtdFX8X4TJ28zszbcbWJmVjPuNjEzqyXPsDQzqx+P8zYzq6eK5+5qJe+5C5/lk5f8ueww2jrlm0eWHYINE089t6jsENoaM6pS6aJQWZ93tdP38PltmJkNQLVTt5O3mVl7Fc/eTt5mZm2428TMrIaqnbqdvM3M2qt49nbyNjNrITw93sysfjxJx8ysniqeu+krOwAzs0pSzi1PUdJOku6WNEfSsW3OHynpTkm3S7pC0jrdynTyNjNbgnL/17UkaQRwBrAzsAmwp6RNWi67BZgUEW8BLgG+1q1cJ28zszakfFsO2wBzIuK+iHgBuBDYrfmCiPiviHgmHV4PTOhWqJO3mVmLvD0mKXePkzSraZvaUtx44IGm43npsf4cCPy2W4z+wtLMrA3lH26yICIm9ajOfYBJwLu7XevkbWbWRg+HCj4ITGw6npAea6lPk4HjgHdHxPPdCnW3iZlZGz0cbDIT2FDSepKWB/YApi9Wl7QFcBYwJSIezlPoMkneko6QdJeknyyL+szMBmWAnd6dRMQi4DDgcuAu4OKImC3pJElT0mWnAWOAn0m6VdL0fop7xbLqNvkkMDki5i2j+szMBqWX0+MjYgYwo+WxE5r2Jw+0zMJb3pLOBN4A/FbSZ4quz8xssERPhwoWovCWd0QcImkn4D0RsaDo+szMesHT47uQNLUxPvK5JxeWHY6ZWaaH31gWofTkHRHTImJSREwaNXbVssMxMwPyT5Avi8d5m5m10VfxfhMnbzOzdpy8ISLWXRb1mJn1gu+kY2ZWR76TjplZPVU8dzt5m5m1VfHs7eRtZraEcocB5uHkbWbWhvu8zcxqpuTJk7k4eZuZtVPx7O3kbWbWRl/F+02cvM3M2qh26nbyNjNbkifpmJnVVbWzt5O3mVmLxp10qszJ28ysjYrnbidvM7N23PIegEf/MZ8LvnpW2WG09d2Z3yk7BBsmxoyq1D/LYcvT483M6qjaudvJ28ysnYrnbidvM7NWkmdYmpnVU7Vzt5O3mVk7Fc/dTt5mZu1UvNfEydvMbEm+k46ZWe3UYXp8X9kBmJnZwLnlbWbWRtVb3k7eZmZtuM/bzKxmskk6ZUfRmZO3mVk7Tt5mZvXjbhMzsxqq+heWHipoZtaGcm65ypJ2knS3pDmSjm1zfgVJF6XzN0hat1uZTt5mZu30KHtLGgGcAewMbALsKWmTlssOBBZGxAbAN4Cvdiu30OQt6Qvp3eYaSRdIOqrI+szMekU5/8thG2BORNwXES8AFwK7tVyzG3Bu2r8E2EHq3HFTWJ+3pK2BDwObAcsBNwM3tbluKjA1HT713K1n3N2jEMYBC3pUFqOXO6NXRUGPY+uhqsYF1Y2tqnHB8IltnR6V84pbbr7p8hWX17icl4+SNKvpeFpETGs6Hg880HQ8D3hrSxmvXBMRiyQ9DqxGh59RkV9Ybgf8MiKeA56T9Kt2F6UXOa3ducGQNCsiJvW63F6oamxVjQuqG1tV4wLHNhgRsVPZMXTjPm8zs2I9CExsOp6QHmt7jaSRwMrAI50KLTJ5XwvsKmmUpDHALgXWZWZWVTOBDSWtJ2l5YA9gess104GPp/2PAFdGRHQqtLBuk4iYKWk6cDvwEPBn4PGi6muj510xPVTV2KoaF1Q3tqrGBY6tElIf9mHA5cAI4OyImC3pJGBWREwHfgj8WNIc4FGyBN+RuiT3QZE0JiKekrQicDUwNSJuLqxCM7NhougZltPSeMZRwLlO3GZmvVFoy9vMli1JK0bEM2XHYcXzaBOrLElrdJuoYK+SNAX4dvpSzIY4J+9lQFJf+r8TUU6SxgPHk00lruTPrUpxSVoNOIJsWvUESa8tOSQr2JBL3pJGlx1Dq4h4Oe2uI2lklf7RN0jauOwYWvyNbEbuFsCHqvgzA9aCV8bllu0FYBHwReB04OXOl5dD0poV/V3WzpBK3mk4ztcknSJp5QrEs62kPdL+4cDPgbOBYxut8SqQdChwmqQ1yo4FshZtGuPaR7aQz+eA3ar0jz79rZ0p6VTgk5JWKDOeiHgSuJJsjYzbI+KxKv28ACS9BTgJ+HDVYqujKrQYekLSJ4Hdgb3I1lEZL+nLEfG/JYa1KnCKpE2B9VN86wDbA6dKOrapVV6K1E96CDAlIh4qM5aGiAhJewOHA/sD/wK8Bxgp6dJukxeKJukDwEeBKcBlwOyIeL7MmJKLyD6tfEfSwoj4RtkBNUjaFTiKLOesmx4r/XdZZ5Vp/Q2GpJWALckGtn8YuCWd+pakDcuKKyJ+Q7bo1gezw7gXuAb4MbAGWauybGsBF0XEXEnLlR1MkzcCP42I24BjgDnAYcDuFWi1rQz8B/AB4EXgSABJG5UZVETMjYgryBowh6ZPB6VLn+g+B3wiIrYj+wS6PRX7NFU3QyJ5R8QTwKeA1wEfTIvKfBzYGti3zG/fI+I/geOA90v6WES8kD4NjAWq0M88F3iXpDdGxIsAkvZNrcsy3QxsJ2nT9DP7Ntl8ga2AMeWGxv3AacCBEbFjRLwg6QjgoCq8AUbELWRTrE+U9Imy4yHrj+8jWyUPstmEI4HPAP+vrKDqbsh0m0TE85KeIfto/U9k3RNXAD9Ia+iWGdsvJe1L9klgY+BWYEOyBFW2a4Ftgf0lXUv2pnIEsGepUcFVZG++e0m6EhgNPAV8K/Xvlukm4JfAy5K2B9Ymayx8vPEGWLaIuD3F9mwFYlko6VLgvZIej4g7JF1GtkDTHpL+qyLdTrUypCbppC+N/hWYTNYdsHtE3FluVK9KrdlLgV8Dn4mI+0oOCQBJryf7omsK2fozp0TE7eVGBZLWAj6UtkXAUVWIC175mU1J2yPAaRHx53Kjqi5JE4BPAJPIGi0fAfYlGw76hdQ9ZgMwpJI3QPrYuibwckS0LrtYOknvBuZGxP1lx9Kq0b1U9ieVVpJeQ/a3+lTZsbRqdJNUpcVdZem7qW3JbtAyA1gR+D7wvqp8WV4nQy55m1n1SXoPcArZl5hudS8FJ28zW+ZSt9PyETG37FjqysnbzKyGhsRQQTOz4cbJ28yshpy8zcxqyMnbliDpJUm3SrpD0s/SbeyWtqztJf067U+RdGyHa1dJa9QMtI4TJR2V9/GWa86R9JEB1LWupDsGGqNZrzl5WzvPRsTmEfFmsqnNhzSfVGbAfzsRMT0iTu1wySrAgJO32XDk5G3d/AnYILU475Z0HnAHMFHSjpKuk3RzaqGPAZC0k6T/kXQz2exI0uP7S/pO2l9D0mWSbkvbtsCpwPqp1X9auu5oSTMl3S7pS01lHSfpHknXkC1i1ZGkg1M5t0m6tOXTxGRJs1J5u6TrR0g6ranuKqwRYvYKJ2/rl7KbDOwMNKZ9bwh8NyI2BZ4mm9o8OSK2BGYBR0oaRTZrbleyRaTW7Kf4bwF/jIjNyFaEnA0cC9ybWv1HS9ox1bkNsDmwlaR3SdqKbAXJzYH3k62B0s3PI2LrVN9dwIFN59ZNdfwz2Rrdo9L5xyNi61T+wZLWy1GP2TIxZBamsp4aLenWtP8nslXg1iKb1n99evxtZEvaXptW9VweuA54E/CXxjrqks4nWxa31XuB/QAi4iXgcUmrtlyzY9oaS/yOIUvmY4HLGjfalTQ9x2t6s6SvkHXNjAEubzp3cVpX/X8l3Zdew47AW5r6w1dOdd+Toy6zwjl5WzvPRsTmzQ+kBP1080PAf0bEni3XLfa8QRLZIllntdTxr0tR1jnAByLiNkn7k60n3dA6Uy1S3YdHRHOSR5YKvOUAAAEMSURBVNK6S1G3Wc+528SW1vVk621vANniUelmBP8DrCtp/XRdf0vLXgEcmp47Qtlt654ka1U3XA4c0NSXPl7S64CrgQ9IGi1pLFkXTTdjgb+nhaT2bjm3u6S+FPMbgLtT3Yc2Fp6StFFaIMusEtzytqUSEfNTC/YCvXr/xuMj4h5JU4HfKFtf/U8snpAbPg1Mk3Qg8BJwaERcJ+naNBTvt6nfe2PgutTyfwrYJyJulnQRcBvwMDAzR8hfAG4A5qf/N8f0V+BGYCXgkIh4TtIPyPrCb1ZW+XyyO+eYVYLXNjEzqyF3m5iZ1ZCTt5lZDTl5m5nVkJO3mVkNOXmbmdWQk7eZWQ05eZuZ1dD/Ad3TM6x0TKKJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import svm, datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "# Compute confusion matrix\n",
    "cm = confusion_matrix(y_compare, pred)\n",
    "np.set_printoptions(precision=2)\n",
    "print('Confusion matrix, without normalization')\n",
    "print(cm)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, products)\n",
    "\n",
    "# Normalize the confusion matrix by row (i.e by the number of samples\n",
    "# in each class)\n",
    "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "print('Normalized confusion matrix')\n",
    "print(cm_normalized)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm_normalized, products, title='Normalized confusion matrix')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}