t81_558_class_08_3_keras_hyperparameters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 8: Kaggle Data Sets**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 8 Material\n",
    "\n",
    "* Part 8.1: Introduction to Kaggle [[Video]](https://www.youtube.com/watch?v=v4lJBhdCuCU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_1_kaggle_intro.ipynb)\n",
    "* Part 8.2: Building Ensembles with Scikit-Learn and Keras [[Video]](https://www.youtube.com/watch?v=LQ-9ZRBLasw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_2_keras_ensembles.ipynb)\n",
    "* **Part 8.3: How Should you Architect Your Keras Neural Network: Hyperparameters** [[Video]](https://www.youtube.com/watch?v=1q9klwSoUQw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_3_keras_hyperparameters.ipynb)\n",
    "* Part 8.4: Bayesian Hyperparameter Optimization for Keras [[Video]](https://www.youtube.com/watch?v=sXdxyUCCm8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb)\n",
    "* Part 8.5: Current Semester's Kaggle [[Video]] [[Notebook]](t81_558_class_08_5_kaggle_project.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 8.3: How Should you Architect Your Keras Neural Network: Hyperparameters\n",
    "\n",
    "* [Guide to choosing Hyperparameters for your Neural Networks](https://towardsdatascience.com/guide-to-choosing-hyperparameters-for-your-neural-networks-38244e87dafe)\n",
    "\n",
    "### Number of Hidden Layers and Neuron Counts\n",
    "\n",
    "* [Keras Layers](https://keras.io/layers/core/)\n",
    "\n",
    "Layer types and when to use them:\n",
    "\n",
    "* **Activation** - Layer that simply adds an activation function, the activation function can also be specified as part of a Dense (or other) layer type.\n",
    "* **ActivityRegularization** Used to add L1/L2 regularization outside of a layer.  L1 and L2 can also be specified as part of a Dense (or other) layer type.\n",
    "* **Dense** - The original neural network layer type.  Every neuron is connected to the next layer.  The input vector is one-dimensional and placing certain inputs next to each other does not have an effect. \n",
    "* **Dropout** - Dropout consists in randomly setting a fraction rate of input units to 0 at each update during training time, which helps prevent overfitting.  Dropout only occurs during training.\n",
    "* **Flatten** - Flattens the input to 1D. Does not affect the batch size.\n",
    "* **Input** - A Keras tensor is a tensor object from the underlying backend (Theano, TensorFlow or CNTK), which we augment with certain attributes that allow us to build a Keras model just by knowing the inputs and outputs of the model.\n",
    "* **Lambda** - Wraps arbitrary expression as a Layer object.\n",
    "* **Masking** - Masks a sequence by using a mask value to skip timesteps.\n",
    "* **Permute** - Permutes the dimensions of the input according to a given pattern. Useful for e.g. connecting RNNs and convnets together.\n",
    "* **RepeatVector** - Repeats the input n times.\n",
    "* **Reshape** - Similar to Numpy reshapes.\n",
    "* **SpatialDropout1D** - This version performs the same function as Dropout, however it drops entire 1D feature maps instead of individual elements. \n",
    "* **SpatialDropout2D** - This version performs the same function as Dropout, however it drops entire 2D feature maps instead of individual elements\n",
    "* **SpatialDropout3D** - This version performs the same function as Dropout, however it drops entire 3D feature maps instead of individual elements. \n",
    "\n",
    "\n",
    "### Activation Functions\n",
    "\n",
    "* [Keras Activation Functions](https://keras.io/activations/)\n",
    "* [Activation Function Cheat Sheets](https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html)\n",
    "\n",
    "As follows:\n",
    "\n",
    "* **softmax** - Used for multi-class classification.  Ensures all output neurons behave as probabilities and sum to 1.0.\n",
    "* **elu** - Exponential linear unit.  Exponential Linear Unit or its widely known name ELU is a function that tend to converge cost to zero faster and produce more accurate results.  Can produce negative outputs.\n",
    "* **selu** - Scaled Exponential Linear Unit (SELU), essentially **elu** multiplied by a scaling constant.\n",
    "* **softplus** - Softplus activation function. $log(exp(x) + 1)$  [Introduced](https://papers.nips.cc/paper/1920-incorporating-second-order-functional-knowledge-for-better-option-pricing.pdf) in 2001.\n",
    "* **softsign** Softsign activation function. $x / (abs(x) + 1)$ Similar to tanh, but not widely used.\n",
    "* **relu** - Very popular neural network activation function.  Used for hidden layers, cannot output negative values.  No trainable parameters.\n",
    "* **tanh** Classic neural network activation function, though often replaced by relu family on modern networks.\n",
    "* **sigmoid** - Classic neural network activation.  Often used on output layer of a binary classifier.\n",
    "* **hard_sigmoid** - Less computationally expensive variant of sigmoid.\n",
    "* **exponential** - Exponential (base e) activation function.\n",
    "* **linear** - Pass through activation function. Usually used on the output layer of a regression neural network.\n",
    "\n",
    "### Advanced Activation Functions\n",
    "\n",
    "* [Keras Advanced Activation Functions](https://keras.io/layers/advanced-activations/)\n",
    "\n",
    "The advanced activation functions contain parameters that are trained during neural network fitting. As follows:\n",
    "\n",
    "* **LeakyReLU** - Leaky version of a Rectified Linear Unit. It allows a small gradient when the unit is not active, controlled by alpha hyperparameter.\n",
    "* **PReLU** - Parametric Rectified Linear Unit, learns the alpha hyperparameter. \n",
    "\n",
    "### Regularization: L1, L2, Dropout\n",
    "\n",
    "* [Keras Regularization](https://keras.io/regularizers/)\n",
    "* [Keras Dropout](https://keras.io/layers/core/)\n",
    "\n",
    "### Batch Normalization\n",
    "\n",
    "* [Keras Batch Normalization](https://keras.io/layers/normalization/)\n",
    "\n",
    "* Ioffe, S., & Szegedy, C. (2015). [Batch normalization: Accelerating deep network training by reducing internal covariate shift](https://arxiv.org/abs/1502.03167). *arXiv preprint arXiv:1502.03167*.\n",
    "\n",
    "Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.  Can allow learning rate to be larger.\n",
    "\n",
    "\n",
    "### Training Parameters\n",
    "\n",
    "* [Keras Optimizers](https://keras.io/optimizers/)\n",
    "\n",
    "* **Batch Size** - Usually small, such as 32 or so.\n",
    "* **Learning Rate**  - Usually small, 1e-3 or so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0819 15:28:03.170156  1808 training_utils.py:1210] When passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.\n",
      "W0819 15:28:03.440148  1808 deprecation.py:323] From C:\\Users\\jheaton\\AppData\\Local\\Continuum\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\optimizer_v2\\optimizer_v2.py:466: BaseResourceVariable.constraint (from tensorflow.python.ops.resource_variable_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Apply a constraint manually following the optimizer update step.\n",
      "W0819 15:28:26.512756  1808 training_utils.py:1210] When passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.\n",
      "W0819 15:28:26.589758  1808 training_utils.py:1210] When passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.\n",
      "W0819 15:28:37.886053  1808 training_utils.py:1210] When passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.7551467691035941\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "import time\n",
    "import tensorflow.keras.initializers\n",
    "import statistics\n",
    "import tensorflow.keras\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation, Dropout\n",
    "from tensorflow.keras import regularizers\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "from tensorflow.keras.layers import LeakyReLU,PReLU\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "\n",
    "\n",
    "def evaluate_network(dropout,lr,neuronPct,neuronShrink):\n",
    "    SPLITS = 2\n",
    "\n",
    "    # Bootstrap\n",
    "    boot = StratifiedShuffleSplit(n_splits=SPLITS, test_size=0.1)\n",
    "\n",
    "    # Track progress\n",
    "    mean_benchmark = []\n",
    "    epochs_needed = []\n",
    "    num = 0\n",
    "    neuronCount = int(neuronPct * 5000)\n",
    "\n",
    "    # Loop through samples\n",
    "    for train, test in boot.split(x,df['product']):\n",
    "        start_time = time.time()\n",
    "        num+=1\n",
    "\n",
    "        # Split train and test\n",
    "        x_train = x[train]\n",
    "        y_train = y[train]\n",
    "        x_test = x[test]\n",
    "        y_test = y[test]\n",
    "\n",
    "        # Construct neural network\n",
    "        # kernel_initializer = tensorflow.keras.initializers.he_uniform(seed=None)\n",
    "        model = Sequential()\n",
    "        \n",
    "        layer = 0\n",
    "        while neuronCount>25 and layer<10:\n",
    "            #print(neuronCount)\n",
    "            if layer==0:\n",
    "                model.add(Dense(neuronCount, \n",
    "                    input_dim=x.shape[1], \n",
    "                    activation=PReLU())) \n",
    "            else:\n",
    "                model.add(Dense(neuronCount, activation=PReLU())) \n",
    "            model.add(Dropout(dropout))\n",
    "        \n",
    "            neuronCount = neuronCount * neuronShrink\n",
    "        \n",
    "        model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=lr))\n",
    "        monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "            patience=100, verbose=0, mode='auto', restore_best_weights=True)\n",
    "\n",
    "        # Train on the bootstrap sample\n",
    "        model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=0,epochs=1000)\n",
    "        epochs = monitor.stopped_epoch\n",
    "        epochs_needed.append(epochs)\n",
    "\n",
    "        # Predict on the out of boot (validation)\n",
    "        pred = model.predict(x_test)\n",
    "\n",
    "        # Measure this bootstrap's log loss\n",
    "        y_compare = np.argmax(y_test,axis=1) # For log loss calculation\n",
    "        score = metrics.log_loss(y_compare, pred)\n",
    "        mean_benchmark.append(score)\n",
    "        m1 = statistics.mean(mean_benchmark)\n",
    "        m2 = statistics.mean(epochs_needed)\n",
    "        mdev = statistics.pstdev(mean_benchmark)\n",
    "\n",
    "        # Record this iteration\n",
    "        time_took = time.time() - start_time\n",
    "        #print(f\"#{num}: score={score:.6f}, mean score={m1:.6f}, stdev={mdev:.6f}, epochs={epochs}, mean epochs={int(m2)}, time={hms_string(time_took)}\")\n",
    "\n",
    "    tensorflow.keras.backend.clear_session()\n",
    "    return (-m1)\n",
    "\n",
    "print(evaluate_network(\n",
    "    dropout=0.2,\n",
    "    lr=1e-3,\n",
    "    neuronPct=0.2,\n",
    "    neuronShrink=0.2))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}