t81_558_class_03_3_save_load.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_1_neural_net.ipynb)\n",
    "* Part 3.2: Introduction to Tensorflow and Keras [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_2_keras.ipynb)\n",
    "* **Part 3.3: Saving and Loading a Keras Neural Network** [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_3_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_4_early_stop.ipynb)\n",
    "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_5_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.3: Saving and Loading a Keras Neural Network\n",
    "\n",
    "Complex neural networks will take a long time to fit/train.  It is helpful to be able to save these neural networks so that they can be reloaded later.  A reloaded neural network will not require retraining.  Keras provides three formats for neural network saving.\n",
    "\n",
    "* **YAML** - Stores the neural network structure (no weights) in the [YAML file format](https://en.wikipedia.org/wiki/YAML).\n",
    "* **JSON** - Stores the neural network structure (no weights) in the [JSON file format](https://en.wikipedia.org/wiki/JSON).\n",
    "* **HDF5** - Stores the complete neural network (with weights) in the [HDF5 file format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format). Do not confuse HDF5 with [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop).  They are different.  We do not use HDFS in this class.\n",
    "\n",
    "Usually you will want to save in HDF5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 398 samples\n",
      "Epoch 1/100\n",
      "398/398 - 0s - loss: 2363.8779\n",
      "Epoch 2/100\n",
      "398/398 - 0s - loss: 515.8320\n",
      "Epoch 3/100\n",
      "398/398 - 0s - loss: 272.4640\n",
      "Epoch 4/100\n",
      "398/398 - 0s - loss: 218.8972\n",
      "Epoch 5/100\n",
      "398/398 - 0s - loss: 203.7403\n",
      "Epoch 6/100\n",
      "398/398 - 0s - loss: 196.9009\n",
      "Epoch 7/100\n",
      "398/398 - 0s - loss: 193.1863\n",
      "Epoch 8/100\n",
      "398/398 - 0s - loss: 190.5076\n",
      "Epoch 9/100\n",
      "398/398 - 0s - loss: 180.3763\n",
      "Epoch 10/100\n",
      "398/398 - 0s - loss: 175.6821\n",
      "Epoch 11/100\n",
      "398/398 - 0s - loss: 180.3716\n",
      "Epoch 12/100\n",
      "398/398 - 0s - loss: 166.5054\n",
      "Epoch 13/100\n",
      "398/398 - 0s - loss: 165.7250\n",
      "Epoch 14/100\n",
      "398/398 - 0s - loss: 156.9933\n",
      "Epoch 15/100\n",
      "398/398 - 0s - loss: 158.5421\n",
      "Epoch 16/100\n",
      "398/398 - 0s - loss: 156.1468\n",
      "Epoch 17/100\n",
      "398/398 - 0s - loss: 150.7693\n",
      "Epoch 18/100\n",
      "398/398 - 0s - loss: 163.5738\n",
      "Epoch 19/100\n",
      "398/398 - 0s - loss: 143.2920\n",
      "Epoch 20/100\n",
      "398/398 - 0s - loss: 139.2389\n",
      "Epoch 21/100\n",
      "398/398 - 0s - loss: 138.5074\n",
      "Epoch 22/100\n",
      "398/398 - 0s - loss: 131.6507\n",
      "Epoch 23/100\n",
      "398/398 - 0s - loss: 131.3154\n",
      "Epoch 24/100\n",
      "398/398 - 0s - loss: 127.9629\n",
      "Epoch 25/100\n",
      "398/398 - 0s - loss: 125.8971\n",
      "Epoch 26/100\n",
      "398/398 - 0s - loss: 128.2931\n",
      "Epoch 27/100\n",
      "398/398 - 0s - loss: 120.6059\n",
      "Epoch 28/100\n",
      "398/398 - 0s - loss: 121.3194\n",
      "Epoch 29/100\n",
      "398/398 - 0s - loss: 117.7072\n",
      "Epoch 30/100\n",
      "398/398 - 0s - loss: 120.7639\n",
      "Epoch 31/100\n",
      "398/398 - 0s - loss: 115.9290\n",
      "Epoch 32/100\n",
      "398/398 - 0s - loss: 111.2488\n",
      "Epoch 33/100\n",
      "398/398 - 0s - loss: 108.1675\n",
      "Epoch 34/100\n",
      "398/398 - 0s - loss: 109.8107\n",
      "Epoch 35/100\n",
      "398/398 - 0s - loss: 105.6261\n",
      "Epoch 36/100\n",
      "398/398 - 0s - loss: 102.7176\n",
      "Epoch 37/100\n",
      "398/398 - 0s - loss: 104.2149\n",
      "Epoch 38/100\n",
      "398/398 - 0s - loss: 105.8699\n",
      "Epoch 39/100\n",
      "398/398 - 0s - loss: 99.8017\n",
      "Epoch 40/100\n",
      "398/398 - 0s - loss: 100.1521\n",
      "Epoch 41/100\n",
      "398/398 - 0s - loss: 106.0208\n",
      "Epoch 42/100\n",
      "398/398 - 0s - loss: 96.6723\n",
      "Epoch 43/100\n",
      "398/398 - 0s - loss: 97.1959\n",
      "Epoch 44/100\n",
      "398/398 - 0s - loss: 96.9985\n",
      "Epoch 45/100\n",
      "398/398 - 0s - loss: 89.7726\n",
      "Epoch 46/100\n",
      "398/398 - 0s - loss: 86.2898\n",
      "Epoch 47/100\n",
      "398/398 - 0s - loss: 85.2615\n",
      "Epoch 48/100\n",
      "398/398 - 0s - loss: 91.1641\n",
      "Epoch 49/100\n",
      "398/398 - 0s - loss: 94.0940\n",
      "Epoch 50/100\n",
      "398/398 - 0s - loss: 88.7233\n",
      "Epoch 51/100\n",
      "398/398 - 0s - loss: 83.5340\n",
      "Epoch 52/100\n",
      "398/398 - 0s - loss: 79.6850\n",
      "Epoch 53/100\n",
      "398/398 - 0s - loss: 88.2901\n",
      "Epoch 54/100\n",
      "398/398 - 0s - loss: 78.3867\n",
      "Epoch 55/100\n",
      "398/398 - 0s - loss: 78.2893\n",
      "Epoch 56/100\n",
      "398/398 - 0s - loss: 76.9312\n",
      "Epoch 57/100\n",
      "398/398 - 0s - loss: 79.6265\n",
      "Epoch 58/100\n",
      "398/398 - 0s - loss: 81.1463\n",
      "Epoch 59/100\n",
      "398/398 - 0s - loss: 70.2810\n",
      "Epoch 60/100\n",
      "398/398 - 0s - loss: 72.4620\n",
      "Epoch 61/100\n",
      "398/398 - 0s - loss: 68.5006\n",
      "Epoch 62/100\n",
      "398/398 - 0s - loss: 68.8582\n",
      "Epoch 63/100\n",
      "398/398 - 0s - loss: 64.0886\n",
      "Epoch 64/100\n",
      "398/398 - 0s - loss: 71.4101\n",
      "Epoch 65/100\n",
      "398/398 - 0s - loss: 76.8993\n",
      "Epoch 66/100\n",
      "398/398 - 0s - loss: 64.9849\n",
      "Epoch 67/100\n",
      "398/398 - 0s - loss: 60.9163\n",
      "Epoch 68/100\n",
      "398/398 - 0s - loss: 61.2893\n",
      "Epoch 69/100\n",
      "398/398 - 0s - loss: 60.9643\n",
      "Epoch 70/100\n",
      "398/398 - 0s - loss: 58.1576\n",
      "Epoch 71/100\n",
      "398/398 - 0s - loss: 63.6915\n",
      "Epoch 72/100\n",
      "398/398 - 0s - loss: 66.7531\n",
      "Epoch 73/100\n",
      "398/398 - 0s - loss: 55.5604\n",
      "Epoch 74/100\n",
      "398/398 - 0s - loss: 52.3791\n",
      "Epoch 75/100\n",
      "398/398 - 0s - loss: 51.7025\n",
      "Epoch 76/100\n",
      "398/398 - 0s - loss: 53.3734\n",
      "Epoch 77/100\n",
      "398/398 - 0s - loss: 56.5933\n",
      "Epoch 78/100\n",
      "398/398 - 0s - loss: 57.8872\n",
      "Epoch 79/100\n",
      "398/398 - 0s - loss: 51.7859\n",
      "Epoch 80/100\n",
      "398/398 - 0s - loss: 47.8782\n",
      "Epoch 81/100\n",
      "398/398 - 0s - loss: 46.0320\n",
      "Epoch 82/100\n",
      "398/398 - 0s - loss: 51.8777\n",
      "Epoch 83/100\n",
      "398/398 - 0s - loss: 46.0369\n",
      "Epoch 84/100\n",
      "398/398 - 0s - loss: 45.0455\n",
      "Epoch 85/100\n",
      "398/398 - 0s - loss: 44.6317\n",
      "Epoch 86/100\n",
      "398/398 - 0s - loss: 40.9859\n",
      "Epoch 87/100\n",
      "398/398 - 0s - loss: 45.3110\n",
      "Epoch 88/100\n",
      "398/398 - 0s - loss: 39.6353\n",
      "Epoch 89/100\n",
      "398/398 - 0s - loss: 41.0722\n",
      "Epoch 90/100\n",
      "398/398 - 0s - loss: 38.8340\n",
      "Epoch 91/100\n",
      "398/398 - 0s - loss: 37.3409\n",
      "Epoch 92/100\n",
      "398/398 - 0s - loss: 36.5442\n",
      "Epoch 93/100\n",
      "398/398 - 0s - loss: 37.4860\n",
      "Epoch 94/100\n",
      "398/398 - 0s - loss: 37.7561\n",
      "Epoch 95/100\n",
      "398/398 - 0s - loss: 37.6984\n",
      "Epoch 96/100\n",
      "398/398 - 0s - loss: 35.7752\n",
      "Epoch 97/100\n",
      "398/398 - 0s - loss: 33.7147\n",
      "Epoch 98/100\n",
      "398/398 - 0s - loss: 34.2420\n",
      "Epoch 99/100\n",
      "398/398 - 0s - loss: 32.8322\n",
      "Epoch 100/100\n",
      "398/398 - 0s - loss: 37.8696\n",
      "Before save score (RMSE): 6.090650375914274\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "save_path = \".\"\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"Before save score (RMSE): {score}\")\n",
    "\n",
    "# save neural network structure to JSON (no weights)\n",
    "model_json = model.to_json()\n",
    "with open(os.path.join(save_path,\"network.json\"), \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "\n",
    "# save neural network structure to YAML (no weights)\n",
    "model_yaml = model.to_yaml()\n",
    "with open(os.path.join(save_path,\"network.yaml\"), \"w\") as yaml_file:\n",
    "    yaml_file.write(model_yaml)\n",
    "\n",
    "# save entire network to HDF5 (save everything, suggested)\n",
    "model.save(os.path.join(save_path,\"network.h5\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below sets up a neural network and reads the data (for predictions), but it does not clear the model directory or fit the neural network.  The weights from the previous fit are used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we reload the network and perform another prediction.  The RMSE should match the previous one exactly if the neural network was really saved and reloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After load score (RMSE): 6.090650375914274\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import load_model\n",
    "model2 = load_model(os.path.join(save_path,\"network.h5\"))\n",
    "pred = model2.predict(x)\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"After load score (RMSE): {score}\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}