t81_558_class_14_03_anomaly.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 14 Video Material\n",
    "\n",
    "* Part 14.1: What is AutoML [[Video]](https://www.youtube.com/watch?v=TFUysIR5AB0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_01_automl.ipynb)\n",
    "* Part 14.2: Using Denoising AutoEncoders in Keras [[Video]](https://www.youtube.com/watch?v=4bTSu6_fucc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_02_auto_encode.ipynb)\n",
    "* **Part 14.3: Training an Intrusion Detection System with KDD99** [[Video]](https://www.youtube.com/watch?v=1ySn6h2A68I&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_03_anomaly.ipynb)\n",
    "* Part 14.4: Anomaly Detection in Keras [[Video]](https://www.youtube.com/watch?v=VgyKQ5MTDFc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_04_ids_kdd99.ipynb)\n",
    "* Part 14.5: The Deep Learning Technologies I am Excited About [[Video]]() [[Notebook]](t81_558_class_14_05_new_tech.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 14.3: Anomaly Detection in Keras\n",
    "\n",
    "Datasets used for anomaly detection:\n",
    "\n",
    "* [Stratosphere IPS Dataset](https://www.stratosphereips.org/category/dataset.html)\n",
    "* [The ADFA Intrusion Detection Datasets (2013) - for HIDS](https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFA-IDS-Datasets/)\n",
    "* [ITOC CDX (2009)](https://westpoint.edu/centers-and-research/cyber-research-center/data-sets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in KDD99 Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/jheaton/.keras/datasets/kddcup.data_10_percent.gz\n",
      "Read 494021 rows.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>181</td>\n",
       "      <td>5450</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>239</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>19</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>235</td>\n",
       "      <td>1337</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>219</td>\n",
       "      <td>1337</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>39</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>217</td>\n",
       "      <td>2032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>49</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type service flag  src_bytes  dst_bytes  land  \\\n",
       "0         0           tcp    http   SF        181       5450     0   \n",
       "1         0           tcp    http   SF        239        486     0   \n",
       "2         0           tcp    http   SF        235       1337     0   \n",
       "3         0           tcp    http   SF        219       1337     0   \n",
       "4         0           tcp    http   SF        217       2032     0   \n",
       "\n",
       "   wrong_fragment  urgent  hot  ...  dst_host_srv_count  \\\n",
       "0               0       0    0  ...                   9   \n",
       "1               0       0    0  ...                  19   \n",
       "2               0       0    0  ...                  29   \n",
       "3               0       0    0  ...                  39   \n",
       "4               0       0    0  ...                  49   \n",
       "\n",
       "   dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
       "0                     1.0                     0.0   \n",
       "1                     1.0                     0.0   \n",
       "2                     1.0                     0.0   \n",
       "3                     1.0                     0.0   \n",
       "4                     1.0                     0.0   \n",
       "\n",
       "   dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
       "0                         0.11                          0.0   \n",
       "1                         0.05                          0.0   \n",
       "2                         0.03                          0.0   \n",
       "3                         0.03                          0.0   \n",
       "4                         0.02                          0.0   \n",
       "\n",
       "   dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
       "0                   0.0                       0.0                   0.0   \n",
       "1                   0.0                       0.0                   0.0   \n",
       "2                   0.0                       0.0                   0.0   \n",
       "3                   0.0                       0.0                   0.0   \n",
       "4                   0.0                       0.0                   0.0   \n",
       "\n",
       "   dst_host_srv_rerror_rate  outcome  \n",
       "0                       0.0  normal.  \n",
       "1                       0.0  normal.  \n",
       "2                       0.0  normal.  \n",
       "3                       0.0  normal.  \n",
       "4                       0.0  normal.  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from tensorflow.keras.utils import get_file\n",
    "\n",
    "try:\n",
    "    path = get_file('kddcup.data_10_percent.gz', origin='http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path) \n",
    "\n",
    "# This file is a CSV, just no CSV extension or headers\n",
    "# Download from: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html\n",
    "df = pd.read_csv(path, header=None)\n",
    "\n",
    "print(\"Read {} rows.\".format(len(df)))\n",
    "# df = df.sample(frac=0.1, replace=False) # Uncomment this line to sample only 10% of the dataset\n",
    "df.dropna(inplace=True,axis=1) # For now, just drop NA's (rows with missing values)\n",
    "\n",
    "# The CSV file has no column heads, so add them\n",
    "df.columns = [\n",
    "    'duration',\n",
    "    'protocol_type',\n",
    "    'service',\n",
    "    'flag',\n",
    "    'src_bytes',\n",
    "    'dst_bytes',\n",
    "    'land',\n",
    "    'wrong_fragment',\n",
    "    'urgent',\n",
    "    'hot',\n",
    "    'num_failed_logins',\n",
    "    'logged_in',\n",
    "    'num_compromised',\n",
    "    'root_shell',\n",
    "    'su_attempted',\n",
    "    'num_root',\n",
    "    'num_file_creations',\n",
    "    'num_shells',\n",
    "    'num_access_files',\n",
    "    'num_outbound_cmds',\n",
    "    'is_host_login',\n",
    "    'is_guest_login',\n",
    "    'count',\n",
    "    'srv_count',\n",
    "    'serror_rate',\n",
    "    'srv_serror_rate',\n",
    "    'rerror_rate',\n",
    "    'srv_rerror_rate',\n",
    "    'same_srv_rate',\n",
    "    'diff_srv_rate',\n",
    "    'srv_diff_host_rate',\n",
    "    'dst_host_count',\n",
    "    'dst_host_srv_count',\n",
    "    'dst_host_same_srv_rate',\n",
    "    'dst_host_diff_srv_rate',\n",
    "    'dst_host_same_src_port_rate',\n",
    "    'dst_host_srv_diff_host_rate',\n",
    "    'dst_host_serror_rate',\n",
    "    'dst_host_srv_serror_rate',\n",
    "    'dst_host_rerror_rate',\n",
    "    'dst_host_srv_rerror_rate',\n",
    "    'outcome'\n",
    "]\n",
    "\n",
    "# display 5 rows\n",
    "df[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "outcome\n",
       "back.                 2203\n",
       "buffer_overflow.        30\n",
       "ftp_write.               8\n",
       "guess_passwd.           53\n",
       "imap.                   12\n",
       "ipsweep.              1247\n",
       "land.                   21\n",
       "loadmodule.              9\n",
       "multihop.                7\n",
       "neptune.            107201\n",
       "nmap.                  231\n",
       "normal.              97278\n",
       "perl.                    3\n",
       "phf.                     4\n",
       "pod.                   264\n",
       "portsweep.            1040\n",
       "rootkit.                10\n",
       "satan.                1589\n",
       "smurf.              280790\n",
       "spy.                     2\n",
       "teardrop.              979\n",
       "warezclient.          1020\n",
       "warezmaster.            20\n",
       "Name: outcome, dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('outcome')['outcome'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "    \n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = f\"{name}-{x}\"\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>...</th>\n",
       "      <th>flag-S3</th>\n",
       "      <th>flag-SF</th>\n",
       "      <th>flag-SH</th>\n",
       "      <th>land-0</th>\n",
       "      <th>land-1</th>\n",
       "      <th>logged_in-0</th>\n",
       "      <th>logged_in-1</th>\n",
       "      <th>is_host_login-0</th>\n",
       "      <th>is_guest_login-0</th>\n",
       "      <th>is_guest_login-1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002879</td>\n",
       "      <td>0.138664</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002820</td>\n",
       "      <td>-0.011578</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002824</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002840</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002842</td>\n",
       "      <td>0.035214</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 121 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration  src_bytes  dst_bytes  wrong_fragment    urgent       hot  \\\n",
       "0 -0.067792  -0.002879   0.138664        -0.04772 -0.002571 -0.044136   \n",
       "1 -0.067792  -0.002820  -0.011578        -0.04772 -0.002571 -0.044136   \n",
       "2 -0.067792  -0.002824   0.014179        -0.04772 -0.002571 -0.044136   \n",
       "3 -0.067792  -0.002840   0.014179        -0.04772 -0.002571 -0.044136   \n",
       "4 -0.067792  -0.002842   0.035214        -0.04772 -0.002571 -0.044136   \n",
       "\n",
       "   num_failed_logins  num_compromised  root_shell  su_attempted  ...  flag-S3  \\\n",
       "0          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "1          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "2          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "3          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "4          -0.009782        -0.005679   -0.010552     -0.004676  ...        0   \n",
       "\n",
       "   flag-SF  flag-SH  land-0  land-1  logged_in-0  logged_in-1  \\\n",
       "0        1        0       1       0            0            1   \n",
       "1        1        0       1       0            0            1   \n",
       "2        1        0       1       0            0            1   \n",
       "3        1        0       1       0            0            1   \n",
       "4        1        0       1       0            0            1   \n",
       "\n",
       "   is_host_login-0  is_guest_login-0  is_guest_login-1  \n",
       "0                1                 1                 0  \n",
       "1                1                 1                 0  \n",
       "2                1                 1                 0  \n",
       "3                1                 1                 0  \n",
       "4                1                 1                 0  \n",
       "\n",
       "[5 rows x 121 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now encode the feature vector\n",
    "\n",
    "encode_numeric_zscore(df, 'duration')\n",
    "encode_text_dummy(df, 'protocol_type')\n",
    "encode_text_dummy(df, 'service')\n",
    "encode_text_dummy(df, 'flag')\n",
    "encode_numeric_zscore(df, 'src_bytes')\n",
    "encode_numeric_zscore(df, 'dst_bytes')\n",
    "encode_text_dummy(df, 'land')\n",
    "encode_numeric_zscore(df, 'wrong_fragment')\n",
    "encode_numeric_zscore(df, 'urgent')\n",
    "encode_numeric_zscore(df, 'hot')\n",
    "encode_numeric_zscore(df, 'num_failed_logins')\n",
    "encode_text_dummy(df, 'logged_in')\n",
    "encode_numeric_zscore(df, 'num_compromised')\n",
    "encode_numeric_zscore(df, 'root_shell')\n",
    "encode_numeric_zscore(df, 'su_attempted')\n",
    "encode_numeric_zscore(df, 'num_root')\n",
    "encode_numeric_zscore(df, 'num_file_creations')\n",
    "encode_numeric_zscore(df, 'num_shells')\n",
    "encode_numeric_zscore(df, 'num_access_files')\n",
    "encode_numeric_zscore(df, 'num_outbound_cmds')\n",
    "encode_text_dummy(df, 'is_host_login')\n",
    "encode_text_dummy(df, 'is_guest_login')\n",
    "encode_numeric_zscore(df, 'count')\n",
    "encode_numeric_zscore(df, 'srv_count')\n",
    "encode_numeric_zscore(df, 'serror_rate')\n",
    "encode_numeric_zscore(df, 'srv_serror_rate')\n",
    "encode_numeric_zscore(df, 'rerror_rate')\n",
    "encode_numeric_zscore(df, 'srv_rerror_rate')\n",
    "encode_numeric_zscore(df, 'same_srv_rate')\n",
    "encode_numeric_zscore(df, 'diff_srv_rate')\n",
    "encode_numeric_zscore(df, 'srv_diff_host_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_count')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_count')\n",
    "encode_numeric_zscore(df, 'dst_host_same_srv_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_diff_srv_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_same_src_port_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_diff_host_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_serror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_serror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_rerror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_rerror_rate')\n",
    "\n",
    "# display 5 rows\n",
    "\n",
    "df.dropna(inplace=True,axis=1)\n",
    "df[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal count: 97278\n",
      "Attack count: 396743\n"
     ]
    }
   ],
   "source": [
    "normal_mask = df['outcome']=='normal.'\n",
    "attack_mask = df['outcome']!='normal.'\n",
    "\n",
    "df.drop('outcome',axis=1,inplace=True)\n",
    "\n",
    "df_normal = df[normal_mask]\n",
    "df_attack = df[attack_mask]\n",
    "\n",
    "print(f\"Normal count: {len(df_normal)}\")\n",
    "print(f\"Attack count: {len(df_attack)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the numeric feature vector, as it goes to the neural net\n",
    "x_normal = df_normal.values\n",
    "x_attack = df_attack.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_normal_train, x_normal_test = train_test_split(\n",
    "    x_normal, test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normal train count: 72958\n",
      "Normal test count: 24320\n"
     ]
    }
   ],
   "source": [
    "print(f\"Normal train count: {len(x_normal_train)}\")\n",
    "print(f\"Normal test count: {len(x_normal_test)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Auto Encoders\n",
    "\n",
    "An auto encoder is a neural network that has the same number of input neurons as it does outputs.  The hidden layers of the neural network will have fewer neurons than the input/output neurons.  Because there are fewer neurons, the auto-encoder must learn to encode the input to the fewer hidden neurons.  The predictors (x) and output (y) are exactly the same in an auto encoder.  Because of this, auto encoders are said to be unsupervised.  \n",
    "\n",
    "![Simple Auto Encoder](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_13_auto_encode.png \"Simple Auto Encoder\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0810 23:28:26.176645 140735657337728 deprecation.py:323] From /Users/jheaton/miniconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:468: BaseResourceVariable.constraint (from tensorflow.python.ops.resource_variable_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Apply a constraint manually following the optimizer update step.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 72958 samples\n",
      "Epoch 1/100\n",
      "72958/72958 [==============================] - 3s 37us/sample - loss: 0.35610s - loss: - ETA: 0s - loss: \n",
      "Epoch 2/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.2610\n",
      "Epoch 3/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.21350s - loss: 0.\n",
      "Epoch 4/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1971\n",
      "Epoch 5/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1833\n",
      "Epoch 6/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1802\n",
      "Epoch 7/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1674\n",
      "Epoch 8/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1594\n",
      "Epoch 9/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1549\n",
      "Epoch 10/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1447\n",
      "Epoch 11/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1546\n",
      "Epoch 12/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1435\n",
      "Epoch 13/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.13600s - loss: 0\n",
      "Epoch 14/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1375\n",
      "Epoch 15/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1340\n",
      "Epoch 16/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1281\n",
      "Epoch 17/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1283\n",
      "Epoch 18/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1251\n",
      "Epoch 19/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.12280s - loss\n",
      "Epoch 20/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1255\n",
      "Epoch 21/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1303\n",
      "Epoch 22/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1193\n",
      "Epoch 23/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.12120s - lo\n",
      "Epoch 24/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1269\n",
      "Epoch 25/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1209\n",
      "Epoch 26/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.12110s\n",
      "Epoch 27/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1171\n",
      "Epoch 28/100\n",
      "72958/72958 [==============================] - 3s 35us/sample - loss: 0.1258\n",
      "Epoch 29/100\n",
      "72958/72958 [==============================] - 2s 33us/sample - loss: 0.1183\n",
      "Epoch 30/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1164\n",
      "Epoch 31/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.12100s - loss: 0\n",
      "Epoch 32/100\n",
      "72958/72958 [==============================] - 2s 32us/sample - loss: 0.1197\n",
      "Epoch 33/100\n",
      "72958/72958 [==============================] - 2s 32us/sample - loss: 0.1142\n",
      "Epoch 34/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1170\n",
      "Epoch 35/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1201\n",
      "Epoch 36/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.11550s - loss:  - ETA: 0s - loss: 0.1\n",
      "Epoch 37/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.11620s - \n",
      "Epoch 38/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1209\n",
      "Epoch 39/100\n",
      "72958/72958 [==============================] - 2s 33us/sample - loss: 0.1202\n",
      "Epoch 40/100\n",
      "72958/72958 [==============================] - 2s 33us/sample - loss: 0.1160\n",
      "Epoch 41/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1184\n",
      "Epoch 42/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1160\n",
      "Epoch 43/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1163\n",
      "Epoch 44/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1182\n",
      "Epoch 45/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1159\n",
      "Epoch 46/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1147\n",
      "Epoch 47/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1130\n",
      "Epoch 48/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1166\n",
      "Epoch 49/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1115\n",
      "Epoch 50/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1142\n",
      "Epoch 51/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.11291s - loss: 0.119 - ETA: 1s - l\n",
      "Epoch 52/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.11290s - loss: 0.\n",
      "Epoch 53/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.11210s -\n",
      "Epoch 54/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1117- ETA: 0s\n",
      "Epoch 55/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1124\n",
      "Epoch 56/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.11300s - loss:\n",
      "Epoch 57/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1098\n",
      "Epoch 58/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1128\n",
      "Epoch 59/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1088\n",
      "Epoch 60/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1129\n",
      "Epoch 61/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1104\n",
      "Epoch 62/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.10970s - loss: 0.11\n",
      "Epoch 63/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1112\n",
      "Epoch 64/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1118\n",
      "Epoch 65/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1103\n",
      "Epoch 66/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1065\n",
      "Epoch 67/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1123\n",
      "Epoch 68/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1069\n",
      "Epoch 69/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1069\n",
      "Epoch 70/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1064\n",
      "Epoch 71/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1100\n",
      "Epoch 72/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1061\n",
      "Epoch 73/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1066\n",
      "Epoch 74/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1077\n",
      "Epoch 75/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1076\n",
      "Epoch 76/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1059\n",
      "Epoch 77/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1055\n",
      "Epoch 78/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1081\n",
      "Epoch 79/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1054\n",
      "Epoch 80/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.10230s - loss: 0.102\n",
      "Epoch 81/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.10780s\n",
      "Epoch 82/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1048\n",
      "Epoch 83/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.10270s - los\n",
      "Epoch 84/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1050\n",
      "Epoch 85/100\n",
      "72958/72958 [==============================] - 2s 32us/sample - loss: 0.1039\n",
      "Epoch 86/100\n",
      "72958/72958 [==============================] - 3s 35us/sample - loss: 0.1019\n",
      "Epoch 87/100\n",
      "72958/72958 [==============================] - 2s 32us/sample - loss: 0.1031\n",
      "Epoch 88/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1070\n",
      "Epoch 89/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1008\n",
      "Epoch 90/100\n",
      "72958/72958 [==============================] - 2s 29us/sample - loss: 0.1040\n",
      "Epoch 91/100\n",
      "72958/72958 [==============================] - 3s 34us/sample - loss: 0.1020\n",
      "Epoch 92/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1029\n",
      "Epoch 93/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1047\n",
      "Epoch 94/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.10201s - loss: 0.21 - ETA: 0s - loss:\n",
      "Epoch 95/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.10391s - loss\n",
      "Epoch 96/100\n",
      "72958/72958 [==============================] - 2s 30us/sample - loss: 0.1030\n",
      "Epoch 97/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1004\n",
      "Epoch 98/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.0999\n",
      "Epoch 99/100\n",
      "72958/72958 [==============================] - 2s 31us/sample - loss: 0.1016\n",
      "Epoch 100/100\n",
      "72958/72958 [==============================] - 2s 32us/sample - loss: 0.09890s - \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x64648ae80>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML \n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x_normal.shape[1], activation='relu'))\n",
    "model.add(Dense(3, activation='relu'))\n",
    "model.add(Dense(25, activation='relu'))\n",
    "model.add(Dense(x_normal.shape[1])) # Multiple output neurons\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x_normal_train,x_normal_train,verbose=1,epochs=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Insample Normal Score (RMSE): 0.3032416019220358\n",
      "Out of Sample Normal Score (RMSE): 0.31275482923174686\n",
      "Attack Underway Score (RMSE): 0.5313390948621244\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x_normal_test)\n",
    "score1 = np.sqrt(metrics.mean_squared_error(pred,x_normal_test))\n",
    "pred = model.predict(x_normal)\n",
    "score2 = np.sqrt(metrics.mean_squared_error(pred,x_normal))\n",
    "pred = model.predict(x_attack)\n",
    "score3 = np.sqrt(metrics.mean_squared_error(pred,x_attack))\n",
    "print(f\"Insample Normal Score (RMSE): {score1}\".format(score1))\n",
    "print(f\"Out of Sample Normal Score (RMSE): {score2}\")\n",
    "print(f\"Attack Underway Score (RMSE): {score3}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}