Lazy LazyPredict.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lazy LazyPredict: An Exercise in Automated Python Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[LazyPredict](https://pypi.org/project/lazypredict/) is an excellent python library that allows the user to automatically run their data through a myriad of models. It will then output information regarding how each model performed, allowing the user to view which model might be best for their use case.\n",
    "\n",
    "This is a great tool to save time that might otherwise be spent manually testing different models. But while exploring this tool, I began to wonder what other manual steps I could accomplish purely with python libraries. And with that thought, I dove into a test of lazier than lazypredict predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Folder Structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Utilized data can be found in the /data folder, retrieved from [Kaggle.com](https://www.kaggle.com/c/titanic/overview)\n",
    "- All images can be found in the /images folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, I install the requisite libraries to do some EDA (exploratory data analysis) and feature engineering for me."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment and run the cells below for libraries that you do not currently have.\n",
    "\n",
    "Note: Some users, particularly Windows, may run into problems with installation. If so, running a command prompt in administrator mode and running the respective code may solve it. Otherwise, please refer to the library's documentation:\n",
    "- [Lazy Predict](https://pypi.org/project/lazypredict/)\n",
    "- [Feature Engine](https://pypi.org/project/feature-engine/)\n",
    "- [Feature Tools](https://pypi.org/project/featuretools/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:13.134983Z",
     "start_time": "2021-06-14T12:17:13.131992Z"
    }
   },
   "outputs": [],
   "source": [
    "#pip install lazypredict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:13.138973Z",
     "start_time": "2021-06-14T12:17:13.135980Z"
    }
   },
   "outputs": [],
   "source": [
    "#pip install feature_engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:13.142961Z",
     "start_time": "2021-06-14T12:17:13.140968Z"
    }
   },
   "outputs": [],
   "source": [
    "#pip install featuretools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.796541Z",
     "start_time": "2021-06-14T12:17:13.143960Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from feature_engine.imputation import MeanMedianImputer\n",
    "from feature_engine.outliers import Winsorizer\n",
    "\n",
    "import featuretools as ft\n",
    "import featuretools.variable_types as vtypes\n",
    "\n",
    "from lazypredict.Supervised import LazyClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.817486Z",
     "start_time": "2021-06-14T12:17:14.797539Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.28</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.92</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.10</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.05</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male 22.00      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female 38.00      1   \n",
       "2                             Heikkinen, Miss. Laina  female 26.00      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female 35.00      1   \n",
       "4                           Allen, Mr. William Henry    male 35.00      0   \n",
       "\n",
       "   Parch            Ticket  Fare Cabin Embarked  \n",
       "0      0         A/5 21171  7.25   NaN        S  \n",
       "1      0          PC 17599 71.28   C85        C  \n",
       "2      0  STON/O2. 3101282  7.92   NaN        S  \n",
       "3      0            113803 53.10  C123        S  \n",
       "4      0            373450  8.05   NaN        S  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Read in and view the data\n",
    "df = pd.read_csv('data/train.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.825465Z",
     "start_time": "2021-06-14T12:17:14.818482Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(891,\n",
       " PassengerId      0\n",
       " Survived         0\n",
       " Pclass           0\n",
       " Name             0\n",
       " Sex              0\n",
       " Age            177\n",
       " SibSp            0\n",
       " Parch            0\n",
       " Ticket           0\n",
       " Fare             0\n",
       " Cabin          687\n",
       " Embarked         2\n",
       " dtype: int64)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Check for missing values\n",
    "len(df), df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like there are some missing values. Cabin may have too many missing values to impute (687 out of 891), but Age may be something I can work with. Here's where Feature Engine comes in. I'll be using the MeanMedianImputer to impute the mean for each missing value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.837433Z",
     "start_time": "2021-06-14T12:17:14.829454Z"
    }
   },
   "outputs": [],
   "source": [
    "#Set the MeanMedianImputer to utilize the mean\n",
    "mean_imputer = MeanMedianImputer(imputation_method = 'mean',\n",
    "                                 variables = ['Age'])\n",
    "\n",
    "#Fit to the data\n",
    "mean_imputer.fit(df)\n",
    "#Impute the mean\n",
    "df = mean_imputer.transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.845412Z",
     "start_time": "2021-06-14T12:17:14.838430Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      0\n",
       "Survived         0\n",
       "Pclass           0\n",
       "Name             0\n",
       "Sex              0\n",
       "Age              0\n",
       "SibSp            0\n",
       "Parch            0\n",
       "Ticket           0\n",
       "Fare             0\n",
       "Cabin          687\n",
       "Embarked         2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good! The NaN values in Age have been changed to the average age of riders.\n",
    "\n",
    "Next, I'll drop the Cabin columns before dropping all NaN values to remove the two in Embarked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.856382Z",
     "start_time": "2021-06-14T12:17:14.846409Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId    0\n",
       "Survived       0\n",
       "Pclass         0\n",
       "Name           0\n",
       "Sex            0\n",
       "Age            0\n",
       "SibSp          0\n",
       "Parch          0\n",
       "Ticket         0\n",
       "Fare           0\n",
       "Embarked       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop(columns = 'Cabin', inplace = True)\n",
    "df.dropna(inplace = True)\n",
    "#Double check for NaN values after dropping\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with Extreme Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extreme values can skew a model's predictions, so I'll take a look at the data and see if I can detect any."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.877326Z",
     "start_time": "2021-06-14T12:17:14.857380Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.00</td>\n",
       "      <td>0.38</td>\n",
       "      <td>2.31</td>\n",
       "      <td>29.65</td>\n",
       "      <td>0.52</td>\n",
       "      <td>0.38</td>\n",
       "      <td>32.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.00</td>\n",
       "      <td>0.49</td>\n",
       "      <td>0.83</td>\n",
       "      <td>12.97</td>\n",
       "      <td>1.10</td>\n",
       "      <td>0.81</td>\n",
       "      <td>49.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>224.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>22.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>29.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>14.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>35.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>31.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>80.00</td>\n",
       "      <td>8.00</td>\n",
       "      <td>6.00</td>\n",
       "      <td>512.33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId  Survived  Pclass    Age  SibSp  Parch   Fare\n",
       "count       889.00    889.00  889.00 889.00 889.00 889.00 889.00\n",
       "mean        446.00      0.38    2.31  29.65   0.52   0.38  32.10\n",
       "std         257.00      0.49    0.83  12.97   1.10   0.81  49.70\n",
       "min           1.00      0.00    1.00   0.42   0.00   0.00   0.00\n",
       "25%         224.00      0.00    2.00  22.00   0.00   0.00   7.90\n",
       "50%         446.00      0.00    3.00  29.70   0.00   0.00  14.45\n",
       "75%         668.00      1.00    3.00  35.00   1.00   0.00  31.00\n",
       "max         891.00      1.00    3.00  80.00   8.00   6.00 512.33"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the Age, SibSp, Parch, and Fare have some particularly high values.\n",
    "\n",
    "For example, the age variable. The mean is 29.70 and the 75th percentile is 35. However, the max is 80. The other variables contain similar gaps between the max value and the rest. In particular, Fare, where the max is 512, despite the mean being 32.20.\n",
    "\n",
    "So, time to do a little value capping with Feature Engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.907247Z",
     "start_time": "2021-06-14T12:17:14.879321Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "      <td>889.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.00</td>\n",
       "      <td>0.38</td>\n",
       "      <td>2.31</td>\n",
       "      <td>29.62</td>\n",
       "      <td>0.48</td>\n",
       "      <td>0.36</td>\n",
       "      <td>29.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.00</td>\n",
       "      <td>0.49</td>\n",
       "      <td>0.83</td>\n",
       "      <td>12.87</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.70</td>\n",
       "      <td>36.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.42</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>224.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>22.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>29.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>14.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>35.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>31.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>68.56</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.80</td>\n",
       "      <td>181.19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId  Survived  Pclass    Age  SibSp  Parch   Fare\n",
       "count       889.00    889.00  889.00 889.00 889.00 889.00 889.00\n",
       "mean        446.00      0.38    2.31  29.62   0.48   0.36  29.89\n",
       "std         257.00      0.49    0.83  12.87   0.89   0.70  36.95\n",
       "min           1.00      0.00    1.00   0.42   0.00   0.00   0.00\n",
       "25%         224.00      0.00    2.00  22.00   0.00   0.00   7.90\n",
       "50%         446.00      0.00    3.00  29.70   0.00   0.00  14.45\n",
       "75%         668.00      1.00    3.00  35.00   1.00   0.00  31.00\n",
       "max         891.00      1.00    3.00  68.56   3.84   2.80 181.19"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "capper = Winsorizer(capping_method = 'gaussian',\n",
    "                    #Which tail to cap values, here I choose the right tail to cap high values\n",
    "                    tail = 'right',\n",
    "                    #Fold indicates the number of standard deviations\n",
    "                    fold = 3,\n",
    "                    variables = ['Age', 'SibSp', 'Parch', 'Fare'])\n",
    "\n",
    "#Fit the capper to the data\n",
    "capper.fit(df)\n",
    "#Cap extreme values in the data\n",
    "df = capper.transform(df)\n",
    "#Call .describe() on the transformed dataframe\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-13T22:54:08.827376Z",
     "start_time": "2021-06-13T22:54:08.805433Z"
    }
   },
   "source": [
    "Looks like the extreme values have successfully been capped. This still retain the values, to aid in predictions. However, capping them will prevent fluctuations in predictions for average values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having capped extreme values, I'll now take a look at automated feature engineering with Feature Tools.\n",
    "\n",
    "First, a look at the data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.913230Z",
     "start_time": "2021-06-14T12:17:14.908244Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      int64\n",
       "Survived         int64\n",
       "Pclass           int64\n",
       "Name            object\n",
       "Sex             object\n",
       "Age            float64\n",
       "SibSp          float64\n",
       "Parch          float64\n",
       "Ticket          object\n",
       "Fare           float64\n",
       "Embarked        object\n",
       "dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll drop a few columns here: PassengerId, Name, Sex, Ticket, and Embarked.\n",
    "\n",
    "However, for Sex and Embarked, I'll dummy the columns first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.922206Z",
     "start_time": "2021-06-14T12:17:14.914228Z"
    }
   },
   "outputs": [],
   "source": [
    "dummy_sex = pd.get_dummies(df['Sex'], drop_first = True)\n",
    "dummy_embark = pd.get_dummies(df['Embarked'], drop_first = True)\n",
    "\n",
    "df = pd.concat([df, dummy_sex, dummy_embark], axis = 1)\n",
    "\n",
    "df.drop(columns = ['PassengerId', 'Name', 'Sex', 'Ticket', 'Embarked'], inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost ready! Now I'll need a named column to function as the index for Feature Tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.927193Z",
     "start_time": "2021-06-14T12:17:14.923204Z"
    }
   },
   "outputs": [],
   "source": [
    "df.reset_index(inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step will be to tell Feature Tools what each column datatype is. This will instruct it on how to create features from them. I'll also drop the Survived column and place it into a separate variable, as that is the value I am trying to predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.933177Z",
     "start_time": "2021-06-14T12:17:14.928189Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "index         int64\n",
       "Survived      int64\n",
       "Pclass        int64\n",
       "Age         float64\n",
       "SibSp       float64\n",
       "Parch       float64\n",
       "Fare        float64\n",
       "male          uint8\n",
       "Q             uint8\n",
       "S             uint8\n",
       "dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.940158Z",
     "start_time": "2021-06-14T12:17:14.934174Z"
    }
   },
   "outputs": [],
   "source": [
    "variable_types = {'Pclass': vtypes.Categorical,\n",
    "                  'Age': vtypes.Ordinal,\n",
    "                  'SibSp': vtypes.Numeric,\n",
    "                  'Parch': vtypes.Numeric,\n",
    "                  'Fare': vtypes.Numeric,\n",
    "                  'male': vtypes.Boolean,\n",
    "                  'Q': vtypes.Boolean,\n",
    "                  'S': vtypes.Boolean}\n",
    "\n",
    "y = df['Survived']\n",
    "df.drop(columns = 'Survived', inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I'll make an entity called survived. Feature Tools will utilize this to engineer new features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.955118Z",
     "start_time": "2021-06-14T12:17:14.945146Z"
    }
   },
   "outputs": [],
   "source": [
    "es = ft.EntitySet('survived')\n",
    "es = es.entity_from_dataframe(entity_id = 'survived',\n",
    "                              dataframe = df,\n",
    "                              index = 'index',\n",
    "                              #Indicate what the variable types are here\n",
    "                              variable_types = variable_types)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! The base entity has been set. Next, I'll add some relationships that I want it to explore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:14.976081Z",
     "start_time": "2021-06-14T12:17:14.958112Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entityset: survived\n",
       "  Entities:\n",
       "    survived [Rows: 889, Columns: 7]\n",
       "    male [Rows: 85, Columns: 1]\n",
       "    Pclass [Rows: 240, Columns: 3]\n",
       "    SibSp [Rows: 4, Columns: 1]\n",
       "  Relationships:\n",
       "    survived.Age -> male.Age\n",
       "    survived.Fare -> Pclass.Fare\n",
       "    survived.Parch -> SibSp.Parch"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#The first variable indicates which entityset to utilize\n",
    "#Then indicate which variables to create relationships between\n",
    "\n",
    "es.normalize_entity('survived', 'male', 'Age',\n",
    "                    make_time_index = False)\n",
    "es.normalize_entity('survived', 'Pclass', 'Fare',\n",
    "                    additional_variables = ['Q', 'S'],\n",
    "                    make_time_index = False)\n",
    "es.normalize_entity('survived', 'SibSp', 'Parch',\n",
    "                    make_time_index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having now set the relationships, I can run Feature Tool's Deep Feature Synthesis to engineer new features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:15.421870Z",
     "start_time": "2021-06-14T12:17:14.977059Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 50 features\n",
      "Elapsed: 00:00 | Progress: 100%|███████████████████████████████████████████████████████████████████████████████████████\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "      <th>male</th>\n",
       "      <th>Pclass.Q</th>\n",
       "      <th>Pclass.S</th>\n",
       "      <th>male.COUNT(survived)</th>\n",
       "      <th>male.MAX(survived.SibSp)</th>\n",
       "      <th>...</th>\n",
       "      <th>SibSp.MODE(survived.Age)</th>\n",
       "      <th>SibSp.MODE(survived.Fare)</th>\n",
       "      <th>SibSp.MODE(survived.Pclass)</th>\n",
       "      <th>SibSp.NUM_UNIQUE(survived.Age)</th>\n",
       "      <th>SibSp.NUM_UNIQUE(survived.Fare)</th>\n",
       "      <th>SibSp.NUM_UNIQUE(survived.Pclass)</th>\n",
       "      <th>SibSp.PERCENT_TRUE(survived.male)</th>\n",
       "      <th>SibSp.SKEW(survived.SibSp)</th>\n",
       "      <th>SibSp.STD(survived.SibSp)</th>\n",
       "      <th>SibSp.SUM(survived.SibSp)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>22.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.25</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>27</td>\n",
       "      <td>1.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>38.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>71.28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>1.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.92</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>2.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>35.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>53.10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>1.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>35.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.05</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>1.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>2</td>\n",
       "      <td>27.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>13.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>1.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>1</td>\n",
       "      <td>19.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>30.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>3.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>3</td>\n",
       "      <td>29.70</td>\n",
       "      <td>1.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>23.45</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>177</td>\n",
       "      <td>3.84</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>69.55</td>\n",
       "      <td>3</td>\n",
       "      <td>35.00</td>\n",
       "      <td>39.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.39</td>\n",
       "      <td>0.41</td>\n",
       "      <td>1.59</td>\n",
       "      <td>128.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>1</td>\n",
       "      <td>26.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>30.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>2.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>3</td>\n",
       "      <td>32.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.75</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>2.00</td>\n",
       "      <td>...</td>\n",
       "      <td>29.70</td>\n",
       "      <td>8.05</td>\n",
       "      <td>3</td>\n",
       "      <td>70.00</td>\n",
       "      <td>186.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.72</td>\n",
       "      <td>2.14</td>\n",
       "      <td>0.50</td>\n",
       "      <td>161.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>889 rows × 50 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Pclass   Age  SibSp  Parch  Fare  male  Pclass.Q  Pclass.S  \\\n",
       "index                                                               \n",
       "0           3 22.00   1.00   0.00  7.25     1         0         1   \n",
       "1           1 38.00   1.00   0.00 71.28     0         0         0   \n",
       "2           3 26.00   0.00   0.00  7.92     0         0         1   \n",
       "3           1 35.00   1.00   0.00 53.10     0         0         1   \n",
       "4           3 35.00   0.00   0.00  8.05     1         0         1   \n",
       "...       ...   ...    ...    ...   ...   ...       ...       ...   \n",
       "886         2 27.00   0.00   0.00 13.00     1         0         1   \n",
       "887         1 19.00   0.00   0.00 30.00     0         0         1   \n",
       "888         3 29.70   1.00   2.00 23.45     0         0         1   \n",
       "889         1 26.00   0.00   0.00 30.00     1         0         1   \n",
       "890         3 32.00   0.00   0.00  7.75     1         1         0   \n",
       "\n",
       "       male.COUNT(survived)  male.MAX(survived.SibSp)  ...  \\\n",
       "index                                                  ...   \n",
       "0                        27                      1.00  ...   \n",
       "1                        10                      1.00  ...   \n",
       "2                        18                      2.00  ...   \n",
       "3                        18                      1.00  ...   \n",
       "4                        18                      1.00  ...   \n",
       "...                     ...                       ...  ...   \n",
       "886                      18                      1.00  ...   \n",
       "887                      25                      3.00  ...   \n",
       "888                     177                      3.84  ...   \n",
       "889                      18                      2.00  ...   \n",
       "890                      18                      2.00  ...   \n",
       "\n",
       "       SibSp.MODE(survived.Age)  SibSp.MODE(survived.Fare)  \\\n",
       "index                                                        \n",
       "0                         29.70                       8.05   \n",
       "1                         29.70                       8.05   \n",
       "2                         29.70                       8.05   \n",
       "3                         29.70                       8.05   \n",
       "4                         29.70                       8.05   \n",
       "...                         ...                        ...   \n",
       "886                       29.70                       8.05   \n",
       "887                       29.70                       8.05   \n",
       "888                       29.70                      69.55   \n",
       "889                       29.70                       8.05   \n",
       "890                       29.70                       8.05   \n",
       "\n",
       "       SibSp.MODE(survived.Pclass)  SibSp.NUM_UNIQUE(survived.Age)  \\\n",
       "index                                                                \n",
       "0                                3                           70.00   \n",
       "1                                3                           70.00   \n",
       "2                                3                           70.00   \n",
       "3                                3                           70.00   \n",
       "4                                3                           70.00   \n",
       "...                            ...                             ...   \n",
       "886                              3                           70.00   \n",
       "887                              3                           70.00   \n",
       "888                              3                           35.00   \n",
       "889                              3                           70.00   \n",
       "890                              3                           70.00   \n",
       "\n",
       "       SibSp.NUM_UNIQUE(survived.Fare)  SibSp.NUM_UNIQUE(survived.Pclass)  \\\n",
       "index                                                                       \n",
       "0                               186.00                                  3   \n",
       "1                               186.00                                  3   \n",
       "2                               186.00                                  3   \n",
       "3                               186.00                                  3   \n",
       "4                               186.00                                  3   \n",
       "...                                ...                                ...   \n",
       "886                             186.00                                  3   \n",
       "887                             186.00                                  3   \n",
       "888                              39.00                                  3   \n",
       "889                             186.00                                  3   \n",
       "890                             186.00                                  3   \n",
       "\n",
       "       SibSp.PERCENT_TRUE(survived.male)  SibSp.SKEW(survived.SibSp)  \\\n",
       "index                                                                  \n",
       "0                                   0.72                        2.14   \n",
       "1                                   0.72                        2.14   \n",
       "2                                   0.72                        2.14   \n",
       "3                                   0.72                        2.14   \n",
       "4                                   0.72                        2.14   \n",
       "...                                  ...                         ...   \n",
       "886                                 0.72                        2.14   \n",
       "887                                 0.72                        2.14   \n",
       "888                                 0.39                        0.41   \n",
       "889                                 0.72                        2.14   \n",
       "890                                 0.72                        2.14   \n",
       "\n",
       "       SibSp.STD(survived.SibSp)  SibSp.SUM(survived.SibSp)  \n",
       "index                                                        \n",
       "0                           0.50                     161.00  \n",
       "1                           0.50                     161.00  \n",
       "2                           0.50                     161.00  \n",
       "3                           0.50                     161.00  \n",
       "4                           0.50                     161.00  \n",
       "...                          ...                        ...  \n",
       "886                         0.50                     161.00  \n",
       "887                         0.50                     161.00  \n",
       "888                         1.59                     128.54  \n",
       "889                         0.50                     161.00  \n",
       "890                         0.50                     161.00  \n",
       "\n",
       "[889 rows x 50 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fm, features = ft.dfs(entityset = es,\n",
    "                      target_entity = 'survived',\n",
    "                      verbose = True)\n",
    "fm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like everything worked! There are now 50 features as opposed to the original 6.\n",
    "\n",
    "Having now undergone data cleaning and feature engineering, I'll move onto LazyPredict."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing Models with Lazy Predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, the train/test split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:15.427854Z",
     "start_time": "2021-06-14T12:17:15.422867Z"
    }
   },
   "outputs": [],
   "source": [
    "#We moved the target variable into y earlier\n",
    "y = y\n",
    "#The dataset we want to use will be fm, set to X for clarity\n",
    "X = fm\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,\n",
    "                                                    y,\n",
    "                                                    random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next I'll instantiate the LazyClassifier and run it using the train/test split.\n",
    "- Note: LazyPredict will preprocess data for models, such as StandardScaler for numeric values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:16.796197Z",
     "start_time": "2021-06-14T12:17:15.428851Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 29/29 [00:01<00:00, 21.41it/s]\n"
     ]
    }
   ],
   "source": [
    "clf = LazyClassifier()\n",
    "models, predictions = clf.fit(X_train,\n",
    "                              X_test,\n",
    "                              y_train,\n",
    "                              y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-14T12:17:16.813152Z",
     "start_time": "2021-06-14T12:17:16.797194Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Balanced Accuracy</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>F1 Score</th>\n",
       "      <th>Time Taken</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AdaBoostClassifier</th>\n",
       "      <td>0.81</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LinearDiscriminantAnalysis</th>\n",
       "      <td>0.81</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CalibratedClassifierCV</th>\n",
       "      <td>0.81</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RandomForestClassifier</th>\n",
       "      <td>0.81</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RidgeClassifier</th>\n",
       "      <td>0.81</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NuSVC</th>\n",
       "      <td>0.81</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BaggingClassifier</th>\n",
       "      <td>0.81</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LinearSVC</th>\n",
       "      <td>0.80</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ExtraTreesClassifier</th>\n",
       "      <td>0.80</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LogisticRegression</th>\n",
       "      <td>0.80</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SVC</th>\n",
       "      <td>0.81</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RidgeClassifierCV</th>\n",
       "      <td>0.80</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>XGBClassifier</th>\n",
       "      <td>0.79</td>\n",
       "      <td>0.77</td>\n",
       "      <td>0.77</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LGBMClassifier</th>\n",
       "      <td>0.78</td>\n",
       "      <td>0.77</td>\n",
       "      <td>0.77</td>\n",
       "      <td>0.78</td>\n",
       "      <td>0.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KNeighborsClassifier</th>\n",
       "      <td>0.76</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.76</td>\n",
       "      <td>0.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassiveAggressiveClassifier</th>\n",
       "      <td>0.78</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.77</td>\n",
       "      <td>0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ExtraTreeClassifier</th>\n",
       "      <td>0.73</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LabelPropagation</th>\n",
       "      <td>0.74</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.74</td>\n",
       "      <td>0.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DecisionTreeClassifier</th>\n",
       "      <td>0.73</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SGDClassifier</th>\n",
       "      <td>0.73</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LabelSpreading</th>\n",
       "      <td>0.73</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Perceptron</th>\n",
       "      <td>0.72</td>\n",
       "      <td>0.71</td>\n",
       "      <td>0.71</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BernoulliNB</th>\n",
       "      <td>0.73</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NearestCentroid</th>\n",
       "      <td>0.72</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GaussianNB</th>\n",
       "      <td>0.74</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.73</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QuadraticDiscriminantAnalysis</th>\n",
       "      <td>0.65</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.66</td>\n",
       "      <td>0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DummyClassifier</th>\n",
       "      <td>0.57</td>\n",
       "      <td>0.54</td>\n",
       "      <td>0.54</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               Accuracy  Balanced Accuracy  ROC AUC  F1 Score  \\\n",
       "Model                                                                           \n",
       "AdaBoostClassifier                 0.81               0.81     0.81      0.81   \n",
       "LinearDiscriminantAnalysis         0.81               0.80     0.80      0.81   \n",
       "CalibratedClassifierCV             0.81               0.80     0.80      0.81   \n",
       "RandomForestClassifier             0.81               0.79     0.79      0.81   \n",
       "RidgeClassifier                    0.81               0.79     0.79      0.81   \n",
       "NuSVC                              0.81               0.79     0.79      0.81   \n",
       "BaggingClassifier                  0.81               0.79     0.79      0.81   \n",
       "LinearSVC                          0.80               0.79     0.79      0.80   \n",
       "ExtraTreesClassifier               0.80               0.79     0.79      0.80   \n",
       "LogisticRegression                 0.80               0.79     0.79      0.80   \n",
       "SVC                                0.81               0.79     0.79      0.81   \n",
       "RidgeClassifierCV                  0.80               0.79     0.79      0.80   \n",
       "XGBClassifier                      0.79               0.77     0.77      0.79   \n",
       "LGBMClassifier                     0.78               0.77     0.77      0.78   \n",
       "KNeighborsClassifier               0.76               0.75     0.75      0.76   \n",
       "PassiveAggressiveClassifier        0.78               0.73     0.73      0.77   \n",
       "ExtraTreeClassifier                0.73               0.72     0.72      0.73   \n",
       "LabelPropagation                   0.74               0.72     0.72      0.74   \n",
       "DecisionTreeClassifier             0.73               0.72     0.72      0.73   \n",
       "SGDClassifier                      0.73               0.72     0.72      0.73   \n",
       "LabelSpreading                     0.73               0.72     0.72      0.73   \n",
       "Perceptron                         0.72               0.71     0.71      0.72   \n",
       "BernoulliNB                        0.73               0.70     0.70      0.73   \n",
       "NearestCentroid                    0.72               0.70     0.70      0.72   \n",
       "GaussianNB                         0.74               0.70     0.70      0.73   \n",
       "QuadraticDiscriminantAnalysis      0.65               0.68     0.68      0.66   \n",
       "DummyClassifier                    0.57               0.54     0.54      0.57   \n",
       "\n",
       "                               Time Taken  \n",
       "Model                                      \n",
       "AdaBoostClassifier                   0.09  \n",
       "LinearDiscriminantAnalysis           0.02  \n",
       "CalibratedClassifierCV               0.24  \n",
       "RandomForestClassifier               0.17  \n",
       "RidgeClassifier                      0.01  \n",
       "NuSVC                                0.04  \n",
       "BaggingClassifier                    0.05  \n",
       "LinearSVC                            0.07  \n",
       "ExtraTreesClassifier                 0.13  \n",
       "LogisticRegression                   0.03  \n",
       "SVC                                  0.03  \n",
       "RidgeClassifierCV                    0.02  \n",
       "XGBClassifier                        0.10  \n",
       "LGBMClassifier                       0.10  \n",
       "KNeighborsClassifier                 0.03  \n",
       "PassiveAggressiveClassifier          0.02  \n",
       "ExtraTreeClassifier                  0.01  \n",
       "LabelPropagation                     0.04  \n",
       "DecisionTreeClassifier               0.01  \n",
       "SGDClassifier                        0.01  \n",
       "LabelSpreading                       0.04  \n",
       "Perceptron                           0.01  \n",
       "BernoulliNB                          0.01  \n",
       "NearestCentroid                      0.01  \n",
       "GaussianNB                           0.01  \n",
       "QuadraticDiscriminantAnalysis        0.02  \n",
       "DummyClassifier                      0.01  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like Ada Boost Classifier will be the way to go!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilizing Feature Tools, Feature Engine, and Lazy Predict, I was able to narrow down to utilizing Ada Boost Classifer for my model as well as being able to utilize a slew of engineered features.\n",
    "\n",
    "There were a few items I would have liked to look further into, but I felt they broke the spirit of using the three libraries for nearly 100% of the EDA/feature engineering process. Overall, I found this a great exercise in utilizing the three libraries and feel that, with a little more tweaking and practice, would be great additions to my usual data analysis process."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}