pytorch/surviving-the-titanic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "0c644fd9b8bbcba6d8de36fdcde4e352a85c04e2"
   },
   "source": [
    "#  Introduction\n",
    "\n",
    "This is a beginner-level kernel that uses a neural network built with PyTorch to predict which passengers survive the Titanic disaster. \n",
    "\n",
    "It focuses on implementing functionalities similar to scikit-learn to facilitate model building, training, cross-validation, and grid search. I will not go into much detail about PyTorch's API, thus some level of familiarity with PyTorch is expected, or you can follow along and check the [official documentation](https://pytorch.org/docs/stable/index.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [],
   "source": [
    "import itertools\n",
    "import copy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import torch\n",
    "\n",
    "from torch.autograd import Variable\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "sns.set_palette('deep')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "0d519ea8dbbbdbbca502c07371a953198b8f4d5d"
   },
   "source": [
    "# Loading the Data\n",
    "\n",
    "To focus on the PyTorch side of things and to keep this Kernel short and to the point, we are going to load the features from my previous kernel: [Surviving the Titanic step-by-step with groups](https://www.kaggle.com/dr1t10/surviving-the-titanic-step-by-step-with-groups)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_uuid": "e6703427fae85efb3ac61b87fe4b0f8b092c2bcf"
   },
   "outputs": [],
   "source": [
    "train = pd.read_csv(\"train.csv\")\n",
    "test = pd.read_csv(\"test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8d0780adcc7275c2d330d95ae923b65c223327e7"
   },
   "source": [
    "These are the features we are going to use for the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "b618a50cd4d27441865d8c6982b2e3ec8ad476e4",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>AgeBand</th>\n",
       "      <th>IsMale</th>\n",
       "      <th>InGroup</th>\n",
       "      <th>InWcg</th>\n",
       "      <th>WcgAllSurvived</th>\n",
       "      <th>WcgAllDied</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Pclass  AgeBand  IsMale  InGroup  InWcg  WcgAllSurvived  \\\n",
       "0         0       3        1       1        0      0               0   \n",
       "1         1       1        2       0        1      0               0   \n",
       "2         1       3        1       0        0      0               0   \n",
       "3         1       1        2       0        1      0               0   \n",
       "4         0       3        2       1        0      0               0   \n",
       "\n",
       "   WcgAllDied  \n",
       "0           0  \n",
       "1           0  \n",
       "2           0  \n",
       "3           0  \n",
       "4           0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6f11db4595bb727f51d28bcbdc6fb549365ac221"
   },
   "source": [
    "and the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_uuid": "f7fc2f69bda08615f5b6cabc83c26e09d887ee16",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pclass</th>\n",
       "      <th>AgeBand</th>\n",
       "      <th>IsMale</th>\n",
       "      <th>InGroup</th>\n",
       "      <th>InWcg</th>\n",
       "      <th>WcgAllSurvived</th>\n",
       "      <th>WcgAllDied</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pclass  AgeBand  IsMale  InGroup  InWcg  WcgAllSurvived  WcgAllDied\n",
       "0       3        2       1        0      0               0           0\n",
       "1       3        3       0        0      0               0           0\n",
       "2       2        3       1        0      0               0           0\n",
       "3       3        1       1        0      0               0           0\n",
       "4       3        1       0        1      1               1           0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1291b38fc8a8f3b7fbeaf82100af166f0c17bfb0"
   },
   "source": [
    "A brief description of the features:\n",
    "* **Pclass**: The ticket class;\n",
    "* **AgeBand**: `Age` split into 4 bands using K-Means. Ranges from 0 to 3 with ascending `Age`;\n",
    "* **IsMale**: `Sex` in binary form;\n",
    "* **InGroup**: Is `1` if the passenger is in a group; otherwise `0`;\n",
    "* **InWcg**: Is `1` if the passenger is in a woman-child-group; otherwise `0`;\n",
    "* **WcgAllSurvived**: Equal to `1` if all members of its woman-child-group survived; otherwise `0`;\n",
    "* **WcgAllDied**: The opposite of `WcgAllSurvived`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "98fc76f497d037e5a8748d05f5cd11055016f097"
   },
   "source": [
    "# Neural Network with PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "87750deca40a7b98517a2416ddf01843aea273a9"
   },
   "source": [
    "## Building the foundation\n",
    "\n",
    "While scikit-learn exposes high-level, easy-to-use classes and methods that allow us to define a model, train, and validate it in a few lines of code, PyTorch is more low-level and is not as straight-forward. We are then going to start by laying the foundation with some simple functions inspired in scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7183c9bfaa844c6ffa2fe3c56c1c4cbad6e5aca4"
   },
   "source": [
    "\n",
    "### TitanicNet\n",
    "\n",
    "Our neural network (TitanicNet) consists of an input layer with `d_in` inputs,`n_hidden` hidden layers with `d_hidden` neurons, and an output layer with `d_out` neurons.\n",
    "\n",
    "The function `titanic_net` creates a network given `d_in`, `d_hidden`, `n_hidden`, and `d_out` as arguments, allowing us to create different network configurations easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "2fdd176ff8d88cf5ec728135af1d719d5a76cf8e"
   },
   "outputs": [],
   "source": [
    "def titanic_net(d_in, d_hidden, n_hidden, d_out):\n",
    "    if d_in < 1 or d_hidden < 1 or d_out < 1:\n",
    "        raise ValueError(\"expected layer dimensions to be equal or greater than 1\")\n",
    "    if n_hidden < 0:\n",
    "        raise ValueError(\"expected number of hidden layers to be equal or greater than 0\")    \n",
    "    \n",
    "    # If the number of hidden layers is 0 we have a single-layer network\n",
    "    if n_hidden == 0:\n",
    "        return torch.nn.Linear(d_in, d_out)\n",
    "    \n",
    "    # Number of hidden layers is greater than 0\n",
    "    # Define the 3 main blocks\n",
    "    first_hlayer = [torch.nn.Linear(d_in, d_hidden), torch.nn.ReLU()]\n",
    "    hlayer = [torch.nn.Linear(d_hidden, d_hidden), torch.nn.ReLU()]   \n",
    "    output_layer = [torch.nn.Linear(d_hidden, d_out)]  \n",
    "    \n",
    "    # Build the model\n",
    "    layers = torch.nn.ModuleList()\n",
    "    \n",
    "    # First hidden layer\n",
    "    layers.extend(first_hlayer)\n",
    "    \n",
    "    # Remaining hidden layers\n",
    "    # Subtract 1 to account for the previous layer\n",
    "    for i in range(n_hidden - 1):        \n",
    "        layers.extend(hlayer)\n",
    "    \n",
    "    # Output layer\n",
    "    layers.extend(output_layer)\n",
    "    \n",
    "    return torch.nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6cabf571e34db6b3fc40fae0cfc9e9e2fd210072"
   },
   "source": [
    "### Training the network\n",
    "\n",
    "The `fit` function trains the network. It expects a model (`model`), the training samples (`X`), and the targets (`y`). \n",
    "\n",
    "Optionally, we can specify the number of epochs (`epochs`), one of `['sgd', 'rmsprop', 'adam']` as the optimization algorithm (`optim`), the learning rate for said algorithm (`lr`), and a verbosity level (`verbose`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_uuid": "e7365798adee2ab4bf057cd181fcee796419b72d"
   },
   "outputs": [],
   "source": [
    "def fit(model, X, y, epochs=250, optim='adam', lr=0.001, verbose=0):\n",
    "    # Optimizer argument validation\n",
    "    valid_optims = ['sgd', 'rmsprop', 'adam']\n",
    "    optim = optim.lower()\n",
    "    if optim.lower() not in valid_optims:\n",
    "        raise ValueError(\"invalid optimizer got '{0}' and expect one of {1}\"\n",
    "                         .format(optim, valid_optims))\n",
    "    \n",
    "    # Define the loss function - we are dealing with a classification task with two classes\n",
    "    # binary cross-entropy (BCE) is, therefore, the most appropriate loss function.\n",
    "    # Within BCE we can use BCELoss or BCEWithLogitsLoss. The latter is more stable, so we'll\n",
    "    # use that one. It expects logits, not predictions, which is why our output layer doesn't\n",
    "    # have an activation function\n",
    "    loss_fn = torch.nn.BCEWithLogitsLoss()\n",
    "\n",
    "    # Define the optimization algorithm\n",
    "    optim = optim.lower()\n",
    "    if optim == 'sgd':\n",
    "        optimizer = torch.optim.SGD(model.parameters(), lr=lr)\n",
    "    elif optim == 'rmsprop':\n",
    "        optimizer = torch.optim.RMSprop(model.parameters(), lr=lr)\n",
    "    elif optim == 'adam':\n",
    "        optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "\n",
    "    # Training loop\n",
    "    for t in range(epochs):\n",
    "        # Forward pass: The model will return the logits, not predictions\n",
    "        logits = model(X)\n",
    "\n",
    "        # Compute loss from logits\n",
    "        loss = loss_fn(logits, y)\n",
    "\n",
    "        # Zero gradients, perform a backward pass, and update the weights.\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()    \n",
    "\n",
    "        # We can get the tensor of predictions by applying the sigmoid nonlinearity\n",
    "        pred = torch.sigmoid(logits)\n",
    "\n",
    "        # Compute training accuracy\n",
    "        acc = torch.eq(y, pred.round_()).cpu().float().mean().item()\n",
    "        \n",
    "        if verbose > 1:\n",
    "            print(\"Epoch {0:>{2}}/{1}: Loss={3:.4f}, Accuracy={4:.4f}\"\n",
    "                  .format(t + 1, epochs, len(str(epochs)), loss.item(), acc))\n",
    "        \n",
    "    if verbose > 0:\n",
    "        print(\"Training complete! Loss={0:.4f}, Accuracy={1:.4f}\".format(loss.item(), acc))\n",
    "\n",
    "    return {'loss': loss.item(), 'acc': acc}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4b5ffe2ac9d60031134207764ba9ba0429151877"
   },
   "source": [
    "### Cross-validation\n",
    "\n",
    "`cross_val_score` performs K-fold cross-validation and returns a list of scores for each fold. Similarly to `fit`,  it expects a model (`model`), training samples (`X`), and training targets (`y`). The samples and targets will be split into `cv` folds. As optional arguments, we can specify the number of epochs (`epochs`), the optimization algorithm (`optim`), the learning rate (`lr`), whether to use CUDA (`use_cuda`), and a verbosity level (`verbose`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_uuid": "93199bee2deeaa4df831111f549c9fba88799962",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def cross_val_score(model, X, y, cv=3, epochs=250, optim='adam', lr=0.001, use_cuda=True, verbose=0):\n",
    "    # Generate indices to split data into training and validation set\n",
    "    kfolds = KFold(cv, False).split(X)\n",
    "    \n",
    "    # For each fold, train the network and evaluate the accuracy on the validation set\n",
    "    score = []\n",
    "    for fold, (train_idx, val_idx) in enumerate(kfolds):\n",
    "        X_train = X[train_idx]\n",
    "        y_train = y[train_idx]\n",
    "        X_val = X[val_idx]\n",
    "        y_val = y[val_idx]\n",
    "\n",
    "        # Convert the training data to Variables\n",
    "        X_train = Variable(torch.Tensor(X_train), requires_grad=True)\n",
    "        y_train = Variable(torch.Tensor(y_train), requires_grad=False).unsqueeze_(-1)\n",
    "        X_val = Variable(torch.Tensor(X_val), requires_grad=False)\n",
    "        y_val = Variable(torch.Tensor(y_val), requires_grad=False).unsqueeze_(-1)\n",
    "        \n",
    "        # Clone the original model so we always start the training from an untrained model\n",
    "        model_train = copy.deepcopy(model)\n",
    "        \n",
    "        # Move model and tensors to CUDA if use_cuda is True\n",
    "        if (use_cuda):\n",
    "            X_train = X_train.cuda()\n",
    "            y_train = y_train.cuda()\n",
    "            X_val = X_val.cuda()\n",
    "            y_val = y_val.cuda()\n",
    "            model_train = model_train.cuda()\n",
    "\n",
    "        # Train the network\n",
    "        metrics = fit(model_train, X_train, y_train, epochs=epochs, optim=optim,\n",
    "                      lr=lr, verbose=0)\n",
    "        \n",
    "        # Predict for validation samples\n",
    "        y_val_pred = torch.sigmoid(model_train(X_val))\n",
    "        acc = torch.eq(y_val, y_val_pred.round_()).cpu().float().mean().item()\n",
    "        score.append(acc)\n",
    "        \n",
    "        if verbose > 1:\n",
    "            print(\"Fold {0:>{2}}/{1}: Validation accuracy={3:.4f}\"\n",
    "                  .format(fold + 1, cv, len(str(cv)), acc))\n",
    "\n",
    "    if verbose > 0:\n",
    "        print(\"Mean k-fold accuracy: {0:.4f}\".format(np.mean(score)))\n",
    "        \n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "380163bfc2f74803701866f7abf4c485002ec2e2"
   },
   "source": [
    "### Grid search\n",
    "\n",
    "The last function is `titanic_net_grid_search`, which exhaustively searches over a specified grid of hyperparameters and returns the network and parameters that yield the highest cross-validation accuracy.\n",
    "\n",
    "It expects the training samples (`X`), the targets (`y`), and the grid of hyperparameters (`param_grid`). The number of folds (`cv`), number of epochs (`epochs`), whether to use CUDA (`use_cuda`), and the verbosity level (`verbose`) are all optional arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_uuid": "22633b8159bcb3b0f092801d9c8e263388bb3e47",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def titanic_net_grid_search(X, y, param_grid, cv=3, epochs=250, use_cuda=True, verbose=0):\n",
    "    # Cartesian product of a dictionary of lists\n",
    "    # Source: https://stackoverflow.com/questions/5228158/cartesian-product-of-a-dictionary-of-lists\n",
    "    grid = list((dict(zip(param_grid, param))\n",
    "                 for param in itertools.product(*param_grid.values())))\n",
    "    \n",
    "    n_candidates = len(grid)\n",
    "    if verbose > 0:\n",
    "        print(\"Fitting {0} folds for each of {1} candidates, totaling {2} fits\"\n",
    "             .format(n_folds, n_candidates, n_folds * n_candidates))\n",
    "        print()\n",
    "    \n",
    "    # Do cross-validation for each combination of the hyperparameters in grid_param\n",
    "    best_params = None\n",
    "    best_model = None\n",
    "    best_score = 0\n",
    "    for candidate, params in enumerate(grid):\n",
    "        if verbose == 1:\n",
    "            progress = \"Candidate {0:>{2}}/{1}\".format(candidate + 1, n_candidates,\n",
    "                                                       len(str(n_candidates)))\n",
    "            print(progress, end=\"\\r\", flush=True)\n",
    "        elif verbose > 1:\n",
    "            print(\"Candidate\", candidate + 1)\n",
    "            print(\"Parameters: {}\".format(params))\n",
    "\n",
    "        # Model parameters and creation\n",
    "        d_in = X_train.shape[-1]\n",
    "        d_hidden = params['d_hidden']\n",
    "        n_hidden = params['n_hidden']\n",
    "        d_out = 1\n",
    "        model = titanic_net(d_in, d_hidden, n_hidden, d_out)\n",
    "\n",
    "        # Cross-validation\n",
    "        cv_score = cross_val_score(model, X_train, Y_train, cv = n_folds, epochs=epochs,\n",
    "                                   use_cuda=use_cuda, verbose=0)\n",
    "        cv_mean_acc = np.mean(cv_score)\n",
    "        if verbose > 1:\n",
    "            print(\"Mean CV accuracy: {0:.4f}\".format(cv_mean_acc))    \n",
    "            print()\n",
    "\n",
    "        # Check if this  is the best model; if so, store it\n",
    "        if cv_mean_acc > best_score:\n",
    "            best_params = params\n",
    "            best_model = model\n",
    "            best_score = cv_mean_acc\n",
    "\n",
    "    if verbose > 0:\n",
    "        if verbose == 1:\n",
    "            print()\n",
    "        print(\"Best model\")\n",
    "        print(\"Parameters: {}\".format(best_params))\n",
    "        print(\"Mean CV accuracy: {0:.4f}\".format(best_score))\n",
    "    \n",
    "    return {'best_model': best_model, 'best_params': best_params, 'best_score': best_score}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5433c9ba4a04a9560f206aadc3d00e49e831d2d5"
   },
   "source": [
    "## Training TitanicNet\n",
    "\n",
    "Split the `train` data frame into training samples (`X_train`), training targets (`Y_train`), and convert the `test` data frame into testing samples (`X_test`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_uuid": "b8a271c2683552d5ab3fb4d46a5cc1c2a3ce62f7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training samples shape: (891, 7)\n",
      "Training targets shape: (891,)\n",
      "Test samples shape: (418, 7)\n"
     ]
    }
   ],
   "source": [
    "# Split the training set into samples and targets\n",
    "X_train = np.array(train.drop(columns='Survived'))\n",
    "Y_train = np.array(train['Survived'].astype(int))\n",
    "\n",
    "# Test set samples to predict\n",
    "X_test = np.array(test)\n",
    "\n",
    "# Always important to check if the shapes agree with each other\n",
    "print(\"Training samples shape: {}\".format(X_train.shape))\n",
    "print(\"Training targets shape: {}\".format(Y_train.shape))\n",
    "print(\"Test samples shape: {}\".format(X_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "0de890ca39f311fc9fde98c271d58f6f37cbae03"
   },
   "source": [
    "It's finally time to train our networks and find the one with the highest performance.\n",
    "\n",
    "(the next code cell takes about 12 minutes to run with GPU enabled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_uuid": "195db1f6cd79a0dd05e4391146995adeefe3f5fc",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 45 candidates, totaling 450 fits\n",
      "\n",
      "Candidate 45/45\n",
      "Best model\n",
      "Parameters: {'n_hidden': 3, 'd_hidden': 7, 'lr': 0.01, 'optim': 'Adam'}\n",
      "Mean CV accuracy: 0.8619\n"
     ]
    }
   ],
   "source": [
    "# Number of folds\n",
    "n_folds = 10\n",
    "\n",
    "# Grid search\n",
    "grid = {\n",
    "    'n_hidden': [0, 3, 7, 10, 15],\n",
    "    'd_hidden': [3, 7, 10],\n",
    "    'lr': [0.001, 0.005, 0.01],\n",
    "    'optim': ['Adam']\n",
    "}\n",
    "best_candidate = titanic_net_grid_search(X_train, Y_train, grid, cv=n_folds,\n",
    "                                         epochs=500, verbose=1)\n",
    "\n",
    "# Our best network\n",
    "best_model = best_candidate['best_model']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7ed41b85a2c4c45e3a0ed4810f6b6c3313bc0e2d"
   },
   "source": [
    "# Predictions and submission\n",
    "\n",
    "The best model returned by `titanic_net_grid_search` is not trained, so before making predictions let's train it with the best parameters that we found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_uuid": "2ceef8945db4a7fbb781d6d2450059667f4e531d"
   },
   "outputs": [],
   "source": [
    "X_train_t = Variable(torch.Tensor(X_train), requires_grad=True)\n",
    "y_train_t = Variable(torch.Tensor(Y_train), requires_grad=False).unsqueeze_(-1)\n",
    "X_test_t = Variable(torch.Tensor(X_test), requires_grad=False)\n",
    "\n",
    "# Train the best model\n",
    "best_params = best_candidate[\"best_params\"]\n",
    "_ = fit(best_model, X_train_t, y_train_t, epochs=500,optim=best_params['optim'],\n",
    "        lr=best_params['lr'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "64827a003be261bc3a6c4a5378feefa3909f8ecc"
   },
   "source": [
    "Generating the predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_uuid": "897df33a092a0f64136e0b1217c0432d745c0c00"
   },
   "outputs": [],
   "source": [
    "# The model outputs logits, we have to apply the sigmoid function and round the result\n",
    "prediction = torch.sigmoid(best_model(X_test_t)).data.round_().numpy().flatten()\n",
    "\n",
    "# Submission\n",
    "_test = pd.read_csv(\"../input/test.csv\")\n",
    "submission_df = pd.DataFrame({'PassengerId': _test['PassengerId'], 'Survived': prediction.astype(int)})\n",
    "submission_df.to_csv(\"submission.csv\", index=False)\n",
    "\n",
    "# Storing the datasets\n",
    "train.to_csv(\"submission_train.csv\", index=False)\n",
    "test.to_csv(\"submission_test.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "38441e12ec5e4f9cc6a52dcef317f4d65eb9ed53",
    "collapsed": true
   },
   "source": [
    "Thank you for reading! Discussion and suggestions are welcome."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Titanic-p3",
   "language": "python",
   "name": "titanic-env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}