solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pothole detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is my slightly revised solution to the hackathon. The original winning solution had some poor design choices that I was not satisfied with. I guess the pressure of the hackathon environment really got to me. I was also very curious to see how high I can get the accuracy. I will try to comment on all the major changes I made to the code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import csv\n",
    "from PIL import Image\n",
    "\n",
    "import keras\n",
    "import keras.preprocessing.image as img\n",
    "from keras.applications import ResNet50, VGG16\n",
    "from keras.layers import Dense, Conv2D, BatchNormalization, Activation, Flatten\n",
    "from keras.layers.pooling import GlobalMaxPool2D, GlobalAveragePooling2D\n",
    "from keras.models import Model\n",
    "from keras.optimizers import SGD, Adam\n",
    "from keras import backend as K\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crop images and save in a new folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Edit:** For the updated solution I omitted this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My idea was to crop the dashboard and the sky parts of the images to save some computing time. I found the bounding box limits of all the training images and cropped the images at these points. In hindsight, I realise that this is not the best method since if the potholes in the test set appears outside these margins, this model will not be able to detect them. Which might be one of the reasons for the big difference between the validation (97%) and test accuracy (86%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get all the filenames\n",
    "all_files = []\n",
    "for path, subdirs, files in os.walk('data'):\n",
    "    for name in files:\n",
    "        all_files.append(os.path.join(path, name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# crop the images and save in data_crop folder\n",
    "for f in all_files:\n",
    "    temp_img = Image.open(f)\n",
    "    temp_img = temp_img.crop((0, 600-435, 800, 600-435+185))\n",
    "    temp_img.save('data_crop' + f.split('data')[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code is not efficient at all, but luckily the number of images is small."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create train and validation folders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we take 500 random images from the `train` folder and move it to the `valid` folder. You can do this either with the images in `data` or `data_crop`. 500 images is probably not enough to obtain a reasonable estimate of the test accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_files = []\n",
    "for path, subdirs, files in os.walk('data/train/'):\n",
    "    for name in files:\n",
    "        train_files.append(os.path.join(path, name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.shuffle(train_files)\n",
    "\n",
    "valid_files = train_files[:500]\n",
    "train_files = train_files[500:]\n",
    "\n",
    "for f in valid_files:\n",
    "    os.rename(f, 'data/valid/' + f.split('data/train/')[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to reset the split you can run the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%mv data/valid/positive/* data/train/positive/\n",
    "%mv data/valid/negative/* data/train/negative/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Edit:** I finally realised what caused the huge difference between validation and test accuracy. The images were taken from different frames of video footage. Therefore, images that were taken say 1 second frames apart are very similar to each other, depending how fast the driver was going at the time. Thus if we randomly sample images from the training set for a validation set, some of the validation images will look very similar to the images in the training set. I suspect the images in the test set are from totally different video footage and not similar to any of the training images. Thus the big difference!\n",
    "For a more appropriate validation set, we need to find a way to group the images by the sequence they were taken in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup data generators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following section creates the batch generators for training and validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since, we are using models pretrained on ImageNet, we subtract the ImageNet means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def imagenet_mean(x):\n",
    "    x = x[..., ::-1]\n",
    "    x[..., 0] -= 103.939\n",
    "    x[..., 1] -= 116.779\n",
    "    x[..., 2] -= 123.68\n",
    "    return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data augmentations include horizontal flip and small horizontal and vertical shifts. The shifts are a bit risky since they can cut off some of the potholes, but I didn't have time to compare results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Edit:** In the original solution, I used 0.05 for both shifts, but now, since I took a way the cropping step and we have more vertical area to play with, I increased the random vertical shift range to 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_gen = img.ImageDataGenerator(\n",
    "    horizontal_flip=True,\n",
    "    width_shift_range=0.05,\n",
    "    height_shift_range=0.1,\n",
    "    preprocessing_function=imagenet_mean\n",
    ")\n",
    "test_gen = img.ImageDataGenerator(\n",
    "    preprocessing_function=imagenet_mean\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choose sizes that fit in your machine's memory. I suppose bigger is better. Note that the current specification strecthes the images vertically. I argued that this might increase the visibility of the 'flatter' potholes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size=64\n",
    "img_size = (300,300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Edit:** In the original solution I stupidly did not set `class_mode='binary'` and therefore the labels were represented by a 2-column array, instead of the now more compact representation as a 1-D vector. Note, during the hackacthon, I had to set `class_mode=None` for the test batches, since their annotations were unknown. We will still only use the test images for the final evaluation of our model and not to do any type of model selection after 'looking' at the test images. We want to imitate the hackathon setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_batches = train_gen.flow_from_directory(\n",
    "    'data/train/',\n",
    "    batch_size=batch_size,\n",
    "    target_size = img_size,\n",
    "    class_mode='binary'\n",
    ")\n",
    "\n",
    "valid_batches = test_gen.flow_from_directory(\n",
    "    'data/valid/',\n",
    "    batch_size=batch_size,\n",
    "    target_size = img_size,\n",
    "    shuffle=False,\n",
    "    class_mode='binary'\n",
    ")\n",
    "\n",
    "test_batches = test_gen.flow_from_directory(\n",
    "    'data/test/',\n",
    "    batch_size=batch_size,\n",
    "    target_size = img_size,\n",
    "    shuffle=False,\n",
    "    class_mode='binary'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the one of the generators output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp_train_batch = train_batches.next()\n",
    "print('X shape: ', temp_train_batch[0].shape)\n",
    "print('Y shape: ', temp_train_batch[1].shape)\n",
    "\n",
    "plt.imshow(temp_train_batch[0][0].astype('uint8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I used an ensemble of 3 pretrained ConvNets: ResNet50, ResNet101 and DenseNet121. Each model I trained on a different train/validation split and averaged their predictions on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# choose the convnet\n",
    "base_model = ResNet50(include_top=False, input_shape=img_size + (3,))\n",
    "#base_model = densenet121_model(img_rows=img_size[0], img_cols=img_size[1], color_type=3, num_classes=2)\n",
    "#base_model = resnet101_model(img_rows=img_size[0], img_cols=img_size[1], color_type=3, num_classes=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "base_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add new classification head. Can use max or average pooling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_map = base_model.get_layer(index=-2).output\n",
    "\n",
    "x = Conv2D(128, (3,3), padding='same')(ft_map)\n",
    "x = BatchNormalization()(x)\n",
    "x = Activation('relu')(x)\n",
    "\n",
    "x = Conv2D(1, (3,3), activation='sigmoid', padding='same')(x)\n",
    "x = GlobalAveragePooling2D()(x)\n",
    "\n",
    "model = Model(base_model.input, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, train only the new classification layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# freeze all the base model layers\n",
    "for layer in base_model.layers:\n",
    "    layer.trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can experiment with different optimising strategies. I found that small learning rates worked the best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "opt = Adam(0.001)#, momentum=0.9)\n",
    "model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start training. I repeated the following sequence as necessary: fit -> save_weights -> decrease learning rate -> repeat."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit_generator(train_batches, \n",
    "                    steps_per_epoch=np.ceil(train_batches.samples/batch_size), \n",
    "                    epochs=5, verbose=1, \n",
    "                    validation_data=valid_batches, \n",
    "                    validation_steps=np.ceil(valid_batches.samples/batch_size),\n",
    "                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save_weights('models/rn50_cls.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "K.set_value(model.optimizer.lr, 0.00001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit_generator(train_batches, \n",
    "                    steps_per_epoch=np.ceil(train_batches.samples/batch_size), \n",
    "                    epochs=5, verbose=1, \n",
    "                    validation_data=valid_batches, \n",
    "                    validation_steps=np.ceil(valid_batches.samples/batch_size),\n",
    "                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save_weights('models/rn50_cls.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i,layer in enumerate(model.layers):\n",
    "    print(i, layer.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fine-tune deeper layers - either conv5 block or conv5 + conv4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for layer in model.layers[:141]:\n",
    "    layer.trainable = False\n",
    "    \n",
    "for layer in model.layers[141:]:\n",
    "    layer.trainable = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "opt = Adam(0.0001)\n",
    "model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit_generator(train_batches, \n",
    "                    steps_per_epoch=np.ceil(train_batches.samples/batch_size), \n",
    "                    epochs=5, verbose=1, \n",
    "                    validation_data=valid_batches, \n",
    "                    validation_steps=np.ceil(valid_batches.samples/batch_size),\n",
    "                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save_weights('models/rn50_block5.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "K.set_value(model.optimizer.lr, 0.00001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model.fit_generator(train_batches, \n",
    "                    steps_per_epoch=np.ceil(train_batches.samples/batch_size), \n",
    "                    epochs=3, verbose=1, \n",
    "                    validation_data=valid_batches, \n",
    "                    validation_steps=np.ceil(valid_batches.samples/batch_size),\n",
    "                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save_weights('models/rn50_block5.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "K.set_value(model.optimizer.lr, 0.000001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model.fit_generator(train_batches, \n",
    "                    steps_per_epoch=np.ceil(train_batches.samples/batch_size), \n",
    "                    epochs=2, verbose=1, \n",
    "                    validation_data=valid_batches, \n",
    "                    validation_steps=np.ceil(valid_batches.samples/batch_size),\n",
    "                    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate on hold-out sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we test the model on the validation set, but it can also be applied to the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# load data in memory\n",
    "valid_batches.reset()\n",
    "x_valid = np.vstack([valid_batches.next()[0] for x in range(int(np.ceil(valid_batches.samples/batch_size)))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "valid_batches.reset()\n",
    "y_valid = np.concatenate([valid_batches.next()[1] for x in range(int(np.ceil(valid_batches.samples/batch_size)))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Little bit of TTA, predict on both horisontal orientations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_valid = np.zeros_like(y_valid)\n",
    "for flip in [False, True]:\n",
    "    temp_x = x_valid\n",
    "    if flip:\n",
    "        temp_x = img.flip_axis(temp_x, axis=2)\n",
    "    p_valid += 0.5 * np.reshape(model.predict(temp_x, verbose=1), y_valid.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean((p_valid > 0.5) == y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# load data in memory\n",
    "test_batches.reset()\n",
    "x_test = np.vstack([test_batches.next()[0] for x in range(int(np.ceil(test_batches.samples/batch_size)))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_batches.reset()\n",
    "y_test = np.concatenate([test_batches.next()[1] for x in range(int(np.ceil(test_batches.samples/batch_size)))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Little bit of TTA, predict on both horisontal orientations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_test = np.zeros_like(y_test)\n",
    "for flip in [False, True]:\n",
    "    temp_x = x_test\n",
    "    if flip:\n",
    "        temp_x = img.flip_axis(temp_x, axis=2)\n",
    "    p_test += 0.5 * np.reshape(model.predict(temp_x, verbose=1), y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean((p_test > 0.5) == y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pothole localisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cam_extract = Model(base_model.input, model.get_layer(index=-3).output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cam_valid = cam_extract.predict(x_valid, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_ind = np.random.randint(low=0,high=500)\n",
    "valid_file = valid_batches.filenames[valid_ind]\n",
    "print(valid_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_cam = cam_extract.predict(np.expand_dims(x_valid[valid_ind], 0))\n",
    "np.max(valid_cam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "overlay = img.array_to_img(valid_cam[0]).resize((800,600), Image.BILINEAR).convert('RGB')\n",
    "bg = img.load_img('data/valid/' + valid_file)#.resize((300,300))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "Image.blend(alpha=0.5, im1=bg, im2=overlay)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_ind = np.random.randint(high=1500,low=0)\n",
    "test_file = test_batches.filenames[test_ind]\n",
    "print(test_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_cam = cam_extract.predict(np.expand_dims(x_test[test_ind], 0))\n",
    "np.max(test_cam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "overlay = img.array_to_img(test_cam[0]).resize((800,600), Image.BILINEAR).convert('RGB')\n",
    "bg = img.load_img('data/test/' + test_file)#.resize((300,300))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "Image.blend(alpha=0.5, im1=bg, im2=overlay)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}