labs/practical_06.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-jlGXqM_t452"
   },
   "source": [
    "# P7 - Convolutional Neural Networks (CNNs)\n",
    "We have now learned about the Perceptron, Linear and logistic regression, Multi-layer perceptron and backpropagation, Auto-encoders.\n",
    "\n",
    "In this pratical session about Convolutional Neural Networks (CNNs) we will use the MNIST datasets.\n",
    "\n",
    "First, we will obtain baselines using a Logistic Regression and a Feed-forward Neural Network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ITJR4snhxdT0"
   },
   "source": [
    "## 0.0 - Imports\n",
    "We will need to import some libraries to be used in this session. Libraries include data visualizers ([matplotlib](https://matplotlib.org/)), neural network package ([torch](https://pytorch.org/)), and other helper packages for data handling ([sklearn](https://scikit-learn.org/), [numpy](https://numpy.org/))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "MWGjU3tDw4bD",
    "ExecuteTime": {
     "end_time": "2023-12-19T09:15:27.445778300Z",
     "start_time": "2023-12-19T09:15:21.984478600Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\envs\\IST_DL21_Env\\lib\\site-packages\\torchvision\\io\\image.py:13: UserWarning: Failed to load image Python extension: '[WinError 127] The specified procedure could not be found'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?\n",
      "  warn(\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.utils import check_random_state\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms\n",
    "from torch.autograd import Variable\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torch.utils.data.sampler import SubsetRandomSampler\n",
    "import time\n",
    "import copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W-od7M6WMN0N"
   },
   "source": [
    "Then, other variable definitions are needed to be set. This includes the size of the dataset we will use, and the configuration of the GPU to be activated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ECqewHJ0MM62",
    "outputId": "f5749bd9-87e6-483b-8d1e-220e4aa3c0d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "# Configure Device\n",
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "odY0Ng9yycgr"
   },
   "source": [
    "### 0.1 - Create Dataloaders\n",
    "#### MNIST dataset\n",
    "Using torchvision we can easily download and use the MNIST dataset to create our train and validation dataloaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "snFv-Hu-zRnW"
   },
   "outputs": [],
   "source": [
    "# Define tranform - Convert data to tensor and normalize using dataset mean and std\n",
    "# mean and std are computed offline using the training dataset\n",
    "# tranforms.Normalize expects a value of mean and std per image channel\n",
    "mnist_transform = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "     transforms.Normalize((0.1307,), (0.3081,))])\n",
    "\n",
    "# Download and create MNIST train and validation dataloaders\n",
    "mnist_train_dataset = datasets.MNIST('../data', download=True, train=True, transform=mnist_transform)\n",
    "mnist_val_dataset = datasets.MNIST('../data', download=True, train=False, transform=mnist_transform)\n",
    "mnist_train_dataloader = DataLoader(mnist_train_dataset, batch_size=64, shuffle=True)\n",
    "mnist_val_dataloader = DataLoader(mnist_val_dataset, batch_size=64, shuffle=True)\n",
    "\n",
    "# MNIST Dataloaders to get data into numpy for Logistic Regression\n",
    "mnist_train_dataloader_numpy = DataLoader(mnist_train_dataset, batch_size=len(mnist_train_dataset))\n",
    "mnist_val_dataloader_numpy = DataLoader(mnist_val_dataset, batch_size=len(mnist_val_dataset))\n",
    "X_y_train = next(iter(mnist_train_dataloader_numpy))\n",
    "X_y_val = next(iter(mnist_val_dataloader_numpy))\n",
    "X_train = X_y_train[0].numpy()\n",
    "y_train = X_y_train[1].numpy()\n",
    "X_val = X_y_val[0].numpy()\n",
    "y_val = X_y_val[1].numpy()\n",
    "\n",
    "dataloaders = dict(train=mnist_train_dataloader, val=mnist_val_dataloader)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vJANJSJ0hsqd"
   },
   "source": [
    "We can check the MNIST dataset properties such as:\n",
    "\n",
    "- shape of train and validation datasets - \\[number of samples, width, height\\]\n",
    "- number of input feature on the flattened/reshaped input for Logistic Regression or MLP\n",
    "- shape of train and validation batches - \\[batch size, number of channels, width, height\\]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KuluBqnnCbn5",
    "outputId": "ae3765db-a0b9-4c4f-e5bd-9f50eed9f991"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datasets shapes: {'train': torch.Size([60000, 28, 28]), 'val': torch.Size([10000, 28, 28])}\n",
      "N input features: 784 Output classes: 10\n",
      "Train batch: torch.Size([64, 1, 28, 28]) torch.Size([64])\n",
      "Val batch: torch.Size([64, 1, 28, 28]) torch.Size([64])\n"
     ]
    }
   ],
   "source": [
    "# get batch to extract properties and plot example images\n",
    "# next(enumerator(dataloader)) -> creates an iterator of the dataloader and gets the next batchß\n",
    "batch_idx, (example_imgs, example_targets) = next(enumerate(mnist_train_dataloader))\n",
    "# info about the dataset\n",
    "D_in = np.prod(example_imgs.shape[1:])\n",
    "D_out = len(mnist_train_dataloader.dataset.targets.unique())\n",
    "print(\"Datasets shapes:\", {x: dataloaders[x].dataset.data.shape for x in ['train', 'val']})\n",
    "print(\"N input features:\", D_in, \"Output classes:\", D_out)\n",
    "print(\"Train batch:\", example_imgs.shape, example_targets.shape)\n",
    "batch_idx, (example_imgs, example_targets) = next(enumerate(mnist_val_dataloader))\n",
    "print(\"Val batch:\", example_imgs.shape, example_targets.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JnFAmoinjY1T"
   },
   "source": [
    "We can plot some examples with corresponding labels using the following function. This function can also receive the predicted labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 459
    },
    "id": "5ZWvjQOvC2ep",
    "outputId": "7abb9b79-83d7-4bfb-d1f4-6e3fdba72359"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAAGlCAYAAABQuDoNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAyrklEQVR4nO3de1xVZbrA8We7RRBFUvES2igBCmlHDfNTedna8fJpTMe8IGYhNiqap8k6atF08p5jqUc7U+NhTmoZNR1Tk8ljpo63bqfbsbGLDirkBTMzRcQLCu/5ww/UlnfJXrg2+93w+34+/sHDWms/G9aDDy/72a9LKaUEAAAAAVcn0AkAAADgChozAAAAQ9CYAQAAGILGDAAAwBA0ZgAAAIagMQMAADAEjRkAAIAhaMwAAAAMQWMGAABgCBqzGqB3797Su3fvQKcB1BjUFOA86so3jjRmLpfLp3/bt2934uEctX379mvmPG/ePFvnh4SEyM033yypqaly8ODBanoWVZeXl1ee+5o1ayp8fubMmeJyueTHH38MQHa1VzDXlIjIhQsXZP78+XLLLbdIeHi4tGrVSkaMGCFff/11pedSU/CXYK8rEZHCwkKZPn26xMTESGhoqLRq1UqGDx8u586du+Z51FXwqOvERVatWuX18auvviqbN2+uEE9MTHTi4RyVmJhYIU+RK8/pvffek/79+/t0nd/97ndy++23y6VLl+SLL76QzMxM2bBhg+zZs0eio6OdTtsvZs+eLUOHDhWXyxXoVGq9YK4pEZHRo0dLdna2jB8/Xm677TbJz8+XF198Ue68807Zs2ePtGnTptJrUFNwWrDXVUFBgXg8Hjly5IhMmDBB4uLi5MSJE7Jr1y65ePGihIeHV3oN6ioIKD+YPHmy8uXSRUVF/nh4R8TFxan4+PhKj9u2bZsSEbV69Wqv+AsvvKBERD377LOW5549e/a681RKKY/HozweT5XOzc3NVSKiOnfurERErVmzxuvzM2bMUCKiTpw44UCmqKpgqqkjR44oEVFTp071iv/tb39TIqIWL158zfOpKVSXYKorpZSaNGmSuuGGG9TBgwdtn0tdBY9qe41Z7969pWPHjvL5559Lr169JDw8XJ566ikRubK8PHPmzArntG3bVtLS0rxip0+flilTpshNN90koaGhEhcXJwsWLJDS0lKv444dOyZ79+6VS5cu2c71k08+kf3798vo0aNtn1vm7rvvFhGR3NxcEfl5mfWbb76R+++/Xxo3biw9evQoP/61116TpKQkqV+/vjRp0kRSUlLk8OHDFa6bmZkpsbGxUr9+fenWrZvs2rVL+/iHDh2SvXv3+pxvSkqKtGvXTmbPni1KqUqPX716dXm+UVFR8sADD8jRo0e9jklLS5OGDRvK0aNHZciQIdKwYUNp1qyZTJ06VUpKSryOLS0tlSVLlkiHDh0kLCxMWrRoIenp6XLq1Cmfn0NtY2pNFRYWiohIixYtvOI33nijiIjUr1/fztMsR01RU9XB1Lo6ffq0rFixQiZMmCAxMTFSXFwsFy9evK7nKkJdiZhXV9X64v+TJ0/KPffcI507d5YlS5ZInz59bJ1/7tw58Xg88tprr0lqaqq88MIL0r17d8nIyJDHH3/c69iMjAxJTEys8A3wRVZWlojIdTVmBw4cEBGRpk2besVHjBgh586dk2effVbGjx8vIiLz5s2T1NRUiY+Pl8WLF8uUKVNk69at0qtXLzl9+nT5uS+//LKkp6dLy5Yt5bnnnpPu3bvL4MGDtUWRmppqazne7XbL008/LV9++aWsW7fumseuXLlSkpOTxe12y/z582X8+PGydu1a6dGjh1e+IiIlJSUyYMAAadq0qSxcuFA8Ho8sWrRIMjMzvY5LT0+XadOmSffu3WXp0qUyduxYycrKkgEDBlSpua4tTKyp2NhYad26tSxatEj++te/ypEjR+STTz6RiRMnSkxMjKSkpNh+niLUVBlqyv9MrKv3339fLly4IHFxcTJ8+HAJDw+X+vXrS/fu3WX37t12n2I56uoKo+rKH8twuuVhj8ejREQtW7aswvEiombMmFEh3qZNGzVmzJjyj+fMmaMaNGig/vGPf3gd9+STTyq3260OHTpUHhszZowSEZWbm2sr98uXL6sWLVqobt26+XR82fLw8uXL1YkTJ1R+fr7asGGDatu2rXK5XOrTTz9VSv28zDpq1Civ8/Py8pTb7Vbz5s3ziu/Zs0fVrVu3PF5cXKyaN2+uOnfurC5evFh+XGZmphKRCsvDZV/vypQtDz///PPq8uXLKj4+XnXq1EmVlpZ65V22PFyWR8eOHdX58+fLr/POO+8oEVHPPPNMeazsezB79myvx+zSpYtKSkoq/3jXrl1KRFRWVpbXce+++642XhsFW0397//+r4qNjVUiUv4vKSlJHTt2rNJzqakrqCn/C6a6Wrx4sRIR1bRpU9WtWzeVlZWlXnrpJdWiRQvVuHFjlZ+ff83zqasrgqGuqnXFLDQ0VMaOHVvl81evXi09e/aUxo0by48//lj+r2/fvlJSUiI7d+4sP3blypWilJK2bdvaeoytW7fK8ePHba+WPfTQQ9KsWTOJjo6WgQMHSlFRkbzyyivStWtXr+MmTpzo9fHatWultLRUkpOTvZ5Ty5YtJT4+XrZt2yYiIp999pn88MMPMnHiRKlXr175+WlpaRIZGVkhn+3bt/u0zPtLv/xN5O2339YeU5bHww8/LGFhYeXxgQMHSkJCgmzYsKHCOVc/5549e3pNAa1evVoiIyOlX79+Xl+DpKQkadiwYfnXABWZWlONGzeWzp07y5NPPilvv/22LFy4UPLy8mTEiBFy4cIFn3KjpqipQDGxrs6ePSsiV/6cunXrVrn//vtl0qRJ8vbbb8upU6fkxRdf9Ck36sr8unJkKtNXrVq18vpG2ZWTkyN///vfpVmzZtrP//DDD1W+dpmsrCxxu90ycuRIW+c988wz0rNnT3G73RIVFSWJiYlSt27FL29MTIzXxzk5OaKUkvj4eO11Q0JCRETku+++ExGpcFzZyLNTRo8eLXPmzJHZs2fLkCFDKny+LI/27dtX+FxCQoK8//77XrGwsLAK36/GjRt7/T0+JydHCgoKpHnz5tqcnPi+1lQm1lRBQYH07NlTpk2bJv/6r/9aHu/atav07t1bVqxYIZMmTar0OtQUNRUoJtZV2WszBw0aJA0bNiyP33HHHRITEyMffvihT9ehrsyvq2ptzOy+6Ff3ort+/frJ9OnTtce3a9euyrmJiJw/f17WrVsnffv2rfDC5crceuut0rdv30qPu/prUFpaKi6XSzZu3Chut7vC8b8swOpQ9ptIWlqarF+/3pHrVaa0tFSaN29e/tq+q1n9cIOZNbVmzRo5fvy4DB482Cvu8XikUaNG8sEHH/jUmFFT1terDDV1fUysq7K3stD939S8eXOfX3xOXVlfrzLVVVfV2phZady4cYUX4hUXF8uxY8e8YrGxsXL27FmfbqqqyM7OlsLCwut60b9dsbGxopSSmJiYaxZr2fs+5eTklE/RiIhcunRJcnNzpVOnTo7l9MADD8jcuXNl1qxZFf5zLctj3759XnmUxXx5f6qrxcbGypYtW6R79+5VntiDt0DW1PHjx0Wk4n9WSikpKSmRy5cvO/ZYOtQUNeUvgayrpKQkERHtkEB+fr4kJCQ49lg61FX11ZURWzLFxsZ6/c1d5Mqo7dU/2JOTk+Wjjz6STZs2VbjG6dOnvX7gV+XtMl5//XUJDw+X++67z+YzqLqhQ4eK2+2WWbNmVfg7u1JKTp48KSJX/gzUrFkzWbZsmRQXF5cfs3Llygo/KETsjyD/UtlvIrt375bs7Gyvz3Xt2lWaN28uy5Yt8xrV3rhxo3z77bcycOBA24+XnJwsJSUlMmfOnAqfu3z5svb54doCWVNlP7T/8pe/eMWzs7OlqKhIunTpYuu52EVNUVP+Esi6at++vXTq1EnWr1/v9e727733nhw+fFj69etXlafkM+qq+urKiBWzcePGycSJE2XYsGHSr18/+fLLL2XTpk0SFRXlddy0adMkOztb7r33XklLS5OkpCQpKiqSPXv2yFtvvSV5eXnl52RkZMgrr7wiubm5Pr1Y+aeffpKNGzfKsGHDqnVJNjY2VubOnSsZGRmSl5cnQ4YMkYiICMnNzZV169bJhAkTZOrUqRISEiJz586V9PR0ufvuu2XkyJGSm5srK1as0P7dPjU1VXbs2GH7RZVlyv5+f/UYdkhIiCxYsEDGjh0rHo9HRo0aJcePH5elS5dK27Zt5bHHHrP9WB6PR9LT02X+/Pmye/du6d+/v4SEhEhOTo6sXr1ali5dKsOHD6/S86itAllTgwYNkg4dOsjs2bPlu+++kzvuuEP2798vf/zjH+XGG2+U3/72t/586tSUUFP+Euj/q/793/9d+vXrJz169JD09HQpKCiQxYsXS7t27Xx6ecD1oK6qsa4cme28itUIcocOHbTHl5SUqCeeeEJFRUWp8PBwNWDAALV///4KI8hKKVVYWKgyMjJUXFycqlevnoqKilJ33XWXWrhwoSouLi4/zu7bZSxbtkyJiMrOzrb1XK3eTflqlb0r8Zo1a1SPHj1UgwYNVIMGDVRCQoKaPHmy2rdvn9dxL730koqJiVGhoaGqa9euaufOndp3U67KCPLVVqxYUf5WB1fn/eabb6ouXbqo0NBQ1aRJEzV69Gh15MgRr2PGjBmjGjRoYPm1uFpmZqZKSkpS9evXVxEREerWW29V06dPr3QMvDYItpr66aef1GOPPabatWunQkNDVVRUlEpJSfHpHcupKWqqugRbXSml1ObNm9Udd9yhwsLCVJMmTdSDDz5o621oqCvz68qlVBXbVAAAADjKiNeYAQAAgMYMAADAGDRmAAAAhqAxAwAAMASNGQAAgCFozAAAAAzh0xvMlpaWSn5+vkRERIjL5fJ3ToDPlFJSWFgo0dHRUqdOcP2eQV3BVNQV4Dxf68qnxiw/P19uuukmx5IDnHb48GFp3bp1oNOwhbqC6agrwHmV1ZVPvwpFREQ4lhDgD8F4jwZjzqhdgvEeDcacUbtUdo/61JixHAzTBeM9Gow5o3YJxns0GHNG7VLZPRpcLx4AAACowWjMAAAADEFjBgAAYAgaMwAAAEPQmAEAABiCxgwAAMAQNGYAAACGoDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAgaMwAAAEPQmAEAABiCxgwAAMAQdQOdAAAAqDl69+6tjW/btk0b79Onjza+fft2hzIKLqyYAQAAGILGDAAAwBA0ZgAAAIagMQMAADAEjRkAAIAhmMoEUOPNnDlTG/d4PLaus2PHDlvXB2oyq/t+xowZtq5jNcXJVCYAAAACisYMAADAEDRmAAAAhqAxAwAAMASNGQAAgCFcSilV2UFnzpyRyMjI6sgHqJKCggJp1KhRoNOwhbqqOrtTllZTX05xuVx+vX6gUFcQcW760kpNrR8rldUVK2YAAACGoDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAj2ygQQcNu2bdPG/T1N6RSrqTX20EQw8Xcd1ta9L+1ixQwAAMAQNGYAAACGoDEDAAAwBI0ZAACAIWjMAAAADFHrpzJDQkK08Tp19D2r1fF9+vTRxu+8805b+dx3333aeEJCgq3rLF++XBufPn26Nn7y5Elb1weqwoeteYOS1R6dgImspoX9PX1p9f8kvLFiBgAAYAgaMwAAAEPQmAEAABiCxgwAAMAQNGYAAACGMH4q02oKMioqSht/8MEHtfHWrVtr47fccout61sdb5WnU1Nodq8zduxYbXz+/PnaOFOZcJLVnnuBYjUltmPHDm3casrSamrNKs4emggkq/tsxowZjlx/1qxZ2jh7Yl4fVswAAAAMQWMGAABgCBozAAAAQ9CYAQAAGILGDAAAwBDGTGW2aNFCG8/MzNTGBw0a5M90pKCgQBvPy8vTxl0ulzZuNU35008/aePLli3TxtPS0rTxXr16aeNAdbA7pegUq2kwf087Wk2b+vv5Atdidf85NX1ptccl05f+wYoZAACAIWjMAAAADEFjBgAAYAgaMwAAAEPQmAEAABjCmKnMiIgIbdzu9OWZM2e08YMHD2rjf/rTn7TxTz/9VBvfvXu3rXycsn//fm18586d1ZwJ8LOaOn1pxWpvTbtTceyVCSc5tTet1ZQl05fVixUzAAAAQ9CYAQAAGILGDAAAwBA0ZgAAAIagMQMAADCEMVOZFy5c0Ma/++47bdxqL8uPP/5YG584cWLVEgMQMB6PJ9ApeLGbj9VUKVAVTk3zWk1ZWu2JacUqH6Y7rw8rZgAAAIagMQMAADAEjRkAAIAhaMwAAAAMQWMGAABgCGOmMo8cOaKN9+rVSxu32hNTKeVYTgCuzWoqy2p60e7emlbHW9W5U3trWj2u3fyZQoOTrPZetWJ1/9mdFrbai9PuHrFWrKZBa2v9sGIGAABgCBozAAAAQ9CYAQAAGILGDAAAwBA0ZgAAAIYwZirTyuHDhwOdghHGjRsX6BQAn1lNWfl7atpqGsxqSnTHjh22jrdiNeVWW6fKcH2spiDtcuq+tDuNbJfV83VqT89gw4oZAACAIWjMAAAADEFjBgAAYAgaMwAAAEPQmAEAABjCpXwYkzpz5oxERkZWRz6wsHHjRm18wIABtq7Trl07bXz//v22czJJQUGBNGrUKNBp2FIb68pqusupKbRAcblcgU7BL6irwHBqetnf92Wg9qYO9nqrrK5YMQMAADAEjRkAAIAhaMwAAAAMQWMGAABgCBozAAAAQxi/VyaqxmrKsqCgoJozAX5mtfed1Z5+VntfBkpN36MPsMOqHqymr02rZ1OxYgYAAGAIGjMAAABD0JgBAAAYgsYMAADAEDRmAAAAhmAq0zCTJk3Sxvv27WvrOi+++KI2fuLECds5Af42c+ZMW8cz3QUEntWUtVXcit16tvp5YffniKlYMQMAADAEjRkAAIAhaMwAAAAMQWMGAABgCBozAAAAQ7iUUqqyg86cOSORkZHVkU+tERUVpY3v2rVLG09ISLB1/S5dumjju3fvtnWdYFFQUCCNGjUKdBq2UFc/s9pbb9u2bdWbSBW5XK5Ap+AX1FVgOFUPVtORO3bs0MZNm2r0oT3xYvV8TdvjtrK6YsUMAADAEDRmAAAAhqAxAwAAMASNGQAAgCFozAAAAAzBXpkB0rp1a228ffv22rjVdMrXX3+tjR89erRqiQF+5O/py1mzZmnjVtNaTj2u1XVMmwZDcLC7B6VVXdmNW+1ZaVVXVuzm6RSradNgw4oZAACAIWjMAAAADEFjBgAAYAgaMwAAAEPQmAEAABiCvTL97IYbbtDGP/nkE208Li7O1vWzsrK08QcffNDWdYIde/oFB7t731mxmna0mgaz4lQ+VoJ9D03qKjj4+z4OFk79XPA39soEAAAIEjRmAAAAhqAxAwAAMASNGQAAgCFozAAAAAzBXpl+NmrUKG3c7vRlUVGRNr5o0SLbOQH+5tSUmN29L+3uAQjUBFbTv1Z7uPp7z0p/s/tzIdiwYgYAAGAIGjMAAABD0JgBAAAYgsYMAADAEDRmAAAAhmAq0yFut1sb/+d//mdHrj99+nRtfPfu3Y5cHzCR1TSlaVOWVnv0AYFkdV/anV4O1BSn1ZTlzJkzqzWP6saKGQAAgCFozAAAAAxBYwYAAGAIGjMAAABD0JgBAAAYgqlMh0yYMEEbHzp0qK3r5OTkaON/+ctfbOcEBIrdvSxNY5V/Td+jD7WD1f3q1BSkx+PRxnfs2OFIPjUdK2YAAACGoDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAiXUkpVdtCZM2ckMjKyOvIJWsePH9fGmzVrpo2fPXtWG3/wwQe18fXr11ctsVqioKBAGjVqFOg0bKmNdbVt2zZt3KlpTaspLqtpsJq+5971oq4A51VWV6yYAQAAGILGDAAAwBA0ZgAAAIagMQMAADAEjRkAAIAh2CvTpk6dOmnjzZs318athl6///57bZzpS9Rkffr0CXQKAGA0VswAAAAMQWMGAABgCBozAAAAQ9CYAQAAGILGDAAAwBBMZdo0Y8YMW8cXFRVp4+np6U6kAwAAahBWzAAAAAxBYwYAAGAIGjMAAABD0JgBAAAYgsYMAADAEExl2tSvXz9bx2/atEkb37ZtmxPpAACAGoQVMwAAAEPQmAEAABiCxgwAAMAQNGYAAACGoDEDAAAwBFOZNkVERAQ6BQAAUEOxYgYAAGAIGjMAAABD0JgBAAAYgsYMAADAED41Zkopf+cBXJdgvEeDMWfULsF4jwZjzqhdKrtHfWrMCgsLHUkG8JdgvEeDMWfULsF4jwZjzqhdKrtHXcqHXy9KS0slPz9fIiIixOVyOZYccL2UUlJYWCjR0dFSp05w/WWeuoKpqCvAeb7WlU+NGQAAAPwvuH4VAgAAqMFozAAAAAxBYwYAAGAIGjMAAABD0JgBAAAYgsYMAADAEDRmAAAAhqAxAwAAMASNGQAAgCFozAAAAAxBYwYAAGAIGjMAAABD0JgBAAAYgsYMAADAEDRmAAAAhqAxAwAAMASNGQAAgCFozAAAAAxBYwYAAGAIGjMAAABD0JjVAL1795bevXsHOg2gxqCmAOdRV75xpDFzuVw+/du+fbsTD+eo7du3XzPnefPm2To/JCREbr75ZklNTZWDBw9W07Ooury8vPLc16xZU+HzM2fOFJfLJT/++GMAsqu9grmmRETOnj0rU6ZMkdatW0toaKgkJibKn/70J5/OpabgL8FeV7904MABCQsLE5fLJZ999lmlx1NXwaOuExdZtWqV18evvvqqbN68uUI8MTHRiYdzVGJiYoU8Ra48p/fee0/69+/v03V+97vfye233y6XLl2SL774QjIzM2XDhg2yZ88eiY6Odjptv5g9e7YMHTpUXC5XoFOp9YK5pkpKSmTAgAHy2WefyeTJkyU+Pl42bdokDz/8sJw6dUqeeuopn65DTcFpwVxXV3vsscekbt26cvHiRVvnUVdBQPnB5MmTlS+XLioq8sfDOyIuLk7Fx8dXety2bduUiKjVq1d7xV944QUlIurZZ5+1PPfs2bPXnadSSnk8HuXxeKp0bm5urhIR1blzZyUias2aNV6fnzFjhhIRdeLECQcyRVUFU03993//txIR9fLLL3vFhw0bpsLCwtTx48eveT41heoSTHX1S++++66qV6+eevrpp5WIqE8//bTSc6ir4FFtrzHr3bu3dOzYUT7//HPp1auXhIeHl//m7HK5ZObMmRXOadu2raSlpXnFTp8+LVOmTJGbbrpJQkNDJS4uThYsWCClpaVexx07dkz27t0rly5dsp3rJ598Ivv375fRo0fbPrfM3XffLSIiubm5IvLzMus333wj999/vzRu3Fh69OhRfvxrr70mSUlJUr9+fWnSpImkpKTI4cOHK1w3MzNTYmNjpX79+tKtWzfZtWuX9vEPHToke/fu9TnflJQUadeuncyePVuUUpUev3r16vJ8o6Ki5IEHHpCjR496HZOWliYNGzaUo0ePypAhQ6Rhw4bSrFkzmTp1qpSUlHgdW1paKkuWLJEOHTpIWFiYtGjRQtLT0+XUqVM+P4faxtSaKrsnU1JSvOIpKSly4cIFWb9+vc1negU1RU1VB1PrqsylS5fk0UcflUcffVRiY2Or9Bx/iboyr66q9cX/J0+elHvuuUc6d+4sS5YskT59+tg6/9y5c+LxeOS1116T1NRUeeGFF6R79+6SkZEhjz/+uNexGRkZkpiYWOEb4IusrCwRketqzA4cOCAiIk2bNvWKjxgxQs6dOyfPPvusjB8/XkRE5s2bJ6mpqRIfHy+LFy+WKVOmyNatW6VXr15y+vTp8nNffvllSU9Pl5YtW8pzzz0n3bt3l8GDB2uLIjU11dZyvNvtlqefflq+/PJLWbdu3TWPXblypSQnJ4vb7Zb58+fL+PHjZe3atdKjRw+vfEV+/rNW06ZNZeHCheLxeGTRokWSmZnpdVx6erpMmzZNunfvLkuXLpWxY8dKVlaWDBgwoErNdW1hYk1dvHhR3G631KtXzyseHh4uIiKff/65rRzLUFNXUFP+Z2JdlVmyZImcOnVKnn76aVs5WaGurjCqrvyxDKdbHvZ4PEpE1LJlyyocLyJqxowZFeJt2rRRY8aMKf94zpw5qkGDBuof//iH13FPPvmkcrvd6tChQ+WxMWPGKBFRubm5tnK/fPmyatGiherWrZtPx5ctDy9fvlydOHFC5efnqw0bNqi2bdsql8tVvsRctsw6atQor/Pz8vKU2+1W8+bN84rv2bNH1a1btzxeXFysmjdvrjp37qwuXrxYflxmZqYSkQrLw2Vf78qULQ8///zz6vLlyyo+Pl516tRJlZaWeuVdtjxclkfHjh3V+fPny6/zzjvvKBFRzzzzTHms7Hswe/Zsr8fs0qWLSkpKKv94165dSkRUVlaW13HvvvuuNl4bBVNNLVq0SImI2rVrV4Vrioi69957r3k+NXUFNeV/wVRXSil17NgxFRERof7zP/9TKaXUihUrbP8pk7oyv66qdcUsNDRUxo4dW+XzV69eLT179pTGjRvLjz/+WP6vb9++UlJSIjt37iw/duXKlaKUkrZt29p6jK1bt8rx48dtr5Y99NBD0qxZM4mOjpaBAwdKUVGRvPLKK9K1a1ev4yZOnOj18dq1a6W0tFSSk5O9nlPLli0lPj5etm3bJiIin332mfzwww8yceJEr5WItLQ0iYyMrJDP9u3bfVrm/aVf/iby9ttva48py+Phhx+WsLCw8vjAgQMlISFBNmzYUOGcq59zz549vaaAVq9eLZGRkdKvXz+vr0FSUpI0bNiw/GuAikysqfvvv18iIyPloYceks2bN0teXp5kZmbKSy+9JCIi58+f9yk3aoqaChQT60pE5IknnpCbb75Zxo0bV+XcqCvz68qRqUxftWrVqsKfN+zIycmRv//979KsWTPt53/44YcqX7tMVlaWuN1uGTlypK3znnnmGenZs6e43W6JioqSxMREqVu34pc3JibG6+OcnBxRSkl8fLz2uiEhISIi8t1334mIVDiubOTZKaNHj5Y5c+bI7NmzZciQIRU+X5ZH+/btK3wuISFB3n//fa9YWFhYhe9X48aNvf4en5OTIwUFBdK8eXNtTk58X2sqE2uqZcuWkp2dLQ8++GD5VHOjRo3kP/7jP2TMmDHSsGFDn65DTVFTgWJiXX388ceyatUq2bp1q9SpU/U1FerK/Lqq1sasfv36to7XveiuX79+Mn36dO3x7dq1q3JuIld+k1+3bp307dtXWrRoYevcW2+9Vfr27VvpcVd/DUpLS8XlcsnGjRvF7XZXON7X/8ScUvabSFpaWpVfpH319SpTWloqzZs3L39t39WsfrjB3Jrq1auXHDx4UPbs2SNFRUXSqVMnyc/Pt3VNasr6epWhpq6PiXU1ffp06dmzp8TExEheXp6ISPl7dh07dkwOHTokv/rVryq9DnVlfb3KVFddVWtjZqVx48YVXohXXFwsx44d84rFxsbK2bNnfbqpqiI7O1sKCwuv60X/dsXGxopSSmJiYq5ZrG3atBGRKx172RSNyJUJndzcXOnUqZNjOT3wwAMyd+5cmTVrlgwePFibx759+7zyKIuVfd6O2NhY2bJli3Tv3t32D0TomVBTbrdbOnfuXP7xli1bRET8Vr9lqClqyl8CWVeHDh2S7777rsJKlojI4MGDJTIyskJuTqKuqq+ujNiSKTY21utv7iJXRm2v/i0kOTlZPvroI9m0aVOFa5w+fVouX75c/nFV3i7j9ddfl/DwcLnvvvtsPoOqGzp0qLjdbpk1a1aFv7MrpeTkyZMiItK1a1dp1qyZLFu2TIqLi8uPWblypbYY7Y4g/1LZbyK7d++W7Oxsr8917dpVmjdvLsuWLfN6Y8ONGzfKt99+KwMHDrT9eMnJyVJSUiJz5syp8LnLly/79YdNTWVKTZU5ceKELFiwQP7pn/7J740ZNUVN+Usg6yozM1PWrVvn9e+RRx4REZGFCxdaruI4hbqqvroyYsVs3LhxMnHiRBk2bJj069dPvvzyS9m0aZNERUV5HTdt2jTJzs6We++9V9LS0iQpKUmKiopkz5498tZbb0leXl75ORkZGfLKK69Ibm6uTy+q/Omnn2Tjxo0ybNiwal2SjY2Nlblz50pGRobk5eXJkCFDJCIiQnJzc2XdunUyYcIEmTp1qoSEhMjcuXMlPT1d7r77bhk5cqTk5ubKihUrtH+3T01NlR07dth+UWWZsr/f79692yseEhIiCxYskLFjx4rH45FRo0bJ8ePHZenSpdK2bVt57LHHbD+Wx+OR9PR0mT9/vuzevVv69+8vISEhkpOTI6tXr5alS5fK8OHDq/Q8aqtA15TH45E777xT4uLi5Pvvv5fMzEw5e/asvPPOO9f1+hhfUFPUlL8Esq50u9CUNQIej6fCi/edRl1VY105Mtt5FasR5A4dOmiPLykpUU888YSKiopS4eHhasCAAWr//v0VRpCVUqqwsFBlZGSouLg4Va9ePRUVFaXuuusutXDhQlVcXFx+nN23y1i2bJkSEZWdnW3ruVq9m/LVKntX4jVr1qgePXqoBg0aqAYNGqiEhAQ1efJktW/fPq/jXnrpJRUTE6NCQ0NV165d1c6dO7XvplyVEeSrlY1i6/J+8803VZcuXVRoaKhq0qSJGj16tDpy5IjXMWPGjFENGjSw/FpcLTMzUyUlJan69euriIgIdeutt6rp06er/Pz8Sp9HTRdsNfXYY4+pm2++WYWGhqpmzZqp+++/Xx04cMCn50pNUVPVJdjq6mpVebsM6sr8unIpVcU2FQAAAI4y4jVmAAAAoDEDAAAwBo0ZAACAIWjMAAAADEFjBgAAYAgaMwAAAEP49AazpaWlkp+fLxEREeJyufydE+AzpZQUFhZKdHS039+41GnUFUxFXQHO87WufGrM8vPz5aabbnIsOcBphw8fltatWwc6DVuoK5iOugKcV1ld+fSrUEREhGMJAf4QjPdoMOaM2iUY79FgzBm1S2X3qE+NGcvBMF0w3qPBmDNql2C8R4MxZ9Quld2jwfXiAQAAgBqMxgwAAMAQNGYAAACGoDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAgaMwAAAEPQmAEAABiCxgwAAMAQNGYAAACGoDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAgaMwAAAEPQmAEAABiCxgwAAMAQNGYAAACGoDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAgaMwAAAEPQmAEAABiCxgwAAMAQdQOdAAAAwNWGDx+ujb/66qva+C233KKN5+XlOZVStWDFDAAAwBA0ZgAAAIagMQMAADAEjRkAAIAhaMwAAAAMwVSmhb59+2rjI0eO1MYfeughbbxOHX3v+/3332vja9eu1cZXrVqljf/f//2fNn7x4kVtHKgO0dHR2vjmzZu18cTERFvX37Rpkza+c+dObXzRokXaeHFxsa3HBapDXFycNn706FFt/Pz58/5Mx++spimffPJJbfzUqVPaeMuWLbVxpjIBAABQJTRmAAAAhqAxAwAAMASNGQAAgCFozAAAAAxRa6Yyb7zxRm3carqrbdu22viGDRu08ZSUFG3822+/1catpkeGDh2qjX/wwQfaeExMjDZ+6NAhbRxwUnJysjb++9//XhtPSEjQxpVSth63f//+2viAAQO08YyMDFuP+/XXX2vjCxYs0MbXr1+vjQNVsX///kCnUK0GDx6sjXfp0kUb/5d/+Rdt/OOPP3Ysp0BixQwAAMAQNGYAAACGoDEDAAAwBI0ZAACAIWjMAAAADOFSPoxDnTlzRiIjI6sjH79p0KCBNm41DfLWW29p45cuXXIsJ50bbrhBGz958qQ2zlTmFQUFBdKoUaNAp2FLMNWV1ZTlv/3bv2njISEhtq5vNXV84MABW9cZM2aMNm536tOK1R60zz33nDa+fPlybTxY6pO6Cm5W9+XBgwe18WXLlvkzHct3R7Cq/yZNmmjjSUlJ2rjdnxeBUlldsWIGAABgCBozAAAAQ9CYAQAAGILGDAAAwBA0ZgAAAIaoNXtlFhUVaeNvvPFGNWcCBJ/Ro0dr43anL//6179q41bTlAUFBbau/9NPP2njU6ZMsXWdLVu2aONdu3bVxq2mUwcNGqSN33fffdp4sExrwixWe9COHTtWGz9y5Ig27u+pzJEjR2rjbdq00canTp2qjQfL9GVVsWIGAABgCBozAAAAQ9CYAQAAGILGDAAAwBA0ZgAAAIaoNVOZACr36KOPauNWU1NWsrOztfGHHnpIG7c7fWnlxx9/tHX8//zP/2jjycnJ2vh//dd/aeMpKSnaeOfOnbVxq6/DzJkztXHgWvr06aONN23aVBtfvHixP9OxNGzYMG3c5XJp43v37vVnOsZixQwAAMAQNGYAAACGoDEDAAAwBI0ZAACAIWjMAAAADMFUpmEmTJigje/YsUMbP3r0qD/TQQ01ZMgQbfwPf/iDNl6vXj1t/IsvvtDGf/vb32rjp06dqjy569C6dWtbx1tNcZ4/f14bf+SRR7Tx9u3ba+NdunTRxpnKRFW0aNFCG3/qqae0caWUNm61p6xT4uLitHGrvWbXrl2rjW/dutWxnIIJK2YAAACGoDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAimMg2TlpamjWdmZmrjJSUlfswGNZXV9KLV9KUVqylLf099WZk0aZI2bjWdlpWVZev6Vs/rj3/8ozb+8ssva+NWexjec8892vjGjRt9yA413W9+8xttvFWrVtr4V199pY2vWrXKsZx0rKYvQ0JCtPF9+/Zp48XFxY7lFExYMQMAADAEjRkAAIAhaMwAAAAMQWMGAABgCBozAAAAQzCVGSB33HGHNm61F9qLL77oz3SAoJKcnGzr+C1btmjjH374oRPpyDfffGPr+LCwMG385ptvdiIdBLnY2FhtfMmSJdq41fTiuHHjtPFz585VKa+rRUZGauOLFy/Wxk+ePKmN//nPf3Ykn5qCFTMAAABD0JgBAAAYgsYMAADAEDRmAAAAhqAxAwAAMARTmQEyZMgQbXz//v3a+KVLl/yYDWAmq6mvadOm2brOwoULtXGnptPs7jEKXMugQYO08dDQUG387Nmztq5jNcW5e/fuypP7hR49emjjVu8u8Prrr2vjeXl5th63pmPFDAAAwBA0ZgAAAIagMQMAADAEjRkAAIAhaMwAAAAMwVSmn7ndbm38N7/5jTaek5Pjz3QAI1lNX1rtcXnbbbfZuv7HH39sOyc7ZsyYYev4M2fOaOMfffSRE+kgyCUlJWnjLpdLG2/UqJE2/vvf/95W3C6rfJRS2njv3r1t5fPFF19o4xs3bqw8uSDGihkAAIAhaMwAAAAMQWMGAABgCBozAAAAQ9CYAQAAGIKpTD/r2LGjNh4fH6+NP/fcc/5MBxARkc8++0wbLyoq0sYbNmyojfft21cbLy0trVpi1+nXv/61Nl5YWOjXx7WaTqtTR/+7r9XehlZTaKhdZs6cqY1/9dVX2nj9+vW18T59+jiSz+23366Nh4WFaeNWU5nR0dHa+IQJE7TxN954QxtnKhMAAADVgsYMAADAEDRmAAAAhqAxAwAAMASNGQAAgCGYyvSzu+66Sxv/8MMPtfEVK1b4Mx1ARKz3jrSaaly/fr02fsMNNziVklZxcbE2PmLECG383Xff9Wc6kpGRoY336tVLG7eaTmVPTFzLgQMHtPEFCxbYuo7VdKddf/jDH7Tx6dOna+NvvvmmNj5p0iRt/OLFi9r4hQsXfMiu5mHFDAAAwBA0ZgAAAIagMQMAADAEjRkAAIAhaMwAAAAMwVSmQ9xutzbeuXNnbdxq2gwIpPfff18bz8zM1MZbtmypjffv318b37t3rzZutQeg1TTYsWPHtHF/GzJkiDZuVf8FBQXa+NKlS51KCfC7lJQUbfz48ePa+Pjx47Vxq7144Y0VMwAAAEPQmAEAABiCxgwAAMAQNGYAAACGoDEDAAAwBFOZDunUqZM2Pm7cOG388ccf18YbNGigjTPNgkCy2iPSyk033aSNnz59WhsvLCy0m5IjfvWrX2njqamp2njHjh1tXX/Lli3a+AcffGDrOkB1SEhI0Majo6O18U2bNmnj/H91fVgxAwAAMASNGQAAgCFozAAAAAxBYwYAAGAIGjMAAABDMJXpEKu9xEpKSrTxo0ePauM33nijNr5///6qJQYEwOHDhwOdgherKdGHH35YG582bZqt63/zzTfaeHp6uq3rAIFk9f9Y3br6VuH111/3Zzq1FitmAAAAhqAxAwAAMASNGQAAgCFozAAAAAxBYwYAAGAIpjIdYrXHpdVeYlZ77r311luO5QTUNlZ7+r3zzjvauN29L62mL+fOnauNnzp1ytb1gUCyqgellDb+t7/9zZ/p1FqsmAEAABiCxgwAAMAQNGYAAACGoDEDAAAwBI0ZAACAIZjKdIjL5dLGW7VqpY0XFxf7Mx2gRnC73dr4I488oo2PHz9eG09ISLD1uIWFhdr4qFGjtPGvvvrK1vWBQAoPD9fG27Vrp41/+umn2vjx48cdywk/Y8UMAADAEDRmAAAAhqAxAwAAMASNGQAAgCFozAAAAAzBVKZDfv3rX2vjVnv3WR0P1EZW05eTJ0/WxhctWqSN16mj/12ztLRUGz937pw2Pm7cOG2c6UvUBImJidp4hw4dtPHbbrvNn+ngKqyYAQAAGILGDAAAwBA0ZgAAAIagMQMAADAEjRkAAIAhmMp0iNX05fLly7Xx77//3p/pAEGlRYsW2vhzzz3nyPU3b96sjT///PPa+NatWx15XMBEVntiWjl16pSfMoEOK2YAAACGoDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAimMm0aOHCgNm41tfLnP//Zn+kANUJ+fr42PnjwYG38ySef1Matpi+t9tYsLi72ITugZnnjjTdsxVG9WDEDAAAwBI0ZAACAIWjMAAAADEFjBgAAYAgaMwAAAEO4lFKqsoPOnDkjkZGR1ZEPUCUFBQXSqFGjQKdhC3UF01FXgPMqqytWzAAAAAxBYwYAAGAIGjMAAABD0JgBAAAYgsYMAADAEDRmAAAAhqAxAwAAMASNGQAAgCFozAAAAAxBYwYAAGAInxozH3ZtAgIqGO/RYMwZtUsw3qPBmDNql8ruUZ8as8LCQkeSAfwlGO/RYMwZtUsw3qPBmDNql8ruUZ82MS8tLZX8/HyJiIgQl8vlWHLA9VJKSWFhoURHR0udOsH1l3nqCqairgDn+VpXPjVmAAAA8L/g+lUIAACgBqMxAwAAMASNGQAAgCFozAAAAAxBYwYAAGAIGjMAAABD0JgBAAAY4v8Br5rU9hGNILYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_img_label_prediction(imgs, y_true, y_pred=None, shape=(2, 3)):\n",
    "    y_pred = [None] * len(y_true) if y_pred is None else y_pred\n",
    "    fig = plt.figure()\n",
    "    for i in range(np.prod(shape)):\n",
    "        plt.subplot(*shape, i+1)\n",
    "        plt.tight_layout()\n",
    "        plt.imshow(imgs[i][0], cmap='gray', interpolation='none')\n",
    "        plt.title(\"True: {} Pred: {}\".format(y_true[i], y_pred[i]))\n",
    "        plt.xticks([])\n",
    "        plt.yticks([])\n",
    "\n",
    "plot_img_label_prediction(imgs=example_imgs, y_true=example_targets, y_pred=None, shape=(2, 3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mj3utDDuzDCj"
   },
   "source": [
    "## Question 3.1.1\n",
    "### Logistic Regression\n",
    "\n",
    "We can use a very simple Logistic Regression that receives our input images as a vector and predicts the digit. This will be our first baseline to compare with the CNNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TniyY4bQzBMS",
    "outputId": "5aa11ea8-84a6-4757-9ac8-7e950fc3f1f9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score with penalty: 0.9037\n"
     ]
    }
   ],
   "source": [
    "# Use standard scaler to transform input data\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(np.reshape(X_train, (X_train.shape[0], -1)))\n",
    "X_val = scaler.transform(np.reshape(X_val, (X_val.shape[0], -1)))\n",
    "\n",
    "# Use scikit-learn Logistic Regression with C=50., multi_class='multinomial', solver='sag', tol=0.1\n",
    "clf = LogisticRegression(C=50., multi_class='multinomial', solver='sag', tol=0.1)\n",
    "clf.fit(X_train, y_train)\n",
    "score = clf.score(X_val, y_val)\n",
    "\n",
    "print(\"Test score with penalty: %.4f\" % score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A8rylkCnrwIy"
   },
   "source": [
    "We can select the coefficients for each class and reshape them into the image shape to plot them. This allows us to visualize what are the pixels that are contributing more to the classification for each of the digits.\n",
    "\n",
    "But what happens if the digits are not centered? Will we still get such a good performance? Lets test that out later!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kK8v34AK6xXJ"
   },
   "source": [
    "## Question 3.1.2\n",
    "### Feed-Forward Neural Network\n",
    "\n",
    "The first step is to create the functions that will allow us to implement a feed-forward neural network and manage the training and validation process.\n",
    "\n",
    "The MLP class will define the architecture of a feed-forward neural network, with a set of hidden layers (fully connected layers [nn.Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)), with a activation function in between them ([relu](https://pytorch.org/docs/stable/generated/torch.nn.functional.relu.html#torch.nn.functional.relu)), and a [softmax](https://pytorch.org/docs/stable/generated/torch.nn.functional.log_softmax.html#torch.nn.functional.log_softmax) in the last layer. Since the dataset poses a multiclass classification problem, the last layer should have a number of neurons equal to the number of classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "In9r_o8vvNaz"
   },
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "    def __init__(self, dim_layers):\n",
    "        super(MLP, self).__init__()\n",
    "        # define layers\n",
    "        self.dim_layers = dim_layers\n",
    "        layer_list = [nn.Linear(dim_layers[l], dim_layers[l+1]) for l in range(len(dim_layers) - 1)]\n",
    "        self.lin_layers = nn.ModuleList(layer_list)\n",
    "\n",
    "    def forward(self, X):\n",
    "        # reshape X\n",
    "        X = X.view(-1, self.dim_layers[0])\n",
    "        # apply relu\n",
    "        for layer in self.lin_layers[:-1]:\n",
    "            X = F.relu(layer(X))\n",
    "        # use softmax for output layer\n",
    "        return F.log_softmax(self.lin_layers[-1](X), dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h6OVD_1xUwWH"
   },
   "source": [
    "##### training validation function for the MLP and CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "B1eUu01N8wIR"
   },
   "outputs": [],
   "source": [
    "def train_val_model(model, criterion, optimizer, dataloaders, num_epochs=25,\n",
    "        scheduler=None, log_interval=None):\n",
    "    since = time.time()\n",
    "\n",
    "    best_model_wts = copy.deepcopy(model.state_dict())\n",
    "    best_acc = 0.0\n",
    "\n",
    "    # init dictionaries to save losses and accuracies of training and validation\n",
    "    losses, accuracies = dict(train=[], val=[]), dict(train=[], val=[])\n",
    "\n",
    "    for epoch in range(num_epochs):\n",
    "        if log_interval is not None and epoch % log_interval == 0:\n",
    "            print('Epoch {}/{}'.format(epoch, num_epochs - 1))\n",
    "            print('-' * 10)\n",
    "\n",
    "        # execute a training and validation phase for each epoch\n",
    "        for phase in ['train', 'val']:\n",
    "            if phase == 'train':\n",
    "                model.train()  # set model to train mode\n",
    "            else:\n",
    "                model.eval()   # Set model to eval mode\n",
    "\n",
    "            running_loss = 0.0\n",
    "            running_corrects = 0\n",
    "\n",
    "            # iterate over the data\n",
    "            nsamples = 0\n",
    "            for inputs, labels in dataloaders[phase]:\n",
    "                inputs = inputs.to(device)\n",
    "                labels = labels.to(device)\n",
    "                nsamples += inputs.shape[0]\n",
    "\n",
    "                # set the parameter gradients to zero\n",
    "                optimizer.zero_grad()\n",
    "\n",
    "                with torch.set_grad_enabled(phase == 'train'):\n",
    "                    outputs = model(inputs)\n",
    "                    _, preds = torch.max(outputs, 1)\n",
    "                    loss = criterion(outputs, labels)\n",
    "\n",
    "                    # if in training phase, perform backward prop and optimize\n",
    "                    if phase == 'train':\n",
    "                        loss.backward()\n",
    "                        optimizer.step()\n",
    "\n",
    "                # increment loss and correct counts\n",
    "                running_loss += loss.item() * inputs.size(0)\n",
    "                running_corrects += torch.sum(preds == labels.data)\n",
    "\n",
    "            if scheduler is not None and phase == 'train':\n",
    "                scheduler.step()\n",
    "\n",
    "            epoch_loss = running_loss / nsamples\n",
    "            epoch_acc = running_corrects.double() / nsamples\n",
    "\n",
    "            losses[phase].append(epoch_loss)\n",
    "            accuracies[phase].append(epoch_acc)\n",
    "            if log_interval is not None and epoch % log_interval == 0:\n",
    "                print('{} Loss: {:.4f} Acc: {:.2f}%'.format(\n",
    "                    phase, epoch_loss, 100 * epoch_acc))\n",
    "\n",
    "            # deep copy the best model\n",
    "            if phase == 'val' and epoch_acc > best_acc:\n",
    "                best_acc = epoch_acc\n",
    "                best_model_wts = copy.deepcopy(model.state_dict())\n",
    "        if log_interval is not None and epoch % log_interval == 0:\n",
    "            print()\n",
    "\n",
    "    time_elapsed = time.time() - since\n",
    "    print('Training complete in {:.0f}m {:.0f}s'.format(\n",
    "        time_elapsed // 60, time_elapsed % 60))\n",
    "    print('Best val Acc: {:.2f}%'.format(100 * best_acc))\n",
    "\n",
    "    # load best model weights to return\n",
    "    model.load_state_dict(best_model_wts)\n",
    "\n",
    "    return model, losses, accuracies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0CBE5tRMZEfr"
   },
   "source": [
    "We will start by creating a simple network with some hidden layers. Thus, in addition to the input, it will have 3 fully connected layer which, in this implemetation, is assigned to the input of the MLP Class. We will use the Stochastic Gradient Descend optimizer ([optim.SGD](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html)) with 0.01 learning rate and 0.5 momentum. The loss function to be optimized will be negative log likelihood ([nn.NLLLoss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html)). Training and validation will be managed by the function \"train_val_model\" previously define."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "200WI3xND6_M",
    "outputId": "173c83bc-e0bf-4ed6-b7fc-b44ad318387d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/14\n",
      "----------\n",
      "train Loss: 0.8482 Acc: 75.17%\n",
      "val Loss: 0.3036 Acc: 91.35%\n",
      "\n",
      "Epoch 2/14\n",
      "----------\n",
      "train Loss: 0.1791 Acc: 94.85%\n",
      "val Loss: 0.1539 Acc: 95.42%\n",
      "\n",
      "Epoch 4/14\n",
      "----------\n",
      "train Loss: 0.1086 Acc: 96.78%\n",
      "val Loss: 0.1077 Acc: 96.65%\n",
      "\n",
      "Epoch 6/14\n",
      "----------\n",
      "train Loss: 0.0747 Acc: 97.84%\n",
      "val Loss: 0.0886 Acc: 97.15%\n",
      "\n",
      "Epoch 8/14\n",
      "----------\n",
      "train Loss: 0.0531 Acc: 98.47%\n",
      "val Loss: 0.0768 Acc: 97.56%\n",
      "\n",
      "Epoch 10/14\n",
      "----------\n",
      "train Loss: 0.0383 Acc: 98.90%\n",
      "val Loss: 0.0772 Acc: 97.54%\n",
      "\n",
      "Epoch 12/14\n",
      "----------\n",
      "train Loss: 0.0286 Acc: 99.22%\n",
      "val Loss: 0.0731 Acc: 97.85%\n",
      "\n",
      "Epoch 14/14\n",
      "----------\n",
      "train Loss: 0.0208 Acc: 99.48%\n",
      "val Loss: 0.0731 Acc: 97.86%\n",
      "\n",
      "Training complete in 4m 13s\n",
      "Best val Acc: 97.98%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA490lEQVR4nO3deXhU9d3+8XuyQ0zCEsgCgQQFQVYFQVzqlpJWhVKroiLww1YfLSqS1kdQIY9ViUu1VKFQaKltLZVq1boglqaIWlEoESuKIBIggNkEkhAkCZP5/fHtZCbJJJlJZuYkmffrus41Z07OnPlMipm73+3YHA6HQwAAABYJs7oAAAAQ2ggjAADAUoQRAABgKcIIAACwFGEEAABYijACAAAsRRgBAACWIowAAABLRVhdgDfq6up0+PBhxcXFyWazWV0OAADwgsPhUGVlpVJTUxUW1nz7R6cII4cPH1ZaWprVZQAAgDYoLCxU//79m/15pwgjcXFxksyHiY+Pt7gaAADgjYqKCqWlpdV/jzenU4QRZ9dMfHw8YQQAgE6mtSEWDGAFAACWIowAAABLEUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEsRRgAAgKUIIwAAwFIhHUaWLpVuuUXas8fqSgAACF0hHUaee076zW+k//zH6koAAAhdIR1GBg0yj3v3WlsHAAChLKTDyOmnm0fCCAAA1gnpMOJsGfnyS2vrAAAglBFGRMsIAABWCukw4uym2bdPststLQUAgJAV0mEkNVWKipJOnZIKC62uBgCA0BTSYSQsTMrIMPt01QAAYI2QDiMSM2oAALBayIcRZtQAAGAtwggzagAAsFTIhxFnNw0tIwAAWCPkwwgtIwAAWCvkw4hzNs3Ro2YDAADBFfJhJDZWSk42+7SOAAAQfCEfRiS6agAAsFKbwsiyZcuUnp6umJgYTZgwQVu2bGnx/CVLlujMM89Ut27dlJaWpnnz5unkyZNtKjgQCCMAAFjH5zCydu1aZWdnKycnR/n5+Ro9erSysrJUUlLi8fw1a9Zo/vz5ysnJ0c6dO/Xb3/5Wa9eu1X333dfu4v2FGTUAAFjH5zDy1FNP6ZZbbtHs2bN11llnacWKFerevbtWr17t8fz3339fF1xwgW688Ualp6dr0qRJuuGGG1ptTQkmWkYAALCOT2GkpqZG27ZtU2ZmpusCYWHKzMzU5s2bPb7m/PPP17Zt2+rDx969e7Vu3TpdccUVzb5PdXW1KioqGmyBxJLwAABYJ8KXk8vKymS325WUlNTgeFJSkj7//HOPr7nxxhtVVlamCy+8UA6HQ6dOndJtt93WYjdNbm6uHnzwQV9Kaxdny8iBA1JtrRQZGbS3BgAg5AV8Ns3bb7+txYsX61e/+pXy8/P10ksv6Y033tBDDz3U7GsWLFig8vLy+q2wsDCgNSYnS926SXa7CSQAACB4fGoZSUxMVHh4uIqLixscLy4uVrJzsY5GFi5cqBkzZuhHP/qRJGnkyJGqqqrSrbfeqvvvv19hYU3zUHR0tKKjo30prV1sNtM68umnpqvG2W0DAAACz6eWkaioKI0dO1Z5eXn1x+rq6pSXl6eJEyd6fM2JEyeaBI7w8HBJksPh8LXegOHuvQAAWMOnlhFJys7O1qxZszRu3DiNHz9eS5YsUVVVlWbPni1Jmjlzpvr166fc3FxJ0uTJk/XUU0/p7LPP1oQJE7Rnzx4tXLhQkydPrg8lHQEzagAAsIbPYWTatGkqLS3VokWLVFRUpDFjxmj9+vX1g1oPHDjQoCXkgQcekM1m0wMPPKBDhw6pT58+mjx5sh555BH/fQo/YK0RAACsYXN0pL6SZlRUVCghIUHl5eWKj48PyHu88YZ01VXSmDHSRx8F5C0AAAgp3n5/c2+a/3Lvpun48QwAgK6DMPJfGRnmsaJC+vpra2sBACCUEEb+KyZG6tfP7DOIFQCA4CGMuGFGDQAAwUcYccOMGgAAgo8w4oaWEQAAgo8w4oYwAgBA8BFG3NBNAwBA8BFG3DhbRg4elKqrra0FAIBQQRhx06ePFBtrFj3bv9/qagAACA2EETc2G101AAAEG2GkEQaxAgAQXISRRpwtI4QRAACCgzDSiLNlhG4aAACCgzDSCN00AAAEF2GkEfduGofD2loAAAgFhJFGBg40s2qqqqSSEqurAQCg6yOMNBIVJaWlmX26agAACDzCiAesNQIAQPAQRjxgECsAAMFDGPGAMAIAQPAQRjygmwYAgOAhjHhAywgAAMFDGPHAGUYOH5a++cbaWgAA6OoIIx706iUlJJj9ggJrawEAoKsjjHhgs9FVAwBAsBBGmkEYAQAgOAgjzWBGDQAAwUEYaQYtIwAABAdhpBnOMELLCAAAgUUYaYazm6agQKqrs7YWAAC6MsJIM9LSpPBw6eRJqajI6moAAOi6CCPNiIyUBg40+3TVAAAQOISRFjCIFQCAwCOMtIAwAgBA4BFGWsBaIwAABB5hpAW0jAAAEHiEkRYQRgAACDzCSAuc3TTFxdLx49bWAgBAV0UYaUFCgtSrl9kvKLC2FgAAuirCSCvoqgEAILAII61gRg0AAIFFGGkFLSMAAAQWYaQV3L0XAIDAIoy0wtlNQ8sIAACBQRhphbNlZN8+yW63tBQAALokwkgr+vc3d/CtqZEOHbK6GgAAuh7CSCvCw6X0dLNPVw0AAP5HGPECM2oAAAgcwogXWGsEAIDAIYx4gZYRAAAChzDiBcIIAACBQxjxAt00AAAEDmHECxkZ5vHrr6XycmtrAQCgqyGMeCEuTurTx+wXFFhbCwAAXQ1hxEt01QAAEBiEES8xiBUAgMAgjHiJMAIAQGAQRrxENw0AAIFBGPESLSMAAAQGYcRLzjCyf7906pS1tQAA0JUQRryUmipFR5sgUlhodTUAAHQdhBEvhYW5Fj+jqwYAAP8hjPjA2VXDIFYAAPyHMOID54waWkYAAPAfwogPmFEDAID/EUZ8QDcNAAD+RxjxAd00AAD4H2HEB87ZNMeOSUePWloKAABdBmHEB927SykpZp+uGgAA/KNNYWTZsmVKT09XTEyMJkyYoC1btrR4/rFjxzRnzhylpKQoOjpaQ4YM0bp169pUsNUYxAoAgH/5HEbWrl2r7Oxs5eTkKD8/X6NHj1ZWVpZKSko8nl9TU6Nvf/vb2rdvn1588UXt2rVLq1atUr9+/dpdvBUIIwAA+FeEry946qmndMstt2j27NmSpBUrVuiNN97Q6tWrNX/+/Cbnr169WkeOHNH777+vyMhISVJ6enr7qrYQd+8FAMC/fGoZqamp0bZt25SZmem6QFiYMjMztXnzZo+vefXVVzVx4kTNmTNHSUlJGjFihBYvXiy73d7s+1RXV6uioqLB1lHQMgIAgH/5FEbKyspkt9uVlJTU4HhSUpKKioo8vmbv3r168cUXZbfbtW7dOi1cuFBPPvmkHn744WbfJzc3VwkJCfVbWlqaL2UGFGuNAADgXwGfTVNXV6e+fftq5cqVGjt2rKZNm6b7779fK1asaPY1CxYsUHl5ef1W2IFuk+vspikslGpqrK0FAICuwKcxI4mJiQoPD1dxcXGD48XFxUpOTvb4mpSUFEVGRio8PLz+2LBhw1RUVKSamhpFRUU1eU10dLSio6N9KS1okpKkbt2kb76RDhyQzjjD6ooAAOjcfGoZiYqK0tixY5WXl1d/rK6uTnl5eZo4caLH11xwwQXas2eP6urq6o/t3r1bKSkpHoNIR2ez0VUDAIA/+dxNk52drVWrVun3v/+9du7cqdtvv11VVVX1s2tmzpypBQsW1J9/++2368iRI5o7d652796tN954Q4sXL9acOXP89ymCjGXhAQDwH5+n9k6bNk2lpaVatGiRioqKNGbMGK1fv75+UOuBAwcUFubKOGlpaXrrrbc0b948jRo1Sv369dPcuXN17733+u9TBBkzagAA8B+bw+FwWF1EayoqKpSQkKDy8nLFx8dbXY6eeUa66y7p+9+XXnrJ6moAAOiYvP3+5t40bUA3DQAA/kMYaQP3bpqO364EAEDHRhhpg/R0M6umslIqK7O6GgAAOjfCSBvExEjO+/zRVQMAQPsQRtqIGTUAAPgHYaSNWPgMAAD/IIy0ETNqAADwD8JIG9EyAgCAfxBG2oiWEQAA/IMw0kbOlpFDh6STJ62tBQCAzoww0kaJidJpp5lFz/bts7oaAAA6L8JIG9lsdNUAAOAPhJF2YK0RAADajzDSDsyoAQCg/Qgj7UA3DQAA7UcYaQe6aQAAaD/CSDu4hxGHw9paAADorAgj7TBwoBQWJp04IRUXW10NAACdE2GkHaKipLQ0s09XDQAAbUMYaSdm1AAA0D6EkXZiRg0AAO1DGGknWkYAAGgfwkg7Mb0XAID2IYy0E900AAC0D2GknZwtI199Zab4AgAA3xBG2qlXL6lHD7NfUGBpKQAAdEqEET9g3AgAAG1HGPEDZtQAANB2hBE/YBArAABtRxjxA7ppAABoO8KIH9BNAwBA2xFG/MDZTVNQINXVWVsLAACdDWHED9LSpPBwqbrarDcCAAC8Rxjxg4gIaeBAs09XDQAAviGM+AkzagAAaBvCiJ8wowYAgLYhjPgJM2oAAGgbwoif0E0DAEDbEEb8hJYRAADahjDiJ84wUloqVVZaWwsAAJ0JYcRPEhKk3r3NfkGBtbUAANCZEEb8iK4aAAB8RxjxI6b3AgDgO8KIHzGjBgAA3xFG/IhuGgAAfEcY8SNaRgAA8B1hxI+cLSP79kl2u6WlAADQaRBG/KhfPykyUqqtlQ4etLoaAAA6B8KIH4WHSxkZZp+uGgAAvEMY8TOm9wIA4BvCiJ8xowYAAN8QRvyMGTUAAPiGMOJntIwAAOAbwoifMWYEAADfEEb8zBlGjhyRjh2ztBQAADoFwoifnXaa1Lev2ad1BACA1hFGAoCuGgAAvEcYCQBm1AAA4D3CSAAwowYAAO8RRgKAbhoAALxHGAkAumkAAPAeYSQAnC0j+/ebO/gCAIDmEUYCICVFio6W7HapsNDqagAA6NgIIwEQFsa4EQAAvEUYCRBm1AAA4B3CSIAwiBUAAO8QRgKElhEAALxDGAkQxowAAOAdwkiAOLtpvvxScjisrQUAgI6MMBIg6enmsaJCOnLE0lIAAOjQ2hRGli1bpvT0dMXExGjChAnasmWLV697/vnnZbPZNHXq1La8bafSvbtZb0SiqwYAgJb4HEbWrl2r7Oxs5eTkKD8/X6NHj1ZWVpZKSkpafN2+ffv005/+VBdddFGbi+1smFEDAEDrfA4jTz31lG655RbNnj1bZ511llasWKHu3btr9erVzb7Gbrdr+vTpevDBBzXIObIzBDCjBgCA1vkURmpqarRt2zZlZma6LhAWpszMTG3evLnZ1/3sZz9T37599cMf/tCr96murlZFRUWDrTNiRg0AAK3zKYyUlZXJbrcrKSmpwfGkpCQVFRV5fM17772n3/72t1q1apXX75Obm6uEhIT6LS0tzZcyOwy6aQAAaF1AZ9NUVlZqxowZWrVqlRITE71+3YIFC1ReXl6/FXbSu83RTQMAQOsifDk5MTFR4eHhKi4ubnC8uLhYycnJTc7/8ssvtW/fPk2ePLn+WF1dnXnjiAjt2rVLpzubD9xER0crOjral9I6JGcYKSyUamqkqChr6wEAoCPyqWUkKipKY8eOVV5eXv2xuro65eXlaeLEiU3OHzp0qD755BNt3769fpsyZYouvfRSbd++vdN2v3grKclM8XU4pP37ra4GAICOyaeWEUnKzs7WrFmzNG7cOI0fP15LlixRVVWVZs+eLUmaOXOm+vXrp9zcXMXExGjEiBENXt+jRw9JanK8K7LZTOvIjh2mq2bwYKsrAgCg4/E5jEybNk2lpaVatGiRioqKNGbMGK1fv75+UOuBAwcUFsbCrk7OMMIgVgAAPPM5jEjSHXfcoTvuuMPjz95+++0WX/vss8+25S07LWbUAADQMpowAowZNQAAtIwwEmAsfAYAQMsIIwHm7Kb58kszqwYAADREGAmw9HQzq6aqSiottboaAAA6HsJIgEVHS/37m326agAAaIowEgQMYgUAoHmEkSBgECsAAM0jjAQBa40AANA8wkgQ0E0DAEDzCCNBQDcNAADNI4wEgbOb5tAh6eRJa2sBAKCjIYwEQe/eUlyc2S8osLYWAAA6GsJIENhsdNUAANAcwkiQMKMGAADPCCNBwowaAAA8I4wECd00AAB4RhgJEve79wIAABfCSJC4t4w4HNbWAgBAR0IYCZIBA6SwMLPOSFGR1dUAANBxEEaCJCrKBBKJrhoAANwRRoKIQawAADRFGAki1hoBAKApwkgQsdYIAABNEUaCiG4aAACaIowEEd00AAA0RRgJImfLSFGRVFVlbS0AAHQUhJEg6tlT6tHD7BcUWFoKAAAdBmEkyOiqAQCgIcJIkDGjBgCAhggjQcaMGgAAGiKMBBl37wUAoCHCSJDRMgIAQEOhHUZqa6X586UnngjaWzrDSEGBVFcXtLcFAKDDirC6AEu99pr02GNSeLg0caJ04YUBf8u0NCkiQqqpkQ4dMs8BAAhlod0y8v3vSzfeKNnt0vXXS6WlAX/LiAhp4ECzT1cNAAChHkZsNmnFCmnIENNMMXNmUPpOGDcCAIBLaIcRSYqLk154QYqJkdavN902AcaMGgAAXAgjkjRqlLR0qdl/4AHpnXcC+na0jAAA4EIYcbr5ZmnGDKlbN6msLKBvRRgBAMCFMOJks0nLl0vbtklXXx3Qt6KbBgAAF8KIu9hY6cwzXc+rqwPyNs6WkbIyqaIiIG8BAECnQRhpzqZN0hlnSBs3+v3S8fFSYqLZLyjw++UBAOhUCCPN+eMfpYMHzTokxcV+vzx37wUAwCCMNOfpp6Xhw6WiImn6dLMwmh8xiBUAAIMw0pzu3c36I7GxUl6e9PDDfr28cxArYQQAEOoIIy0ZNsys0CpJDz5oQomf0E0DAIBBGGnNTTdJP/qR5HCY8SNffeWXy9JNAwCAEdp37fXW009LH35oWkpiY/1ySWc3zb590qlT5gZ6AACEIr4CvdGtm5nq26OHWRzND1JTpagoqabGTNpJT/fLZQEA6HTopvFWz56uIOJwSPv3t+ty4eGuAEJXDQAglBFGfFVRIV1zjXTuudLhw+26FMvCAwBAGPFdZKRJD6Wl0g03mAEfbTRsmHl89FHpwAE/1QcAQCdDGPFVt25m/ZG4OOmdd6ScnDZfKjvbzKrZu1e6+GIzmBUAgFBDGGmLwYOl3/zG7C9eLK1f36bL9Osnvf22uQXOvn0mkDB+BAAQaggjbXXdddKPf2z2Z8wwU2LaIC3NBJIhQ0xXzSWXSHv2+K1KAAA6PMJIezz5pHT22VJZmQkkDkebLuNsIRk6VCosNC0ku3f7t1QAADoqwkh7xMSY8SMjRkiPPNKuNUhSUkwgGT7cTNK5+GLp88/9VyoAAB0VYaS9Tj9d+vhj6fzz232ppCRp40Zp5Ehzs+BLLpE+/bT9JQIA0JERRvwhzO3XuGNHu+bp9ukj/fOf0pgxUnGxdOml0ieftL9EAAA6KsKIP732mjR+vHT99VJtbZsvk5hobhB8zjlmOZNLLzWNLwAAdEWEEX8aPtzccGbzZum++9p1qV69pH/8wyz0+vXX0mWXSfn5fqoTAIAOhDDiT4MGSb/7ndn/+c9NS0k79OwpbdggnXeedOSIdPnl0tatfqgTAIAOhDDib9//vjR3rtmfNavdN9RLSJDeeku64ALp2DEpM1P64IP2lwkAQEdBGAmExx83/StHj0rTpkk1Ne26XHy8WeT1W98y9+mbNEl6/30/1QoAgMUII4EQFSX95S9Sjx7Shx9KK1e2+5KnnSatW2cGs1ZWSllZ0rvvtr9UAACsRhgJlPR06dlnpQcekG67zS+XjI2VXn/ddNUcPy595ztmoTQAADozm8PRxjXMg6iiokIJCQkqLy9XfHy81eVY7ptvpKlTpb//3dxE+LXXzOBWAAA6Em+/v2kZCZbqaumJJ9o9fkQyAeRvf5OuuMIEk6uuMsEEAIDOiDASLFOmSP/7v2bzg5gY6aWXpMmTpZMnzeXffNMvlwYAIKgII8Fy553m8Ze/NCnCD6KjpRdfNF021dXmsZ1LmwAAEHSEkWC56irpnnvM/s03S3v3+uWyzok7P/iB6QH6wQ+kV17xy6UBAAiKNoWRZcuWKT09XTExMZowYYK2bNnS7LmrVq3SRRddpJ49e6pnz57KzMxs8fwu7ZFHzN19y8ul664zzRl+EBkp/fnPZkmT2lrp2mulv/7VL5cGACDgfA4ja9euVXZ2tnJycpSfn6/Ro0crKytLJSUlHs9/++23dcMNN2jjxo3avHmz0tLSNGnSJB06dKjdxXc6kZHS889LvXtL27ZJP/mJXy/93HPSjTdKp06ZYPKXv/jt8gAABIzPU3snTJigc889V0uXLpUk1dXVKS0tTXfeeafmz5/f6uvtdrt69uyppUuXaubMmV69Z5eb2vvmm2YqTI8e0mefSSkpfru03W56gf7wByksTPrjH01AAQAg2AIytbempkbbtm1TZmam6wJhYcrMzNTmzZu9usaJEydUW1urXr16NXtOdXW1KioqGmxdyne/K61YYW7D68cgIknh4dLq1SaQ1NVJM2aYQAIAQEflUxgpKyuT3W5XUlJSg+NJSUkqKiry6hr33nuvUlNTGwSaxnJzc5WQkFC/paWl+VJm5/A//yNlZLieL10qHTjgl0uHh0urVkm33GICyaxZrpsJAwDQ0QR1Ns2jjz6q559/Xi+//LJiYmKaPW/BggUqLy+v3woLC4NYpQX+9S8z9ffMM6X77jN3w2unsDDT+HL77ZLDYVpKVq3yQ60AAPiZT2EkMTFR4eHhKi4ubnC8uLhYycnJLb725z//uR599FH9/e9/16hRo1o8Nzo6WvHx8Q22Li0+Xrr4YrN6WW6uNHiw9Otfm5Go7RAWJi1b5lri5NZbTUABAKAj8SmMREVFaezYscrLy6s/VldXp7y8PE2cOLHZ1z3++ON66KGHtH79eo0bN67t1XZVI0dKGzeaBUKGDJFKSszN9UaPNoNd23H7IJvNrLM2b555fvvtpkcIAICOwudumuzsbK1atUq///3vtXPnTt1+++2qqqrS7NmzJUkzZ87UggUL6s9/7LHHtHDhQq1evVrp6ekqKipSUVGRjh8/7r9P0RXYbNL3vift2CE9/bSZ/vvZZ2bgRzvXI7HZpCefdK25dued0pIl7S8ZAAB/iPD1BdOmTVNpaakWLVqkoqIijRkzRuvXr68f1HrgwAGFhbkyzvLly1VTU6NrrrmmwXVycnL0f//3f+2rviuKjDRpYcYMs0jamDHmRjSSmbdbWiq10iXmic0mPfaYFBFheoLmzTO9QD/9qX/LBwDAVz6vM2KFLrfOSFs9+6x0xx3mZns/+YkUG+vzJRwOKSdHeugh8zw3V/JieRgAAHwWkHVGYLHXX5eqqkyaGDLEzNe12326hM0m/exn0oMPmucLFphZxo3GJAMAEDSEkc7khRektWvN+iSHD5v5uuPGSW4Dir21aJG0eLHZX7lSOv106YEHzG1zAAAIJsJIZ2KzmRvs7dwpPfGElJAgbd8uZWaaJg4fLVgg/fOf0vjxpsHlkUekQYPMpb/5xv/lAwDgCWGkM4qONiNP9+wxg10jIqTJk9t0qUsvlT74QHr5ZWnYMOnIETMkZfBg02JSW+vn2gEAaIQBrF3B4cNSaqrreW6uaUW5+27XTBwv2O3mPjY5Oa6V6QcPNoNdr73WLKIGAIC3GMAaStyDyKFDZoTqggVmefk1a8wNarwQHi79v/8n7d5tFkrr00f64gvp+uvN0JS33mrX+msAAHhEGOlqUlJM/0r//qZ5Y/p06bzzpPfe8/oS0dHSXXdJX35pZt3ExUkffSR95zumW8fLGzQDAOAVwkhXExZmFkzbtUt6+GHptNOkrVuliy6SfvADyYebDsbFmVk3e/dK2dkmpGzaJJ1/vmuxWAAA2osw0lV17y7df7/pZ7n1VhNS1q1rUz9LYqJZTv6LL6Qf/tBc6tVXpVGjpJkzpYKCANQPAAgZhJGuLjnZ3AH4449N982AAa6fvf66VFPj9aXS0qTf/Eb69FPpmmtMrvnjH83QlLvuYuE0AEDbEEZCxYgRpvvG6f33zXTgs86SnnlG2rfP60sNHWrWX9u6Vfr2t83032eeMQunLVzIwmkAAN8QRkLVkSOm1eTLL02zRkaG6Xd54AFpyxavZuCMGyf9/e9mAVjnwmkPP2wWTvv5z1k4DQDgHdYZCWXHj0urVkmvvGJm27gHkK1bTdrwksNhLnP//WaBWEnq18+sWTJ7tlmXDQAQWrz9/iaMwPj6a+nNN83I1E8+MQNDnKuc3XmntH+/NGWKdNVVpkWlGc0tnPbww2acCQunAUDoIIyg7RwOs4KrZNJFSopUWur6+YQJZrzJlClmLIrzXDfV1dKKFSaElJWZY+ecY27ON2mSx5cAALoYwgj8w+EwLSWvvmq2rVsb/jwrS1q/vtmXV1ZKv/iFGUNSWWmOXXKJWbH+vPMCVzYAwHosBw//sNkaDmw9fNhMEb7qKnPfmzFjXOeePCnddJP0pz+ZAbJyLZz25ZfSvHlm4bS335YmTpSmTjW9QQCA0EbLCNruxAkTQHr1Ms/XrZOuvNLsh4ebVV+d3TlnnCHJjCN58EHp2WfNeFmbTbr6amnWLNPIEhVlzUcBAPgf3TQIvj17pN/9TnrtNdO1427oUGn5ctNHIzPjZuFC6a9/dZ3Su7c0bZppXDnvPMaVAEBnRxiBtQoKTCh59VVzQ5tTp6TPPpOGDTM/f/99qahIO1In6XcvnKY1a6SiItfLBw0yoWT6dGnIEGs+AgCgfQgj6DjKy6WNG83d9ZzNHdddZ5ZxjYqSLrlEdZO+ow8SsvTrd4bppZdtOn7c9fJzzzXB5Prrpb59rfkIAADfEUbQseXkSM89Z24J7C4tTbXfvkIvXvYrPbcmTG+9ZWYXS2YYyqRJJph873tSbGzwywYAeI8wgo7P4TCDR9avl956y3TnVFebteU//FCSVFIifZz9rP700XD94bOxcvx3AlhsrBn4etNN0mWXscIrAHREhBF0PidOSO+8Y0LKd79rjlVUmJGtp07J3qO3Pus/SX8oztIfS7NULLMSbHKydMMNJpicfTYDXwGgoyCMoGvYv98sUJKXZ4KJm4O9R+uR6p9oxXHX3YiHDTOh5MYbpfT0INcKAGiARc/QNQwcKL30kllT/p13zJ34xo6VJPX/+mMtXVyp114zU4JPjz6oi3cu12/u36uMDOlb35J+/ev69dcAAB0ULSPonEpKpA0bzLol/fpJkr55arm6/eTHkqTdGqy3lKX1+o7+FXGJLr0qVjfdZNZki4mxsG4ACCF00yD0vPSS9PTT0r/+ZdY1+a9qReldXaTbtEJlCWfo2mtNV85FF3EXYQAIJMIIQldFhVnXZP16s+3bJ7stXCNTvtbOwwmSpCn6m/r3PqnkmzJ11azeGj2aYAIA/kYYASQzM+eLL6Tt21V3zXV6912zvMmtvztP59o/lF1h2qlhKo7oL6Wk6LQhqUqaOEj9c37omi5st5tFTgAAPiGMAM1xOHTqp/NV9eI6JRzY0eTHOzVUE+J26sILzSDYO1efre5HCmVLTZVSUqTUVLOlpJgpO1ddFfzPAACdAGEE8MahQ7J/vEP7P/xKh7YcVvnnX+nTr3ppfvWD9acUKUlJKvH8+qFDzcJtTpddZmb+OEOL++PAgdK4cQH+QADQcXj7/c26lQht/fopvF8/DbpCGvTfQ9+1S1mfmJnEmzZJ33r7U0Ud+UqpOqwUmcf+YV9peK/Dij4tVcf/IU2c+N/l6XfskEpLm961WJLOOkv69FPX82uvNeNb+veXTj9dOuMM10boBhBCaBkBWuFctX7TJldA+eqrhudERJhGj+vO2qELMg5rZO/D6nbsK+nwYbN99ZUJHH/8o+tFffqYVhRP3JbEl2QG4vbqZYJKr17+/5AAEAB00wAB4nBIe/a4gsmmTdKBAw3PCQszS9N/61vSxRebacRNMsTf/26CSmGh9OWX5qJ79kjFxdJ3viO9+abr3MRE6euvzX7Png1bUc45R5o6NZAfGQDahDACBNH+/Q1bTvbsaXrOyJEmmHzrW2ZLSmrmYpWVpvvmv4u5qbra3K54zx4TXhqbNMncaNApM9Pcz8c9sJxxhrmJDzfuARBEhBHAQocOSe++6woon33W9JyhQ00ouegi04py5ple3H24qkrau9fVirJnjzR8uHTXXebnFRVSQoLn18bGmpv2rFzpOvbee2ZgbVKSFBXVps8KAM0hjAAdSEmJCSfOlpP//Md097iLjja5YvTohlvPnj680cmTZpl8Z1D54gvzuH+/VFcn/c//SCtWmHOPH5fi4lyvjY833UF9+pgtK0u64w7zM4dDWrfO9fPERPNaWloAtIAwAnRgR4+aRolNm6QPPjDhpLLS87lpaU0Dyumn+7gOW02NtG+fFBkpZWSYYwUFJnAUFDRYPr/erbeaOw1K0rFjTVNRVJQruHz/+9KiRea4w2FaX9yDS58+ZtAMi8cBIYUwAnQidXUmK3z8ccOtoMDz+d27mzEo7gFl1KiGDR0+vfmxY2ZKclmZeSwtlYYMMYNcJDNWZcoU189PnGh4DfcWF0/BRTKtKL16SdOnS7/8peu9Fy0yx3v2dD06t969pW7d2vChAHQEhBGgCygvN0uWuAeUTz6RvvnG8/mDBjVtRUlPD0BvyokTDYNLcrI0Zoz5WWmpaVVx/qysTDpyxPXa226Tli83+0eOmMDRnOuuk9auNft1dWaAjXtYcQ8vQ4ZI553nem11ten7AmAZwgjQRdntZhhI41aUgwc9nx8XZ1pN3APKyJGmdSVoTp0yU5NLS81AWmdX0ZEjUk6OeTx61GzO/SNHpFtucQWXo0dbXmPFPbjY7WY0cExMw8Di3J840bTmOM/961+l2lpTZ+PHQYOk733P9T4LFpig4+nc4cPNz52mTDGDit3Pc+6PGSP9+c+uc8ePN4OLwsNN7eHhrv2hQxueO2uWaa1yP8f5mJoqLVniOvfhhz2fGx4u9egh/e//us4tKJBOO810rTEeCH5AGAFCzNdfm7En7gHl00/NcJHGbDZp8GBXOBk2zMzmOeOMDtSY4HCYL+3ISPP85EkziNYZWtyDy9Gj0qWXSvPnm3N9aXGpqWn5Q0+ZIv3tb67nUVEmUHhy+eXSP/7het6zp+m28uS886TNm13PU1KkoiLP544ZI330kev5GWeYtWk8GTxY2r3b9XzUKM8rAktm+rh7ir3kEjOQ6bTTTGAcNMhsGRlmoNIVV3i+DtAMloMHQkzv3ub7+NJLXcdqa6Vdu5q2ohQXm++r3bulF15wnR8WZrp1zjyz6ZaSEuT/s2yzuYKIZFo5rr7au9c6Q4B7WHEPL0OGuM6NjDRzrCMjzRYR4dqPjDSLyrmbN8/1OvdzIyLML8/dqlWme6nxeZGRplXC3VtvmcBlt5vt1CnXfmxsw3OXLDEtLs5z3M9tPHBozhwTctzPce43/nJw9v8dP24CjHuISU01c9ad7rrL/I6dYcX5mJpq/iF1NO++a7oMnd2L7vt9+kh/+IPr3IcfNv/x9OjRsFuwRw/TapSaatWn8I7DYZYBOHbMhO1Bg1w/W73arNJ47Jj5b8H5uGaNuTWFRWgZAUJQcbErmPznPyaw7Nplvt+aExdnvsMbh5QhQ4Lc5YPAOnnSTAXfu9dsBQXmsUcP80XmNHBg06WHJdPKNGGCaWFxevdd8w8oI6P5dXBa43CYgOBcD8fhkJ57rmm4cD4/5xzpT39yvT4uzoQsTy65RNq40fW8b19zDU9Gj5a2b3c9z8oyX+aNQ0vPntKAAdL117vOPXTIDMhOSGh9ZpndbgaNOYO0MzQcPWr+g7vpJte5N91kpvG7n+ecITdsWMOFjkaMaHiPLKdt25oGbz+gZQRAs5KSzMKtkya5jjkcJqQ4g8muXdLnn5vHggIz9XjbNrM1lpbmCidDh7r2+/fvmP8nGS2IiXH9D9iSp54yg5fcQ8v+/WY8jd3e8NyZM810McmM23FvTRk1yizGJ5l/hA895Ln1oqzM3BV73Tpzrs1mWn2amxPfuIXo3HNN0EpMbDjtPDHRfGG7+/GPzfgd9wDg3Pr2bXjuRx+1HFzcw8hll7m60OLjG4aX4cOlpUtd5w4c2LAlyt2wYQ3DyPbtngNGRETT/wCvucYMBHcGJudj41a9IKNlBECrqqvNEAX3oOLc3CfKNNatm+fWlDPPbOM0ZHRsp06ZMSjffOP6gnc4pAsvNP/P3dOX9sSJ0vvvu54nJDTfRDdunLR1q+v5zJkm+HgKGKmpDbvjAuVf/2raHegMMf37S4884jo3I8MVyhprfFdvZwtGbGzTFpeMjIaDlN9807QauZ/To4d5rcUDkRnACiAoyso8h5Q9ezyvpeaUkuIKJoMGmRbtgQPNlpxMi0qXVFlpWlCcXT8FBeZ/+J/+1HXOffeZL1BPAaNPn6bjZzqbmpqGLS7O/dhYM1ja6dgxc8x93FQnRBgBYKlTp8x3TeMun127TAt4SyIjTdePM5y4B5UBA8zWYWb9AGgWYQRAh3XsmCuY7N5tWq737zfjIQ8eNBNQWpOc7DmoOPcTEixvoQZCHmEEQKd06pQZt3fggAkozpDi3N+/v/kVaN3FxbmCiafQkpJCVxAQaIQRAF2Sw2EWePMUVJz7ZWWtX8fZFTRggHns18+MeUxNde0nJ7tmkgLwHVN7AXRJ7mMbx471fE5VlVRY2HxgOXjQTD5wzkptSd++TUOK+36/fqYWWlmAtqNlBEDIOXXK3K7FGVIOHjTPDx82XUTO/eZWfW8sIsJ0+3gKK+778fGMY0FooZsGANqhrs50BzUOKe5h5dAhMzPI27+i3bt7DikpKWbWat++rpmsEbRbowugmwYA2iEszASDPn3MQprNqa01K9c2DimN948dk06cMGt/ffFF6+/fq1fDgOJ89HSsd2/CCzo3/vkCQDtERpqFNlu7x9iJE66A0jiwfPWVWZzUueq5w2EW9TxyxEx/bo3N1jS8tBRkevdu/dYoQDARRgAgCLp3l844w2wtsdtNCCkpMeHE+ei+737s669dM4y+/tosLtcam80EEmdA6dPHrB7eeEtIaPo8Lo5xL/A/wggAdCDh4a6A4I1TpxqGl+ZCi3PfGV6c96DbudO3+sLCGoYUT4GlpVATH0+rDJoijABAJxYRYbpeGt9MtjmnTplA4h5USkvN3eqPHTOb+777VltrBvY6b6vSVnFxzYcV535zjz16cCuArogwAgAhJCJCSkoymy8cDunkyYbhxFNoaemYc+XcykqzFRa27TNERzcfVLwJM6edxrowHQ1hBADQKptN6tbNbCkpbbtGdbUrmDQOLZ6ONz5WUeG6TnGx2dr6WRISGgaV+HjTYuPcGj9v7mfR0Yyh8QfCCAAgKKKjfetSasxuNy0qjUOKt4/Hjkk1NaaVx/l8//72faaIiJbDii8/i40N3SnaIfqxAQCdTXi4q6tl4MC2XcPZ1dQ4qFRWmpYXZxeS++bpeFWVud6pU+0fQ+MuJsZ0I7lvcXFNj/mydYb7KxFGAAAhIybG3AAxObl917HbpePHPYcXX4NNZaUJNZIJSydPenezR29FRnoXWu64Qxo0yH/v6wvCCAAAPgoPd407aS+Hw4yDOX7cv1t1tbl+ba13rTfTphFGAAAISTababGJiTH3JfKX2lrTneRteElL8997+4owAgBAFxQZ6Rpj09Ex0xoAAFiKMAIAACzVpjCybNkypaenKyYmRhMmTNCWLVtaPP+FF17Q0KFDFRMTo5EjR2rdunVtKhYAAHQ9PoeRtWvXKjs7Wzk5OcrPz9fo0aOVlZWlkpISj+e///77uuGGG/TDH/5QH330kaZOnaqpU6dqx44d7S4eAAB0fjaHw+Hw5QUTJkzQueeeq6VLl0qS6urqlJaWpjvvvFPz589vcv60adNUVVWl119/vf7YeeedpzFjxmjFihVevWdFRYUSEhJUXl6u+Ph4X8oFAAAW8fb726eWkZqaGm3btk2ZmZmuC4SFKTMzU5s3b/b4ms2bNzc4X5KysrKaPV+SqqurVVFR0WADAABdk09hpKysTHa7XUmNbveYlJSkoqIij68pKiry6XxJys3NVUJCQv2WZuXkZwAAEFAdcjbNggULVF5eXr8VtvU+0wAAoMPzadGzxMREhYeHq7jRfZuLi4uV3MxC/8nJyT6dL0nR0dGKjo72pTQAANBJ+dQyEhUVpbFjxyovL6/+WF1dnfLy8jRx4kSPr5k4cWKD8yVpw4YNzZ4PAABCi8/LwWdnZ2vWrFkaN26cxo8fryVLlqiqqkqzZ8+WJM2cOVP9+vVTbm6uJGnu3Lm6+OKL9eSTT+rKK6/U888/r3//+99auXKlfz8JAADolHwOI9OmTVNpaakWLVqkoqIijRkzRuvXr68fpHrgwAGFhbkaXM4//3ytWbNGDzzwgO677z4NHjxYr7zyikaMGOG/TwEAADotn9cZsQLrjAAA0Pl4+/3dKe7a68xLrDcCAEDn4fzebq3do1OEkcrKSklivREAADqhyspKJSQkNPvzTtFNU1dXp8OHDysuLk42m81v162oqFBaWpoKCwtDtvsn1H8Hof75JX4HfP7Q/vwSv4NAfn6Hw6HKykqlpqY2GE/aWKdoGQkLC1P//v0Ddv34+PiQ/AfoLtR/B6H++SV+B3z+0P78Er+DQH3+llpEnDrkCqwAACB0EEYAAIClQjqMREdHKycnJ6SXng/130Gof36J3wGfP7Q/v8TvoCN8/k4xgBUAAHRdId0yAgAArEcYAQAAliKMAAAASxFGAACApUI6jCxbtkzp6emKiYnRhAkTtGXLFqtLCorc3Fyde+65iouLU9++fTV16lTt2rXL6rIs8+ijj8pms+nuu++2upSgOnTokG666Sb17t1b3bp108iRI/Xvf//b6rKCwm63a+HChcrIyFC3bt10+umn66GHHmr1/hmd2TvvvKPJkycrNTVVNptNr7zySoOfOxwOLVq0SCkpKerWrZsyMzP1xRdfWFNsgLT0O6itrdW9996rkSNHKjY2VqmpqZo5c6YOHz5sXcF+1tq/AXe33XabbDablixZEpTaQjaMrF27VtnZ2crJyVF+fr5Gjx6trKwslZSUWF1awG3atElz5szRBx98oA0bNqi2tlaTJk1SVVWV1aUF3datW/XrX/9ao0aNsrqUoDp69KguuOACRUZG6s0339Rnn32mJ598Uj179rS6tKB47LHHtHz5ci1dulQ7d+7UY489pscff1zPPPOM1aUFTFVVlUaPHq1ly5Z5/Pnjjz+up59+WitWrNCHH36o2NhYZWVl6eTJk0GuNHBa+h2cOHFC+fn5WrhwofLz8/XSSy9p165dmjJligWVBkZr/wacXn75ZX3wwQdKTU0NUmWSHCFq/Pjxjjlz5tQ/t9vtjtTUVEdubq6FVVmjpKTEIcmxadMmq0sJqsrKSsfgwYMdGzZscFx88cWOuXPnWl1S0Nx7772OCy+80OoyLHPllVc6br755gbHrr76asf06dMtqii4JDlefvnl+ud1dXWO5ORkxxNPPFF/7NixY47o6GjHn//8ZwsqDLzGvwNPtmzZ4pDk2L9/f3CKCqLmPv/Bgwcd/fr1c+zYscMxcOBAxy9+8Yug1BOSLSM1NTXatm2bMjMz64+FhYUpMzNTmzdvtrAya5SXl0uSevXqZXElwTVnzhxdeeWVDf4dhIpXX31V48aN07XXXqu+ffvq7LPP1qpVq6wuK2jOP/985eXlaffu3ZKkjz/+WO+9956++93vWlyZNQoKClRUVNTgv4WEhARNmDAhJP8mOpWXl8tms6lHjx5WlxIUdXV1mjFjhu655x4NHz48qO/dKW6U529lZWWy2+1KSkpqcDwpKUmff/65RVVZo66uTnfffbcuuOACjRgxwupygub5559Xfn6+tm7danUplti7d6+WL1+u7Oxs3Xfffdq6davuuusuRUVFadasWVaXF3Dz589XRUWFhg4dqvDwcNntdj3yyCOaPn261aVZoqioSJI8/k10/izUnDx5Uvfee69uuOGGkLl53mOPPaaIiAjdddddQX/vkAwjcJkzZ4527Nih9957z+pSgqawsFBz587Vhg0bFBMTY3U5lqirq9O4ceO0ePFiSdLZZ5+tHTt2aMWKFSERRv7yl7/oT3/6k9asWaPhw4dr+/btuvvuu5WamhoSnx8tq62t1XXXXSeHw6Hly5dbXU5QbNu2Tb/85S+Vn58vm80W9PcPyW6axMREhYeHq7i4uMHx4uJiJScnW1RV8N1xxx16/fXXtXHjRvXv39/qcoJm27ZtKikp0TnnnKOIiAhFRERo06ZNevrppxURESG73W51iQGXkpKis846q8GxYcOG6cCBAxZVFFz33HOP5s+fr+uvv14jR47UjBkzNG/ePOXm5lpdmiWcf/dC/W+i5Aoi+/fv14YNG0KmVeTdd99VSUmJBgwYUP93cf/+/frJT36i9PT0gL9/SIaRqKgojR07Vnl5efXH6urqlJeXp4kTJ1pYWXA4HA7dcccdevnll/XPf/5TGRkZVpcUVJdffrk++eQTbd++vX4bN26cpk+fru3btys8PNzqEgPuggsuaDKde/fu3Ro4cKBFFQXXiRMnFBbW8M9feHi46urqLKrIWhkZGUpOTm7wN7GiokIffvhhSPxNdHIGkS+++EL/+Mc/1Lt3b6tLCpoZM2boP//5T4O/i6mpqbrnnnv01ltvBfz9Q7abJjs7W7NmzdK4ceM0fvx4LVmyRFVVVZo9e7bVpQXcnDlztGbNGv3tb39TXFxcfZ9wQkKCunXrZnF1gRcXF9dkfExsbKx69+4dMuNm5s2bp/PPP1+LFy/Wddddpy1btmjlypVauXKl1aUFxeTJk/XII49owIABGj58uD766CM99dRTuvnmm60uLWCOHz+uPXv21D8vKCjQ9u3b1atXLw0YMEB33323Hn74YQ0ePFgZGRlauHChUlNTNXXqVOuK9rOWfgcpKSm65pprlJ+fr9dff112u73+b2OvXr0UFRVlVdl+09q/gcbhKzIyUsnJyTrzzDMDX1xQ5ux0UM8884xjwIABjqioKMf48eMdH3zwgdUlBYUkj9vvfvc7q0uzTKhN7XU4HI7XXnvNMWLECEd0dLRj6NChjpUrV1pdUtBUVFQ45s6d6xgwYIAjJibGMWjQIMf999/vqK6utrq0gNm4caPH/+5nzZrlcDjM9N6FCxc6kpKSHNHR0Y7LL7/csWvXLmuL9rOWfgcFBQXN/m3cuHGj1aX7RWv/BhoL5tRem8PRhZccBAAAHV5IjhkBAAAdB2EEAABYijACAAAsRRgBAACWIowAAABLEUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJb6/5ZWYKdPRk1xAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# instantiate model\n",
    "model_mlp = MLP([D_in, 256, 128, 64, D_out]).to(device)\n",
    "\n",
    "# instantiate optimizer and criterion\n",
    "optimizer = optim.SGD(model_mlp.parameters(), lr=0.01, momentum=0.5)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "# train and validate model\n",
    "model_mlp, losses, accuracies = train_val_model(model_mlp, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=15, log_interval=2)\n",
    "\n",
    "# plot losses\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Xw-k2B1ldd-i"
   },
   "source": [
    "## Question 3.1.3\n",
    "### Plots of the coefficients of the Logistic Regression for each class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 396
    },
    "id": "2pucfjpaDF9_",
    "outputId": "e21dec94-a4f1-40d6-cf20-d956a53515b0"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxsAAAHFCAYAAACadeS/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACRFElEQVR4nO3deZwcZZ0/8G/13dPdc2SOJJM7IYEEQiBBlPsmoIKoXB7ciC5E5Le+wGNVWBU84oqCq+Kx4Lq6KigC4gqCsNyIQOQIgRByX5NM5uqZnr7q+f3BZkw930+YmqPSPeHzfr14vegn1TXVVU891TVTn+frGGOMEBERERERjbJQpTeAiIiIiIj2TrzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkg2gtMnz5dLrzwwor9/AsvvFCmT5/uactms3LppZfKhAkTxHEcueqqq2TNmjXiOI7cdttte3wbjz32WDn22GP3+M+tpKVLl8rMmTMlHA7LQQcdJCIipVJJrrnmGpkyZYqEQiE544wzRETEcRy57rrrhrT+hx9+WBzHkYcffnhUt5tGzzPPPCOHH364pFIpcRxHli1bVulNIqK3mUilN4CIdm/VqlXyzW9+U/785z/Lpk2bJBaLyfz58+Xss8+Wyy67TJLJZKU3cbduuOEGue222+SLX/yizJo1S+bOnRv4z1y+fLn85je/gTc/bzf333+/XHPNNfLRj35UrrvuOmlqahIRkf/4j/+QpUuXylVXXSULFy6UqVOnVnhLB3fDDTfIvHnzBm6MqlE1bmOxWJSzzjpLEomE3HjjjVJTUyPTpk2r9GYR0duMY4wxld4IItLuvfdeOeussyQej8v5558vBxxwgBQKBXnsscfkt7/9rVx44YXyox/9SETe/MvGscceW5G/GIi8+aXGdV2Jx+MDbe9617skEonIY489NtBmjJF8Pi/RaFTC4fCob8cdd9whZ511ljz00EPqrxiFQkFERGKx2Kj/3Gr02c9+VpYuXSq5XM7zmc8991x57LHHZMOGDZ7l+/v7JRKJSCTi/3dQrutKoVCQWCwmoVBwfyhPp9Ny5plnVqx/+1GN27hixQqZO3eu/PjHP5ZLL7200ptDRG9T/MsGURVavXq1nHvuuTJt2jT5y1/+IhMnThz4tyuuuEJef/11uffeeyu4hV7RaFS1tbW1ybx58zxtjuNIIpHYU5vl8Xa5ydipra1Nksmk+txtbW1SX1+vlh/OcQmFQhU7nm8Hvb29kkqlhv3+trY2ERF4vIdrpNtERG9Dhoiqzic+8QkjIubxxx/3tfy0adPMBRdcMPC6vb3dfPrTnzYHHHCASaVSJpPJmFNOOcUsW7ZMvfemm24y8+bNM8lk0tTX15tFixaZX/ziFwP/3t3dbT71qU+ZadOmmVgsZpqbm82JJ55onn322YFlLrjgAjNt2jRjjDEPPfSQERH13+rVq83q1auNiJhbb73Vsw2vvPKKOeuss0xTU5NJJBJmzpw55vOf//zAv69Zs8b80z/9k5kzZ45JJBJm3Lhx5swzzzSrV68eWObWW2+FP/ehhx4yxhhzzDHHmGOOOcbzc7du3Wouvvhi09LSYuLxuDnwwAPNbbfd5llm5zYvXbrU3HLLLWbmzJkmFouZQw45xPz1r3/1cXSM6ejoMFddddXAPpw0aZI577zzzLZt24a0LcYYUy6XzY033mjmzZtn4vG4aWlpMZdddpnZsWPHwDJoPwy2f0TEXHvttZ6ftWHDBnPxxRebiRMnmlgsZqZPn24+8YlPmHw+b4z5x7HeuY6dnnrqKbN48WJTW1trksmkOfroo81jjz3mWebaa681ImJWrlxpLrjgAlNXV2dqa2vNhRdeaHp7e9/ys+za13e1ZcsWEw6HzXXXXaf+bcWKFUZEzM033+w5Lp/61KfM5MmTTSwWM7NmzTJf//rXTblcVvv8O9/5jjnggANMPB43TU1NZvHixeaZZ57xtY3PPfecOeWUU0wmkzGpVMocf/zx5sknn/T8jJ3H5+GHHzb/9E//ZJqbm019fb0xxt85aLvgggvUNu3a/x988EFz5JFHmpqaGlNXV2dOP/10s3z5cniMXn75ZfOhD33I1NfXm4MOOmi3P5OICOFfNoiq0D333CMzZ86Uww8/fFjvf+ONN+T3v/+9nHXWWTJjxgzZunWr3HLLLXLMMcfI8uXLpbW1VUREfvzjH8uVV14pZ555pnzqU5+S/v5+eeGFF+Tpp5+WD3/4wyIi8olPfELuuOMOWbJkicybN0/a29vlsccek1deeUUWLlyofvbcuXPl5z//ufy///f/ZPLkyfLpT39aRESam5tl27ZtavkXXnhBjjrqKIlGo3LZZZfJ9OnTZdWqVXLPPffI9ddfLyJvhlyfeOIJOffcc2Xy5MmyZs0a+cEPfiDHHnusLF++XGpqauToo4+WK6+8Um666Sb5/Oc/P5AR2V1WJJfLybHHHiuvv/66LFmyRGbMmCG33367XHjhhdLZ2Smf+tSnPMv/8pe/lJ6eHvn4xz8ujuPIN7/5TfnABz4gb7zxBvzLzk7ZbFaOOuooeeWVV+Tiiy+WhQsXyvbt2+Xuu++WDRs2SFNT05C25eMf/7jcdtttctFFF8mVV14pq1evlu9973vy/PPPy+OPPy7RaFR+/vOfy49+9CP561//Kj/5yU9EROTggw+Wn//853L99ddLNpuVr33ta2+5fzZt2iSHHnqodHZ2ymWXXSb77befbNy4Ue644w7p6+vb7V+K/vKXv8ipp54qixYtkmuvvVZCoZDceuutcvzxx8ujjz4qhx56qGf5s88+W2bMmCFf+9rX5LnnnpOf/OQn0tLSIt/4xjdEROTnP/+5XHrppXLooYfKZZddJiIis2bNgj97/Pjxcswxx8hvfvMbufbaaz3/9utf/1rC4bCcddZZIiLS19cnxxxzjGzcuFE+/vGPy9SpU+WJJ56Qz33uc7J582b5zne+M/DeSy65RG677TY59dRT5dJLL5VSqSSPPvqoPPXUU3LIIYe85Ta+/PLLctRRR0ltba1cc801Eo1G5ZZbbpFjjz1W/vd//1fe+c53erbz8ssvl+bmZvnSl74kvb29IjL0c1DkzX4yadIkueGGG+TKK6+Ud7zjHTJ+/HgREXnggQfk1FNPlZkzZ8p1110nuVxObr75ZjniiCPkueeeU3mns846S2bPni033HCDGD55TURDVem7HSLy6urqMiJi3ve+9/l+j/2Xjf7+fvXb2dWrV5t4PG6+/OUvD7S9733vM/vvv/9brruurs5cccUVb7nMrn/Z2HWb3vOe96htEOsvG0cffbTJZDJm7dq1nmVd1x34/76+PvUzn3zySSMi5j//8z8H2m6//Xb4m3Zj9F82vvOd7xgRMf/1X/810FYoFMxhhx1m0um06e7u9mxzY2Oj568Hd911lxERc8899+gdsosvfelLRkTM7373O/VvOz+j32159NFHjYh4/vJkjDF/+tOfVPsFF1xgUqkU3A/omIv1l43zzz/fhEKhgd/eo+22/7Lhuq6ZPXu2Wbx4sTp+M2bMMCeddNJA287fml988cWedb///e83jY2NnrZUKrXbv2bYbrnlFiMi5sUXX/S0z5s3zxx//PEDr7/yla+YVCplXnvtNc9yn/3sZ004HDbr1q0zxhjzl7/8xYiIufLKK3e7H95qG8844wwTi8XMqlWrBto2bdpkMpmMOfroowfadv5l48gjjzSlUsmzDj/nILLz+Nx+++2e9oMOOsi0tLSY9vb2gba///3vJhQKmfPPP3+gbecx+tCHPjTkn01EtBOnviWqMt3d3SIikslkhr2OeDw+ENgtl8vS3t4u6XRa9t13X3nuuecGlquvr5cNGzbIM888s9t11dfXy9NPPy2bNm0a9vbszrZt2+SRRx6Riy++WM2K5DjOwP/vOutWsViU9vZ22WeffaS+vt7zeYbij3/8o0yYMEE+9KEPDbRFo1G58sorJZvNyv/+7/96lj/nnHOkoaFh4PVRRx0lIm/+Femt/Pa3v5UFCxbI+9//fvVvOz+j3225/fbbpa6uTk466STZvn37wH+LFi2SdDotDz300BD3Aua6rvz+97+X0047TQ455JDdbrdt2bJlsnLlSvnwhz8s7e3tA9vX29srJ5xwgjzyyCPiuq7nPZ/4xCc8r4866ihpb28fOA+G6gMf+IBEIhH59a9/PdD20ksvyfLly+Wcc84ZaLv99tvlqKOOkoaGBs++PPHEE6VcLssjjzwiIm8eP8dx1F9K3mo/7FQul+X++++XM844Q2bOnDnQPnHiRPnwhz8sjz32mPqcH/vYx9TkCaN5Dm7evFmWLVsmF154oYwbN26g/cADD5STTjpJ/vjHP6r32MeIiGgoeLNBVGVqa2tFRKSnp2fY63BdV2688UaZPXu2xONxaWpqkubmZnnhhRekq6trYLnPfOYzkk6n5dBDD5XZs2fLFVdcIY8//rhnXd/85jflpZdekilTpsihhx4q11133aBfsP3auZ4DDjjgLZfL5XLypS99SaZMmeL5PJ2dnZ7PMxRr166V2bNnq1mUdj5WtHbtWk+7fTO088ajo6PjLX/OqlWrBv18frdl5cqV0tXVJS0tLdLc3Oz5L5vNDgSCR2rbtm3S3d096HbbVq5cKSIiF1xwgdq+n/zkJ5LP59XxGu5+3Z2mpiY54YQT5De/+c1A269//WuJRCLygQ98wLOtf/rTn9R2nnjiiSLyj3D1qlWrpLW11fPF3K9t27ZJX1+f7Lvvvurf5s6dK67ryvr16z3tM2bMUMuO5jm4sy/tbpt23hwOtk1ERH4xs0FUZWpra6W1tVVeeumlYa/jhhtukC9+8Yty8cUXy1e+8hUZN26chEIhueqqqzy/WZ47d668+uqr8oc//EH+9Kc/yW9/+1v5/ve/L1/60pfkX//1X0XkzWfqjzrqKLnzzjvl/vvvl6VLl8o3vvEN+d3vfiennnrqiD+vH5/85Cfl1ltvlauuukoOO+wwqaurE8dx5Nxzz1W/KQ/K7qbqNXvwGXbXdaWlpUV+8YtfwH9vbm7eY9uC7DwWS5cuHSgiaEun057XQezXc889Vy666CJZtmyZHHTQQfKb3/xGTjjhhIFaIzu39aSTTpJrrrkGrmPOnDnD/vkjgWrnVPocrOZ6PkRU/XizQVSF3vve98qPfvQjefLJJ+Wwww4b8vvvuOMOOe644+SnP/2pp72zs9PzhUtEJJVKyTnnnCPnnHOOFAoF+cAHPiDXX3+9fO5znxuY1nTixIly+eWXy+WXXy5tbW2ycOFCuf7660f8RWfnoyWD3VjdcccdcsEFF8i//du/DbT19/dLZ2enZ7nBHmvZ1bRp0+SFF14Q13U9f1FYsWLFwL+PhlmzZg36+fxuy6xZs+SBBx6QI444ItAvgM3NzVJbWzvkG96doeja2tqBvxCMhqEcVxGRM844Qz7+8Y8PPEr12muvyec+9znPMrNmzZJsNjvods6aNUvuu+8+2bFjx1v+dQNtY3Nzs9TU1Mirr76q/m3FihUSCoVkypQpfj7SqJ2DO/vS7rapqamJU9sS0ajiY1REVeiaa66RVColl156qWzdulX9+6pVq+S73/3ubt8fDofVb4Zvv/122bhxo6etvb3d8zoWi8m8efPEGCPFYlHK5bJ67KWlpUVaW1sln88P9WMpzc3NcvTRR8t//Md/yLp16zz/tuv2o89z8803S7lc9rTt/JJk34Qg7373u2XLli2eZ/tLpZLcfPPNkk6n5Zhjjhnqx4E++MEPyt///ne588471b/t/Ex+t+Xss8+WcrksX/nKV9S6SqWSr8/tRygUkjPOOEPuuece+dvf/rbb7bYtWrRIZs2aJd/61rckm82qf0ezkfmRSqWG9Nnq6+tl8eLF8pvf/EZ+9atfSSwWU5W9zz77bHnyySflvvvuU+/v7OyUUqkkIm8eP2PMwF/6drXrfkDbGA6H5eSTT5a77rpL1qxZM9C+detW+eUvfylHHnnkwGOTuzPa5+DEiRPloIMOkp/97Gee7X3ppZfk/vvvl3e/+92DrmPVqlWyatUqT9vmzZtlxYoVUiwWB9q6urpkxYoVw37UkYj2DvzLBlEVmjVrlvzyl7+Uc845R+bOneupIP7EE08MTIu6O+9973vly1/+slx00UVy+OGHy4svvii/+MUvPCFVEZGTTz5ZJkyYIEcccYSMHz9eXnnlFfne974n73nPeySTyUhnZ6dMnjxZzjzzTFmwYIGk02l54IEH5JlnnvH8lWEkbrrpJjnyyCNl4cKFctlll8mMGTNkzZo1cu+998qyZcsGPs/Pf/5zqaurk3nz5smTTz4pDzzwgDQ2NnrWddBBB0k4HJZvfOMb0tXVJfF4XI4//nhpaWlRP/eyyy6TW265RS688EJ59tlnZfr06XLHHXfI448/Lt/5zndGFNDf1dVXXz1Q2fziiy+WRYsWyY4dO+Tuu++WH/7wh7JgwQLf23LMMcfIxz/+cfna174my5Ytk5NPPlmi0aisXLlSbr/9dvnud78rZ5555qhs9w033CD333+/HHPMMXLZZZfJ3LlzZfPmzXL77bfLY489BgvFhUIh+clPfiKnnnqq7L///nLRRRfJpEmTZOPGjfLQQw9JbW2t3HPPPUPelkWLFskDDzwg3/72t6W1tVVmzJihpoy1nXPOOfLRj35Uvv/978vixYvV9l599dVy9913y3vf+1658MILZdGiRdLb2ysvvvii3HHHHbJmzRppamqS4447Ts477zy56aabZOXKlXLKKaeI67ry6KOPynHHHSdLlix5y2386le/Kn/+85/lyCOPlMsvv1wikYjccsstks/n5Zvf/Oagn72np2fUz8GlS5fKqaeeKocddphccsklA1Pf1tXVyXXXXTfo+0844QQREc8N1Oc+9zn52c9+JqtXrx6YOvfOO++Uiy66SG699da3HK+IaC9XqWmwiGhwr732mvnYxz5mpk+fbmKxmMlkMuaII44wN998s+nv7x9YDk19++lPf9pMnDjRJJNJc8QRR5gnn3xSTf96yy23mKOPPto0NjaaeDxuZs2aZa6++mrT1dVljDEmn8+bq6++2ixYsGCgINmCBQvM97//fc92jmTqW2OMeemll8z73/9+U19fbxKJhNl3333NF7/4xYF/7+joMBdddJFpamoy6XTaLF682KxYsUJ9bmOM+fGPf2xmzpxpwuGwr6J+O9cbi8XM/Pnz1bbtWtTPJqAQHtLe3m6WLFliJk2aZGKxmJk8ebK54IILzPbt24e0LTv96Ec/MosWLTLJZNJkMhkzf/58c80115hNmzYNLDPSqW+NMWbt2rXm/PPPN83NzSYej5uZM2eaK664YtCifs8//7z5wAc+MNCvpk2bZs4++2zz4IMPDiyzc1rVXQsbGvOPKWB3Ldi4YsUKc/TRR5tkMvmWRf121d3dPbD8rlMK76qnp8d87nOfM/vss4+JxWKmqanJHH744eZb3/qWKRQKA8uVSiWzdOlSs99++w0U1Tv11FM9RfXeahufe+45s3jxYpNOp01NTY057rjjzBNPPAE/tz3VsN9zENnd1LfGGPPAAw+YI444wiSTSVNbW2tOO+203Rb1s4/RtGnT1Pm+s4ggKrS5u35MRG8PjjGs0ENERERERKOPmQ0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoEBE/C7muK5s2bZJMJiOO4wS9TTRGGGOkp6dHWltbJRQK7r6V/Y+QPdX/RNgHSWP/o0rjNZgqaSj9z9fNxqZNm2TKlCmjsnG091m/fr1Mnjw5sPWz/9FbCbr/ibAP0u6x/1Gl8RpMleSn//m62chkMiIisnLlyoH/J+rp6ZHZs2cH3ifY/wjZU/1P5O3TB9HvLEtGt5Vd0AjEwnvvb0HZ/0bfaPcWf7107OI1eHQ5xvW5oL+/IrH//YOvm42dfzbLZDJSW1s7sq2jvU7Qf1Zl/6O3sif+rP926YO82Rg69r/Rw5uN4eE1eHTwZmN4/PQ/BsSJiIiIiCgQvv6yUSnDvld3S74WKzvej18Gt6HoF3N+f1lnry8E3uf7M9p33ODO+u1yF11N4PFDvx3x8ZsQdPzQL5ALoNGA5WrCPnqEz9/Q2J/J+H0fQcM+74ES+J0RGqP8/DHC73jngt9klUEn3NrnHYvr4mG1TNIp6x/g5zeMIXD54m8cfRnV34OjY+X3N8S2kYwr4Gc6ozlO2etH/c/vqka4KWPdaI5/fvufr79auGAZEHw2kYRef6Fv8J8Z1n3GDLcfjaBvV+L6zW8MREREREQUCN5sEBERERFRIHizQUREREREgeDNBhERERERBSLwgHg/SF0nhjkdIgpVOSAM7hT7wZt18Mf+8GhnOCh56zOAHrKCPyakw5Ew5OOjEigKFcEwHAOTw2YHalHAH/ZJtM/9hNPA+0o+pxh1QT/Nud4NjoEPEB5muNP3Gfw27H++9k25oNvCsUHfhoJ9ZTDGdhf1MQyD42/3L9TdUL9Hy6GhEvU5BXwmFLb08z44cUasRrXlwHy+RetD1cbG5u/iAp0E1ee1D12Xh/8zfY5Rw5yYwzdrXSbs8+eh/j3MCUPGohFNJ2u1OSXw3Q4p6fHVKeZ0m91PfQbE4Tjj59iX/X1v8/Nd0YCwOZy0AIXl9VJ6/WiZERiboykREREREVU93mwQEREREVEgeLNBRERERESBCDyzgZ7Zhc+LoeyF1eb4fHYTPZsH2c/KjeR5VPTcnbVcCD27WAbrQkVkrOe5HZ+FYEwsqdtAQRr7GWwUqxkLz5D6yVkgaDG1D9DxQ/3W53F2XG8xM/ScZjwCnuNHz4ZGwbOg1nPvoWyHXqaY1+sC1Lah4kQocwDaTDiql4t6+6QDMg0u6LdjAuoP4PxFRUWN9fugfpDFyIK2LT1F1dZf1ssVrba+oi6wly/p96XjevszMfCcsaWxRh/7cQm9rvpErWoL5bq8DSUwzqMsW1nvi5p4SrUVrPeiMWEsjIG+gHGrrV9/4njE2xYCY08RdNxwSJ/3KH+WLXj7Vh6sy/H5O1F7W0VEkhHve6PggoDeh7632E2hfFYtY4/pIni8g8/b29d4vcTY7H9+Mxt+cjjgOyA6vwVkO1CW134vuu6g9RtQ1BR+B7SvWWB8gvkMkDMzkbheTi3k71rjgL5rZ9vQ95hhFyAU/mWDiIiIiIgCwpsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQIx6QNyOzcD6fShg7TdE5IeP4lgiOphjIhm1TMHR6yqD6lUo/BaxEmUonIb2DyxUCIrU2AwIE+N16aBU2A7+OCBUNMjramDnC/3WjyyA7he1bsX9FMJ5cyNA/waBQDee9rzuAhuRzYGQFuh/yagOJmaswFe0TofOUNHNHTm9rl4rhIwGjjQIuqXtnSgiqXKvagv193heuwl9LtqBzFBer2dP8xPidMHvdFDQ0y4qJyKSKw4eoM2BAHdPQffBjpwOOm7o9o4Fb7Tp4GBPv14XKhBYD8LfdTXeMWn/Cfq47t+cVm1JENpNWKHMcB+Y8AAEPOHEBeUG1RSrsdoMKMa26/+P5jVrmPzU/OwD/QP9nrEMPk9P3tsG5iKAkwqgayTqfz0F73sTYb1daDKClpTuazEw2Nv9KAXGI3SNDPXqvmWPUU5RF540Md2X0fcdN9Wo2+z8r4/CvZW+Bvv6+ahYLeh/8LuQ1Sdh4TwwoQ7aLqcweFE/09etlnELoJBgCYTSwfUvXOc9zm5UT9gjET1ZhQETWPQ73g6CJgbB44E+P5MR/V0gbfctFKgPWZMe+S2yKPzLBhERERERBYQ3G0REREREFAjebBARERERUSB4s0FERERERIEY/QridsgMBHpgBW4UqrXfhypF+g2bo8qHVqCnDKqedvoIy4rgcGfZarMD4yIimTgI0IIQW03UG+hBlbHhfkXBcnRMrNcwiIXaqlzJZ8lVFLpWewUFCUHoDFYvBe811jFNJ3VoFU084ID4W21ch9NiOSvkCI5fIawDu32gf3f1e8+9eESvC1XdLbq6rQRClNF+qzq0n2qyVRDQxUFv72t0DPtBaDcB9qmtAALi3f16jMoWdFtHvx4/27q9FeTtMUsEh8HRcjnwM9MJ73LdIGyOwsQOqtBrMShsicY2VBW4oMO9jrU+EwUV66tsDITXAR/vQ9cwdEz7rAF0Bwh5Z/P6mKLQOGqz+5YLtiEEPmSuOHi1ehGRkLU30hH9ueFEE2BssQPhbtd2vUxEB4yddL1uQ9XHrUrjJjb6X89Gm4+rJvwugarO94I2WxisK2lX6RaREGhDxzTcbR3Tnk61jNvdrleVB9f4uP6ZTsSayGCcvt6W082qrcfVx77XGl9zRXC+gnMMXWuawQQL9vUngsY/O7A/hIri1TVyEhERERHRXoM3G0REREREFAjebBARERERUSB4s0FERERERIEYUQLJV/XIEYQ4HdcKu6Bq2KjKof0+ETF2UEdEhcZR2A4UNIV6QEhue583mBgGoceGpN6uCWlQCdx6HQLrCjs6NBeP6zBuKNel2sQK3xsQXLcD9D7yXIFCFXxjxnscyqACPDrOkRD4MNY+QUFCtC9DdthZRNwceG/MG8AKpfT7mtJNqg317/DWjarNWME2J6MD6HUTD1Bt+RLoR1Yl3iQIM9fGdFvC5FWb06/b7FC9E7XL6YoYqy+beOUD4mgMtOd3QAWc0amDQrt2dfAuMM6gauFR0MnTIHA6q9muVKsr16LgetxnpWe7IjQa79Kg34A5MsSxg+RhcPkqgirBYGIRpwT6pf1esH6za/C0CsLiqM/Y5yZaBk00gCYy2Jr17icUQG3v0/u3E0xGgALoKatPhhK636JJBVAovSaqx626hHf9ruhxJR3Vod0MGH/KViX6UFRXYXb6OnUb6n8++qSJ6GuXhEFbBcHvgNb3NLvquYiIMfqdaI4W+2sO+s6B+i2qJi9gogi3s83zurj6ZbXMtudfU22bnl6t2gpZfUwnLmr1vJ7ynmPUMvED3qXaMhPmqTb7bwOFsj4Xu8C5gs7ZhiQIoFvjBNqHSbta+xDGwMqPlkREREREtFfizQYREREREQWCNxtERERERBSI0a8a4+cZLh8F/EREP/uHijOBZx9NyF/BH3s5vxkEVIQKPRfXYRVAQs9b1/X5OwTJiPcZ0mRUP09XAM8uoqcqE2H93LT9vDLKQ9hrr3BkA7OeaUU9Ae4l1LeK3mdoHVSsr6iLg7m9urBTuaNNtZmSt39EJk5Xy4RA1sig53bBeRdK1Xq3CxTTQ8XNMnFdLC1j/V4i6ej+Hurv1G19HaoN7Uc37n1uuir7lrx5NnnOKHCeFKxChujZ1+6C/oToOfregnf9UfuZWREZn9bPlyMoV5a3MiFNKd23EhE0hoAcB1jO/uioQCC6YqDxRxWUQudBXGdODDpvfRTeNO4g21CBopJ2/0PZPbsblUC/WtupPz8qBFksez8jKrCXjulRtrFGj1vjQF7HXg4XBlVNsrVXj9dbenQOxy62i/YXykiWY7otmfBm7BLg+IdgrhTliECxXfv7jo9rcFWyrkWoqF/Y8ZezQNELWxzkLR2Qrwz36uJ8xS3rPK+7Vuns45bn1qm2x5/dotpW9+rvoos39nhe18+ZopaJzdS5SZQFTacaPa9RYcRJtSBrBL6vNoLMRsLa2aOdyeVfNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgrEkALi/WUjsV1SI3agBPET9BPBhcr8BPBgGBwVYwIFeEzM25Yt6J+H2lDhFDtoKSJi57VRuBOF8uxguYguwoLCVCgfngXpurhVTE5EB3lNQhc6ioj7lq/3NBQ8gxl5Cyrqh3+Aj8+HJkRAfRKFTYuFQZexg9MiIh2ODnoXa8eptuaktR0oBA8+Y01Jh+vsglNOPquWCeV7VJvbtV235XU4NTx+qneZGl2AsNoKWomIlMDva+waeGiYDIGJG1DRM7ugHpoYAoW1XRAKrAWFOu3if40J3XfRJAICxjIU2LbP0VxJb5cL9g/ar7FhFtHzFOL7P04fKmzqc+KSKlIDQvn2LkZBT1SID42LdlG8lpQOoKKwaaM99ohINLtVtUW6vUFbAwqVuXXjVVtD40TVZofZRfS1FF2nUdFDA84f+/yMoTEKBMRDICAOCxTbEyDod+lr0h6epMAxLv5O51nIe56iyy2aKAJN5mKPnehno/EJTRyE9lWozhu6rtt3ulpmfIe+rs17dYdqawITJUw/yhsIr1+4SG9Wow6NGzSJj7X9tXH981ABVnT9iQsoKmn3PzBu2oWOUT/eHf5lg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokAMKSCeCDu+QuFBQQFEtDUoVFuM16o2O/y9PafDLm1ZHapFVVQnZnRwblKtN2CDgt+bs7rqKao03mZVp0SVVlHoClUlTkb0Ya9VwR8QArPDR8MMbAZpNIteqpAWDKT7qyIb6teh67I9KQKYJAH124df0+G09V06hHjK7GbP69l1oMp4dptqcwog0Gh9TlSBWQq6L0tJ93kD9oXYbX4mlqhABWcbyOcqKKALcnwwoNuS8p6rqXynWsYBFev9npt2qN906yq7BvTLUH2LaivXTVJt/fE672sQ0IXbFUUBdO/5iEKg8BrhgP6MJnFQy1Tf+OYHmmjAVpvQ14DxoHq8XR18aq0OrsY3/l215R54SLVtXvaqauvb4h3L6mbpPtT8ntNVm8zQAfH1XXpMarOur3ZFcRGRODgZ4yCgO9xrCwr7ovPT7rtoAhT1vj19DXZC3p/pY5xG1dIjaLIPGP62jqnPz4v2ebleB7El4518ID51f7XIxHS9auvd0qHa9gGTJ8w4/1zPa/fgU9UynWX9mWpQ0Nt4v1dEwTUYTULkgOttCFRTtydgKaf0pDMS8S4z6GQBu/5M30sSERERERENAW82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAjGkgLgNhUNUqAlVGEThKJSF8hEGciM6lFMCoVpUJXRz1hswRAHuNKjSODGtw03NMRCizHuDOf0NuuLo2i69rrWdOtBobxvK6aMgar6ktyse0dXU1R4bQmXISkHZXD/TF6D8pBPSp4K9LgMq06NQm4mAQCrajp5Oz2u0x3Mz9Pp/9Mgbqq17u66i+q4p9Z7X4bAOEoe6dFVfJ6z7vHG8ewNWF43qfhWK6fPTARXsJZ7UbfY2WMfIfl0RPsa3MhjcUPXXJAhFp7s3eH/c68+qZUrbNqq2clZXvS3166BmocfbbwrdOkwYjumwZf3+c1Rb/MDDVVus9UDVZgs5/iYdcWPe/hVClYPR8QD9xCT1NcJY60dhc8+xrYYJMsDnDfs4L6bX63OwHvTJxoj3uhNa8bhaZtuDf1Jta//8omrb/Hc91iSt7UABcTPtYNX2p9d1wPXuZZtUW8wK2oZngsld0nqMSoKA7riEd/+EevTnCfXr8w6NUwZ8b1Ft1dC/BjPcSTpQGLykxyfHmpzCjelzshzSbZ15/b2nF0xOUXa953wdCEU3T5iq2hrm6H46bqEe68qLTvO8/ttmPb42JPX42prWbamyNxDuFPS6YIV10CfROGl/LXJKesIXewweyjV4DPRmIiIiIiIai3izQUREREREgeDNBhERERERBYI3G0REREREFIiRJSxBgMlPRUEUPEEVH92EVQkchXFBQCVf0gngLhAYyluBoUm1OrSFKqbG1j2n17X8r6qt2N3peZ2YpatT7jvvaNUWHqfDsms6veGgjd16H/YV9WdEFXvr4vozTUoOHtAdC+wqzD6K6b4JTloQGXQZCYFQG6iYWupoU23dr3mD3rFaHbAuLNIf4PVnV6u2WE1atbVYFYFDOzaoZQyo8C2gYqob956LJgECtqhSLgiiORkdoHeKVjVUcF7bEyCgCRGqgvWZo6iqNdgv8awOnBaf+R/P645lL6tlshu3q7a+7To82Num2/o7vPu9XNDjRbxWb//kNl1Bd0pDs2oLWRV6x6X1MiXwO6+Cn4OL+haqbA+uG+raIiKFmkbPa1Qd3slnd/l/vT+rURx8kASYBGKcq4Ok4U2rPK/7XnhKLbPjZT0eob6WbtHj26xT5npet1ywRC1z72bdF/776fWqrRtMrDKl1TtOocrpEzP6ut9co/dPZMdaz2tnh56YwYmAyuM1darNrsQsImLi1hg+3PD1nuRjG9F3NDiRA+DGU57XOaOPS1uPvoYt36b736vbsqotHfdu2yn7NKpl3C49GUGyRU/2EzvgMNX26Cbvdry0VZ9jh7Tq/hHO6H7kFL39GwXq4fFAExTopdRyJpZCSw0b/7JBRERERESB4M0GEREREREFgjcbREREREQUiMCrYsEMByooUqOfgcu5ViEx8Lweeigf5TO2ZvXzbfUJ73Nx0zJ6/c4zv1dtWx56SLW1v6Kfh++zCq21vks/ZzoxpZ8dnjj7GNWWLXg/EypAuB48s5or6H0xtU4XdHIbrOcGx8Dzoui5Qz/lwexcx+5WVnYGPz0iKMcB+ndxsz727dazzqkJuqBQT04fv+zWNaptn8PfpdomW899llfp3EioTj+jakBxPtc6P137+WLBGQq0q8MxUPzPfv4dFSfyse49zkfhLZThQcW/ZP1y1WRnNNA407vVXz6jY5sugpezDloaFDMLx3RbGRQILHdsU22xbd4+HiqAvE6mRbW5oPCo/Zx3CBS1CuW6VJsUQI4DXG+29Q3+HPnE5D/GThMFz0zvaaD/RYz3c4wDOQVwSMXZ0anayiBrZks216u2KUfp4zdu7jTVVvvuD3teP9Kv+8JPHntdte0Az+TXNemfuWCqd9sWjNdZs9a03j/R7bpwqtniza+Uc6D/Zep1W1Jf410fz9HD8c0eF/fwddqItZ0+8mgqjyeC87dR/b1kR8G7FzZndVb1uc26WO3/vqrHojWb9HKL5jR5Xr933ya1TCil+0xy6nTVVqqfrNreeN075naB720Iupaqoo+o6Cjar+D6A99rjSUwWw2yRn7xLxtERERERBQI3mwQEREREVEgeLNBRERERESB4M0GEREREREFYkQBcVgYxA6ZuP6KjORdHYfKWgWmfBV6EpFNPf4K3u3X5C1kF371Eb2u/7lPtW19bp1q2/6qLvzSn/MGpdLjdZGUlvYtqi05W4eIMjHvPgs7en/FQbizDAL0IZCQLlqh4IgLQkV28NlnYZ7R4hrvfABhkKCzP63fum8GBC1LVn9zwD5HAXHTr4O4PetBUb/13pAwCoi/AIoApcdPV22L5upgZXy7N1hZ2KpD6jEQfjOoMJUVDEOnIuprYZTGR6FGq80Bi4St4Kv9umr4CI2H+nRRvOIWPa7YokkQ9k2D4+XqIp01Tbot1eIdk5JNOviPAsANc6aoNhSOFVQ00uKAoHcChBOdove8QvvQ9OoQaCipx10T0pOB5PLeTreyXZ/HnXX/2IfZHn1NqQpW/wPTquDih6Df2uHYcL0eo5oOBMV94zrsmzjwSNW2MT3L8/r+p/Q5gMaVebP1xBaHg4JsU+q8fb45BYr0wgsJGICscRH1Kwf0K3RtgRNg+Lme2u/bw9dgR7zB9WxRfw57kgnH1edJOakL2XXl9bpWdXgnlHhmo54A4qHl+tq6ZpX+PpbP6X3VbhV9RN8xnQmzVFu8QV9vOyN6IoA1O7xj1LZufd5tnaDHugYwzmcj3v5WE9X9NhbWfS1mQOFhHwWyfRUNHMIEBfzLBhERERERBYI3G0REREREFAjebBARERERUSB4s0FERERERIEYUUAcBprsgHgeVMoF1QtROKjXCh919eufh4LfqMrkpFodWKsrebctt+wJtUz7Kxt128odqm1rt/6ZdlAqXq8DZWFQwbmMqjuKN2iJwuCTQGVwtH8yMR1is0N4sDK2HewHQf8ghRyr+jcKJ/kI54K8IV7Oeo2ClnAbfIRiRUTC1nFoeeeBapln1+kQbN14HdJcOE1XRC6++EfP655Va9QyDROnq7YQ2H47LBaOgz4KwuAoe4n6jYl5g5wwVFnh/gehMdDeLhTGs6vBiki4cYJqazjIe/5mpo5Xy5RANe9QFATJm3WoMTp1jvd9tbpvoYryaPtRENRN6NCkXkjvQ1QJ3A6Imx59biAmrEPBaJxwrJrNURC2nFn/j37fHRp+Nd3R0gO6X8bufuB98PwK6z7jWKFodL1CQelQrV7OzTTrNmvr5rfqCSsOsaqAi4hMBde65hp9nNFEKrbugj4/65pm6nXFrc8JArROMafa7OCtCA7o2pW2YbVm+7sB/K4QHLuCeBSM+WXrPColdBh8W6/uuPaEQCIiy7dlPa9RGHzDuk7VhsLgkejgE+is7tDHb8Jk3RdQ6Hptp257fo13jMqCCuKTx+nxNQeC9y0p77GeVKvH4ExMn+3JqD4vEhF93EL2pBHou80I+h//skFERERERIHgzQYREREREQWCNxtERERERBQI3mwQEREREVEgRpawRAFNK+xnh55ERMo1IMya08GWohXe2dqrQ9htvTqUky/pYMv88TqoGNnhrbDc8ZqusGxXeRYRyWZBZUVgwizv52w8YIZaBlWn7AHhIHtfNIEwXF1cH4/+sl5XfQIE6eygl9/S25UEQo52+BvkoGD1a8QOF6KwMwrFIvF6XZl56vH7e15HjzhDLbP2Xl0JNVGjQ1mHTalXbdnHXvO8LnTrSs2mT/dvVF3Y/pwo4AgnFUCGW/W2whXsITQGWvum4OjjFaubqNqcuTpIGZ3pPT6xvD6GaBtcO8wqIsW0DpdvtIKUHTndn7vyej83girzyZg+QRzjbRuX1NMspAq66rcB1w17QhInoT+jRMC+iOnlekI6lNlT8F5f0Pm+a5i4BwRa97QICOj2lLwDnKOmuhBJgPBxCEwEIEnvdRPtc5PTfdLt0EHeCAiSt072jounzdHB8pijtx9OIJDfrrcj5V1fu6tDtZ39qM+rJqlLeM+f2hiouN6v+7JTBhOGwAri3u0wezj87YddQRxVX7ercKPrLcrt9xb0cejo8+67Avhu54BzYNx43ddqwCRB9db3KPR9cl23bkOTFf3X3/T3Rzu8nq7X29AD1tVf0vuiJuodO9H45Pe7jVMG32HL3u1w4/o7i7rus4I4ERERERFVGm82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAjGkgLgdDoKswAgMR4EgadnoZEufFRiyQ9IiIm09OslVl/RXMdHt9Ya53IIO6sTSel0NzTpINwFUc5x67GzP68yhR6tlSqBSaXdWb4e99TjkrZoE5MMlDYJtETtEiKoeD/J6T8uDNFTcTk2BzxGC1YO1sLECqWUQSAZhagHh2dScffV2WFV2dyR1aDjbv1W1oZDZPnX6VO6wKkuHQAcxeV0xdSihr0GBIBqaNMKxz39Q9dlErc/tN5AeJHSeWIHtHhBAzRV133UcUKk75K2sHknrnoqq+KJqvBs3ZlXbi23eAPpza3RVbrvKrojIXFDpeRKo6txkTWawf4sObiZr9OQdIbBfyzHv/jEg+G1AFey86L60HlT7/csq72QMtWCMPWCX7Qe7fY9DZ0CX1d8K4PjFwVjQXKMnKEgmvVXbQyl93IvrX1Nt/Vt1WNu8ppdLND7peR0B63fBJBzlkv5eEWmepNpC8w73vC5FJ6hlNnaDNDhgrGt8CezXTExvP6o2HSr4GHdRiLfSoXHjerbTAWNwXLzHBk04UpPU4eOyq8+3fZq853jfPnoCgW1gTJlQn1Rt9rpE9KQ68Yj+PJvBd8xHV+9Qbc+8qK/Vtplg3JwP2g4EExrVJ7wB8ST4TpsE2x92wTUYfJdxSt7PGQJjqYkM/xpcBVdrIiIiIiLaG/Fmg4iIiIiIAsGbDSIiIiIiCsSQMhtGvM/p+3pkFT0fD4pxhUQXe/IDPTdZAEEFlF1w0t5nVOv20c98Fnr6VFvdNP0MXMOcKaqt8WhvRsOdeYhaZgd4ntsuiiMiEreez0PFdBD0PDcqBKWOE3g2dGOP91nMnizI4+xBqhChiH7uFRVPQm0oR1DwFqvC79PPczo1+hnMyH7vVG3lTIvn9Svb9XO8RdBxD56hi2KG+vSz9vEG73Ox5Rx4djOun21Vz2WKiAn5OD/R/imh50UH7zcGFKWrhqJ+fsbAfuv87QVFOjeAqmF9RT0WbO/z7j+7sJOISEta58XyoCgUejb95Q3e4mhvbNJFycLg+X67GJYIzso1JL37or+kx7YecBiTSd3H7XExBwp89Rf0+vOgv61s14XoNnd6z79jDwYF5nYZd/2OwaPJ7n8oXZWMeo/X39bqAniN4PiJ6LFgUsZbyC7SD4qAxvR4AQvkrtMFSp2Qd1tRRhJJNuoxduLx+jl3SXiv8e0gD9kDsprga4W0uN5rIurLYVCAMBrX11KnqPeFnfWqikyaxTghMbtsFzoD7GKE4W6dZTAlPRZNSunzLTXFe/z2A7kLEPdV+QYRkbr44NewLCgsuKpDfzfI9usxJTNOnz/7W9v/oYWT1TL7NurzJ+Pq7512pkKKoH/0g5wP6kewqKRV1C8BsnS93n4bAkWBd6f6ejMREREREe0VeLNBRERERESB4M0GEREREREFgjcbREREREQUiCEFxIfFZ6g2HdNhFLsYUbRXx5FQERYEhS/ddLPndc3s/dQykzK60FYoqUNK0Zn7qzYz2duWjYDPCAJJKHhoF3BJgM+NCiOiMHjYAbEuO2xrh9VEZHLGG97rNv7CfKPFMa4npB2BwScf6wHBYjsMLiISBqFrmxvVoTC7X4ngAmTbXW+wtwuEL+dP0YW23jV9HNgS/cGTU6eD5bzCoBCWC8LZJuKjmJTfwDboW8Yu4ocKfzre95VDe77AlZ/CpsmQ91gkQMAahVK3ZPW4uD3rDfyhSRFQsVM0McT6Tj0BwYYd3rZiXm9XtFYHGCeColnTQVvaKpqFJvToAJNkZEP6umEHwrvB+9AYaIfsRUS6wOecPM471k/I6PFt12J4eVRFNWD2GAgy8mpCguVb9bgyARxTdI00zd6xYEq9DrhGW2eottQEUOivXW9Hodd7bPpBGDcEJkVITdBjSARsxzbj7ZMbu3Vhy27QF9Ixvf6iNUFB2NF9LREBxTrRuAgmzhAQJK82oVK/hEr/2E40mYgaeUBxuFBeX2+dgg5FN1nnc2NUT4aB9mVoa5tebOs6vVzGOxFFYtoCtUy+Vl+Dj5rVpNoOnlKv2hZM8E5kMEfPayDRbSt0YwEVC7b2IyrwCPqaiYJisTEwMYx9XQbrd60Qv1v2/x2Qf9kgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoECMKiKMsrh0OMhEd6EHViO0K2SIiNVYl1KYaHViZAkKJXf06JINCgp113uqodQcep5ZJzNGhNhSucUHF296Qd7lsQaf5QiBumoqjgLh3X6CQatmgMLheDhXeFjP4fad9vH1ksUdVwYSksMt2RlEhdDtYDILGBkxaEAJtxs+6Yjp85dbovtDn6j6fs4KJEzP6XDlipq6qOrc5rdpMXG9/bL+F3oaCDiCbjL8wuwp1o3A+2j9RUI0cLWe1oQq+Y4VTtqt+63ELVgJP6eNvh1K7crpy7VYQLC+A5PAOsFzMGldmgJDjwum6P79jsl6uNaM/pz3ZRRkEmu3K4CIivWDijA7rs6NAMwrLo9A4upZkrFAwqky86yUpWolf1Tkhz3nmonHL2u4yCuWD6yFqUz++RY89U2Ydqtoa8zrgGm/Q7+1Zp6tL22qadf/LHAAmZAHh3h3WJALoewCCJp6xJ1tBxz8T0+d1qB9MNAJC09VYMdxmIglvKByF3+2xHFxPQijUndfftdz2zZ7X5e2b1TKl9i2qrb+9W7UVe3WfrJ0x0fM6ntDX8wkz36XaDhyv+zKa2Kc54W0Ld6xXy5g+va1S0uO8E7OupQnQh8B4gNrQNVhA2F+9b5DXb6X6ezcREREREY1JvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKxIgC4ijEGbaDJz6DUCjEFrMCWSiAiKwBlXK39+nAzaod3sBka0ZXhUw3tKg2FyQHcyXdZqyQJircXRvT+yIeAjvWDF7hOywgHOQjwIXaqjGfGw15Q3mDVXIWERyYQgFlEI4yaW+bqnItOACdMzokiEKwdghxMujfE1K6DRUudkGF73L9FM9rp6grtKLPDaH9aPMR/N4de/fgscXXqgJlxHtuqAkJRH/mXNHf2ZQG4VK7DYWdcyBMjcZTVPV7phX4nQcmH9hnnH7fhBSoClzSAcxS2Nu/esAkGd15PTav69JjeNb6nCjEiyqnR0NgjAXvnd/iDbI2JvVndHbz/3uK3f/siUNERDrFu58a03psQBMI9ICJVdC11JZv1KHamQvfo9rqGieottTmNZ7XxtV9OTJ+qmpzJ+6r2jqiOkje0e3d/kREn2MRMLA01eix3g4A1yXARDcFEPYtg1A6Gjfs6wu4xleaGv+Gu43g8zvg2Ls5b6XxcvcOtUzvFt2Wa9OhfAdcON2Cd+wxoHJ3BIS6J4NJYNA10snpz6SWAaF0KYHvbdZEIwZVEAdtaEIjuByYuMlmnylDGQP5lw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJAjOihQPwMtfXsfxQ8jwaebQuDlYWtx47D4LHxcUn9bKVd/ElEZGOPfhZv+bas53VXXhfVQoW27OI+Ivg5+rjVOC4JnvEE+QynqLdVrOeOUWE633xkNvpBxsB+HhosEqii++Z//9geH28Cn9UF99j5eL1q0zkC/YELBZDVAdkZ1GdaQt7niUN9+jlTlAkp1+hCfDATYj1X6vTr/uegymV+CgOhPoSeWUX5LPAj7b3jpxglLE4ZMEd8PKdqPQ+bBwWa0LhlF/ATEZla533eFhUlQ8XtkJa0Hsum1nn714w6/SxvfMcavbKtupCWA3JDTsMkz+tIQvddmIEr6r5kF0JEGRe7MJ+ISNHV60KZDbuILIjTVZzd/wzYd3a2YH9QiK+tV/ej/pTeT2093lzjFlAYEhVSFNGF3CZNOUS1JcfP8by2C2KKiJSSdaqtx9XHub1PP+dun1ItIAOXjut+1AjOT/v6XVPURehCuS7VBo3Ron6+xj8f300MGCvcqM4W2IXsUL4h2Vir2kIgp4UyG+Haeu92WRkREZFQx0bVFsmBa3UMFNu1czjguJu43n4Dviva74XfrQGDwsJ+sjY+865+VX/vJiIiIiKiMYk3G0REREREFAjebBARERERUSB4s0FERERERIEY/aoxdnE4UAjNKepCQVEQrkEhQRsKZtcl9Mfqyus2O0i+aofers09OhCHgpaTa3WQNxPzBnPsIoUiIk5BB5JQcRuxm1DoBxUKAgFjP9C2VmOhP5u91WibCyDQuAMU37F3AcpBoiJrCdApG2MgNL7uFc/r4ppX1DLhpom6bdahqq0Qyag2O+SfjOkgWkxA8SA/wTBQLNIB70NF/UI+QuNoX1ciED4c9qajMHISfECQSbXnhZAaEKJEfbA2rsc7NEFFU8Q7Bka2LlfLlNbqfun2gnBsSvfBiFWIKp3WAXEU9J5Sp8ctO4icAe8LgXGxIar3BZqwodYKCod629UybqpRtVUSnGzB+mhokpO6uO5s/WUwcYu1sn5QDDCb12PB85t1/9gCrpstaW9/TkV1sNzRl2Apg9C/A6LLE62Chui4p8BMI3bIXkSkpmxN6IHC4GgMRNdgVJANhJrHJHt8R2Fkn22hTL3ndQRMthGK6f0bqdcFbJ2I7vOhlHVNBP3K9Ot1QSggbh17E9P9G4Xl/QS48+Dkh5MXoWLPgD0xEbp2j+QSvJf0biIiIiIiqja82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgrEiALiKHyrAiQo9ARCVNGiDuGkY94KiTkQToug9adBFVwQ0mzLequVooAcCiGiwF1DQi83zmqLFLJqGRwQB6FdGwoQRUAQDawLBX/swBoKFdnvwxXkgxMNDV413O6TaF8ao48VqsBtQwFlFMhC4cJQXh97Y4XdQpkGtYyT0IEyF4Rg+0t6+0tWoBYdL9C9Meuc9RsGRxWBUTgybL0XnevRKghQGrH6mI8QXTMIZttjgwgO+9rQMYyAsdMp6PBqqAcEWjs2e16W2nVl8HJHm2ozaBILwPR4K+2GE7oa77SGKarNruYtos9RP+esiEg92Nd1qDy4NVa4oHK12c3/7yl2/0tGdIcolO3P5u9YoX3eYM1aYE+qIiISBRNioKr2G3v6VVuXFS6fCELkKTBIwW0FxzltHWc08UQBXPdrwnpBJ+cNvaOJXGC1ZjRxS0R/TmNd01H3ti83e7oPqvEPUJO0gKrZThlU0gaTCYVS9Z7XYVBBPNSgJ51AQW/IvqaAADriJHUYvJzQE2S41oQYO/p1nyn06z0aDenlUN+1JcB4kATffZGIj8mE7E0YSv+r/NWbiIiIiIj2SrzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCsToVxC3GBRaRjmUsg7yJvLeQGMChVJDOhTWCCrxTgMVdaXBG9KC2wp+Jgq9OkUdAJasNxDnlMD7UMgMfCaxQ1Yo5I0qPyMosOZjmTEJ7JMUqJo9BVSAt6FJDJw8qKTcrdvMtnWqrdTuDeeavA5Qun3dejtAxdQJdeP1z7T7cwmE8ow+WVCgz+4PdpjxzUZwfuql4HKu9XuP9GAzAVQz6/OF+nV/cEAYMgomiwjlvW1OSfcRyet+6Xbr6tflXt2XVJAShPDDzZP0diVBpWfQL+0KvW4ZVADOblNtLShAm7Sq8YKJBlCoFoXq2/pAANN6XQuCyckx0C3rI97xrQMERIsgQJuM6M+LJlaxZQt6PEWVxtt79fUvae1jFEBHE3NMrUuqtsl1us/Ykwgko6ByvAOuy7mcarPHQDiBALp2+6gGjaB+W4lJCUYMfH64n2I6/F222pwCuAZH9ft8fz+ylguB67kpg+9o4LsimozGHq9r4/p93Xm9/jCqBO5jcoAwmKAAZMZFwHdYezxFbxtJ/xsDQycREREREY1FvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKROABcQSFSx0QWDOOVbUYhMhDILSLqkeiILb+gYMHiHb7VhgMC731axFxIzrkiCos+9kuuA1vd6jCc0GH/2KgzQ5RwaAvqvae08u5RR3ICje0eNcFqoWjPuOE9XFG2+ZaaVYXhJL9hMHf/AE+hooRTCpgv3O0w2l7lB0kjekwK5osAu1j+70hNGFFFIyLtY2+2tS6QN9C4woKZ6PtL9vvRdWE0TkEmtQY7uhzNozWBdrGx0G1XyuM2ltEE26gnlld7GMzo1Z/jq6iPqY5EOqeM87b//YZp/tyvqTPTFSVG4XGba7R66qJ6m3NxHUbDPRb6dh4SK/fGHDe1TSoNjQxjF7I5xiIlttbJmWxzzf0XQWEwZHhfm+Dk/iAySlK272TtJQ62vT6o3qsC6FrNWKN8zHwfa+lTo/LjjWxhojuk+h7NOq3BVf/zBj6XmQft2FObLA7e0nvJiIiIiKiasObDSIiIiIiCgRvNoiIiIiIKBCjntnw81y1g57hQ8+OW20mCgqpoJwFyGz4Knjn95nJERTPU8Az+cbH++DnBmDxtVF+Fq+qoeca4z6ft7TflwLPvKPnRdGxmYze6uMYgufNy6hIEsr+2P3Ib/7IxzLwyXWfmSc/6x8z+QzA3nYH5BvQOY6OoVP2Fiorx0AxPfRcM8ouoOMzmuOin/f6zZrBccvPdoBigGhd6Jll63VqLBeW3BX4rHVRfdzrYv4KdarVoywDyiSh99oZDb+5SbQcOA2MWDk18D0DZkjR+q3zGI1R8H3DzIKO1THQ3p+wphz4cCaWVm3RslXE1Ohl4LiQ1wVS0T4PW3kJVKzUlHTWw4AMpgOyHVL09j9YDLVGfyaYibO3H/VbcN6FYygDPLzvnSOxl4ymRERERERUbXizQUREREREgeDNBhERERERBYI3G0REREREFIjKJIVREMVPITugIiEqFL5EhhnQ9cOMIIg2VoNnwwFDfChgPcz1uz7fGAIpOb/v9bMuxA7hOeDnhYdZowzu172lKNUo89sH4XvtcRGc4767kd+JLXy8b0QTVKiVjaAQmvqBHANtIzpX/QSZ0bUbtQ23746k/wV4DYZDp89ifW/3/uf3uqP6FupWoGiqyYxXbbCQaqN1bPxM2iIiTimv2lwfE12UUFHTYY6lqBgqCnmHBRXM3vPXan47ICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCoSvB4fN/xXe6enpGZUfOszHxKtHFWQ2fBfoQs+LjtJ27OwPxi7MNMr2RP97W2Q2wDLo2dnh7gu/5/Vo9ZY91f92/RkVHwOHm7sYyXuZ2YDelv0P8duvKtH/hnsN9pm9GK6xNgZWrP9ZxxkWkERvgwXvdHE+dTH1eVF2yjqzYZzBMxsmrJcJOrOBiv9V4jugr5uNnSucPXv2CDaL9lY9PT1SV1cX6PpF2P8IC7r/7fwZIuyDpLH/UaXxGkyV5Kf/OcbHLYnrurJp0ybJZDLiOGP+7xI0Sowx0tPTI62trRIKBfdEHvsfIXuq/4mwD5LG/keVxmswVdJQ+p+vmw0iIiIiIqKhYkCciIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQLytbzYcx5Hf//73ld4Mepti/6NKYx+kSmL/o0pi/9tz9tqbjS1btsgnP/lJmTlzpsTjcZkyZYqcdtpp8uCDD1Z600TkzSnDvvSlL8nEiRMlmUzKiSeeKCtXrqz0ZtEoqfb+97vf/U5OPvlkaWxsFMdxZNmyZZXeJBpl1dwHi8WifOYzn5H58+dLKpWS1tZWOf/882XTpk2V3jQaJdXc/0RErrvuOtlvv/0klUpJQ0ODnHjiifL0009XerNolFR7/9vVJz7xCXEcR77zne9UelMC46uo31izZs0aOeKII6S+vl6WLl0q8+fPl2KxKPfdd59cccUVsmLFikpvonzzm9+Um266SX72s5/JjBkz5Itf/KIsXrxYli9fLolEotKbRyMwFvpfb2+vHHnkkXL22WfLxz72sUpvDo2yau+DfX198txzz8kXv/hFWbBggXR0dMinPvUpOf300+Vvf/tbRbeNRq7a+5+IyJw5c+R73/uezJw5U3K5nNx4441y8skny+uvvy7Nzc2V3jwagbHQ/3a688475amnnpLW1tZKb0qwzF7o1FNPNZMmTTLZbFb9W0dHx8D/i4i58847B15fc801Zvbs2SaZTJoZM2aYL3zhC6ZQKAz8+7Jly8yxxx5r0um0yWQyZuHCheaZZ54xxhizZs0a8973vtfU19ebmpoaM2/ePHPvvffC7XNd10yYMMEsXbp0oK2zs9PE43Hz3//93yP89FRp1d7/drV69WojIub5558f9uel6jOW+uBOf/3rX42ImLVr1w79A1NVGYv9r6ury4iIeeCBB4b+gamqjJX+t2HDBjNp0iTz0ksvmWnTppkbb7xxRJ+7mu11f9nYsWOH/OlPf5Lrr79eUqmU+vf6+vrdvjeTychtt90mra2t8uKLL8rHPvYxyWQycs0114iIyEc+8hE5+OCD5Qc/+IGEw2FZtmyZRKNRERG54oorpFAoyCOPPCKpVEqWL18u6XQa/pzVq1fLli1b5MQTTxxoq6urk3e+853y5JNPyrnnnjuCPUCVNBb6H+3dxmof7OrqEsdx3nL7qPqNxf5XKBTkRz/6kdTV1cmCBQuG/qGpaoyV/ue6rpx33nly9dVXy/777z+yDz0WVPpuZ7Q9/fTTRkTM7373u0GXFeuu1rZ06VKzaNGigdeZTMbcdtttcNn58+eb6667ztc2Pv7440ZEzKZNmzztZ511ljn77LN9rYOq01jof7viXzb2PmOtDxpjTC6XMwsXLjQf/vCHh/V+qh5jqf/dc889JpVKGcdxTGtrq/nrX/86pPdT9Rkr/e+GG24wJ510knFd1xhj9vq/bOx1AXFjzLDf++tf/1qOOOIImTBhgqTTafnCF74g69atG/j3f/7nf5ZLL71UTjzxRPn6178uq1atGvi3K6+8Ur761a/KEUccIddee6288MILI/ocNDax/1GljbU+WCwW5eyzzxZjjPzgBz8Y9rZTdRhL/e+4446TZcuWyRNPPCGnnHKKnH322dLW1jbs7afKGwv979lnn5Xvfve7ctttt4njOMPe3rFkr7vZmD17tjiOM+QA0JNPPikf+chH5N3vfrf84Q9/kOeff17+5V/+RQqFwsAy1113nbz88svynve8R/7yl7/IvHnz5M477xQRkUsvvVTeeOMNOe+88+TFF1+UQw45RG6++Wb4syZMmCAiIlu3bvW0b926deDfaGwaC/2P9m5jqQ/uvNFYu3at/PnPf5ba2tqhf2CqKmOp/6VSKdlnn33kXe96l/z0pz+VSCQiP/3pT4f+oalqjIX+9+ijj0pbW5tMnTpVIpGIRCIRWbt2rXz605+W6dOnD/uzV7VK/lklKKeccsqQw0Hf+ta3zMyZMz3LXnLJJaaurm63P+fcc881p512Gvy3z372s2b+/Pnw33YGxL/1rW8NtHV1dTEgvpeo9v63Kz5GtXcaC32wUCiYM844w+y///6mra1t9x+Gxpyx0P+QmTNnmmuvvXZI76HqU+39b/v27ebFF1/0/Nfa2mo+85nPmBUrVrz1hxuj9rq/bIiI/Pu//7uUy2U59NBD5be//a2sXLlSXnnlFbnpppvksMMOg++ZPXu2rFu3Tn71q1/JqlWr5Kabbhq4YxURyeVysmTJEnn44Ydl7dq18vjjj8szzzwjc+fOFRGRq666Su677z5ZvXq1PPfcc/LQQw8N/JvNcRy56qqr5Ktf/arcfffd8uKLL8r5558vra2tcsYZZ4z6/qA9q9r7n8ibIbply5bJ8uXLRUTk1VdflWXLlsmWLVtGcU9QpVR7HywWi3LmmWfK3/72N/nFL34h5XJZtmzZIlu2bPH8JpHGpmrvf729vfL5z39ennrqKVm7dq08++yzcvHFF8vGjRvlrLPOGv0dQntUtfe/xsZGOeCAAzz/RaNRmTBhguy7776jv0OqQaXvdoKyadMmc8UVV5hp06aZWCxmJk2aZE4//XTz0EMPDSwjVjjo6quvNo2NjSadTptzzjnH3HjjjQN3tfl83px77rlmypQpJhaLmdbWVrNkyRKTy+WMMcYsWbLEzJo1y8TjcdPc3GzOO+88s3379t1un+u65otf/KIZP368icfj5oQTTjCvvvpqELuCKqDa+9+tt95qRET9x9/q7T2quQ/u/Isa+m/X7aOxq5r7Xy6XM+9///tNa2uricViZuLEieb0009nQHwvUs39D9nbA+KOMSNI0xAREREREe3GXvkYFRERERERVR5vNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKRMTPQq7ryqZNmySTyYjjOEFvE40Rxhjp6emR1tZWCYWCu29l/yNkT/U/EfZB0tj/qNJ4DaZKGkr/83WzsWnTJpkyZcqobBztfdavXy+TJ08ObP3sf/RWgu5/IuyDtHvsf1RpvAZTJfnpf75uNjKZjIiIrFy5cuD/90ajfb9uRnl91aanp0dmz54deJ94u/S/MugwEdQpywVf6zPh2Mg2qMrtqf4n8vbpg+Qf+9/o4zV4aHgNHl3sf0MzlP7n62Zj55/NMpmM1NbWjmzrqhg72vAE/WfVt0v/483G8OyJP+u/XfogDR373+jhNXh4eA0eHex/w+On/zEgTkREREREgfD1l43RNuy7R+P6W84tDfcnaI7P+zGwbb4+53DXHwKHzue63i532yOVK3n3VNnoPYeO8Y7+sq/1R0Ped0dCem1h8ANq4/ovFv0l3f/C1pHOFfUy9mfcnYL1Z5emmrBaJhnR/Q/tH/AxaTcCHyt9bcTwx8A9vS6Obf747lfD7Uc+r8GOtX7jt3/4FApy/bwGD9uoXgJQH62S8c/u377X5Qfqfz5Vov/xLxtERERERBQI3mwQEREREVEgeLNBRERERESB4M0GEREREREFYtQD4oEGf/yGzkp6alDHBaFde30o4IN+ZtnfdpiIFeRFgR7QZkI6fKuE9bb6neoUHiMruLS3BNjQZ82CoHQChJttuaLeKyg03lvQ61++Lava+orePpkHIW+0/vas7t81scH7TNnV6+rp1305DBLcLbVxz+v9mlJqmUm1CdU2LqG3C7WFcl3ebU3WqWXGKr9joq9zzm/wMejQ5GgaxdCkM4J17c1jnjKKwW8VghWB10h4Dbam8UbrQtdgp1TU6xrupALoeguqIdvXVxOJq2Xg9Tys2xwf4fKx2h8Dnf/D77iG+hH63mYvB0Pe4EgMdxIiND6hPommkrWXA/3K+Jy0wM84Odr9j3/ZICIiIiKiQPBmg4iIiIiIAsGbDSIiIiIiCkRFivr1gUJiqXKvXtB6rswp+3xOs4wyGz6f+/SzfvSMXRQ8v+njfW5cP/teAIelz8oZ9INcgAHbWg+ej4+BZ/LtLUPPzo52waUg2J8sV/b35CGIM0ivvc/LaJ/72666uD6mcSsnEgbPaXbkdB9tF92/N3fm9HJWtmPNpm61TB/If8Tius/sM63B87oA8iVdIP+BchydSb0vZtZ5MxqoMCLKeoxVTqlfN/p53tbvM/Pu8J9t9gM+x+xzrFRv8/N8MmpDz9Wj942g+NpeYQTFcNVz7mgZkMWA/RtlKUv5wd9XyKsmU9DLGZjLtIvh6uMeSuhrsMSTev2xtHfVKJ/h85l8P9B3FvhMfrXzOy74GJ9ghhYZZm7XQd8d0fdEtJyP88zv+GTCUd1mZ4BRRtfP+0Tw/re2A2VvRpLjeBuNuEREREREtCfxZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQI0obweItPgI8KXfwoBj8eSiU098z6PveXBAUNrHWZ7KdahnTr4ProdpGvVwatNV4Q7VuolYt05UHRWScwYNGqMBcGNw6bsmiom16uQYrfJuOgiBdoc96rUPJFWf1vyQofFMCAXkUEK+NefdBIqLfVwQB9CQoEFgPQohpa/1xcGBQUb88mGDBDrOLiKzv9p5Tb+zoU8v0FHT/QOHvCRnvBAgtKR06i0d0gBuF3tNRvZw9aQQKg2/r8wb8enIg8FcF/BS1QkFPGAhVk2SA8XUUQ7uhIjinwbgrJZ8FVsPe4wiDjyDoiJYTK+iI3oeKpeGf6aPQ2iDB3kCLl/nkJ8QJg6uoeFlR9w84KYtaaPACeCICA62udY00sRq1TNnRxyoPxl00z0Ai7G20r2EiItLfpdtAwNhE9WQXahvA9xjU5qb094Ueawx3jd6vibD3cxd8ToASFF/ngN8w+HAnKAB9FI11Dvi+4hS9/cFkdV9wezpBW4duA5MW2BMUODHdh0KpjG7LNKi2cLre2wDW5cb09wx0TpkomADBRxFquxjgUMZA/mWDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQIwoII6iSXaABAZ6CqBaOFoXCiba25DQ4RoUvkSBGDtY5IBwDQpawiqQIHBoBxqzIMTbASolo9Duui5v+CgPQrwuCBOnY3q7JqZ1tfOYFaSrASHnYsQbNMpHfFbz3IPsYx8CQbFwRAerIgZ8FuuYxsM6DpUL+QvoRUEoPWYHE8FxR0G3FAha1ifrVFtzjbc/79eozwEUMERBS7t/IKioNOqTUR/r8rNd1RDQ9cVnQNcF/VJBoWhYdRmFIXVQNWT1wdL2zWqZcvsW1eb26Wr0qGq5E/GGs2FAMlOv21J6Mo2QFZA0NbrPm4ge2wQFe1FA0l5usGrNw6y+PprgNdiuuuyjcreI4KrIfis229uFQv/g+trueo/X5h067Lu5R39fsK+HIiJbunWbPdnFxHrdF945qV61TarV25+yJk2JoDEd/foW9JMC6Dr2mIeuN3Gn/Jav9zTY/9RCYFyAk06A/mdNWIEmpbFD3iK7mezHR9V5FAYvd7Sptv62bbqtXY+J/e3e8TUU1WNKvF5/h0221Ku2WHOL53W4Tk8yAIPl9U2qzU3osdPPZA2DTZrxVviXDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiAIxsgriIPhTsu5fYnkd7vJTKRf/QH+VSl0QGi84ermSVY1TYvVqGRSMRflWVJlUfSaQpiqB0tWbe3R4ryPnDc6hytUbO3V4KgdCx8jBU+o9r0+YoYNGdlVnGIbbg1BA2N4tasICwf0PhSjFeANlqsKmiCRRiBSsP9SnK907+az3tc9KqCaqg1socFdT8q6vBiyDKkabkK7ebf9MNw4qlYLQLQrZGhSEttLfKHhoBzTLoMr9nobGwL6y97M4IHFvjK5qnQOTRZSt1YN5G0RErysaBRXeE/qcTuS9AUZ0QUAB8WKXDkOW+3X/iqa8xzqcBpXNI6DCdwL0LyvMCStjo8rp/frcM+gaZJ1/sMrurqHJEYQlR4uf6uCwCjgKg6OArrU/DTpWsbRqQ+NDzuhxpSvnPQ7o2teV18cq7HPX19d4tzcBTqAymMSiK6/7UZ81wYvfiTRyRb3+ZHTwyQXCaNywAsZoMpw9CZ+DpUGXccD3wlCfrsrt9HrbUFjb7WrXbSAMHgKTU4SbJnpeR2bOV8uY5n1U245efUy39+l+WpfwHp8ptfr8iXdvVG3OtrWqrdzunbzD7dXjGgq4h8D+iUycrtrs8cxF37ft0PgQJsmo/GhJRERERER7Jd5sEBERERFRIHizQUREREREgRjSA3+ODF5IK1r2PiuHnjkP9+jn7lAxJjuPYaJ6GZTP6APPQ3f16+fpslZlnXhEf7rmGr2LomX9nHu4Wz/XbD/7XpvSRVjclH7+vuTqZ4Vz1vOi67v0NmT79bO5z6/aodr6uvVzse1Z7/O6R02r19tl3vp1NbAfmYXPlJZRZgMUufLzPKLR/RsVgnRApkcVlQTFieBz4SiHAooduRte9bwurl2hlsm16edkY7W6T8amzPS8joyfqrerVhcPgud1Qhdss5dDBYXs56H9FBoMWt4dfBvQ89+oyGN3Xvc3e9zqK45uEa+mlHesaWo8QC9z2HTVlunWxf/Q+eIr64P6CMhHlax+jwp84SKy4NyD7/VuP8wg7foZK1DUT12D/RSM9LudoDCtsdrswl8iIv2OPn6oKGc3OA/s02B6g97ndlZLRKQxqbcVnFIwX2lDxXZRJtJergd8HvS+rb362gKimrJfk/fcSOiIi/r+tadHQF/9z34PONdCuS7VZto3qLbSNm+eodSm8w3GDraJSCimvwM6oAietM7xvFyfmKQW+Y/H9M/81R/0tXTLS0+rtnEzF3hen/uBA9Uyn3jXFNU2qUX3+XDIaguBYqsgn1HuAVmYGv29OZTyntuwoLVq8Y9/2SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQo18Rxh08nObmQKG/FEpDee+FTKROLZIHBa1yJRD4AomsXMkbgEmDUFHS0SEZFAaXbh16d6yiapG8LsIyDgTuUuN0SClspd9QIaIC+NwlECjd8MpK1Va2QlY9x+lCNmmrEFEe/Lw9CYWV7MAcKhbpuKCAHypyZfddvwWU/IQ20dtAsT43qfu8gEKWkR1rVFveCoRv/etytUzPBj2BQO1UHfRuSXhDoKF0PdgsUKwPNIlVzBBxovqN9t6PSAUCusa1+oWOaPb7OC9AphGOUTusSR+2gKJnKDTud3xIRr3jbktK962GpB4Xi26rakPb1pb1tsUiehKLqfX6vJrTiELB3m0dB84NVFwVTVLi6xwFYfNKB8QVH0VLUQBfUOFOsK6SVbCvp6A/cwF05izqk2B32cX5EiDRPSGl+0d8499VW3Hda/oHWKFaB4xRdfscoto64s2qzd5+NOnMGlBYt71PX1tSMf2Zptd7w/GFsj4enXk7pL6H+6BxB+33jj0BC7q2gjZTAsUn7ffFwXUBBL9DtbrNmbFAtbXHx3teL98y+LVJRGTRIfo7WuGgM1TbgVPrPa9P3EdfWxvi4LsvGHqcGu/EKtG4nkyhBIpuuqigMCj66stwJ58Q/mWDiIiIiIgCwpsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQIx+QNyucgg4NWndCMK3djgWVS/tyumASj9IoqFqn5utQGMyooM6uYQO3KRBJXPHDsaLSLnDGxo3m1arZSSsf2YCVGfeb9I8z+vG6fVqmTX1OjDU1q3Dka8++pRq2/jyK57XudI71TLRkHdfRFDJ1iqDKog7IDxrhypFdGDbhHVfQJWOUcViARWx7a1AYXA0AUI8pLe/DCqHdq/2VnnuWquXMaDfOnZqU0TECp6hoJ6A0BkMp4JKxSqMi8K5fgP6ATJOyBOkRUFsGwrG5kBjGQTEa6wAt/1aRKQIjmEIVDbP1AxedRmFzdt6dZjzhfWdqu2Vdbot2+kdf1K1uj8cup8O4yIHjvdeN+IgTJwC55kDJuYIoWrndkAanNudxX8s01Osgt/VoYCmHdAFwW/02YphfU539Hv7Qx+oto36x46cDvtGwbhSF/f2yQlpvV2h/m7VVmrT1ab7Vr6i2qIp72eKTp2jlgmBsQZVQLc/0+bs4BMiiIhsBqHxAybpsT4e8fZnFLxvqfGOw6H4Hu6DTgj3p8HeYwP7HIW6Q+O8Ae4ouAaUU/p9hfR41bamS/fTdVYgPAH66MUgDD4ejKVhUBXd/pwGHC8DJiFyE7p/uHFv1W+noCdaCoOJH0Kogngipdrs8c84o/v9rgpGSyIiIiIi2hvxZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQI0tcohCnj3Cam9RBbxS0ddPe4GB7Xoe2ukDwewsIab3RoUO7G63w4sZu/b4FEzKqbX6LDgzV5HVYx2x43fN66yNPqmV61m1TbdGUDlE27j/Den2AWmbCrANV26zj9lNt7dnFqu2xh73b2pbVYar9Gr0BdGeUA0RDhcLfw63qa0BVbjfuDVGVQmAZkA+OoTB4XE+KYAeytud0X86X9Dk2JaoDh0Wrr4mIFHq8fbKmSYfCEo21qq1hzhTVFps+19vQMkMtUwITOJiEXj+qKBvu9YbXVRVaEXVsURXaPW1jjw7C2hnDGJhIIQp+zRON6cYaK3QfBetqKYFQLVjODuOKiKSsn4ny7pvBWNBUo3/mpHG63//tDW+F+jDYrhiYmAMpWSdbHoR4w2DyhJqonjjDjeqJM+xrEKqondllfxlwvPY0NLGFnyq/xag+L7vBtdS+vnaA4Pf2Pt3W1T94NWgRkbxV1T5b0J8n1qzHrYmTZqu2ZLZTtYWsqsuhKfuqZfoy+nq+vQMEva3vFZvA5CudYF/UgXMlAyqIJyPe/lQDBomsFdDvBYH9PcpP/0Nvi+vvVW6jniiiaE3Sgj5ve05vQ9tm/X1sfZe+btZaY+Jhk/V2ZdY+rdpKoFq9Sep+Gp7prVrel9Rh9rZe3WcKru4f9oQ8teB7Rn2LvgaHU2BimBK4dqKJW9QbWUGciIiIiIiqDG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAhF4SV4UvDUxUL0QhF16rNwPCgflSv4q3i4D1W1fWuOtrIgCz+vmtqi25tRk1TYDhBDL3d5w5I5XN6tlti3frtpK/TrwtPnZTZ7XTfutUMs0zNEB9KbjjlNtP/ng6art+9O9wSIUAM1ZYb7+UoXDaYiP6qawMiY4fnb17lwBVNsGq8qDbQiDBbutMGR7nz7u9QkQFOtYp9py29pUW2aqt4pq04H76O2q16G8yGS9nDt+lud1V0K/z642LCKSz+lzsS6uA8ETrCCgU9Rhvn4rXJcP7fmAeNF987+d7NCeiEiXdf7WgWOYiOj3oYrY9inWVKMryqM+mAbh0saQDr2Geq1+AwJ/M2rBhAfj9CQW3TP1uPi0VSn5tXYd3EyAgPjkjK5mbe9r39NT+KygbQsVdB+UXd4Wdis/QQEKsdv7xQGpf1StHgXu7e4dDemfl47p41dGMw0Abb3ePvm0NaGAiMjEej02X7BIT1Ax7bCpqs219k9O9PmzAwSM7eA60pzSfWhCRp8XaRAGB8OGZK3rC5p8pNmqXI2qsgfKuJ4xwtckLeA7YHmcvn6sA5Nt9Fgh/K3gux2q2p4H1dfjYF8tmOCduKV264tqmcIa/V0r3KC3X+a8SzVtDtV7Xj/9hq7m/fKWHtWGzs8Wq2+hyYtmNehxcxyosB4CExqp44bGFmviFqesr/m7w79sEBERERFRIHizQUREREREgeDNBhERERERBWJkmQ30LKzVZuIgnwGe4SsYva5+K4+B7oxc8Mgget5yxcZu1bZ+xVbP62hCP2/ZNa1etaHtQM+YFzq9PzPZoJ89bdpvnGoz4NnZeK132yIJ/expsVcXGcqvWq7aEiH9jO0/HXKa5/V68PxknfUjHb0Jexbqf/Zzh6CDOK5+ztCN62cd7QJKBXBc0PPyIfA0uSt6OzZ2e58/dcFzzrPH6e1y1+l8RnKfuaotamUvXJSVCuuD6IJCfF1R+9l73dfsolci+FycVKs/U8sE77aFu3S+qZzwnit+nwsfTdGQtyBfQ0KfS0XreVu0nSCWBfMfcSvbEQbPzKPiX+kunespLXtItfW87i0Gmd2oi4yicQUVg5xwms6CHbzP8Z7XqNggiKrAnEvK+pz2axGROCjqh8YJE9U5FLuIp4T0Nuz6WH3BGTz3ETg0BlrbbUCfQflE1I/sfNXEtB4v0PPluRLIhICx4MHV3ozGcy9sUcu4YP0Z0D/2bdaFU+2M0/i0/tyoH42r0etPg+8ythjozKgNRArU9aUurrfLriO5x+tKOiFPn4OZISsPVY7o8X5TVg+AG7r09aMr712ur4gKT+p1JSJ6u/Zr0v1jYtg7tpl+Xfw5Ov9I1dbfpHONqzt1nuSJ9d5M7h+W6etaV4/+3M3gu2LNTPv6pxaBbSgzI0bnRPT79Dlgf3c3Uf+5Nf5lg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokCMelE/Y4dKUEEXEGIxIERph9PyoBBWAYTHUIioBNoSKW8IJzNOh3KO3KdJtU1O61Boadkzqi23rdPzunbGRLXMlBMnqbZQul61iRXqdvt04F1KOtTt1OjCL05SB93CPd6w/NTMeLWMlK0wkAtSrgFyxCpYhQoK2YE1EI5EocpyCExaYBWwQUFI1JdR0HtHTh8bu/jkrHE6tJrId6k2p0kXT5M5h6umLsc7qQAqLBhzQKFMVy+3o9e7L9Z36dDw1l4ddEP7LApCwiXHCoiD45Z0vOdw0fFfUGi02H3QDnCLiNRawU6UY7dD5CI4KG3vKxiK7gFh+hf+V7Wtu+vPqm3V/as8r1fu0BNdZEGwd+EEHbasA+Nb/f4neV5PTOtJOMKgP4CPqQL0KFCPZmwwoICfierQat7q9yj+HZMS/P+KQeF3EOy0xYtZ1ZYogaKPOe/44xT0+1CodlxcX0uLTbNUm62zTV/X8lld6O/ev+mx8ilQ/O+Di7zX1/2b9fvSeb3+ZjAuGnsCGRTOR/seTFKC+mSf631vEowt6pq7h6/Bw9EPLpLt4HqIvrfZ4x+6dqAw+JQ63Rcm1ep97rje/uxO1BOtZKN6MozVYIKUv23S1+r/sSY82LJZB7OTab1ds0HBvkMn13teT6/XY+k4MGFJqE8XEnTs73IiYkCQXy/kvvXrt8C/bBARERERUSB4s0FERERERIHgzQYREREREQWCNxtERERERBSIIQXEzf/9t5PjJ6ALgihhEKyKFnr1qore4FmyrMNQmYYpqq1tvA7XbD5ggmrL7tPoeT2vVQeB3j1bV/iOrHxMtXW+8opqM2Vv4CkzVYeuY/stUm1O01S9LisgHs51qmXcXh2uM6BadqhB7wvXPpbguKmwP6pMGaBh9T+kpMNdInWqxQ5UF0UH3VDQN5vX+9wOg4voQFwGlIM1IPDa2zhbtW3p1edG0fWG8Man9OkeBUncEgg09lnV1FFV7M4+HfprBoHgmqgOseWsEHIM9S27ojOo8Bw0uw+i3mb3GzhHAao8jwL8Vmo8WtRh3HCPrijfvep11db24lbVtqLdGwh/LeuvIuxC0BabNE219VjHFU2e0JUDIdekrlSdjHrfmwfBU7T+aEj3twjY13YTHF927XMV6H+KjzHQKeljGurXYVYUJC1t8Paj0uY1apn+bTpgHYrqfVN78pmqLR3zXhMjMX3e9+b0d4MtazpVW808/d59m7wTT2R6N6llyi89qtoM+JmhOu/3hXBDi1rGqdETJ5iYbnOT+npTk2q0FgLnRRWMgUNVAtfIZESfky0gKG1fd+IgDB4NgwrwCT1+oPkkeq3wN9rW/qI+x9IxsP0pfa3LFbzX+HhSH693L9KTBL1vrv6uOKXW+5li4Bx2uvUEDg6YOAh9r5Cod/IEg/rWCPof/7JBRERERESB4M0GEREREREFgjcbREREREQUCN5sEBERERFRIEaWLkIhTjtYDAK7TkGHHEM9OrxodnjDXOUt6/QmNOmqtUdM2k+1vfPY6arNzgLFujeqZYp/vkW1bXjyeb1crw4dN8zxhtcTs+erZcwU3VauaVBtdhDQJHSY3Unr/Yqql5ZB9VI76IPCkWYE1SMDgcJJ9jaBcCQKTIaNDuMlrfB0ARSsRtWht/fp9beB6trpmLXPQWarJ6zDhVt6QOALbEdj0htiy4Dd5cCwvO4feSvoi6qR79OkK9PDIB0IAhassK+DwpH22IImMdjD4iG94+0q3/2gAjeqBI4qYtstTlFX+C736GAvCug2zKxXbQvsKvZtegxJZHTwccGlR+ifech7VVtHv/ekQecGyHlLIwiI2+O13WdEREBWVBzQV1FYVLWB7ao6aAy0zwswTqPQqNu1Xa+qwzv5QLlP949Svz6m6UYdgEahaLfDu5MjYPKIRF2Taqtr1JXA332Q/i4w06qybF54Ti2Tffkl1YYkGtu96yrosTMycbp+Y1RvKz4m1vrAhDgmZq2r0tdg8B3QsarOx+O6MnUaTIaC2uxz3A6Mi4iEHd2X+8t6uTYwiYpdfbwurrehJarX77h6AgG3SX8nm1Dv/ez1rXryotPn6okGZsXAd+Tt27zbACYLgdD3dFAtXH2/Q1hBnIiIiIiIqg1vNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiAIxpIC4Y1xcVfUtlBxQtRiuHNz3WCG2vjVvqEVyf31Wryr8P6otnBi82nWhW4d+utdsUW35bh32bZyrq0DWTPdWAg9NnavXVdOo2noKeh9HrDLEiaQOkYdBsBwdLwcEa5289dlBOE2FESsdTvPBARXUUVuov0e1Zaz92V/SidEdIByJqoW/ukWvf2azN/ydK+r1h0BK1a4qLSIyIQUqLmc3exvyYLKAWh2qzPbp5ewq2JNqdWgYVXeti+vzPwmWQ5/JZvfloY5Fo6FQNp7QIsgTSk3E+1nCYGxDledjILWsJi4AnzmU0MH85IxZqk2PUCLj5nZ614UqP++3j2qLvus01ba2pIOwvdZYFgXl1FtTemy2JzcQEamNe9tQl0ETCzgumEgAhPbtTg4r6KLrVCX5OQfQNqPJFUBoPJT09q3YJN3XEnP1NcyZur9qe110EPb3z7/ieZ3P6e2avN8U1Xbe8bpPHjtDX/8yjrU+0P9CMX9fg+x94UT0mOsk9DlQBsF4E9Hjp1P0fq8wYfhNqbKM6+1zoG+5saTnNZgLQ+ri+vxGE6TYE5905fW12wWzo/SA2Vw2Z/X3tqYa7z4en9LB6XDHBr1hQCiuA+JzW71tB7fqZWbW6fEvsvEV1eZaVe1dMNFNqHac3jA0RqAJWCxoIh0T1fvHryobOYmIiIiIaG/Bmw0iIiIiIgoEbzaIiIiIiCgQQ8psGCckZpdn9GAewGpDz9W6oCAdagunmz2va7ZvVsv0rHtSte14VS/X34GKlw3OAR+gboreVruAn4hIfM7BntelhslqGZTPQAVpQmXvduTL+plE9Nw7eDRcoiH9vGharMxG2MfzytX2/LJPqJBdKAtyLFbRnNp0q1pmHSgstrlbr/+F9V2qLWc9V7ovKIo3o14/I9lcA3IQXetVWyjvzYmUxk1Xy2zs1X1tW69+dtvWWAMyIiCLgdt0Py1YHdWAokMqRwTOgaDFwo51nqEHjb37NAYqzYX95DNEJNTnLdgXsrNVIuJE9TO/4UadxUmqFpGa2d73hsfrcaw0baFqW5nV68rm9fbbmZ2mGr0VDQn9/Hasr121OT3DK+KIshfomXkT8T5vXwZ5w/Cux7YCmSEj3lqDgyedcCFIlNtDNQzDzd6kj9M0VS3T2zBDtT26rlu1/e7vr6u2F5d5r9XRhB5X3nf4NNV26mydE5mY0sfLsc+fjM511EyfqdoQu4BwqFZvg5sCbahIL+iTjl3YOKr7aKX5+Q5oC4HrbQpkQp0yKFZrnafxGp2JQcUAUf6xDLId9nvTRd1v0XlertPfBdp26HPKzmjMa9bX+HCXLibt5vU5K1bW1AnpcVNKIIuBMknou5vVZtB3wBEYm98WiYiIiIio6vFmg4iIiIiIAsGbDSIiIiIiCgRvNoiIiIiIKBCBF/VDRZa6SvoeJwuC0n1Fb5hwzpFnq2UmxHXgMF7/jGrrWLH2LbdTRCSa0eGjZKMOg9fP0SG56AxdxMgd5w2E9zs68FUG+xPkmKTgWgXNXBCyBWFlJAyq56SsAkLGR4BoTwfEHfEXiPTwW9Cmp0M12cXSUCHFcUl/hZd6u3RI7uV+73ZMAwHxCWndZ9DxQ8HEvlpv/1vfowN4azp04DgNCvHZxY9Q8LsGVG9Cy6EiT3bXdeN6X0jYG2Y28Soo6gcmZbDPHTRmRkAfROHvUM47sYA9aYGILvYkImKKYMIDUPwv3OAttFZunaeWeV3Xo5TNPbpAFir0OCnj7TexHDjPtoMweE4HNZ0wCERaUCE0NNmAC+YfsAtWoekHwrsWhavABBl+rsGOFb51CrrPCAightL1qs2kvGNevlGHqf+6QXeQe1/WxXBXrNHHvrbBe/1+x/zxapkPH6gnO5hY1OuXDjBhjR2OBwVdwy164hYnpjuIk/EWTHNj+nwqW5PaiODCxuj8HwsTrujxz8f3hKK+9qFJC0IFEIq2oBEAxejrUYVAsH+dXutndm/Xy4Dzoiekvys2JPUxndng7UfNrh7Xwln9Mw0Kf8cGH/8kAibDQBNkgMkH1M8c5e+A1d+7iYiIiIhoTOLNBhERERERBYI3G0REREREFAjebBARERERUSCGViLQCXkDISDkVAp7AzFhVwcVQyBUkivpcNer7d7gY29Bh7ZmHfJB1da476Gqbdz6V1SblLyBWQPCY7BKY0SHEMPjdWi8HPWGiIqgnHd/SbcZkBC3KyyjyuyoQiaCgrx2OBKFEP2tvcKGWdXXlHR42u1o87yOgNDg1PE6ULtgop5U4MGMDmR1bvP276de10HZmeN0EE0krVoKZX1MN3R7w8UdOf0ZURh8EmhLWB0O9aE0aEPnv7ggXO5YEyBUaf+Lhx1PKDwHJmVI2icn+Cx2iFcEV7a3J9gw/Trsa/p1QNwe20Rw6FVqmzwvu8O6b/UVdRi8JqrHxcak7jfxHm+F6NCODWqZcjsI+yIhK3gPKqejkLOk9HL2ZANvrtC7flCY2NMHK9IffVyDVX9DYyK4rpmEPvZ29euuvL5G5sG1e0K9nrileeEk1XbwpDrP68On6LGzccNfVVt563rVZlc7F9F9HlZJrwNVvxN1qq1sTVphEhm1TE7AhCHguh8F57oNTtJSYfb4h8YsmwO+lzglcF0AVe2d/qzntdunA9Zul75ulju26eV6QXVwiz1hhohIdP6Rqi0FrnWz6vWYEu71blto60q9XWisBpN52OFvtF8NGNdMVJ+LJqbPdfs74GhPWFB9vZmIiIiIiPYKvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKxJAC4ka8ASsHBEhCPko814CqwmmQxhtvBfs2Z3VQcU2nrjrZAqopz9jnRNU2LuENySX7d6hlQv26OqpTyKo2O0gnIpKLecNu27I6zNcPwnVhsBPtpjIInXXn9frrEvoQo3BT2aoyGTYgeFhhqv/5eRPooyhE5YDQv2tVFS9vXaeWSYDw5dHTDlBtHUdOV233/N0bnt0OqjLf/cJm1fZIjd7WdEK3Tbaq806qBQH3Ot3WktJ9pi7uPVciIPjt5AcPC4qICKxo6t0O10c4rRIBXbsP2sH5Nxeywu4gxOuUdShw2NAkFiEwNmfqVZtdeb63qPszCoOnavT660Uf/3CPd5KF4raNehtAwBOdj07SG5pEAXEYfEZVxVGb1S99jS97mJ9rsA0GdEE1djeiJ7Fwk1ZQGuR65zTpSSymgYB4PKL36NQa77ZFt76klsmveFa1wclcanRgO2Qt56R18LtUqyuUo+u53T/QuYLC8mjiDAdsP5y0QL1x+BWcR4Pd/yQCJp2wgt7wPALjE/x5Be+Y4oLgdwlcl7Mb9XL5Tv1dLpLw9vn6JAhmA+GsXj/8rpj1jm3lnk69DDgXUT+1j7UB+9CA42Fi+vws2+e1iNhznURkeJPt7A7/skFERERERIHgzQYREREREQWCNxtERERERBSIIWU2HBn8OVb1fLLPZwqbQUGoRNj73jrwXPpr7bqg1bouneNABe8m1Xqf12sBz2mm0bOb4DN15fXzbW3d3ueyt/fqB17R89BxkGnps54FLbr650XBM3zomfIYyITYRQOTPp+p3JP89D87DwCfzQZFbsIpXUzKtZ6vRMXTyptWq7aanM70nLXvIaptXrO3sM5f3tDPrr/RpteF8hnzW/X22+tvqtHnWENC979oXhc/cnqtPAksFIayGKCgUHjwYQc8Dq2KrFXimXpffdAGzlXjgLWg/Wc9x+0kdTEmJwKe9QbPr5sa/Zyu/Wx6LK/HySYwNtfFQSaka6tef846Z0ABKwg8x6zyGCDXIahwIepvYHyzY3AojlNpvvqf/Ww36GtwXIzrvlV2vPsuFtZ9uSmi96/fArNlu1Bj40y1TPTIZv1GVPQzCp5Ntz6nCz7jjn6dnyjD+JmV/0BxLfCx46gjoWJ49vP2VVjUb7T6H8qnOCGdWfSV7QDjQCQBittldB7DXi4ECjyGCvq6j9pK23W+Uo13KIuGCvihrKnVP0wMXFvBOeCi4pOgmLTqp+gaP4I+WX29mYiIiIiI9gq82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgrEkALivtgFrUDIpOTo8F+irEPd41xvuKa+QQcc6+N6Xdv6/IUQy9amted0UGwbCHwZ0UW6sgX93mze2+Y3DB4FgbKIFY6MgDBpDLwvAQopoVIteSswZMC6QL64+oFwKApCuqDIUrh5kneZbh3gdnt1mLq07jXV5mxdr9oOtMJoC2ZM1eufW6/aEMfVgUOn4C1K6HSDQo2dug0VPbSD3gaFklF4DJz/Thn8TPXevej3ICjkGAL7GJzTqrghCPuhooFov7tx/V77mNXqum6wuJOT1xMXIHZxPlR4DRU4g0X9rPB3CAQrYVErsF8RPwVpxyJU1AsVxUN9Jmwd+yjYSQ4IADvgurk5q6/Lz2/2FkJbtqFTLZPt1/17AigaOK1BL9eQ9G5bIqLH66JdzUxEWtL6/KyL22F5/bkzoDixg8LgsNgsmPBgLLI/G7pWgCS9AQU5w3HvcXbB+BEBY0UKBL1NUfe/kFXEz77mi4BJLgRf992ONtUWamjxbmsTKCAZA+MY6B920FsV3BSREriuoOKTBux/1Z9HeYKCveiKTkRERERE1YQ3G0REREREFAjebBARERERUSB4s0FERERERIEYUuTX/N9/A0CAxLErOINlyqB6oQtCbOFcl+d1qFcHdBtrdBAoDkLRuZIOyRSsYBiq/olCg6i6cTqmw031Ce++SEf1vkAhM5BXU9uaBJ8xDDa2ZJfFFZEyaCtabWi77Hf5qxE7elT/A+ytRmFnAU0qiCsiTtSblnUSOlge6etSbWUQFCt3bFNtpa3rvNvw0l/1NoTRdoHqqwl9/tjht1C6Xi0TtgJsIiKS1utSQXsUuh1JoMx6bwyEko31u5E93f92/swh90Gwr3JGjxcG7T4rIImK8SbBuQqGWDWGoAXheY+OKxiv0cQTTsYbyrQDnyIioVSt/pmlgl6XXR0crAtBk5SgYwiD9jafYfOg+LkG2+eqfU3e3ftCOT2WOWXvcUjG9BjYH9dBVXSNsSchERF5fbs3fPvws5vUMm1r9diZatBB4XHj9bZNmeBd7oDJeltnNeqAbhpc5DPWsIvOFXSNNwKq2kfRxBzWcSv0qWXchPdcqcprsHW+wQkaHH1+C5q4xfpSFpqkjzsK1jtFPeEQZJ0HcAzI9agmJ6zHb1R9PDxhuud1KTNeLePWNKg2P+N3Nq/7aH/Z3+RIaKIHNLmBbSTfAfmXDSIiIiIiCgRvNoiIiIiIKBC82SAiIiIiokDwZoOIiIiIiAIReNotD0KJNWEYz1MtxgroiquDQGFXB43Cjv5YSVCp24405op6u1DAGgXQ60Al8zo7cAOCigisHmltBgqu51CqyKfmmFVRFgSgKhHIHSp7Gx1UqRUF1kDxVruiqQPC5k4UBF5rdAgx0jJZtdmVSVEoVlx/fQZVqQ7VWMFbECyHFZft80500B5WuwX71Q49iggOtY5ytdJKyloZw2RE74P2Xh3k2wEqJUet45oHE12gAuVIH5rZwpIBE11EwSQFaDiNhHS/aaid6nmNArThrA4Aw7HSDp7qJXyPsU4xrxutLo3OjTHBPg/RPgFB2FBBV0o2VkBc+nVYNl4PKj+DIPnEtB4zzpo/wfP6wIl6soC2Xn2s+sF5gELpzSnvuDWvWW9XCMy6EAEX2Lq4t++OS+jPjaqF+61gb4+BBgSmxwT789p9SERMVJ9b+RiYKCIxzvO6uwCOO5jZJ1Gjx5n+sn6vHZRG3xNrHFB5HJwHqKL3toJ3/Vuyel1dO7KqrS6h+8yO3ODh73FJfY7FwPhdl9J9N1IGle4tIxkT954rPBERERERVRXebBARERERUSB4s0FERERERIHgzQYREREREQVi9APiVsipu1BWizSAYFU0r0MydrDNQSW+QdAtEwFBWJCirIt5l0MFdlEQOwS21Snrzyl9diVN/bkRB2x/yAoK5yO6gm8WhKfsyuAiOPSergVltccgcLjAQj5D4/Zy6H2oeimqhApCcmLl4VClY7+BV7RtZR/b7/cz2aE/GPz2uV17UxgcyVjVcYuogjDQXKP3e2e/d1z5+9ZutczGTh3s29at21C4vKvPGzqMgYBkPdiuMBgY0wm93H5WVefpDXrcGp+qV23oGlEDwuU2WAQXnHsFR493McfnuTbGoOAqhMZAa985/fraF1r5pGqLguthU1JX6m62qsLvl9DHxa3X248+k4mBivL2BBugfxgQEIf7wv4+gr6zoLEN/Uxwja90dfrR0mV9D6kDX3vsCUdERNZ26PM0V/L2o809erKA9j4dnC6CiVU6wHJJa0IMNMlAIoImzdB9xjXtqu31Nm8fscdbEZE0CIPvO0FXSrcnMphUq68raKKiejCWZlxdnb7P8Z6fNQZMojECe/dVn4iIiIiIKoY3G0REREREFAjebBARERERUSBG/SFB+4m35iR4YA/kLHw9x+3q590cUAjNAcWJELt4HtoC+Bx9WW8/fAbTXpcDCqih50WBUtT7rLMBzxbaRYdEROIOyJKEwDP59nb52qrqo4r6oYX8ZgbsIks+nuMVETGgzTHgeWL7ff62CvNT/A/kllAByVHNVPjNiYzeT6w41yp8FAGZgck6uiAFBxVk8vbgQ1r1s+qNNXrsaQPP83b06e3YsMP77C7KbHSC54xRZgMtZ0OFBbtqdTHAiRndNjnj/ZwFMAaGwXhaF/eXz4Dnwhhkfw5U2NRN6GfCYf7Mta4fNQ16mUyzaguV0bUaFA3zUajRKehsRLionzn3k8UzEVCwFBSYwwVKrf0D824+v1KBzIJal781VZ1aKziVK4MsWq/+DpUExZIdx/v9cb8mnfvJglwwKoDn1us9aud0+4rg+xKQzevtn1qnB/Xp9d7rfgMoulcTBZkQFBa2NILv1qhAJcrt9ob1fqyxrjWuz6yhX3vH6EpERERERFWHNxtERERERBQI3mwQEREREVEgeLNBRERERESBCLyKDAx8ofBVbPDwLQrejoiPQKBTBKE2FHBH7M8JAroICiKHxfvZw6gAFQzjgn3tayv2Dr4/q5++MMz3DWk7fK0MHHtUL3IUA6++gvfDXNfeDhWwAjlpKYOipXYhu2l1el0T0zp0iPZxGQ0Z1oFE27ClR4ct+8HK8iUdrmxJeQO5cRACjYAwZG0MFKKygqd9oEghyEJKHmxr1EeBwL2F72swKEaq3uv3GgwnyQDv9TOxxUiKnfq4BvueJGO44+lePiHGYBKgAN7EmuFN0gLqFksYTKwRFj1Olof5u/UImIygP1yv2uIhfVQde3IQNLlQDExaAPqMfc0AuxWqCelxGX3HDLpPvn1GXCIiIiIi2qN4s0FERERERIHgzQYREREREQXCV2bD/N9zvD09PaPyQ30/720/l1mRzEZeNw47s+Hvk8Picfa2on2xhwuo7ewPBjznPZpGu/8N10hyCqPK73mwl2c29lT/2/VnjFYf9JvZsLkglFCwK1PJbjIboNE+jmXwzl6Y2dDL5cv62eCs6x0/iz4zGw4odGXsYmE+Mxto2HVHIbMxlvsfzE/42pDRzmz42HejmtnQnWEsZzbG6jXYd//zldkAbaIXHH5mI6fa+sN6TPSX2QD5iRjIcYxiZiNswPrRd8xhGEr/8/UTd65w9uzZI9gs2lv19PRIXZ2ubDya6xdh/yMs6P6382eIsA+Sxv5HlcZrMFWSn/7nGB+3JK7ryqZNmySTyYgDyqHT25MxRnp6eqS1tVVCPmfaGg72P0L2VP8TYR8kjf2PKo3XYKqkofQ/XzcbREREREREQ8WAOBERERERBYI3G0REREREFAjebBARERERUSB4s0FERERERIF4W99sOI4jv//97yu9GfQ2xf5HlcY+SJXE/keVxP635+y1NxtbtmyRT37ykzJz5kyJx+MyZcoUOe200+TBBx+s9KaJiMiFF14ojuN4/jvllFMqvVk0Sqq9/4mIvPLKK3L66adLXV2dpFIpecc73iHr1q2r9GbRKKn2PmiPfzv/W7p0aaU3jUZBtfe/bDYrS5YskcmTJ0symZR58+bJD3/4w0pvFo2Sau9/W7dulQsvvFBaW1ulpqZGTjnlFFm5cmWlNyswo1NGsMqsWbNGjjjiCKmvr5elS5fK/PnzpVgsyn333SdXXHGFrFixotKbKCIip5xyitx6660Dr+PxeAW3hkbLWOh/q1atkiOPPFIuueQS+dd//Vepra2Vl19+WRKJRKU3jUbBWOiDmzdv9rz+n//5H7nkkkvkgx/8YIW2iEbLWOh///zP/yx/+ctf5L/+679k+vTpcv/998vll18ura2tcvrpp1d682gEqr3/GWPkjDPOkGg0KnfddZfU1tbKt7/9bTnxxBNl+fLlkkqlKrp9gTB7oVNPPdVMmjTJZLNZ9W8dHR0D/y8i5s477xx4fc0115jZs2ebZDJpZsyYYb7whS+YQqEw8O/Lli0zxx57rEmn0yaTyZiFCxeaZ555xhhjzJo1a8x73/teU19fb2pqasy8efPMvffeu9ttvOCCC8z73ve+EX9Wqj5jof+dc8455qMf/ejIPyxVpbHQB23ve9/7zPHHHz/0D0tVZyz0v/333998+ctf9rQtXLjQ/Mu//MswPzVVi2rvf6+++qoREfPSSy8NtJXLZdPc3Gx+/OMfj/DTV6e97i8bO3bskD/96U9y/fXXw7vD+vr63b43k8nIbbfdJq2trfLiiy/Kxz72MclkMnLNNdeIiMhHPvIROfjgg+UHP/iBhMNhWbZsmUSjURERueKKK6RQKMgjjzwiqVRKli9fLul0+i239eGHH5aWlhZpaGiQ448/Xr761a9KY2Pj8D88VdxY6H+u68q9994r11xzjSxevFief/55mTFjhnzuc5+TM844Y8T7gCprLPRB29atW+Xee++Vn/3sZ0P/wFRVxkr/O/zww+Xuu++Wiy++WFpbW+Xhhx+W1157TW688caR7QCqqLHQ//L5vIiI50mCUCgk8XhcHnvsMbn00kuH+/GrV6Xvdkbb008/bUTE/O53vxt0WbHuam1Lly41ixYtGnidyWTMbbfdBpedP3++ue6663xv53//93+bu+66y7zwwgvmzjvvNHPnzjXveMc7TKlU8r0Oqj5jof9t3rzZiIipqakx3/72t83zzz9vvva1rxnHcczDDz/sax1UvcZCH7R94xvfMA0NDSaXyw3r/VQ9xkr/6+/vN+eff74REROJREwsFjM/+9nPfL+fqtNY6H+FQsFMnTrVnHXWWWbHjh0mn8+br3/960ZEzMknn+xrHWPNXnez8dRTTw27o/3qV78yhx9+uBk/frxJpVImHo+b5ubmgX+/9tprTSQSMSeccIL52te+Zl5//fWBf/vxj39sIpGIOfzww82XvvQl8/e//31I271q1SojIuaBBx4Y0vuouoyF/rdx40YjIuZDH/qQp/20004z55577hA+LVWjsdAHbfvuu69ZsmSJ7+Wpeo2V/rd06VIzZ84cc/fdd5u///3v5uabbzbpdNr8+c9/HvqHpqoxVvrf3/72N7NgwQIjIiYcDpvFixebU0891ZxyyilD/9BjwF53s9He3m4cxzE33HDDoMvu2tGeeOIJEw6HzVe/+lXzzDPPmNdee818+ctfNnV1dZ73vPrqq+bb3/62Oemkk0wsFvN06HXr1pkf/OAH5v3vf7+JRqPmpptuGtK2NzU1mR/+8IdDeg9Vl7HQ//L5vIlEIuYrX/mKp/2aa64xhx9++NA+MFWdsdAHd/XII48YETHLli0b0uek6jQW+l9fX5+JRqPmD3/4g6f9kksuMYsXLx7aB6aqMhb63646OztNW1ubMcaYQw891Fx++eX+P+wYstfdbBhjzCmnnDLkcNC3vvUtM3PmTM+yl1xyiepouzr33HPNaaedBv/ts5/9rJk/f77vbV6/fr1xHMfcddddvt9D1Wks9L/DDjtMBcTPOOMM9dcOGpvGQh/c6YILLvA8qkBjX7X3v66uLiMi5o9//KOn/bLLLjMnnXTSbn8ejQ3V3v+Q1157zYRCIXPffff5fs9YslfW2fj3f/93KZfLcuihh8pvf/tbWblypbzyyity0003yWGHHQbfM3v2bFm3bp386le/klWrVslNN90kd95558C/53I5WbJkiTz88MOydu1aefzxx+WZZ56RuXPniojIVVddJffdd5+sXr1annvuOXnooYcG/s2WzWbl6quvlqeeekrWrFkjDz74oLzvfe+TffbZRxYvXjz6O4T2qGrvfyIiV199tfz617+WH//4x/L666/L9773Pbnnnnvk8ssvH92dQRUxFvqgiEh3d7fcfvvte2cg8m2s2vtfbW2tHHPMMXL11VfLww8/LKtXr5bbbrtN/vM//1Pe//73j/4OoT2q2vufiMjtt98uDz/8sLzxxhty1113yUknnSRnnHGGnHzyyaO7M6pFpe92grJp0yZzxRVXmGnTpplYLGYmTZpkTj/9dPPQQw8NLCPW83pXX321aWxsNOl02pxzzjnmxhtvHLirzefz5txzzzVTpkwxsVjMtLa2miVLlgwEGpcsWWJmzZo18IzfeeedZ7Zv3w63ra+vz5x88smmubnZRKNRM23aNPOxj33MbNmyJajdQXtYNfe/nX7605+affbZxyQSCbNgwQLz+9//frR3A1XQWOiDt9xyi0kmk6azs3O0Pz5VWLX3v82bN5sLL7zQtLa2mkQiYfbdd1/zb//2b8Z13SB2B+1h1d7/vvvd75rJkyebaDRqpk6dar7whS+YfD4fxK6oCo4xxlT0boeIiIiIiPZKe+VjVEREREREVHm82SAiIiIiokDwZoOIiIiIiALBmw0iIiIiIgoEbzaIiIiIiCgQvNkgIiIiIqJA8GaDiIiIiIgCwZsNIiIiIiIKBG82iIiIiIgoELzZICIiIiKiQPBmg4iIiIiIAvH/AbVUIUR4FYlDAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x500 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "coef = clf.coef_.copy()\n",
    "plt.figure(figsize=(10, 5))\n",
    "scale = np.abs(coef).max()\n",
    "for i in range(10):\n",
    "    l1_plot = plt.subplot(2, 5, i + 1)\n",
    "    # iterate over the classes and reshape coefficients to ensure proper visualization\n",
    "    l1_plot.imshow(coef[i].reshape(28, 28), interpolation='nearest',\n",
    "                   cmap=plt.cm.RdBu, vmin=-scale, vmax=scale)\n",
    "    l1_plot.set_xticks(())\n",
    "    l1_plot.set_yticks(())\n",
    "    l1_plot.set_xlabel('Class %i' % i)\n",
    "plt.suptitle('Classification coefficient vectors for...')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HXhHtX1TkUba"
   },
   "source": [
    "## Question 3.2\n",
    "### Convolutional Neural Network\n",
    "\n",
    "Convolutional layers capture patterns corresponding to relevant features independently of where they occur in the input. To do so, they slide a window over the input and apply the convolution operation with a set of kernels or filters that represent the features. Although it is not their only field of application, convolutional neural networks are mainly praised for their performance on image processing tasks.\n",
    "\n",
    "The training and validation management for the CNN implementation will be performed as the feed-forward network, however we will have to define the network's architecture.\n",
    "\n",
    "For that we will implement a CNN class to define how many layers it comprises and how the layers will be connected.\n",
    "\n",
    "The initialization (`__init__`) function will define the architecture and the `forward` function will implement how the different layers are connected. This architecture will be a sequece of 2 convolutional layers ([nn.Conv2d](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html)) (1st: output channels 10, kernel size 5; 2nd: output channels 20, kernel size 5), then 2 fully connected layers ([nn.Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)) (1st: output features 50; 2nd: output features 10 (the number of classes)). Once again, the final layer will be a [softmax](https://pytorch.org/docs/stable/generated/torch.nn.functional.log_softmax.html#torch.nn.functional.log_softmax) function  that will choose the most probable class of the 10 in the input.\n",
    "\n",
    "Between the second convolution layer and the first fully connected, we will set a dropout layer ([nn.Dropout2d](https://pytorch.org/docs/stable/generated/torch.nn.Dropout2d.html)). The idea behind dropout is to disable a percentage of randomly selected neurons during each step of the training phase, in order to avoid overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "PZ0mCl24EoaM"
   },
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    \"\"\"Basic Pytorch CNN for MNIST-like data.\"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        # define convolution and linear layers\n",
    "        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n",
    "        self.conv2_drop = nn.Dropout()\n",
    "        self.fc1 = nn.Linear(320, 50)\n",
    "        self.fc2 = nn.Linear(50, 10)\n",
    "\n",
    "    def forward(self, x, T=1.0):\n",
    "        # Batch size = 64, images 28x28 =>\n",
    "        #     x.shape = [64, 1, 28, 28]\n",
    "        x = F.relu(F.max_pool2d(self.conv1(x), 2))\n",
    "        # Convolution with 5x5 filter without padding and 10 channels =>\n",
    "        #     x.shape = [64, 10, 24, 24] since 24 = 28 - 5 + 1\n",
    "        # Max pooling with stride of 2 =>\n",
    "        #     x.shape = [64, 10, 12, 12]\n",
    "        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n",
    "        # Convolution with 5x5 filter without padding and 20 channels =>\n",
    "        #     x.shape = [64, 20, 8, 8] since 8 = 12 - 5 + 1\n",
    "        # Max pooling with stride of 2 =>\n",
    "        #     x.shape = [64, 20, 4, 4]\n",
    "        x = x.view(-1, 320)\n",
    "        # Reshape =>\n",
    "        #     x.shape = [64, 320]\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.dropout(x, training=self.training)\n",
    "        x = self.fc2(x)\n",
    "        x = F.log_softmax(x, dim=1)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mv9vdZZ7OlSh"
   },
   "source": [
    "As previously, lets describe the model to be trained. We will use the ADAM optimizes ([optim.Adam](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam)), with learning rate 0.001, and the same negative log likelihood ([nn.NLLLoss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "ImTlr5JeEsb6",
    "outputId": "1cc8475e-39a4-4910-cd3b-878a668047e5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/24\n",
      "----------\n",
      "train Loss: 0.4578 Acc: 85.30%\n",
      "val Loss: 0.0919 Acc: 97.52%\n",
      "\n",
      "Epoch 2/24\n",
      "----------\n",
      "train Loss: 0.1463 Acc: 95.67%\n",
      "val Loss: 0.0479 Acc: 98.60%\n",
      "\n",
      "Epoch 4/24\n",
      "----------\n",
      "train Loss: 0.1136 Acc: 96.66%\n",
      "val Loss: 0.0402 Acc: 98.80%\n",
      "\n",
      "Epoch 6/24\n",
      "----------\n",
      "train Loss: 0.1018 Acc: 97.12%\n",
      "val Loss: 0.0320 Acc: 99.18%\n",
      "\n",
      "Epoch 8/24\n",
      "----------\n",
      "train Loss: 0.0868 Acc: 97.53%\n",
      "val Loss: 0.0324 Acc: 99.01%\n",
      "\n",
      "Epoch 10/24\n",
      "----------\n",
      "train Loss: 0.0808 Acc: 97.62%\n",
      "val Loss: 0.0305 Acc: 99.09%\n",
      "\n",
      "Epoch 12/24\n",
      "----------\n",
      "train Loss: 0.0737 Acc: 97.90%\n",
      "val Loss: 0.0279 Acc: 99.11%\n",
      "\n",
      "Epoch 14/24\n",
      "----------\n",
      "train Loss: 0.0718 Acc: 97.87%\n",
      "val Loss: 0.0320 Acc: 98.99%\n",
      "\n",
      "Epoch 16/24\n",
      "----------\n",
      "train Loss: 0.0686 Acc: 98.00%\n",
      "val Loss: 0.0262 Acc: 99.15%\n",
      "\n",
      "Epoch 18/24\n",
      "----------\n",
      "train Loss: 0.0689 Acc: 98.03%\n",
      "val Loss: 0.0281 Acc: 99.11%\n",
      "\n",
      "Epoch 20/24\n",
      "----------\n",
      "train Loss: 0.0626 Acc: 98.12%\n",
      "val Loss: 0.0265 Acc: 99.17%\n",
      "\n",
      "Epoch 22/24\n",
      "----------\n",
      "train Loss: 0.0630 Acc: 98.09%\n",
      "val Loss: 0.0265 Acc: 99.11%\n",
      "\n",
      "Epoch 24/24\n",
      "----------\n",
      "train Loss: 0.0628 Acc: 98.14%\n",
      "val Loss: 0.0254 Acc: 99.24%\n",
      "\n",
      "Training complete in 7m 13s\n",
      "Best val Acc: 99.24%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGdCAYAAAA44ojeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0VklEQVR4nO3deXxU9b3/8Xf2sCTBEAhEA2FRcAN6QSJtVXqNIlp3W1yuYq4Xf1VcMNdqsRW0reJ2latSuRe3Xq1KqaLVKlZRbK1RKjaiFlAQZU1CUJIQlsTk/P74eDIzIYFMMjPnJPN6Ph7nMSczZ2Y+mYyeN9/v93y/CY7jOAIAAPCRRK8LAAAAaImAAgAAfIeAAgAAfIeAAgAAfIeAAgAAfIeAAgAAfIeAAgAAfIeAAgAAfCfZ6wLao6mpSVu2bFFGRoYSEhK8LgcAALSD4ziqra1VXl6eEhPDaxPpEgFly5Ytys/P97oMAADQARs3btQhhxwS1nO6REDJyMiQZL9gZmamx9UAAID2qKmpUX5+fvN5PBxdIqC43TqZmZkEFAAAupiODM9gkCwAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPCduA4o998v/eQn0urVXlcCAACCxXVAeeop6X/+R1q1yutKAABAsLgOKAMG2G15ubd1AACAUHEdUHJz7baiwts6AABAqLgOKLSgAADgTwQUEVAAAPCbuA4odPEAAOBPcR1QaEEBAMCfCCiyFhTH8bYWAAAQENcBxe3i2b1bqq31thYAABAQ1wGlVy+pd2/bp5sHAAD/iOuAIoV28wAAAH+I+4DidvPQggIAgH/EfUDhSh4AAPwn7gMKc6EAAOA/cR9QaEEBAMB/CCgEFAAAfCfuAwpdPAAA+E/cBxRaUAAA8B8CCtPdAwDgO3EfUPr3t9uGBunrr72tBQAAmLgPKOnpUp8+tk83DwAA/hD3AUViunsAAPyGgCKmuwcAwG8IKOJKHgAA/IaAIrp4AADwGwKK6OIBAMBvCCiiiwcAAL8hoIjp7gEA8BsCimhBAQDAbwgoCgSUbdukxkZvawEAAAQUSVK/fnbb2Cht3+5tLQAAgIAiSUpJkXJybJ9uHgAAvEdA+RZzoQAA4B8ElG8xFwoAAP5BQPkWV/IAAOAfBJRv0cUDAIB/EFC+RRcPAAD+QUD5Fl08AAD4BwHlW3TxAADgHwSUb9HFAwCAfxBQvuW2oGzfLjU0eFsLAADxrkMBZd68eSooKFB6eroKCwu1fPnydj3vmWeeUUJCgs4666yOvG1U9e0rJSZKjmNr8gAAAO+EHVAWLlyokpISzZ49Wx988IFGjx6tSZMmqbKycr/P++KLL3T99dfruOOO63Cx0ZSUJPXvb/t08wAA4K2wA8q9996radOmqbi4WEcccYTmz5+vnj176tFHH23zOY2Njbrooot06623aujQoZ0qOJoYKAsAgD+EFVDq6+u1YsUKFRUVBV4gMVFFRUUqLS1t83m//OUv1b9/f1122WXtep+9e/eqpqYmZIsFBsoCAOAPYQWUqqoqNTY2Ktc9k38rNzdX5W2c1d9++2098sgjWrBgQbvfZ86cOcrKymre8vPzwymzw5gLBQAAf4jqVTy1tbW6+OKLtWDBAuXk5LT7eTNnzlR1dXXztnHjxihWGUAXDwAA/pAczsE5OTlKSkpSRYszeEVFhQa4Z/cg69at0xdffKHTTz+9+b6mpiZ74+RkrVmzRsOGDdvneWlpaUpLSwuntIigiwcAAH8IqwUlNTVVY8eO1dKlS5vva2pq0tKlSzVhwoR9jh85cqQ++ugjlZWVNW9nnHGGfvCDH6isrCxmXTftRRcPAAD+EFYLiiSVlJRo6tSpGjdunMaPH6+5c+eqrq5OxcXFkqRLLrlEBx98sObMmaP09HQdddRRIc/v06ePJO1zvx/QxQMAgD+EHVCmTJmibdu2adasWSovL9eYMWO0ZMmS5oGzGzZsUGJi15ygli4eAAD8IcFxHMfrIg6kpqZGWVlZqq6uVmZmZtTe56uvbEZZSdqzR/JgGAwAAN1GZ87fXbOpI0oOOkhKSbF9unkAAPAOASVIQgLdPAAA+AEBpQUGygIA4D0CSgtcagwAgPcIKC3QxQMAgPcIKC3QxQMAgPcIKC3QggIAgPcIKC0wBgUAAO8RUFqgiwcAAO8RUFqgiwcAAO8RUFpwW1B27pTq6rytBQCAeEVAaSEjQ+rRw/bp5gEAwBsElBaY7h4AAO8RUFrBQFkAALxFQGkFlxoDAOAtAkor6OIBAMBbBJRW0MUDAIC3CCitoIsHAABvEVBaQRcPAADeIqC0gi4eAAC8RUBpRXALiuN4WwsAAPGIgNIKN6Ds2SPV1npbCwAA8YiA0opevWzKe4lxKAAAeIGA0gYGygIA4B0CShsYKAsAgHcIKG1gLhQAALxDQGkDXTwAAHiHgNIGungAAPAOAaUNdPEAAOAdAkob6OIBAMA7BJQ20MUDAIB3CChtCA4oTHcPAEBsEVDa0L+/3TY0SF9/7W0tAADEGwJKG9LSpIMOsn3GoQAAEFsElP1goCwAAN4goOwHA2UBAPAGAWU/mAsFAABvEFD2gy4eAAC8QUDZD7p4AADwBgFlP+jiAQDAGwSU/aCLBwAAbxBQ9oMuHgAAvEFA2Q83oFRWSo2N3tYCAEA8IaDsR79+UkKC1NQkbd/udTUAAMQPAsp+JCdLOTm2zzgUAABih4ByAFzJAwBA7BFQDsC9koeBsgAAxA4B5QBoQQEAIPYIKAfAXCgAAMQeAeUAmAsFAIDYI6AcAF08AADEHgHlAOjiAQAg9ggoB0AXDwAAsUdAOQA3oFRVSQ0N3tYCAEC8IKAcQN++UlKS7W/b5m0tAADECwLKASQmSv372z7jUAAAiA0CSjtwJQ8AALFFQGkHprsHACC2CCjtQAsKAACxRUBpBwIKAACxRUBpB7p4AACILQJKO9CCAgBAbBFQ2oHp7gEAiC0CSjsw3T0AALFFQGkHN6Ds2CHt2eNpKQAAxAUCSjv06SOlptp+ZaWnpQAAEBcIKO2QkMA4FAAAYqlDAWXevHkqKChQenq6CgsLtXz58jaPfe655zRu3Dj16dNHvXr10pgxY/TEE090uGCvcCUPAACxE3ZAWbhwoUpKSjR79mx98MEHGj16tCZNmqTKNvo+srOz9fOf/1ylpaVauXKliouLVVxcrFdffbXTxccSc6EAABA7YQeUe++9V9OmTVNxcbGOOOIIzZ8/Xz179tSjjz7a6vETJ07U2WefrcMPP1zDhg3Ttddeq1GjRuntt9/udPGxRAsKAACxE1ZAqa+v14oVK1RUVBR4gcREFRUVqbS09IDPdxxHS5cu1Zo1a3T88ce3edzevXtVU1MTsnmNgAIAQOyEFVCqqqrU2NioXLe/41u5ubkq38+Zu7q6Wr1791ZqaqpOO+00PfDAAzrppJPaPH7OnDnKyspq3vLz88MpMyro4gEAIHZichVPRkaGysrK9Pe//1233XabSkpKtGzZsjaPnzlzpqqrq5u3jRs3xqLM/aIFBQCA2EkO5+CcnBwlJSWpokUzQkVFhQa4Z/BWJCYmavjw4ZKkMWPGaNWqVZozZ44mTpzY6vFpaWlKS0sLp7SoI6AAABA7YbWgpKamauzYsVq6dGnzfU1NTVq6dKkmTJjQ7tdpamrS3r17w3lrz9HFAwBA7ITVgiJJJSUlmjp1qsaNG6fx48dr7ty5qqurU3FxsSTpkksu0cEHH6w5c+ZIsvEk48aN07Bhw7R37169/PLLeuKJJ/TQQw9F9jeJMrcFZedOqa5O6tXL23oAAOjOwg4oU6ZM0bZt2zRr1iyVl5drzJgxWrJkSfPA2Q0bNigxMdAwU1dXpyuvvFKbNm1Sjx49NHLkSD355JOaMmVK5H6LGOjdW+rRQ9q921pRhg71uiIAALqvBMdxHK+LOJCamhplZWWpurpamZmZntUxdKi0fr30t79J3/2uZ2UAANAldOb8zVo8YWCgLAAAsUFACQMDZQEAiA0CShhoQQEAIDYIKGEgoAAAEBsElDDQxQMAQGwQUMJACwoAALFBQAkDAQUAgNggoIQhuIvH/7PHAADQdRFQwuAGlD17pJoab2sBAKA7I6CEoWdPyZ0Ij4GyAABEDwElTG4rCuNQAACIHgJKmBgoCwBA9BFQwsRcKAAARB8BJUy0oAAAEH0ElDARUAAAiD4CSpjo4gEAIPoIKGGiBQUAgOgjoISJgAIAQPQRUMLkdvFUVkpNTd7WAgBAd0VACVP//nbb0CB9/bW3tQAA0F0RUMKUliZlZ9s+A2UBAIgOAkoHMN09AADRRUDpAAbKAgAQXQSUDnADCl08AABEBwGlA+jiAQAguggoHUAXDwAA0UVA6QCmuwcAILoIKB1ACwoAANFFQOkAAgoAANFFQOkAt4tn2zapsdHbWgAA6I4IKB3Qr5+UkGBr8VRVeV0NAADdDwGlA5KTLaRIDJQFACAaCCgdxFwoAABEDwGlgxgoCwBA9BBQOojp7gEAiB4CSgfRxQMAQPQQUDqILh4AAKKHgNJBdPEAABA9BJQOoosHAIDoIaB0EF08AABEDwGlg9wWlO3bpYYGb2sBAKC7IaB0UN++UlKS7VdWelsLAADdDQGlgxITA60oDJQFACCyCCidwEBZAACig4DSCQyUBQAgOggoncBcKAAARAcBpRPo4gEAIDoIKJ1AFw8AANFBQOkEungAAIgOAkon0MUDAEB0EFA6gS4eAACig4DSCW5Aqa6W9uzxthYAALoTAkonZGVJqam2zzgUAAAih4DSCQkJDJQFACAaCCidxEBZAAAij4DSSQyUBQAg8ggonUQXDwAAkUdA6SS6eAAAiDwCSifRxQMAQOQRUDqJLh4AACKPgNJJdPEAABB5BJROoosHAIDII6B0khtQ6uqknTu9rQUAgO6CgNJJvXtLPXvaPuNQAACIDAJKBDBQFgCAyCKgRADjUAAAiKwOBZR58+apoKBA6enpKiws1PLly9s8dsGCBTruuON00EEH6aCDDlJRUdF+j++KuJIHAIDICjugLFy4UCUlJZo9e7Y++OADjR49WpMmTVJlZWWrxy9btkwXXHCB3nzzTZWWlio/P18nn3yyNm/e3Oni/YIuHgAAIivsgHLvvfdq2rRpKi4u1hFHHKH58+erZ8+eevTRR1s9/ne/+52uvPJKjRkzRiNHjtTDDz+spqYmLV26tNPF+wUtKAAARFZYAaW+vl4rVqxQUVFR4AUSE1VUVKTS0tJ2vcauXbvU0NCg7OzsNo/Zu3evampqQjY/YwwKAACRFVZAqaqqUmNjo3LdJoNv5ebmqrydZ+cbb7xReXl5ISGnpTlz5igrK6t5y8/PD6fMmKOLBwCAyIrpVTx33HGHnnnmGS1evFjp6eltHjdz5kxVV1c3bxs3boxhleGjiwcAgMhKDufgnJwcJSUlqaJFU0FFRYUGuM0Ibbjnnnt0xx136PXXX9eoUaP2e2xaWprS0tLCKc1TwV08jiMlJHhbDwAAXV1YLSipqakaO3ZsyABXd8DrhAkT2nzeXXfdpV/96ldasmSJxo0b1/FqfcptQdm7V9q2zdtaAADoDsLu4ikpKdGCBQv029/+VqtWrdIVV1yhuro6FRcXS5IuueQSzZw5s/n4O++8UzfffLMeffRRFRQUqLy8XOXl5drZjRau6dFDGj3a9h96yNtaAADoDsIOKFOmTNE999yjWbNmacyYMSorK9OSJUuaB85u2LBBW7dubT7+oYceUn19vc477zwNHDiwebvnnnsi91v4wE032e1990k7dnhaCgAAXV6C4ziO10UcSE1NjbKyslRdXa3MzEyvy2lVU5N09NHSP/8p3XqrNGuW1xUBAOCtzpy/WYsnQhITA6Hkvvuk6mpv6wEAoCsjoETQeedJhx9uXTz33+91NQAAdF0ElAhKSpJuvtn277tP8vkEuAAA+BYBJcJ+/GNp5Ejp66+lBx7wuhoAALomAkqEBbei3HsvrSgAAHQEASUKpkyRRoyQvvpKevBBr6sBAKDrIaBEQXAryn/9l1Rb6209AAB0NQSUKDn/fOmww6wVZd48r6sBAKBrIaBESVKS9Itf2P4990jdaGZ/AACijoASRRdcIB16qLR9O60oAACEg4ASRcnJtKIAANARBJQou/BCafhwqapK+s1vvK4GAICugYASZS1bUerqvK0HAICugIASAxddJA0bJm3bJj30kNfVAADgfwSUGEhOln7+c9u/6y5aUQAAOBACSoz8279JQ4daK8r8+V5XAwCAvxFQYiQlJbQVZdcub+sBAMDPCCgxdPHF0pAhUmUlrSgAAOwPASWGaEUBAKB9CCgxdsklUkGBVFEh/e//el0NAAD+RECJsZQU6aabbP/OO6Xdu72tBwAAPyKgeGDqVGnwYKm8nFYUAABaQ0DxQGoqrSgAAOwPAcUjl14qDRokbd0qLVjgdTUAAPgLAcUjLVtR9uzxth4AAPyEgOKh4mIpP1/askV6+GGvqwEAwD8IKB5KTZVmzrT9O+6gFQUAABcBxWP//u/SIYdImzdLjzzidTUAAPgDAcVjaWmhrSh793pbDwAAfkBA8YHLLpMOPljatEl69FGvqwEAwHsEFB8IbkW5/XZaUQAAIKD4xGWXSXl51ory2GNeVwMAgLcIKD6Rni797Ge2f/vtUn29t/UAAOAlAoqPTJsmDRwobdxIKwoAIL4RUHwkuBXlpz+V/vhHb+sBAMArBBSfufxyaeJEqbZWOvNM6ZZbpKYmr6sCACC2CCg+k54u/fnP0tVX28+33mpBpbra27oAAIglAooPpaRI998vPf64XYL80kvS+PHSP//pdWUAAMQGAcXHpk6V/vY3W1Dw00+lwkJp8WKvqwIAIPoIKD43dqy0YoWNS9m5UzrnHOnmm6XGRq8rAwAgeggoXUC/ftJrr0nXXWc///rX0hlnSDt2eFoWAABRQ0DpIpKTpXvvlZ580gbSvvyydMwx0scfe10ZAACRR0DpYi66SHrnHWnwYGntWunYY6U//MHrqgAAiCwCShf0ne9I778vnXiiVFcn/ehHttgg41IAAN0FAaWLysmRliyRrr/efr7jDum006SvvvK2LgAAIoGA0oUlJ0t33y09/bTUo4f06qs2LmXlSq8rAwCgcwgo3cD550ulpVJBgfT559KECdLChV5XBQBAxxFQuonRo21cykknSbt2WWi54Qbpm2+8rgwAgPARULqRvn3t8uMbbrCf775bmjxZ2rbN27oAAAgXAaWbSU6W7rzTunh69pRef10aPtzu273b6+oAAGgfAko39eMfS+++K40ZI9XUSD/7mTRihPR//yc1NXldHQAA+0dA6caOPtrW8fm//7MFBzdutAUI/+VfbOp8AAD8ioDSzSUmShdfbKsh33mnlJUlffihdPLJ0imncEkyAMCfCChxIj3dBs+uWyfNmCGlpNi8KWPGSMXF0qZNXlcIAEAAASXO9O0r3XeftGqVNGWK5DjS449Lhx4q3XSTVF3tdYUAABBQ4tawYdIzz9hA2uOOk/bskebMsSt+HnxQqq/3ukIAQDwjoMS5wkLprbekF16QRo6Uqqqkq6+WjjxSevZZa2EBACDWCChQQoJ0xhnSRx9JDz0k9e8vrV0rnXee9L3vSe+843WFAIB4Q0BBs+Rk6Sc/sXAya5ZN9FZaaiHl3HPtSiAAAGKBgIJ9ZGRIt94qffaZNG2aXar83HPSEUdI//qv0ty5tighAADRkuA4/h9lUFNTo6ysLFVXVyszM9PrcuLOJ59IN94o/elPofcfeaR1DZ1xhjR+vAUZAABcnTl/E1DQbp9/Lv3xj7b95S9SY2Pgsdxc6Yc/lM48UzrxROseAgDENwIKYu7rr6VXXrGw8sortt6Pq0cP6aSTrGXlhz+08AIAiD8EFHiqvt5aVNzWlS+/DDyWkGCXMrtdQUccYfcBALo/Agp8w3HscmU3rPz976GPDx1qQWXyZFu0MCfHmzoBANFHQIFvbdkivfSShZXXX5f27g19/OCDpdGjQ7dDD5WSkrypFwAQOZ05f3fouot58+apoKBA6enpKiws1PLly9s89pNPPtG5556rgoICJSQkaO7cuR15S3RReXnS5ZdbSNm+XVq8WLr0UmtJkaTNm6WXX7Zp9s8/Xzr8cLvMefx4u8T5wQelv/6VNYIAIN4kh/uEhQsXqqSkRPPnz1dhYaHmzp2rSZMmac2aNerfv/8+x+/atUtDhw7Vj370I1133XURKRpdU69e0lln2SbZwNqPPpI+/DCwffSRtGuXdQ217B4qKLDVl4NbWwoKuLwZALqjsLt4CgsLdcwxx+jBBx+UJDU1NSk/P19XX321fvazn+33uQUFBZoxY4ZmzJgRVpF08cSPxkZp3ToLK2VlgeCyaVPrx2dkSN/9rq3MfM45UlZWTMsFAOxHZ87fYbWg1NfXa8WKFZo5c2bzfYmJiSoqKlJpaWlYbwy0JilJOuww2370o8D927dLK1eGtrZ88olUWyu9+qptV1whnXqqdMEFdnlzjx7e/R4AgM4JK6BUVVWpsbFRuS0mtsjNzdXq1asjVtTevXu1N2g0ZU3wJBuIS337Sj/4gW2uhgZp1SobgPvUU7a/eLFtGRnWlXThhTZxXEqKZ6UDADrAl733c+bMUVZWVvOWn5/vdUnwoZQUadQo6Re/sNaUsjKbkn/QIGtZeeIJu5w5L0+aPl16+22pqcnrqgEA7RFWQMnJyVFSUpIqKipC7q+oqNCAAQMiVtTMmTNVXV3dvG3cuDFir43uKSHBBs3ecYe0fr2FkenTpX79pKoq6Te/kY47ThoyxEJMWZnN2QIA8KewAkpqaqrGjh2rpUuXNt/X1NSkpUuXasKECRErKi0tTZmZmSEb0F6JidL3vmeXKG/ZIi1ZIk2dat0+GzZId90lfec7ttjhr34lrV3rdcUAgJbC7uIpKSnRggUL9Nvf/larVq3SFVdcobq6OhUXF0uSLrnkkpBBtPX19SorK1NZWZnq6+u1efNmlZWVaS1nBcRAcrI0aZL0+ONSRYX0hz/Y1T5paTZmZdYsmxhu/HjpvvtCp+kHAHinQzPJPvjgg7r77rtVXl6uMWPG6P7771dhYaEkaeLEiSooKNDjjz8uSfriiy80ZMiQfV7jhBNO0LJly9r1flxmjEirrpaef94G177+eujYlEMPtcUOTzrJBuVy6TIAdAxT3XfGK6/YaMuiosi+LrqMigpp0SJp4UKptNTmYnElJVnrihtYCgu5IggA2ouA0lGLF1t7f//+NrFGBAf6omuqrpaWLZNee822Tz8NfTwjQ5o4MRBYRoxgdWYAaAsBpaN277Z/En/0kZ1tlixh3nSE2LAhEFZef90mjAuWn2+NbyedZLf9+nlTJwD4EQGlM/75T2ncOAsrd94p3XBDZF8f3UZTk12e7AaWt9/ed3XmMWMCYWX4cCk319YgAoB4REDprAULbMnd5GQ763w74BfYn1277OviBpYPP2z9uN69LagEbwMG7Htfbq4dCwDdBQGlsxzHVptbtMhm8vrHP7h0A2GrqLBuILd1ZfNmac+e8F6jV699Q0tenq1NNHKk3fbsGZ36ASDSCCiRsGOHtc9/+aX0v/8rTZsWnfdB3HAcm3K/oiJ0Ky/f976KCmuRaY/Bgy2sjBhht+42YAADdgH4CwElUt591+ZJv+CC6L0H0IadO1sPL5s2SWvWSKtX27T9bcnMDA0s7jZsmJSaGrvfAwBcBBQgTlRVBcJK8Pb5520vhJiUJA0damFl9Gjp5JOlCRNsyBUARBMBJRoqK6XbbrMre9LTY/OeQAft3WtrCrUMLqtXW8tMS3362NVGkydLp5wiDRwY85IBxIHOnL/5N1Rrmprs/94rV9pAgvvv97oiYL/S0mzxwyOPDL3fcWzBxNWrbe2hd96RXn1V+uorGxO+aJEdN2aMhZVTT5WOPZbWFQDeowWlLa+8Yv+3lqQXXpDOOCM27wtEWWOjtHy5fcVfeUV6//3Qx2ldARApdPFEy3/+p3TvvVJ2tk1yccghsXtvIEYqK61V5eWXpT//2VpXgo0ZY1l98mRaVwCEh4ASLfX10ne/K61YIR1/vPTGGzbiEOimwmldOeooWxmivVtCwv4fa+/W1vEA/IeAEk1r10rf+Y6NNLzlFmn27Ni+P+ChA7Wu+E1SkoWovn2t4TP4tq397GybwZeQA0QeASXannxSuvhimxmrrIyrehCXgltXXn3V5mxparKBuE1N7d+Cj/eLlJR9A0xWlgUet5UneD94a+3+lvdlZEgFBTbJXkEBSxogfhBQYmHBApsOn3lYgIhxHAs+jtP65gaa/W3Bx3zzjU0KvX27tfa0dttyv74+9r93374WVFrbBg+2QAN0BwQUAOgAx7ElBloLLzU1Fn4aG1tvCWrt/raO/eorW0Xjiy+kr78+cF0tA4zb8lJQYMuF+aEFprFR2rBB+vRTW3fq4INtQsDBg5m5GAHMgxJLjiM98IB181x+udfVAOiEhARboLFXLyk/PzbvWV0dCCstty+/DA1LK1a0/hr9+1sYGDrUljIIvh040LqVIsFxbPbiTz+1bc2awP7atTZBYEuJiXbBo1tP8DZsmHWjMd4H7UELSrgWL5bOOcdmxlq+XBo1ytt6AHQrNTVtB5j16w/cApOebq0swaHF3R8yROrRY9/n7NolffbZviFkzRrrMmtLaqp06KHWerJliy25cKBFLzMz9w1X7jZoEK0v3Q1dPLHkONIPf2iXNRx+uPT3v9s/vwAgBnbssKCybp0FAvf2888t2DQ27v/5AwdaMBg0yBaj/PRTaePGto9PSLBjDzvMthEjAvuDBoXOvOA49ppuPcHbunUWYvYnMdFasoYOtTDlbu7Pubm0vnQ1BJRY27bNVl3bulX6j/+wAbQA4LGGBgsbwcEl+Lampu3nZmeHhg93f/jw1ltdOmL3bmsJahlc3P3du/f//B49AuNwWoaYIUPsyiv4CwHFC2+8IRUV2T8ZnnnGrvABAJ9yHBvf4oaCDRusRcINJH37el+f2/qyfn3g1t3ftOnAl6ZnZ4cGlpyc8CYB3N/2zTd2xVdDg922trXnsW++sc998GDbBg0K7OflRX+m5tpaC7GtbY89FvkJ0wkoXvn5z6Xbb7dO1bIy+y8CABBx9fV2Em0ZXtyfq6q8rrDzkpIsIASHluAgM2iQ1LNn28/fvduCXFsBZONGG6Tdlr/8RTruuMj+TlzF45VbbpGWLZPee0/6298IKAAQJampNnZm2LDWH6+tDQwkdgPMjh0HnkenvVtystXQ1paS0r7HExNtdMCXX4ZuGzda64r781//2vrv2a9fILT07WsTJrrho70hLSvLxvq03A49tCN/meihBaWzvvzSvm3HHut1JQCALqqx0cJGy+CyYUNgf+fOA79Oz56th4/gLZYTAdLF4yfz5llb5GWXMessACAiHMcuMQ8OLVVVdlVWcPg46CB/XelEQPGL3butk7CqyiLqZZdJ11xD1w8AIC515vwdofkGIck6F2+7zeZHqa2V5s61a/TOO8/GqPg/CwIA4AsElEhKS7Pp7z/+2JZ8Pflkuy7u2Wel73/frvgBAAAHRECJhsRE6ZRTbE36jz6yrp70dJsi37VpU/tWDQMAIA4RUKLtqKOkhx+24dmHHx64/8Yb7YL3q66yRTAiqaHBrrd76y1p0SJbahQAgC6EgBIrwXMwf/ONrcK1a5dd9TNihHTGGdKbb7ZvnMrOndInn1g3UnArzCOP2DBud7WwiROlH//Y5oa+8MLIByEAAKKEgOKF5GRbZPD116XTTrNQ8uKL0r/+q/Qv/2IrJrtKS6Vrr5XOOkv6zndsZp6MDGuZOfVU6f33A8cmJgbmg05NtQG6o0dbIHr66QPPEw0AgE8wk6xXEhKkE0+0bc0a6b//W3r8cZsyP7ilY/Vq6f77931+VpZNJRjc4nLqqdK779r9/ftbYJGkf/zD1g4aMSJwbEmJLQgxbZotYAEAgI8wD4qfbN9uKyP/v/9ns+1INsj2ySdDV5UaNKhzy3Zu2mTdPo2Ntjzo1Kk2X0vwGBkAADqJidoQnr17bQXmuXOtxcY1aZI0Y4ZdHp1I7x8AoHOYqA3hSUuzVpMPPrArfc4+27qcXn1VmjzZWnEAAPAQASWeJSRIxx8vPfectHatdN11Ul6eNGVK4Jh//MMWfgAAIIbo4kGob76xq4wkG4BbWGgtLeeeK11xhY2BSU+3sSs9e9rVQgAAtKIz52+u4kGo5KCvRG2tXdLc2Cj9/ve2BTv/fLt8WbJjBg0KhJeWt9/7nk1O57r7bumkk+wyaD8tvQkA8AUCCtqWmSktXSqtXGmXQb/4olRXZ6s2O46FD9fu3dKWLW2/VlpaYN9xpBtusP38fJuk7owzpBNOCD0OABC36OJB+BzHptNvagqElMZGCzJ79lhYaXk7eLC1mEjWjXTeedJrr9lsuq6MDFvD6PLLpaKi2P9eAICIoosHsZWQsO/Yk6Qkm+m2PZKTpeeft+DyxhvSCy9Y60x5ua0dNGFCIKDU1EiVlTYrbrR88420caOtX9Srl3VVDRgQvfcDABwQAQXe6dHDpvo/7TRrjXn/femPf7Rp/V3PPScVF9skcmeeaV1BhYXhz9Oye7e9n+vaa6V//lP6/HPpyy+tBcg1bJhd1eS65RYLMYMGWZeUe0trHgBEDQEF/pCYKI0fb1uw9eutdWbVKtvuuMOm8T/9dAsrJ58c6GZau1Zat85Cx/r1dutuw4eHrlv06qu2xIArLc26oXbtsoUWg/3P/1jrTktZWdL3vy+99FLgvldekXr3tgCTmxsaigAA7cYYFPjf119LS5ZY68rLL1u3j2RdTeXlFlgk6bDD2l6x+aCDpK++Cvz8xBPWajN0qG0DBwZaZRwncGWR49gVR19+afPBbNhg3UHuKtI/+IF1U7kGDgwNM717W339+0vHHBO6rtKf/mTBqF8/ezwnR0pJ6fjnhI5btszC7ebNthTEpk1Sfb2NiTr33H1DK4B2Yap7xI/6eumvf7WwUlkZuMxZshPJmjWB0BG8FRTYvC2RUltrQaWxUTr6aLuvqcladL74wh6rrw99Tsswk5trv0Ow7GwLK4WFtnika84caedOC0yOY+/l7ufn21pKrtmzLYy5jwcf36+f9OtfB4798EMbE9Svn62UnZQUiU/HH6qq7O/gBo5NmwIBJCXFWrtcY8fafD+tycqStm0jPAIdQEAB/MZxAgN8KyvtBNe7d2Dwr+NYmNm61R6rqrIQ4WoZZnJybDHJ1hxzjLR8eeDnwYPbnv33yCOljz8O/HzUUdInn9h+QoIFpH79bDvsMOnhhwPHLl0aCDn9+tlVV6mpduKOxtpNjmNXgQV3k/3tbxYwqqtt27EjsJ+cLD32WODYceOkFStaf+2ePS3wuS1lM2ZIn34qHXKIbQcfbOOWFi+2AOiGRcexMVJjx1ogPuII5vE5EMex9b/27rW/Z8vbo48OdNN+9JF11ebmWmvkgAF0k3ZxBBSgq2tstFaPbdss0KSl2dVMrptusjloEhIsDCQkBLaWLSj33GMnbPfx4OMHDpT+4z8Cx37/+za2J7j7y9UyzBx5pA0sbs1hh4WO6Zk82X5OSbEQ424pKXbyWbQocOwvf2mtFy0DR3W11btxY+DY735XKi1tvYbeva1ly3X22XasGzjc8OFuEye2L1w0NQUC2MqVNrmga+RICyrnndf1Jh10/9fv1rx5s3VP1tW1vl1zTWCeoocftm5X97FduyxsuMFj+XL720kW/v77v9uuY/VqacQI2//FL6Tbbgt9PCsrEFbmzbNQKFl37oYNdv/AgdaN25U+/zjBZcZAV5eUFGiZcP8HHOz229v/Wtdf3/5j337bbr/5xlpo3IC0bdu+l5KPHGkngNZafFq2oGzcaAOVW3PIIaE/v/yy9N57rR+7Y0foz2PHWl1ZWftuffqEjh969tnItOwEv0ZBgbXSPPus9Oc/28n1tttsGzrUxiudc07n3zNYfX3o3+Ktt2zg9+7drW+/+U3gM7j1VhsQ3taxNTWBrs+ZM21sVlsuucTCpWRrdD37bNvH7t4d2A+e0FGykJOebltaWiAoSdKhh1owLy+31sU9ewJhdfXq0L/F735nv58rNTUQVgYOlO66y15Pspa3994LBOW0tND973/fvj+SVFER+EdCcLhuarLw1b9/IKht3mxjl/butb+T21Lk7v/wh7a+mWTv//zz9t9aTo6F5uCtV6+2P884RUABYN0jubmBE1Brgk9I7v+s6+tta9kQ+/vfW2uG+3hDQ2C/ZfC55ho71g0ZLYNHsAceaP/vFI1up8xM6dJLbaupsSu4/vAHG8/y+eehJ5nPP7eT7IQJobU4jrU4BB+7eLG1zlRW2gnSPUlWVNgJbefOwLH33WdzB7Vl7tzACfSzz9pucZKs9cMNKLm5Fh579bL7evUK3YLHJ/34xxak3cd69gwsbZGWFjgpS9LNN1sLoHvC318rx9SptrmfU3V1IKxs3WqX+LuysqyGrVtt0Hp9fWAgu2RX/LmWLAkde9XS++9b+JUsgM6c2fax77wTaN1cuFD6z/9s+9jXXgt8FmVloTW1tGiRtcRJFgCfe27fENOvX/u/124XaUpKYAmT9esDrZXV1fYddvdvv33///17gIACIHyJiXZCamt8QGutQG258MLI1BRrmZlW+4UXWoBYssTGDrkeesi62/LypDFjrNXJDR8JCYEuO8lOii++2PZ71dUFAs24cXYydhfsdP8Orf09rrnGuqDaOjY7O3Ds3Xfb1h4nnGBbe3S0ZSAhwQJrnz7WetfSddfZJtmJuKLCwoobaPLzA8eOGiX927+Fhurg/eCuh7Q0ayVxH6uvt5AoWcBy9yVrCRkxIhC+0tJC94M/39GjrbsrKclaId1B25s32/cneHLId95pPVClpFjr0COPBMazvfii/RzcNeqGj4YGu0LN/Vu9/LJ01VWtf97XXOO7gMIYFACIhhtvlObPD1wW31JNjQ00lqxb5sMP7cTotmQF7/fpw/gKLzU1BcZxRUNNjbU+ua2Ly5ZZK6QbYDZvtgDmnq7fftsWYJVsXE5boUOy1rYzzrD9P/3JWnEyMwMtlO7+pZeGtnxF7FdjkCwA+M/evXb10+bN+4YPxhwgHA0N1jq0ebNdfde7t93/ySc2xiY4bATv9+4dne7OdiKgAAAA3+nM+du7WAUAANAGAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPAdAgoAAPCdZK8LaA93weWamhqPKwEAAO3lnrfd83g4ukRAqa2tlSTl5+d7XAkAAAhXbW2tsrKywnpOgtORWBNjTU1N2rJlizIyMpSQkBCx162pqVF+fr42btyozMzMiL0u9o/P3Rt87t7gc/cGn7s3Wn7ujuOotrZWeXl5SkwMb1RJl2hBSUxM1CGHHBK118/MzOQL7AE+d2/wuXuDz90bfO7eCP7cw205cTFIFgAA+A4BBQAA+E5cB5S0tDTNnj1baWlpXpcSV/jcvcHn7g0+d2/wuXsjkp97lxgkCwAA4ktct6AAAAB/IqAAAADfIaAAAADfIaAAAADfieuAMm/ePBUUFCg9PV2FhYVavny51yV1a7fccosSEhJCtpEjR3pdVrfzl7/8Raeffrry8vKUkJCg559/PuRxx3E0a9YsDRw4UD169FBRUZE+++wzb4rtRg70uV966aX7fP9POeUUb4rtJubMmaNjjjlGGRkZ6t+/v8466yytWbMm5Jg9e/Zo+vTp6tu3r3r37q1zzz1XFRUVHlXcPbTnc584ceI+3/ef/OQnYb1P3AaUhQsXqqSkRLNnz9YHH3yg0aNHa9KkSaqsrPS6tG7tyCOP1NatW5u3t99+2+uSup26ujqNHj1a8+bNa/Xxu+66S/fff7/mz5+v9957T7169dKkSZO0Z8+eGFfavRzoc5ekU045JeT7//TTT8ewwu7nrbfe0vTp0/Xuu+/qtddeU0NDg04++WTV1dU1H3PdddfpxRdf1KJFi/TWW29py5YtOuecczysuutrz+cuSdOmTQv5vt91113hvZETp8aPH+9Mnz69+efGxkYnLy/PmTNnjodVdW+zZ892Ro8e7XUZcUWSs3jx4uafm5qanAEDBjh333138307duxw0tLSnKefftqDCrunlp+74zjO1KlTnTPPPNOTeuJFZWWlI8l56623HMex73ZKSoqzaNGi5mNWrVrlSHJKS0u9KrPbafm5O47jnHDCCc61117bqdeNyxaU+vp6rVixQkVFRc33JSYmqqioSKWlpR5W1v199tlnysvL09ChQ3XRRRdpw4YNXpcUV9avX6/y8vKQ735WVpYKCwv57sfAsmXL1L9/f40YMUJXXHGFtm/f7nVJ3Up1dbUkKTs7W5K0YsUKNTQ0hHzfR44cqUGDBvF9j6CWn7vrd7/7nXJycnTUUUdp5syZ2rVrV1iv2yUWC4y0qqoqNTY2Kjc3N+T+3NxcrV692qOqur/CwkI9/vjjGjFihLZu3apbb71Vxx13nD7++GNlZGR4XV5cKC8vl6RWv/vuY4iOU045Reecc46GDBmidevW6aabbtLkyZNVWlqqpKQkr8vr8pqamjRjxgx973vf01FHHSXJvu+pqanq06dPyLF83yOntc9dki688EINHjxYeXl5WrlypW688UatWbNGzz33XLtfOy4DCrwxefLk5v1Ro0apsLBQgwcP1u9//3tddtllHlYGRN/555/fvH/00Udr1KhRGjZsmJYtW6YTTzzRw8q6h+nTp+vjjz9mXFuMtfW5X3755c37Rx99tAYOHKgTTzxR69at07Bhw9r12nHZxZOTk6OkpKR9RnJXVFRowIABHlUVf/r06aPDDjtMa9eu9bqUuOF+v/nue2/o0KHKycnh+x8BV111lV566SW9+eabOuSQQ5rvHzBggOrr67Vjx46Q4/m+R0Zbn3trCgsLJSms73tcBpTU1FSNHTtWS5cubb6vqalJS5cu1YQJEzysLL7s3LlT69at08CBA70uJW4MGTJEAwYMCPnu19TU6L333uO7H2ObNm3S9u3b+f53guM4uuqqq7R48WK98cYbGjJkSMjjY8eOVUpKSsj3fc2aNdqwYQPf90440OfemrKyMkkK6/set108JSUlmjp1qsaNG6fx48dr7ty5qqurU3FxsdeldVvXX3+9Tj/9dA0ePFhbtmzR7NmzlZSUpAsuuMDr0rqVnTt3hvwrZf369SorK1N2drYGDRqkGTNm6Ne//rUOPfRQDRkyRDfffLPy8vJ01llneVd0N7C/zz07O1u33nqrzj33XA0YMEDr1q3TDTfcoOHDh2vSpEkeVt21TZ8+XU899ZReeOEFZWRkNI8rycrKUo8ePZSVlaXLLrtMJSUlys7OVmZmpq6++mpNmDBBxx57rMfVd10H+tzXrVunp556Sqeeeqr69u2rlStX6rrrrtPxxx+vUaNGtf+NOnUNUBf3wAMPOIMGDXJSU1Od8ePHO++++67XJXVrU6ZMcQYOHOikpqY6Bx98sDNlyhRn7dq1XpfV7bz55puOpH22qVOnOo5jlxrffPPNTm5urpOWluaceOKJzpo1a7wtuhvY3+e+a9cu5+STT3b69evnpKSkOIMHD3amTZvmlJeXe112l9ba5y3Jeeyxx5qP2b17t3PllVc6Bx10kNOzZ0/n7LPPdrZu3epd0d3AgT73DRs2OMcff7yTnZ3tpKWlOcOHD3d++tOfOtXV1WG9T8K3bwYAAOAbcTkGBQAA+BsBBQAA+A4BBQAA+A4BBQAA+A4BBQAA+A4BBQAA+A4BBQAA+A4BBQAA+A4BBQAA+A4BBQAA+A4BBQAA+A4BBQAA+M7/B3vPIN8auunQAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# instatiate the model, optimizer and criterion\n",
    "model = CNN().to(device)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "# train and validate model\n",
    "model, losses, accuracies = train_val_model(model, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=25, log_interval=2)\n",
    "\n",
    "# plot the losses\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ULZ91b0cPhy5"
   },
   "source": [
    "We have now completed training and validation with 3 different models: Logistic Regression, Feed-Forward Network, and Convolutional Neural Network.\n",
    "\n",
    "We have seen that with the CNN, the performance of the model in the validation set, outperforms the other models (~99% accuracy against ~90% and ~98%)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "al8Axr0sdd-n"
   },
   "source": [
    "## Question 3.3\n",
    "### Number of learnable parameters of the MLP and the CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PHyGUuZbTvhr"
   },
   "source": [
    "The difference in performance between CNNs and MLP is small but how many learnable parameters are we using in the MLP and in CNN models?\n",
    "\n",
    "We can find it out using the following lines of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "acy0l3-YQjT2",
    "outputId": "3ba26bee-5fa5-42ff-ea44-91e242d99f1a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of parameters in the MLP model: 242762\n",
      "Number of parameters in the CNN model: 21840\n"
     ]
    }
   ],
   "source": [
    "# get MLP trainable model parameters\n",
    "model_parameters_mlp = filter(lambda p: p.requires_grad, model_mlp.parameters())\n",
    "params_mlp = sum([np.prod(p.size()) for p in model_parameters_mlp])\n",
    "print('Number of parameters in the MLP model: {}'.format(params_mlp))\n",
    "\n",
    "# get CNN trainable model parameters\n",
    "model_parameters_cnn = filter(lambda p: p.requires_grad, model.parameters())\n",
    "params_cnn = sum([np.prod(p.size()) for p in model_parameters_cnn])\n",
    "print('Number of parameters in the CNN model: {}'.format(params_cnn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sj28CWvrMbOw"
   },
   "source": [
    "You can see that we have ~11x more learnable parameters to achieve almost the same performance.\n",
    "\n",
    "We can experiment and try to find out the number of layers and corresponding sizes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QKB4FCyodd-p"
   },
   "source": [
    "## Question 3.4\n",
    "### Change number of MLP hidden layers to approximate the number of learnable parameters of the CNN. What is the impact on the performance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2RmgJhIPMECw",
    "outputId": "cbc4b2c9-7132-46f5-9b06-8bc62af38db2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of parameters in the MLP model: 25450\n"
     ]
    }
   ],
   "source": [
    "# instantiate smaller MLP\n",
    "model_mlp_test = MLP([D_in, 32, D_out]).to(device)\n",
    "model_parameters_mlp_test = filter(lambda p: p.requires_grad, model_mlp_test.parameters())\n",
    "params_mlp_test = sum([np.prod(p.size()) for p in model_parameters_mlp_test])\n",
    "print('Number of parameters in the MLP model: {}'.format(params_mlp_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B_oq9682QWCF"
   },
   "source": [
    "And how does that model perform? We are about to find out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 746
    },
    "id": "w6oa0TeBQU9E",
    "outputId": "4e0844b7-5fe0-4b96-daa4-3762fdfc1f35"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/14\n",
      "----------\n",
      "train Loss: 0.4714 Acc: 86.95%\n",
      "val Loss: 0.2781 Acc: 91.79%\n",
      "\n",
      "Epoch 5/14\n",
      "----------\n",
      "train Loss: 0.1459 Acc: 95.84%\n",
      "val Loss: 0.1422 Acc: 95.90%\n",
      "\n",
      "Epoch 10/14\n",
      "----------\n",
      "train Loss: 0.1048 Acc: 96.97%\n",
      "val Loss: 0.1180 Acc: 96.39%\n",
      "\n",
      "Training complete in 4m 22s\n",
      "Best val Acc: 96.89%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABI50lEQVR4nO3dfVzV9f3/8ecBBBQFNZQLRQE1ry9KkllZbVLYWqXrwpqpUd9arlrGVulKnVlDzTXLnC43W9dZm7lqzWUsrX55lUR5SWpeK3hRcBQTkPP5/fHucCGgHDiHzwEe99vtc+PD4XPe5/0xJ8+9P6/3++2wLMsSAACAHwuwuwMAAADnQmABAAB+j8ACAAD8HoEFAAD4PQILAADwewQWAADg9wgsAADA7xFYAACA3wuyuwPe4HK5dPDgQbVp00YOh8Pu7gAAgFqwLEvHjx9XbGysAgLOPobSJALLwYMHFRcXZ3c3AABAHezbt0+dO3c+6zVNIrC0adNGkrnh8PBwm3sDAABqw+l0Ki4uruz3+Nk0icDifgwUHh5OYAEAoJGpTTkHRbcAAMDvEVgAAIDfI7AAAAC/R2ABAAB+j8ACAAD8HoEFAAD4PQILAADwewQWAADg9wgsAADA7xFYAACA3yOwAAAAv0dgAQAAfo/Achb5+dLjj0t33ml3TwAAaN4clmVZdneivpxOpyIiIlRQUODV3ZoLC6XWrc35kSNSZKTXmgYAoNnz5Pc3IyxnERYmJSaa802b7O0LAADNGYHlHPr1M18JLAAA2IfAcg4EFgAA7EdgOQcCCwAA9iOwnEPFwNL4y5MBAGicCCzn0LOnFBQkFRRIBw7Y3RsAAJonAss5BAdL559vznksBACAPQgstUAdCwAA9iKw1AKBBQAAexFYaoHAAgCAvQgsteAOLFu2SKWl9vYFAIDmiMBSC4mJUmio9P330q5ddvcGAIDmh8BSC4GBUp8+5pzHQgAANDwCSy1RxwIAgH3qFFjmz5+v+Ph4hYaGKjk5WevWravV+9544w05HA6NHDmy0uu33367HA5HpWPEiBF16ZrPEFgAALCPx4FlyZIlSk9P17Rp05SVlaWBAwcqNTVVhw8fPuv7du/erd/+9rcaNmxYtT8fMWKEDh06VHa8/vrrnnbNpwgsAADYx+PA8vTTT+uuu+5SWlqa+vTpo4ULF6pVq1ZavHhxje8pLS3VmDFjNH36dCUmJlZ7TUhIiKKjo8uOdu3aedo1n3IHlpwcqbjY3r4AANDceBRYiouLtWHDBqWkpJQ3EBCglJQUrV69usb3Pf744+rYsaPuvPPOGq9ZuXKlOnbsqJ49e2rChAk6duxYjdcWFRXJ6XRWOnytc2cpPFw6fVr6+muffxwAAKjAo8By9OhRlZaWKioqqtLrUVFRys3NrfY9n376qf72t79p0aJFNbY7YsQIvfTSS8rMzNSsWbO0atUqXX311SqtYdGTjIwMRURElB1xcXGe3EadOBw8FgIAwC5Bvmz8+PHjGjt2rBYtWqTIyMgar7vlllvKzvv3768BAwaoW7duWrlypYYPH17l+smTJys9Pb3se6fT2SChpV8/6bPPCCwAADQ0jwJLZGSkAgMDlZeXV+n1vLw8RUdHV7l+586d2r17t6699tqy11wul/ngoCDl5OSoW7duVd6XmJioyMhI7dixo9rAEhISopCQEE+67hWMsAAAYA+PHgkFBwdr8ODByszMLHvN5XIpMzNTQ4cOrXJ9r169tHHjRmVnZ5cd1113nX784x8rOzu7xlGR/fv369ixY4qJifHwdnyLwAIAgD08fiSUnp6u8ePHKykpSUOGDNHcuXNVWFiotLQ0SdK4cePUqVMnZWRkKDQ0VP3cv+V/0LZtW0kqe/3EiROaPn26brjhBkVHR2vnzp16+OGH1b17d6Wmptbz9rzLfSvffCMVFkphYfb2BwCA5sLjwDJ69GgdOXJEU6dOVW5urgYNGqTly5eXFeLu3btXAQG1H7gJDAzUV199pRdffFH5+fmKjY3VVVddpRkzZtjy2OdsOnSQoqKkvDxp61YpKcnuHgEA0Dw4LMuy7O5EfTmdTkVERKigoEDh4eE+/ayUFCkzU3rhBen22336UQAANGme/P5mLyEPUccCAEDDI7B4iMACAEDDI7B4iMACAEDDI7B4qE8f8/XAAem77+ztCwAAzQWBxUPh4VLXruZ882Z7+wIAQHNBYKkD92OhjRvt7QcAAM0FgaUOqGMBAKBhEVjqgMACAEDDIrDUQcXA0viX3QMAwP8RWOqgVy8pIED69lspN9fu3gAA0PQRWOogNFTq0cOc81gIAADfI7DUEXUsAAA0HAJLHRFYAABoOASWOiKwAADQcAgsdeQOLJs3Sy6XvX0BAKCpI7DUUffuUnCwVFgo7dljd28AAGjaCCx1FBQk9e5tznksBACAbxFY6oE6FgAAGgaBpR4ILAAANAwCSz0QWAAAaBgElnpwB5Zt26SSEnv7AgBAU0ZgqYcuXaTWraXiYmnHDrt7AwBA00VgqYeAAKlvX3POYyEAAHyHwFJP1LEAAOB7BJZ6IrAAAOB7BJZ6IrAAAOB7BJZ6cgeWHTuk77+3ty8AADRVBJZ6ioqSzjvPbIC4bZvdvQEAoGkisNSTw8FjIQAAfI3A4gUEFgAAfIvA4gUEFgAAfIvA4gUEFgAAfIvA4gXuwLJ3r+R02tsXAACaojoFlvnz5ys+Pl6hoaFKTk7WunXravW+N954Qw6HQyNHjqz0umVZmjp1qmJiYtSyZUulpKRo+/btdemaLdq2lTp3NuebN9vaFQAAmiSPA8uSJUuUnp6uadOmKSsrSwMHDlRqaqoOHz581vft3r1bv/3tbzVs2LAqP5s9e7aeffZZLVy4UGvXrlVYWJhSU1N16tQpT7tnGx4LAQDgOx4Hlqefflp33XWX0tLS1KdPHy1cuFCtWrXS4sWLa3xPaWmpxowZo+nTpysxMbHSzyzL0ty5c/XYY4/p+uuv14ABA/TSSy/p4MGDWrZsmcc3ZBcCCwAAvuNRYCkuLtaGDRuUkpJS3kBAgFJSUrR69eoa3/f444+rY8eOuvPOO6v8bNeuXcrNza3UZkREhJKTk2tss6ioSE6ns9JhNwILAAC+41FgOXr0qEpLSxUVFVXp9aioKOXm5lb7nk8//VR/+9vftGjRomp/7n6fJ21mZGQoIiKi7IiLi/PkNnzCHVg2brS3HwAANEU+nSV0/PhxjR07VosWLVJkZKTX2p08ebIKCgrKjn379nmt7brq3dusenvkiHSOch4AAOChIE8ujoyMVGBgoPLy8iq9npeXp+jo6CrX79y5U7t379a1115b9prL5TIfHBSknJycsvfl5eUpJiamUpuDBg2qth8hISEKCQnxpOs+16qV1K2b2QRx0ybpJz+xu0cAADQdHo2wBAcHa/DgwcrMzCx7zeVyKTMzU0OHDq1yfa9evbRx40ZlZ2eXHdddd51+/OMfKzs7W3FxcUpISFB0dHSlNp1Op9auXVttm/6MOhYAAHzDoxEWSUpPT9f48eOVlJSkIUOGaO7cuSosLFRaWpokady4cerUqZMyMjIUGhqqfu7f4j9o27atJFV6feLEiXriiSfUo0cPJSQkaMqUKYqNja2yXou/69dPWraMwAIAgLd5HFhGjx6tI0eOaOrUqcrNzdWgQYO0fPnysqLZvXv3KiDAs9KYhx9+WIWFhbr77ruVn5+vSy+9VMuXL1doaKin3bMVIywAAPiGw7Isy+5O1JfT6VRERIQKCgoUHh5uWz82bzahpU0bqaDAFOECAIDqefL7m72EvKhHD6lFC+n4cckPJi4BANBkEFi8KDhY6tnTnPNYCAAA7yGweBl1LAAAeB+BxcsILAAAeB+BxcsILAAAeB+BxcvcgWXLFqm01N6+AADQVBBYvCwhQWrZUioqknbutLs3AAA0DQQWLwsIkPr2Nec8FgIAwDsILD5AHQsAAN5FYPEBAgsAAN5FYPEBAgsAAN5FYPEBd2D5+mtTfAsAAOqHwOIDsbFS27ZmWnNOjt29AQCg8SOw+IDDwWMhAAC8icDiIwQWAAC8h8DiIwQWAAC8h8DiIwQWAAC8h8DiI+7Vbnftkk6csLcvAAA0dgQWH4mMlKKjzfmWLfb2BQCAxo7A4kM8FgIAwDsILD5EYAEAwDsILD5EYAEAwDsILD7Uv7/5SmABAKB+CCw+1KeP+XrokHTsmL19AQCgMSOw+FDr1lJCgjlnlAUAgLojsPgYdSwAANQfgcXHCCwAANQfgcXHCCwAANQfgcXHKgYWy7K3LwAANFYEFh/r2VMKDJTy86WDB+3uDQAAjROBxcdCQqTzzzfnPBYCAKBuCCwNgDoWAADqh8DSAAgsAADUT50Cy/z58xUfH6/Q0FAlJydr3bp1NV67dOlSJSUlqW3btgoLC9OgQYP08ssvV7rm9ttvl8PhqHSMGDGiLl3zSwQWAADqJ8jTNyxZskTp6elauHChkpOTNXfuXKWmpionJ0cdO3ascn379u316KOPqlevXgoODtZ7772ntLQ0dezYUampqWXXjRgxQi+88ELZ9yEhIXW8Jf/jDiybN0sulxTAuBYAAB5xWJZnk22Tk5N10UUX6bnnnpMkuVwuxcXF6f7779ekSZNq1caFF16oa665RjNmzJBkRljy8/O1bNkyz3r/A6fTqYiICBUUFCg8PLxObfhSaakUFiYVFUk7dkjdutndIwAA7OfJ72+P/r9+cXGxNmzYoJSUlPIGAgKUkpKi1atXn/P9lmUpMzNTOTk5uuyyyyr9bOXKlerYsaN69uypCRMm6NhZdgssKiqS0+msdPizwMDyjRB5LAQAgOc8CixHjx5VaWmpoqKiKr0eFRWl3NzcGt9XUFCg1q1bKzg4WNdcc43mzZunK6+8suznI0aM0EsvvaTMzEzNmjVLq1at0tVXX63S0tJq28vIyFBERETZERcX58lt2II6FgAA6s7jGpa6aNOmjbKzs3XixAllZmYqPT1diYmJuuKKKyRJt9xyS9m1/fv314ABA9StWzetXLlSw4cPr9Le5MmTlZ6eXva90+n0+9BCYAEAoO48CiyRkZEKDAxUXl5epdfz8vIUHR1d4/sCAgLUvXt3SdKgQYO0detWZWRklAWWMyUmJioyMlI7duyoNrCEhIQ0uqJcAgsAAHXn0SOh4OBgDR48WJmZmWWvuVwuZWZmaujQobVux+VyqaioqMaf79+/X8eOHVNMTIwn3fNr7sCybZtUXGxvXwAAaGw8fiSUnp6u8ePHKykpSUOGDNHcuXNVWFiotLQ0SdK4cePUqVMnZWRkSDL1JklJSerWrZuKior0/vvv6+WXX9aCBQskSSdOnND06dN1ww03KDo6Wjt37tTDDz+s7t27V5r23NjFxUlt2kjHj0vbt0t9+9rdIwAAGg+PA8vo0aN15MgRTZ06Vbm5uRo0aJCWL19eVoi7d+9eBVRYaKSwsFC/+tWvtH//frVs2VK9evXSK6+8otGjR0uSAgMD9dVXX+nFF19Ufn6+YmNjddVVV2nGjBmN7rHP2TgcZpRl9WrzWIjAAgBA7Xm8Dos/8vd1WNzuvltatEh67DHphyVoAABotny2Dgvqh8JbAADqhsDSgAgsAADUDYGlAbkDy86d0smT9vYFAIDGhMDSgDp2lDp0kCxL2rrV7t4AANB4EFgaGI+FAADwHIGlgRFYAADwHIGlgRFYAADwHIGlgRFYAADwHIGlgblXuN2/X8rPt7UrAAA0GgSWBhYRYfYVkqTNm+3tCwAAjQWBxQY8FgIAwDMEFhsQWAAA8AyBxQbuwLJxo739AACgsSCw2KB/f/N10yaz6i0AADg7AosNevWSAgKkY8ekvDy7ewMAgP8jsNigZUupe3dzTh0LAADnRmCxCYW3AADUHoHFJgQWAABqj8BiEwILAAC1R2CxiTuwbN4suVz29gUAAH9HYLFJ9+5ScLB04oS0d6/dvQEAwL8RWGzSooWZ3izxWAgAgHMhsNiIOhYAAGqHwGIjAgsAALVDYLERgQUAgNohsNjIHVi2bpVOn7a3LwAA+DMCi426dpXCwqTiYmnHDrt7AwCA/yKw2CggQOrb15zzWAgAgJoRWGxGHQsAAOdGYLEZgQUAgHMjsNiMwAIAwLkRWGzmDizbt0unTtnbFwAA/FWdAsv8+fMVHx+v0NBQJScna926dTVeu3TpUiUlJalt27YKCwvToEGD9PLLL1e6xrIsTZ06VTExMWrZsqVSUlK0ffv2unSt0YmOltq3Nxsgbttmd28AAPBPHgeWJUuWKD09XdOmTVNWVpYGDhyo1NRUHT58uNrr27dvr0cffVSrV6/WV199pbS0NKWlpem///1v2TWzZ8/Ws88+q4ULF2rt2rUKCwtTamqqTjWDIQeHg8dCAACci8eB5emnn9Zdd92ltLQ09enTRwsXLlSrVq20ePHiaq+/4oorNGrUKPXu3VvdunXTAw88oAEDBujTTz+VZEZX5s6dq8cee0zXX3+9BgwYoJdeekkHDx7UsmXL6nVzjQWBBQCAs/MosBQXF2vDhg1KSUkpbyAgQCkpKVq9evU5329ZljIzM5WTk6PLLrtMkrRr1y7l5uZWajMiIkLJycm1arMpILAAAHB2QZ5cfPToUZWWlioqKqrS61FRUdp2lgKMgoICderUSUVFRQoMDNSf//xnXXnllZKk3NzcsjbObNP9szMVFRWpqKio7Hun0+nJbfgdAgsAAGfnUWCpqzZt2ig7O1snTpxQZmam0tPTlZiYqCuuuKJO7WVkZGj69One7aSN3Kvd7tkjOZ1SeLi9/QEAwN949EgoMjJSgYGBysvLq/R6Xl6eoqOja/6QgAB1795dgwYN0m9+8xvdeOONysjIkKSy93nS5uTJk1VQUFB27Nu3z5Pb8Dvt20uxseZ8yxZ7+wIAgD/yKLAEBwdr8ODByszMLHvN5XIpMzNTQ4cOrXU7Lper7JFOQkKCoqOjK7XpdDq1du3aGtsMCQlReHh4paOx47EQAAA18/iRUHp6usaPH6+kpCQNGTJEc+fOVWFhodLS0iRJ48aNU6dOncpGUDIyMpSUlKRu3bqpqKhI77//vl5++WUtWLBAkuRwODRx4kQ98cQT6tGjhxISEjRlyhTFxsZq5MiR3rtTP9evn/TBBwQWAACq43FgGT16tI4cOaKpU6cqNzdXgwYN0vLly8uKZvfu3auAgPKBm8LCQv3qV7/S/v371bJlS/Xq1UuvvPKKRo8eXXbNww8/rMLCQt19993Kz8/XpZdequXLlys0NNQLt9g4MMICAEDNHJZlWXZ3or6cTqciIiJUUFDQaB8PrV8vDRkiRUVJNUyOAgCgSfHk9zd7CfmJPn3M17w86cgRe/sCAIC/IbD4ibAwKTHRnPNYCACAyggsfoQ6FgAAqkdg8SMEFgAAqkdg8SP9+5uvBBYAACojsPiRiiMsjX/uFgAA3kNg8SPnny8FBZn9hPbvt7s3AAD4DwKLHwkOlnr2NOc8FgIAoByBxc9QeAsAQFUEFj9DYAEAoCoCi58hsAAAUBWBxc+4A8uWLVJpqb19AQDAXxBYziU/Xxo3TsrObpCPS0iQWraUTp2SvvmmQT4SAAC/R2A5l0mTpJdflm69VTp50ucfFxhYvhEij4UAADAILOfyxBNSbKy0bZuUnt4gH0kdCwAAlRFYziUyUnrpJcnhkP7yF+ntt33+kQQWAAAqI7DUxvDh0sMPm/P/+z+fL0NLYAEAoDICS209/riUlCR9+600dqxPp/C4A8vXX0tFRT77GAAAGg0CS20FB0uvvSaFhZkksWePzz6qUycpIkI6fdp8FAAAzR2BxRM9ekj/+pf01VdSYqLPPsbh4LEQAAAVEVg8NXy4dN55Pv8YAgsAAOUILHVlWWb20K9+5ZPmCSwAAJQLsrsDjdbXX0tpaZLLJV1yiTRmjFebJ7AAAFCOEZa66tlTmjrVnE+Y4PV19Pv2NV+/+UYqLPRq0wAANDoElvp49FEzunL8uBlhKSnxWtMdOkhRUeZ8yxavNQsAQKNEYKmPoCDp1VfNHOQ1a8xaLV7EYyEAAAwCS3117SotXGjOn3xSWrXKa00TWAAAMAgs3nDLLdLtt5vz9eu91iyBBQAAg1lC3vLss9L48dIVV3itSQILAAAGIyze0qaNV8OKJPXpY74ePGi2MAIAoLkisPjCzp1mRdycnHo1Ex5uSmQk6R//8EK/AABopAgsvvCb30j/+59066313m757rvN11//WtqwwQt9AwCgESKw+ML8+Wa/oS++kB57rF5NTZok/exnJveMGiUdOeKlPgIA0IjUKbDMnz9f8fHxCg0NVXJystatW1fjtYsWLdKwYcPUrl07tWvXTikpKVWuv/322+VwOCodI0aMqEvX/EOnTtLf/mbO58yRVqyoc1MBAdLLL5uNovftk26+WTp92kv9BACgkfA4sCxZskTp6emaNm2asrKyNHDgQKWmpurw4cPVXr9y5Urdeuut+uijj7R69WrFxcXpqquu0oEDBypdN2LECB06dKjseP311+t2R/7i+uvNkv2SNG5cvYZG2raVli2TWreWVq6UHnnEGx0EAKDxcFiWZXnyhuTkZF100UV67rnnJEkul0txcXG6//77NWnSpHO+v7S0VO3atdNzzz2ncePGSTIjLPn5+Vq2bJnndyDJ6XQqIiJCBQUFCg8Pr1MbPnHypHTRRWZt/Z/9THrnHcnhqHNzS5dKN9xgzl99VfrFL7zUTwAAbODJ72+PRliKi4u1YcMGpaSklDcQEKCUlBStXr26Vm2cPHlSJSUlat++faXXV65cqY4dO6pnz56aMGGCjh07VmMbRUVFcjqdlQ6/1KqV9PrrUkiIdOiQ9N139Wru5z+Xfvc7c/5//ydlZ9e/iwAANAYeBZajR4+qtLRUUe5d+X4QFRWl3NzcWrXxyCOPKDY2tlLoGTFihF566SVlZmZq1qxZWrVqla6++mqVlpZW20ZGRoYiIiLKjri4OE9uo2ENGCB9+KH02WfSGSGtLh5/XBoxQvr+e1OEe5ZcBwBAk9GgK93OnDlTb7zxhlauXKnQ0NCy12+55Zay8/79+2vAgAHq1q2bVq5cqeHDh1dpZ/LkyUpPTy/73ul0+ndoufTSyt9bVp0fDQUGSq+9JiUlSd98Y3YF+M9/zD6MAAA0VR6NsERGRiowMFB5eXmVXs/Ly1N0dPRZ3ztnzhzNnDlTH3zwgQYMGHDWaxMTExUZGakdO3ZU+/OQkBCFh4dXOhqFkhIzzfmBB+rVTLt2pgi3VSszePPoo97pHgAA/sqjwBIcHKzBgwcrMzOz7DWXy6XMzEwNHTq0xvfNnj1bM2bM0PLly5WUlHTOz9m/f7+OHTummJgYT7rn/9auNTs6z5snvftuvZrq31964QVzPnu29OabXugfAAB+yuNpzenp6Vq0aJFefPFFbd26VRMmTFBhYaHS0tIkSePGjdPkyZPLrp81a5amTJmixYsXKz4+Xrm5ucrNzdWJEyckSSdOnNBDDz2kNWvWaPfu3crMzNT111+v7t27KzU11Uu36ScuvdSsgitJaWlmk6B6uPlm6aGHypvbuLGe/QMAwE95HFhGjx6tOXPmaOrUqRo0aJCys7O1fPnyskLcvXv36tChQ2XXL1iwQMXFxbrxxhsVExNTdsyZM0eSFBgYqK+++krXXXedzj//fN15550aPHiwPvnkE4WEhHjpNv3Ik09KF1xgqmXHj5dcrno194c/SCkpZgb1qFH1nogEAIBf8ngdFn/kt+uw1CQnR7rwQpMyZs8uHyapo2PHTBHu7t3S1Vebp02Bgd7pKgAAvuKzdVjgJT17Ss88Y85/9zvp88/r1dx550lvvy21bGlmDE2b5oU+AgDgRwgsdrnzTrNsbXCwGRqpp0GDpL/+1Zw/+aRZFRcAgKaCwGIXh0N6/nkpK0u68UavNPmLX0gPPmjOx483OwIAANAUEFjs1L69eTzk5oVyotmzpSuukE6ckEaOlAoK6t0kAAC2I7D4i48/lgYOlPbsqVczQUFmTZa4OGn7dum22+o9EQkAANsRWPyBZUmTJ5uFVG67TaphD6Xa6tDB1LCEhEjvvSfNmOGlfgIAYBMCiz9wOKSXX5batJE+/dQsrlJPSUnSwoXm/Pe/l955p95NAgBgGwKLv0hMlP78Z3M+fbrZ3bmebr9duu8+cz52rFn+BQCAxojA4k9uu00aM8Y8EhozxisVs08/LQ0bJjmdpgjX6ax/NwEAaGgEFn8zf76UkGDWZpkwod4zh1q0kN56S+rUSdq2zSu7AQAA0OAILP4mIkJ67TWztn5JiVRcXO8mo6Kkf/7TrFG3bJmUkVH/bgIA0JAILP7oRz+SNmww85O9tAFkcnJ5icyUKWYJfwAAGgsCi78aONDMHpKk48el77+vd5N33in98pfmKdMvfiHt2FHvJgEAaBAEFn93+rR0003S8OHSkSP1bu6ZZ6ShQ6X8fGnUKLMiLgAA/o7A4u+2b5fWrpVWrzaPirZtq1dzISHSP/4hRUdLmzZJd9zhlR0BAADwKQKLv+vdW1qzxqzT8s03Znjkf/+rV5OxsSa0uGcQPfWUl/oKAICPEFgag549TWi5+GLzLCc1VXrhhXo1eckl0rPPmvPJk6UPPqh/NwEA8BUCS2PRoYOUmSndcoupa7njDmnOnHo1+ctfmkJcl8s0+803XuorAABeRmBpTEJDpVdflR57TGrdWkpJqVdzDof03HPSkCHSd99JP/+5dPKkl/oKAIAXEVgam4AAs/1yTo40aFD563Xc4Tk01Cwq17Gj9OWX0l13UYQLAPA/BJbGKja2/Pyzz6QBA6StW+vUVOfOpvg2KMgssjt3rne6CACAtxBYGjvLkn7zG2nLlnrNILrsMrNRoiQ99FC9JyIBAOBVBJbGzuGQ3nnHzCAqKKjXDKL77pPGjTNPl0aPlvbs8XJfAQCoIwJLU1DdDKLf/c7jbZkdDmnhQunCC6WjR00Rrhd2BAAAoN4ILE1FxRlEktmS+dZbpaIij5pp2VJaulSKjJSysqR77qEIFwBgPwJLU+KeQfT3v5tlbEtLzVcPde1qNooODJReeslMfQYAwE4ElqZo/Hjp449N2gio23/iH/+4fMn+Bx+Uli3zXvcAAPAUgaWp+tGPpFatzLllSb/6lalz8cDEidKYMWagZtQoM3jjYVkMAABeQWBpDv72N2nBAmnECGnx4lq/zeEwE47uu898P3WqdPPN0okTPuonAAA1ILA0B7fdZgpwT582mwd5MIOoRQtp3jzpr3815//8p5lBzb5DAICGRGBpDtwziKZMMd9nZJgp0B7MWb7zTmnVKik6Wtq4UbroIo+fMAEAUGcElubC4ZAef1x68UUzVPLWW9JPfiIdPlzrJoYOlT7/3ISVb781a9TNncu0ZwCA7xFYmptx46QVK6R27aQNG8wmih7o1MlMQBo/3hTjPvigdPvt0qlTvukuAABSHQPL/PnzFR8fr9DQUCUnJ2vdunU1Xrto0SINGzZM7dq1U7t27ZSSklLlesuyNHXqVMXExKhly5ZKSUnR9u3b69I11Mbll0urV0uvvy4NG+bx20NDTTHu3Lnla7Vcdpl04ID3uwoAgFSHwLJkyRKlp6dr2rRpysrK0sCBA5WamqrDNTxaWLlypW699VZ99NFHWr16teLi4nTVVVfpQIXfbrNnz9azzz6rhQsXau3atQoLC1NqaqpO8X/bfadnT+mGG8q/37zZLDhXSw6H9MAD0n//K7VvL61fLyUlmY2jAQDwOstDQ4YMse69996y70tLS63Y2FgrIyOjVu8/ffq01aZNG+vFF1+0LMuyXC6XFR0dbT311FNl1+Tn51shISHW66+/Xqs2CwoKLElWQUGBB3eCMseOWVbXrpYlWdakSZZVWurR23futKz+/c3bW7SwrEWLfNNNAEDT4snvb49GWIqLi7VhwwalpKSUvRYQEKCUlBStXr26Vm2cPHlSJSUlat++vSRp165dys3NrdRmRESEkpOTa2yzqKhITqez0oF6aNfOFKVI0syZHs8gSkw0Iys33CCVlEh33WXWbikp8VF/AQDNjkeB5ejRoyotLVVUVFSl16OiopSbm1urNh555BHFxsaWBRT3+zxpMyMjQxEREWVHXFycJ7eBMzkc0vTplWcQ/fjHHs0gat3avG3GDPP9/PnSlVdKR474qM8AgGalQWcJzZw5U2+88YbefvtthYaG1rmdyZMnq6CgoOzYt2+fF3vZjFWcQbR2rZScLG3ZUuu3Oxxms+h//Utq08as25KUJGVn+67LAIDmwaPAEhkZqcDAQOXl5VV6PS8vT9HR0Wd975w5czRz5kx98MEHGjBgQNnr7vd50mZISIjCw8MrHfAS9wyibt2k3bulRx/1uInrrjN5p0cPae9eszLukiXe7yoAoPnwKLAEBwdr8ODByqywxKnL5VJmZqaGDh1a4/tmz56tGTNmaPny5UpKSqr0s4SEBEVHR1dq0+l0au3atWdtEz7Us6e0Zo1ZYMWDvYcq6t1bWrfObF/0/femLGbyZLN2CwAAnvL4kVB6eroWLVqkF198UVu3btWECRNUWFiotLQ0SdK4ceM0efLksutnzZqlKVOmaPHixYqPj1dubq5yc3N14ocd9BwOhyZOnKgnnnhC77zzjjZu3Khx48YpNjZWI0eO9M5dwnORkWaxlXbtzPeWZWpcPJhq3rat9N570sMPm+9nzpSuvVbKz/d6bwEATV1dpiHNmzfP6tKlixUcHGwNGTLEWrNmTdnPLr/8cmv8+PFl33ft2tWSVOWYNm1a2TUul8uaMmWKFRUVZYWEhFjDhw+3cnJyat0fpjU3gIULzbzlpCTL2rvX47e/9pplhYaaJnr0sKwtW3zQRwBAo+LJ72+HZTX+nWCcTqciIiJUUFBAPYuvfPihNHq02USoQwfpzTelK67wqImsLGnkSGnfPlOU++qrZsQFANA8efL7m72EUDspKWbnw0GDzFzllBTpT3/yaOfDCy80TQwbJh0/Ll1/vfTkk2yeCAA4NwILai8hQfp//0+67TZTPZuebs5Pnqx1Ex07msGaCRNMUHnsMenmm6UfSpoAAKgWgQWeadXK7Hb4zDNm58PXXzfTgTwQHCz9+c/S88+bder+8Q/pkkukXbt81GcAQKNHYIHnHA7p17+W/vc/E1w8rGVxu+su6aOPpKgo6auvpIsuMk0CAHAmAgvq7rLLpPvvL/9+507pqac8Kkq55BJT15KUJB07Jl11lfTss9S1AAAqI7DAO4qKpFGjzKIrN95oqmprqXNn6eOPpbFjTWnMAw9Id9zh0ZIvAIAmjsAC7wgJMVs0t2ghLV1q9iHKyan121u2NOvSPf20FBAg/f3v5knTwYM+6zEAoBEhsMB77r7bDJXExkpbt0pDhkjvvFPrtzsc0oMPSsuXl++/mJRkdgkAADRvBBZ4149+JG3YIF16qeR0msVWpk2TXK5aN3HlldL69VLfvtKhQ2Y/xqeeMk+dAADNE4EF3hcdLWVmmkdEkvTf/0olJR410a2b2TR61CipuNiUxvTqZXZ9piAXAJofAgt8IzhYmjfPrL//j3+YGhcPtWlj3rp4sXnKtHu32fV56FCzfh0AoPkgsMC3fvELMw3IbcYM6a23av32gAApLU36+mvp8celsDBT23LppWYy0o4dPugzAMDvEFjQcD76SJo61azF/8gjZg5zLYWFSVOmmIBy990myPzzn1KfPtLEiWYNFwBA00VgQcMZNkz67W/N+ezZ0tVXe5w0oqOlv/xF+vJL8/aSErPYbvfu0h//SGEuADRVBBY0nKAgM93njTfMnkQrVph5y1984XFT/fpJ778vffCBNGCAlJ9vslDv3hTmAkBTRGBBwxs92iyukphoKmkvvthsolgHV14pZWWVF+bu2kVhLgA0RQQW2KN/f7OJ0NVXmzX46zEkEhhIYS4ANHUEFtinXTvp3XfNs51f/KL89TqGl7MV5j74IIW5ANCYEVhgr8BAM8rilptrVstdt67OTVZXmDt3LoW5ANCYEVjgXyZPNmFl2DDpr3+tV1NnK8x9800KcwGgMSGwwL8884w0cqRZj/+uu6Rf/rLeQyLVFeaOHm1qfSnMBYDGgcAC/xIebgpPnnjCbN/8/PPSFVeY3Z/robrC3DVrygtzd+70TvcBAL5BYIH/CQiQHn1U+ve/pbZtTbLo08eslFtPFQtz77qrvDC3d29TmPvtt/XvPgDA+wgs8F9XX22mPl97ramkveSS8p+99ZYpUDl9uk5NR0ebwZszC3O7dZOefprCXADwNw7Lavylh06nUxERESooKFB4eLjd3YEvfP+91LKlOXe5pIQEae9ekzxuu00aP95U2dbRihWmIPerr8z3CQnSzJnSTTeZJ1MAAO/z5Pc3IyxoHNxhRZJOnpRGjZIiI8006DlzzEJ0gwdLzz4rHT3qcfNnK8z97DMv3gcAoE4ILGh8Wrc2z28OHJCWLTPhpUULkzgeeMDUv9RBxcLc6dPLC3MvuUT62c+k//zHDO4AABoegQWNV3CwdP310tKl0sGDZnRl8GBp3Ljyaz7/3ISYL76o9cIrYWHS1KnS9u3lhbn//rf0059KPXqY/RvrMIgDAKgHaljQtN1zj1n2VjKPjW6/XRozRoqKqnUTX38tLVggvfCCVFBgXgsJMY+MfvUracgQ6lwAoC6oYQHcbrpJuvlmkzA2bpR+8xupUyfzjOett2o1y+j886U//ck8gfrrX6ULLjCziF56yewikJQk/e1vprQGAOAbjLCgefjuO2nJEunFF01hiiR17izt3m2KVzxgWWb3gAULpDfeKJ8CHRFhBnAmTJB69vRq7wGgSfLk9zeBBc1PTo4ZHomMNKvFSWYhluHDTaHKbbeZMFMLx46ZR0ULFkjffFP++vDh5nHRdddJQUE+uAcAaAIILICn3n3XpAvJFKSkpJjhkpEjpVatzvl2l8tssvjnP0vvvVde39upk3T33dL//Z+ZLg0AKOfzGpb58+crPj5eoaGhSk5O1rp162q8dvPmzbrhhhsUHx8vh8OhuXPnVrnm97//vRwOR6WjV69edekaUDeXX24KUYYNM2ljxQpTnBsdbdLGrl1nfXtAgDRihPTOO2akZfJkqUMHU/cybZrUtaspp/noI3aJBoC68DiwLFmyROnp6Zo2bZqysrI0cOBApaam6vDhw9Vef/LkSSUmJmrmzJmKjo6usd2+ffvq0KFDZcenn37qadeAugsPl+64Q/r4Y7MT4rRpUny8dPy4CTLff19+7bZtlZ//nCE+XvrDH6R9+6TXXjMbLJ4+Lf3jH9JPfiL17SvNm1c+4wgAcG4eB5ann35ad911l9LS0tSnTx8tXLhQrVq10uLFi6u9/qKLLtJTTz2lW265RSEhITW2GxQUpOjo6LIjMjLS064B3pGYKP3+9ya4rFwppaeb3RHdfv97s+nQ+edLv/61WVGumilCISHSrbdKn3xi9iy65x6zxsvWreZtsbHSL38pZWc30H0BQCPmUWApLi7Whg0blJKSUt5AQIBSUlK0evXqenVk+/btio2NVWJiosaMGaO9e/fWeG1RUZGcTmelA/C6gADzqOiPf6y80Epxsamk3b7dDJX89KdS+/ZSaqr0zDPVPvMZMMAU5h48KD33nNl8+uRJswHjBReY1XRfeUU6daoB7w8AGhGPAsvRo0dVWlqqqDMW3YqKilJubm6dO5GcnKy///3vWr58uRYsWKBdu3Zp2LBhOn78eLXXZ2RkKCIiouyIi4ur82cDHlu61EwPevttM0TSpYuZ2/zBB9Krr1YON598Yh4r/SA8XLr3XmnTJjN4c/PNJvt89pk0dqwUFydNmnTOkhkAaHb8YuG4q6++WjfddJMGDBig1NRUvf/++8rPz9ebb75Z7fWTJ09WQUFB2bFv374G7jGavfBwM4No4UKzlsuWLdLTT0v3319+jdNpilbat5d+/GNp1iyzHbRlyeEwgzdLlphNp2fMMDOpjx41l3XrZta2e/99qbTUrpsEAP/hUWCJjIxUYGCg8vLyKr2el5d31oJaT7Vt21bnn3++duzYUe3PQ0JCFB4eXukAbONwmBqXBx80wyRue/aYCtzTp81wyqRJ0sCBZq7zHXdIPxSWx8RIjz1mRlXeftvsHG1ZZv+ia64x+xc9+aRZPgYAmiuPAktwcLAGDx6szMzMstdcLpcyMzM1dOhQr3XqxIkT2rlzp2JiYrzWJtDg+vc3dS7bt5vClZ/9zKzpcuiQWW3uyy/Lrz16VEFfrNfI61z64AMTTh58UGrb1gSZxx6TevUytS+PPipt2MD0aADNi8ePhNLT07Vo0SK9+OKL2rp1qyZMmKDCwkKlpaVJksaNG6fJkyeXXV9cXKzs7GxlZ2eruLhYBw4cUHZ2dqXRk9/+9rdatWqVdu/erc8++0yjRo1SYGCgbr31Vi/cImCz7t1N4cq775ralxUrzJ5GP/1p+TX//KfZRTEqSrrtNp2/7hU9PemwDhyQFi829bwtWpgZRn/4g9m/KD7ebES9ahWPjQA0A1YdzJs3z+rSpYsVHBxsDRkyxFqzZk3Zzy6//HJr/PjxZd/v2rXLklTluPzyy8uuGT16tBUTE2MFBwdbnTp1skaPHm3t2LGj1v0pKCiwJFkFBQV1uR3AfrNmWVabNpZlBk7M4XBYVlKSZT32mGUdOWJ9951lvfKKZd1wg2W1alX50shIy7rjDst67z3L+v57u28GAGrHk9/fLM0P+IuSEmn1arOuy/Ll5Qu0BAaaaty2bc33L7ygkgOHtb70Qr229QK9viJS335b3kzr1mbwZtQo85X/SQDwV+wlBDQFhw5J//2vKWKZPr389YsvNsHmB1aXLjoSd6HWlVyoN3deqJeP/VSSmVodHGy2RRo1ymyV1LFjA98DAJwFgQVoyp55xizckpUlnTGTzoqL0/p/7NXSpWbG0dCv/y6nwpWlC7XP0VWXDnNo1CgTYLp2tan/APADAgvQXBQUmNlGWVnmaN9e+mGDUctlqbRDlIK+PSJJ+lbtlKULy47v+yTpolu76+c/N7OyK653BwANgcACwKzzf999Jshs2mRqZCr4SFfoJ/pIktkWaWrvt9T/pl7qd1NvBQQH2dFjAM0MgQVAZcXF0ubNZSMxJWuztClquB7TE/rwQ6llcb7y1U6SdEqhyu04QEFDLlT0NRcq6KILpX79zG6OAOBFBBYAteZ0Sh+/sFOJT96hzke+ULiq7uH1zYgJavfan9WuncyujW+/bVax69nTLIYHAHXgye9vxn2BZi48XPrZA92kB1bp1EmXPnplp3Jez1Lx2iz1/t5UvPxpeW/NP8/sLDCm11b99o3bzJsdDrOCXZ8+phCmTx9p2DCzWB4AeBEjLACqdfq02e7o7aWWPlx+Wlu2t5AkDdFazdbD6qvNitSxqm+cM8es5CtJO3dKzz5rgow71ERGNuBdAPBnjLAAqLegIOmKK6QrrnBIaqFDh6SPP5ZWrkzWhFWrtHWrFKkj6q2t6qMt6qMtuihsq9Z+MlixcWY36qjPPzeBpaIOHcrDy513mn0GAOAcGGEBUCd5eSbArFpljk2bql7z8/gs3RP+mvoGbFXHY1sUtG935QuWLZOuv96c/+c/0owZ5Y+W3KGmSxcpwONtzwA0AoywAPC5qCjpppvMIZndA9wBZuVK6auvpKW7L9RSXVj2ngHdCnVT/236SfQW9QvcqvDBg8sb/OILs4JvhVV8JZmi3p49pb/+Vbrwh7Z27ZJ275Y6d5Y6daLwF2gGGGEB4BPffit98okJL6tWma2RzvzXpls38+jo8sul4d33qNO+NWZL6i1bzPH11+Xrx3z5pTRggDmfOVOqsCu82rYtDy+dO0uTJpUX/ubnSy6X1K4dq+MBfoZpzQD8Tn6+CTDuR0hZWSZHVBQfb8LLFVeYr/GdSuTY9Y2Uk2M2RXKPpMybJ82fL+3fLxUWVv2wiuEmI0P63e+k0NDyUOMONp06SaNHm+EiAA2OwALA7xUUSP/v/5UHmM8/l0pLK18TF1ceXi6+2DwZqlTOYllmIZn9+6UDB8q/PvBA+TbVDz1kZi7V5KuvpP79zfns2aZIuGKg6dzZHD16mOtCQ735xwA0awQWAI3O8eNmT0d3gFm3zkytrqhtW+lHPzLH0KFScrIUEVGLxk+dkg4erBps9u83tTFt25rr7r1X+vOfa25nw4byOppPPzUjP+7iYHcbAGqNwAKg0SssNPW3q1aZYt7166Xvv698jcNh8oI7wAwdKvXqVY9JRUePmoLeM4PN3r2mnmbHDikszFx7zz3SX/5S/t7Y2PLZTX36SL/4hdSmTR07AjQPBBYATU5JiXl6s2ZN+WSib76pel1EhBl5cQeY5GQfDX4895z07rtmj6YDB6r+PD+/fPhn/nxTRFwx0HTsSBEwmj0CC4BmIS9PWru2PMCsX2+2OjrTmaMwvXt7eWmXgoLKs5sOHZJefbX851deKX34YeX3tG9fHl7mzZOCg73YIaBxILAAaJZOn646CrNzZ9XrwsOrjsK0a+fDji1daopytmwxwWbnzvI53pGR0pEj5dfeeqtZY6biaEyfPlJ0tNSiBYvooUkhsADADw4fNgHGHWLWrat+FKZXr/IAM3SoyQg+ywbff29qYrZskU6ckO66q/xnCQkmsFSnRw/zPreUFFNXExJiRmhCQsqP2FjplVfKr83IkPbtq3pdcLBJcL/8Zfm1n3xiHmlVvK51azP9+7zzCE3wGgILANTg9Glp48bKozA7dlS9LjxcGjKkPMAMHCjFxDRA2cnmzeWPltxHTo4p4unXz3TerVcv87PqxMebAmK3pCQzy6k6HTqYZOd2+eWm0rk6oaGmItodWhYskPbsMSNAUVHlX6OizGMv6nRwFizNDwA1CAqSLrjAHBMmmNeOHKlcC7NunVne5cMPK5eehIeb+pczj4QEKTDQSx3s29ccFZ0+bYaFzlyoZulSMx+8uFgqKjKH+zwkpPK199xjRljcP6/4ntatK1/bq5cZBarYZkGBWb64bdvKIyxvvFFzuGnVyvSvYrjZt69quImONgXKhBucBSMsAHCG06fNZo7uUZi1a6Xt26uuzOsWEiKdf37VIHP++U1snbniYum77yqvDLxokRkVys01VdB5eeb8u+/MkNTBg+XXXnaZedxUndatTUp0h5YFC8zsK3eg6dDBXBMWZr7GxfnuPtFgeCQEAF5WVGRCy9atlY+cHLMuXXUCAszoS69eVcNMk19nrqjI1MFUDDd/+UvlcOP+WlBgam4qTg8fNswszledsDBT++N2000mCIWFVT1atza1PO4gtGyZWVenumvDwnxcvIQzEVgAoIGUlpoSjjODzNat5vd1TaKjq3+81CB1Mv7m1KnyERm3BQtM/Y471Bw5YmpnCgtNCNmzp/zac9XcVFxx8Gc/k/7975r7UlJinhtK0pgx0r/+ZYJMmzZmVCc+3hwJCSYotWxZ17uGCCx2dwcAZFnm9+y2bVWDTHXrzLlFRFQ/IuPVOpmmZt8+U1/jDjQVj9LS8mIlSXrqKbNxVXXXlpRU/o9z3XVmccCanDxZHlgefdQ8O3SHmYSE8vOoKEZtakBgAQA/5nRWH2R27qy5TiY01ASXfv1MTW6/fubo0qUZjsg0lG+/NcNkhYXmsdXevWbm1e7d5vW33iq/dvhw6X//q76d0FAzguQuaPrPf8xfAneo6dCh2f5HJLAAQCNUlzqZ1q3LJxa5Q0zfvs300ZKd1q83/7Hcgcb9dd8+E0hyc8uvTUmRMjPLvw8Lq/yo6ZlnykdkTp0yVd1N9D8mgQUAmpDSUvP7b/NmM3tp0yZzvm2beYpRnXbtKo/GuL9GRjZs35u9khJTfxMbW/7aQw+ZKWi7dplZVBV/DXfsaJ4luqWkmDAUHy917myKn2JizNdOnaRRoxrsVnyBwAIAzUBJiVn0rmKI2bTp7FOwo6Kqhpi+fcv3aUQDKyqq/KippES6997yn3fvXv3+EpIJLYcOlX9/7bVmJeTo6MrBJibGBKYrr/TprdQFgQUAmrFTp8xjpIohZtOmygvfnqlz56r1Mb17m6cVsNHJk2ZG1O7dZjQmN9eElNxc8zzw738vv7Znz8pbN1R05po4N91ktjs/M9i4R25+9CNf3lUZAgsAoIoTJ0yZxZmPlvbvr/56h8OUVPTta5Yn6dat/OjcmVlLfmfHDjPLqWKocZ+3bSstWVJ+7fnnm6G46py5Js7Yseb7zEyv19L4PLDMnz9fTz31lHJzczVw4EDNmzdPQ4YMqfbazZs3a+rUqdqwYYP27NmjP/3pT5o4cWK92jwTgQUA6i4/3yx5cuajpYrbC50pONiUVXTrZp5aVAwzCQlNbIXfpig724QQd7Cp+LVDB7P+jNv555tZUhVra7zEp3sJLVmyROnp6Vq4cKGSk5M1d+5cpaamKicnRx07dqxy/cmTJ5WYmKibbrpJDz74oFfaBAB4T9u20sUXm6OiI0fKw0tOjiml2LnTPFoqLjZPH6p7AuFwmKcKFUNMxVDT5Ff5bQwGDTJHbSxebPaEspnHIyzJycm66KKL9Nxzz0mSXC6X4uLidP/992vSpElnfW98fLwmTpxYZYSlPm1KjLAAQEMqLTWPkdwBZseO8vOdO8/9u619+8phpmKoYTp28+KzEZbi4mJt2LBBkydPLnstICBAKSkpWr16dZ0664s2AQC+Exgode1qjp/8pPLPLEs6erRygKkYaPLyzHps335rZuueqWVLKTGx+kDTpUvVTajRfHgUWI4eParS0lJFVdzMSlJUVJS2bdtWpw7Upc2ioiIVFRWVfe90Ouv02QAA73I4TAlEhw7VTzQ5ccJMTqku0Ozda7b92bzZHNWJiSlfY61r16pf2dqn6fK4hsUfZGRkaPr06XZ3AwDgodatpQEDzHGmkhIzg7dimHEf33xjZvgeOmSOmgbgO3asOcx07Wr2METj5FFgiYyMVGBgoPLOqBTOy8tTdHR0nTpQlzYnT56s9PT0su+dTqfi4uLq9PkAAP/QooWpY+neverP3I+a3EuSuL9WPD9+3MxsOnxYWreu+s8477zqw4z7KwXB/sujwBIcHKzBgwcrMzNTI0eOlGQKZDMzM3XffffVqQN1aTMkJEQhPMgEgGaj4qOmpKSqP7csMz27pjCzZ4/Zf/DYMXNkZVX/ORERNYeZ2FgzgsP6M/bw+JFQenq6xo8fr6SkJA0ZMkRz585VYWGh0tLSJEnjxo1Tp06dlJGRIckU1W7ZsqXs/MCBA8rOzlbr1q3V/YcYfa42AQA4G4fD7J/Urp10wQXVX+N0nn2E5uhRs9zIl1+aozoBAWZ7A/dq9zExVc9jYsw1LVr45l6bK48Dy+jRo3XkyBFNnTpVubm5GjRokJYvX15WNLt3714FuHeZlHTw4EFdUOFvz5w5czRnzhxdfvnlWrlyZa3aBACgvsLDpf79zVGdwkITXqoLNXv3mhlOLld5HU1NozRS+YjQ2YJNbKxZCT842Ac32wSxND8AALVQWmrqYw4dMtvyuIPLmee5ueba2jrvvKojNNWFnKa4ejB7CQEAYBOXyzxeOluocZ+XlNS+3fbtzQrCnTqZvZzc5xW/b9++cS2859Ol+QEAQM0CAkxxbseOZ1/93rJMAfC5Qs3Bg1JRUfmCexs31txmaGj1QabieXR046yvIbAAAGADh0OKjDRHTXU1kgk2331ngsuBA2ZbhAMHqp4fPSqdOlW+ds3ZPjc6+tzBpnVr799zffBICACAJuDUqXOHmoMHpdOna9deeHjVR09Tp3q3SJgaFgAAUIXLZQqHqwsz7vP9+80WCmcKCTFbJ3izRoYaFgAAUEVAgHkcFB0tDR5c83VOZ9Ug4+2w4ikCCwAAqCQ83By9e9vdk3IB574EAADAXgQWAADg9wgsAADA7xFYAACA3yOwAAAAv0dgAQAAfo/AAgAA/B6BBQAA+D0CCwAA8HsEFgAA4PcILAAAwO8RWAAAgN8jsAAAAL/XJHZrtixLkuR0Om3uCQAAqC3372337/GzaRKB5fjx45KkuLg4m3sCAAA8dfz4cUVERJz1GodVm1jj51wulw4ePKg2bdrI4XB4tW2n06m4uDjt27dP4eHhXm27MWju9y/xZ9Dc71/iz6C537/En4Gv7t+yLB0/flyxsbEKCDh7lUqTGGEJCAhQ586dffoZ4eHhzfIvqVtzv3+JP4Pmfv8SfwbN/f4l/gx8cf/nGllxo+gWAAD4PQILAADwewSWcwgJCdG0adMUEhJid1ds0dzvX+LPoLnfv8SfQXO/f4k/A3+4/yZRdAsAAJo2RlgAAIDfI7AAAAC/R2ABAAB+j8ACAAD8HoHlHObPn6/4+HiFhoYqOTlZ69ats7tLDSIjI0MXXXSR2rRpo44dO2rkyJHKycmxu1u2mTlzphwOhyZOnGh3VxrUgQMHdNttt+m8885Ty5Yt1b9/f33++ed2d6tBlJaWasqUKUpISFDLli3VrVs3zZgxo1Z7njRWH3/8sa699lrFxsbK4XBo2bJllX5uWZamTp2qmJgYtWzZUikpKdq+fbs9nfWBs91/SUmJHnnkEfXv319hYWGKjY3VuHHjdPDgQfs67APn+jtQ0T333COHw6G5c+c2SN8ILGexZMkSpaena9q0acrKytLAgQOVmpqqw4cP2901n1u1apXuvfderVmzRitWrFBJSYmuuuoqFRYW2t21Brd+/Xr95S9/0YABA+zuSoP67rvvdMkll6hFixb6z3/+oy1btuiPf/yj2rVrZ3fXGsSsWbO0YMECPffcc9q6datmzZql2bNna968eXZ3zWcKCws1cOBAzZ8/v9qfz549W88++6wWLlyotWvXKiwsTKmpqTp16lQD99Q3znb/J0+eVFZWlqZMmaKsrCwtXbpUOTk5uu6662zoqe+c6++A29tvv601a9YoNja2gXomyUKNhgwZYt17771l35eWllqxsbFWRkaGjb2yx+HDhy1J1qpVq+zuSoM6fvy41aNHD2vFihXW5Zdfbj3wwAN2d6nBPPLII9all15qdzdsc80111h33HFHpdd+/vOfW2PGjLGpRw1LkvX222+Xfe9yuazo6GjrqaeeKnstPz/fCgkJsV5//XUbeuhbZ95/ddatW2dJsvbs2dMwnWpgNf0Z7N+/3+rUqZO1adMmq2vXrtaf/vSnBukPIyw1KC4u1oYNG5SSklL2WkBAgFJSUrR69Wobe2aPgoICSVL79u1t7knDuvfee3XNNddU+nvQXLzzzjtKSkrSTTfdpI4dO+qCCy7QokWL7O5Wg7n44ouVmZmpr7/+WpL05Zdf6tNPP9XVV19tc8/ssWvXLuXm5lb630JERISSk5Ob5b+Jkvl30eFwqG3btnZ3pcG4XC6NHTtWDz30kPr27dugn90kNj/0haNHj6q0tFRRUVGVXo+KitK2bdts6pU9XC6XJk6cqEsuuUT9+vWzuzsN5o033lBWVpbWr19vd1ds8c0332jBggVKT0/X7373O61fv16//vWvFRwcrPHjx9vdPZ+bNGmSnE6nevXqpcDAQJWWlurJJ5/UmDFj7O6aLXJzcyWp2n8T3T9rTk6dOqVHHnlEt956a7PaDHHWrFkKCgrSr3/96wb/bAILzunee+/Vpk2b9Omnn9rdlQazb98+PfDAA1qxYoVCQ0Pt7o4tXC6XkpKS9Ic//EGSdMEFF2jTpk1auHBhswgsb775pl599VW99tpr6tu3r7KzszVx4kTFxsY2i/tHzUpKSnTzzTfLsiwtWLDA7u40mA0bNuiZZ55RVlaWHA5Hg38+j4RqEBkZqcDAQOXl5VV6PS8vT9HR0Tb1quHdd999eu+99/TRRx+pc+fOdnenwWzYsEGHDx/WhRdeqKCgIAUFBWnVqlV69tlnFRQUpNLSUru76HMxMTHq06dPpdd69+6tvXv32tSjhvXQQw9p0qRJuuWWW9S/f3+NHTtWDz74oDIyMuzumi3c/+41938T3WFlz549WrFiRbMaXfnkk090+PBhdenSpezfxT179ug3v/mN4uPjff75BJYaBAcHa/DgwcrMzCx7zeVyKTMzU0OHDrWxZw3Dsizdd999evvtt/W///1PCQkJdnepQQ0fPlwbN25UdnZ22ZGUlKQxY8YoOztbgYGBdnfR5y655JIqU9m//vprde3a1aYeNayTJ08qIKDyP5GBgYFyuVw29cheCQkJio6OrvRvotPp1Nq1a5vFv4lSeVjZvn27PvzwQ5133nl2d6lBjR07Vl999VWlfxdjY2P10EMP6b///a/PP59HQmeRnp6u8ePHKykpSUOGDNHcuXNVWFiotLQ0u7vmc/fee69ee+01/etf/1KbNm3KnlFHRESoZcuWNvfO99q0aVOlXicsLEznnXdes6njefDBB3XxxRfrD3/4g26++WatW7dOzz//vJ5//nm7u9Ygrr32Wj355JPq0qWL+vbtqy+++EJPP/207rjjDru75jMnTpzQjh07yr7ftWuXsrOz1b59e3Xp0kUTJ07UE088oR49eighIUFTpkxRbGysRo4caV+nvehs9x8TE6Mbb7xRWVlZeu+991RaWlr272L79u0VHBxsV7e96lx/B84MaS1atFB0dLR69uzp+841yFykRmzevHlWly5drODgYGvIkCHWmjVr7O5Sg5BU7fHCCy/Y3TXbNLdpzZZlWe+++67Vr18/KyQkxOrVq5f1/PPP292lBuN0Oq0HHnjA6tKlixUaGmolJiZajz76qFVUVGR313zmo48+qvZ/9+PHj7csy0xtnjJlihUVFWWFhIRYw4cPt3JycuzttBed7f537dpV47+LH330kd1d95pz/R04U0NOa3ZYVhNethEAADQJ1LAAAAC/R2ABAAB+j8ACAAD8HoEFAAD4PQILAADwewQWAADg9wgsAADA7xFYAACA3yOwAAAAv0dgAQAAfo/AAgAA/B6BBQAA+L3/D1tS4SsrOdOUAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#instantiate optimizer and criterion\n",
    "optimizer = optim.SGD(model_mlp_test.parameters(), lr=0.01, momentum=0.5)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "# train model\n",
    "model_mlp_test, losses, accuracies = train_val_model(model_mlp_test, criterion,\n",
    "                                                     optimizer, dataloaders,\n",
    "                                                     num_epochs=15,\n",
    "                                                     log_interval=5)\n",
    "\n",
    "#plot losses\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rpmgachOUCnX"
   },
   "source": [
    "We can see a drop in performance compared with the previous MLP model. So we can understand that although we have less learnable parameters, due to properties of CNNs (e.g., invariance and parameter sharing), which allow them to have fewer weights as some parameters are shared.\n",
    "\n",
    "## Question 3.5\n",
    "### What happens in each of the classifiers if for some reason your validation set does not always have the digits centered in your input image?\n",
    "\n",
    "CNNs are expected to be invariant to the location where important features occur in the input. In fact, it's not unusual that there is a dataset shift where the data acquisition process suffers some modification. We will do this by applying a transformation with horizontal translations to our validation dataset and see how robust each model is to these shifts.\n",
    "\n",
    "We can do this by going back to **0.1 - Create Dataloaders -\n",
    "MNIST dataset** cell to define the test transform using the following code\n",
    "\n",
    "```\n",
    "mnist_transform_val = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "     transforms.RandomAffine(0, translate=[0.1, 0]),\n",
    "     transforms.Normalize((0.1307,), (0.3081,))])\n",
    "```\n",
    "\n",
    "and replace\n",
    "\n",
    "`mnist_val_dataset = datasets.MNIST('../data', download=True, train=False, transform=mnist_transform)`\n",
    "\n",
    "with\n",
    "\n",
    "`mnist_val_dataset = datasets.MNIST('../data', download=True, train=False, transform=mnist_transform_val)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "5FH6naktdd-r"
   },
   "outputs": [],
   "source": [
    "# These random translations will be added in the end of this notebook, for now we skip this.\n",
    "mnist_transform_val = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "     transforms.RandomAffine(0, translate=[0.1, 0]),\n",
    "     transforms.Normalize((0.1307,), (0.3081,))])\n",
    "\n",
    "# Create MNIST validation dataloaders using transform validation with translation\n",
    "mnist_val_dataset = datasets.MNIST('../data', download=True, train=False, transform=mnist_transform_val)\n",
    "mnist_val_dataloader = DataLoader(mnist_val_dataset, batch_size=64, shuffle=True)\n",
    "\n",
    "# MNIST Dataloader for to get data into numpy for Logistic Regression\n",
    "mnist_val_dataloader_numpy = DataLoader(mnist_val_dataset, batch_size=len(mnist_val_dataset))\n",
    "X_y_val = next(iter(mnist_val_dataloader_numpy))\n",
    "X_val = X_y_val[0].numpy()\n",
    "y_val = X_y_val[1].numpy()\n",
    "\n",
    "dataloaders = dict(train=mnist_train_dataloader, val=mnist_val_dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VahBzmmedd-s"
   },
   "source": [
    "#### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0Fe3K78qdd-s",
    "outputId": "ce719bf2-3d63-4ac6-fb6b-c93f0bdd8fd0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score with penalty: 0.7169\n"
     ]
    }
   ],
   "source": [
    "# score the model on the validation dataset after translation\n",
    "X_val = scaler.transform(np.reshape(X_val, (X_val.shape[0], -1)))\n",
    "score = clf.score(X_val, y_val)\n",
    "\n",
    "print(\"Test score with penalty: %.4f\" % score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IKXir18Udd-t"
   },
   "source": [
    "#### MLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "Vehhij_Odd-t",
    "outputId": "d1318f43-d74f-48f3-f8fc-5c1fe9eb4643"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/14\n",
      "----------\n",
      "train Loss: 0.8217 Acc: 76.59%\n",
      "val Loss: 0.9435 Acc: 73.96%\n",
      "\n",
      "Epoch 2/14\n",
      "----------\n",
      "train Loss: 0.1752 Acc: 94.87%\n",
      "val Loss: 0.7523 Acc: 80.27%\n",
      "\n",
      "Epoch 4/14\n",
      "----------\n",
      "train Loss: 0.1089 Acc: 96.74%\n",
      "val Loss: 0.7016 Acc: 82.31%\n",
      "\n",
      "Epoch 6/14\n",
      "----------\n",
      "train Loss: 0.0756 Acc: 97.75%\n",
      "val Loss: 0.6575 Acc: 84.23%\n",
      "\n",
      "Epoch 8/14\n",
      "----------\n",
      "train Loss: 0.0547 Acc: 98.39%\n",
      "val Loss: 0.6414 Acc: 84.47%\n",
      "\n",
      "Epoch 10/14\n",
      "----------\n",
      "train Loss: 0.0403 Acc: 98.85%\n",
      "val Loss: 0.6263 Acc: 85.81%\n",
      "\n",
      "Epoch 12/14\n",
      "----------\n",
      "train Loss: 0.0295 Acc: 99.19%\n",
      "val Loss: 0.5816 Acc: 86.22%\n",
      "\n",
      "Epoch 14/14\n",
      "----------\n",
      "train Loss: 0.0215 Acc: 99.48%\n",
      "val Loss: 0.5697 Acc: 86.74%\n",
      "\n",
      "Training complete in 5m 23s\n",
      "Best val Acc: 86.74%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/DUlEQVR4nO3de1xUdf7H8fdwRwJSSRTFUDPNvJX3tNtGWpmt61bmmprVtpWVRrVqm9luJVpWVrq6+qvW326ubtv1l65ldLFWS4Oly2bewsQLoKWAqEBwfn98dxhAbgMzcxjm9Xw8zoMzhzPDZ1hj3vu9OizLsgQAAGCTILsLAAAAgY0wAgAAbEUYAQAAtiKMAAAAWxFGAACArQgjAADAVoQRAABgK8IIAACwVYjdBTREeXm5Dhw4oOjoaDkcDrvLAQAADWBZlgoLC5WQkKCgoNrbP/wijBw4cECJiYl2lwEAABohOztbnTp1qvX7fhFGoqOjJZk3ExMTY3M1AACgIQoKCpSYmFjxOV4bvwgjzq6ZmJgYwggAAH6mviEWDGAFAAC2IowAAABbEUYAAICtCCMAAMBWhBEAAGArwggAALAVYQQAANiKMAIAAGxFGAEAALYijAAAAFsRRgAAgK0IIwAAwFaEkdJS6eRJu6sAACBgBXYYOXZMuuYa6cYbpbIyu6sBACAgBXYY+fpr6f33pVdfle69V7IsuysCACDgBHYYGTpU+t//NefPPy8tXGhvPQAABKDADiOSNH689NRT5vy3v5VWrbK3HgAAAgxhRJJSUqQZM8z5TTeZrhsAAOAThBGnp56SrrvOzK6ZPFkqLra7IgAAAkKI3QU0G0FBZvyIwyHNmiWFh9tdEQAAAYEwUllEhLRmjd1VAAAQUOimqcvmzdK4cSyKBgCAFxFGanPihAkir79uxpCUl9tdEQAALRJhpDaRkWaab2io9Mor0n332V0RAAAtEmGkLpdeKq1cac4XLZKeftrWcgAAaIkII/WZMEF68klzft990urV9tYDAEALQxhpiPvuk+65x5xPmSJ99pm99QAA0IIwtbchHA7TRbN/v9ndt29fuysCAKDFIIw0VHCw9PLLUkiIOQcAAB5BN407wsNdQcSypGeflY4etbUkAAD8HWGksX77W7O53tix7GMDAEATEEYa68Ybpeho6aOPzKBWFkUDAKBRCCON1a+fWZ01NNTsZ/PAA3ZXBACAXyKMNMVll0kvvWTOn35aeuYZe+sBAMAPEUaaauJEacECc56SIv397/bWAwCAnyGMeMIDD0h33WW6bEpL7a4GAAC/QhjxBIfD7F3z2WempQQAADQYYcRTgoOl885zPT540KzYCgAA6kQY8Ybt26Vhw6Qrr5Ty8+2uBgCAZo0w4g3h4WYhtK++kn7xCxZFAwCgDoQRb0hKktatk047TfrgA2nqVBZFAwCgFoQRbznvPOm118zGen/7mzRrlt0VAQDQLBFGvOnyy6UXXzTnTz4pPfecvfUAANAMEUa8bdIkKTXVnC9fzvgRAACqCbG7gIAwc6bUqpXZXC883O5qAABoVggjvuBwSPfcU/Xa8eMmoAAAEODoprHD4sXSuedKBw7YXQkAALYjjPja8ePS889Le/awKBoAACKM+F6rVtL69VJ8vPTll9IvfymVlNhdFQAAtiGM2KFLF9eiaGlpZnO9n36yuyoAAGxBGLHL+edLr74qhYVJ//iHWaW1rMzuqgAA8DnCiJ1GjpT+/nezSutf/yq9/bbdFQEA4HNM7bXbz38uvfyytHu3OQcAIMAQRpqD66+v+rikRAoNNeuTAADQwtFN09wUFpo9bX7/e7srAQDAJ2gZaW7WrpU2bjRHZKRZSh4AgBYsoFtGDh+WPvlEKiiwu5JKbrjBtbHerFns9AsAaPECOoyMGCFdeKG0ZYvdlVQza5b08MPmfPp0s9svAAAtVECHkR49zNft2+2to0aPPCI98IA5v/126S9/sbUcAAC8JaDDSM+e5uu339pbR40cDmnBAumuuyTLku6/3wxuBQCghQnoAazNumVEMoHk2WfNQNZJk6ToaLsrAgDA4wI6jDTrlhGnoCDpiSeqXisqkqKi7KkHAAAPa1Q3zZIlS5SUlKSIiAgNGTJEW+oZAbpo0SL16NFDkZGRSkxM1L333quTJ082qmBPcraMZGebz3e/sHGj1LWr9OGHdlcCAIBHuB1G1qxZo5SUFM2dO1cZGRnq16+fRo0apby8vBrvX7VqlWbNmqW5c+dq27ZteuGFF7RmzRo9+OCDTS6+qdq2leLizPmOHfbW0mBLlkh5edLVV0ubNtldDQAATeZ2GHn66af161//WlOnTlWvXr20bNkytWrVSi+++GKN92/atEnDhw/Xr371KyUlJWnkyJGaMGFCva0pvuJsHWnWXTWVrVxpVmgtKpKuvFL6/HO7KwIAoEncCiMlJSVKT09XcnKy6wWCgpScnKzNmzfX+JwLLrhA6enpFeHju+++07p163TVVVfV+nOKi4tVUFBQ5fAW57iRZjuItbqICOmNN6SLLjKrtY0cKX35pd1VAQDQaG6FkcOHD6usrEzx8fFVrsfHxysnJ6fG5/zqV7/SH/7wB40YMUKhoaHq1q2bLrnkkjq7aVJTUxUbG1txJCYmulOmW/xiEGt1rVpJb78tDRkiHTkiJSf72RsAAMDF6+uMfPjhh5o3b57++Mc/KiMjQ6+99prWrl2rRx99tNbnzJ49W/n5+RVHdna21+pr9tN7axMdLa1fL513nnTokLRwod0VAQDQKG5N7Y2Li1NwcLByc3OrXM/NzVX79u1rfM6cOXM0adIk3XrrrZKkPn36qKioSLfddpt+97vfKSjo1DwUHh6u8PBwd0prtMrdNOXlZiat3zj9dOndd83U38ces7saAAAaxa2P3rCwMA0YMEBpaWkV18rLy5WWlqZhw4bV+Jzjx4+fEjiCg4MlSZZluVuvx3XpIoWGSidOSPv22V1NI8TFmTASFmYeW5Z09KitJQEA4A632wFSUlK0YsUKrVy5Utu2bdMdd9yhoqIiTZ06VZI0efJkzZ49u+L+MWPGaOnSpVq9erWysrK0YcMGzZkzR2PGjKkIJXYKCZHOOsuc+/2wi/Jy6e67pWHDzPRfAAD8gNsrsI4fP16HDh3Sww8/rJycHPXv31/r16+vGNS6d+/eKi0hDz30kBwOhx566CHt379fZ5xxhsaMGaPHH3/cc++iiXr0kLZtM2Fk5Ei7q2mCw4elt94yq7glJ5uF0dq0sbsqAADq5LCaQ19JPQoKChQbG6v8/HzFxMR4/PVnz5bmz5fuvNOsKebXdu6ULr5YOnhQGjBASkuTYmPtrgoAEIAa+vntT8M1vcYvp/fWpnt36b33zFiS9HTpqqukY8fsrgoAgFoRRuTH03tr06uXtGGDmW2zaZM0ZowZoQsAQDNEGJErjOzfLxUW2luLx/TvL73zjlmPZONG6ZNP7K4IAIAaEUYktW4ttWtnzv1mw7yGGDxYWrdOWrPG7GcDAEAzRBj5rxY1bqSyESOka691PT50SPrpJ/vqAQCgGsLIf7W4cSM12btXuuAC6eabzZokAAA0A26vM9JStdiWkcq++krKypJ27ZIiI6VlyySHw+6qAAABjjDyX86WkRYdRkaPlv76V2niRGn5cikiQlq0yPOBpKREKiiQiouljh1d199806x/UlDgOgoLpUGDpNtuM8vhAgACDoue/dfu3WZZ+IgIsyxHM1ip3ntWrpRuusmcz5wppaZWDSS7d5v9bQoKpPz8quEhNlaaNs117/jxZqG1yvcUF5vvnXOO9M03rnt795b+85+aaxo4UPrXv1x77AAA/F5DP7/5v6L/lZRkPgdPnjRDK7p0sbsiL5oyxaw7cscd0oIF0kcfSZs3u75/8cVmnnNNevWqGkb+85/aA0b1gbI/+5l09tlSTIzrkKQ//lG68EKCCAAEKMLIfwUHm8VL//MfM4i1RYcRSbr9dpO8UlJOXVylUycpKKhqaHAenTtXvXfRIqm01LSYVL7vtNNO7XZ57rmaa7nnHqlVK9fjb74x/yP84hdNfpsAgOaPMFJJz54mjHz7rXTFFXZX4wMzZkjXXScdP171+qefNvw1kpObXodzkRfJzPL5zW/MIm0//7n0/PNSYmLTfwYAoNliam8lATG9t7qOHU2TUHNRViZddJFpVXnzTdMt9Oyz5joAoEUijFQSENN7m7vQUOnxx6V//9usiXLsmGnBGTrUXAMAtDiEkUoCYnqvv+jdW/r4Y7MWSmys9PnnZsbNhx/aXRkAwMMII5U4w0hOjpnRCpsFBZnxI9u2SddfL/XpIw0fbndVAAAPI4xUEhsrdehgzgNq3Ehz16GD2ezv449NN45k1jKZMUM6cMDW0prsnXfMgm/79tldCQDYhjBSTUAOYvUX0dGu8/nzzcDWc84x65T4y147ublV13Bp1UpasUIaMkRKT7evLgCwEWGkGgax+omf/1waPNis+Dptmum++eoru6uqWX6+9Oc/S6NGSQkJ0rx5ru/17y+1bWtaeC66SHrjDZuKBAD7EEaqoWXET/TvL23aZNYhiY42a6Ocf740a9ap66bY4cQJ6R//kH75Syk+Xpo6VXr3XdOCs2eP677oaLP8/siRpu5x46SFC6Xmv0sDAHgMYaQaWkb8SHCwdNddZoDruHFm+fkFC6S777a7MtNqc9110muvmfEt55wjPfqo2cdn7dqq98bGmmt33GFCyAMPmHEkpaX21A4APkYYqcbZMrJzJ+ts+Y2OHaVXXzWLpJ19tvS73/nuZ5eXmw3+7r+/ani46iqzdP7MmVJmplna96GHzG6MNQkJkZYsce2i/PLL0o4dvngHAGA7du2tpqzMbKty8qS0a5fUrZtXfxw8rbzcTAl2mjnTBIBbbql6vSksS/ryS+lvfzPH3r3m+j//6dpH4PhxswV0Y37m22+bYMPePAD8HLv2NpJzw7yvvjJdNYQRP1P5w3/rVumJJ8z5//6v9Kc/meXlGysnR/qf/zEB5JtvXNejo01waN/eda3yxn/uuvrqqo8//9yMQbnwwsa/JgA0Y3TT1MA5boRBrH7uvPOkp5+WoqLMxnv9+0tz5phmr4aqPGU4J8c8/5tvpPBwM07llVfMdN2VK83re9q+fdKYMWZDwr/8xfOvDwDNAGGkBgxibSFCQqR77zXhYcwY0/Xx2GNmJde0tNqfd+SI9MIL0mWXSTff7Lrer590663SSy+ZAPLqq9K110qRkd57D23amGnLJSXS5MkmDPnLmioA0ECMGanByy9LN95oln346COv/zj4gmWZmS133y0dPCjFxZkptlFR5vvHj0v/93+mC2bdOtdg1NNOkw4fNi0hdikvN4Ny5883j6+/3qxb4s0QBAAe0NDPb1pGakDLSAvkcJg1P7ZtM4ukObtvJNPa0K6ddMMNZkZOaalpPUlNNYOH7AwikhkHk5oqvfiiWQ7/73+XLr3UtM4AQAvAANYanH22+ZqXZ1rsW7e2tx54UGystHhx1WsREVJRkdSlizRhgjl697anvrpMnWpqHDdO+uwzae5cs6sxAPg5umlq0amT2UJk82Zp6FCf/EjY5eBB6fvvzf4wDofd1dRvxw7TbfPSS6YbCQCaKbppmsi5+BldNQGgQweTOP0hiEim6e6VV1xBxLLqHpALAM0cYaQWTO+F31iwwEz9nT6dZYMB+CXCSC0YxAq/4WzRee456ZprpMJCe+sBADcRRmrB7r3wGzNnmm6biAgzLXn4cNcS9QDgBwgjtXC2jOzaZTaDBZq1a6+VNm40S9J/9ZXZNXjLFrurAoAGIYzUolMns6ZUaamUlWV3NUADDBpkpvz27WvWIElOln74we6qAKBehJFaBAUxowZ+qHNnsw/P6NHSwoVS27Z2VwQA9SKM1IFxI/BL0dHSW29Jt93mupadbfa3AYBmiDBSB2bUwG8FVfpP+4cfzPLxl19Otw2AZokwUge6adAibN9u9jbYuNEs7rZjh90VAUAVhJE6sPAZWoQLLpA2bZLOPNNMDxs6VPrwQ7urAoAKhJE6ODfMO3yY1m34ud69zUyboUPN7o+XX252AQaAZoAwUoeoKCkx0ZzTOgK/Fx8vvf++NH68WTznlluk5cvtrgoACCP1YRArWpTISGnVKmnOHKlbN2ncOLsrAgDCSH2Y3osWJyhI+sMfpMxMKS7OXCsoMGNKfvUradkyads2sxswAPhAiN0FNHe0jKDFOu001/knn5j9bPbulf72N3PtjDOkCy+ULr5YuvpqqWtXe+oE0OLRMlIPpvciIFx6qRlP8sgj0s9+ZrpzDh2SXntNmj5devdd1725uWYwbGmpbeUCaFloGamHs2Xku+/M397QUHvrAbwiMtIEkksvNY9LSqTPPzdrk3z0kXTJJa57X39duuMOM8J7+HDpootM68mgQVJ4uC3lA/BvDstq/h3DBQUFio2NVX5+vmJiYnz6sy3LrK5dVGS60Z3hBAhYTz0lPf64mSJcWUSEmTq8YoV01ln21AagWWno5zdhpAEGDJAyMqQ33pB+/nOf/3ig+Skvl/7zH9Nq4mw9ycszg2OPHJGc/53+6U/Snj2m9WT4cNd1X/vpJyk/Xzp61PX16FEzbmbkSHtqAgJAQz+/6aZpgJ49TRj59lvCCCDJhI4+fcxx112mCXHHDumrr6oGjpUrpc2bpfnzzXPOO88Ek4suMoNjG7KrsGWZMFG5j/Sdd1yBonrAOOssM1vIKSFBOniw5tceMqRqGPnPf6RevSSHo8G/CgBNRxhpAKb3AvVwOMx/KM7/WJymTTNpfuNGafduKT3dHM88YxZhO3jQPNeyzC7DNYWL/HwzqPaf/3S97rXXSseO1VzL0KFVw0hwsOs8Kko6/XQpNtZ87dPH9b3sbOn886V+/cxA3iuvJJQAPkIYaQCm9wKNNHGiOSRp/34TSpzdOpVbIBwOafXq2gPG0aNVHw8fLhUXu0KF8+vpp5v1UirbtElq1cq02NQ1Aj09XQoJkbZulUaPNq0mjzwijRpFKAG8jDEjDfDFF1L//lLr1maPGv4uAR5QfXras8+aVgxnqKgcME4/3Ywk97a8POnJJ6UlS6QTJ8y1oUOl3//e7OfDf/yAWxjA6kHHj5txbpZl/ladcYbPSwDgS7m50hNPSH/8o3TypGlZyc6W2rSxuzLArzT085tFzxqgVSupc2dzTlcNEADi480U5qwsacYM6YEHqgaR9PSWsVz+li1mbZmzz5YWLzazpAAbEEYayDluhEGsQABp394Mtn3kEde1jRulgQPNQnAffmhTYY2Uk2NWcHSKjDTvYedO6e67zSwn/h8XbEAYaSAGsQKQJH39tRQWZkKJc9XajRvtrqp2OTmmu+nSS6WOHaWHHnJ9r3dvsxbMokWmL/pf/zKziebNY7l/+BRhpIGY3gtAknTnnWaa8p13mlDy4YdmOfzLLjMbDjYH1QPItGmmzvJyM/DN2cXkcJgp1dOnmzVWrrzSbAXwu9+Z57aErij4BcJIA9EyAqBCp05mxs2uXdLtt5tZQe+/L02ZYhZos9tVV1UNIIMHm1lCWVnSe+/VPCuoc2dp7VrpL38xi9Fdfz2zh+AzrDPSQM6Wkawss7wB+4EBUGKitHSpNGuW6doYMcKsVSKZFobMTBMEvCUnx+ys/Oab0quvmq4WSfrlL01Auu46s0BcUlLDXs/hkG68UbriCrOWgdOmTSZkXXSRx98CIDWyZWTJkiVKSkpSRESEhgwZoi1bttR5/9GjRzVt2jR16NBB4eHhOvvss7Vu3bpGFWyXDh3MMgdlZaaFFgAqnHmmGXsxaZLr2p//bBZOu/JKM2vFU2rqgnn3XdOq4TR7tvTZZ9L99zc8iFQWF+daufb4cWnyZNMVdeedUkGBR94GUJnbYWTNmjVKSUnR3LlzlZGRoX79+mnUqFHKy8ur8f6SkhJdfvnl2rNnj/7xj39o+/btWrFihTp27Njk4n3Judq1RFcNgAbYu9d8oK9fb0LJ6NHS5583/vUyM2seA+LsghkxwnVvkAd74MvKzHgYybQCnXtu1eADeILlpsGDB1vTpk2reFxWVmYlJCRYqampNd6/dOlSq2vXrlZJSYm7P6pCfn6+JcnKz89v9Gt4wo03WpZkWfPm2VoGAH+xa5dl3XSTZQUHmz8ekmVdfbVlff55/c89eNCyvv3W9Tgry/Uagwdb1pNPmmu+8sEHltWtm6uGX/3KsvLyfPfz4Zca+vntVnwuKSlRenq6kpOTK64FBQUpOTlZmzdvrvE5b731loYNG6Zp06YpPj5evXv31rx581RWVtaUDGULBrECcEu3btJLL5k/GpMnmxaLt9+WHnyw5vurd8Hcd5/re0lJpusnK6tpXTCNdckl0pdfmgXggoKkVavM/kJ79/quBrRYbg1gPXz4sMrKyhQfH1/lenx8vL6t5RP6u+++0/vvv6+JEydq3bp12rVrl+68806VlpZq7ty5NT6nuLhYxcXFFY8LmkkfJdN7ATTKWWdJK1eaKbOPPmrGXjjl5prxJh98YNYrqbwKan6+eezsdpkyxbd1V9eqlVkm//rrpZtvNmNlEhPtrQktgtdn05SXl6tdu3Zavny5goODNWDAAO3fv19PPvlkrWEkNTVVv//9771dmtsqt4xYFrPeALjp7LPN1NnKnnhCevpp1+PBg92fBeNrAwea8S/Hjrn+EP74o5nZc/PNnh2zgoDg1r+YuLg4BQcHKzc3t8r13NxctW/fvsbndOjQQWeffbaCnSOzJZ1zzjnKyclRSUlJjc+ZPXu28vPzK47s7Gx3yvSas84y/93l55t1gwCgyY4fN90yznVA7OiCaYywsKr79dx3n/TrX5v3smOHfXXBL7kVRsLCwjRgwAClpaVVXCsvL1daWpqGDRtW43OGDx+uXbt2qbxS0+OOHTvUoUMHhYWF1fic8PBwxcTEVDmag4gIqUsXc864EQAesXSpWTDNHwJIbSxLOv98KSrKdDX162dafJrDAnDwC263paWkpGjFihVauXKltm3bpjvuuENFRUWaOnWqJGny5MmaPXt2xf133HGHfvzxR02fPl07duzQ2rVrNW/ePE2bNs1z78KHGDcCANU4HGajva+/lkaOlE6elGbONFOaMzPtrg5+wO0wMn78eC1cuFAPP/yw+vfvr8zMTK1fv75iUOvevXt18ODBivsTExP1zjvvaOvWrerbt6/uueceTZ8+XbNmzfLcu/AhZtQAQC2Sksy6Kn/+s1nBNSPDjC957TW7K0Mz57Cs5r8TUkFBgWJjY5Wfn297l82f/mS2orjySsnPFpEFAN/JyTGtJR9/LH3zTdXxJXYqL5cOHJB27jR7C+3cKe3ZYwYF3nqr1LWr3RW2KA39/GZvGjc5W0bopgGAOrRvL73yigklziBiWdKyZWb/m+ho7/1sZ+DYtcuMZXF+CC5aZNZ4OXGi5uddfrkrjOTnmzEwIXxM+gK/ZTc5w0hWlukWjYiwtx4AaNYqz7RctcqssTJ/vmlmvuKKpr/+7t1mAHDllo7du12BIy1N+tnPzHlsrLkeHGxmI3TvblpEOnc2410uvND1unPmmHqvuUYaN05KTuYPvhcRRtzUrp3595yfb/7d9+5td0UA4CcSEkwIyMoyfd2TJknPPCO1bVvz/ZVbOCqHjQcfNGNRJLNHz223nfpcZ+A4edJ1bexYEzjOPNPsalyXjz+WfvjBrKD70ktmR+SrrpJ+8QvztZnM8mwpGDPSCEOHmqUAXnnFrEsEAGigoiLT6rBokem2OeMM6bHHpPHjzf/Tk8wuxPffb8JHTV0qK1aY8R2StHWrNHeuaeFwtnR0796wwFGXn34ygeT1182xb5/re927t5y1VH780Szzf9FFXlmsjjEjXtSjhwkjjBsBADdFRZkVZ6+/XrrlFjO49Te/MWMzbr7Z3BMaKn31lTl3tnA4Q0b37lV3KB40yDuzCUJCzAJul14qPfusWXH2tddMMBk1ynXfTz+Z1pLkZPO1c2fP1+IpubmmJenLL6UvvjCHM2Tt3Gl+xzYhjDQC03sBoImGDjVTf1NTzeaAhw+7vnf++SZgeKKFwxMcDhN6Bg0y9VbaO00bN5rND99+W5oxQxowwISSceOkc86xp978fFfgGDFC6t/fXN+0SbrhhlPv79JFOnTI1jBCN00jvPaa9Mtfmi7LrVvtrgYAYJvcXGn1avPB8PHHpuvJqUcP6bnnzEJw3pKfL733nqul44svpO+/d33/0Uelhx4y53v2mO6wfv1cR58+ru4xL6CbxosqT+9lwzwACGDx8dL06ebIy5Peest05bz3nvmQqDw498svzRiNESPcnzJcWGi6rr74wmy4eNll5np2ds2DFxMTTdio3NqRlGTGGDRDtIw0QnGx6fYsK5P27zcDxAEAqFBQIG3YYLprnP+P9aabpJUrTUCpa8rwyZNmJdvKYzt273Z9/6abzAwfSSotNTOEzj1X6tvXBJC+fZvNInO0jHhReLjpYtu1ywRfwggAoIqYGNOfX1nbtiYkVJ8yPGqUuXfCBHNfWZkJKtXbCjp0MGFjwADXtdBQ6dNPvftefIAw0kg9e5ow8u23ZrA1AAB1euopacGCU6cMv/qqWanWGUaiokzLSWysa2xH375mGnQLRRhppB49zOBppvcCABqspinDb7116hiSN96wpTy7EEYaiem9AIAmqTxlOMB5frm1ANGjh/lKGAEAoGkII43kbBnZu1c6ftzeWgAA8GeEkUaKi5NatzaDnXfutLsaAAD8F2GkkRyOqoufAQCAxiGMNAGDWAEAaDrCSBM4B7HSMgIAQOMRRpqAlhEAAJqOMNIElVtGmv8OPwAANE+EkSbo1s0smldUZDbMAwAA7iOMNEFoqNS1qzmnqwYAgMYhjDQR03sBAGgawkgTMYgVAICmIYw0EdN7AQBoGsJIE9EyAgBA0xBGmsjZMpKdbWbVAAAA9xBGmqhtW7NpniTt2GFvLQAA+CPCiAc4W0foqgEAwH2EEQ9gei8AAI1HGPEAWkYAAGg8wogH0DICAEDjEUY8oHIYKS+3txYAAPwNYcQDunQx+9ScOCHt22d3NQAA+BfCiAeEhEhnnWXOGTcCAIB7CCMewrLwAAA0DmHEQ1gWHgCAxiGMeAjTewEAaBzCiIcwvRcAgMYhjHiIs2Vk/36psNDeWgAA8CeEEQ9p3Vpq186cs2EeAAANRxjxIAaxAgDgPsKIBzG9FwAA9xFGPIiWEQAA3EcY8SCm9wIA4D7CiAc5W0Z27pTKyuytBQAAf0EY8aCkJCksTDp5Utq71+5qAADwD4QRDwoOlrp3N+cMYgUAoGEIIx7GIFYAANxDGPEwpvcCAOAewoiH0TICAIB7CCMeRssIAADuIYx4mDOMHDwo5efbWwsAAP6AMOJhsbFS+/bmnNYRAADqRxjxAue4EcIIAAD1I4x4AYNYAQBoOMKIFzCIFQCAhiOMeAEtIwAANBxhxAucLSNsmAcAQP0II17QubMUESGVlEh79thdDQAAzRthxAsqb5hHVw0AAHVrVBhZsmSJkpKSFBERoSFDhmjLli0Net7q1avlcDg0duzYxvxYv8L0XgAAGsbtMLJmzRqlpKRo7ty5ysjIUL9+/TRq1Cjl5eXV+bw9e/bo/vvv14UXXtjoYv2Jc9wILSMAANTN7TDy9NNP69e//rWmTp2qXr16admyZWrVqpVefPHFWp9TVlamiRMn6ve//726du3apIL9BS0jAAA0jFthpKSkROnp6UpOTna9QFCQkpOTtXnz5lqf94c//EHt2rXTLbfc0vhK/QzTewEAaJgQd24+fPiwysrKFB8fX+V6fHy8vq3lU/eTTz7RCy+8oMzMzAb/nOLiYhUXF1c8LigocKfMZuHss83XvDzpyBGpdWt76wEAoLny6myawsJCTZo0SStWrFBcXFyDn5eamqrY2NiKIzEx0YtVekd0tNSxozmnqwYAgNq51TISFxen4OBg5ebmVrmem5ur9s6taivZvXu39uzZozFjxlRcKy8vNz84JETbt29Xt27dTnne7NmzlZKSUvG4oKDALwNJjx7S/v0mjAwdanc1AAA0T261jISFhWnAgAFKS0uruFZeXq60tDQNGzbslPt79uypr776SpmZmRXHNddco0svvVSZmZm1Bozw8HDFxMRUOfwR40YAAKifWy0jkpSSkqIpU6Zo4MCBGjx4sBYtWqSioiJNnTpVkjR58mR17NhRqampioiIUO/evas8//TTT5ekU663REzvBQCgfm6HkfHjx+vQoUN6+OGHlZOTo/79+2v9+vUVg1r37t2roCAWdpWY3gsAQEM4LMuy7C6iPgUFBYqNjVV+fr5fddns3SudeaYUGiodPy6FuB39AADwXw39/KYJw4s6dZIiI6XSUikry+5qAABonggjXhQUxLgRAADqQxjxMmcYYdwIAAA1I4x4GdN7AQCoG2HEy+imAQCgboQRL2N6LwAAdSOMeJlzw7zDh6UffrC3FgAAmiPCiJdFRUnOVe9pHQEA4FSEER9gECsAALUjjPgA03sBAKgdYcQHaBkBAKB2hBEfoGUEAIDaEUZ8wNkysnu32acGAAC4EEZ8oGNHM6vmp59MIAEAAC6EER9wOOiqAQCgNoQRH2EQKwAANSOM+AgtIwAA1Iww4iO0jAAAUDPCiI9U3r3XsuytBQCA5oQw4iPdu5uBrEeOmE3zAACAQRjxkVatpM6dzTldNQAAuBBGfMg5boRBrAAAuBBGfKjyuBEAAGAQRnyIlhEAAE5FGPEhpvcCAHAqwogPObtpsrKk4mJ7awEAoLkgjPhQhw5SdLRUVsaGeQAAOBFGfIgN8wAAOBVhxMcYNwIAQFWEER9jei8AAFURRnyM6b0AAFRFGPGxyt00bJgHAABhxOfOOssMZM3Pl/Ly7K4GAAD7EUZ8LCJC6tLFnDNuBAAAwogtmN4LAIALYcQGTO8FAMCFMGIDpvcCAOBCGLEB03sBAHAhjNig8oZ5J0/aWwsAAHYjjNggPl6KjTXrjOzaZXc1AADYizBiA4eDQawAADgRRmzC9F4AAAzCiE1oGQEAwCCM2ITpvQAAGIQRm1Se3suGeQCAQEYYsUm3blJQkFRYKB08aHc1AADYhzBik/BwqWtXc84gVgBAICOM2MjZVZORYW8dAADYiTBio6FDzddZs6QXXrC3FgAA7EIYsdF990k33CD99JN0663Sb38rlZXZXRUAAL5FGLFRRIS0apX0yCPm8ZNPSr/8pXTsmK1lAQDgU4QRmzkc0ty5JpSEh0tvvildeKG0b5/dlQEA4BuEkWZiwgTpww+ldu2kzExp8GBp61a7qwIAwPsII83I0KHSli1S795m7ZGLLpL+8Q+7qwIAwLsII83MmWdK//qXdNVV0smT0nXXSY8/ziqtAICWizDSDMXESG+9Jc2YYR4/9JA0ebJUXGxrWQAAeAVhpJkKDpaeeUZautSc//Wv0mWXSYcO2V0ZAACeRRhp5m6/XVq/XoqNNd03Q4ZI33xjd1UAAHgOYcQPJCdLn35qNtfLypKGDZPeecfuqgAA8AzCiJ/o2dMEkgsvlAoKpNGjpSVL7K4KAICmI4z4kbg4acMG6aabzLLxd90l3X23WU4eAAB/RRjxM+Hh0osvSvPnm8eLF0tXXy3l59tbFwAAjdWoMLJkyRIlJSUpIiJCQ4YM0ZYtW2q9d8WKFbrwwgvVunVrtW7dWsnJyXXej/o5HNLMmdKrr0qRkWb8yAUXmPEkAAD4G7fDyJo1a5SSkqK5c+cqIyND/fr106hRo5SXl1fj/R9++KEmTJigDz74QJs3b1ZiYqJGjhyp/fv3N7n4QDdunPTxx1JCgplhM3iwmXEDAIA/cViWe2t7DhkyRIMGDdLixYslSeXl5UpMTNTdd9+tWbNm1fv8srIytW7dWosXL9bkyZMb9DMLCgoUGxur/Px8xcTEuFNuQNi/X7rmGikjQwoLk154QbrxRrurAgAEuoZ+frvVMlJSUqL09HQlJye7XiAoSMnJydq8eXODXuP48eMqLS1VmzZtar2nuLhYBQUFVQ7UrmNHaeNG6Re/kEpKpEmTpDlzpPJyuysDAKB+boWRw4cPq6ysTPHx8VWux8fHKycnp0GvMXPmTCUkJFQJNNWlpqYqNja24khMTHSnzIAUFWU21XM2Tj32mHTDDdLx4/bWBQBAfXw6m2b+/PlavXq1Xn/9dUVERNR63+zZs5Wfn19xZGdn+7BK/xUUJKWmSi+9JIWGSq+8Il1yidkBGACA5sqtMBIXF6fg4GDl5uZWuZ6bm6v27dvX+dyFCxdq/vz5evfdd9W3b9867w0PD1dMTEyVAw13003Se+9JbdpIW7eaga2ZmXZXBQBAzdwKI2FhYRowYIDS0tIqrpWXlystLU3Dhg2r9XlPPPGEHn30Ua1fv14DBw5sfLVosIsukrZsMSu37tsnjRhhdgIGAKC5cbubJiUlRStWrNDKlSu1bds23XHHHSoqKtLUqVMlSZMnT9bs2bMr7l+wYIHmzJmjF198UUlJScrJyVFOTo6OHTvmuXeBGnXrJm3ebPa2KSqSxo6VnnpKcm/+FAAA3uV2GBk/frwWLlyohx9+WP3791dmZqbWr19fMah17969OlhpkMLSpUtVUlKia6+9Vh06dKg4Fi5c6Ll3gVqdfrq0bp30m9+YEHL//dJtt5lZNwAANAdurzNiB9YZaTrLkp57TkpJMVN+L73UzL6pY4Y1AABN4pV1RuC/HA5p+nQzbuS006QPPpCGDpV27LC7MgBAoCOMBJjRo6VNm6Qzz5R27jSBZPVqswswAAB2IIwEoD59pM8+M0HkyBFpwgSpRw9p6VLpxAm7qwMABBrCSICKjzddNXPnmnEju3dLd94pde4s/eEP0g8/2F0hACBQEEYCWESE9Mgj0t69ZnBrUpJ0+LAJKImJ0t13S1lZdlcJAGjpCCNQVJQJHjt3mvEj559vumsWL5bOOsvscZOebneVAICWijCCCiEh0vjx0uefm+XkR40y04DXrJEGDpQuu0x65x0WTQMAeBZhBKdwOEzwWL/e7GkzcaIUHCy9/750xRVS//7SX/8qlZbaXSkAoCUgjKBO/fqZ4LF7tzRjhunS+fJLadIks9z8M89IhYV2VwkA8GeEETTImWea4JGdLT3+uNSunTlPSTEzcB58UMrJsbtKAIA/IozALa1bm+Dx/ffS8uXS2WdLR49KqakmsPz619L27XZXCQDwJ4QRNEpEhAke27ZJr78uDRtmNt/7n/+RzjlH+sUvzEqvAADUhzCCJgkKksaONcHjk0+ka64xs23eeEMaPlwaMUJ6800zKwcAgJoQRuAxw4eb4PHNN9Itt0hhYdK//mXCSq9e0gsvSMXFdlcJAGhuCCPwuHPOMd01e/ZIs2ZJsbFmHMmtt5pVXufPN+NMAACQCCPwog4dzMDWvXulhQulTp3MjJvZs81y8/fdZ2bkAAACG2EEXhcTY4LH7t3SypVS797SsWPS009LXbtK48ZJL74oHTxod6UAADs4LKv5L+5dUFCg2NhY5efnKyYmxu5y0ESWZVZ3ffJJs3NwZeefL40ebY6BA83KrwAA/9TQz2/CCGyVmWlm3qxda/bEqSwuTrryShNMRo40a5wAAPwHYQR+JyfHtJisXSu9+65UUOD6XnCwdMEFJphcdZXp6nE47KsVAFA/wgj8WmmpmRa8dq20bp2ZLlxZYqIJJaNHSz/7mdkzBwDQvBBG0KJkZZlQsm6d2T345EnX98LDpUsucY016drVtjIBAJUQRtBiHT8uffihaTVZu9bsk1NZjx6uYDJihFl8DQDge4QRBATLMvvjOLtzPvlE+ukn1/ejo6XLLzddOlddZdY+AQD4BmEEASk/3wx+dXbp5OVV/f5557laTQYNYuowAHgTYQQBr7xcSk83oWTtWmnr1qrfj4uTrrhCGjXKzNTp0oUZOgDgSYQRoJrcXNfU4XfeqTp1WJLatZOGDZOGDjVfBw5klg4ANAVhBKhDaam0aZMJJhs3ShkZ5lplwcFSv36ucDJsmJmpQ+sJADQMYQRww8mTJpBs3uw6Dhw49b4zzqjaejJoEK0nAFAbwgjQRNnZVcNJba0nfftWbT3p1o3WEwCQCCOAx508Kf3731UDyv79p953xhkmnFRuPTntNN/XCwB2I4wAPpCdLX36adXWk5KSqvcEBZnWk8rdO2edResJgJaPMALYoLj41NaTfftOvS8uztV6MmSI1KuXWZCNgAKgJSGMAM3Evn1VW0/S009tPZHMarE9ekg9e5rDeX7WWVJEhO/rBoCmIowAzZSz9cQZUNLTzUaA5eU13x8UJCUlnRpSevY041NoTQHQXBFGAD9SXCzt3i19+605tm93nVdfnK2y00+vOaR06yaFhvqsfACoEWEEaAEsy6wcW1NI+f578/2aBAebQFI9qPToIbVt69v3ACBwEUaAFu7ECWnnzlNDyvbtUlFR7c+Li6saUnr0MONSkpKkyEiflQ8gABBGgABlWWb9k8ohxfk1O7vu5yYkmCXvu3QxXysf7dub8SsA0FCEEQCnKCqSduyo2oqyfbsZr1JYWPdzIyJODSnOx126sLAbgFMRRgA0mGVJP/4offdd1SMry3zdu1cqK6v7Ndq1O7U1xXkkJJhxLAACC2EEgMeUlpounuphxXkcOVL380NDzZiU2sIK/1kDLVNDP79DfFgTAD8VGuoKDjU5etTVilL92LPHhJmdO81Rk5gY03rSsaM5ajpv314K4S8W0CLRMgLAq8rKzCq0NXX/fPeddOhQw17H4ZDi4+sOLB07mrVXWAgOaB7opgHgFwoLzeyfAwfM15rODx6UfvqpYa8XGekKJ7UFlg4dWGIf8AW6aQD4heho17ontSkvl/Ly6g4s+/ebQbgnTpjZQbt31/1z27Z1hZQOHczS+u3amcN5fsYZ5iC4AN5FywiAFuPECdOKUldLy/79Zvl9d8TE1BxWajqPi2MpfsCJlhEAAScysu6BtpKZxnzkSNWQkptrWl4OHTJfncehQ6Z7qKDAHPW1tji1aVN/eHF+bdOGac8AYQRAQHE4TABo00bq06fuey3LzBSqHFQqB5bq54cPmy6lH380x/bt9dcTFGS6jOLizFH5vPrh/F5sLIN00bIQRgCgFg6H1Lq1OXr0qP/+sjITQuoKLJXPf/zRhJdDhxo+q0gyU5yrh5baQozzenQ0AQbNF2EEADwkONg16LVXr/rvLy01rSmHDkk//GDOKx81XSsqMl1HubnmaKjQ0NqDStu2Zkp0bOypX2NjWd8F3sc/MQCwSWiomcnToUPDn3PyZM0hpbbwcviwGdhbWmoG9x486H6dUVGnBpSaQktt1047jVYZ1I0wAgB+JCLCtV5KQx0/Xn/LS36+6zh61HwtKjLPLyoyx4EDjas5KMgVTmoKLZW/Vj+cz2GGUstGGAGAFq5VK3MkJrr3vNJSM4vIGU4qB5XK53VdKy0142KOHKl/D6O6REXVHFRqCzDVH4eHN/5nw/sIIwCAGoWGmvEkbds27vmWZbqV6gst1a9XPo4dM6/lbJ3Zv79xtURG1h1eYmJMd1JNR3S06zwqyrT0wLMIIwAAr3A4TAiIjHRvXExlznVeqoeU6mGmtuv5+eZ1TpwwR05O096TZFqZ6gosDQk1lY+oKAYJB/jbBwA0ZyEhrnVhGqOszOx/VFd4cbbAHDtm7nWeVz4KC013k2TG4Bw/bqZoe0pEhAklUVGugFLTY3fPIyP9Y/AwYQQA0GIFB7u6Y5rCssw2ArWFlbqCTG3fKyw0YUky3VnOmVKe5HA0PNhMm1b36sXeRBgBAKAeDodpvYiIMOvIeIJlSSUlJpQUFZmA4hwbU/2xu987ccL1M5zhpz7XXUcYAQAgoDgcZpZPeLhZfM6TystNV1JdIab6Y3dnW3kSYQQAgBYmKMg1QNYfMEEJAADYqlFhZMmSJUpKSlJERISGDBmiLVu21Hn/K6+8op49eyoiIkJ9+vTRunXrGlUsAABoedwOI2vWrFFKSormzp2rjIwM9evXT6NGjVJeLXOcNm3apAkTJuiWW27Rv//9b40dO1Zjx47V119/3eTiAQCA/3NYlmW584QhQ4Zo0KBBWrx4sSSpvLxciYmJuvvuuzVr1qxT7h8/fryKior09ttvV1wbOnSo+vfvr2XLljXoZxYUFCg2Nlb5+fmKiYlxp1wAAGCThn5+u9UyUlJSovT0dCUnJ7teIChIycnJ2rx5c43P2bx5c5X7JWnUqFG13i9JxcXFKigoqHIAAICWya0wcvjwYZWVlSk+Pr7K9fj4eOXUssZuTk6OW/dLUmpqqmJjYyuORDvnGwEAAK9qlrNpZs+erfz8/IojOzvb7pIAAICXuLXOSFxcnIKDg5Wbm1vlem5urtq3b1/jc9q3b+/W/ZIUHh6ucPZ7BgAgILjVMhIWFqYBAwYoLS2t4lp5ebnS0tI0bNiwGp8zbNiwKvdL0oYNG2q9HwAABBa3V2BNSUnRlClTNHDgQA0ePFiLFi1SUVGRpk6dKkmaPHmyOnbsqNTUVEnS9OnTdfHFF+upp57S6NGjtXr1an3++edavny5Z98JAADwS26HkfHjx+vQoUN6+OGHlZOTo/79+2v9+vUVg1T37t2roCBXg8sFF1ygVatW6aGHHtKDDz6o7t2764033lDv3r099y4AAIDfcnudETuwzggAAP7HK+uMAAAAeJpf7NrrbLxh8TMAAPyH83O7vk4YvwgjhYWFksTiZwAA+KHCwkLFxsbW+n2/GDNSXl6uAwcOKDo6Wg6Hw2OvW1BQoMTERGVnZwfsWJRA/x0E+vuX+B3w/gP7/Uv8Drz5/i3LUmFhoRISEqpMbqnOL1pGgoKC1KlTJ6+9fkxMTED+A6ws0H8Hgf7+JX4HvP/Afv8SvwNvvf+6WkScGMAKAABsRRgBAAC2CugwEh4errlz5wb0PjiB/jsI9Pcv8Tvg/Qf2+5f4HTSH9+8XA1gBAEDLFdAtIwAAwH6EEQAAYCvCCAAAsBVhBAAA2Cqgw8iSJUuUlJSkiIgIDRkyRFu2bLG7JJ9ITU3VoEGDFB0drXbt2mns2LHavn273WXZZv78+XI4HJoxY4bdpfjU/v37deONN6pt27aKjIxUnz599Pnnn9tdlk+UlZVpzpw56tKliyIjI9WtWzc9+uij9e6f4c82btyoMWPGKCEhQQ6HQ2+88UaV71uWpYcfflgdOnRQZGSkkpOTtXPnTnuK9ZK6fgelpaWaOXOm+vTpo6ioKCUkJGjy5Mk6cOCAfQV7WH3/Biq7/fbb5XA4tGjRIp/UFrBhZM2aNUpJSdHcuXOVkZGhfv36adSoUcrLy7O7NK/76KOPNG3aNH366afasGGDSktLNXLkSBUVFdldms9t3bpVf/rTn9S3b1+7S/GpI0eOaPjw4QoNDdU///lPffPNN3rqqafUunVru0vziQULFmjp0qVavHixtm3bpgULFuiJJ57Q888/b3dpXlNUVKR+/fppyZIlNX7/iSee0HPPPadly5bps88+U1RUlEaNGqWTJ0/6uFLvqet3cPz4cWVkZGjOnDnKyMjQa6+9pu3bt+uaa66xoVLvqO/fgNPrr7+uTz/9VAkJCT6qTJIVoAYPHmxNmzat4nFZWZmVkJBgpaam2liVPfLy8ixJ1kcffWR3KT5VWFhode/e3dqwYYN18cUXW9OnT7e7JJ+ZOXOmNWLECLvLsM3o0aOtm2++ucq1cePGWRMnTrSpIt+SZL3++usVj8vLy6327dtbTz75ZMW1o0ePWuHh4dbf/vY3Gyr0vuq/g5ps2bLFkmR9//33vinKh2p7//v27bM6duxoff3119aZZ55pPfPMMz6pJyBbRkpKSpSenq7k5OSKa0FBQUpOTtbmzZttrMwe+fn5kqQ2bdrYXIlvTZs2TaNHj67y7yBQvPXWWxo4cKCuu+46tWvXTuedd55WrFhhd1k+c8EFFygtLU07duyQJH3xxRf65JNPdOWVV9pcmT2ysrKUk5NT5b+F2NhYDRkyJCD/Jjrl5+fL4XDo9NNPt7sUnygvL9ekSZP0wAMP6Nxzz/Xpz/aLjfI87fDhwyorK1N8fHyV6/Hx8fr2229tqsoe5eXlmjFjhoYPH67evXvbXY7PrF69WhkZGdq6davdpdjiu+++09KlS5WSkqIHH3xQW7du1T333KOwsDBNmTLF7vK8btasWSooKFDPnj0VHByssrIyPf7445o4caLdpdkiJydHkmr8m+j8XqA5efKkZs6cqQkTJgTM5nkLFixQSEiI7rnnHp//7IAMI3CZNm2avv76a33yySd2l+Iz2dnZmj59ujZs2KCIiAi7y7FFeXm5Bg4cqHnz5kmSzjvvPH399ddatmxZQISRv//973r55Ze1atUqnXvuucrMzNSMGTOUkJAQEO8fdSstLdX1118vy7K0dOlSu8vxifT0dD377LPKyMiQw+Hw+c8PyG6auLg4BQcHKzc3t8r13NxctW/f3qaqfO+uu+7S22+/rQ8++ECdOnWyuxyfSU9PV15ens4//3yFhIQoJCREH330kZ577jmFhISorKzM7hK9rkOHDurVq1eVa+ecc4727t1rU0W+9cADD2jWrFm64YYb1KdPH02aNEn33nuvUlNT7S7NFs6/e4H+N1FyBZHvv/9eGzZsCJhWkY8//lh5eXnq3Llzxd/F77//Xvfdd5+SkpK8/vMDMoyEhYVpwIABSktLq7hWXl6utLQ0DRs2zMbKfMOyLN111116/fXX9f7776tLly52l+RTl112mb766itlZmZWHAMHDtTEiROVmZmp4OBgu0v0uuHDh58ynXvHjh0688wzbarIt44fP66goKp//oKDg1VeXm5TRfbq0qWL2rdvX+VvYkFBgT777LOA+Jvo5AwiO3fu1Hvvvae2bdvaXZLPTJo0SV9++WWVv4sJCQl64IEH9M4773j95wdsN01KSoqmTJmigQMHavDgwVq0aJGKioo0depUu0vzumnTpmnVqlV68803FR0dXdEnHBsbq8jISJur877o6OhTxsdERUWpbdu2ATNu5t5779UFF1ygefPm6frrr9eWLVu0fPlyLV++3O7SfGLMmDF6/PHH1blzZ5177rn697//raefflo333yz3aV5zbFjx7Rr166Kx1lZWcrMzFSbNm3UuXNnzZgxQ4899pi6d++uLl26aM6cOUpISNDYsWPtK9rD6voddOjQQddee60yMjL09ttvq6ysrOJvY5s2bRQWFmZX2R5T37+B6uErNDRU7du3V48ePbxfnE/m7DRTzz//vNW5c2crLCzMGjx4sPXpp5/aXZJPSKrxeOmll+wuzTaBNrXXsizr//7v/6zevXtb4eHhVs+ePa3ly5fbXZLPFBQUWNOnT7c6d+5sRUREWF27drV+97vfWcXFxXaX5jUffPBBjf/dT5kyxbIsM713zpw5Vnx8vBUeHm5ddtll1vbt2+0t2sPq+h1kZWXV+rfxgw8+sLt0j6jv30B1vpza67CsFrzkIAAAaPYCcswIAABoPggjAADAVoQRAABgK8IIAACwFWEEAADYijACAABsRRgBAAC2IowAAABbEUYAAICtCCMAAMBWhBEAAGArwggAALDV/wPjC/yB6geBRQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#instantiate the model\n",
    "model_mlp = MLP([D_in, 256, 128, 64, D_out]).to(device)\n",
    "\n",
    "#instantiate optimizer and criterion\n",
    "optimizer = optim.SGD(model_mlp.parameters(), lr=0.01, momentum=0.5)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "#train model and obtain loss and accuracy for translated validation dataset\n",
    "model_mlp, losses, accuracies = train_val_model(model_mlp, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=15, log_interval=2)\n",
    "\n",
    "# plot losses\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3dxwYacqdd-u"
   },
   "source": [
    "#### CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "OFq2OFmydd-u",
    "outputId": "ce6751bd-6403-4583-ba92-f5c2bd59c8ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/24\n",
      "----------\n",
      "train Loss: 0.4581 Acc: 85.48%\n",
      "val Loss: 0.2038 Acc: 94.53%\n",
      "\n",
      "Epoch 2/24\n",
      "----------\n",
      "train Loss: 0.1420 Acc: 95.77%\n",
      "val Loss: 0.0888 Acc: 97.39%\n",
      "\n",
      "Epoch 4/24\n",
      "----------\n",
      "train Loss: 0.1101 Acc: 96.67%\n",
      "val Loss: 0.0766 Acc: 97.92%\n",
      "\n",
      "Epoch 6/24\n",
      "----------\n",
      "train Loss: 0.0956 Acc: 97.13%\n",
      "val Loss: 0.0739 Acc: 97.80%\n",
      "\n",
      "Epoch 8/24\n",
      "----------\n",
      "train Loss: 0.0909 Acc: 97.26%\n",
      "val Loss: 0.0616 Acc: 98.15%\n",
      "\n",
      "Epoch 10/24\n",
      "----------\n",
      "train Loss: 0.0817 Acc: 97.58%\n",
      "val Loss: 0.0641 Acc: 98.03%\n",
      "\n",
      "Epoch 12/24\n",
      "----------\n",
      "train Loss: 0.0762 Acc: 97.69%\n",
      "val Loss: 0.0565 Acc: 98.12%\n",
      "\n",
      "Epoch 14/24\n",
      "----------\n",
      "train Loss: 0.0739 Acc: 97.73%\n",
      "val Loss: 0.0530 Acc: 98.42%\n",
      "\n",
      "Epoch 16/24\n",
      "----------\n",
      "train Loss: 0.0709 Acc: 97.87%\n",
      "val Loss: 0.0549 Acc: 98.26%\n",
      "\n",
      "Epoch 18/24\n",
      "----------\n",
      "train Loss: 0.0702 Acc: 97.86%\n",
      "val Loss: 0.0516 Acc: 98.37%\n",
      "\n",
      "Epoch 20/24\n",
      "----------\n",
      "train Loss: 0.0667 Acc: 97.95%\n",
      "val Loss: 0.0567 Acc: 98.26%\n",
      "\n",
      "Epoch 22/24\n",
      "----------\n",
      "train Loss: 0.0631 Acc: 98.05%\n",
      "val Loss: 0.0503 Acc: 98.41%\n",
      "\n",
      "Epoch 24/24\n",
      "----------\n",
      "train Loss: 0.0631 Acc: 98.10%\n",
      "val Loss: 0.0470 Acc: 98.60%\n",
      "\n",
      "Training complete in 9m 12s\n",
      "Best val Acc: 98.60%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGdCAYAAAA44ojeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA3QklEQVR4nO3deXhU9b3H8c8kkAVJAiGQEAibKCiyWAKRWhFrFNSquPSi1YropbWiVXOtWyu4x61KVSqttrXXlXoVq9biEoW6oChIcaUKIigmJCBJCJCEzLl/fB0mExLIZJYzmXm/nuc852TmzJlfjiPzyW/1OI7jCAAAIIYkuV0AAACAlggoAAAg5hBQAABAzCGgAACAmENAAQAAMYeAAgAAYg4BBQAAxBwCCgAAiDld3C5Ae3i9Xm3cuFEZGRnyeDxuFwcAALSD4ziqra1Vfn6+kpKCqxPpFAFl48aNKigocLsYAACgAzZs2KD+/fsH9ZpOEVAyMjIk2S+YmZnpcmkAAEB71NTUqKCgYPf3eDA6RUDxNetkZmYSUAAA6GQ60j2DTrIAACDmEFAAAEDMIaAAAICYQ0ABAAAxh4ACAABiDgEFAADEHAIKAACIOQQUAAAQcwgoAAAg5hBQAABAzCGgAACAmENAAQAAMSehA8o990i/+IX06adulwQAADSX0AHl8cel+fMJKAAAxJqEDih9+th+0yZ3ywEAAAIRUERAAQAg1hBQREABACDWEFBEQAEAINYQUERAAQAg1hBQREABACDWEFAkVVa6Ww4AABCIgCKpqkpqanK3LAAAwC+hA0qvXrb3eqUtW9wtCwAA8EvogNKliz+k0A8FAIDYkdABRaKjLAAAsYiAQkABACDmEFAIKAAAxBwCCgEFAICYQ0AhoAAAEHMIKAQUAABiDgGFgAIAQMwhoBBQAACIOQQUAgoAADGHgPJdQKmpkXbudLcsAADAJHxAycqSuna1Y1Y1BgAgNiR8QPF4aOYBACDWJHxAkQgoAADEGgKKCCgAAMQaAooIKAAAxBoCiggoAADEGgKKCCgAAMQaAooIKAAAxBoCiggoAADEGgKKCCgAAMQaAooCA4rjuFsWAABAQJEk9e5t+4YGW5MHAAC4i4AiKT1dysiwY9bjAQDAfQSU7/hqUeiHAgCA+wgo36GjLAAAsYOA8h0CCgAAsYOA8h0CCgAAsYOA8h0CCgAAsYOA8h0CCgAAsYOA8h0CCgAAsYOA8h0CCgAAsYOA8h0CCgAAsYOA8h1fQKmqkpqa3C0LAACJrkMBZd68eRo0aJDS0tJUVFSkZcuWtet1TzzxhDwej6ZOndqRt42oXr0kj8cWC9y82e3SAACQ2IIOKAsWLFBJSYnmzJmjFStWaPTo0Zo8ebI27aNtZN26dbr88st1xBFHdLiwkdSli4UUiWYeAADcFnRAueuuuzRz5kzNmDFDBx98sObPn69u3brpz3/+c5uvaWpq0llnnaXrr79eQ4YMCanAkUQ/FAAAYkNQAaWhoUHLly9XcXGx/wJJSSouLtbSpUvbfN0NN9ygPn366Pzzz2/X+9TX16umpiZgiwYCCgAAsSGogFJVVaWmpibl5uYGPJ6bm6vy8vJWX/PGG2/oT3/6kx544IF2v09paamysrJ2bwUFBcEUs8MIKAAAxIaIjuKpra3VT3/6Uz3wwAPKyclp9+uuvvpqVVdX7942bNgQwVL6EVAAAIgNXYI5OScnR8nJyaqoqAh4vKKiQnl5eXucv2bNGq1bt04nnnji7se8Xq+9cZcuWr16tfbff/89XpeamqrU1NRgihYWBBQAAGJDUDUoKSkpGjt2rMrKynY/5vV6VVZWpgkTJuxx/vDhw/XBBx9o5cqVu7eTTjpJRx11lFauXBm1ppv2IqAAABAbgqpBkaSSkhJNnz5dhYWFGj9+vObOnau6ujrNmDFDknTOOeeoX79+Ki0tVVpamg455JCA1/fo0UOS9ng8FhBQAACIDUEHlGnTpqmyslKzZ89WeXm5xowZo0WLFu3uOLt+/XolJXXOCWoJKAAAxAaP4ziO24XYl5qaGmVlZam6ulqZmZkRe5/PPpMOPFDKyJCiNLIZAIC4Fcr3d+es6ogQXw1Kba20Y4e7ZQEAIJERUJrJzJRSUuy4stLdsgAAkMgIKM14PPRDAQAgFhBQWvAFFGpQAABwDwGlBWpQAABwHwGlhd69bU9AAQDAPQSUFqhBAQDAfQSUFggoAAC4j4DSAgEFAAD3EVBaIKAAAOA+AkoLBBQAANxHQGmheUCJ/VWKAACITwSUFnzDjBsaWDAQAAC3EFBaSE+31YwlmnkAAHALAaUV9EMBAMBdBJRWEFAAAHAXAaUVBBQAANxFQGkFAQUAAHcRUFpBQAEAwF0ElFYQUAAAcBcBpRUEFAAA3EVAaQUBBQAAdxFQWkFAAQDAXQSUVvgCyubN0q5d7pYFAIBEREBpRa9eksdjiwVu3ux2aQAASDwElFYkJ0s5OXZMMw8AANFHQGkD/VAAAHAPAaUNBBQAANxDQGkDAQUAAPcQUNrgCyiVle6WAwCARERAaUPv3ranBgUAgOgjoLSBJh4AANxDQGkDAQUAAPcQUNpAQAEAwD0ElDYQUAAAcA8BpQ2+gFJbK+3Y4W5ZAABINASUNmRmSikpdsxQYwAAoouA0gaPh2YeAADcQkDZCwIKAADuIKDsBQEFAAB3EFD2goACAIA7CCh7QUABAMAdBJS9IKAAAOAOAspeEFAAAHAHAWUvCCgAALiDgLIXBBQAANxBQNmL5gHFcdwtCwAAiYSAshe9e9u+sVGqrna3LAAAJBICyl6kpdmaPBLNPAAARBMBZR/ohwIAQPQRUPaBgAIAQPQRUPaBgAIAQPQRUPaBgAIAQPQRUPaBgAIAQPQRUPaBgAIAQPQRUPbBNxdKZaW75QAAIJEQUPaBGhQAAKKPgLIPBBQAAKKPgLIPvoCyebO0a5e7ZQEAIFEQUPahVy/J47HFAjdvdrs0AAAkBgLKPiQnSzk5dkwzDwAA0UFAaQf6oQAAEF0ElHYgoAAAEF0ElHYgoAAAEF0ElHYgoAAAEF0ElHYgoAAAEF0ElHYgoAAAEF0ElHYgoAAAEF0ElHYgoAAAEF0ElHYgoAAAEF0ElHbwBZRt26Tt290tCwAAiaBDAWXevHkaNGiQ0tLSVFRUpGXLlrV57tNPP63CwkL16NFD++23n8aMGaOHH364wwV2Q0aGlJpqx5WV7pYFAIBEEHRAWbBggUpKSjRnzhytWLFCo0eP1uTJk7WpjfaP7Oxs/frXv9bSpUu1atUqzZgxQzNmzNCLL74YcuGjxeOhmQcAgGjyOI7jBPOCoqIijRs3Tvfdd58kyev1qqCgQBdffLGuuuqqdl3je9/7nk444QTdeOON7Tq/pqZGWVlZqq6uVmZmZjDFDZvCQmn5cun556UTTnClCAAAdCqhfH8HVYPS0NCg5cuXq7i42H+BpCQVFxdr6dKl+3y94zgqKyvT6tWrNXHixDbPq6+vV01NTcDmNmpQAACInqACSlVVlZqampSbmxvweG5ursrLy9t8XXV1tbp3766UlBSdcMIJuvfee3XMMce0eX5paamysrJ2bwUFBcEUMyIIKAAARE9URvFkZGRo5cqVevfdd3XzzTerpKREixcvbvP8q6++WtXV1bu3DRs2RKOYe0VAAQAgeroEc3JOTo6Sk5NVUVER8HhFRYXy8vLafF1SUpKGDh0qSRozZow++eQTlZaWatKkSa2en5qaqlTfsJkYQUABACB6gqpBSUlJ0dixY1VWVrb7Ma/Xq7KyMk2YMKHd1/F6vaqvrw/mrV1HQAEAIHqCqkGRpJKSEk2fPl2FhYUaP3685s6dq7q6Os2YMUOSdM4556hfv34qLS2VZP1JCgsLtf/++6u+vl4vvPCCHn74Yd1///3h/U0ijIACAED0BB1Qpk2bpsrKSs2ePVvl5eUaM2aMFi1atLvj7Pr165WU5K+Yqaur04UXXqivvvpK6enpGj58uB555BFNmzYtfL9FFPTubXsCCgAAkRf0PChuiIV5UDZskAYMkLp2lerrbfI2AADQtqjNg5LIfDUojY1SdbW7ZQEAIN4RUNopLU3yhT+aeQAAiCwCShDoKAsAQHQQUIJAQAEAIDoIKEEgoAAAEB0ElCAQUAAAiA4CShAIKAAARAcBJQgEFAAAooOAEgQCCgAA0UFACQIBBQCA6CCgBIGAAgBAdBBQguALKJs3S7t2uVsWAADiGQElCNnZkm+h5qoqd8sCAEA8I6AEITlZysmxY5p5AACIHAJKkOiHAgBA5BFQgkRAAQAg8ggoQSKgAAAQeQSUIBFQAACIPAJKkAgoAABEHgElSAQUAAAij4ASJAIKAACRR0AJEgEFAIDII6AEiYACAEDkEVCC5AsodXW2AQCA8COgBKl7dyk11Y4rK90tCwAA8YqAEiSPx1+LQkABACAyCCgdQD8UAAAii4DSAQQUAAAii4DSAQQUAAAii4DSAQQUAAAii4DSAQQUAAAii4DSAQQUAAAii4DSAQQUAAAii4DSAQQUAAAii4DSAc0DiuO4WxYAAOIRAaUDeve2/a5d0tatrhYFAIC4REDpgNRUKSvLjmnmAQAg/AgoHUQ/FAAAIoeA0kEEFAAAIoeA0kEEFAAAIoeA0kEEFAAAIoeA0kEEFAAAIoeA0kEEFAAAIoeA0kEEFAAAIoeA0kEEFAAAIoeA0kEEFAAAIoeA0kG+gLJli9TY6G5ZAACINwSUDsrOlpK+u3tVVe6WBQCAeENA6aCkJP+igTTzAAAQXgSUEBBQAACIDAJKCOgoCwBAZBBQQuALKJWV7pYDAIB4Q0AJATUoAABEBgElBAQUAAAig4ASAgIKAACRQUAJAQEFAIDIIKCEgIACAEBkEFBCQEABACAyCCgh8AWUujrbAABAeBBQQtC9u5SWZsfMhQIAQPgQUELg8dDMAwBAJBBQQkRAAQAg/AgoISKgAAAQfgSUEBFQAAAIPwJKiAgoAACEHwElRAQUAADCj4ASIgIKAADhR0AJEQEFAIDwI6CEiIACAED4EVBC5AsolZWS1+tuWQAAiBcdCijz5s3ToEGDlJaWpqKiIi1btqzNcx944AEdccQR6tmzp3r27Kni4uK9nt/Z9O5t+127pK1bXS0KAABxI+iAsmDBApWUlGjOnDlasWKFRo8ercmTJ2tTG20cixcv1plnnqnXXntNS5cuVUFBgY499lh9/fXXIRc+ZH/7mzRxovSHP3T4EikpUo8edkwzDwAA4RF0QLnrrrs0c+ZMzZgxQwcffLDmz5+vbt266c9//nOr5z/66KO68MILNWbMGA0fPlwPPvigvF6vysrKQi58yNatk15/XVq4MKTL0A8FAIDwCiqgNDQ0aPny5SouLvZfIClJxcXFWrp0abuusX37djU2Nio7O7vNc+rr61VTUxOwRcRJJ9n+1VelEN6DgAIAQHgFFVCqqqrU1NSk3NzcgMdzc3NVXl7ermtceeWVys/PDwg5LZWWliorK2v3VlBQEEwx22/YMOmAA6TGRumllzp8GV8/FAIKAADhEdVRPLfeequeeOIJLVy4UGlpaW2ed/XVV6u6unr3tmHDhsgUyOORTj7Zjp99tsOXoQYFAIDwCiqg5OTkKDk5WRUVFQGPV1RUKC8vb6+vvfPOO3XrrbfqpZde0qhRo/Z6bmpqqjIzMwO2iPE18/zjHzYUpwOaDzUGAAChCyqgpKSkaOzYsQEdXH0dXidMmNDm626//XbdeOONWrRokQoLCzte2kiYMEHq1UvaskV6880OXYIaFAAAwqtLsC8oKSnR9OnTVVhYqPHjx2vu3Lmqq6vTjBkzJEnnnHOO+vXrp9LSUknSbbfdptmzZ+uxxx7ToEGDdvdV6d69u7p37x7GX6WDunSRfvxjaeNGKTW1Q5cgoAAAEF5BB5Rp06apsrJSs2fPVnl5ucaMGaNFixbt7ji7fv16JSX5K2buv/9+NTQ06PTTTw+4zpw5c3TdddeFVvpwuf/+kF5OQAEAILw8juM4bhdiX2pqapSVlaXq6urI9kfpoI8/lkaMkLKzpc2b3S4NAACxIZTvb9biae6LL6T33gv6Zb4alC1bbMQyAAAIDQHF5/HHpSFDpF/+MuiXZmfblPeStGJFmMsFAEACIqD4TJxo+7fflloMo96XpCTpjDPs+LbbwlwuAAASEAHFp18/qbBQchzp+eeDfvlVV9m8bwsXWp8UAADQcQSU5nyTtnVgVtmDDpJOOcWOvxthDQAAOoiA0pwvoLz8srR9e9Avv+Ya2z/+uLR2bRjLBQBAgiGgNDdqlDRggLRjh9Rsttz2GjtWmjxZamqS7rgjAuUDACBBEFCa83j8tSjPPdehS/hqUf78Z5ucFgAABI+A0tLPfiY99ZR0110devnEidIPfiA1NHT4EgAAJDwCSksjR0qnniqFsE6QrxZl/nxmlgUAoCMIKBEwZYp06KFSXZ10zz1ulwYAgM6HgNKazZul666T/uu/OvRyj8dfi3LPPVJtbfiKBgBAIiCgtCYpSbr5ZunJJ6XPP+/QJU45RRo2TNq61Zp6AABA+xFQWtOzp3/q+w6O5klOttllJem3v7WRywAAoH0IKG05+WTb//3vHb7EWWfZtCoVFdJf/hKmcgEAkAAIKG058UTbv/FGh4fidO0qXXGFHd9+u9TYGKayAQAQ5wgobRk82IYcNzVJ//xnhy9z3nlSbq705ZfSY4+FsXwAAMQxAsrehLB4oE96ulRSYselpZZ3AADA3hFQ9ubkk6UePaTs7JAuc8EFdpnVq6WFC8NSMgAA4hoBZW/GjpU2bQp5nHBmpnTxxXZ8yy2S44ShbAAAxDECyt4kJVlP1zC45BJpv/2k99+XXnwxLJcEACBuEVDaw3GkDz4IqeqjVy/p5z+345tvDlO5AACIUwSUffF6pVGjbFu5MqRL/c//SCkpNnL59dfDUzwAAOIRAWVfkpKkAw6w4xBG80hSfr40Y4Yd33JLiOUCACCOEVDawzfcOIRZZX2uuMIyz6JF0vLlIV8OAIC4REBpjxNOsCWK339f2rAhpEsNGSKdeaYdl5aGoWwAAMQhAkp79O4tff/7dtzBxQObu/pq2z/9tPTJJyFfDgCAuENAaS/f4oEh9kORpBEjpKlTbVDQrbeGfDkAAOIOAaW9fP1QXn1VqqkJ+XLXXGP7Rx+V1q0L+XIAAMQVAkp7DRsmzZ4tPf+8LbATonHjpGOOsbV57rgjDOUDACCOeBwn9ider6mpUVZWlqqrq5WZmel2ccJm8WLpqKOk1FTpiy+kvn3dLhEAAOETyvc3NSguOvJIacIEqb5euvtut0sDAEDsIKAE6+23pcsvl5YtC/lSHo/061/b8f33S1u2hHxJAADiAgElWPPnS7/9rbRgQVgud/zxNov+tm3SvfeG5ZIAAHR6BJRgNZ9VNgzddzwe/4ie3/1Oqq0N+ZIAAHR6BJRgHXus9WpdsyZss6ydfrot9/Ptt9If/xiWSwIA0KkRUILVvbt09NF2HIZJ2yQpOVm66io7vvNOaefOsFwWAIBOi4DSEWFcPNDn7LOlggKpvFx66KGwXRYAgE6JgNIRP/qR7d95xxJFGKSkSL/6lR3fdpu0a1dYLgsAQKdEQOmIfv2kwkIpL0/6/POwXfb8821dwnXrpMcfD9tlAQDodAgoHfX889JXX0k/+EHYLtmtm3TZZXZ8zTXS2rVhuzQAAJ0KAaWjcnOlpPDfvlmzpAMPtOwzcaL06adhfwsAAGIeASVUTU1SdXXYLpeZKS1ZIo0YIX39tU2H/8EHYbs8AACdAgElFI89JuXnS//zP2G9bF6eLSQ4Zoy0aZM0aZK0fHlY3wIAgJhGQAlFnz6WIJ57TvJ6w3rpnBzp1VeloiJbo+eHP5TeeiusbwEAQMwioIRi4kRrk9m0KSyLB7bUs6f08sv2NjU1Nont4sVhfxsAAGIOASUUKSm22p8U1knbmsvIkP75T+mYY6S6Oum446QXX4zIWwEAEDMIKKHyzSobpmnvW9Otm13+Rz+yafBPOilieQgAgJhAQAnVlClSly7Sxx+HddK2ltLSpKeesoUFGxpsv2BBxN4OAABXEVBC1bOndRKRrLNsBKWk2AyzZ59tU+H/5CfSX/8a0bcEAMAVXdwuQFz47/+Wxo3zr3IcQV26WChJT5ceeEA691xpxw7pggsi/tYAAESNx3Ecx+1C7EtNTY2ysrJUXV2tzMxMt4sTExxHuvRS6Z577Oe77vJPkw8AQCwI5fubJp5wi1Le83ikuXOlK6+0n0tKpJtvjspbAwAQcQSUcHr3Xenww22u+ijweKTSUumGG+zn3/xG+vWvo5aRAACIGAJKOP3lL9LSpdIll9gaPVHg8UjXXivdcYf9fMstNvM+IQUA0JkRUMLphhukHj2kf/9b+tOfovrWl18u3XefHd99t3ThhWGffR8AgKghoIRTTo503XV2/JvfSFu3RvXtZ82yXOTxSPPnS+edF7WKHAAAwoqAEm4XXigNHy5VVko33hj1tz/vPOnRR6XkZBuOfNZZUmNj1IsBAEBICCjh1rWrDa+RbAzw6tVRL8KZZ0p/+5sVZcEC6eCDpd/9TqqujnpRAADoEAJKJEyeLJ1wgk33+vvfu1KEU0+VnnnGusR8/rnNmdKvnzUDffKJK0UCAKDdmKgtUv7zHxtufN551t7ikm3bpEceke6915YL8jnmGOnii20xZheLBwCIY6F8fxNQEoTjSK++akHl2Wf9w5CHDLFalfPOs9oWAADChZlkY92OHdI777haBI/Hlgp65hlpzRobltyzp7R2rc2b0q+frefz0UeuFhMAAEkElMhbu9ZG9UyZIlVVuV0aSdLgwTax21dfSX/8ozRypLR9u/SHP0iHHOIPMgxRBgC4hYASaQMHWlXF1q3S7NlulyZAt27SzJk2r9zixdaxNinJmoJOOUXaf3/p9tulLVvcLikAINEQUCItOdnG+EpWRbFqlbvlaYXHIx15pPTUU9IXX0hXXSX16iV9+aUtRtivnwWZlSuZQh8AEB10ko2WH/9Y+r//k446Siors1QQw3bskB5/3DrVrlzpf7xXL6mwMHDr1y/mfx0AgAsYxdMZrFtnfVHq66Wnn7Y2lE7AcaQ337Sg8swzUkPDnufk5e0ZWnJzo15UAECMIaB0FtdeK910k/VS/fhjKS3N7RIFpb5e+uAD6b33bHv3XRv101pn2oKCwMAydqzVvgAAEgcBpbPYtk0aNkwaP96Gz/Tu7XaJQrZ9u3Wy9YWW996zmWpb+1QNHiyNG2eB5fDDpQkTaBoCgHhGQOlMqqps1eM4Vlsrvf9+YGj57LM9zxsxwmazPftsab/9ol9OAEBkRX2itnnz5mnQoEFKS0tTUVGRli1b1ua5H330kU477TQNGjRIHo9Hc30L6SWqOA8nkpSRIU2cKJWUSI89ZrP+f/ut9Mor0q23SqedZoHko49scrj+/aVf/cq66QAAIHUgoCxYsEAlJSWaM2eOVqxYodGjR2vy5MnatGlTq+dv375dQ4YM0a233qq8vLyQCxw3Nm6Uzj3XOnIkgB49bAK4K6+0wUxffy3dfbdNtb91q3TnnTbvyimnSK+9xnBmAEh0QTfxFBUVady4cbrvvvskSV6vVwUFBbr44ot11VVX7fW1gwYN0qWXXqpLL700qELGVROPz8yZ0oMPWkeMN99M2M4YTU3SP/9po4Reesn/+CGH+Jt/unVzr3wAgI6LWhNPQ0ODli9fruLiYv8FkpJUXFyspUuXBvXGe1NfX6+ampqALe5cf721cyxdahOOJKjkZOlHP5JefNEGNl14od2WDz+Ufv5za/654gqafwAg0QQVUKqqqtTU1KTcFpNc5Obmqry8PGyFKi0tVVZW1u6toKAgbNeOGfn50jXX2PEVV0h1de6WJwYcdJA0b56tEeRr/vn2W1s3iOYfAEgsMTnV/dVXX63q6urd24YNG9wuUmSUlEiDBlmHjNtuc7s0MaNHD+nSS61z7XPPScccI3m9NlHcD38ojRolPfCADXEGAMSnoAJKTk6OkpOTVVFREfB4RUVFWDvApqamKjMzM2CLS2lp1jtUsmqCL790tzwxxtf889JLezb//Oxn/uafVatsEjkAQPwIKqCkpKRo7NixKisr2/2Y1+tVWVmZJkyYEPbCJYRTT5UmTZJ27pRuvNHt0sSs5s0/d90V2PwzerR1pB061ALN5ZdbDcvrr0uVlTQJAUBn1CXYF5SUlGj69OkqLCzU+PHjNXfuXNXV1WnGjBmSpHPOOUf9+vVTaWmpJOtY+/HHH+8+/vrrr7Vy5Up1795dQ4cODeOv0kl5PNLcudLDD9tU+NirHj2kyy6TfvlL6YUXpN//XnrrLammRlqzxrZ//CPwNT172jJIw4fbRL6+4yFDpK5dXfk1AAD70KGZZO+77z7dcccdKi8v15gxY3TPPfeoqKhIkjRp0iQNGjRIDz30kCRp3bp1Gjx48B7XOPLII7V48eJ2vV9cDjNG2DiOVF4urV4tffqpbb7jL79suwalSxfrfNs8uOTmSt2722Rzzff77ZewI8EBoMOY6j6eOI59qw4a5HZJ4sKOHTbNfsvgsnp1cAOnPB4LKS2DS1v7rCyb4mbkSIINgMQVyvd30E08iKBvvpH+679s+Mp//mPfcghJerqN+hk1KvBxx7HBUy1Dy5YttpbQtm3+vePYtm2bbcHo3186/njbjj7awgsAYN+oQYklDQ32Tbp6tXT66TYv/Nix/AnuIsex4czNA0t79uXl1kl3xw7/tVJSpCOPtLBywgnSAQe493sBQDTQxBNP/vlP+wbz6d9fmjrVtkmTbOwtOoUdO6QlS6wz7z/+Ia1dG/j80KH+sDJxoo06B4B4QkCJN//4h/TQQxZWfB0lcnLsz3JfQGlqIqx0Io5jrXa+sPKvf0mNjf7nu3WTiov9zUHxOHkygMRDQIlXO3dKr7xiU6hmZ0u3326Pe7027OSgg6xm5cQTpd693SwpglRbK5WVWVh54QVb3Lq5kSP9YeX737cRRwDQ2RBQEs2KFdY3xScpSTr8cFus5uSTbYIPdBqOI/373xZUXnjB1o/0ev3P77efDerq39+/FRQE/kx/agCxiICSaBzH5nt/5hnbVqwIfL60VLrqKjdKhjDYvNmm93/hBWvl27x536/JyAgMLK0FmR49/P2tHcf6ZNfVBbdt22b77dulfv2kyZOt4+9++0X0lgDopAgoie7LL6W//93Cyr/+ZW0HRx5pz731lvTHP0pFRbaNHMn0qZ1IU5PN4/LVV/5tw4bAn7dsad+1unWzmhZf2GhqCk8ZU1Ksk++UKRZYRoxg4BkAQ0CB3+bN9qeyrwPtxRdL993nfz49Xfre9/yB5bjj7M9vdFp1dTany95CTFVV26/v2tVqQNq7de9uH6MPP5QWLdpzjct+/fxhpbjYlhoAkJgIKGjbkiXW0fadd6Rly6Tq6sDn162TBg6047fesrGx48ZJ3Oe4smOHhZiaGv/U/b4tlAo13+ikRYtsW7zY+nb7JCVJhx1mYWXKFOs6xeAzIHEQUNA+Xq+1F7zzjm2ffSa9+KK/Pn7qVGsq8nhshFBRkTR+vL9piKEk2IcdO2yCukWL7KP13Tqhu/XqJR1zjIWVY4+V+vZ1p5wAooOAgvC46CLp+ef3rLOXbJjzpk38+YugrF9vQeXFF6WXX7YanOZGj7awMmKE1KePbbm5NmqerlJA50dAQXhVVPibhHz7ESOsCchnzhz7djnhBCk11b2yBsvrtTaIHTtsKEpurvXylKQvvrBepNnZ7pYxTjU22sfpxRethuW99/Z+fna2P7Dsa89q00BsIqAgsrxeGyqSk2M/b9hg/VYcx75FzjhDmj7d+q5E61uisVFavtz62Kxfb2Fj+3YLHvffbz01Jemuu6S77/YHkuaL40g2RPvQQ+24tFS69Vbp2mulX/7SH1wQEZWVVqtSVmYfqYoKq6SrrAx+hFF6uoWV3r3tP1tysn9LSgr8ub1bSoo1SfXqZR/9nBz/ca9e1PAA7UFAQXR98400d670yCOBU6AOGyadc45t/ftH7v1vv1268ca2lxZetcr6zEh23uzZrZ+XkmIB57DD7Oe775ZKSux46FDpt7+1WXr50zyqfHnYF1ha7ls+tn27O+XMzGw7vDTf5+TYSKbMTOugnJTkTnkBNxBQ4I6mJvvz93//V3r6aX/txF/+Ip17bmjX3rVLWrlSeu01Gxpyxx3SwQfbcw8+KM2cabU3Rx4pHXKI1fF362Z/Sk+d6q/t+eor+xZLT7fnfVt6+p79abxe6a9/la65xtY9kmyc7F13+QMPYk5dnT+wVFVZ5VpTU+ub19v2cy23nTstKFVV2eh9337zZqs87AiPx0b1Z2a2vWVl7f35vDwq99B5EFDgvpoa6amnpCeekP7v//xzq8yfL735pjUBHXVU251sm5oskCxebKHk9dcDe1T+7nfW7CLZN8RXX1loiMSfo7W10i23WDBpaLD3uO026fLLw/9e6HSamqStWwNDS2v75sfffmuZOxxSUqxL2Jgxth16qHUHi/Y/jVVV0ief2LZxo1U6jh4tDR9O8xf8CCiIXYceasFDsmafs8+2JqBhw6xuvnt3e+7ll23caXNZWTZF6VFHST/6kXTAAVEtutaula64woLX4sX+2XmBIDmO1cjU1LS+VVe37/Hq6sBVsJvbf//A0DJmjJSfH1oLpddrXbx8QeTTT/3HbS3BkJJilZ2jRllg8W2+Sk0kFgIKYpPj2LCNv/7Vala2bvU/t99+0gUXSHfeaT/X1VnH28MOs0AyaZL9CxsLw5o//NCakXweeMD+5T/+ePqnIKocx+ZWXLlSev99269caZ2MW9O7956h5cAD9/zfqr7epkVqHkA++URavXrPfuXNDRxoUybl59uEfatW7TmU3Cc/PzCwjB7delkQXwgoiH319TbHyl//aivg7dol/eAH1pTj4/XGfg/Cr7+2f1W3b7fpUe+6y983BnBJVZWtiO0LLO+/b2GjtdFQ6elWuzFihL+ZZu3atkdOde1qH/mDDrLmm4MOsm3YMOvO1ZwvQP3734Hb2rWtXzstzbJ/89AyZIjVEtXXW63T3vb7Oqepyf5Jac/m8ez9+bw8uw8HHGCDBGP9n6pYQUBB51JZaSOBDj64881OW1Mj3XSTjWJqbLQ//y68ULruOuZPQUzZsUP66KPAmpZ//9sqK1uTmblnCDnoIGnw4ND/N62pkT74wB9YVq2yza0RWKFKT7egcuCB/tDiO+7Vi4rV5ggoQLR9/rn0q1/ZCtKSjSO9/nprtorFHoLbtkkvvWRtAQUF0oQJzDOfgLxe++iuXGnLEPTu7Q8kfftG94vV65XWrNmztuWrr6xmJTU19H1ystXqeL2hbbt2Wbk++8xqg/bW4blnzz1Di+9nX5e7REJAAdzy6qvSpZfan4cpKVZfPmSI26Uy5eXSc8/Z+kqvvGL13j5PPimdfrodv/KKzS0zYIB1KhgwwL8VFDCmFWimsdGasf7zHwss//mPf2urL5BP37422qlHDwtRzQNVa9u+nuva1QYatrfJa1/7uXPDPxYhlO/vTla/DsSYH/7QZqN98EEbS+oLJ9u3W5+bwkKrI492ne9zz0knnxw4YYdvmIdvTKjPhx/aKKrWeDxWS3TSSf5zFy+2IDN8uF2H+mwkkK5d7Uu8tS/y7dutVqh5aPGFGF/L9jffRL/M7fWb30R/sOTeEFCAUHXpYk07zS1fLk2bZse9ellQGTfOv8/PD897e73Su+9aiBgxwoZxSzYaKilJ+t73LKhMnWp9floLEyecYPXS69fbQpHN9zt3Wu9An1dekS67zP9zjx72+/i2o46y4eGIPY5jnVJeecUC6Xvv2dCeOXOsyQ8h69bNpmdqbV7Hb7+1sLJmjfUDal5z0XJr7fHWHmtstJoU37avJq99nRMrlb8+NPEAkVBWJl11lTWqtzZxxR/+IP3sZ3ZcW2v1tL16te/a9fXWtPTMM1ZT4vuTbOJEm7rfp7LSOhl0lOPYMI+sLH8zzzPPSA8/bHXcH39s/0o299Zb/i+7Dz6wKV4LCy3IwB0ffmgTDb7yin+G5OaefdaWdAAigCYeINYcfbTVbNTX2xf1u+/aX6zvvmt/xTb/E+upp6QZM6wpyFcTUVgojR3rn5FXssBw7rm2rEDzdYgyMmxOllNPDSxDKOFEstqWlteYOtU2yYLXhx/a77RsmdUajRnjP/cPf5DmzbPjYcPs9xo/3vZjxtifbAiv2lprgsvLs/ssWfh95BE7TkuzIHvMMfb8kiX22fG56SYL1b/4hdWG0XwHF1GDAkRbXZ3VqfrGbs6ZI91ww57neTzWz+Pxx22CCMn6gjz3nDURnXyybZMm2fVizbXXSo8+Kn3xxZ7Pde1qtSs9e9rPy5bZffH1AExPDzym2ah1jY12715+2WpI3nnHhpice66tiSVZM+D119tMyN//ftvBcNcu6xjtq5EbNsyaLqdP9/93iqTVq20SxM8/908+0nxJ6h//2D7vkg2pueWWtpetnjTJQphkwX77dpscElHHKB6gs9u61Wog3n3XX9uyfr09t2mTvybjvffsH9yxYzvPTFFVVf7fa9ky29LTrZ+Lz+GHW/NQazIyAqcnPf10W9+pZYhJS7NOAE8+6f8SXrvWgmB+fuebc2dvGhul006zdataruo9dKh05pmth959WbVKuv9+q3HxXTc9XTrjDGnWLPvchcpxrCPGG2/Y+OaiInv89detdqctN90k/frXdvz++9a/qi3XXCPdfLMdf/21Ba9hw+w1vm3MGJoeo4CAAsSjigr7wvD9JRgvHMeWCW7e5+bss+1Lp3kPwB07bN+7t90LnyOPlP71r9av3b27hRlf04Svxik52ab/bD6UeuBA6b//O7bmWt+2zWoHWm4bNlg5n33Wf+7IkdbE1quXrbrt2wYNCr0ctbUWUu6/35ooJennP7fFP4O1a5c1G73xhn/z9YWZNUu67z473rnTOmCPGmXhu+US1Icf7g8zGzdaE2LL5al9Px97rHX+lmzm6ubNWM0NGWI1fb7V1x2HZq0wI6AAiE+OY7UFzedi+fJLq3HyBZjmYSYpyT96SrIvqZdfbr2jcsswc8459mXsCy/5+f6amYwM6Sc/8b/WNw1qW5NVpKfv+X7V1XsGj4YG/1/6kvU9Wr689XvRrZsFGF95X33Vml5Gj45cbZrjWM3W/Pm2mrevqfG996yz9AUXWC1Iy9f4yrhli93LlrU8KSnWB+aMM6SLLopM2Zv75hsLwCtW2Pb++9bRW7Lfwzf6bckSOz700MDaln79CC4dREABgLY0NVkNTMsh1F6v9Pvf+88bPdqCR2tycmxUlM9RR1ln1NakpQWusPfTn9pkebW1e57bMnRMnWrNNv372yR5/fsHbsceGxtNe+eea+tqSdbfY9o0G0P7xhsW7BYu9J87eLCNsT38cFt/6wc/sHDidifpLVssqIwcKfXpY4/dfbdUUrLnub172+8xd65/lNrq1dZs2aePPe/bM7FhAEbxAEBbkpPtSzM/f+/zfTzxhPVZ8YWY8nL/5BMt5yjPy7MvrOaTVOzYYbUHLb94y8v94aRnT3/Y8AWQXbv8yyP87W+d4wvupz+1GqFnn7Wg1jys9egRuPDn66/bvY+FYNVcdraNtmtu5kwbaearaVmxwkbdVVba5vX6z33pJemXv9zzullZFlYefNDfp2bVKqud6d3bNt88/L7tgANsMSTJ7mtlZeDzvq1LF/ssxuJyGhFADQoAhIPjWNiorw8MNGvW2OP9+8ffSJING2zkzeLFNhHgEUdYDcmAAfHTJLJzp/X1+fprCxy+EU1PPmn9YCorrSN7ZWXgktBvvmmjpiTpnnukSy5p+z1efNFqxyTpj3+0/j5tWbjQP9R/0SKbT2nwYOtP03Kfl+f6fwdqUADAbR6P/WXb8q/b/fd3pzzRUFDQsdFCnUlamvUNKiwMfPzHP7bNx+u1vlG+sNJ8rqPBg+3cTZtsVFtDQ2Dn3m7d/Od26WJ9npo/39TkX7aieafuzz+3kLhhQ+sdxx95RDrrLDv+6CMLNM0DTIz/wU8NCgAAsc5xLKj45oiRLBB9+qnNNbR2re19xxs2WPOar1lz3rw9OyRnZ/sDy/XX79nhOQzoJAsAAPwaG/2T10nSCy9I//u//gBTVRV4/qpVrS8iFCKaeAAAgF/Lpsbjjw+cD6a2NrDGJQabIgkoAAAkmowMmxRv1Ci3S9KmGBv3BQAAQEABAAAxiIACAABiDgEFAADEHAIKAACIOQQUAAAQcwgoAAAg5hBQAABAzCGgAACAmENAAQAAMYeAAgAAYg4BBQAAxBwCCgAAiDmdYjVjx3EkSTU1NS6XBAAAtJfve9v3PR6MThFQamtrJUkFBQUulwQAAASrtrZWWVlZQb3G43Qk1kSZ1+vVxo0blZGRIY/HE7br1tTUqKCgQBs2bFBmZmbYrou94767g/vuDu67O7jv7mh53x3HUW1trfLz85WUFFyvkk5Rg5KUlKT+/ftH7PqZmZl8gF3AfXcH990d3Hd3cN/d0fy+B1tz4kMnWQAAEHMIKAAAIOYkdEBJTU3VnDlzlJqa6nZREgr33R3cd3dw393BfXdHOO97p+gkCwAAEktC16AAAIDYREABAAAxh4ACAABiDgEFAADEnIQOKPPmzdOgQYOUlpamoqIiLVu2zO0ixbXrrrtOHo8nYBs+fLjbxYo7//rXv3TiiScqPz9fHo9HzzzzTMDzjuNo9uzZ6tu3r9LT01VcXKzPPvvMncLGkX3d93PPPXePz/+UKVPcKWycKC0t1bhx45SRkaE+ffpo6tSpWr16dcA5O3fu1KxZs9SrVy91795dp512mioqKlwqcXxoz32fNGnSHp/3Cy64IKj3SdiAsmDBApWUlGjOnDlasWKFRo8ercmTJ2vTpk1uFy2ujRgxQt98883u7Y033nC7SHGnrq5Oo0eP1rx581p9/vbbb9c999yj+fPn65133tF+++2nyZMna+fOnVEuaXzZ132XpClTpgR8/h9//PEoljD+LFmyRLNmzdLbb7+tl19+WY2NjTr22GNVV1e3+5zLLrtMzz33nJ588kktWbJEGzdu1KmnnupiqTu/9tx3SZo5c2bA5/32228P7o2cBDV+/Hhn1qxZu39uampy8vPzndLSUhdLFd/mzJnjjB492u1iJBRJzsKFC3f/7PV6nby8POeOO+7Y/djWrVud1NRU5/HHH3ehhPGp5X13HMeZPn26c/LJJ7tSnkSxadMmR5KzZMkSx3Hss921a1fnySef3H3OJ5984khyli5d6lYx407L++44jnPkkUc6l1xySUjXTcgalIaGBi1fvlzFxcW7H0tKSlJxcbGWLl3qYsni32effab8/HwNGTJEZ511ltavX+92kRLKF198ofLy8oDPflZWloqKivjsR8HixYvVp08fDRs2TL/4xS+0efNmt4sUV6qrqyVJ2dnZkqTly5ersbEx4PM+fPhwDRgwgM97GLW87z6PPvqocnJydMghh+jqq6/W9u3bg7pup1gsMNyqqqrU1NSk3NzcgMdzc3P16aefulSq+FdUVKSHHnpIw4YN0zfffKPrr79eRxxxhD788ENlZGS4XbyEUF5eLkmtfvZ9zyEypkyZolNPPVWDBw/WmjVrdM011+i4447T0qVLlZyc7HbxOj2v16tLL71Uhx9+uA455BBJ9nlPSUlRjx49As7l8x4+rd13SfrJT36igQMHKj8/X6tWrdKVV16p1atX6+mnn273tRMyoMAdxx133O7jUaNGqaioSAMHDtTf/vY3nX/++S6WDIi8M844Y/fxyJEjNWrUKO2///5avHixjj76aBdLFh9mzZqlDz/8kH5tUdbWff/Zz362+3jkyJHq27evjj76aK1Zs0b7779/u66dkE08OTk5Sk5O3qMnd0VFhfLy8lwqVeLp0aOHDjzwQH3++eduFyVh+D7ffPbdN2TIEOXk5PD5D4OLLrpIzz//vF577TX1799/9+N5eXlqaGjQ1q1bA87n8x4ebd331hQVFUlSUJ/3hAwoKSkpGjt2rMrKynY/5vV6VVZWpgkTJrhYssSybds2rVmzRn379nW7KAlj8ODBysvLC/js19TU6J133uGzH2VfffWVNm/ezOc/BI7j6KKLLtLChQv16quvavDgwQHPjx07Vl27dg34vK9evVrr16/n8x6Cfd331qxcuVKSgvq8J2wTT0lJiaZPn67CwkKNHz9ec+fOVV1dnWbMmOF20eLW5ZdfrhNPPFEDBw7Uxo0bNWfOHCUnJ+vMM890u2hxZdu2bQF/pXzxxRdauXKlsrOzNWDAAF166aW66aabdMABB2jw4MG69tprlZ+fr6lTp7pX6Diwt/uenZ2t66+/Xqeddpry8vK0Zs0aXXHFFRo6dKgmT57sYqk7t1mzZumxxx7T3//+d2VkZOzuV5KVlaX09HRlZWXp/PPPV0lJibKzs5WZmamLL75YEyZM0GGHHeZy6Tuvfd33NWvW6LHHHtPxxx+vXr16adWqVbrssss0ceJEjRo1qv1vFNIYoE7u3nvvdQYMGOCkpKQ448ePd95++223ixTXpk2b5vTt29dJSUlx+vXr50ybNs35/PPP3S5W3HnttdccSXts06dPdxzHhhpfe+21Tm5urpOamuocffTRzurVq90tdBzY233fvn27c+yxxzq9e/d2unbt6gwcONCZOXOmU15e7naxO7XW7rck5y9/+cvuc3bs2OFceOGFTs+ePZ1u3bo5p5xyivPNN9+4V+g4sK/7vn79emfixIlOdna2k5qa6gwdOtT51a9+5VRXVwf1Pp7v3gwAACBmJGQfFAAAENsIKAAAIOYQUAAAQMwhoAAAgJhDQAEAADGHgAIAAGIOAQUAAMQcAgoAAIg5BBQAABBzCCgAACDmEFAAAEDMIaAAAICY8/957uHDvW3OPQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# instantiate model, optimizer and criterion\n",
    "model = CNN().to(device)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "# train model and obtain loss and accuracy for the translated validation dataset\n",
    "model, losses, accuracies = train_val_model(model, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=25, log_interval=2)\n",
    "\n",
    "# plot losses\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-5gcf_gMlcqI"
   },
   "source": [
    "After rerunning the different models we can see that the accuracy of the Logistic Regression drops from ~90% to ~72%, the MLP drops from ~98% to ~87%, and the CNN drops from ~99% to ~97%. This shows that the learned features are more robust to variances in location, as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nU3NwQ7Nuvhv"
   },
   "source": [
    "# Bonus Case - Attention with small images and CNNs.  (And how to create a dataset that takes numpy arrays)\n",
    "\n",
    "In this case we will use the Scikit-Learn's digits dataset\n",
    "\n",
    "## Scikit-Learn Digits\n",
    "\n",
    "This dataset is provided by scikit-learn and the digit images are returned as numpy ndarray. We will use PIL (Python Image Library) to convert the numpy ndarray to a image, tranform it to a tensor and normalize it.\n",
    "\n",
    "In this case we don't have a predefined Digits Dataset provided by torchvision so we will need to write a custom Dataset class and implement three functions:\n",
    "\n",
    "`__init__`, `__len__`, and `__getitem__`.\n",
    "\n",
    "Scikit-Learn return the digits images and labels as ndarrays. Each digit image is an 8x8 array.\n",
    "\n",
    "To use the previous CNN, we will use a transform to resize the images to the MNIST image size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A4v-XFzcv9If"
   },
   "outputs": [],
   "source": [
    "SKLEARN_DIGITS_TRAIN_SIZE = 1247\n",
    "SKLEARN_DIGITS_VAL_SIZE = 550\n",
    "\n",
    "class NumpyDataset(Dataset):\n",
    "\n",
    "  def __init__(self, data, targets, transform=None):\n",
    "    self.data = torch.from_numpy(data).float()\n",
    "    self.targets = torch.from_numpy(targets).long()\n",
    "    self.transform = transform\n",
    "\n",
    "  def __getitem__(self, index):\n",
    "    x = np.expand_dims(self.data[index], axis=2)\n",
    "    y = self.targets[index]\n",
    "    if self.transform:\n",
    "        x = self.transform(x)\n",
    "    return x, y\n",
    "\n",
    "  def __len__(self):\n",
    "    return len(self.data)\n",
    "\n",
    "digits_transform = transforms.Compose([\n",
    "                                       transforms.ToPILImage(),\n",
    "                                       transforms.Resize(28),\n",
    "                                       transforms.ToTensor(),\n",
    "                                       ])\n",
    "\n",
    "# Get sklearn digits dataset\n",
    "X, y = load_digits(return_X_y=True)\n",
    "X = X.reshape((len(X), 8, 8))\n",
    "y_train = y[:-SKLEARN_DIGITS_VAL_SIZE]\n",
    "y_val = y[-SKLEARN_DIGITS_VAL_SIZE:]\n",
    "X_train = X[:-SKLEARN_DIGITS_VAL_SIZE]\n",
    "X_val = X[-SKLEARN_DIGITS_VAL_SIZE:]\n",
    "\n",
    "digits_train_dataset = NumpyDataset(X_train, y_train, transform=digits_transform)\n",
    "digits_val_dataset = NumpyDataset(X_val, y_val, transform=digits_transform)\n",
    "digits_train_dataloader = torch.utils.data.DataLoader(digits_train_dataset, batch_size=64, shuffle=True)\n",
    "digits_val_dataloader = torch.utils.data.DataLoader(digits_val_dataset, batch_size=64, shuffle=True)\n",
    "\n",
    "dataloaders = dict(train=digits_train_dataloader, val=digits_val_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dhQU3v7Zv9Ih"
   },
   "outputs": [],
   "source": [
    "# Get some examples of images and targets\n",
    "_, (example_train_imgs, example_train_targets) = next(enumerate(digits_train_dataloader))\n",
    "_, (example_val_imgs, example_val_targets) = next(enumerate(digits_val_dataloader))\n",
    "\n",
    "# Info about the dataset\n",
    "D_in = np.prod(example_imgs.shape[1:])\n",
    "D_out = len(digits_train_dataloader.dataset.targets.unique())\n",
    "\n",
    "# Output information\n",
    "print(\"Datasets shapes (before transformations):\", {x: dataloaders[x].dataset.data.shape for x in ['train', 'val']})\n",
    "print(\"N input features:\", D_in, \"Output classes:\", D_out)\n",
    "print(\"Train batch:\", example_train_imgs.shape, example_train_targets.shape)\n",
    "print(\"Val batch:\", example_val_imgs.shape, example_val_targets.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Vx78pb7Ov9Ih"
   },
   "outputs": [],
   "source": [
    "plot_img_label_prediction(imgs=example_train_imgs, y_true=example_train_targets, y_pred=None, shape=(2, 3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xbBAH9OTv9Ii"
   },
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "W46ofUE6v9Ii"
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "print(X_train.squeeze().shape)\n",
    "X_train = scaler.fit_transform(np.reshape(X_train, (X_train.shape[0], -1)))\n",
    "X_val = scaler.transform(np.reshape(X_val, (X_val.shape[0], -1)))\n",
    "\n",
    "# Turn up tolerance for faster convergence\n",
    "clf = LogisticRegression(C=50., multi_class='multinomial', solver='sag', tol=0.1)\n",
    "clf.fit(X_train, y_train)\n",
    "#sparsity = np.mean(clf.coef_ == 0) * 100\n",
    "score = clf.score(X_val, y_val)\n",
    "\n",
    "print(\"Test score with penalty: %.4f\" % score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "E_G2Rt0Gv9Ij"
   },
   "outputs": [],
   "source": [
    "coef = clf.coef_.copy()\n",
    "plt.figure(figsize=(10, 5))\n",
    "scale = np.abs(coef).max()\n",
    "for i in range(10):\n",
    "    l1_plot = plt.subplot(2, 5, i + 1)\n",
    "    l1_plot.imshow(coef[i].reshape(8, 8), interpolation='nearest',\n",
    "                   cmap=plt.cm.RdBu, vmin=-scale, vmax=scale)\n",
    "    l1_plot.set_xticks(())\n",
    "    l1_plot.set_yticks(())\n",
    "    l1_plot.set_xlabel('Class %i' % i)\n",
    "plt.suptitle('Classification coefficient vectors for...')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PfYdXpde4bg0"
   },
   "source": [
    "### Feed-forward using digits dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UDKn4WS636Bg"
   },
   "outputs": [],
   "source": [
    "model = MLP([D_in, 512, 256, 128, 64, D_out]).to(device)\n",
    "\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "model, losses, accuracies = train_val_model(model, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=20, log_interval=2)\n",
    "\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "R9ImEBeo4MW6"
   },
   "outputs": [],
   "source": [
    "model = CNN().to(device)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "model, losses, accuracies = train_val_model(model, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=50, log_interval=10)\n",
    "\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uIhei09Ruvf-"
   },
   "source": [
    "# Bonus Information - Visualizing CNN filters\n",
    "\n",
    "Some work have been done to demonstrate the type of features learned by different filters in different layers.\n",
    "\n",
    "For instance, considering a known CNN called **VGG16** which has the following architecture\n",
    "\n",
    "![image](https://media.geeksforgeeks.org/wp-content/uploads/20200219152327/conv-layers-vgg16.jpg)\\[taken from: https://www.geeksforgeeks.org/vgg-16-cnn-model/ \\]\n",
    "\n",
    "these would be some of the filters from some of the layers:\n",
    "<table border=0 width=\"500px\" >\n",
    "\t<tbody>\n",
    "\t\t<tr>\n",
    "\t\t\t<td width=\"19%\" align=\"center\"> Layer 2 <br /> (Conv 1-2)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l2_f1.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l2_f21.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l2_f54.jpg\"> </td>\n",
    "\t\t</tr>\n",
    "\t\t<tr>\n",
    "\t\t\t<td width=\"19%\" align=\"center\"> Layer 10 <br /> (Conv 2-1)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l10_f7.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l10_f10.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l10_f69.jpg\"> </td>\n",
    "\t\t</tr>\n",
    "\t\t<tr>\n",
    "\t\t\t<td width=\"19%\" align=\"center\"> Layer 17 <br /> (Conv 3-1)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l17_f4.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l17_f8.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l17_f9.jpg\"> </td>\n",
    "\t\t</tr>\n",
    "\t\t<tr>\n",
    "\t\t\t<td width=\"19%\" align=\"center\"> Layer 24 <br /> (Conv 4-1)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l24_f4.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l24_f17.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l24_f22.jpg\"> </td>\n",
    "\t\t</tr>\n",
    "\t</tbody>\n",
    "</table>\n",
    "\n",
    "or obtain the class activations:\n",
    "<table border=0 width=\"500px\" >\n",
    "\t<tbody>\n",
    "    <tr>\t\t<td width=\"27%\" align=\"center\"> Input Image </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> Layer Vis. (Filter=0)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> Filter Vis. (Layer=29)</td>\n",
    "\t\t</tr>\n",
    "<tr>\n",
    "\t\t\t<td width=\"27%\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/input_images/spider.png\"> </td>\n",
    "\t\t\t<td width=\"27%\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/cnn-gifs/master/spider_layer_graph.gif\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/cnn-gifs/master/spider_filter_graph.gif\"> </td>\n",
    "\t\t</tr>\n",
    "\t</tbody>\n",
    "</table>\n",
    "\n",
    "\\[examples taken from: http://www.github.com/utkuozbulak/pytorch-cnn-visualizations \\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VU21L86BvAXK"
   },
   "source": [
    "# Bonus Information - Predefined architectures, pre-trained models and transfer learning\n",
    "\n",
    "Packages like [torchvision](https://pytorch.org/vision/stable/index.html) and [timm](https://rwightman.github.io/pytorch-image-models/) offer you the possibility of using predefined architectures or even use pre-trained models that can be used to fine tune the models for that same task or used for transfer learning.\n",
    "\n",
    "Besides datasets, transforms and others, **Torchvision** has a large number of predefined architecture with the possibility of loading the pre-trained weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w_1YpmkV-PbU"
   },
   "source": [
    "#### Torchvision classification models examples\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l5yOWA9l4lrF"
   },
   "outputs": [],
   "source": [
    "import torchvision.models as models\n",
    "\n",
    "# construct a model with random weights to be trained\n",
    "resnet18 = models.resnet18()\n",
    "\n",
    "# load a pre-trained model\n",
    "resnet18 = models.resnet18(pretrained=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nAsfqUIyDBD7"
   },
   "source": [
    "For examples of different models and how to use pre-trained weights please visit https://pytorch.org/vision/stable/models.html#\n",
    "\n",
    "\n",
    "\n",
    "Another possibility is **timm** which contains models for classification only.\n",
    "In **timm** you are not restricted to have inputs only with 1/3-channels, allowing you to use architectures or pre-trained models using images that have 2 or > 3-channels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4zS8Ykbo-ZUg"
   },
   "source": [
    "#### timm classification models examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "w0FQYnKQ-lr_"
   },
   "outputs": [],
   "source": [
    "if 'google.colab' in str(get_ipython()):\n",
    "  !pip install -q timm\n",
    "import timm\n",
    "\n",
    "# list all models\n",
    "print(timm.list_models())\n",
    "\n",
    "# list pre-trained models\n",
    "print(timm.list_models(pretrained=True))\n",
    "\n",
    "# list models architectures by wildcards\n",
    "print(timm.list_models('*resne*t*'))\n",
    "\n",
    "# construct a model with random weights to be trained\n",
    "model = timm.create_model('resnet18')\n",
    "\n",
    "# load a pre-trained model\n",
    "model = timm.create_model('resnet18', pretrained=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OSEwqB8ADIao"
   },
   "source": [
    "For more details on how to use this package visit https://rwightman.github.io/pytorch-image-models/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kaank-9kDI72"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}