1. ANN/1.1.2 MLP and MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### (exceprt from Python Machine Learning Essentials, Supplementary Materials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Classifying handwritten digits](#Classifying-handwritten-digits)\n",
    "    - [Obtaining the MNIST dataset](#Obtaining-the-MNIST-dataset)\n",
    "    - [Implementing a multi-layer perceptron](#Implementing-a-multi-layer-perceptron)\n",
    "- [Training an artificial neural network](#Training-an-artificial-neural-network)\n",
    "- [Debugging neural networks with gradient checking](#Debugging-neural-networks-with-gradient-checking)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifying handwritten digits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtaining the MNIST dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST dataset is publicly available at http://yann.lecun.com/exdb/mnist/ and consists of the following four parts:\n",
    "\n",
    "- Training set images: train-images-idx3-ubyte.gz (9.9 MB, 47 MB unzipped, 60,000 samples)\n",
    "- Training set labels: train-labels-idx1-ubyte.gz (29 KB, 60 KB unzipped, 60,000 labels)\n",
    "- Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, 7.8 MB, 10,000 samples)\n",
    "- Test set labels: t10k-labels-idx1-ubyte.gz (5 KB, 10 KB unzipped, 10,000 labels)\n",
    "\n",
    "In this section, we will only be working with a subset of MNIST, thus, we only need to download the training set images and training set labels. After downloading the files, I recommend unzipping the files using the Unix/Linux gzip tool from the terminal for efficiency, e.g., using the command \n",
    "\n",
    "    gzip *ubyte.gz -d\n",
    " \n",
    "in your local MNIST download directory, or, using your favorite unzipping tool if you are working with a machine running on Microsoft Windows. The images are stored in byte form, and using the following function, we will read them into NumPy arrays that we will use to train our MLP.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get MNIST Dataset\n",
    "\n",
    "**Note**: The following commands will work on Linux/Unix (e.g. Mac OSX) Platforms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!mkdir -p ../data/mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz --output ../data/mnist/train-images-idx3-ubyte.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!curl http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz --output ../data/mnist/train-labels-idx1-ubyte.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!curl http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz --output ../data/mnist/t10k-images-idx3-ubyte.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!curl http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz --output ../data/mnist/t10k-labels-idx1-ubyte.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load MNIST Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import struct\n",
    "import numpy as np\n",
    " \n",
    "def load_mnist(path, kind='train'):\n",
    "    \"\"\"Load MNIST data from `path`\"\"\"\n",
    "    labels_path = os.path.join(path, \n",
    "                               '%s-labels-idx1-ubyte' \n",
    "                                % kind)\n",
    "    images_path = os.path.join(path, \n",
    "                               '%s-images-idx3-ubyte' \n",
    "                               % kind)\n",
    "        \n",
    "    with open(labels_path, 'rb') as lbpath:\n",
    "        magic, n = struct.unpack('>II', \n",
    "                                 lbpath.read(8))\n",
    "        labels = np.fromfile(lbpath, \n",
    "                             dtype=np.uint8)\n",
    "\n",
    "    with open(images_path, 'rb') as imgpath:\n",
    "        magic, num, rows, cols = struct.unpack(\">IIII\", \n",
    "                                               imgpath.read(16))\n",
    "        images = np.fromfile(imgpath, \n",
    "                             dtype=np.uint8).reshape(len(labels), 784)\n",
    " \n",
    "    return images, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows: 60000, columns: 784\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train = load_mnist('data/mnist', kind='train')\n",
    "print('Rows: %d, columns: %d' % (X_train.shape[0], X_train.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows: 10000, columns: 784\n"
     ]
    }
   ],
   "source": [
    "X_test, y_test = load_mnist('data/mnist', kind='t10k')\n",
    "print('Rows: %d, columns: %d' % (X_test.shape[0], X_test.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the first digit of each class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHnJJREFUeJzt3Xm81XMex/H3IWUXjYx5YJBEMynL2IaHpt0yGSUykhJD\nZaxRmploG4yhIipbCzN2ZpgxlkqMBllHC7KMXYoSUhFn/uDzvZ/j/u695957lu859/X8x+fxPcv9\n9XXu/Zzv9/f9fr6pdDotAABis16xLwAAgCQkKABAlEhQAIAokaAAAFEiQQEAotSougdTqRRL/LKU\nTqdTNT2H/sxeNv0p0afZoj9zi/7MvaQ+rTZBffei/FxNGUmlsvqsSqI/s1Gb/pTo05rQn7lFf+Ze\nVX3KFB8AIEokKABAlEhQAIAokaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAA\ngCiRoAAAUSJBAQCiRIICAESJBAUAiBIJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIA\nRIkEBQCIEgkKABAlEhQAIEokKABAlEhQAIAokaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAg\nSiQoAECUSFAAgCg1KvYF1Mc777wT4gkTJkiSxo0bF9rOPvvsEJ955pmSpO23375AVwcAqA9GUACA\nKJGgAABRSqXT6aofTKXS1T1eDO+9916I27ZtG+JPPvmk2tdtueWWkqRly5bl/JpSqZTS6XQqi+dF\n15+1cd1114X4tNNOkyR98803oe2VV14J8a677lrnn5Ntf3733Kj7dO3atZKkr776KrQ9/vjjkjI/\nyyeeeGKIGzXK7cx77P350UcfhXjdunWSpHnz5oW2I488MsTrrVf779T9+/eXJE2ZMiW0rb/++rV+\nHxN7f+bbSy+9FOJOnTqF+IUXXpAkbb311rV+z6r6lBEUACBKJCgAQJRKZhXfW2+9JUlq3759aFux\nYkWIU6lvR4dbbLFFaGvSpEmIly5dKkl64403QtuPf/zjENdnyF/OZs2aFeJzzjknxElTLfb/oCHy\nU8yXX355iGfPni1Jeuqpp6p9vZ/uGzFiRI6vLh5LliwJ8YwZMyRJ1157bWizKeO33347tPnPWl0+\nY9OmTZNUMc0vSWPGjAmx/ztRTK+++mqI7W/bvvvuW6zLqZL/LHfs2DGvP4sRFAAgSlGOoOyGso2a\nJKlbt26SMvc+JWnXrl2Ix44dG+KDDjpIktSyZcvQ5r+5DRgwoB5XXL4WL14c4jVr1hTxSuLhF9rY\n/jv7ryStXr06xHaDfKeddgptzZo1kyQ9++yzoc3fwB84cKCkut1sjt2wYcNCfPPNNxf0Z/s9krbI\nR5JatGhR0Ouoip+tePnllyXFNYKyz7If6fm/D/nACAoAECUSFAAgSlFO8Z133nmSpIkTJ9b6tY8+\n+miIV61aFeKjjjpKknT33XeHtueff76ul1j2Fi1aJEm66KKLEh/fa6+9JEkPPfRQaNtkk03yfl2F\n5qc17cb6pEmTQtvKlSurfX2bNm0kZX4uba/PNttsE9o+/PDDSu9ZjlN8v/zlL0OcNMX3ox/9SJI0\nZMiQ0Ob32iUtzvn3v/8tSbrnnntydp3FcOWVV4a4S5cuRbySZJ9//rkk6eKLLw5tVkJOys/nlREU\nACBK0Yyg/OIH+2aVtAPbRkKS1LNnzxD36dNHUmYx2N133z3EQ4cOlSTdeeedoa3cdnjX12uvvRbi\nww47TJK0fPnyxOdecsklkjKX9ZejuXPnhtj+zTVp3bp1iB977DFJ0uabbx7aPv744xxdXenxv79J\nny0bIW266aZZv+epp54qKfP33S9TNyeddFKI/RaTWHz99dfFvoRq+YUlxvd5PjCCAgBEiQQFAIhS\nUaf4/O75PffcM8S2K9/vGj/++OMlZRYstRv5vr13796hbeONNw6x3Xz1N1lvuummENv+jIZ8XtT1\n118f4qT9Zj169AjxL37xi4JcU7FZFYKqWFHcDh06hDa//85P7Rm/v6+h8b9/SX1TF88995ykzKKz\nSXbYYYcQ57ogb129//77IfZ/D2OUNCXbuXPnvP5MRlAAgCiRoAAAUSrKONeG4pdeemlo84VfbX+I\nLw9j5V8aN24c2nxZIx9n64svvgjxZZddJilzL0JDkNQHUsVUjJXlkaTRo0cX7sIicc0114T4gAMO\nkFRRdkuq+KzWZg+YFS5G3dmZWlJFmSn/WU5i+ytj4vcR1nT9xeD3ks6fP7/S4/7vQz4wggIARKlg\nIyjbPS9V7BL3O8n9fpoHH3xQkrTLLruENn8iaT7873//y+v7x8YWovjTSpP4ShK77bZbPi8pSptt\ntlmIBw0alJP3tCM4kB3bSyZJ5557riRp4cKFoe3LL7+s9vUHH3ywpLqdxptvCxYsSGyvy4xQPvzu\nd78LsS3o2GOPPUKbn9HKh/j+jwEAIBIUACBSBZvi86VHkopEPvnkkyG2vSXeRhttlJ8La6CswOZ/\n/vOfxMd79eolSerXr1+hLqmk+RJan376aYitnJbf0+fPgTKHH354iHfeeed8XGIU/MnDt99+uyTp\n/vvvr/Y19913X4irO1G3adOmIbbTeqWKs+A22GCD2l1sEe23334F+Tlr164NsX0u/Tl5t912W6XX\n+IVkG264YR6vjhEUACBSJCgAQJQKNsU3ePDgENu0h69snDStlw9VnS3TECqbP/300yE+8cQTKz3u\nz+qx0lH5HsKXEltJ6svTjBgxQlLVx5fb5y1pBZkvqzV16tQQx7jarD4++OCDELdv3z7Er7/+ek5/\njv/8WjX+UuWnQqvjP4v+b5udP+ZXJ9tqx6uuuiq0+QrqtpfPn0Xlf//t85/vCuZeef0mAADKRl5H\nUP7EWr+XwW502o34QvLfTv0N13322afg11Io9m1s//33r/Z5ft9ZOZ6Omy3/rfLdd98NsX3794V0\nrSCxHw0deuihIb7lllskVZxG6vm9gf/85z9D/Otf/1qStP7669fp+mPmZyqynbWo6URd4xdG+JNe\nY9lTlMQXtPZ/j7p37y5JatWqVbWvf+KJJ0Ls+9OK4fpztWzhhT+t2PaISRX95H/3/efaqkoU8qRn\nRlAAgCiRoAAAUcrrFN+aNWtC7Nfb29lMfu9HPvgplKQisEcffXSIhw8fntdrKabLL79cUs0334cO\nHVqIy4mWTe298MILoS1pP4ovINuxY0dJUosWLULb6tWrQ/ziiy9Kkp566qlK77NkyZIQ9+/fP8S2\nD8r/7FjOL6qLbbfdNsR+oc4dd9whKfOmfLalc2644YYQX3jhhfW9xKIZNWpUiP1naM6cOVm9vmXL\nliG2qWGpYrreF9zOlt+X5j+jxSh1xggKABClonwts6WL/gZervhR06RJk0J8/vnnS5J23HHH0OYL\nIea76GGh+dM5fZWD7/Pf3At58zMWfkGEHdtgn5Xvs2+offv2DW32WfZHJRxxxBEhtgopTZo0CW12\nrIkfqfll5occcogk6ZhjjglttpxdSv692W677RKvOTa+KPTJJ59c5/exorFSaY+gPL/1I2kbSKH8\n4x//SGw/6aSTCnwljKAAAJEiQQEAolSUKb4TTjgh5+9pU1r+lF5/M9umsqxCQrnz+7rsBGOva9eu\nkqSJEycW7Jpi4ffVjB8/PsS2SMSfATVt2rQQW5/53fVvvfWWJOmUU04JbX7PX5s2bSRJt956a2iz\nm81+4dBvf/vbEN94442SpOnTp4c2K6zq+aKyixcvrvR4OXvuueeKfQkNTo8ePQr+MxlBAQCiRIIC\nAEQpr1N8VZU1sWmTP/zhD/V6fysjI1VMkaxYsSK0nXHGGSEeN25cvX5WqVm6dGmIk/Y/2XRWua1e\nzIZfpeT3ftnqOH/+0N577x3iV155RZI0efLk0GZFYv3eJz9taiv/Nt9880rX4Vf2+WO0bdqxZ8+e\noS1pajrGz7RfFTl//nxJ0k9+8pPQVt8zmR5++GFJxSmThsJjBAUAiFJeR1C++KGPrQCn30U9YMCA\nENtN6oULF4a2KVOmSKo4CVaS3nzzzRDbLuzevXuHNj+Cagh8EUi/ECCJ/8be0AwaNCix3fbQ+f1x\nK1euDPGCBQuqfE+/585/lutzdIYv5Onj2Lz66qshvuiii0Jsp7EuX748tGU7gvIj0nnz5oXYfr+T\niu/6wqscE5MbfubLFgQV8sRnRlAAgCiRoAAAUSrKPii7keqn+Hzxx6222kpSxU3Wqvhzd7p16yZJ\nOv3003N2naXC9oD5kkZ+asluxvuSMA35vCdf7soXw7TixnPnzk18XZ8+fSRJnTt3Dm32GWzatGlo\nK7cTcWvSr1+/ECcVxfWLOZIWiyTxC1XsdFgp81aBsf05vvxRMQqbliPf3zXdNsiHhvWbBAAoGXkd\nQfnlpZ06dQrxzJkzKz3Xn1zqC52a5s2bS5IGDhwY2uq7TL1c2A3jpH6TKkYMDf04DTNr1qwQ+xNJ\nbeTkj4c49thjQ2w33svxpNt8Gj16dM7ey47q8dVoRo4cKam0jyQpBbNnz5ZUccRMITCCAgBEiQQF\nAIhSXsfE/oaov4E/Y8YMSTXvUxozZkyIrRhns2bNcnmJaIB8BYf27dsnxsie7XeSMk+uvuKKK2r9\nXq1bt5aU+bfDn7hrfwf8NCzyx++DKgZGUACAKJGgAABRSlU3hEulUuliD/FKQSqVUjqdrrxBo/Lz\n8tKftorP9ulImftIWrVqJUlatGhRzn92PmTbn989l89oDQrZn1YuSpIeeOABSZlHu/uzyewI8e7d\nu4c2m2ZNOtY+FuX++fSrXP2evwsuuECSNHbs2Jz/zKr6lBEUACBKjKByoNgjqHJT7t9QC43+zC36\nM/cYQQEASgoJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIARIkEBQCIEgkKABAlEhQA\nIEokKABAlEhQAIAokaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAAgCiRoAAA\nUSJBAQCiRIICAESJBAUAiBIJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIARIkEBQCI\nEgkKABAlEhQAIEqNanpCKpUqxHU0GPRn7tGnuUV/5hb9WXepdDpd7GsAAKASpvgAAFEiQQEAokSC\nAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAAgCiRoAAAUaq21FEqlaLMRJbS6XSN9Uzoz+xl058S\nfZot+jO36M/cS+rTGmvxUQqpZrWptUV/1qy2tcvo0+rRn7lFf+ZeVX3KFB8AIEokKABAlEhQAIAo\nkaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAAgCiRoAAAUSJBAQCiRIICAESJ\nBAUAiBIJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIARIkEBQCIEgkKABAlEhQAIEok\nKABAlEhQAIAokaAAAFEiQQEAokSCAgBEqVGxLwBx+uijjyRJP//5z0PbunXrQvz6668X/JoANCyM\noAAAUSJBAQCixBQfgpEjR4Z48uTJkqRly5aFtr59+xb8mgA0XIygAABRSqXT6aofTKXS1T2Ob6VS\nKaXT6VQWz4umP1etWiVJ6tWrV2h78MEHQ5xKffvP2W+//ULbzJkzQ7zRRhvl7dqy7c/vnhtNn8aK\n/swt+jP3qupTRlAAgCiRoAAAUYp6kcQ333wT4rVr11b73OnTp0uqmLqSpEWLFoV4/PjxkqThw4eH\ntokTJ4bYpqwuv/zy0DZw4MC6XHa0bG+TJA0ZMkSS9NBDDyU+d+rUqZKkn/3sZ6Etn9N6QC59+eWX\nIe7WrZukzL17//3vf0PctGnTwl0YaoURFAAgSiQoAECUijLFt3LlSknS119/Hdr8kNumnT755JPQ\ndu2119b65+y4444hPvfccyVJN9xwQ2jbYostQnzwwQdLkjp06FDrn1MqPv300xDffPPN1T7X+m63\n3XbL5yUBtfLZZ59l/NfbZJNNQvzss8+GeM6cOZKktm3bhjamq0sDIygAQJQKNoJ69913Q9yuXTtJ\n0ooVK3L+c9ZbryLn+tGSfWMaMGBAaGvevHmIN910U0nS1ltvnfNrKjZbHHHooYeGtqS9GU899VSI\n99lnn/xfWAPw17/+VZK0Zs2a0DZ//nxJ0pVXXpn4mj333FOS9Mwzz+T56uLwwQcfhNj65M0330x8\nro2MkooV+wVO1sdSxWe9ZcuWoc0vwCpnvh+nTZsmSXrggQdC29NPP13pNX/5y19CvP3224f44Ycf\nliT169cvtPlZqnxgBAUAiBIJCgAQpYJN8TVr1izE22yzjaS6TfF16dIl8T3vvvtuSVKTJk1CW/v2\n7Wv9/uXolltukZQ5LdKnTx9JmXvBNttss8JeWBlYvHixpMw9d75k1PXXXy8peUrVykl934svvihJ\n2muvvULbc889V/+LjdTcuXND/Kc//ana52644YaSpDPPPDO02e++LYT6PuvnwYMHh7ZyXyRhfXrM\nMceEtg8//FBS5mexR48eIX7nnXckVfxt+D57nS8gffXVV+foipMxggIARKlgIyj/jcVu1t15552h\n7YADDghxz549K73+oIMOkiT9/e9/D22NGzcO8ZIlSyRJEyZMyM0Flzi/IOKxxx6TJO26666h7Yor\nrpDEqCnJ559/HuITTjghxH4rhLFZAL/s2X9DtVH8o48+mvXPtxv4th2jXF1zzTWSpPPPP7/SY+ec\nc06IbcZFkgYNGiRJ2njjjUObjZx81RMbLUjSD3/4Q0mZp0OXC7/Ywy+IOPzwwyVlfpZ/9atfSZLG\njBkT2vzCEdv2c9JJJ4W2W2+9tdLPPPDAA+t51dljBAUAiBIJCgAQpaJUkrCh+B577BHa/HSdDfn9\nDdPRo0dXep5nw/iLL744txdbQvy+GV8E1m4Sn3zyyaFtgw02KNyFlQhb6GBTIZL0xhtv1Pp9bLpZ\nqthf56daPv74Y0nSEUccEdqS9v3sv//+tf7ZpcT65Isvvghtu+yyiyTpwgsvDG3Wh97y5ctDbFNW\nvt99VYlJkyZJkho1iro2dp088sgjIe7atWulx4899tgQ33jjjZIyF5J5jz/+uKTkaT2pYs/TUUcd\nVadrrQtGUACAKJGgAABRKuqYt6qh5pZbblmpzUqgWFFXqep9JA2NldGZNWtWtc/7wQ9+EOLNN988\nq/e+4447Qpw03TV06NCs3qcUjBo1SlLN03q2F0eSZsyYIUnae++9Q1tSuSy/ivWqq66SVHU5H1tt\ned1112Vx1aXL9uj4z5jt9xoxYkRou+SSS0Js58L5VX433XSTpMx+96t5jzzyyFxedhTs7+HZZ58d\n2vzfQ+s///tZ1d9bc9ZZZ1X7+G233SYpcwVlvjGCAgBEKcq7hpbJ582bF9ruueceSdLChQtD209/\n+tPCXlik7JuT7y+/P8IK6PrRZxKrOOHf09+sfu211yq9ZtiwYSH2x3mUyv6qBQsWhNgX0UzSokUL\nSdL9999fqa023n777Wof79u3r6TCflMthu22206S1LFjx9BmIyirDiFJxx13XIiPP/54ScnFYm1f\nlZS8l7LUTZ48OcQ2cvKjot69e4f4ggsukJS8GGrdunUh9nv7Xn31VUmZ+/h8QeNiFJBmBAUAiBIJ\nCgAQpSin+Gyvkz9F1xYA+Buefr+KlTHxa/QbyiIK27/jy0D5c7FsGippYcR7770XYr/IwspReX7a\nbuedd5aUOUXQq1evENsNVX9qcYzGjh0bYr9XyVjJGKniZn1tpvVsAYuffr333nur/TnleFM/ie1L\natq0aaXHrHCplLkfzKaf/O+27Zvs3LlzXq6zmPw5YrYXVKr49/tpPdvnVBXbO+b3Rvl9VObUU08N\n8SmnnFLLK84tRlAAgCiRoAAAUYpyis9stdVWIbYzdrp16xbaxo8fXyn2w1y/kiepXEops/0gUvK+\nHX9U8xlnnCEp8/wsOwb+0ksvDW1Tp04NsVWQ9tN25513XoitPM3uu+8e2pYuXVrLf0Xx+b0f77//\nvqTM/TR+qrMunyE78v03v/lNpcd89W1/zHa5fVZrYuWNasOfWWTVzLPd21dKrMK4lFmh3YwbNy7E\nq1atCrGdFGFT7ZL0xBNPSMpcbeunSpNKolVVWq5QGEEBAKIU9QjK23fffSVl7oPyu6htN7o/y8Tv\nlbBv/6WyP6cmL7/8coj9TU/j9yeddtppkjK/YQ0ZMkSSdPPNN4c2v6DBvvH//ve/D2026vI/07+m\ne/fuie8Vs/322y/EtTmzqTr+9NvTTz+90uO2N8X/P2pooyapYq/eww8/HNqSTh727Hyu6dOn5+/C\nIrL++uuH2ApiSxWFcf0sU02LwnbYYQdJmYtS/GIUmzXxJzkXGyMoAECUSFAAgCiVzBSf2XbbbUPs\nb2DbNFanTp1Cm9/j8sorr0jKvGlYyl544YVqH7f+8PyCB39elHnyySdDbAVL/QIMf2S88X1cToVj\n68MvfkiadrnrrrskSYcddljBrilGAwcOlCRdf/31oa2maaqGsrfR+MLEdl6TVLE3bNmyZaGtdevW\nIbapUCubJVWckWWPSZlTfPb/IyaMoAAAUSq5EZTnv120b99eUuZNRV8U8W9/+5ukipGUJLVq1SrP\nV5g/diqrVHFjuX///onPtWoRfoGJvcaWQUuZIyRbEHHooYdWeo1/XdICjYbIL/dNKtTr+RFWQ/DZ\nZ5+F2M9g2HEiflR0yCGHSMrsoz//+c8htq0ADZGdaCtlnh6cLSsGa38LpczP52677Vb3i8sTRlAA\ngCiRoAAAUSq5KT4/xPdnxtguaT+t59mUQdKN/lJnUyQ13UD2w3l77jPPPBPa7AwZSVq9erWkzDO3\n/HNrOp2zobCd/r5vkvrZdvZLmScbNwTPPvtsiH0hUuNPDrbznuz3Wcqc4mvbtm0+LrFBsMKzSZ9P\nKXM6PxaMoAAAUSJBAQCiFPUUn1/jf/XVV0vKLGj67rvvVvt6v6LPVsCUyz4KfxaWnYfj+8ZP19nq\nvZUrV1Z6H7/6zK/Ss7Inl112WWgrlzJR9fXVV1+F2Mr0VLW/zkod+SLH5fIZrImtmK3q+HWb+mvT\npk1oszO5Bg8enPia2pzFhUy+n0sFIygAQJSiGUH500zvu+8+SdKoUaNC2+LFi7N6nw4dOoTYTkCV\npL333ru+lxgVKzgqVRQa9X3YsmXLEGf7jT2pWGy7du3qdZ3lxI44Oeecc0LblClTKj3Pj6Zs9NBQ\nRk3ev/71L0nSihUrQps/8XrPPfeUlHmkxOzZsyVVnP4qZY7sfSUZ1M78+fOLfQm1xggKABAlEhQA\nIEpFmeKzc4l8oUJ/Qubzzz+f1ft06dIlxCNHjpRUc6HOcuFPzJ0zZ46kzMKtfo9YEpum8lOfNuUi\nled+sfqyRSZJ03q+UOfRRx9dsGuKme23STq1VaqY2ps3b15os4LGfq+YL0J85JFH5udiG4Ckk7dj\nxwgKABClvI6grBqBJJ111lkhtrLx/lTYmtjRBCNGjAht/ga+XzTQ0Fg/2KnCyB2/1eGKK66o9Pge\ne+whSXrkkUcKdk2l4sMPP6zU1rx58xDbSPPee++t9DxbYCHFdcJrKbNTyWsqZhyTuK8OANBgkaAA\nAFHK2RTfm2++GeI//vGPkqSZM2eGtrfeeivr99p4440lSaNHjw5tgwYNkiQ1bty4PpcJ1Ir/DF5z\nzTWVHr/wwgslZe4hw7ds+tPzC0xsf9PWW28d2mwKvxSrHsTO9pD5AtAvvfRSiG1KdqeddirshVWD\nERQAIEokKABAlHI2xXfXXXeF+IYbbqj2ubYq57jjjqu4kEYVl2JldvyR7kCh+OO0kwrsDh8+PMQH\nHnhgQa6pFNmeJV/E2IrnSlLnzp0lVex9kqTevXsX6OoarvHjx4e4a9euIbai0xMnTgxtVjS6WBhB\nAQCilPKFGCs9mEqlq3sc30qlUkqn0zWWraA/s5Ntf3733Jz3qT/BddiwYSG2AryPPfZYaPM3+GNV\n7P4sN6Xen1b0WJL69+8f4ttvv12SdMopp4S2CRMmhDifC9Sq6lNGUACAKJGgAABRYoovB5jiy61i\nT6H4vSF+P86TTz4pSdpnn31y+vPyrdj9WW7KqT/9dJ+dn+f3/r333nshzueCCab4AAAlhQQFAIgS\nU3w5wBRfbpXTFEoM6M/coj9zjyk+AEBJIUEBAKJEggIARIkEBQCIEgkKABAlEhQAIEokKABAlEhQ\nAIAokaAAAFEiQQEAokSCAgBEiQQFAIgSCQoAECUSFAAgSiQoAECUSFAAgCiRoAAAUSJBAQCiRIIC\nAESJBAUAiBIJCgAQJRIUACBKJCgAQJRIUACAKJGgAABRIkEBAKJEggIARIkEBQCIEgkKABClRjU9\nIZVKFeI6Ggz6M/fo09yiP3OL/qy7VDqdLvY1AABQCVN8AIAokaAAAFEiQQEAokSCAgBEiQQFAIjS\n/wHTK4KPiGYqugAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x110041390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "for i in range(10):\n",
    "    img = X_train[y_train == i][0].reshape(28, 28)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/mnist_all.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize 25 different versions of \"7\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmAzdX/x/HnHdmy7/TNnqVEhG8bX9VXG0lUtiJfpZQl\n9YtUKipU0kZUIimUIkskCtW3JHvlK3xR9qUsgxnDmPn98fmec2fMdmfc5dyZ1+MfM5/7udfb2733\nfM75nPM+vuTkZERERFwTE+kARERE0qMGSkREnKQGSkREnKQGSkREnKQGSkREnHROZg/6fD5N8QtQ\ncnKyL6tzlM/ABZJPUE4DpXwGl/IZfOnlNNMG6n9PCk00uYjPF9B7FVA+A5GdfIJymhXlM7iUz+DL\nKKca4hMRESepgRIRESepgRIRESepgRIRESepgRIRESepgRIRESepgRIRESepgRIRESepgRIRESdl\nWUnCVVOnTgXgxIkT/PLLLwC88cYb9vFGjRoBsHLlyvAHJ7lCYmIicXFx9veff/4Z8FcGWL9+fabP\nL1CgALfffjsARYsWJSZG14PZYfJ87NgxABYsWMDkyZMBWLdunf3clyhRIjIBOsTkaubMmZk+7vP5\n2Lt3LwD9+vUD4OOPP7aVHFq2bAm4k1NfZmU4fD5fsgtlOjZt2gTAf/7zH7788ksA3n33XSDjMiLm\ny+Diiy8GYPXq1SGLz+fzBVyLz4V8ZiYpKYlDhw6lOW7esOecE/prmkDz+b9zQ5bT3r17M27cuKC8\nVvfu3XnzzTcBKFy4cFBeM1Cu5DMQsbGxACxevJgJEyYAMG/evDTnFSlShH379gFw7rnnhi9A3Mln\nXFwcy5cvB+Chhx4CvO/I9KRsoNJ7zByvXLky4DVeXbp0AaBChQrBDTwdGeVUl3QiIuIkp3pQx44d\no2vXroDXhTfMFf3Ro0ftlcDVV18NwDfffJPpa1atWhWALVu2BDtcK1w9qFWrVgHQuHHjHL8GQEJC\nAgBr1qxhzJgxAJw8edL+OWfOnDTPefvttwHo2bPnWf3dgXDlCvWee+5h69atQM56PYmJiXz11Vf2\nd9OLb9iwYXACDJAr+czI7t27ARg+fLjtNSUkJFCrVi0A7rzzTsDL5/PPPw9Ajx497ChKuLmSzx49\netghz6wE2oNKqWbNmoD3OQDo378/BQoUyGm4mcoop040UKZbeuutt9ovhIzs2bMH8Mb0wWvU/vrr\nLwBuvvlmAH7//Xd7fseOHQGYMmVKUGNOKVwN1KlTpwDInz9/wM85fvw44G/cfv75Z/ulmbIhatGi\nBQCdO3fmk08+AeDrr7+2j5svaPN6oeTKF8CpU6fsUHG+fPmy/fykpCTat28PeLl+4YUXABg4cGDw\nggyAK/lM6bfffuOWW24BYNeuXQDEx8fz+OOPA96QaLVq1QD/+33Xrl12CGr06NH07t075HGmx5V8\nNmjQIN0hPXMvvmLFigG/Vt++fQHssCmkbdTWrFlD/fr1cxxvZjTEJyIiUcWJWXzPPvssQLq9p0KF\nCtlubOPGjSlXrlyqxwsXLszo0aOB1D2n2rVrAzB+/PhQhBwR2ek5Gc2bNwdg7dq1QOrufHJyMvfe\ney8Ar732GuDdfK5evTrg70ElJyfz8MMPn13wUSgn+U7pzOFS05sXb9jevDfNaMhdd91FkyZNgKz3\nXCpSpEhoA4wCixYtYuTIkYD/8wvY70szoSyQGXmmp29Go8IxMSIQ6kGJiIiTItqD+vXXXwFvfcOZ\nzA26+fPn258zsn379jTHunXrBoR/CqprzD0Uk4fmzZszYsQIAKpXr07x4sUB/xXrBx98wAMPPJDq\nNerWrcugQYPCFbLkAVdccQVXXHFFtp7z2GOP2Z87deoU7JCiToUKFXjxxRcB7L2hcePG2bWf06dP\nB6BDhw5nta7J3MsqWbLk2YSbIxFtoIYNGwb4F+IBtG7dGsDeUM6ocTpx4gQAP/30U5pZZ61bt6Zt\n27ZBjzcaTZw4EYDzzz8fgNKlS6d7nplE0atXL+Lj4wHsLKply5bZYRgJ3A8//BDpEHKVlEP44jGT\nd+6++27Am2h21113Af5JUB06dAj49dKbGdm5c2fAv0YqnDTEJyIiTopoD6p///6Afx1EuXLlmDRp\nEkCWV+ym1NF9991njzVt2hTwppTrit/ToEGDTB//4IMPAOyU3fj4eDtJwvQAItG1zw0OHDhgf65R\no0aaG89r1661a9LSc9lll4Ustmh1zTXXAIRsPU60K1GiBHPnzs3288z0cvP9C15vDOC5554LSmw5\noR6UiIg4yYmFutlhVuM3a9YM8KbymunAH330EUDY7z9Fay2+1atX2xvVZhFw3bp1mTZtGgCXXHJJ\nROJyZSFkduzfv99W4zh69CgAN954Izt27AC8ySrFihVL9ZzY2Fi7GLJ48eK2OoqpnHDHHXcEJbZo\nzGdKpj5fhQoV6N69O0DQaiTmRLTnMz1mkkXKhb/ff/89AJdffnnI//6McurEOqjsMMN4KddJzJgx\nA4BWrVpFJKZoY9ZEXXvttbZhqlGjBgDfffcdZcqUiVhsrjp58qT98G7cuBHwipiatXurV6+2E3cy\nYt6zJtft2rWzwyiVKlXKcAJLXrd48WLAK3/0yCOPRDia3GfJkiW2FJx5jzZp0iQsDVNWNMQnIiJO\niqoe1KuvvkpSUhJAqr11TK9KsrZ27Vqbr9OnT3PhhRcC/u68JkSkZvaDuuCCC+w+Olkxk0y2bdtm\njy1btkzv0xwyFU1iYmIoX758hKPJPcx7e+DAgXZ4ukqVKoC/CkWkqQclIiJOiooe1OnTpwFvd1zT\nczJjpZ9++illy5aNWGzRwkwuufbaa20+AZYuXQqo55QRsxCyTJkytgf18ssvA94KezMhJ2WP3lyN\nNmjQwE6SMBtnSvaZZShXXnmlMzu95gavvPIK4H03mO9Ts/THlTw730CdOnWKRYsWAd7WxEafPn0A\nb6ZUVoUl8zIzA8qsc4qNjaVSpUqAtz20GvfMFSxYEPDWi5lcXnnllUDGuwubrUkqV65sGygRV5jh\nu2eeeQbwtoUxxbVdKyGlIT4REXGSsz0os8L+kUcesbu5gr8XddtttwFZl+XPy+Li4uzuraaOWY0a\nNZg/fz7g35JEshbuXXDF/x1gikm3adMmkuHkCnFxcXaI2nx3xsTE2F6Va5NQ1IMSEREnOduDOnLk\nCECq3tNFF13E7bffHqmQos7OnTttz8ncL1m4cKFdKCqhVbZsWaKhioCrli9fDmCr6w8cODCS4eQK\ny5cvZ8mSJamOde/enb/97W8RiihzzjVQpsCmmWEC/oKnZyZWMle0aFG786jZzVWNU/h07tw5zVYw\nErj3338/1e+u7PIazR566KE0xwYNGnTWu0eHiob4RETESc71oExp97Fjx9pjZjqkK3Pzo8V5553H\nzp07AShUqFCEoxHJGbNGz+z+LNl3//33A7B+/Xp7zGxVtG7dOruEonHjxuEPLhNOVTPfu3ev3dZ5\nypQpADzxxBP07dsX8PaLclG0VjN3VW6pFh0fH28XmVavXj3VYt5witZ8NmrUCPCX5DFFeiMtGvNp\nFpynN+s5OTnZFoutVq1aOMOyMsqphvhERMRJTg3xffjhh7bnVKtWLQD69u3rbM9JJDOFCxemZs2a\nkQ4jKk2dOtVuC/Pkk09GOJrcbeDAgc7O4lMPSkREnORUD6p169YMGjQI8Gqfgbv3nUQkdPbt22d/\n7tKlSwQjyb3MxIkRI0ZEOJKMOTVJIlppkkRwReNNaJcpn8GlfAafJkmIiEhUyXKIT8VYg0v5DD7l\nNLiUz+BSPnMu0yE+ERGRSNEQn4iIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiI\nOEkNlIiIOEkNlIiIOCnTUkc+n09lJgIUaLHYcMSSG2SnGGeoY8kNlM/gUj6DL72cZlmLT6WQspad\nWlvKZ9ayW7tMOc2c8hlcymfwZZRTDfGJiIiT1ECJiIiT1ECJiIiT1ECJiIiT1ECJiIiT1ECJiIiT\n1ECJiIiT1ECJiIiTslyoK3nP4cOHAZgyZQoA/fr145ZbbgFg2rRpABQqVCgywYVQXFwcAM2aNQNg\n3bp1PP7449l+neHDhwPe4kOzSPPee++lfPnyAHTs2BGA+vXrn3XMItmRmJjI5s2bAZg9ezYAX3zx\nBd9++22q85544gmGDRsW9vjOpB6UiIg4yZdZGQ6fz5fsUpmO06dPc/LkSQDeeuste3zPnj0AjBw5\n0h5r2LBhqvMuu+yykMX1vyvlgGrxuZTPuLg4li9fDsBDDz0EeGVZjh07BsCOHTvsuSbuhx9+GICX\nX345ZHEFms//nRu0nJory86dOwOwdu1a++9O2RsyZVmSk5PT/dm897J6/vfffx/S96URqXwGy8SJ\nE+3Pjz76KABHjhzhoosuAvzvSYBWrVoBULFixZDFEy35NN+L8fHxLF26FIAFCxYwY8aMVOelfO+m\ndPr06ZDHaGSUU+eH+E6dOsXo0aMBWLRoEQsWLADSr92U8ti6desA7H9GOL4IXGbebIcPH7aN0b59\n+1iyZEmq8zJ6sxp33nln6IKMsFq1agGwcuVKAPbv329/Tk+ZMmWy9b7avn07ANWrVwfgr7/+ymmo\nudKJEyfs53v69On25yNHjgCpP98+n48NGzYAcN9999njderUAWD9+vVhidk1n3zyiW3Qf/zxRwBi\nY2PTXBwBNGjQwP754Ycfpnqd3r17hyPcLGmIT0REnORcDyo2NhbwD7cMGTKEefPmpTkvX758ANSt\nW9ceM0NTf/zxR6jDjBrmyumDDz4AvJv16V1NBapq1arBC85x5cuXt0NGwTBr1iwgZ3nPjdauXQvA\nqlWrABg6dCi7d+8+q9fcu3cvANu2bbM91bykZ8+eHD16NM3xdu3aAdC+fXvatGkDQIECBQD49ddf\n0/SgnnrqqRBHGhj1oERExElO9KBOnToFwBtvvGEnOuzfvz/dcwsXLgx4N5gBGjVqZJ9vbgTecMMN\n9vxKlSqFJGaXbdq0CYBffvmFfv36Ad79pvR0794d8N8X+de//kXTpk3TPGfo0KEAlC5dOiQx5wXm\n3l9MTN68Lly9ejXLli0D4L333mPjxo2AdxM/u5o2bcqKFSvSHC9ZsiRAnuw9ASxfvpzVq1enOta+\nfXsKFiyY4XMWLVpkR1WuvPJKAIoXLx66ILMhog2UaVgGDx4MpJ6FZ1SvXt3evL7++utt41OvXj17\nzs8//wykbpjMup0+ffqEIHI3mYkhLVu2BODQoUNpzmnSpAm1a9cGvHU4/fv3ByB//vwAnHfeeWku\nDpo0acIjjzwSsrjzgv3799uGyQzxXX311RGMKPxatmxph/Cz0rBhQ/u5f+GFF9I8vn37dq655po0\nx10ZmoqUOnXq2IkiWTFDgdOmTbPvyZtuugkg0wYtnPLmpZyIiDjPiSG+UqVKAd4U5vvvvx/wr2Mo\nW7YsxYoVA/wTI1LatGkTrVu3TnWsb9++tldmegZ5gbkKMlfq9evX5/XXXwf8Oa5SpQolSpRI81xT\nRWHfvn32dcxV1Isvvsi5554b2uBzueHDh9thlNdeew0gT+a0QoUKgDe1+bHHHgPgs88+A6BFixZ2\nbdN5552X6TBTekPWNWvW5Pbbbw92yLnWSy+9BHiTJMz/S69evSIZUhrqQYmIiJOiqpJESub+VefO\nnZk5cybg9ZzAu5dlplCGg2uVJMwUfTOGnxnTc7rxxhsBb/KJmSTx4osvAt6VbThFy0r9QJj7eRdc\ncAHHjx8H4PfffwegcuXKYYnBlXyuWbOGsmXLAmf/b7///vuZMGEC4O+JTpo0ifbt259dkAFwJZ9n\nq0aNGoD3fjSf//nz50cklqitJHEmU+rIFNrcvHkzjRs3Bvzld/LSsF56AmmYDLMK38yKvPzyy1m0\naBGQN4eggm3x4sUAHD9+3M4sK1OmTCRDiphGjRqd9Ws8//zzAKnK9ZhyZuFonHKDXbt2AXDw4EHA\nuyVgChy7RkN8IiLipKjrQW3duhXwD2OVKlWKjz76CFDPKbvmzJnD559/DvgnWCxatEg9pyDZv3+/\nnZ7v8/mYOnUqoJ5pdiUmJgLw22+/MWrUKMCr22dGTG6++eaIxRaNXnnlFcA/zfzaa6+1BY5dE1UN\n1JYtW2yBw6JFiwJeYc+8uigvp0wB1G7dutnyUGbWpL48g2fOnDm29E7FihX5+9//HuGIotPkyZOB\n1EVhe/ToYdfwSeD27NmTqjo8eLl0lYb4RETESVHRgzIlUVLuQPrrr78C3roeyZ4RI0YAXnFd03P6\n6aefIhlSrmJmRr700kt26NSsR5PAmc/9PffcA3g3800v1Kwlk+yJj48PuJqHC9SDEhERJznfgzp1\n6pStuZWYmGgLbqrnlD0nTpwAvGodZt1Y7dq1+e677wBvawkJDjNld8uWLfb+6JnVTiRzCQkJXHrp\npYC/MsoVV1zBwoULAd0rza6kpCTA277IrMsylTrMDtIucraBMns69enTx95o7tq1K6+++mokw4pa\nZofR2bNn2w/8vffeq4YpBMxNaJ/PZ/fe0RdqYMwC/L59+5KQkABgh/WGDRumPObQjh07AJgyZYod\ndn7zzTcjGVJANMQnIiJOcrYHZbbemDdvnr2RP2zYsEiGFJVMQc6UO2aaPaA0TTe4TJkY0+P3+Xw8\n/vjjkQwpqiQkJNh1YymnQpvtNsJdcis3mTZtWppjpnfvMucaqOXLlwNeVxS8qsY//PADAOeff37E\n4opGmzZtsh/0lHtDDRo0CNDC5mAz71kzhKIh1MCYBv22226zn/9ixYrZLeGrVq0asdhyo27dugH+\ntaQu0xCfiIg4yakeVHx8vN0B1+wB9d1332nGXjaZWToXXnihPWZ2IDa7D0twnTx50q7bMfm/+OKL\nIxmS88zMUjN0v3z5cpo3bw7AO++8o55TEH3xxReA9958+umnAf/sSJe5H6GIiORJTvSgzJXU+++/\nz6pVqwAYPXo0oPHnnBg/fjzg3QsxE0zM/REJjX379tl7Jibn3bp1s/tB6V5UagkJCQwYMADwPvcA\nbdu25b333gPIdDddyZ6jR4/a9Y4+n48DBw4AREUNUyc2LNy5cyfgLb4dOnQogJ39dM45TrShmXJl\nw0JTndisG9m8eTMrVqwAgrMXT7hE44ZwO3bsoFq1aoB/iO/SSy+lTp06QGQvEFzKp1nn1L9/f2bP\nng3AddddB2AbJ9e5lM9AHDt2zDb4Pp+PcuXKAd5CcoAiRYpELDYjo5xqiE9ERJwU0e6JGdrr1asX\n4PWgzCSJaOg5ucbsNmz2ymrXrl2qiRISOsWLF7dDeynXQZnhVoHTp0/Tu3dvAD755BM7Yedst3+X\nzBUtWtSuIdu2bRtjxowBoHDhwpEMKyARHeJ78sknAf8Y9MqVK+2HPJq4MsSXW0TbEIrrXMnnv/71\nL7Zv3w7A559/HhVfkOlxJZ+5iYb4REQkqmTZgwpjLFEt0B5UOGLJDbJzhRrqWHID5TO4lM/gSy+n\nmTZQIiIikaIhPhERcZIaKBERcZIaKBERcZIaKBERcZIaKBERcZIaKBERcZIaKBERcZIaKBERcZIa\nKBERcVKmJcNVpiNwKnUUXColE1zKZ3Apn8GXXk6z3NNCpZCy5vMF9F4FlM9AZCefoJxmRfkMLuUz\n+DLKqYb4RETESWqgRETESWqgRETESWqgRETESWqgRETESVnO4hMRCYVJkyYBcM8999hjw4YNS3Ne\njx49AChfvnxY4hJ3qAclIiJOynTLd5/Pl6w5/Fnz+XwBL9RVPrMWaD7/d65ymgVX87lt2zYAZs+e\nbY/NmTMHgG+//TbN+XPnzuWmm24KS2yZcTWf0SyjnKoHJSIiTsqVPaikpCSSkpJSHYuJiSEmJjTt\ncTT3oEw8J0+eBGDFihX2KnbkyJEAPPTQQ9x2220ANG/ePOQx6Qo1uKIpn4mJiQDs2LGDF154AYAJ\nEyYAUKBAAT7++GMA2rRpE5kAia58ZtdPP/0EwKOPPsp3330H+O8Bmv+HUMgop1HbQJkG6NSpU0yb\nNg2AP//8E4CVK1cyffr0VOePHDmS//u//wtJLNHQQG3evBnA5sowDdPw4cMzfX69evUAWL58OQDn\nnntusEO0ouUL4M8//+Srr75Kc9zE07dvXw4dOpTmcfPebd68uc17s2bNQhZntOTzTKdOnQJg/Pjx\ngJfPQoUKATBjxgxuvPHGiMQVrfk807FjxwCYOXMm7777LgA//PADQKoL/BIlSgCk+14OFg3xiYhI\nVImKHtTp06cB2LlzJx9++CEAmzZtAuCDDz5Ic35ycnKa4oO33HILs2bNCkl8rvWgzJXRzp07bbd8\n8uTJABw4cCDVuSYek6/8+fNTu3ZtAH7//XcAjh8/bs83vdTSpUuHKHo3r1C3bt3Kvn37AFi4cCEA\nb731Vpp8QtqcZvZ4wYIFAfj+++9p2LBh0OM2f49r+cyJdu3a2eHnXr168eabb0YkjmjM5+HDhwEY\nPXo0o0aNAvzDqXFxcZk+t1OnTgBMnTo1ZPFllFOn1kElJSXZZJnu5LvvvsvOnTsBeO+993L82t26\ndTv7AB1nZkP1798fgD/++CPT8wcPHmy/IP/5z38C3tDdhRdeCHhfCADz58+3wylFixYNfuCO2rBh\ngx0WXrNmjW2MsmqAzEyzjB6fP3++/TkhIQGAEydOBCfoXGzEiBHMnTsXgC+//JL4+HgAChcuHMmw\nnGXuIS1ZsoRXX30VgCNHjgT8/HLlygH+i9tI0BCfiIg4yYke1Ndffw3Ap59+yttvv53puebmfNu2\nbYHUPaMff/wRgOeee87e5GvVqhUAt956a3CDdpDpaabsOV1xxRUAlC1bFoBLL73U9owuvvjidGc2\nmtdJeaXfokULwJtJlVccPnzYDuelVK1aNQDy5cvH0KFDAahevbp9/PLLL0/39UwvqUiRIvZYgwYN\nALjooouCEnNuVrVqVZv733//3b7P69atG8Go3PLnn38yduxYAJ5//nnAP5R3pgoVKgDQunVrJk6c\nmObxl19+GYBzzolcM6EelIiIOCmsTWNSUpKttXXw4EF73FylbtiwIc1zChcubNc89OrVi4oVKwLp\nXzWZK6qCBQva8ek77rgDyP4umNHovvvuA+CGG26wxypXrgxg7zUFwvSgjIoVK9rXzktq1apFrVq1\nAG/dTePGjQHo2LFjtl/rxIkTtjefkrlfWLx48bOING8oXLgw119/PQDvvPOOnTChHpR/8tJVV11l\nl5SkVKVKFcAbSapRowYAHTp0ALzp+2fq1KmTfTySwtpALVu2jJkzZwKwbt26dM8xQ0mme1qqVKks\nhz92794NYGfpxcfHc9dddwFw3XXXAXmjgcqfPz8AF1xwwVm9zvvvv5/q988++4xSpUqd1WtGo7Jl\ny6Z70ZQdR48eBbyJE8uWLQP8a0wefPDBPDF5J1iSkpLsjNLk5GQ7sUdg7969gHdxab77zISmevXq\n0bVrV8C7EDIT0cxa0RkzZtjX6dy5M+Atys3ORW2oaIhPREScFNZ1UL1792bcuHEADBgwgNtvvz3N\nOXXq1AGyN+Tx1ltvAV55DvDW7WzZsgXAdmdDybV1UGdj1KhR9sa/+T9Yt24dZcqUCVsM0bjOJCNm\nWrSZmAJQv359wJv+W7JkyZDHkFvyGRsba9ff3XjjjXz22WeAf+QgXFzO5/79++1nNV++fOmes2DB\nAoBUQ85mKNuMbJmKHeGiShIiIhJVwnoPasyYMfTs2RPwpuoG4+px79699r6WGVv95ptvqFq16lm/\ndl5iavJ9+eWXduza1OUKZ+8pN9mwYQN33323/d1U6Fi6dCngr3EmgTH3msGbch7unlM0yGpTx2+/\n/dbeZzLuvPNOWxg63D2nrIS1gfL5fEEv5/Lqq6/aL1JTafuyyy7LsHsrqZmGyTRCx48f5+mnnwb8\nM38ke2JjYwGvAK/5+aKLLmLJkiWAGqacMmv6wPuMS+DMrOYBAwbYahLmfTh06FA7O9o1GuITEREn\nOVFJIidMCf6xY8faoT2zf0wkVz5HG1OI1xSYbdy4MUOGDIlgRNHPTMlPubThnXfe0VBpDpnC0IcP\nH7ZVOtq3bx/JkKJKfHy8vbWyYsUK/v73vwP+ShHhmEiWU+pBiYiIk6Kuq2Gmj48ePRrw7pmYzbaa\nNm0KZDy9UlI7efKknWpqcvb6669HMqSotWbNGru43CzEbdiwIU8++SSQcX0+ydrgwYMBr5KE2dEg\nL1XVP1sLFy60W2WUKFGCTz75BPBXmXFZ1DVQplTSr7/+ao+ZyREa2sue1157zc4oM2V8rrrqqghG\nFH3MzeennnrKri8xBXj79u2b7lo/CYxZQzZv3jzA2/lVZY0CZ3a/7t69uz02ffr0qGiYDA3xiYiI\nk6JiR13jkUce4Z133gH8K59nzJhhp0NHqgcVbZUk9uzZA3g5NMNRpkBsKHfKDZTLK/VT2rBhAz16\n9AC8m8+GKdZZpUoVJ4aboyWf4N/AsWvXrrbnZIZHzbY8keZ6Pk3PKWXRaLPd0Lhx45zc4FGVJERE\nJKpExU0bs43G22+/bcf8zVh0lSpVdO8pm8xkkri4OAYMGAC40XOKFikX4pqe00UXXWR79yk3L5Ts\nufHGGwGv4sHNN98M+KtuS9Y2btxod3AwS0e6d++e7oaE0SAqvtnNflGmcQLsWh01Ttnz6aefcuDA\nAcDbi8jMMpPAZbTOSTP1csZ8rp955hm+/fZbwJttZnbXdmHbh2gxfPhw2zBVqlQJSD1JItpoiE9E\nRJzkfPdj+/btPPHEE/b3sWPHAme/KV9eY6bld+3alVOnTgHQtm1b7eQaoISEBLp06QKkXuekwq85\nt3XrVgC7LueVV16xj40fP97Z+nAuMttkzJw50xZ8/eWXX4DoHr53voFaunQpf/31l/3dzEBxYXZU\nNDFDogkJCUyZMgWAf/zjHxGMKLoMGTKE2bNnA6nXOalhypk1a9bY3bNNqTKAn376CYAGDRpEJK5o\nY0qVtWyEYHr+AAAWO0lEQVTZEvAW3y9evBiI7obJ0BCfiIg4yfke1JmuvPLKSIcQlfbu3Wt/btOm\nDZD6Jr+kz6zLMcNR4O06DKTa60myp1GjRnY2pORMUlIS3bp1A/wz9pYuXZpqW5Jopx6UiIg4yflK\nEsnJyfamNPjH/126+o+GShLNmjUDvF1yzZWrqwU3XVqpb3pOZjdcgMTExJD9faHgUj5zA1fyOXny\nZDuF/Pjx4wBOVokIREY5db6BigbR0EBFE1e+AEANlKSlfAafSh2JiEhUyXKShEtDabmB8hl84c5p\nbv8/zO3/vnBTPnMu0yE+ERGRSNEQn4iIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiIOEkNlIiIOEkN\nlIiIOEkNlIiIOEkNlIiIOCnTUkc+n09lJgIUaLHYcMSSG2SnGGeoY8kNlM/gUj6DL72cZlmLT6WQ\nspadWlvKZ9ayW7tMOc2c8hlcymfwZZRTDfGJiIiT1ECJiIiT1ECJiIiT1ECJiIiT1ECJiIiToqqB\n2rVrFz179qRnz57ExMQQExNDx44dOXr0KEePHo10eCIiEkRR1UCJiEjekemW7z6fL9mFOfyzZs0C\noH379pQrVw6Aq6++GoC1a9dy8OBBADZs2ABA2bJlwxqfz+cLeKGuC/l0XaD5/N+5YclpbGwspUqV\nAiApKQmAmJgYnn32WQBq1qxpzzXxNGvWjO+//94ev+qqqwCoXLlyyONNycV8RjPlM/gyyqnzDdRv\nv/1Go0aNAChUqBDLli0DoG7dugDs27eP8847D4AFCxYAcN1114U1RlcaqO3btwNQvXp1AE6fPh2y\nvyuUXPwCiI+Ptw3MunXrzN+d7rkmntKlS3Po0CF73DRws2fPBqBp06bkz58/ZDEbLuYzmuWFfCYk\nJPD++++nOpYvXz7uueeekPx9GeVUQ3wiIuKkLEsdRYoZRpk1axbnnOOFuWrVKmrUqJHqvGLFilGo\nUCEAWrduDcDhw4c599xzwxitG8wVvflz+fLlXHbZZZEMKdcoXLgwTzzxBAAdO3YM6Dkpe08pf2/e\nvDngvU/D0YOKRkePHuXHH39MdSwxMZHu3btn+rxevXrZPytVqhSq8HKFY8eOATBjxgx7bPLkyQB8\n9913nDp1Ckg9UrB06VIAPvjgg7DE6OwQ3/r16wFo0KABEydOBODuu+9O99y+ffsCMHbsWMC7X1Ck\nSJEwROlxdYgvKSmJxo0bAzB//nwAypcvn+3XXb58OcuXLwfggQceAAjpF6vrQygpvzi3bt0KwJAh\nQ+wx0xCd2UAZJt7Dhw9TrFixEEXp53o+Dxw4AHhDSNOnTwfgpZdeYtu2bTl+zYoVK7Jnz56gxHcm\n1/N5pqVLl7Jv3z4ARo4cyaZNmwB/JyAuLi7br2meGywa4hMRkaji7BBfyi5k27ZtIxhJ9DhziK9x\n48asXbsWgAsuuACAJk2a2Ekkt99+O7Vq1UrzOqa3NG3aNAAmTpzI8ePHAWjXrh0Q/ploLrn88svT\n/NylSxd77KOPPgLgrrvuSvf5JocFCxYMVYhOMT3706dPM3r0aAA2b95sHzejJfnz57dX92erZ8+e\nQXmdaLN48WIWL14MwPjx4wE4ePBg0CZMvfLKK0F5nUCpByUiIk5ytgdl3HHHHZmO0ycnJ5OYmAh4\nvQMI7f0Rl5mxbvNn8eLFbW7MPSiAhQsXAl4Py9wovffeewF49913bQ/M3MuqWLGiHXMuU6ZMqP8Z\nUa9z586At04qPYMGDQKgQIECYYspEn766SfAPynk5MmT2Xq+6aVXq1YN8O6fpPTNN98A8Nhjj6V5\nbnojA7nN7t27ef311wH/aMfu3buzfX+oQ4cONG3aFID7778f8N6j5p6+ccMNN9h70OHibANVpUoV\nwLuBar5k8+XLl+a8uLg43nnnHQB69+4N5P4PfkbOHOJLqVWrVml+fvLJJzlx4kSq8ypUqGBnqZkZ\nk0uXLqV///4hiTk3eeuttwB/w3Tm/4MZ3jIXUrndqFGjAOxssIyYWbgFCxa0OSpVqhT/+Mc/AO9C\n60ybNm1i9erVqY7FxMQwYMAAAP75z3+eXfBRoEmTJuzduzfL87p27cozzzwDeGvzzlSkSBF7UR8b\nGwvAhx9+mOa8Ll26hH1YWkN8IiLiJGd7UA8++GBA55npkwB33nlnqMKJCmZIxFx5Hj582F69pjfs\nacpGpfTcc8+l+9rnn38+QJ5cXxaoI0eOZPhYoUKFaNCgQRijibyPP/4YgFtvvRXwr7s50zXXXAP4\nJ/IEYuPGjfb1jXLlytlRFFNdJjd6+umnAdLtPd199922t2SW2pQqVcquJU3PiRMn7C0AM/wcGxtL\nixYtABg8eDAAV155ZZD+BYFTD0pERJzkbA8qUObqC7z7J+LvSd533332KutspoVPmTIlKHHlZvHx\n8XZ6eXouueQSW8svrzGTRoLh999/B1JP+jFuuummqK0/mR1mosKePXto1qwZANdffz3gLcRP7159\nZg4ePGiXPhiFCxe2EzAi2fN3roFKucr+THFxcbz33nsA/PLLL4C3xsLcjN6/fz/gNVSFCxcOR7hO\nMiVekpOT7QSSjIbuAhGqFfm5gXmfjhw50r4n02MmDEjOJSUl0adPHwDmzZtnj9euXRvwcpzeJIDc\nxny+zTqnnDIXrynXmZoh/FmzZjkxJK0hPhERcZITPSgz1Xnbtm289tprgLce50zJyclppu7GxMRQ\nsmRJwH+VOmrUKHtTPy8ya0Ay2g4iUKZG186dO20hXknN7EX2wgsv2GMp94u66aabgNTVJyR7du/e\nDXgjJKYyCvh7EnfccQdAWOtvRjPzfWsmW6Qswr1kyRLAnUoxEW2gPvnkEwA762Tjxo32MbPpYI8e\nPVI9x1Te3bJlC+DN3DEbFZr9dvI600DNnTv3rBYs/vXXX4CX65wUmc3tYmNjbfXslBcDKddBne1F\ngvjLnpkZZsaXX34JQP369cMeUzTbuXMngN1bD/wzLLN7/yrUNMQnIiJOimgPqn379oC/asT69evp\n1KkT4F+3k3L9TlJSEn/88Qfg70EtXrxYPacMpKwekRNmBX/FihW55JJLghFSrjJ79mxbmDMjZ+5f\nJp5Dhw7ZCSampFZKiYmJ9ko/5ciKUaVKFfu9IYHbv38/9913H+Av0luvXj2GDx8OuLd+TD0oERFx\nkrMbFqZnzJgxPPTQQwDcdtttgLe1QUZFOcPFlQ0LQ+Xaa6+1P2fVYwiGaNkQrmbNmrZHn5KJp1ix\nYvz6668AEZ2042I+//vf/2ZaOeKZZ57h2WefTXUsX7589n5Uhw4dIna/xMV8ZiUhIQGA119/3d7L\nM1PKR4wYYTd9jZSMcurELL6smB03+/XrZ286Dx06FMi4YrRIqJg9izLaMdeYO3dunp5NmpmMGiez\n5m7ChAnpPt6wYUPAvZv5LktISLAXloMGDbINk1nQH+nGKTP6dhcRESc534NKSkqya0nA644C1KlT\nJ1Ih5Rlm/57Y2Nh0tzzIq1atWgVkXBzWrIMyV/sSmN27d9utSPbs2WOXNgwcOBCAPn365JldiIPB\nfH4ffvhhuxVM69at7XfoxRdfHLHYAuV8A7VmzRrWrFkDeCWMzIZaGtoLPVMpfu3atbZCumS+7xbo\nvZlTTZs2TVVWq0uXLgD83//9X6RCikpbt24F/A37zJkz7Yzejz76KKoWNOuTJCIiTnK+B/Xf//7X\n/vzWW29RokSJCEaTt5hJAMnJyVx33XURjsYdZuJDwYIF7ewoyT7TW+rWrZv93ZQvmj17tipE5EB8\nfLwtuzVz5kzA20X37bffBqKvHJR6UCIi4iTne1AVK1a0e56kLAsvoTd9+nTAu9eiShJ+5v3YoEED\nVqxYkeZx08tXHb7MjRkzBoCvvvrKHrv99tsB736UZN8ll1ySatQJYPXq1fztb3+LUERnx/kGqkWL\nFnzzzTeRDiNPS05OPuuySbnR3Llzad68OQCbN2+2x3/88UcAihYtGpG4okWxYsWA1BuNdu/ePULR\n5A5jx46lTZs2gL+C/t69e6O2LJSG+ERExElRVerIVbm11NHgwYMBb1uD9IayQiUaS8m4TPkMLuUz\n+DLKqRqoIMitDVSk6AsguJTP4FI+gy+jnGqIT0REnJTlJAnNRAou5TP4lNPgUj6DS/nMuUyH+ERE\nRCJFQ3wiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuIkNVAiIuKk\nTEsd+Xw+lZkIUKDFYsMRS26QnWKcoY4lN1A+g0v5DL70cpplLT6VQspadmptKZ9Zy27tMuU0c8pn\ncCmfwZdRTjXEJyIiTlIDJSIiTlIDJSIiTlIDJSIiTlIDJSIiTlIDJSIiTlIDJSIiTspyHZTkLsnJ\nyYwbNw6A3r172+PTpk0DoFOnThGJS0TkTOpBiYiIk9SDymPeffdd+vbtC0BMjP/6ZPHixYB6UNnx\nxx9/8J///AeAQ4cOceeddwLQpEkTAM477zx69eplzz/nHO/jdt1114U5Urdt27YNgBo1alCyZEnA\ny2egtm7dCsC///1vOnToAEChQoWCHGX0SUpKAuDUqVNMnz4dgL/++ss+vnDhQgAWLFhgj91zzz0A\n1KtXjwsvvBDwv19Tfl+Eiy+zMhw+ny852GU6Dh8+DMDXX3+d7uN79+4FoF+/foCX5PQSY5IfExND\nxYoVAbj88ssB6Ny5MzfffDMQnjeqz+cLuBZfpMqe/PnnnwA0b96cTZs2pXn8oosuAmDZsmUULVo0\nrLGdKdB8/u/coOV0zJgxALRv3x6Ae++9l8TExAzP37RpEzt27Aj49QsWLAjAvn37AChWrFhOQ82W\nSOUzUKaBuuCCC2zJmz59+tjPcJUqVQCoVKkSx44dA2D9+vWcOnUKgI4dOwJw/PhxypYtC8DKlSup\nXLlySOJ1PZ/x8fEAvPTSSwA8++yz6Z5n4sqqdNOkSZMA6Nq1a5AiTCujnGqIT0REnBT2HtS6desA\naNy4cUDnJycnp9vCZ9b6Jycnc+DAAQBKly6d01AD5nIPyvSc/vGPfwCwceNG6tWrB3hXRE8//TQA\nJ0+eBGDXrl22RxopkbhC3bJlCy1atABgz549Z/16mbniiisA+OKLL8LSi3L9ij9lDyo9ZtivVKlS\nnDhxAsj6/2jr1q1UrVo1iFH6uZzPXbt20bx5c8AbggaoWbMm5557rj1nxIgRAOTPn9/EaB/bvHkz\nkHoC1ZNPPglk3BMLhoxyGvZ7UIULFwa84Y2jR4+medwMg9SuXRuA06dPky9fvjTnnT59GoBjx46x\nffv2UIUb1Q4cOGC/dDdu3Ah4wyRDhgwBvKEsM9S6aNEiAOLi4sIfqAOqV69OnTp1gNRffubLsXv3\n7mf1+pMmTbLD28uWLQO8i4avvvoKgDJlypzV6+dmJm+HDx+mSJEigJe7b7/9Ns257dq1A4j4RVak\nTJgwwTZMXbp0AWD8+PEB3+ooV66c/dm8980960jQEJ+IiDgp7D0o0zOaPn06N910U4aPr169Gsj6\nBt6RI0e4/vrrAVi1alUwQ41aphfUokUL23MyV1CLFy+2OU7Pq6++yujRo0MfpGNiYmIYOXIk4E2O\nAJg6daodGjE36nPq+uuvp1WrVqmO/fzzz6xcuRKAG2644axeP5q9//77aY716dPH9gBSKlGiBODN\nkCxVqlSaxx9//HHAPxKT1zz22GPceuutgP+7NJDek5lw8txzz9ljDz74IJC6VxVu6kGJiIiTIrYO\n6tJLL033+C+//ALAZ599Bvin/GbkxIkTxMbGpjluelN5cc2JWW/TsmVLLr74YgA7GSKz3lNeZ96T\npvceTDVq1Aj6a+YWZuJUSvXr1+eyyy7L8DmffvppmmMFChSgZs2aQY0t2hQsWJAGDRpk6zmJiYkM\nHjwYgJkzZwJerynlRIlIiVgDVbJkSSZMmAD4F4eBf21ItWrVAnqdChUq2IWRZgYKwNq1a4G82UAV\nKFAAgDfeeCPCkUhmrr76alq2bBnpMCLOrHeaM2eOHY66+uqr0z3XrIN6+eWX0zw2YMAAe2NfAjdp\n0qQ0+Zw6daoTE000xCciIk6KWA8qX758dOvWDfAPrTzxxBO2B5XREGCgUvbKRCLFrNt5/vnn0zyW\nP3/+dJdQ5DW33XYb4FWRMaWKMloTdf/99wOwYsUKe8yMkphhKgmMKRE1aNAge8wM65m1VJGmHpSI\niDgposVizRTy+vXrAzB37tygvbZ5rbvvvjtor5kX9OjRI9Ih5CozZswA4MMPP4xwJO4y941MxYKM\nbN68mVmzZtnfTY1Os/Dc3HuVwDRs2BDwahiaReqvvPIK4J9oFWluRHEWDh8+zG+//Qb4yx8lJSXR\npk2bSIYlwokTJ+jTp0+a4+bL4L333gt3SFHJTIx49NFH7ZBpoUKF7B5mpki0BKZnz56AP68lSpRg\n3rx5gDsNk6EhPhERcZJbzWUO/PHHH6xZswbwDxlGYt+S3GLixIl5spJEqKS3Rq9z586AVxdRsjZ7\n9mwAPv/8c3usQoUK3HLLLZEKKWp9+umndnmPGXGaPHky1atXj2RYGdI3uYiIOCnqe1Aikjs98sgj\ngFeN2zATIQYMGBCRmKKVqc85Z84cO9LUv39/AKfv16uBklTSu6kvOXPm2icz9JzdUjR5UVJSEuPG\njQP8e5WBf5bZAw88EJG4olFCQoKdsbdlyxZb/NXk0mUa4hMRESflih7UmTtWJiUlRSiS6Ge2M5Cc\nM5tpnrnrq9kCom3btmGPKVokJiYC0KlTp1Q9J4C6devSsWPHSIQVlRISEgCvOsSWLVsAbyPH5cuX\nRzKsbFEPSkREnBT1PajHH388zaaGkyZNUk8gh8aMGZNu3TgJ3IYNGwDvfZiSrv6z9sILLwD+7XYA\n6tWrB8DSpUspXbp0ROKKRqaAQcoF4S+99BLnn39+pELKtqhtoL788ksAfvjhB3vMlIdXeaOcO3Lk\nSKRDiHqbNm1Kc6xUqVJ2Vpqk78iRI+neuDefdTVO2WPWO4F/+yJTmDdaaIhPREScFLU9qFatWgGk\nGt7TBn1n7+eff05V70yyb+DAgWmOtWjRwg5VSWrmZv6QIUNS9eBHjRoFQPny5SMSV7TasWMH4FWI\nMBYtWgRE32c6ahsoM1NPZY2C69///jdDhw4FYMSIERGORvKC3bt3A2kvMPv27QugPbOyyeyOe/To\nUcArTlyhQoVIhpRj+nYXEREnRW0PyvSczpzBJ9ljhgHMnjqrVq3iqaeeimBEudOwYcMiHUJUMTtr\nS/bExcWlmjgG3t54RYoUiVBEZ0c9KBERcVLU9qBSMiv0y5QpE+FIoo+5AT127NgIR5J7PPzwwwD0\n69fPbq1hpvlK5sxN/NWrV+veUw4cPHiQ1atXA141DsDZrTQC4TuzTFCqB32+5Mwej6SZM2cCMHXq\nVHsztUWLFhGJxefzkZycnOVYo8v5dEmg+fzfucppFpTP4FI+gy+jnGqIT0REnJRlDyqMsUS1QHtQ\n4YglN8jOFWqoY8kNlM/gUj6DL72cZtpAiYiIRIqG+ERExElqoERExElqoERExElqoERExElqoERE\nxEn/D4pZX87LFMIKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x109d0c198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "for i in range(25):\n",
    "    img = X_train[y_train == 7][i].reshape(28, 28)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/mnist_7.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the following lines to optionally save the data in CSV format. \n",
    "However, note that those CSV files will take up a substantial amount of storage space:\n",
    "\n",
    "- train_img.csv 1.1 GB (gigabytes)\n",
    "- train_labels.csv 1.4 MB (megabytes)\n",
    "- test_img.csv 187.0 MB\n",
    "- test_labels 144 KB (kilobytes)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#np.savetxt('train_img.csv', X_train, fmt='%i', delimiter=',')\n",
    "#np.savetxt('train_labels.csv', y_train, fmt='%i', delimiter=',')\n",
    "X_train = np.genfromtxt('train_img.csv', dtype=int, delimiter=',')\n",
    "y_train = np.genfromtxt('train_labels.csv', dtype=int, delimiter=',')\n",
    "\n",
    "#np.savetxt('test_img.csv', X_test, fmt='%i', delimiter=',')\n",
    "#np.savetxt('test_labels.csv', y_test, fmt='%i', delimiter=',')\n",
    "X_test = np.genfromtxt('test_img.csv', dtype=int, delimiter=',')\n",
    "y_test = np.genfromtxt('test_labels.csv', dtype=int, delimiter=',')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing a multi-layer perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.special import expit\n",
    "import sys\n",
    "\n",
    "\n",
    "class NeuralNetMLP(object):\n",
    "    \"\"\" Feedforward neural network / Multi-layer perceptron classifier.\n",
    "\n",
    "    Parameters\n",
    "    ------------\n",
    "    n_output : int\n",
    "      Number of output units, should be equal to the\n",
    "      number of unique class labels.\n",
    "\n",
    "    n_features : int\n",
    "      Number of features (dimensions) in the target dataset.\n",
    "      Should be equal to the number of columns in the X array.\n",
    "\n",
    "    n_hidden : int (default: 30)\n",
    "      Number of hidden units.\n",
    "\n",
    "    l1 : float (default: 0.0)\n",
    "      Lambda value for L1-regularization.\n",
    "      No regularization if l1=0.0 (default)\n",
    "\n",
    "    l2 : float (default: 0.0)\n",
    "      Lambda value for L2-regularization.\n",
    "      No regularization if l2=0.0 (default)\n",
    "\n",
    "    epochs : int (default: 500)\n",
    "      Number of passes over the training set.\n",
    "\n",
    "    eta : float (default: 0.001)\n",
    "      Learning rate.\n",
    "\n",
    "    alpha : float (default: 0.0)\n",
    "      Momentum constant. Factor multiplied with the\n",
    "      gradient of the previous epoch t-1 to improve\n",
    "      learning speed\n",
    "      w(t) := w(t) - (grad(t) + alpha*grad(t-1))\n",
    "    \n",
    "    decrease_const : float (default: 0.0)\n",
    "      Decrease constant. Shrinks the learning rate\n",
    "      after each epoch via eta / (1 + epoch*decrease_const)\n",
    "\n",
    "    shuffle : bool (default: False)\n",
    "      Shuffles training data every epoch if True to prevent circles.\n",
    "\n",
    "    minibatches : int (default: 1)\n",
    "      Divides training data into k minibatches for efficiency.\n",
    "      Normal gradient descent learning if k=1 (default).\n",
    "\n",
    "    random_state : int (default: None)\n",
    "      Set random state for shuffling and initializing the weights.\n",
    "\n",
    "    Attributes\n",
    "    -----------\n",
    "    cost_ : list\n",
    "      Sum of squared errors after each epoch.\n",
    "\n",
    "    \"\"\"\n",
    "    def __init__(self, n_output, n_features, n_hidden=30,\n",
    "                 l1=0.0, l2=0.0, epochs=500, eta=0.001, \n",
    "                 alpha=0.0, decrease_const=0.0, shuffle=True, \n",
    "                 minibatches=1, random_state=None):\n",
    "\n",
    "        np.random.seed(random_state)\n",
    "        self.n_output = n_output\n",
    "        self.n_features = n_features\n",
    "        self.n_hidden = n_hidden\n",
    "        self.w1, self.w2 = self._initialize_weights()\n",
    "        self.l1 = l1\n",
    "        self.l2 = l2\n",
    "        self.epochs = epochs\n",
    "        self.eta = eta\n",
    "        self.alpha = alpha\n",
    "        self.decrease_const = decrease_const\n",
    "        self.shuffle = shuffle\n",
    "        self.minibatches = minibatches\n",
    "\n",
    "    def _encode_labels(self, y, k):\n",
    "        \"\"\"Encode labels into one-hot representation\n",
    "\n",
    "        Parameters\n",
    "        ------------\n",
    "        y : array, shape = [n_samples]\n",
    "            Target values.\n",
    "\n",
    "        Returns\n",
    "        -----------\n",
    "        onehot : array, shape = (n_labels, n_samples)\n",
    "\n",
    "        \"\"\"\n",
    "        onehot = np.zeros((k, y.shape[0]))\n",
    "        for idx, val in enumerate(y):\n",
    "            onehot[val, idx] = 1.0\n",
    "        return onehot\n",
    "\n",
    "    def _initialize_weights(self):\n",
    "        \"\"\"Initialize weights with small random numbers.\"\"\"\n",
    "        w1 = np.random.uniform(-1.0, 1.0, size=self.n_hidden*(self.n_features + 1))\n",
    "        w1 = w1.reshape(self.n_hidden, self.n_features + 1)\n",
    "        w2 = np.random.uniform(-1.0, 1.0, size=self.n_output*(self.n_hidden + 1))\n",
    "        w2 = w2.reshape(self.n_output, self.n_hidden + 1)\n",
    "        return w1, w2\n",
    "\n",
    "    def _sigmoid(self, z):\n",
    "        \"\"\"Compute logistic function (sigmoid)\n",
    "\n",
    "        Uses scipy.special.expit to avoid overflow\n",
    "        error for very small input values z.\n",
    "\n",
    "        \"\"\"\n",
    "        # return 1.0 / (1.0 + np.exp(-z))\n",
    "        return expit(z)\n",
    "\n",
    "    def _sigmoid_gradient(self, z):\n",
    "        \"\"\"Compute gradient of the logistic function\"\"\"\n",
    "        sg = self._sigmoid(z)\n",
    "        return sg * (1 - sg)\n",
    "\n",
    "    def _add_bias_unit(self, X, how='column'):\n",
    "        \"\"\"Add bias unit (column or row of 1s) to array at index 0\"\"\"\n",
    "        if how == 'column':\n",
    "            X_new = np.ones((X.shape[0], X.shape[1]+1))\n",
    "            X_new[:, 1:] = X\n",
    "        elif how == 'row':\n",
    "            X_new = np.ones((X.shape[0]+1, X.shape[1]))\n",
    "            X_new[1:, :] = X\n",
    "        else:\n",
    "            raise AttributeError('`how` must be `column` or `row`')\n",
    "        return X_new\n",
    "\n",
    "    def _feedforward(self, X, w1, w2):\n",
    "        \"\"\"Compute feedforward step\n",
    "\n",
    "        Parameters\n",
    "        -----------\n",
    "        X : array, shape = [n_samples, n_features]\n",
    "          Input layer with original features.\n",
    "\n",
    "        w1 : array, shape = [n_hidden_units, n_features]\n",
    "          Weight matrix for input layer -> hidden layer.\n",
    "\n",
    "        w2 : array, shape = [n_output_units, n_hidden_units]\n",
    "          Weight matrix for hidden layer -> output layer.\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        a1 : array, shape = [n_samples, n_features+1]\n",
    "          Input values with bias unit.\n",
    "\n",
    "        z2 : array, shape = [n_hidden, n_samples]\n",
    "          Net input of hidden layer.\n",
    "\n",
    "        a2 : array, shape = [n_hidden+1, n_samples]\n",
    "          Activation of hidden layer.\n",
    "\n",
    "        z3 : array, shape = [n_output_units, n_samples]\n",
    "          Net input of output layer.\n",
    "\n",
    "        a3 : array, shape = [n_output_units, n_samples]\n",
    "          Activation of output layer.\n",
    "\n",
    "        \"\"\"\n",
    "        a1 = self._add_bias_unit(X, how='column')\n",
    "        z2 = w1.dot(a1.T)\n",
    "        a2 = self._sigmoid(z2)\n",
    "        a2 = self._add_bias_unit(a2, how='row')\n",
    "        z3 = w2.dot(a2)\n",
    "        a3 = self._sigmoid(z3)\n",
    "        return a1, z2, a2, z3, a3\n",
    "\n",
    "    def _L2_reg(self, lambda_, w1, w2):\n",
    "        \"\"\"Compute L2-regularization cost\"\"\"\n",
    "        return (lambda_/2.0) * (np.sum(w1[:, 1:] ** 2) + np.sum(w2[:, 1:] ** 2))\n",
    "\n",
    "    def _L1_reg(self, lambda_, w1, w2):\n",
    "        \"\"\"Compute L1-regularization cost\"\"\"\n",
    "        return (lambda_/2.0) * (np.abs(w1[:, 1:]).sum() + np.abs(w2[:, 1:]).sum())\n",
    "\n",
    "    def _get_cost(self, y_enc, output, w1, w2):\n",
    "        \"\"\"Compute cost function.\n",
    "\n",
    "        y_enc : array, shape = (n_labels, n_samples)\n",
    "          one-hot encoded class labels.\n",
    "\n",
    "        output : array, shape = [n_output_units, n_samples]\n",
    "          Activation of the output layer (feedforward)\n",
    "\n",
    "        w1 : array, shape = [n_hidden_units, n_features]\n",
    "          Weight matrix for input layer -> hidden layer.\n",
    "\n",
    "        w2 : array, shape = [n_output_units, n_hidden_units]\n",
    "          Weight matrix for hidden layer -> output layer.\n",
    "\n",
    "        Returns\n",
    "        ---------\n",
    "        cost : float\n",
    "          Regularized cost.\n",
    "\n",
    "        \"\"\"\n",
    "        term1 = -y_enc * (np.log(output))\n",
    "        term2 = (1 - y_enc) * np.log(1 - output)\n",
    "        cost = np.sum(term1 - term2)\n",
    "        L1_term = self._L1_reg(self.l1, w1, w2)\n",
    "        L2_term = self._L2_reg(self.l2, w1, w2)\n",
    "        cost = cost + L1_term + L2_term\n",
    "        return cost\n",
    "\n",
    "    def _get_gradient(self, a1, a2, a3, z2, y_enc, w1, w2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "\n",
    "        Parameters\n",
    "        ------------\n",
    "        a1 : array, shape = [n_samples, n_features+1]\n",
    "          Input values with bias unit.\n",
    "\n",
    "        a2 : array, shape = [n_hidden+1, n_samples]\n",
    "          Activation of hidden layer.\n",
    "\n",
    "        a3 : array, shape = [n_output_units, n_samples]\n",
    "          Activation of output layer.\n",
    "\n",
    "        z2 : array, shape = [n_hidden, n_samples]\n",
    "          Net input of hidden layer.\n",
    "\n",
    "        y_enc : array, shape = (n_labels, n_samples)\n",
    "          one-hot encoded class labels.\n",
    "\n",
    "        w1 : array, shape = [n_hidden_units, n_features]\n",
    "          Weight matrix for input layer -> hidden layer.\n",
    "\n",
    "        w2 : array, shape = [n_output_units, n_hidden_units]\n",
    "          Weight matrix for hidden layer -> output layer.\n",
    "\n",
    "        Returns\n",
    "        ---------\n",
    "\n",
    "        grad1 : array, shape = [n_hidden_units, n_features]\n",
    "          Gradient of the weight matrix w1.\n",
    "\n",
    "        grad2 : array, shape = [n_output_units, n_hidden_units]\n",
    "            Gradient of the weight matrix w2.\n",
    "\n",
    "        \"\"\"\n",
    "        # backpropagation\n",
    "        sigma3 = a3 - y_enc\n",
    "        z2 = self._add_bias_unit(z2, how='row')\n",
    "        sigma2 = w2.T.dot(sigma3) * self._sigmoid_gradient(z2)\n",
    "        sigma2 = sigma2[1:, :]\n",
    "        grad1 = sigma2.dot(a1)\n",
    "        grad2 = sigma3.dot(a2.T)\n",
    "\n",
    "        # regularize\n",
    "        grad1[:, 1:] += (w1[:, 1:] * (self.l1 + self.l2))\n",
    "        grad2[:, 1:] += (w2[:, 1:] * (self.l1 + self.l2))\n",
    "\n",
    "        return grad1, grad2\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict class labels\n",
    "\n",
    "        Parameters\n",
    "        -----------\n",
    "        X : array, shape = [n_samples, n_features]\n",
    "          Input layer with original features.\n",
    "\n",
    "        Returns:\n",
    "        ----------\n",
    "        y_pred : array, shape = [n_samples]\n",
    "          Predicted class labels.\n",
    "\n",
    "        \"\"\"\n",
    "        if len(X.shape) != 2:\n",
    "            raise AttributeError('X must be a [n_samples, n_features] array.\\n'\n",
    "                                 'Use X[:,None] for 1-feature classification,'\n",
    "                                 '\\nor X[[i]] for 1-sample classification')\n",
    "\n",
    "        a1, z2, a2, z3, a3 = self._feedforward(X, self.w1, self.w2)\n",
    "        y_pred = np.argmax(z3, axis=0)\n",
    "        return y_pred\n",
    "\n",
    "    def fit(self, X, y, print_progress=False):\n",
    "        \"\"\" Learn weights from training data.\n",
    "\n",
    "        Parameters\n",
    "        -----------\n",
    "        X : array, shape = [n_samples, n_features]\n",
    "          Input layer with original features.\n",
    "\n",
    "        y : array, shape = [n_samples]\n",
    "          Target class labels.\n",
    "\n",
    "        print_progress : bool (default: False)\n",
    "          Prints progress as the number of epochs\n",
    "          to stderr.\n",
    "\n",
    "        Returns:\n",
    "        ----------\n",
    "        self\n",
    "\n",
    "        \"\"\"\n",
    "        self.cost_ = []\n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        y_enc = self._encode_labels(y, self.n_output)\n",
    "\n",
    "        delta_w1_prev = np.zeros(self.w1.shape)\n",
    "        delta_w2_prev = np.zeros(self.w2.shape)\n",
    "\n",
    "        for i in range(self.epochs):\n",
    "            \n",
    "            # adaptive learning rate\n",
    "            self.eta /= (1 + self.decrease_const*i)\n",
    "\n",
    "            if print_progress:\n",
    "                sys.stderr.write('\\rEpoch: %d/%d' % (i+1, self.epochs))\n",
    "                sys.stderr.flush()\n",
    "\n",
    "            if self.shuffle:\n",
    "                idx = np.random.permutation(y_data.shape[0])\n",
    "                X_data, y_data = X_data[idx], y_data[idx]\n",
    "\n",
    "            mini = np.array_split(range(y_data.shape[0]), self.minibatches)\n",
    "            for idx in mini:\n",
    "\n",
    "                # feedforward\n",
    "                a1, z2, a2, z3, a3 = self._feedforward(X[idx], self.w1, self.w2)\n",
    "                cost = self._get_cost(y_enc=y_enc[:, idx],\n",
    "                                      output=a3,\n",
    "                                      w1=self.w1,\n",
    "                                      w2=self.w2)\n",
    "                self.cost_.append(cost)\n",
    "\n",
    "                # compute gradient via backpropagation\n",
    "                grad1, grad2 = self._get_gradient(a1=a1, a2=a2,\n",
    "                                                  a3=a3, z2=z2,\n",
    "                                                  y_enc=y_enc[:, idx],\n",
    "                                                  w1=self.w1,\n",
    "                                                  w2=self.w2)\n",
    "\n",
    "                delta_w1, delta_w2 = self.eta * grad1, self.eta * grad2\n",
    "                self.w1 -= (delta_w1 + (self.alpha * delta_w1_prev))\n",
    "                self.w2 -= (delta_w2 + (self.alpha * delta_w2_prev))\n",
    "                delta_w1_prev, delta_w2_prev = delta_w1, delta_w2\n",
    "\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training an artificial neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn = NeuralNetMLP(n_output=10, \n",
    "                  n_features=X_train.shape[1], \n",
    "                  n_hidden=50, \n",
    "                  l2=0.1, \n",
    "                  l1=0.0, \n",
    "                  epochs=1000, \n",
    "                  eta=0.001,\n",
    "                  alpha=0.001,\n",
    "                  decrease_const=0.00001,\n",
    "                  minibatches=50, \n",
    "                  random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 1000/1000"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<__main__.NeuralNetMLP at 0x109d527b8>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn.fit(X_train, y_train, print_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNW9//H3lx1FJYiyK6ioYFS8RDAxUTSKRvNTExM1\nJsYYbhKjcX1uFI33kcQsGHNjjAYSr1s0iRHNFVFRcRuXRMUFBEEEoqjsuLC4sM18f3+cartnpmeY\nYbq6qqs/r+fpp6tOVVd9+zDMd07VqXPM3REREUmjdkkHICIi0hQlKRERSS0lKRERSS0lKRERSS0l\nKRERSS0lKRERSa3YkpSZDTCzx81sjpm9YmbnRuU9zOxhM5tvZtPMrHvBZy4xswVmNs/MRheUDzez\n2dG2a+KKWURE0iXOltQm4AJ33wc4CDjbzIYAY4GH3X1P4NFoHTMbCpwMDAWOBiaYmUXHmgiMcffB\nwGAzOzrGuEVEJCViS1LuvtzdZ0bLHwCvAv2A44A/R7v9GTghWj4euN3dN7n7ImAhMNLM+gDbufv0\naL9bCz4jIiIZVpZ7UmY2EDgAeA7o5e4rok0rgF7Rcl9gccHHFhOSWsPyJVG5iIhkXOxJysy6Af8A\nznP3dYXbPIzJpHGZRESkqA5xHtzMOhIS1G3uPjkqXmFmvd19eXQpb2VUvgQYUPDx/oQW1JJoubB8\nSZFzKdmJiCTI3W3Le7VOnL37DLgRmOvuvyvYNAU4PVo+HZhcUH6KmXUys0HAYGC6uy8H1prZyOiY\npxV8pgH/5OWuV+51+eWXJx5DGl+qF9WN6qZ0r7jE2ZI6GPgWMMvMZkRllwDjgUlmNgZYBJwE4O5z\nzWwSMBfYDJzl+W9+FnAL0BWY6u4Pxhi3iIikRGxJyt2fpumW2hFNfOaXwC+LlL8I7Fu66EREpBJk\ndsSJ2tqkI0iPUaNGJR1CKqlemqa6aZrqprwszmuJ5RQ6TuS/y1tvwYABzXxARERKxszwSuo4ISIi\n0laZTVL33JN0BCIi0laZvdwHkJGvJiKSerrcJyIiVUdJSkREUktJSkREUktJSkREUitTSer73086\nAhERKaVMJakdd0w6AhERKaVMJamGrOSdIUVEpJwylaT0XJSISLZkKknV1SUdgYiIlFKmkpRaUiIi\n2ZKpJLXNNklHICIipZSpJLXLLo3Lnnmm/HGIiEhpZCpJFbvc9/rr5Y9DRERKI1NJSkREsiXzSWr+\n/KQjEBGRrZX5JPWznyUdgYiIbK1MJammuqBv3lzeOEREpDQylaQ+85ni5U8/Xd44RESkNDKVpIYN\nK16uh3xFRCpTppJUU5Yvh+efTzoKERFpLfOMNDPMzN296MjnXbvCxx+rRSUiEhczw91LPvdEVbSk\nPv446QhERGRrVEWSEhGRylR1Scodli1LOgoREWmJqkpS//43DBkCffsmHYmIiLREVSWpSZPgtdeS\njkJERFqqqpKUiIhUlqpKUuqCLiJSWaoqSRX63OeSjkBERLakKh7mbUpGvrqISOL0MK+IiFQdJSkR\nEUktJSkREUktJSkREUmtqk9SdXVJRyAiIk2p+iTVvr16+YmIpFXVJykREUmvqk5SuRaUWlIiIulU\n1Ukqp2fPpCMQEZFiqjpJXXFFeH///WTjEBGR4qp6WKRCGakGEZFEaFgkERGpOplLUhMmJB2BiIiU\nSuaS1N57b93nXnghv/z66zB1amniERGRrZe5JLW1zjgjv3zOOXDsscnFIiIiQaxJysxuMrMVZja7\noGycmS02sxnR60sF2y4xswVmNs/MRheUDzez2dG2a+KI9ZVXYPbsLe8nIiLlE3dL6mbg6AZlDvzW\n3Q+IXg8AmNlQ4GRgaPSZCWaf9NWbCIxx98HAYDNreMyS2G+/xmXPPKOefyIiSYk1Sbn7U0Cxp5CK\ndVM8Hrjd3Te5+yJgITDSzPoA27n79Gi/W4ETmj5n22JueIzPfS4kKhERKb+k7kmdY2Yvm9mNZtY9\nKusLLC7YZzHQr0j5kqi8bGpry3k2ERHJSSJJTQQGAcOAZcD/JBBDi+gyn4hIsjqU+4TuvjK3bGY3\nAPdGq0uAAQW79ie0oJZEy4XlS4ode9y4cbz+em5tVPQSEZFSq6mpoaamJvbzxD4skpkNBO51932j\n9T7uvixavgA40N1PjTpO/A0YQbic9wiwh7u7mT0HnAtMB+4Hfu/uDzY4j7s7jz4KRxyx9fHW1UG7\ndvWXn3wSvvCFrT+miEjWxTUsUqwtKTO7HTgU6GlmbwOXA6PMbBihl98bwA8A3H2umU0C5gKbgbM8\nn0HPAm4BugJTGyaoQnvu2baY2+nJMRGR1MjcALNhuTTHvOEG+M//VEtKRGRL4mpJKUk1o2dPeOcd\n2GMPWLCgNMcUEckiJaktiCNJFcpINYmIxEJTdYiISNXJZJKKoyX14oulP6aIiDQvk5f7Bg+GhQtL\nf46MVJWISMnpcl8rfPazSUcgIiKlkMkkdcstMGJEPMe+7jq1qEREyiWTSapdu3juS0GYEFFERMoj\nk0kK1NoREckCJSkREUmtzCapOMbgW7lyy/uIiEjpZDZJde1a+mPeeGPpjykiIk3LbJIaOrT0x5w1\nq/THFBGRpmXyYV6AzZuhY8d4zjVuHJx2GnTpAn37xnMOEZFKogFmt6Bhkgpl8Z6zf394++14zyEi\nUgk04kQKrV2bX3aH9euTi0VEJIuUpNqgsOF27bXxdNYQEalmSlJttHRpeJ8/P9k4RESyKNNJatWq\neI+/bh306wdz58KGDfGeS0SkGmU6SfXsCc8/H/95HnoIliyJ/zwiItUm07378tviP/8xx8DUqRqO\nSUSqk3r3pVxziXDmTPif/ylPshQRyRK1pErMPXRN37wZevQIZTvtBO+8k98uIpI1aklViFmz4NBD\nYY898mW5BCUiIq2jllQMuneH1avzrabC82ekukVE6lFLSkREqo6SVAz0zJSISGkoScXg44+b3rbP\nPmEUdRER2bKquCf1zjuhh125bdgAnTsX35aRahcRAXRPqk169kzmvLvvnsx5RUSyoipaUmF7GYNp\ngYxUu4gIoJZUm/3ud0lHUN/hh+eXN22CM85o3efvvBNGjChtTCIiaVM1LSl3aJeylHzddXD22WFw\n2v79m29dvfYa7LVXfv300+HWW9UiE5F0UEuqjdJ2uQ/gRz9qett3vwv3359f33tvqKuLPyYRkTTp\nkHQA1e4734GPPqpfdt99cPPNsGYNHHtsvjyNiVZEJE5Vc7kvvx+cdBJMmlSGoFqp4TBKX/lKGAPw\npJPgwANDSyq3rVOncC8rI/98IlLh4rrcV5UtqV12STqClrvqquITKm7aVP5YRETKrWruSRUaMiTp\nCFom10ravLn+eiEz3asSkeyquiS1eXPru3uX03vvNS4rTE7vvAMXX9z09oa+8AUlMRGpXFV3Tyq/\nf3gfPBgWLIgpqFZ66qmQVBoaMADefhtqa+GOO+DUU+tv37wZ2rcvfkwz2LgROnYsfbwiIjnqgh6T\niROTjiCvWIKCkKDaIiN/h4hIFaraJPXoo2FsvUobX2/9+tZ/RklKRCpV1V7uK5Trzp12O+8MK1cW\n3/a//ws1NfCTn+Q7huRG2Vi/vunR2EVESkGX+2I0bVrSEbRMUwkK4Hvfg7/+Fe66K9yHmjkz3IsC\ntaREpHIpSQH9+iUdQenkOoScfDLMmxeWly9PLh4RkbbYYpIys9taUibpMn8+DBsWljVauohUqpa0\npD5duGJmHYDh8YSTjO22SzqC0il2b+3dd8sfh4hIKTSZpMzsUjNbB+xrZutyL2AlMKVsEZZB795b\n12sujX72s8ZldXUwdWr9so8+ggsvLE9MIiJba4u9+8xsvLuPLVM8W60tvftypk+Hiy6CJ54oUVAp\n0qcPLF2aX582DY46Sp0qRKQ0kuzdd5+ZdYuCOM3Mfmtmu5Y6kDQYMSJ0477ppqQjidcTT4QEJSKS\ndi1JUhOBj8xsf+BC4HXg1pYc3MxuMrMVZja7oKyHmT1sZvPNbJqZdS/YdomZLTCzeWY2uqB8uJnN\njrZd0+JvJ58obDE115VdRCRNWpKkNrt7HXAC8Ad3vw5oaVeDm4GjG5SNBR529z2BR6N1zGwocDIw\nNPrMBLNPpvmbCIxx98HAYDNreMySOuwwOPhg+NrXwth+WdDUZb2hQ8sbh4hIa7QkSa0zs0uBbxEu\n/bUHWjRcqbs/BbzfoPg44M/R8p8JyQ/geOB2d9/k7ouAhcBIM+sDbOfu06P9bi34TCwGDoSnn4Y7\n7wzLWfDOO8XLX321vHGIiLRGS5LUycAG4LvuvhzoB1zVhnP2cvcV0fIKoFe03BdYXLDf4uhcDcuX\nROVlkZWOBbW1cNll4WHf669POhoRkZbZ4sy87r7MzP4KHGhmXwamu3uL7km14NhuZiVLA+PGjftk\nedSoUYwaNarNx9xvvzAYbRaS1S9+Ed4feaTxtltuCfNsTZsGa9fCiSfCjBlwwAFlDVFEKkRNTQ01\nNTWxn6clXdBPIrScch2zDwF+7O53tugEZgOBe91932h9HjDK3ZdHl/Ied/e9zWwsgLuPj/Z7ELgc\neDPaZ0hU/g3gUHc/s8F52twFvZi6uvCAbJcuJT90asybB3vvXb/sgw+gWzfYsCEMwCsi0pwku6Bf\nBhzo7t92928DBwL/3YZzTgFOj5ZPByYXlJ9iZp3MbBAwmNBqWw6sNbORUUeK0wo+E7t27cII4hdd\nVK4zlt/q1Y3LdtopvO+7b3ljEREp1JIkZcCqgvV3o7Itf9DsduBfwF5m9raZnQGMB440s/nA4dE6\n7j4XmATMBR4AzipoGp0F3AAsABa6+4MtOX8pXXFFuc+YrI8/Du/z5+fLamrg3nsTCUdEqlRLLvdd\nBewP/I2QnE4GZrl7qtoWcV3uq3+OWA+fmP33h5dfbnp7rlp79gzjAGbh/pyIlFZcl/uaTFJmNpjQ\nE+9pMzsRODjatBr4m7svLHUwbaEkFZ9cte68M6xapSQlIo0lkaTuBy5x91kNyvcDfuHu/6/UwbSF\nklR8ctWa+/7FqnnmzNAiq9Y6Eql2SXSc6NUwQQFEZYNKHUil6NYt6QjSZ+nS0FV9xozQG/Dxx5OO\nSESyorkk1b2ZbRnukN28akxSCxeGZ8WacuON4X3jxtDB5PDDyxOXiGRfc0nqBTP7fsNCM/se8GJ8\nIaXfSSeF98WLm98vKw44AI44Ir++eDHcdlvx7//ee43LNm3KT2XfErW1rY9RRLKpuREnzgfuNrNv\nkk9Kw4HOwFfiDiyNnn8ett0WBg2Cjh2hX79weatz56Qji9cHH9RfHzAgv/zcc/nlK6+Evn0bf37i\nRDjvvJZ3uOjQIbTKOrZohEgRybImk1Q0IsTngMMIU8g7cJ+7P1au4NLmM5/JL//lL+G92kdjmDED\nNm8Oy5Mnw1lnNd5n3brWH7e2VklKRLYwdl/UXe6x6CXSyJln1l/P9e5bujQk8J49yx+TiGRHS0ac\nEGmxP/whvPfrF+blKvTaay0/jp7FEhFQkiqJ1nQKqCYN57B6663SHPd3vwsD/4pI9ilJlcBee4V3\nzdNU3/Ll4X3ZsvBuFlpIDzwAs2c33r/w3lVzLakLLgjDM4lI9ilJldCOO4ZBWK8qmBLy2msTCyc1\ncpf5zj4bvvlNOOYY+Pa3G++3/fb5Luy63Cci0IIBZitFOYZFav78cM89cNxx+XWAl14KXar32y+x\n0BLl3vRQScuWQe/e+XUzWLIk3M/64IPQ3b8YM1ixIowlKCLpkOR8UtICTzwBxx6bXz/qqDAn07Bh\n1T0nU3Nj+Y0c2bhsxYr4YhGRyqMkVSKHHALt2+fXH3wQVq7M/5IePjy8b7NN+WNLq7feyo8uMXBg\neD/++PB+yinNfzYjFwBEZAuUpMrkhRfg5pvDqBUbN8IzzyQdUTqsWhVG7XjzzbD+9tvh/b77Gu/7\nj3/kk9OaNS07/urVGpldpJLpnlSC9Msz3Jfq06f4tob/nGYhme26a1i///7QCaM5b7wBu+2mlpdI\n3HRPSjKpuSRTVwfTpuWXof7gs7p/JZJ9SlKSqBkzmt72y1+GDig77AAnnBDKFi3Kb1dLVCT7lKQk\ntf77v8P72rVw772Nt+eS1EUX5Qf8hdB9fePG+vvU1MQWpojESEkqQbvtBkOGbP3nx48vXSyV4sMP\nG5dddVX+AeqZM2G77Ro/LDxnTvyxiUjpqeNEwtauDZezirngArj66uLbunQJg7mOGRNfbJXgpZfg\nP/4jLNfVQbvoz66BA0O3/09/Gn7601BWgT8eIhUjro4TSlIJc8//Yi109dXwgx+E3m+77954+4kn\nhk4H1Z6kWqMCfzxEKoZ692VU7p7JiSfCddfVL+/aNVwSnDMHbrghmfhERJKkJJUCv/pVuPnfpUtY\nP+EEOPro/PahQ/PPBuXkRhSXttl1V42oLpJmSlIpMHYsjBgB++8f1u++Oz/9R06PHvnlM8+E732v\n+GVCadr994f3P/4xP6LFW2/B7bfn91myBD7+uPyxiUhxuidVQcxCMnvuubC+YQOMG5fv5Xf55flO\nAgB77AELF5Y9zFTbZZf85Itr14bpQXK++EV49FH44Q9hwoRk4hOpVLonJUBodeV07gyHHx6W3UPC\ngvAQLIQZbAEuu6xs4aVe4ezAhQkKQoKCMJ4ghES1yy6wfn391pV7fo4sEYmXklQF+eij/MgLOXvu\nWX/0dQjzV0GYOsQdrriiPPFlzWOPhQFv99oLBgzI/xHwz3/C3nsnGppI1VCSqiBduzYeCmjXXWHz\n5vz6okX171/l3HcfzJoVa3iZk2t1vfVW6FyRu5Sqe1Yi5aMklTENewHmHHts48kXP/wQvvWt+GOq\nNHfdBf37h2lVGho/HkaPDstmcOihxY9x/PHw/vvxxShSLZSkMqhhz8Bi3MMEjLvtli8bPx523DG+\nuCrJkiXFyy+5pP76k08W32/KFHjlldLGJFKNlKQy6POfb/kzVBdckF+++OIwX9OUKWH9Zz8rfWxZ\n9MUvhgktRaT01AW9ytxxR+i6XjgAa+4+V2H1TZ0KBx2kllVrFNafGTzxBBxySHLxiJSTuqBLSZx8\ncuMRwgF69qy/fswxoQPGjBlh/EDZsnXrwgPDuWT11FNhypAFC2DlylC2Zo3mwRJpDbWkBLPQlb25\nZ39yv1i7dQvzNUnrHHQQPPMMPPBA+AOgqR/VuXNDL85Bg8obn0hbqSUlsXnzzaY7ADT0+OPxxpJV\na9aEJPWNb4T1p54qvt8++4ThsTZsKF9sImmmlpS0SOF9K12uarvrrw/jLzZUWLf6cZZKopaUJOrz\nn4eHH046iuz4/vfDw9Xnnx8ewC52309/DIioJSVb4ZxzwtxXalXFb+HC4pNeiqSNWlKSGtde2/hS\nVG5cOymtH/0o6QhEkqWWlLTJ+PFhFIYNG8Ko7KAJGUvt61+HSZOgrg5+//vQuvrud2Hp0vAc2xtv\nwCmn1J9fbP368O9hFpY7dMgPPCwSh7haUkpS0ia1teGXX+GlvzFj4MYbk40r6zp1Cs9g9euXH8Jp\nyhQ48sjw75Cb5XnhQhg5MpQXTu4oUmpKUlugJJW8yZNh9eow6GpuTMBevWDFimTjqjZ9+4ZWFoQO\nGtdfD5/6VCjLJa8nn4TFi+HUU5OLU7JFSWoLlKTS5cUXYfjwMP7f5ZeHMQI/+gj+9KekI6teP/xh\nmAjz/fehd+9Qpv8yUirqOCEVZfjw8J77Jfjb38LPf15/n4YTOObkZhaW0po4MdynyiWoQgsXlj8e\nkZZQkpJY5TpTQOPxASdMgO98p/Fniv0SlXg88EDoWDF4MMyenXQ0Io3pcp/Eav16mDMn37IqfK7q\n/fdDj7QddsiX/elP4Ub/sGHljVPguefCQ8VHHAHbbpt0NFJpdLlPKlKXLvkEBWEa+69+Fdauhe7d\nYfvt88MDLV0abvTvv3/oNdhwgkGJ18iR4RLsVVeFPy42bICxY3XfSpKVWEvKzBYBa4FaYJO7jzCz\nHsAdwK7AIuAkd18d7X8J8N1o/3PdfVqD46klVaHmzw+zCRf75/v73+Hcc2HVqvLHVe1OPBH+8Y+w\n/PHH4Q8Os3A5dtmy8G+y007JxijpkcWWlAOj3P0Adx8RlY0FHnb3PYFHo3XMbChwMjAUOBqYYGZq\nBWZEw3tVhU45JT8XE4RLUXfcAdOnxx9XtcslKAjTh1x2WVhevjx0hNl55zDtyJVXhulbDjqoZcdt\n3x5uu6308Uo2JdmSegP4jLu/W1A2DzjU3VeYWW+gxt33jlpRde5+ZbTfg8A4d3+24LNqSWVYsdmD\nx4yBm25KJh6p7+CD4Z//hDvvhK99LZRdfDH8+tfhMuKzz+b3LfZvKZUvqy2pR8zsBTPLTVrQy91z\nj36uAHpFy32BxQWfXQz0K0+YkhYnnVR//YgjkolDGvvnP8P7178ektCYMSFBQeiQMWcOHHZY/bEI\n164NsxZv3Jifp+zZZ8MAxiI5Sbak+rj7MjPbCXgYOAeY4u6fKtjnPXfvYWbXAs+6+1+j8huAqe7+\nfwX7qiWVYWvXhu7shV3aIawfdRR85SvwzW+G9f/6L/jNb8L2Ll1CJ4BPfxpeeaX8cUvL5VpjUL+V\n9f77YcQMSbe4WlKJDTnp7sui91VmdjcwAlhhZr3dfbmZ9QFydyOWAAMKPt4/KqtnXMFQ3KNGjWLU\nqFHxBC9lt/32xcsbzmC7dGm4x3X11eGv+smTQ/ns2aETRm5mXEmfXIIC2LQJOnYMyz16wEMPwejR\nycQlxdXU1FBTUxP7eRJpSZnZNkB7d19nZtsC04CfAkcA77r7lWY2Fuju7mOjjhN/IySyfsAjwB6F\nTSe1pKTQBx+EQVjPOw/WrYO//CWUa/6ryrFwIWy3XRj/8brr4Oyzk45ImpOpsfvMbBBwd7TaAfir\nu/8q6oI+CdiFxl3QLyV0Qd8MnOfuDzU4ppKUbNGXvxwGwP3xj/MJ6zvfgVtuSTIqaam1a8OAuS++\nCH/7W9LRSKFMJak4KElJa/XvDz/5CRx4IFx4ITz11Jb3X7y4+X2kfEaPhmnTQrLae2/Yb7/Qvb3Q\nxo3hEYb+/ZOJsZooSW2BkpS01ezZ4Rddod/8JnTEAN3ArwTnnw9DhoSZopcty5f/5jfwpS/BmWfm\n/xh57z39e5aSktQWKElJKRx6KPz0p6G7dO7H6a23wgOrf/hD6Kjx0kuw556hM8e770KfPsnGLKX3\n73+H7vGrV4dHH+bPD2NMLlsGBxwQ9sn9fOg+Z6AktQVKUlIq69aFHoINew425d//hj32yN/bKhxO\nSCRnm23CwL3dusHAgeEPoW23DY9NdOwYLlV27hxG9+jUKcx4nSvv1Cksd+gQ1nOv3D4dOjTe1r59\nGMC52CsOSlJboCQlSVm/Pvxief310F26c2c4/HD48EOYNSvp6ETKRUmqWUpSkiabNoXLQXV1oUW2\nww75y0K77BIuIUK4R/LHPyYXp0jpZG9YJJHM6tgxXKLp0iU/X5Z7GE387ujhi/vuC9O5L14cLgHl\nfPWr5Y9XJK3UkhIps9raMAp4w1mJcy2tWbPCQK1Dh2qEDKkkutzXLCUpqXRm8PDD+YFz3WHFinCj\ne8kSOPLIcBmxRw9YtAgGDIC33040ZJECutwnkmnu9Ud2z00wuPPOodvzokWhC/Rrr8HEifDmm/mB\ndO+/P5GQRWKnlpRIBaurC12N3fW8jiRNLSkRaaBdu/xDpS+9FEZVyOnSJb9cVxfezz8/vEQqhVpS\nIhn2wQcheR1ySHhdc024dJhrdY0fH0ZU2G23ZOOULFDHiWYpSYm0XG1tGKFgzhzYa68waeS99xbf\nd889w7BAIs3T5T4RKZH27cMEkUOHhuUpU8I4hPvsE0bQgNBN3h0mTEg2VqluakmJSCPTpsGoUeGB\nZAjJ6sMP4dVXYeTI+tO7iwRqSYlImYwenU9QEO5hdesW5t7KDem0dm2YNTfnmGMaH2f77eONU7JP\nSUpEWqV/f3jllTC1+/LlcPXV8PTT4VmtTZvCA8jf+lYYk3DNGnjkkTC5JITR4gE++9n88d59t/zf\nQSqHLveJSOLuuQeOOy7f67CuLrw+/BB22il09Pj1r/MTUEoaqXdfs5SkRKrDRx/Bv/4FZ58NF14Y\nOn8cckjSUYmS1BYoSYlUt8KZctesge7dw/ruu4eJKSVuSlLNUpISkdZYvjxMVnnrrfDEE6Hr/aWX\nwsEHJx1ZpVKSapaSlIjEYfPmcE+sU6fQSqutDSN5/N//wV13wdSpSUeYFkpSzVKSEpFqVFsbEumm\nTfmEWlsb1jdsCGW5jii5bbmy3OzRuW2bN+cvm9bW1v8MNN4/Vw5w6qlKUs1SkhIRSY6ZHuYVEZEq\noyQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQl\nIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKp\npSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKppSQlIiKpVTFJysyONrN5ZrbAzC5OOh4REYlf\nRSQpM2sPXAccDQwFvmFmQ5KNqnLU1NQkHUIqqV6aprppmuqmvCoiSQEjgIXuvsjdNwF/B45POKaK\nof9Uxalemqa6aZrqprwqJUn1A94uWF8clYmISIZVSpLypAMQEZHyM/f0//43s4OAce5+dLR+CVDn\n7lcW7JP+LyIikmHubqU+ZqUkqQ7Aa8AXgaXAdOAb7v5qooGJiEisOiQdQEu4+2Yz+xHwENAeuFEJ\nSkQk+yqiJSUiItWpUjpONKsaHvQ1s5vMbIWZzS4o62FmD5vZfDObZmbdC7ZdEtXHPDMbXVA+3Mxm\nR9uuKSjvbGZ3ROXPmtmu5ft2W8/MBpjZ42Y2x8xeMbNzo3LVjVkXM3vOzGaa2Vwz+1VUXvV1A+H5\nSzObYWb3RuuqF8DMFpnZrKhupkdlydWNu1f0i3D5byEwEOgIzASGJB1XDN/zC8ABwOyCsl8DF0XL\nFwPjo+UOXTAmAAAFyUlEQVShUT10jOplIflW83RgRLQ8FTg6Wj4LmBAtnwz8Penv3MJ66Q0Mi5a7\nEe5dDlHdfFI/20TvHYBngc+rbj6pmwuBvwJTonXVS4j3DaBHg7LE6ibxCilBhX4WeLBgfSwwNum4\nYvquA6mfpOYBvaLl3sC8aPkS4OKC/R4EDgL6AK8WlJ8C/LFgn5HRcgdgVdLfdyvraDJwhOqmUb1s\nAzwP7KO6cYD+wCPAYcC9UVnV10sU7xvAjg3KEqubLFzuq+YHfXu5+4poeQXQK1ruS6iHnFydNCxf\nQr6uPqlHd98MrDGzHjHFHQszG0hobT6H6gYAM2tnZjMJdfC4u89BdQNwNfBjoK6gTPUSOPCImb1g\nZt+LyhKrm4ro3bcF6vkBuLtX87NiZtYN+AdwnruvM8s/rlHNdePudcAwM9sBeMjMDmuwverqxsy+\nDKx09xlmNqrYPtVYLwUOdvdlZrYT8LCZzSvcWO66yUJLagkwoGB9APUzeJatMLPeAGbWB1gZlTes\nk/6EOlkSLTcsz31ml+hYHYAd3P29+EIvHTPrSEhQt7n75KhYdVPA3dcA9wPDUd18DjjOzN4AbgcO\nN7PbUL0A4O7LovdVwN2EsVMTq5ssJKkXgMFmNtDMOhFuxE1JOKZymQKcHi2fTrgfkys/xcw6mdkg\nYDAw3d2XA2vNbKSFpsZpwD1FjvU14NFyfIG2ir7HjcBcd/9dwSbVjVnPXC8sM+sKHAnMoMrrxt0v\ndfcB7j6IcK/kMXc/jSqvFwAz28bMtouWtwVGA7NJsm6SvklXoht9XyL06loIXJJ0PDF9x9sJo21s\nJFzPPQPoQbj5Ox+YBnQv2P/SqD7mAUcVlA+PfugWAr8vKO8MTAIWEHqBDUz6O7ewXj5PuK8wk/AL\neAZhShfVDewLvBTVzSzgx1F51ddNQfyHku/dV/X1AgyKfl5mAq/kfp8mWTd6mFdERFIrC5f7REQk\no5SkREQktZSkREQktZSkREQktZSkREQktZSkREQktZSkRIows9poqoLc66ISHnugFUy5spXHuLyZ\nbYWxTy4oH2Rh6o4FZvb3aKQOkVTTc1IiRZjZOnffLqZjDySMvL3vVnz2F4QBdI8kPMR8k7u/3GCf\norGb2STgLnefZGYTgZfd/Y9b8RVEykYtKZFWiCaEuzKaFO45M9s9Kh9oZo+Z2ctm9oiZDYjKe5nZ\n3RYmHpxpZgdFh2pvZtdbmKjxITPrEu1/roUJHF82s9sbnt/df0IYYeWbwHUNE1QzcRthWoq7oqI/\nAye0pS5EykFJSqS4rg0u9309KndgtbvvB1wH5MYLvBa42d33J0yk9/uo/PeEKTKGAf8BzI3KBxOS\nzKeB1cCJUfnFhEkc9wd+0DAoM7sCeAD4C/AjM9uvSOxdzOxFM3vGzI6PynaM4s5NTVE4dYJIauly\nn0gRzVwyewM4zN0XRfd0lrl7TzNbBfR299qofKm772RmK4F+7r6p4BgDgWnuvme0fhHQ0d1/YWYP\nAB8QBvCc7O4fNhHf5e7+0ya29fEw1cIg4DHgcGAd8Iy7D472GQBM3ZpLjiLlpJaUSNsU/pVnTexT\nrHxDwXIt+bndjgX+QGh1PW9m7YuetIkEFW3LTbXwBlBDmAjyXaC7meX+z/cntKZEUk1JSqT1Ti54\n/1e0/C/CtA8Q7hc9GS0/CvwQwMzam9n2TR00um+0i7vXAGOBHYBtWxOYmXU3s87Rck/gYMI0Jg48\nDuQuWxZOtyCSWlmYmVckDl3NbEbB+gPufmm0/CkzexlYD3wjKjsHuNnMfkyYEO6MqPw84HozG0No\nMZ1JmH674XV2B9oDt1mYRdeAa9x9bSvjHgL8yczqCH+E/srdczOrXgz83cx+TpjC48ZWHluk7HRP\nSqQVontSw71CZlkVqXS63CfSOvqrTqSM1JISEZHUUktKRERSS0lKRERSS0lKRERSS0lKRERSS0lK\nRERSS0lKRERS6/8D1eIjaZLpzHsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x109c374e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot(range(len(nn.cost_)), nn.cost_)\n",
    "plt.ylim([0, 2000])\n",
    "plt.ylabel('Cost')\n",
    "plt.xlabel('Epochs * 50')\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/cost.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "batches = np.array_split(range(len(nn.cost_)), 1000)\n",
    "cost_ary = np.array(nn.cost_)\n",
    "cost_avgs = [np.mean(cost_ary[i]) for i in batches]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xn8VHW9x/HXh32RNVkFxIUQl1AwTM2kFNPM5VqBpqVm\n5XbTFhfw3pS2q3hvZaa5pYZe18I9NTTE5RrihqCAgUgKsiiKgAvyg8/943N+zjD+fvADZubMnHk/\nH495zJnvOTPznaPw5nvOdzF3R0REpBI1S7sCIiIijVFIiYhIxVJIiYhIxVJIiYhIxVJIiYhIxVJI\niYhIxSpZSJlZXzN7xMxeMrMXzeyMpLyrmT1kZv80s4lm1jnvPWPMbI6ZzTazg/LKh5rZjGTf70pV\nZxERqSylbEmtAX7k7rsAnwNON7NBwGjgIXf/NPD35DVmtjMwCtgZOBj4g5lZ8llXACe5+wBggJkd\nXMJ6i4hIhShZSLn7YneflmyvAmYB2wCHA+OTw8YDRybbRwC3uPsad58PzAX2MrNeQAd3n5ocd0Pe\ne0REJMPKck/KzPoDewBPAT3cfUmyawnQI9nuDSzIe9sCItQKyxcm5SIiknElDykz2wqYAJzp7ivz\n93nMyaR5mUREpEEtSvnhZtaSCKgb3f2upHiJmfV098XJpbylSflCoG/e2/sQLaiFyXZ++cIGvkth\nJyJSQdzdNn7UhpWyd58B1wIz3f2SvF33AMcn28cDd+WVH21mrcxsO2AAMNXdFwMrzGyv5DO/lfee\n9fitt+IjR+LueuQ9LrjggtTrUMkPnR+dG52b4j+KpZQtqX2B44DpZvZ8UjYGuAi43cxOAuYDIwHc\nfaaZ3Q7MBOqA0zz3S08D/gS0Be539wcb/MY2beDDD0vyY0REpPxKFlLu/gSNt9QObOQ9/wX8VwPl\nzwK7bfRLFVIiIpmSrRknFFINGj58eNpVqGg6P43TuWmczk15WDGvHabJzNynTIEzzoCnnkq7OiIi\nNc3M8EruOJEKtaRERDJFISUiIhUrWyHVtq1CSkQkQ7IVUmpJiYhkSvZC6oMP0q6FiIgUSfZCSi0p\nEZHMyFZItWwJdXWwdm3aNRERkSLIVkiZRWtq9eq0ayIiIkWQrZACXfITEckQhZSIiFSs7IWUxkqJ\niGRG9kJKLSkRkczIXkh16ADLl6ddCxERKYLshVT//vDqq2nXQkREiiB7IbXDDvDKK2nXQkREiiB7\nIdW9O7z1Vtq1EBGRIsheSLVvD++9l3YtRESkCLIXUu3awfvvp10LEREpguyFlFpSIiKZkb2QUktK\nRCQzshdSakmJiGRG9kJKLSkRkczIXki1b6+QEhHJiOyF1FZbwTvvgHvaNRERkS2UvZDq2RM6doQX\nX0y7JiIisoWyF1JmsOuuMG9e2jUREZEtlL2QAujcWTOhi4hkQDZDqkuXuC8lIiJVTSElIiIVSyEl\nIiIVK5sh1bmzQkpEJAOyGVJqSYmIZIJCSkREKlZ2Q0pd0EVEql42Q6pzZ5g7F555Ju2aiIjIFjDP\nyBx3ZuYf/5b334+JZkFz+ImIpMDMcHfb0s/JZkuqXTu4+27Yfvu0ayIiIlsgmy0pgFWroHv3WADR\ntjjMRURkE6gltTFbbQWtW8Pbb6ddExER2UzZDSmAvn1hwYK0ayEiIpsp2yHVpw+8/nratRARkc2U\n7ZAaOBAmTUq7FiIispmy23ECYPFi2GkneO21WK1XRETKQh0nmqJnz3gsXJh2TUREZDNkO6RA8/iJ\niFSx7IdUu3Zwxx1p10JERDZDtu9JxY54zsjvFBGpBronJSIimVfSkDKz68xsiZnNyCsba2YLzOz5\n5HFI3r4xZjbHzGab2UF55UPNbEay73ebVIlf/CKe1ZISEak6Jb3cZ2b7AauAG9x9t6TsAmClu/+m\n4NidgZuBzwLbAA8DA9zdzWwq8O/uPtXM7gcudfcHC97f8OU+iCmS3nhD3dBFRMqkKi73ufvjQENd\n6xqq+BHALe6+xt3nA3OBvcysF9DB3acmx90AHLlJFVEPPxGRqpTWPakfmNkLZnatmXVOynoD+RPt\nLSBaVIXlC5PypuveHRYt2oLqiohIGtIIqSuA7YDdgUXAr0v+jXvuCU89VfKvERGR4mpR7i9096X1\n22b2R+De5OVCoG/eoX2IFtTCZDu/vMEpJMaOHfvx9vDhwxk+fHi8GDQI5s2L6ZGmT4evfnVLf4aI\niOSZPHkykydPLvrnlnyclJn1B+7N6zjRy90XJds/Aj7r7t/M6zgxjFzHiR2TjhNPAWcAU4G/sqkd\nJ667Dp54IpaVv+029fQTESmxYnWcKGlLysxuAfYHtjaz14ELgOFmtjvgwKvAyQDuPtPMbgdmAnXA\naXmpcxrwJ6AtcH9hQG1Ux46wYoVW6BURqTLZn3ECYOJE+PKXYxHE119XS0pEpMSqogt6xagfH1W/\nAOLy5bBqVXr1ERGRJqmNkNpqq/Vfd+kChx6aTl1ERKTJaiOkOnX6ZNn06eWvh4iIbJLaCKm+fWHK\nlPXLmjdPpy4iItJktRFSAJ/6VG77uONinSkREalotRlSAwZoLj8RkSpQG13Q602fHmOl+vSBbt3g\ngw+gZcvyVFBEpIYUqwt6bYVUvm7d4KWXYvJZEREpKo2T2lJvvQVjxqRdCxER2YDaDalLLonBvRMm\npF0TERFpRO1e7ps7NzpQADz5JOy9d2kqJiJSg3S5b0vtuCOMGwcHHADPPZd2bUREpAFlX0+qopxz\nTvT2e+WVtGsiIiINqN2WVL2ddtIUSSIiFap270nVW7kSevWK3n5t2hS/YiIiNUj3pIqlQwfYbjuY\nOTPtmoiISAGFFMDgwXDXXbBoUdo1ERGRPAopiJD6xS/goIPSromIiORRSAF85jPxrElnRUQqikIK\nYNCgeLYG7vHtvntu2XkRESkrhRTErOgACxZEJ4p8L7wA06aVv04iIqKQAqBZM3j22diePx+uvhq2\n3z63f926VKolIlLranvGiXxDhsTlPnc4+eQoe+yxeF6zJr16iYjUMLWk8l1++fqvTzopnu++u/x1\nERERzTjRwAfF8wUXwEUXwerV8Xrt2rgsKCIiG6UZJ0rlww/jkt9JJ+UCCuJelYiIlJVCqlDr1vHc\ntWs8jx4Nhx2mHn4iIilQSDWmXbt4PuqoGCulkBIRKTuFVGPMYMSIWMpDISUikgqF1IZMnBizpO++\ne2713vHjoxOFiIiUnEKqKbbbLjpRTJ0KJ5ygaZJERMpEIdUUZrDnnnDNNfF68eJ06yMiUiMUUk3V\np09umXmFlIhIWSikmqpXL3jxxdjW4ogiImWhkGqqHj3g/fehZUu1pEREykTTIjXVggXQty80bx69\n+5YtiwG/ixbBpz4FrVqV7rtFRKqMpkUqtz59okv6z38er8eMiYlne/eGP/0p1aqJiGSVQmpTjBgB\nhx4a21dfDUceGdtr18Z8f7NmpVc3EZEMUkhtqsGD4bLL1i9bsQIeeQR23jmdOomIZJRCanN06bL+\n65Ur414VaBVfEZEiUkhtjsKQWrEiN1XSm2+Wvz4iIhmlkNocXbpEV/R6v/89LF0a2wsXplMnEZEM\nUkhtjj59YK+94Kyz4DOfibJnn43nZcuiE0XhJLRPPFHeOoqIZIDGSW2pZctilvTtt4fHHoNbbonB\nvj/6UYQVwDvvxJiqceNgv/1g773LX08RkTIq1jgphVQxXHopnHlm7nW/fvDaaxFS558Pu+wCRx8d\n+775TbjppnTqKSJSJmUbzGtmNzalrKYdccT6r197LZ5Hj4Zf/AK+//3cPjPYddfy1U1EpIq1aMIx\n6/2NamYtgKGlqU6V2nbbhsuffjqeV6zIlT3yCLzxRunrJCKSAY22pMzsPDNbCexmZivrH8BS4J6y\n1bBaPP88zJgBl1+eK2vf/pPHvftuPGt1XxGRjdroPSkzu8jdR5epPpst1XtS+ZYtgwsvjM4UF10E\n//oXrFr1yeMWL46Z1UVEMqhsHSfM7PPANHdfZWbfAvYAfufu/9rSLy+migmpekuWQM+eudejRsWs\nFDffHK+ffjpW+xURyaByzoJ+BfC+mQ0GfgzMA25oyoeb2XVmtsTMZuSVdTWzh8zsn2Y20cw65+0b\nY2ZzzGy2mR2UVz7UzGYk+37X5F+Xph494OWXo0s6xLiqAw6I7VGj4IEH4Kc/Ta9+IiJVoCktqefd\nfQ8zuwBY6O5/NLPn3H3IRj/cbD9gFXCDu++WlF0MvOXuF5vZuUAXdx9tZjsDNwOfBbYBHgYGuLub\n2VTg3919qpndD1zq7g8WfFdltaTyzZsX3dLXrIGZM6PzxNlnx75166LHn4hIhpSzJbXSzM4DjgPu\nM7PmQMuNvAcAd38ceKeg+HBgfLI9HkjWu+AI4BZ3X+Pu84G5wF5m1gvo4O5Tk+NuyHtPddh+e2jR\nAtq2haFD4Sc/iZYVxGq/TeGeGxwsIlIjmhJSo4DVwHfcfTHRyvnvLfjOHu6+JNleAtT3HugNLMg7\nbkHyXYXlC5Py6mUG994b2/W9/TamSxf45S9LVycRkQq00XFS7r7IzG4CPmtmXwWmunuT7kk14bPd\nzIrWPBg7duzH28OHD2f48OHF+uji69YNPv3pGEPVu/fGj3/3XXjuudLXS0RkM0yePJnJkycX/XOb\nck9qJNFyejQp+gJwtrv/uUlfYNYfuDfvntRsYLi7L04u5T3i7juZ2WgAd78oOe5B4ALgX8kxg5Ly\nY4D93f2Ugu+p3HtSjRk2LDpYrFoVIXTffTHH34QJ0YX9y1+O2SuaNYvW19FH5zpiiIhUsGLdk2rK\njBP/CXzW3ZcmX9wN+DvQpJBqwD3A8cC45PmuvPKbzew3xOW8AUSrzc1shZntBUwFvgVcupnfXVk6\ndoxgqjd7NixaBMcdB9/7Xiz7MW5crgXVqtX677/tNthjj2iRiYhkUFPuSRmQv5LfsqRs4280uwV4\nEhhoZq+b2YnARcAIM/sn8KXkNe4+E7gdmAk8AJyW1zQ6DfgjMAeYW9izr2rtuOP6r994I0KqWTO4\n5pooO+88+MtfYnvlytyxU6ZEy+r888tTVxGRFDTlct9/A4OJ7uFGdKSY7u7nlL56TVeVl/vefjum\nUzrwwHg9ciRMmwbHHAN33gnz58c9q0sugc9/Hk44IaZe+te/oH//eM/3vw9XXZXSDxARaVjJL/eZ\n2QCiJ97ZZvY1YN9k15NEYMmW6to1N8B3l13g9ttju08feOGFWO23R4/oer7rrjB3Lnz4YS6gAFq3\nLnu1RUTKZUOX+y4BVgC4+wR3/7G7/5i4h/TbclSuZsycCZMm5V7Xh1D37jHo96ijIoy6d4f77499\np54K//u/MGsW3LiBlVPeeQdefbVkVRcRKaVGL/eZ2TPu3uDkcmb2ortX1KJIVXm5r9ATT8R8fm3a\nNLw/f2YK9wi2+pbYySfDlVd+8j3HHAO33qqBwCJSVuWYcaLzBvY18reobJHPf77xgMo3Z04851/2\na+y+lKZcEpEqtqGQesbMvl9YaGbfA54tXZWkUZMmRceK+l6Bffuuv78+kNatg7feiu36buvLl5en\njiIiRbShy309gTuBj8iF0lCgNfBv7r6oLDVsokxc7tscDz8MI0bkXj/2WExoe/HFMHBgjL2aNQue\neSbmDRQRKYOyrCdlZgZ8kVhC3oGX3H1So29IUc2GFGz8kt4OO8CvfhVLhIiIlEFZZpxI/taflDyk\nUj3xBCxYEIN7G7LfftF9vdCjj8Ylw+23L239REQ200YH81aLmm5J1XOHa6+N0OnUCY49NhZevPJK\nOOWUGHfVrVvueDPYe2948sn06iwimVS25eOrhUKqAV/5SqwAvHRpjLGCGJM1aFBsm8HgwdEZQ0Sk\niMq56KFUq4MOghNPjNbTfvtF2d/+FhPXfuMb8Tp/0UV3qKsrfz1FRBqhllSteOMNOO00uPvumHZp\nQd46klddFXMA/vKX8fjww/TqKSKZoJaUbJrevSOkYP2A6tEjd0/quedg9WrYbbfc/g8/1GwVIpIa\nhVQtOeAAuOKK9csuvTQWXQRo3jyeX3wRfvvbmE6pbVsttCgiqWnKooeSFc2bR4+/++6Dli3hrrtg\n221jmqXVq+Hxx3PHPv00vPRSbOevYyUiUkYKqVrToUOE1AcfwJtvxmXARYvgz3+Oe1VTpsQ6Vmee\nGeE0alT0DhQRSYEu99Wqtm2hXz9o0QL23Rd+/evoAdi/f6xttWABLFkSXdQXL8697847YyFGEZEy\nUEgJDBkSY6Xqu6lvvXWsGvz++7D77jH3X72jjoLrr0+nniJSc3S5TyKkIJYKgRjku/XW8bzPPnF/\nas6cuFQI0Z1dRKQMFFISUyN997u5WSkAnnoqQqpTp7g/ddllcMghsU8r/YpImWgwr2zcyy/DTjvF\no0uXmLFi3rxcl3URkQIazCvlM3AgDBgQa1MddRS89lp0uDCLmdQB7rgDvvOd2P7mN+HUU9Orr4hk\nhlpS0jS33grHHBMT1P7kJzFxbT13OP54uOGG2K5f30r/PURqllpSUl6f+xwcdljMoH7jjTHQd+ed\nY9911+WCSQN/RaSI1JKSzbduXe6+VPfuMej36aejIwbEvH/1+++/H1q3jqmZRCTz1JKS9DVrBo88\nEttLl8KwYbEUyJAhEVotWsBf/hIzWxx6KBx4YNzDqp8rUERkI9SSki3329/C889HaI0fD+eeGy2q\nSZNyx7RoAW3a5I7p2BF23DFmvRCRzFFLSirHj34UnSbGjYvX3/52BFC+ujr40pdiSqV//CMu+40f\n3/hnnn46vPJK6eosIlVBLSkprrq6aDW9+WZMp7T//rDrrtHJ4vDDo9X17LNxbLt2MHYsnH127v2r\nV8d9rJYt4eKL198nIlWjWC0pzTghxdUi+V+qW7d4PPdcTFLbLGm09+wZ96a+/vW4X3XOOfDFL8Ke\ne8b4qltuiWNA0y+JiC73SYntsUcuoCAGBUOMuYK4f3XTTbFdv7ji4sXRrX3WLJg+HbbZBt59d/3P\nPfHEaHWJSKbpcp+Ulzs8/DCMGBGvFyyAvn1h+HCYPDlmXZ82LS4JDh2ae9+kSdHighiL1bEjPPRQ\n3NuyLb6iICJFpo4TUp3McgEFsdAiREAB/PCH8J//CdtvH69POinGXT30ELz3XpTNnRvPI0bEwOKm\nuPvuWOhRRKqKWlKSvmnToqPE3LlwxBGf3P+HP0RvP4CrropWVP3lwiOPjIUY8736aizemN/CMoOb\nb869T0RKSi0pyY7dd4/VgBsKKIAvfCHubQGcfHIEzZAhcN998Nhj63ewuPLKaIUddljMiJFP/4gR\nqToKKal8u+4avQTdo1UFMGpUzGIxYgRMnBhhdf318Pjjsf+vf41ygI8+iuc1a8pfdxHZIuqCLtXl\n1FPh2GNhq63i9aGHxuDhhjz6KBx8cO5+19tvl6WKIlI8CimpPh075raPPTZ6B7ZtGy2pq66CCRMi\noJ54Av78Zxg5Mo5dsiSd+orIZlPHCckO95jotkePmMS2Q4co32cfOPPMCLC//z3dOorUCHWcEClk\nFgEFcTmwXTs47jh48MG47DdvHlxxRQwWPvTQmEPQPcZgLVuWbt1FpEFqSUl2rV4NrVrluqLPnRtz\nCLZokRszdfnl0b39/PNjNozly+Hee2PJERHZbMVqSSmkpLaMHw/XXgtXXx2ztz/4YCwhsvXWEWjz\n5sVxo0bBrbfGrO3bbAO33x6zYrRtm2r1RaqFQqqAQko2y5FHRlj9+tfRVf3ss2NKpp/+FGbOhGee\ngaOPzq08vGBBhBbEe7797ZhIV0TWo5AqoJCSolm9OsLrwQfj9QUXwM9+ltt//vnwk59Ap04xhdMR\nR8Qs7iLyMYVUAYWUFJU7XHNNzHAxY0Y83n47Lg1+97sx+W39ulj1x4vIxxRSBRRSUhLLl0PnzuuX\nPfAA/OAHsR5W/XRNQ4bAbrvBn/5U9iqKVCKFVAGFlJTVunWxTpZ7XAqsvxx4zjnRxf3002NC2xNO\niLkJV6+Oe1v1k+GuXRvPzZunUn2RUlNIFVBISaruuWf9CXK7d4e33oJhw2JWjF/9KsLrzTejJ2GP\nHvCNb8Bll6VXZ5ES0mBekUpy+OG5RRkhevzdfDNMmRKT3y5eHOUHHBAT3y5dCi+/nE5dRapIai0p\nM5sPrADWAmvcfZiZdQVuA7YF5gMj3X15cvwY4DvJ8We4+8SCz1NLStK1dGlcArz+evja16B37+hc\nse++sf/rX495BXfZJZYXad8evve9CLQvfQkGDvzkZ86YEc+77Va+3yFSBFV/uc/MXgWGuvvbeWUX\nA2+5+8Vmdi7Qxd1Hm9nOwM3AZ4FtgIeBT7v7urz3KqSksi1dGp0tTj8dLrwQxoxZf/9JJ8H//E90\n1PjZzyKgJkyA1q3hww/TqbPIZipWSKU9C3rhDzgc2D/ZHg9MBkYDRwC3uPsaYL6ZzQWGAVPKVE+R\nLde9O5x2GhxyCGy7bbSeunWLAcMjR8ZMGN26xXF/+AMcf3y8b/XqeLRunW79RVKQZktqHvAucfnu\nKne/xszecfcuyX4D3nb3Lmb2e2CKu9+U7Psj8IC7T8j7PLWkpLq98EIMCq6rg0suiZnb163L9QA8\n4QQ48cRYBLJr1yhbty6mbirsJi+Ssix0nNjX3fcADgFON7P98ncmibOh1FEiSbYMHhwDiF9/PQIK\n4h7Xf/xHBNPEibD//vCpT0XPwPnz4xJhly5w223Qrx/ccUe0ukQyIrXLfe6+KHl+08zuJC7fLTGz\nnu6+2Mx6AUuTwxcCffPe3icpW8/YsWM/3h4+fDjDhw8vTeVFSuWEEz5Z9stfxsMdTjkFDjssAmvQ\noGhRDRsWY7B69YqOGKefDj//eUyWe/rpMHYs7LVXuX+J1JjJkyczuX4V7CJK5XKfmbUDmrv7SjNr\nD0wEfgYcCCxz93FmNhroXNBxYhi5jhM75l/f0+U+qTlr10ZLa9myWFrk2GPhuediyqZBgyLUZs+O\nY5s1y82Q0b9/qtWW2lDVvfvMbDsgGXpPC+Amd78w6YJ+O9CPT3ZBP4/ogl4HnOnufyv4TIWUyLp1\nMQFumzYxk/spp8T8g3vuGaHWoUN00nj//RhYfOaZ8NWvwjvvxCKRa9bEgpEiW6iqQ6oUFFIiBerq\nYoHHfLNmwQ9/GMH03nvwm9/EPa76FteAATHI2Lb47xapcQqpAgopkc3w+ONw7rlw1lkwaRI89BD0\n7AkrV8ZaWSNHRkeMf/wDvvIV9SKUJlNIFVBIiRTBX/8Ko0dHr8Hx42HatBi31a9fDCiePDkuJU6Y\nEDNp7LBD2jWWCqWQKqCQEimB2bNj/NbIkXF/67rr4jIiRG/CCy+ERx+Ne1uDB6dbV6koCqkCCimR\nMpg9O2Zwb98+gunKK6O8S5cYwzV0KNx6a4TZ0KHRq3DJEpgzJ3oV9u27wY+X7FBIFVBIiaSgri5m\nvHj5Zfi//4twOvxwGDcu9vfrB6+9ljt+4cLoVbhkSXTqaNMG9tknnbpLSSmkCiikRCrI/PnRurr/\n/hhUvMsucMMNsehj167Qpw9Mnx5BddZZMZ9h27aw3XbR2/D116PVpV6GVUshVUAhJVLh3KMF1bNn\nvK6ri96FDz0Ev/89rFoV5e3axTiugQNjOqizzooZ4du1i9k2OnZM7zdIkymkCiikRKqcO3z0UW5g\n8W23wYMPxnyEX/hCHPPYY3DUUXGJ8Gtfi8HHO+6oFlcFUkgVUEiJZJx79DR89tkIsIcfjkuHbdvG\npcEOHWIw8kEHxSrJc+ZES6xVq7RrXpMUUgUUUiI1xD1aT2vWwNy58Mor8MEHMa5rypQYz9WlS8xr\nOHhwPOrqYO+9Yy2vfv2i04ZZBJtaYkWnkCqgkBKRj61bF8Gzdm3MljF7dtznmj07ehu+9lquh+HK\nlbGgZLt2MaN8t27QsiVsvXV0+GjePI4bODBaZe++GzNv9OypVtoGKKQKKKREZJO5w/Ll0cpavhym\nTo0QWrMmQmzmzDhm1arosVhXF2G2alXsb98+gqpDB+jdO8KtY8d4dOgQ4deqVTxatMg96oMvf7uh\nsg1tN2uWe5htfLupx21ou/4B679uoCWaleXjRUTSYxaXBSFaUAMGNP29a9dGsK1ZE509li6N7ZUr\nY+zYihXREeSjj2L+w9WrozVXVxePtWs3vL2h/WvWRGvRPZ6bur2576kvq28I1G8XNgzqA+vf/q04\n/31QS0pERLZUfmglfw9by5ZqSYmISAVo5JJfMTQryaeKiIgUgUJKREQqlkJKREQqlkJKREQqlkJK\nREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQq\nlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJK\nREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQqlkJKREQq\nlkJKREQqlkJKREQqVtWElJkdbGazzWyOmZ2bdn1ERKT0qiKkzKw5cBlwMLAzcIyZDUq3VtVj8uTJ\naVehoun8NE7npnE6N+VRFSEFDAPmuvt8d18D3AockXKdqob+MG2Yzk/jdG4ap3NTHtUSUtsAr+e9\nXpCUiYhIhlVLSHnaFRARkfIz98r/+9/MPgeMdfeDk9djgHXuPi7vmMr/ISIiNcTdbUs/o1pCqgXw\nMnAA8AYwFTjG3WelWjERESmpFmlXoCncvc7M/h34G9AcuFYBJSKSfVXRkhIRkdpULR0nNqjWB/qa\nWV8ze8TMXjKzF83sjKS8q5k9ZGb/NLOJZtY57z1jkvM128wOSq/25WFmzc3seTO7N3mtcwOYWWcz\n+4uZzTKzmWa2l85NSH7rS2Y2w8xuNrPWtXxuzOw6M1tiZjPyyjb5fJjZ0OSczjGz3230i929qh/E\n5b+5QH+gJTANGJR2vcp8DnoCuyfbWxH37wYBFwPnJOXnAhcl2zsn56llct7mAs3S/h0lPkc/Bm4C\n7kle69zmdM6OAAAEvElEQVTE7x0PfCfZbgF00rlxkt83D2idvL4NOL6Wzw2wH7AHMCOvbFPOR/2V\nu6nAsGT7fuDgDX1vFlpSNT/Q190Xu/u0ZHsVMIsYR3Y48ZcQyfORyfYRwC3uvsbd5xP/Aw0ra6XL\nyMz6AF8B/gjU9zaq+XNjZp2A/dz9Ooh7v+7+Ljo3ACuANUC7pONWO6LTVs2eG3d/HHinoHhTzsde\nZtYL6ODuU5Pjbsh7T4OyEFIa6JvHzPoT/9p5Cujh7kuSXUuAHsl2b+I81cv6OfstcDawLq9M5wa2\nA940s+vN7Dkzu8bM2qNzg7u/DfwaeI0Ip+Xu/hA6N4U29XwUli9kI+cpCyGlnh8JM9sKmACc6e4r\n8/d5tK03dK4yeR7N7KvAUnd/nlwraj21em6Iy3tDgD+4+xDgPWB0/gG1em7MbAfgh8Slqt7AVmZ2\nXP4xtXpuGtOE87FZshBSC4G+ea/7sn5S1wQza0kE1I3ufldSvMTMeib7ewFLk/LCc9YnKcuifYDD\nzexV4BbgS2Z2Izo3EH9OFrj708nrvxChtVjnhj2BJ919mbvXAXcAe6NzU2hT/hwtSMr7FJRv8Dxl\nIaSeAQaYWX8zawWMAu5JuU5lZWYGXAvMdPdL8nbdQ9zsJXm+K6/8aDNrZWbbAQOIm5mZ4+7nuXtf\nd98OOBqY5O7fQucGd18MvG5mn06KDgReAu6lxs8NMBv4nJm1Tf58HQjMROem0Cb9OUr+n1uR9CI1\n4Ft572lY2j1GitTr5BCiR9tcYEza9Unh93+euN8yDXg+eRwMdAUeBv4JTAQ6573nvOR8zQa+nPZv\nKNN52p9c7z6dm/itg4GngReI1kInnZuPf+s5RGjPIDoFtKzlc0NciXgD+IjoB3Di5pwPYGhyTucC\nl27sezWYV0REKlYWLveJiEhGKaRERKRiKaRERKRiKaRERKRiKaRERKRiKaRERKRiKaREisTM1ibL\ngdQ/ziniZ/fPXyJBpFZUxcq8IlXifXffI+1KiGSJWlIiJWZm881snJlNN7OnkslL61tHk8zsBTN7\n2Mz6JuU9zOxOM5uWPD6XfFRzM7vaYmHLv5lZm+T4M5LF+V4ws1tS+pkiJaGQEimetgWX+76RlDux\n1MNngMuA+vkVfw9c7+6DiQUZL03KLwUecffdiQlfZyblA4DL3H1XYDnwtaT8XGLRy8HAySX8fSJl\np2mRRIrEzFa6e4cGyl8Fvuju85PZ6he5+9Zm9ibQ093XJuVvuHs3M1sKbOOxiGf9Z/QHJrr7p5PX\n5wAt3f1XZvYAsIqYqPMud3+v1L9VpFzUkhIpv/x/GTa4xlUj5avztteSu6d8KHA50ep62syab3EN\nRSqEQkqkPEblPT+ZbD9JLB8CcCzwWLL9d+BUADNrbmYdG/vQZLmDfu4+mViwsBPQvqg1F0mReveJ\nFE9bM3s+7/UD7n5est3FzF4APgSOScp+AFxvZmcTi8WdmJSfCVxtZicRLaZTiKW5C6/NO9AcuNHM\nOhGtr9+5+4oi/y6R1OielEiJJfekhrr722nXRaTa6HKfSOnpX4Iim0ktKRERqVhqSYmISMVSSImI\nSMVSSImISMVSSImISMVSSImISMVSSImISMX6f+5I6h05kVFUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a101588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(len(cost_avgs)), cost_avgs, color='red')\n",
    "plt.ylim([0, 2000])\n",
    "plt.ylabel('Cost')\n",
    "plt.xlabel('Epochs')\n",
    "plt.tight_layout()\n",
    "plt.savefig('./figures/cost2.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 97.74%\n"
     ]
    }
   ],
   "source": [
    "y_train_pred = nn.predict(X_train)\n",
    "acc = np.sum(y_train == y_train_pred, axis=0) / X_train.shape[0]\n",
    "print('Training accuracy: %.2f%%' % (acc * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 96.18%\n"
     ]
    }
   ],
   "source": [
    "y_test_pred = nn.predict(X_test)\n",
    "acc = np.sum(y_test == y_test_pred, axis=0) / X_test.shape[0]\n",
    "print('Training accuracy: %.2f%%' % (acc * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAEZCAYAAAA32jqgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXmYFNXVh9/TA8POsDnIvqgICIyAioQACmrQoBIlRlAR\nRI1JBI2CRFQCiAvGaFxQQdwQP0WDGwIGZBkQCQRlF1D2fWeYYRmY5Xx/VHXb03T3dPdUzXQP932e\nerq77nbq1/fWXeuWqCoGg8FgMMQDnpI2wGAwGAwGL6ZSMhgMBkPcYColg8FgMMQNplIyGAwGQ9xg\nKiWDwWAwxA2mUjIYDAZD3OBopSQibURkkZNxns0YPZ3HaOosRk9nMXrGUCmJyP0iskxEskXkHX83\nVV0FZIhIzzDh54vIwBhsRUT6iUh+rOEjTONiEfleRI7b15nmYlrJIvKWiGwVkUwRWS4iPbzubugp\nIs1E5AsR2S8ih0TkaxFpVsRLiSRd1/87v7Qmi8geW9PNIvKY182tPCoi14vIGhHJEpFFItKiCJcQ\nLp2advwHReSonWd6uZFWkLQvsMv9+95zpaTMJ4nIGBHZZeeZH0QkxcX05ovISTuvZInIOq+bi/mz\n2K6xqPfQWHpKu4AngbdDuH8A/DFM+Jie1hWR6sBwYE2scUSQRjLwBTAJqAa8B3whImXdSA8oA2wH\nuqhqVeBx4GMRaeTnx2k9U4DPgWZAbWAp1jW7RnH8dwE8AzSxNb0WGORf2eOwpiJyATAZuBdL32nA\nlyKSFJXVkXEMuAtIVdUUYCRWnqnsQlqBjMPKL4H6JGyZtxkFXA5cbueZ24FsF9NT4C+qWsU+Ahsw\nbuhZLNfoyD1UVWM6sCqmd4KcrwecAMoGcXsKyAVOAlnAy1Gk9wbwJ2AecFcYf+/afmcBmcB8oGGE\naVwD7Aw4tw34Taw6xaDrSuB3buvpF0cNIB+o7rSe0f53Lul5IbATaOeWpsD9wFd+v8WOv5tbmtrx\neIDrgd1Asss63gpMAf4OvB/glshlvrptV5NizJPzgIFh3J3On1FdYxH1LPI9tChzShLspKruAnKw\nbgaBbo8BC/mllTAYQESmicgjIRMSuQxohyVUJPQFRgO1gBVYLQ9vXOHSughYFXBupX3edUSkNlYP\nZq33nBt6BtAF2KOqR8L4iVXPWP47RxCR10TkOJaWY1T1B6+bC5oqBcuDx/4dLt/ErKntZxXWjeld\nrEbM6XD+i4KIVMVqaf+VIOU+wct8a6yb/O/tId8NIvLnCNMsCs+IyAER+VZEuvo7uKBnLNdYYvfQ\nMpF6DEK4LmQWVtctFAUytqpeH9KjNQQyDutPUJGgdWEgX6nqt3b4x4CjIlJPVXeFSwuoDBwNOJcJ\nVIkk0aJgd28/AN5V1Z8CnB3TMyDN+sCrwEOFeI1Jzxj/O0dQ1T+LyF+ArsC/ReQHVV3q58VJTb8B\nxto3l8XAMCAZqBgmTKx51GtPG3uo5I/AVBFprqrHCgsXI08CE1V1t4iEKveJWubrYw25XgA0xmoU\nzhGRn1T1m0gSjoFhWI2l00AfYJqIXKyqm/38OJk/Y7nGEruHOt5TsqkCZIRxj2ZM9M/AqoAbSri0\nFWu4xvqhehw4DNSNIK0soGrAuRQsUV1DRDzA+1hjvPcH8eKknt40z8Hqno9T1SmFxB2rntH+d46i\nFvOBT7AKvz+OaaqqG4A7sSr43UBN4Ef8dAsSd6ya+qd7WlVfwcq33aMJGykicrEd97+8p0J4TdQy\nf9L+HK2qp1R1NfARcF0U9kaFqi5V1eOqmqOqk4BFQdJzUs9or7FE76GO95REpB5WK3FDNOHC0A3o\nKiJeAWsAbUUkzdt1DTQBaOBnT2U7zO4I0loLPBxwrg3wSpQ2R4xYzcC3gHOA61Q1L8DdaT29E8iz\ngM9V9ZnCvBO7ntH+d25RFjjk/eGGpqo6FZhqx18NGAj8L4T3omgajDLA8RjDFkZXrNb1drvHUhlI\nEpEWqnoJJHyZDxxq8lJir09wQc9or7FE76GxLAlPEpHyWAUhSUTKBawy6grMUdWcEFHsA86LIsn+\nQHMgDbgYWIa14uix0EG4TkQ62cMbTwKL7XHawpgP5InIYPu6BmMtApgbhb3R8jrW9d2gqqeCuDuq\npz0/8B/gW1UdHmGwWPXsT/T/XZEQkXNE5FYRqWTn1d8Av6fgCkOn8ygi0t5O7xxgAvBFkGFYf2LS\nVEQ6iMivxXqcoIKIDAPKA/+Nxt4omAA05Zf/8A1gOvAbPz8JW+ZVdRPWHM1jtqYtgD8AX0Vhb8SI\nSIqI/EZEyotIGRG5DegMfO3nzVE9Y7zGkruHRroiwm8lxUg7Ef9jhJ/7dKBnmPCXY7UADgP/ss/N\nAP4WYfqFrcR5B+tGPwurKzkfaOTnHjYtfikEJ+zPtGg1ikLLRrZ+J2xbvUcft/TEGmbKx1pa7E0v\nE6jvhp7R/HcOaVrLtvEI1vDHUqwK39+P43kUq9BnYvXIXgcquJFHsRamrLDTOmBfy0VuahqQ/t+B\nSW7rGU2+caDM1wVm2mE3Afe4nD+X2v/fEeA7oHsx5M+Ir9EBPYt0DxU7EkcQkTbA66raybFIo7fh\nHawliU+UlA1OYfR0HqOpsxg9ncXoWbQ5pTNQ62nkEhPTpviWeLmM0dN5jKbOYvR0FqNn6dyQVSnB\nScpSiNHTeYymzmL0dJYS1dPR4TuDwWAwGIpC1MN3YR6eK9WoqitdWqOnsxg9neds1NTo6TyRahrT\nnNLZ1rtyeycCo6ezGD2dpyQ1zcnJ4Z577uH888/n8ccfdz290q5nSRCNplEP34mIno2CutmyN3o6\nGrfR0/n4S1TTgwcPkpqaSsWKFTl48CDly5d3Nb3SpueePXto27Yt+/btY9u2bTRs2LDY0vYSjaal\ncaGDwWAwGBKUuK6UsrOzz7pursFgCE7dunXxeOL6lhV3DBs2jEaNGnHgwAFat25NtWrh9niND+L6\nH77zzjsZOHAgy5cvjyrc8ePHyc/Pd8mqxCc/P9/o4wBZWVm88sortGzZEhFBRBg6dCgLFiwgLy+v\n8AgMUXHrrbeSnJxc0mYkFFOmTCEvL4/WrVuzYMECqlYN3Cs1/ojrSunSSy/l3XffZcOGUPsSBmfC\nhAk8//zzLlmV2CxfvpzU1FRat27NN998Y3qiUaKqpKenk56eTpcuXXjggQdYv369r1L65z//yRVX\nXMGll15KVlYWWVlZJW1ywjN+/HjKlSvH4MHFuYdvYjNixAhGjBjBrl27aNOmDfPnz0+IXhLE+UKH\nTz/9lN69e/N///d/3HrrrRGFWbt2LZdccgmnTp1yrDdQmibmMzIyGD9+PBMmTGDr1q307NmTt956\ni1q1ahWbDYms54wZM+jZs6fvd0pKCg8++CAtW7YErPz31FNPkZeXx5/+9CcAXnnlFVeHnUrbxLyX\no0et1/Kcd955iAgHDhwolnRLg57nnnsuAAcOHGDnzp3UqVPH1fQKIypNY9hQUIuL9u3bq4johx9+\nGHGYF198UUVEr7jiCsfssK/ZrQ0aHbMzWlavXq01a9bUKlWqaGZmZrGlm6h6Lly4UNu2basioiKi\naWlpeuTIkTP8fffdd9qwYUOfv4yMDNdsUnVXTy3BPDpr1iydNWuWioi+++67xZZuous5a9YsLVOm\njJYpU0b/8pe/aF5ens8tMzNT58+fr/Pnz4/bMu/o3ndOsXfvXgA2b95ciM8zmTZtGoAZvouAVq1a\nMX36dDp16sSWLVto06ZNSZsU18yaNYsVK1ZQt671rrN58+YFHRLp2LEjqamp7Nixo7hNLDUcP36c\noUOHAtYCh969e5ewRYnD4cOHfaNEnTt3xuPxsHLlSkaOHMnKlSvZtm0bAI0aNSIlJQWA+vXrM3z4\ncFq0aFHiw3xxWSl5FzZkZFgvXox0ci43N5dTp05RsWJFLroo4lfCn9V06NCBNm3asGzZMlMpRcgD\nDzwAQPXq1UvYktLLkiVLWLXKejfdH//4RypVqkROTg75+fmUK1euhK2Lb0aPHu37/tvf/pbly5fT\nqVMnTp0q+Lo2b+UEsGrVKmbMmEG7du2YPXt2iVZMcb3QwWAwGAxnF3FXKZ0+fZpHH32URx99FICB\nAwdy3XWhXiVfkKVLl/Ldd9/xwAMPuP7UdyLhXQX24YcfMmDAAB5++GFWrFhBdnY2e/fuJSMjg+uv\nv76kzUwY6tevT/369cP6ufPOO33fZ86c6bZJpYrTp0/zxhtv+H4/+uij5Ofnc++993L99ddz8uTJ\nErQuvpk7dy4//fTLC4/HjRtH3759OXXqFD179mTp0qVs3br1jOPZZ5+lQYMG/PDDD4wfP74Er4D4\nW+gwbNgw3wSxiOiuXbsiDtunTx8VER0+fLijNpGgE/PZ2dk6a9YsrVatmlarVk1TUlK0WbNm2qRJ\nEwW0QYMGetVVV+m9997rmg3BSFQ9n3jiCRURbd26tbZu3Tqs37vvvtuXh4cNG+aaTaqJPzEfyIQJ\nE1REtHfv3tq7d29VtRblxHJPiIVE1vO9995Tj8dzxlGvXj09fPhw2LAbNmxQj8ejycnJ+t133zlq\nVzSaxtWc0vfff8+ECRN8vy+44AJSU1MjCpudnc3OnTvdMi0hWbNmDZ999hljxowB4PbbbyclJYW8\nvDxWrlxJ+/bt2bFjB7fccgtHjx71TXoawrN161YAfv75Zy644IKgflasWFGMFpUe9u7dyzPPPAPA\noEGDADhx4gR9+/YFoEGDBiafRsm5557L3LlzC50DrVOnDnXq1GHPnj0sWLCAjh07FpOFAURae6nL\ntXx2drZeeeWVBXpJO3fujDj8zp07feG2bt3qqG0kYMt+48aNWq1aNV2+fHlQ9wULFiigzZs31+Tk\nZK1bt65u2bJFt2zZ4oo9/iSinqqq27Zt0wYNGvjyWYcOHXT69Om6YcMG3bBhg37zzTd6yy236C23\n3KJVqlTx+evXr5/m5ua6Zpebemox9ZRyc3P1+++/10aNGvl0O336tJ4+fVqXLFniO9ekSRPNzMzU\nnJwc12xJZD0De0p16tTRcePGRRze21uqXr26o3ZFo2ncCHro0CFfxrv77rv17rvvLrC+vjD8K6VD\nhw45alsi3kSvvvpq7dy5s2ZnZ5/htmLFCm3YsKF27txZjx07psuWLdNmzZpphQoVtEKFCrp//35X\nbPKSiHp6mThxYoGGU6RHVlaWazYl8k1UVfXkyZN68803R6Vn06ZNdc2aNa7Yk8h6BlZK69atiyp8\nZmZmiVdKcTF8t3r1at8WIhdddBEvvvgiAB6PB1UlJyfH59fj8fjW4JctWxYo+K6OgQMHmqW6QOPG\njenRo8cZy2ePHj3KjTfeSMWKFZkxYwaVKlWiffv2LF682Nddv/rqq0lPTzfDJEG48847ad++PQAv\nvPACq1atok2bNsyePRuwtBs8eDDLly/nj3/8Y0mamhBkZ2czcuRIPv30UwAqV67M2LFjqV69Om++\n+SZgPQ8WSLly5Vi2bJl59KMQCluQE0g8bIsVF5XSJ598Qnp6OmCNH/s/+JqXl8dTTz0FWL26Ll26\nsHDhQgDmz58PWM/azJkzB7CeHSmOl3TFO4899hjdu3encePGtGvXDrDmOfr370+TJk2YOnUqlStX\n9vmvUaMGixcvBqyHP7t27crSpUvNBpgBlClThosvvhiASZMmcezYMSpXrszhw4cBS0ewbrb+7Nq1\niwsvvLB4jY1z8vLyGDlyJM8995zv3Pjx4+nTpw+5ubm+OSWwGp4dO3bk+uuvp1evXjRu3Ng8r+Qw\neXl5vvnne+65p8TsiLsl4QaDwWA4i4l0nE9dHA+tU6dOxGPJrVq10qZNm+pll13mO3fllVfqeeed\n58oiB9XEnQN55ZVXVET0nnvu0XvuuUfLly+vzZo1K3TPq0OHDmlKSorecccdevr0acftSlQ9o2Hh\nwoUF8u3nn3/uWlpu6qkuarp3794CGj344IO+eeQ5c+YooICKiN5+++2u2BCMRNVTVXX37t1asWJF\n35zS5MmT9eTJkxGF3bVrl3o8Hq1WrZru3bvXUbui0TQuBP3ss898GfO8887Tq666Sq+66irfcwpv\nvvmmrlq1SletWqUnT570rb5Zs2aNrlmzRu+8804VEe3UqVPEf0A0JOpN9NSpU9qrVy/t0KGDdujQ\nQb/55pugCx+CsWrVKq1cubJefvnljq90SlQ9jx07pq1atdKxY8fq2LFjg27G6iWwUjILHc7koYce\n8unTvHnzAhvXTp482ef217/+VfPz812xIRiJqqeXwYMHF1jskJaWpp999lnQhWN5eXmal5en+/fv\n17S0NPV4PPrII484blM0msbFnNINN9zAli1bAGtOqEKFCsAvCxkC8e7W4J3kbNy4MQDdunUzOzn4\nkZyczCeffOKbY0tKSoo4bOvWrdm0aRNNmjThtttu46OPPjrr5+oqVKjAkCFDGDBgAACZmZkMGTKk\nxDewTEROnjzJ559/DliLFmbOnFlgYc3WrVt9Zblfv35nfd6LhqFDh/r0mjhxIqtXr+bmm29m0KBB\nnHPOOT5/1157LZMmTQKs16ucc845TJkyhU6dOpWI3T4irb20mGr5WDh+/LgeOnTIlaEm1cRt2TtB\nenq6JiUl6SeffOJYnIms54kTJzQtLU3T0tJURLRt27b6+eef69KlSwv4Gzt2rK+l3759+4h7qLHg\npp7qkqZZWVm+Yfsff/wxqPuOHTt0x44djqddGImoZyh2796tDzzwgCYnJwfd6cF7NGzYUKdPn+6a\nHdFoGtcv+YsXEvmldE7wySef0KdPH7777jsuu+yyIseX6Hp6Xz7XtWtX307WZcqUKdDSz8jI8L0S\nfcaMGfTo0cM1e0rDS+niidKo5759+3jmmWd45ZVXfOcaNWrE8OHDAWu0KtLdc2IhGk1NpRQBiX4T\nLSr5+fnccMMNrFu3jk2bNhU5vtKiZ2ZmJqNHj+aFF14I6n7ppZcCkJ6e7uqwcmm8iZYkRk/nMZWS\nw5SWm2hR8HatnXitd2nSU1XJz89n3rx5rF+/nn/+85+kpaXRunVr3073FStWdNUGcxN1FqOn80Sj\nqXlOyWAwGAxxg+kpRUBpatnHA0ZPZzEte2cxejqP6SkZDAaDISExlZLBYDAY4oaYHp41D7I5i9HT\nWYyezmM0dRajZ2iinlMyGAwGg8EtzPCdwWAwGOIGUykZDAaDIW4wlZLBYDAY4gZTKRkMBoMhbjCV\nksFgMBjiBlMpGQwGgyFuMJWSwWAwGOIGUykZDAaDIW4wlZLBYDAY4gZHKyURaSMii5yM82zG6Ok8\nRlNnMXq6i4g8LyL3lbQdxUlMlZKI3Coi60TkmIhsFJFfA6jqKiBDRHqGCTtfRAZGmd6vReR/InJU\nRDaJyD2x2B1lmv1EJD9aW2NI55iIZPkduSLyMrijp4g0E5EvRGS/iBwSka9FpJkDlxIqvYtF5HsR\nOS4iy0Qkza20/NKsLyLT7OvbIyKviEgSJL6mItI5IL9k2fn0d26kZ6fZQkTmikiGiPwsIr28bm6V\neb+wrpdDEbleRNbYWi4SkRYOx3+/nfezReSdIO7dRWS9XUbmikhDP+fngeEiUjZE3I1tfSK+l4tI\nfxHJC8hDXaK/sojSutW+tqMiclBEPhWRuuHCRF0picjVwLPAnapaGegMbPbz8gHwxzBRRLXZnn0z\n+QyYoKopwB+AF0SkTVSGR5dmdWA4sIYo7Y0WVa2sqlVUtQpwLnAS+NjPi6N6AinA50AzoDawFPgi\nyjgiQkSS7bgnAdWA94AvQhUwB3kZOAjUAS4GugJ/9nNPWE1VdaE3v9h5pidwDPjajfREpAzWtXwJ\nVAfuBSaLyAV+3pzW05u26+XQvo7JWNeVAkwDvvQ2YhxiF/Ak8HaQ9GsBU4HHsPRdBkzxuqvqXmA9\ncEMhaUS7w+si/3ykqguiDB9xOkAX+97dCDgBvBA2hPc115EewHfAgDDu9eyEywZxewrIxbrxZgEv\nR5BeXSAfKO93binwhxD+3wXeAGYBmcB8oGGU1/gG8CdgHnBXtBrFegB3Ahvd1DNIHDVsfas7rSdw\nDbAz4Nw24Dcu67gB6OH3+zngjdKgaZC43gHeclHLVkBWwLn/AKPd1jPScljEPHo/8JXfb7GvpZsL\nWj4JvBNw7l7gW7/fFe30m/mdGw68HSLO7XZey7KPDhHY0R9YGIXd+cAgYBNwwC5PEsP1V8ZqmL4Y\nzl9UPSW79dAeSLW78TvsoZHyXj+qugvIAS4MDK+qjwELgb+oVTsPtuOdJiKPBEtTVXcDq4C7RCRJ\nRH6FVeN+G8bUvsBooBawAqsl572GkGnZ7pcB7bAyeXFzJ1avwofTegahC7BHVY+E8ROrnhdh/Xf+\nrLTPu8l/gL4iUkFE6gHXAjO9jgmuKX7+KgE3YxX04sSDVVkB7ugZQzmMVU+lYC/DY/92I48G681c\nhFUmLGNUTwAb8dMXq6cUati7s/2ZYuu7REQaisgREakfIowCbUXkgIhsEJHHI+gZ9sK697cDbgTu\nAoggLe/0SwZWg6EhMCxsSlHWdN5ey1KsYYqaWJXDmAB/O4Ffh4hjHjAwynQvw6qhc+wjZHisVtP/\n+f2uhNVSqxdBOknA/4DL/Gwtlp4SVkWbCzQK4uaonn5h69txB+11OqDnE8CHAecmA393WcsawA92\nXsknSCszUTUNiOcOYJPLWpbFaiEPtb9fA5wCZrqlZ7TlsIh59EKs4c+uQLKdZ/OAYS5oGaynNBF4\nJuDct0A/v99Xh/qfgcZ2HvdEYUcT730Gq/JbC/wtjP984Bq/338Cvonh+uti9WZfCucv2jmlk/bn\nK6q6T1UPYY0PXhfgrwqQESaeiMeH7ZbuV0BfVS2L1bIYJiKBafrHvdP3Q/U4cBhLkML4M7BKVZf6\nmxCprUXkDqwu9bYgbo7p6UVEzsHKIONUdUoYr0XRMwuoGnAuBavF5AoiIlg9pU+whkJqATVEZGyA\n10TV1J8zetZOo6o5WK3k3wJ7gL9izXnuDPDqpJ7RlsOY9VTVDVg6vgrsxmpo/8iZ1+cEwa7hGMHL\nSJbf78K0jQpV3eK9z6jqGqweZu9Cgu3w+76d6PMqao16PQH0C+cvqkpJreGIsH+WXYkkY43rB40m\nmjSBX2HNS8y2bfgJmI41JBPUBKCBnz2VsVrOuyNIqxvwO3vF1h477X+KvRrOZfoRZBjGBT29E8iz\ngM9V9ZnCvBO7nmuBwAUpbezzblELa5jhVVXNUdXDWC1pXyMmwTX1hmmA1bp3tVICUNXVqnqFqtZS\n1WuB87BGS7y2OK1ntOWwSHqq6lRVba2qtYCRWL2P/0Vpc0RJBTm3Fr+hOXtI9jwKlpEWWEOSkcYZ\nC4U1vhsGfN8VYzplsebMQhLLkvB3gEEico5dEP+KtWLFS1dgjt3CCsY+LNEjZQ1woYhcKRbnYa04\nWhkmzHUi0sle/fUksFitce/C6A80x8okF2OthBmJtTLGNex5srpYrftAHNVTRKpi9SS+VdXhEQaL\nVc/5QJ6IDBaRciIyGGsoYG6k9sbAQawW/Z/sOchqWC1h//ySyJp6uQNrBdWWKMLEhIi0FpHyIlJR\nRIZgDd2/6+fF6TLfn+jLYcx6ikh7O6+cA0wAvrAbv45gx10eKAMk2WXBO4fzGdBKRG6y/fwdWBGQ\nflf85kQDOIBVpqLJr9eKSG37e3PgcazVo+EYIiLV7MbQYPxWCBaSVl87DCLSCGvhy9SwgWIYFywD\njAOOYBX+fwHJfu7TgZ5hwl+O1aI6DPzLPjeD8GOa/bC61JlY3chnwvh9B3gdq9WahXVjbOTnHjat\ngLiKZU4JazL3vRBujuqJdYPOxxo28K7YyQTqu6Env9xUTtifacWgZwesyfUjWIX2I+Cc0qKp7Wcd\nYVbBOqznc7YWWbZ2Td3Mo0HCFzanVNQ8utD+vw7Z8VRwWL+Rdv7wP0b4uXe3/88TWA22hn5udbDu\neWXCxD8K2G/n98uwejJZYfLfP4C9dn7dZNuXFCb+fKxVipuwGn3/wJ7DiiCtMbb9x4AtWI8TlQ+V\nlqpay/qcwn526HVV7eRYpNHb8A7WcN8TJWWDUxg9ncdo6ixGT3cRkeexHhMpidXAXhvygfNVdXOh\nnh2gjJORqfV0d4llTpviWpjgOkZP5zGaOovR011UdUhJ21DclMYNWRWXd2E4yzB6Oo/R1FmMnu5S\nrNo6OnxnMBgMBkNRiHr4TkTOylpMVV0ZIjB6OovR03nORk2Nns4TqaYxzSmdbb0r63lM9zB6OovR\n03nOJk2Nns4TjaalcU7JYDAYDCE4ePAgN998MzNnhnr0qWSJek5JRPRsrOXdHG5yQ8/c3Fzeeecd\nduzYwdatWwF4//33vdfia7nUrFmTl156iVtvvRWPp3jaKImoZzzjpp52/GeVpqVZz4MHD9K1a1c2\nbNhA06ZN+eknx54RDks0mpqeksFgMBjihxieTtbi5LLLLlNAb7755mJN1x/7mt16Wt4Vm4cNG6Ye\nj+eMo1evXvrQQw9p69atC5z//e9/r5mZmZqZmemKPf4kop7xjJt6aglpunv3bm3QoIE2aNBAX3nl\nlWJNuzTqqar68ccfa7NmzTQpKUmTkpK0V69exZZ2NJo6+vCs04wZM4Zly5YhIlx55ZUlbU7CMGrU\nKF566SWefvppbrrpJho3buxzS0pKwuPxkJeXR35+PtnZ2fTr14/PPvuMn3/+GYD09HSqVg3cuNgQ\nSHZ2Nvv27WPy5MkAPPHEEwUmdOvXt14xs2rVKlJSUkrExkQkNzeXjh07snOntfdzcSw8KO3k5+ez\nYsUKNm7ciIjQpk0b3nuvuF/DFSGR1l5ajLX86NGjdfTo0ZqUlKQion/4wx80Ly/P5z5jxgxNSUkp\ncDzyyCOu2UOCtewfeOABbdiwoR47diziMF26dPH1mi655BJXe0yJpmcg6enpmp6eru3bt/e1OpOS\nktTj8Wjnzp31kksuKXB+y5Ytrtrjpp5aAi37n3/+WUXEdyxZsqRY0y9teqqqfvbZZ748mpSUpD17\n9izW9KMIDAMUAAAgAElEQVTRNO4E3bdvn6ampmpqaqqKiDZr1kw/++wzn/u0adM0OTm5QKbt3bu3\nrlq1yjWbEu0mumvXLv3xxx81Nzc34jBZWVnaunVr37DerbfeWqAh4CSJpqc/6enpWqVKFa1SpYom\nJSVp7dq1dfjw4fr555/rnj17NDs7W0+ePKmVK1f2VUqjR4921abSdBPNzs7W3r17q4hot27dtFu3\nbnr69OliS1+1dOmpqpqZmamdOnVSj8ejgHo8Ht28eXOx2pDQlVLr1q19lU3z5s113759Prcvv/xS\nK1SooCKiM2bM0KysLM3KynI90ybyTTQaFi5cqAsXLtSKFSuqx+PRdevWuZJOoup5+vRp7d27t/bq\n1Ut79eqla9euDZr3Tp8+7au0kpKSdOPGja7ZpFq6bqL//e9/VUS0Ro0aunbtWl27dm1Iv5s3b9ZJ\nkyZpdna2ozaUJj1VVTdt2lSgNz9gwICoGqxOkLCV0oEDB7RmzZq+SmnixIkF3G+++WYVEa1Zs6Zu\n3brVNTsCSdSbaKz069dPPR6P1qtXz5X4S7ueEyZM0KSkJE1LS9O0tDQ9ceKEq+mVlptofn6+3nbb\nbSoi+sQTT4T1O2/ePE1JSdGkpKQCDVcnKC16etmzZ4/WqVPHVynVqVNH9+zZU6w2JGylNGLECBUR\nHTBgwBm1+cyZM7VSpUoqIjp79mzXbAhGab+JBjJ+/Hj1eDxatmxZ/emnnxyPvzTruXXrVl8v6eOP\nP9aPP/7Y9TRLy0103bp1KiJauXLlsP6++OILTU1N1eTk5AJD+05RWvT0Z/jw4QXmlEaMGFGs6Uej\nadw8p3T48GFeeuklAKpWrUrVqlVJSkpi/fr1fPDBB/z+97/nxIkTNGrUiIsvvriErU1s8vLyChxW\nnvmFm266yedv165Y33p89qGqzJ49mxMnTlC9enU6depEp04l/VaHxMG7ivHee+8N6Sc9PZ17772X\nAwcOcP3119OrV6/iMi+hGTZsWIHfkyZNYvfuiN4WX+zETaU0e/ZsMjMzAdi1axe7du2ib9++XHLJ\nJdxxxx0cP34cgG3bttG1a1fWrVtXkuYmFNnZ2WRnZzNjxgyGDRtG7dq1SU5O9h39+/dn2LBhLFmy\nhCVLllChQgVq1KhR0mYnHF9//TX33XcfAO+99x5169albt26JWxV4vHYY2e+9Tw/P5+vvvqKa6+9\nlv3793PNNdfwwQcflIB1iUnVqlUZM2YM+fn5qCrbt2+nc+fOJW1WUOKmUjIYDAaDIW7GQz/66KMC\ny7wLO9LS0vTLL7/UL7/80hV7/CGB50B++umnAku9Izk6deqktWrV0kqVKrkySZ/Ieoajf//+mpSU\npM2aNdOcnJxiS9dNPbUYNX3sscdURM54Rm7Tpk3at29fX9kfNWqUq4tHSouegZw8edL3SENSUpIm\nJyfrqFGjdNSoUbpz505X045G07jZkHXKlCn06dOnwLmuXbtSuXJlFi5cSGZmJpdddhllypThu+++\nA6BBgwYA/Pjjj1SqVMlxm7wk6gaip06dolGjRmRnZwPQv39/X5f9V7/6FapKSkoKX3/9NRkZGTz8\n8MMAZGVlAVCpUiXfkKqTJKqe4di5cyeNGjVCRJg6dSo33nhjsaVdWjYQffzxx3n66ad55JFHSEtL\n4+233waseaScnBxEhIcffpjRo0dToUIF1+woLXqGYuDAgXz++eccPXrUd65evXqsW7eOihUrupJm\nVJpGWnupy7V8RkaGrlq1SseOHaurVq3SVatW6alTp1RV9corr1QR0XXr1unJkyd1zpw5WqtWLV/L\nacKECa7Y5IUEbdk/+OCD6vF49Pbbb9fbb7+9UP/ff/+9fv/991qpUiX1eDxapUoVV+xKVD1DcerU\nKe3fv78CessttxR7+m7qqcWo6ZgxY8KOjvTo0aNYeqClRc9wTJs27YzdSOJlF5e42fsuJSWF1q1b\n07p16zPcvHuIAZQvX55u3bpx00038eabbwLw1FNPcc899xSbrYlCtKtrvIsbypSxssUf//hHx20q\njcyYMYP333+fSpUq8Ze//KWkzUlYHn74Yc4991w+//xzfvjhB/bs2eNzu+iii/j00099edNQNFq2\nbFnSJoQkIRY6DBky5Ixzd911l+/7gQMHWLNmTXGaVOrIyclhwIABDBgwgKysLGrXrs3KlStL2qy4\n59ChQwwYMACAsWPH0qVLlxK2KHEpX768b2ipXbt2vvO//vWvee+991wdsitN/PTTT74jGDNnzuTa\na68t0DvJz88vZitDk1DNDv/5jQsuuIDKlSsDcOzYMbZs2UKrVq1KyrS45vTp04C1rDbYi/yys7Np\n164dGzZsAOCcc85h8eLF1KlTp1jtTDRUlWeffdY3B9ezZ88Stqh0kJ2dzfTp033zG6+99pop2xGS\nlZXF5Zdfzg033AAUHO2YMmUKX375Jbt27SIvLw8R8c0xf/nll67Oy0dFpON8WoLjoUePHtWWLVtq\n3bp1dfz48b7zgwcP1sGDB6uIuLoKjwSdA5kzZ46WL1/et6pu3LhxZ/jZvHmznnvuuQVW302fPt01\nm1QTV89AFi9e7BuTd3OX+sJwU08tZk2PHz+uV111lYqIzp07V+fOnVtsaXtJZD2PHj3qmyPy7t4Q\nOHeUlJSkNWrU0OnTp/v2D3WbaDRNiOG7qlWr8vjjj3Pw4EFGjhzJxo0bycnJ8T0UaghOt27dfC0m\ngAceeIARI0awZs0aVqxYwd/+9jcuvfRS9u/fj4gwe/ZsZs+eTY8ePUrQ6sRh7dq1vu+PPvpoAbdF\nixYVtzmlgkmTJjFnzhxSU1NJS0sjLS2tpE1KKEQk7Lu7GjduTL9+/Vi0aBHXXXcdlStX9o04xQ2R\n1l5aDLV8Yfg/q9CjR48CK3NMTyk4O3fu9K2+C/dsUnp6uqt2+JPIevozcOBATUpK0nHjxmleXp7m\n5OToggULtH379vrjjz8Wmx1u6qnFqGlubq726NFDy5cvr7t37y6WNIOR6Hpu2LBBP/roI9/RvHlz\nHTp0qH700UeuphuOaDSNO0HDkZmZqaNHjw66XNRUSqHJzs7W7OxsnTNnjt51110FKqMXXnhBd+3a\n5dq7k4KR6Hp6qVevniYlJemoUaN09+7d2qFDB61WrZq+9957pUZPLUZN9+zZE9GGrG5TWvSMJ6LR\nNCGG7wwGg8FwlhBp7aVxUstnZmbqyJEjC/SS7r77bt2+fbtraVJKWvbxQmnRc8SIEWdMIj/22GPF\nlr4XN/XUYtQ0JydHhw8frtdee22xpBeK0qJnPBGNpnGzzVA8Uxq3xSlJSoueJ06coHv37vzvf/+j\nc+fOjB07lrZt21K2bNliSd9Lad8Wp7gxejpPNJqaSikCSstNNF4wejqLuYk6i9HTeaLR1MwpGQwG\ngyFuiGlHBxHXGhFnJUZPZzF6Oo/R1FmMnqGJevjOYDAYDAa3MMN3BoPBYIgbTKVkMBgMhrjBVEoG\ng8FgiBtMpWQwGAyGuMFUSgaDwWCIG0ylZDAYDIa4wVRKBoPBYIgbTKVkMBgMhrjBVEoGg8FgiBtc\nrZRE5HkRuc/NNM42jKbOYvR0FqOns5yVehb2bgvgfmAZkA28E+BWFvg3sAXIB7oGuJ8LbAfKhoi7\nsR3OE+m7NoBbgfXAUeAg8ClQN9Lw0RzAG0CW35ENZDoQbzhNLwdmA4eA/cDHwLluamqHewLYAWQA\n84CWLml6p33tR+30xgJJLurZ0nY7bF/bIuDXbuvpF35OUcJHmEZT4CsgEzgAjHVLzwB/I+xr61aK\n8qfjZb6Q/OnVwz/Nx4pBT0fzTCFp/RXYY5f5t4DkcP4j6SntAp4E3g7hvgC4HdgLFNhIT1X3YlUg\nNxSSRjS7Ey4CuqhqCtAIOAG8EEX4iFHV+1S1ivcAPsSqJIpKOE2rYRWMRvaRBbzjZ5PjmorIDcB9\nQGegBrAYeD/S8FFSAXgAqAl0ALoDQ4oYZzg9dwG/t9OrDnyE1ZACXMujVgCR27A2PXZtg0kRScZq\nxHwD1AbqAZOLGG1hZR4ROQ/oDez2P5/o+dOlMl+onkBVv3Sf8rPHDT3dyDOh0voNMAzohnU/awqM\nChem0EpJVT9T1S+wWu6Bbjmq+rKqLgLyQkQxH/htCLcF9meGiGSJSIcI7Nmhqvvtn2KnuyeUfxHJ\nF5FBIrJJRA6IyHMSwxa9IlIJuBl4L9qwgRSi6deqOlVVj6nqSWAc0CnA23wc1BS4CPhWVbeqaj7w\nAVYPIyhF0VRV31DVRaqaq6q77bQCry8qCtHzqKpuUavJloTVqgzML/NxVk9EJAWrJ/EIhdwwiphH\n+wM7VfVfqnpSVU+r6uoIwwYlnJ5+vIp1s8kJ4jafBM2fAfE4UuYj1DPcvXg+zurZnyjyTBH1vBOY\nqKrrVDUDGG2nH5Jo5pRi3Wt9PZAWwq2z/ZlitxCWiEhDETkiIvVDGiLyaxHJwOp6NsQqHOHoBbQH\n2gE3AnfZ8RSalh83A/tVdWEEfiMlEk27AGsCzjmt6Rygo4hcICJlsTLSzELsckJTgK6ceX2xElJP\nO7+cxKokegc4O55HgaeB14B9kZkes56XA9tEZIZ9w5gnIq0iTLMwguopIr8HslU1VB4pLfnT6TIf\nrrxvE5EdIvK2iNQMcHNaz1jyTKx6tgRW+v1eBdQWkeqhEoqmUop1CCILa0gqGGf8Saq6XVWrq+rO\nkIaofquq1YD6WC21fxRiw1hVzVDVHcC/gD6RpuXHncCkCPxFQ1hNRaQN1lj60AAnRzVV1aVYrcEN\nWMOhNwMPhTe96JqKyF1Ymfz5wvxGSEg97fySgjV890lAS89RPUXkEqAj8EqkhhO7nvWx5llfAuoA\n04Ev7Jt3UTlDTxGpAjyFNQQbilKRP3G+zAfLnweAS7Aa1+2BKlg9QX+cvofGkmdi1bMy1lySl0z7\ns0qohIqjp1QFa3LScezhnyeAfoV43eH3fTtQN5p0RKQhVove6UopXMv+fGAGMNgeHvXHUU1F5H6s\nuZ36QDmsLvZcEakQJlhRNe2F1Zu4VlUPR2dx6GjDOarqCeBvQDOgtZ+TY3qKiAerh/SgPdQUkW3E\nrucJYKGq/sceEn0ea/6seaQ2hyGYzSOB91V1exh/pSF/ulHmg1Ugx1X1B1XNt6cl7geusYcOvTh9\nD40lz8Sq5zGgqt/vFPszK1SA4ugptQBWOBynP2WxRA5Hw4Dvu6JM4w7sMe0owxVG0OsXkUZYE5Gj\nVTWw1QTOa9oD+FBVd9uF4z2sRQEtwoSJWVMR6QFMAHqq6toY7A1FJNeehJXv/fOMk3pWxWrxThGR\nPcBS+/xOEQk3dxarnqv8f8QydxKGYNfeDRgsInvs62sAfCwi/r35hM6fNm6U+Wiu3f/e7LSeseSZ\nWPVcC1zs9zsN2KeqR0IFKLRSEpEkESmPtYooSUTKiUiSn3s52x3A/7uXroQe/z2ANfF8XmF2+KXX\nV0Qa2N8bYQ0lTC0k2BARqWaHGwxMiTQ9m37Au1GGCUk4TUWkHjAXeFVVJ4SIwlFNsTLpLSKSKiIe\nEbnDtm1jmDAxaSoi3bCGJ25S1WVR2BguznB6XiUiF9t+qmKt1Nygqv7X5pie9mRuHazClwZcZzu1\n45cKKhix5tHJwOUi0t2+5gdtm9dFGP4MCinz3bEWHqRh3Wx2A/di9Q69JGz+9MOxMl9I/rxMRC60\nr6sm8DIwT1X9exJO6xlLnolVz0nAQBFpYc8jPYHfauKgaOFrzEdiXbT/McLPfat9Ls/vs6HtVger\n21cmTPyjsJ7HOQJchlULZwH1Q/gfY8d5DOv5qGeB8mHiz8fqEm/Ceq7pH9hr+gtLy/bT0fZTqTCt\nIj3CaQr8nTOfW8j0C+uGphWBiVjL+o9iPVNxjRuaYlW4pwOub7qLevbGKmxZWKvuPgQauKlnQNjG\ndpkI+RyJA3n0d8DP9n83F2jhlp5B/G6h4HNKCZ0/bT+OlvlC8uetwGas+9lurIow1e38GU2ecUDP\nv/r9d28R4pkr7yF2IFcQkeeBjar6hmuJFG5DPnC+qm4uKRucxGjqLEZPZzF6OsvZqKerlVI8UJoy\naLxgNHUWo6ezGD2dpbj1PBs2ZC3dtW7JYDR1FqOnsxg9naVY9Sz1PSWDwWAwJA5log0gImdlLaaq\nTi619WH0dBajp/OcjZoaPZ0nUk2jrpTsyGMJlrA4++jHmRg9ncXo6Txnk6ZGT+eJRtOzYU7JYDAY\nDAmCqZQMBoPBEDeYSslgMBgMcYOplM5i7r//fjweDyKCx+PxHRUrVmTlypWFR3CWU6tWLUSErKyQ\ne0saYuTrr7/mkUce4ZFHHvHl0fr167N///7CAxsK8O9//7twT3FE1EvCRUTPxkk6N1eLFaee2dnZ\nvPzyy0ycOJFt27aRm5trbe0RMBGZlpbGt99+S8WKFR23obToec4553Do0CGOHj1KlSohd+J3HTf1\ntOMvNk2feuopxowZQ05OTtDFAJdccglLlixx1YbSoOfp06cBeOaZZ9i0aROTJjn9goPoiEbTmFbf\nOc2JEyd48sknmTdvHu3atePhhx8GrBUqtWvXLtECXxrIz7feoLBu3Tp++9vfsmPHL7vQp6Sk8Oc/\n/5nU1FQWLrTeZfbpp5+ycuVKhg4dyrhx40rEZsPZR4cOHfjhhx98+bVx48aAVVHddNNN3H333axZ\n49T7IEs3hw9bb4MZPXo0W7ZsKWFroqPEKqUVK1Zwzz338PPPP5OTk8PJkyepW7cuS5cuZeLEiYB1\nMy1Xrhxly5Zl4MCBDBgwAIDzzz+f8uUDNyM3hOLECestDW3atPH1iv7+97+TmppK9+7dueCCCwDo\n06cPAEuWLGHXrl3Urx/py2MNhqIxZswYX4XUrFkz5s6dS40aNQAoV64cYA03N2vWjLfeeosJEya4\n3mNKZAYPHgzAZZddRnJycglbEx0lVik9+uijfP/991x66aU0bdqUdu3acd1115GTk0PVqtY7oXJz\nc/nvf//LtGnT2L59O507d+bo0aOkpqbyj3/8gx49egDWMIohNK+++mqB3/fddx8PP/wwlSpVKnDe\nW9F7K7FOncK9/sdgcI6JEyf6ekipqakcPHiQOnXqFPDTqFEjLrroIvbu3cudd95ZEmYmBGvXriUn\nJweAxYsXB/Vz+PBhjh8/ToMGDQBYv369b6QEoH379rRr1859Y4NQYnNK+/bto2/fvlx88cX885//\njChMdnY2O3fu5KOPPuLFF1+kadOmAHzwwQc0a9asyDaFIpHnQI4dO+brCe3bt4+OHTuSnp5OmTJn\ntkd++uknAJo3t15AOXToUMaOHeu4TYmsJ1j5EKBOnTocPXrUN6fUvXt3Bg8ezI033uhq+oGUhjmQ\nZcuWcdVVV/kWjVSsWJG6da2Xmz788MNceOGFjBs3jqlTp1K1alWWLl3qy9dOk+h6jh07lv/9739A\nwUUOWVlZ9OrVC4BDhw5x4sQJ331z586drF692ue3du3atGjRgjlz5jhiUzSamtV3BoPBYIgfYnhh\nlTrFyZMn9fTp01GFyc3N1QMHDuhdd92lIqIion379tUPP/xQT5w44Zht/tjXXOSXfQU7nNQzGEeP\nHtVq1apptWrV1OPx6O233x7S77hx43TcuHHq8Xj0qquu0v3797tiUyLrqaq6evVqXb16tS//ZWZm\nqqpqixYtNDk52TXdQuGmnlpMmqqqHj58WD/++GP1eDxhj08++cRVOxJZz8zMTL3lllv08OHDevjw\nYd22bZseO3ZMVVX79OmjaWlp2rp1a5+W3jL/8ssvn6Fzv379HLMrGk1LdPVdtIsVVqxYQa9evTh0\n6BDHjx/3nZ81axZTpkxh9erVtGjRwmkzE5qqVavSoUMHAGbPns3KlSvJycmhbNmyBfzl5ub6lo3e\ncMMNTJ48+Yw5J0N4OnXqxPr165kxY4aZ84iB6tWrc/3117Nz507Gjx/P999/D8CMGTN8fm677TZ6\n9uxZUibGPXfddReffvqp7/e6dev46quvqFSpEpUrV2bmzJmoKkeOHAHgwgsvBCAzM5O3336b9evX\nc/r0afr06cPrr79eItcQV7V8YcyYMcPXOr388ss1NTVVU1NT9YsvvtD09HTX0iXBW/bp6emanp6u\nHo9HRUTHjRt3hp+5c+dq2bJltWzZsjp9+nRX7Ul0PUP1lH7++WcVEb3yyitdt8EfN/XUEizz69at\n03Xr1vla7g0aNNB9+/a5nm6i6rllyxZNSUnRK664Qrdv367bt2/XQYMG6eHDhyMKn52drQ8++KB6\nPB595JFHHLUtGk3jRtBIyM3N1QceeEArVKigK1as0KysLM3KynI93US/iZ46dUpPnTqlN9xwg4qI\n1qpVSzdt2lTAz4cffqjvvPOOvvPOO67bk+h6btq0STdt2qRlypRREdF7771Xt27dqtnZ2dq8eXNt\n3ry55uTkuG6Hl0S9iRbGzJkzdebMmQWGlBYvXux6uomq55AhQ9Tj8eiHH34YU/hDhw75dC7JSinh\ndnQ4efIkzz77LB988AFXXHEFAK+99pqra/ETfbWYl4yMDKpXr46IkJKS4uoKpnCUFj29Ozp4adWq\nFRs3biQ7O5sRI0Zw7rnnAlCpUiV69OiBx+OhZs2ajtuR6KvFgrF48WLfMF1GRobv/KJFi7j88std\nTTsR9XzjjTcYPHgw1113HZ9++ikeT3Rr2F577TWGDBnCE088wdChQxERkpKSHLMvGk0TrlIC66Ha\n1atX+56jufbaa3nxxRdde9iztNxEAZYvX07Xrl05fvw41apVY+bMmaSmppKRkcG0adN8/jZv3kzT\npk05//zzfQ/VOkVp0XPOnDkMGjSI9evXR+S/TJkybNiwgSZNmjhqRyLeRAuje/fuzJ8//4zzplIK\njnd/wF69ejF16tSIw7344osATJs2jd69e9OjRw/fozZOknDbDEWLx+MhLS2NTZs2AXDFFVcwcuRI\nXn75ZVf2aistnDp1iiNHjniHEMjIyKBjx45Wl9lv77saNWpQr149MjMz6d27d0mZG/d0796d5cuX\n89NPP/Gf//yHI0eO8Mknn7Bt2zbfw4sASUlJNGnShHLlyvkeDDeEZtmyZUE3BK5QoYLZciwE3jJ8\n+PBhjh07RuXKlcP63759O+PGjfNVSg0aNKBPnz5Ur169OMwNi3lOyWAwGAxxQ0IO34G1C+6iRYsA\n6N27N4cPH6Zx48aubD6Y6MNNGzduBOB3v/sda9euPWNHcFVlxowZvuX01apVIyUlxTV7El3Pwrjn\nnnt46623mD17NgDNmjXzbefiBok43BSKzZs3c+mll5KXl8e9994L4NvxpWPHjnz77beu25CIenqH\n7wBuueUW3n77bcDqXfqze/dupk6dypAhQ7j99tv51a9+BVijTeedd56jNvlTqofvsrKy+PLLL3n6\n6adZt26d73yFChXo27dvCVoWn2RlZfnG4L3PJtSuXZs//OEPrFmzhrlz5wLWMyKNGjUqMTtLE/36\n9eOtt97y/XazQipt7N27l4yMDPr27cvNN98M/FIpmWe/QtOiRQv27NnD0aNH+fjjj31D9K+99hpD\nhw717a5+8OBBnnzySbZt20ZKSsoZlVZcEOkyPe9BCS0P3bx5s44ZM0arVavmez7Ee9SoUUNnzZrl\nWtok8BLmgwcPFtDqtttuK7Bc+bbbblNA//3vf7tqhz+JrGck7N+/X0VEmzZtqk2bNtW8vDxX03NT\nTy1mTQcNGqQej0ebNGmi9erV03r16qnH49FOnTppRkZGsdiQqHoOGTJER40apd26dQu6E0ZycrI+\n99xzunPnTlfSD0c0msZ9T2nBggVMnz6d8ePHk5mZCVg72Aa+csEscAhO2bJlfUNxmZmZ3HjjjQU2\nYxURRISvv/7a1zI1FA2Px0NSUpJvKHnbtm2Or7grrXh3qt62bVuB86+//rqrQ8qlgWHDhlGjRg0G\nDRrENddcA1Dg3Wnjx48v9s2CYyHuK6UOHTqQnp7O1VdfTcuWLbnppps499xzqV27dkmblhBUrVqV\n++67D4DnnnuOgQMHkpubyzXXXMP27dt9q5zOP//8kjSzVFGzZk06dOjAd999V9KmlAratm1Lw4YN\nS9qMuKdWrVqANRTv3SU8EUnYhQ7FSaJPzHsf8Kxfvz6nTp06Y6FDcnIy33//PS1btnTVDi+Jrmck\nPPnkk/z9738HYNOmTa72lBJxYj4Ubdu2ZdWqVQCkpaUBMG/evGLtJZUmPeMF8+oKQwFq1qxJzZo1\n2bBhA4MHD/a9yROsDRnnzZtXbBXS2cJDDz1E48aNady4sXk2KQZat27N/PnzmT9/vhm2O8swlZLB\nYDAY4gYzfBcBZ8NwU3Fi9HQWM9zkLEZP5zHDdwaDwWBISEylZDAYDIa4IaYl4YGrtwxFw+jpLEZP\n5zGaOovRMzRRzykZDAaDweAWZvjOYDAYDHGDqZQMBoPBEDeYSslgMBgMcYOplAwGg8EQN5hKyWAw\nGAxxg6mUDAaDwRA3mErJYDAYDHGDqZQMBoPBEDeYSslgMBgMcYOrlZKIPC8i97mZxtmG0dRZjJ7O\nYvR0lrNST1UNewD3A8uAbOCdIO4VgdeAA0AGkO7ndi6wHSgbIu7GQD7gKcwOvzDlgBeBXcBhYBxQ\nJtLw0R7AE8AO+9rmAS0diDOkpsBtQJbfcdzWqG1p0BS4FVgPHAUOAp8Cdd3S03bvBawFMu3PG93M\no27lmxDp3Glf+1E7vbFAkst63g38bOfPmUAdN/UE+gN5AeWiSyLkTyAZeAvYaue/5UCPAD/d7TSP\nA3OBhi7rWaz3UL9050RiayQ9pV3Ak8DbIdwnANWA5kB14EGvg6rutcW+oZA0otmd8G9AO+AioJn9\n/fEowkeMiNwA3Ad0BmoAi4H3HYg6pKaq+oGqVvEewJ+BTaq63HZPaE2BRVg3lBSgEXACeKGIcYbU\nUyuddrAAACAASURBVERSgQ+Ah1S1KjAU+D8ROQfc0dPFfBOMCsADQE2gA9YNbkgR4wyn5xXAU1h6\n1QC2AB963V3KnwCL/MuFqi6IMnzE6eBs/iyDVal0sfPf48DHItIIQERqAVOBx7Dun8uAKd7ApaC8\nAyAit2FpUfhmq1HUck9yZqu+OVaLonKYcMOBt0O4bceqOb2tnw4R2PE/oLff7z7A9jD+84FBwCas\n3txz2BvRRpDWo8AUv98XAScdbDmcoWkQP/OAJ0qLpgHxVAbeA150S0/gV8C+gHP7/XVxQc+o8o1T\netpx/RX40kU9nwde9ftdx7a/iYt69gcWRmF3XOZPv3hXAr+zv98LfOvnVhGrImzmop7FWt6BFGAD\nVqPJkZ6Sl2A18WXANmC0iBwQkVUiclOAn/VAWog4O9ufKWq1fpaISEMROSIi9SO0xQPUF5EqYfz3\nAtpjtQhuBO4CiCCtOUBHEblARMpiDZXMDJNOtIRt3ditqc7ApACnRNYUEfm1iGRgDWc0BIaFSSca\ngum5EsgVkZ4ikiQivbCGpVb5+XFaz1jyTcx6BtAVWBOh38IIpqdyZl4BaOV3zmk9FWhr32M2iMjj\nIpJUiO3xmD8RkdpYvZO19qmLsPIoAKp6AtiIu3pCMZZ34GmsKZ59Yfz8QhFbTcOxar4RWF2zLli1\ndXM/P1djDT8Fi7Mx0Y+HPgl8C9TCGm9dgjXeXDtMLX+N3+8/Ad9EmV4+kIPVUmjsYIspbE8Ja15i\nbpDzCa2pX7i6wCzgJTf1BHpijdfn2J/XuqlntPnGQT3vwmo513BLT6zhwf1Aa6yhw/F2XvmDi/mz\nCdDI/t4K64b+t2LQ0+n8WRb4Bnjd79xE4JkAf98C/VzUs9jKO3AJ8ANWxReRrUXtKZ3EKnRjVDVX\nrXHeecA1fn6qYE32OsVTWJOFK7CE/QzIVdVwtfAOv+/bsTJboYjI/ViFsD7W5OBoYK6IVIjB7qBJ\nFOLeD2v4IJCE1dQfVd2NVfH2izZsCM7QU0TaYc17dlbVslg9ibdExL/l6aieMeabIulp9wCfxqpw\nD0dncehoA0+o6hxgJNY8yBb7yAJ2+nlzVE9V3aKq2+zva7D07F1IsLjKnyLiwZpXzMZaSOLlGFA1\nwHsKlqZeErK829f8GvCgqub7O4ULF02lFGyCyjsEEpiIv98WWBcfaZzhjVDNVtVBqlpfVc/HWj2y\nrJBgDQO+74owuR7Ah6q6W1XzVfU9rMnIFtHaHYKQ1y8inbDG6/8dxDmRNQ2kLNYYuhMEu/buwH9V\n9QcAVV2G1TK8ys+Po3oSW76JWU8R6YFV8fZU1bWF+Y+CoNeuqq+pajNVPRdrdVoZCg4ZOq1nMApr\n0MVN/hTrNbNvAecAN6tqnp/zWvyG5kSkEnAevwzvQeKW96pYQ35TRGQPsNQ+v9O+v4U0sLDuVxJQ\nHngGa26jHPaSU6zM+DPWyo0yQCescVj/SbpZ+E2qBcRdEcgFLoiyS10XK1NejlVrXxXGfz4wG2uF\nYANgHXB3hGk9DSwEUrEq8DuwWjBVI7U3Wk39/EwA3g0RPpE17Qs0sL83AtKBl93SE6vXfgBIs3+3\nxVrqe5VfeKf1jCrfFFHPbsAh4NdF0TAKPcthDaEJ1s1pPtZIiZv581rsoSWsxVWrCVj8E+f58w2s\nFZiVgrjVwuoF3WRr/hzwnct6Fmd5T/U7LrHjqkOIJe6qGlGlNNKOyP8Y4efeEvgOqxu6hoLPgNTB\n6vaFXAMPjMIaoz6CtXCioV2A64fw3xlryOC4LU6fQuzPx+oub8K6Gf0De0wzgrQqYo357sVaZbgM\nv7HVImTSwjQtb+txZZCwia7pGNv+Y3aazwLlXdZzqG1rlv35V5f1jCrfFFHPucBpCj7DM90tPbFu\nTCvt/28P1lCQ+IV1Q89/2FoeszUaSZhnseIpf2JVbPlYvS3//6iPn5/uWOXuBGc+p5TQ5T0gnsZY\nc1dh55TE9uwKIvI8sFFV33AtkcJtyAfOV9XNJWWDkxhNncXo6SxGT2c5G/V0tVKKB0pTBo0XjKbO\nYvR0FqOnsxS3nmfDhqylu9YtGYymzmL0dBajp7MUq56lvqdkMBgMhsShTLQBROSsrMVUNdq9uiLC\n6OksRk/nORs1NXo6T6SaRl0p2ZHHEixhsR4zcA+jp7MYPZ3nbNLU6Ok80Wh6NswpGQzFxo4dO6hV\nqxYXXnghBw8e5ODBgyVtksGQUEQ9pyQiejbW8m4ONxk9HY27RPTMyclh6NChTJ48mYwMa0eYSy+9\nFIDFixe7mrabetrxn1V51OjpPNFoanpKBoPBYIgbEq5SevPNN/F4PIgIderUoU6dOuzZs6ekzYpr\njh8/zvHjx7n//vvxeDw+/dq2bcuCBQs4depUSZuY0OzZs4ff/OY3vPrqqxw5csR3/rrrruO6664r\nQctKBzt27OChhx5ix44dhXs2hCQ7O5s33niDDh06UKlSJSZOnFjSJgUnhm0ztLg5ePCg/u1vf9Py\n5ctrUlKSejyeAkerVq1cTd++5pi3bQl3uK3npk2btEuXLtqlSxf1eDzauHFj7d+/v/bv39937rbb\nbtOTJ0+6aoc/iaxnIJmZmfrII49oUlKSL296v6ekpGhKSoqmp6e7aoObemoJlXlV1R07duiOHTu0\nUaNGmpSUpLfeequuXr1aVVWzs7N19erVvsNJSqOee/fu1caNGyvWM0e+Y/LkyTp58mTX049G07gX\n9MCBAzps2LAClVCDBg20ZcuWvt9JSUk6YsQI12xI1Jvo8ePHtVWrVj6dOnfurMePH/e5nzp1SmfP\nnq0ej0efffZZzc3N1dzcXNfs8ZKoegZj1KhRvkoosFLyP/6/vTMPj6JK+/ZdFQIhRAIEgaiR+CmM\nKA7Rl4sAIsg2Ok6UyDIoviwOMIIzss44DPuiEDcYUUFlkbAo44agfrixJKAsw7zKMjIOX/wkEIiA\nQDaSEJLn/aO6y07SnXQnXd1dybmvq650V51T58mvn6rnrFWLFi2S7du3W2JDXbyJnj59WmJjYyU2\nNraSlt27d5eEhATze2Jiol/Lrmt6FhYWyj333COAPPvss1JSUiK7du2SFi1ayNy5c2Xu3LmW21Cn\ngtJdd91l3lTHjh0ra9askcLCQnn00UfLBar4+HjLbLDrTfT06dOi67qMHDlSRo4c6TZNfn6+qWFO\nTo7k5ORYZo8Tu+pZkczMTImJiSnnh0Cllrxz/+OPP26JHXXtJnr69GmJi4srV+msGPgTExMlPz9f\n8vPzpbi42K/l1zU9p06dKoC89957UlpaKiIieXl5sn37domLi5O4uDjLbfBF0xqtUwoU58+f54cf\nfgBg0KBBLF++nLAw929B7t27dwAtswdbtmwBoFMnT29SLs+mTZsAGDlypGU21SVefPFFLl68aK7B\nuOuuu9iyZQsZGRmkpKTw7rs/vwpL13XWrVvHgw8+SL9+/TydUgH87W9/Iysri7Iy471wDRs2pHfv\n3gwfPhyARx55JJjm2YotW7awePFipk6dSnJyMrqus2fPHgYOHMiuXbtMTUOJkA5Kq1atIisri/j4\neBYvXuwxIDVt2pS//vWvAbYu9PFmYLhx48aMHDmS1FR3L7hVVMWSJUvMgNS/f3/eeecdoqKiSEhI\nYOHCheWCEkBUVBQtW7YMhqm2Yvfu3Wiahq4b87Cee+45JkyYEGSr7MmqVau44YYbmDdvnqnnt99+\nS3Z2Nhs2bOCBBx4IsoWVCcmgtHfvXgBmzpwJwN133811110HwKVLl/jwww95//33zfTJycm0a9cu\n8IbWAXRdp3Fjf73dvf4yadIkGjRoQHFxMcXFxbz22muV0gwcOJCEhIQgWGcv2rdvb94DACIjI4No\njT358MMPAfjoo49Yvnw5TZo0MY/t32+8APbKlSskJiYGxb6qCMmglJOTAxiiAWRlZXHggPG23iNH\njjB69GjAaCEBqpXkgQ4djLdvr169GoDJkye7TXfHHXcAsH79ekB139WEYcOG0alTJ5o3b87mzZvd\nphk6dGiArbInI0aMYO3ateb3yZMn89VXXzFjxgxuvPHGIFpmH15//XXACOhDhgwx9xcUFJjd+hER\nEUGxrVq8HXySAA7SHTp0SA4dOiRRUVFuB42d2+233y6333675fZg04H50tJSGTJkiKnX/PnzpbCw\nUMrKyqSkpESKiorklVdekcjISNF1Xdq2bStt27a1fAaeXfV0smjRIlm0aJHbSQ3u9um6Lv/4xz8s\ns8dKPSUIA/OZmZkyePDgShMdoqKiZPny5ZaXb3c9i4qKpEWLFtKiRQu55557yh3bsGGDOR38qaee\nstQOV3zR1HaLZxUKhUJRh/E2ekmAorwrrrV8XdfLrU3SdV3efPNNefPNNy23AxvX7C9evGi2KJ26\njRs3Tvr16ydt2rSRAQMGyM6dO6Vdu3bm8S1btlhqk531nDlzpjRv3lyaN2/ucbqyu3VKVmKlnhKE\nlpKTZcuWybJlyyQuLk4aN25s+ueyZcssLdfuej7zzDNma+i1114z9587d07at2+vWkq1YfXq1Ywa\nNYr4+HiSkpL44IMP6NGjh3n8hhtu4IYbbgiihaFPdHQ06enppKen889//pOJEyfSuHFjZsyYwd69\ne3n//ffp1asX06ZNM/OsW7cuiBaHLqdPn2blypXk5uaSm5tr7o+OjmbLli08//zzHvMuXryY0tLS\nQJhZZxg/fjzjx48nMzOT1NRUNE1D0zQWLVoUbNNCGlff/Ne//kVeXh579uyhe/fuXLx4kdatWwfR\nOi/wNnpJgKK8O86dOyciRl9pnz59RNd16dChg+Tl5UleXp7l5WPjmr23nDx5UiIiIiQiIkKGDBli\naVl21XPWrFmVWkVJSUly9OhRM828efOkYcOGbltQK1eutMQuK/WUIPpoSUmJlJSUyOHDh2Xs2LGm\nlhXHSfyN3fX8+OOPy41xNm/eXAAJDw+XvXv3ysqVKwWQsWPHWmqHK75oGpKz7yoSExMDGFMZd+7c\nCUBCQgJRUVFBtKpuce2115oL6Y4dO0ZJSQnh4eFBtip0yMrKctuCfPLJJ7n55pvN77NnzyY1NZXj\nx49XSvv1119baqOdKCoq4sqVKx6v4ezsbO6//36gvG6dO3cmJSUlIDbalfvuu4833ngDgHnz5gFw\n//33M27cOBITEzly5AgQuv4YckGpqKiI2bNnA5CYmMigQYPMY88884z5eerUqQG3ra7jvLmuWrWK\nAwcO0K1btyBbFDocP36czMzMcvt69uxpvjOpuLiYTZs28fDDD5uLFF0pKyujf//+AbHVDqSlpXHd\ndddx6623Vjr2wgsv8PTTT5tLQzRNo127dowePZrx48eXW3OjcM+IESPK/XXH9ddfHyhzfCLkgtJn\nn33GCy+8AMCECRPMoFRSUsKlS5eCaVqdoKSkhLCwMLc3Tld27dqlgpILzvEMV7755hueffZZUlNT\nKS0t5eTJk+ZrQSqSm5urbqYO5syZQ0pKCocPH6akpITi4mIOHjzIpk2bWLJkiZnOqeOUKVOYPn06\nzZo1C5bJdZJTp04F2wS3hFxQcuWLL74gPz+fRo0aMX36dNLS0gDjWViNGjUKsnX249ixYzz99NO8\n+uqrbhfOubYEqqphKQxyc3OZP3++x+NOjZctW8ZVV10VKLNCnqeeeoqWLVvy6quvkpGRwccff4yI\nmIG/ZcuWjB492uy+69q1a5Atrpt899135OfnA4TUUEjIBaXIyMhyz2iaPn0658+f56233gKgQYMG\nJCcn07Fjx2CaaStKSkoA6N69O0OHDnUbkM6ePWs+Gqd///7qGW0VuPXWW7nttts4fPhwleni4+PN\nZzQ6ZzSqAF+exx57jNdff52lS5ea+xo1asSDDz7I8OHDSUhIoE2bNkG0sG7Tu3dvdF3nwoULnDx5\nEqDcuGjQ8XZGhARo5oiISPv27aV9+/ZuV8ZbPTPMHdh0tpiTsrIyKSsrk/nz50tYWJj85je/kdTU\nVLlw4YK5TZgwwdQ4KirKnPFoBXbVc82aNR7XJP35z3+WjRs3WlZ2VVipp1igaV5ennz22Wfy8ssv\ny9GjR+XUqVOSnZ3t1zJqg930rAkLFy4UICTfp6QZ6b1H0zTxNY+vfPrppwCVXiXdqVMnduzYQXR0\ntKXlV0TTNESk8kCBf85tuZ5OioqKmDVrFp988gnffvttpePOFtTWrVvp2bOnZXbUFT1DBSv1dJy/\nXmlaH/Rcu3YtI0eONB8QvHv3bkvHPH3RNKQXzyoUCoWifhGSLaWsrCzAqLHPmjWLxx9/nNjYWH77\n29+aTwYPJHWtZl9YWMjBgwe58847zX1/+MMfzKn4Vo8n1TU9g019qNkHkvqgZ1ZWFgkJCTz22GMA\nTJ8+3dJXhPiiaUgGpVBD3UT9i9LTv9SHm2ggUXr6H9V9p1AoFApbooKSQqFQKEKGGq1TcrdiXVFz\nlJ7+Renpf5Sm/kXp6Rmfx5QUCoVCobAK1X2nUCgUipBBBSWFQqFQhAwqKCkUCoUiZFBBSaFQKBQh\ngwpKCoVCoQgZVFBSKBQKRciggpJCoVAoQgYVlBQKhUIRMqigpFAoFIqQwdKgpGna85qmjbOyjPqE\n0tP/KE39i9LTv9RLPat6LS3QEFgF/ADkAl8D97ocDwfeBf4/UAb0qpC/DZAJhHs4f7wjn+7tq3KB\nV4E8l60IyPU2vy8b8BDwbyAHOAe8D1xTi/NVp2dX4HPgJ+AM8DbQxko9K+TfVpv8Xpy/EbAEyALO\nA68ADWp5zuo0vQU44CjvIvAl0MNiH/Wr31RTll+vh+r0rJB2tkObPnXFRx1l/B/gI8f/fxZ4xkL/\ndOrh+hvOsNg/OwKfOv63Mqt0dClvFnDCcf3tAG6pKn11LaUGDkF6ikhTYCbwtqZpbV3SpAP/DWQD\n5R6kJyLZGBfnA9WU4/XTCUVknIhc5dyAtzBu3lbwJcb/Hg20BS4Bi2txvur0bIZxk2nr2PKAN5yZ\nrdDTzKBpjzjss/JhiNOAO4BbgfaOzzNrec7qNM0ChgAxQHNgI0ZFCrBMU3/7jUcsuB68uebRNO1G\nYDBwqoI9tvZRTdMaYlQMvwBaA9cC62txSq/0BJq6/I5PO3dapOdljOtgtA95aoSmaQ8A44C7gBbA\nHmBdlZlqEPUOAg+62X8CQ/iK+6cDqz2cK5PytYREH21pglH7uKuKNGXAE0AGRs3gWRwPovWxrCgg\nFVji51qEWz0dx+6gQq3XCj2BaOA7IJFqal210RP4BzDY5fvDQKY/9azGRxsAfwC+DqCPVus3fvTR\naq8Hf+kJbAV+jdFL0qfCMTv76O+BNH/7pCc9+bmlE1ZFekv8E7gJL1pKtdTzr8DfXb7fChRWmcdH\nMVsDhUB7N8c8BaWBwD89nK9tRQcDrgcuANd5Yc8I4P95Ieg2jFZInMOxR3tbFtADo9lZhtH0bOhH\n5/Sop+P4JOArq/XE6EabiBddAbXREyMoDXH5/ojjfFdZranjNywBjgM3BkBTr/2mtj7qy/XgDz0x\nWp6bHJ/dBSU7++hqYC3wfzFuwDuAjlbp6fL/nMS4h64GYqzW05HOl6BUUz27YATOdhjDPc8C71dZ\nng9ihmM0aZd7OO4pKPUHMjzkqdbBqrFpGzDbC0F/5fJ9PPBFDcq6BvgMeNFPzlmdnr/EGFu600o9\ngc7A/2BMevH2gq+RnsACYDfQEqOvfB9QCrQOkKaRwDOO/1dz2W+lj1brN3700Wqvh9rqCVwF/Ae4\n3vHdXVCys49+htG9dQ9Gy/pPGC0Et2M6ftCzCUaPiA60At4BPrFST5e8vgSlGvun47ovw6gUZgDx\nVaX3avadpmk6Rj9gEfBHb/K4cBVGjdGvaJp2PdALo1ZTHSdcPmdi3Ch8QkROYQzYjfA1b0Wq01PT\ntJswamoTROTLCof9pqfDjmXAJBEpcz1UTdaa6vk0xkDvNxjBaRNwRUR+9DK/R7zxURG5hDGu1R64\nzeWQJT7qKNNbv6mVj/p4PXhzPk96zgXWiUima/IK2e3so5eAXSLyqYhcEZHnMcYjb/bWZnd40lNE\nCkTkf0SkTETOOI79StO0Ji7ZLfNPH6iRnpqm/RHoC1yHMdFpPrBd07TGnvJUG5Q04xWJq4CrgUEi\nUuqNMS50wLgJuUN8PJcrw4HdIvKDF2mvr/A5q4ZlhmM4bY2pTk/HAOjnwHwR2eDmFP7UsynwX8Df\nNU07Dex37D+padqdVeSrkZ4iUiQiT4jIdSJyE8aMuAM+2lwJH300DMPvXX9Hq3zUiTd+U1sf9eV6\nqJJq9OwDTNA07bTDZ+IwBu7/7JLGtj4KHHL9ovnhFbE1vIe63put9k9vqKme9wJvicgpR+BNxZhw\n1MFjDi+aXq9izJho4uF4IyACI5L2ByLcNIcHe8gbCVwB2tWg6fkdMMrLpufn/NwfehQY42UZw4A4\nx+e2QBqw1FdbvdUTY6ZPBjC1ivx+1ROjy8C5dXboFYvnKai10fMax6ZhTH/PBPrVRk8vNO0HJGAE\no6bAUipPdPC3pj75TW009fV68IOeLVz8pbXjNxzkmtbmPtoeKMCo3YcBk4Fj1GLpQjV6dgF+gRGE\nYoC/A9us9E9HvgiM5RJlGPfwRlb4J7AQ2OX47XSMylMexmxD93mqOaFzEO0S5efRP+yS5gdHmlKX\nv87+5liMYOXxBwXmYazJueD4ga53lFHVoGc3Rxq3gdKNoH/EuNmfA57D0f9aXVnAUw778zH6zlOo\nEHR9dIQq9QTmUHnNQq5Lfkv0dMkb7/j9quuvr6medzl0LHA49sPV2eQHTQc7ysoDTmNMmY6zUlNf\n/aY2mvp6PdRWTzfpy40p2d1HHWkexAhEOcB2oIOF/vkQ8L3DV04Ba4BWFvtnvMMm1/v291boiRE0\nV2IsGcrB6Bn5laeyRMQY7LUKTdOex5gN9KplhVRvQxlwk4h8Hywb/IXS0/8oTf2L0tO/1Ec9LQ1K\noUBdctBQQOnpf5Sm/kXp6V8CrWd9eCBr3Y66gUfp6X+Upv5F6elfAqpnnW8pKRQKhcI+NPA1g6Zp\n9TKKiUitp4a6Q+npX5Se/qc+aqr09D/eaupzUHKcvCbZbIsflipUidLTvyg9/U990lTp6X980bQ+\njCkpFAobc/nyZZYuXcq+ffuCbYoiAKigpFAoFIqQQQUlhUIR0uzZs4cpU6awcePGYJtiC9LS0mjQ\noIG5paenB9sknwi5oPTpp5+iaRqaptGkSRP+8pe/8P33armBIjS4fPky8fHxxMfHk5SUxOXLl4Nt\nUp1n8uTJiAitWrUKtim2oG/fvui6bm79+vXjo48+4tixY8E2zSt8nhKuaZpYOUiXnp7O0KFDAWMw\n8OzZszRq1IihQ4eyYsUKGjSo0dyMWqFpmqWzxQI16FlQUMDcuXPJzs7m5ZdfJjo6moKCArp27Uq/\nfv1YsmRJQOyws55FRUXExcUB8NNPP7FixQpGjhzJyZMnAXj00Uc5evQoPXr0oFOnTma+uLg4RowY\nga77vx5opZ6O8wfMR11xjiF1794dTdPIzc0lMjLS8nLtrmeDBg0q+VlZWRmdO3dm/fr13HTTTZaV\n7QmfNK3Bs5wkUFy5ckXee+89GTp0qOi6LsOGDZPi4uKAle/E8T/X6plinrZA6pmamiq6rkuzZs3k\n22+/FRGRN954Q3RdlzZt2gTMDrvrmZGRIRkZGaJpmjRp0kTat28vuq6LruuiaZpcc8010rp1a3Of\nc/+cOXPkypUrfrfHSj0lwD7qysSJE2XixImiaZokJycHrFy765mWlibh4eHltrCwMAkPD5etW7da\nWrYnfNE08M0OHwgLC2PgwIH8+te/pqSkhLfeeovHHnuMnj17Bts0W1Faajwpf/fu3QDMmzePDh2M\nJ8f/9NNPQbPLjogIe/bsMb+XlJQAhqYAw4cPp2XLlpSVlXH+/Hkz3bBhw1iwYAEJCQkkJycH1mgb\nsnLlSlavXg1AmzZtWLp0aZAtsg/x8fGUlZWV2+f8npSUxPbt20P7Hupt9JIARfmq0DRNunTpEvBy\nsXnNfu3atbJ27Vqz1v7555+bxyZPnqxaSj4wZ86ccq2fjz/+2Kt8H3zwgei6LsuWLfO7TVbqKUG4\n5gsKCuT22283dV64cKF57Mcff5QZM2ZIZmamZeXbXc+LFy/KoEGDZNCgQZVaSs4t0PiiaUi3lCoS\nGxvLkSNH2LZtGzNnzuQXv/gFAGvWrAmuYSHMm2++yYgRxktPu3fvTpcuXejdu7d5fMeOHa4Xi6Ia\nkpKSyMjIAGDGjBncfHP1LyTNyMggOTmZjh07MmDAAKtNtD0vvvgiBw8eZMyYMQBMnDjRPPbQQw+R\nlpaGpmksWLAgWCaGNNHR0Sxfvtz8vmXLliBa4zshN/tOoVAoFPUXW7WUbrnlFrZv387QoUO5cOEC\nYWFhwTYppMnNzWXcuHHmIz4GDhzI1KlTK6VzTsF3JT8/n/DwcBo1ahQQW+1C586dWbdunVdpd+3a\nBcD999+PpmmMHz+ea665xkrzbE9OTg4vvfQSkZGRjB49GsCccXfixAnS0tIQEY4cORJMM0Oeq6++\nGoAOHTrYrqVkq6C0bds2NE3jwoULAPTo0SPIFoU2FQPNc889x/r16+nRoweTJk0CjGniYAwmO7ul\nAMaOHUurVq3UgsUakp6ezn333QdAw4YNWbx4MePGjQuyVaFP3759OXPmDK+99hpdunQpd2z+/Pmm\nT+/fv58+ffrwyCOPEBsba6Zxaq4wWLBgAV9++SU7d+4st/+OO+5gx44dREdHB8ewqvB28EkCNEhX\nFZqmlZtmm5KSIikpKZaXi40H5tPS0kTTtEraVfdd13VJTEy0xCY761kdxcXFMm3aNGnSpInExMRI\nTEyM/Oc//7G0TCv1lABqumLFCrfTv4uLiyUzM9P0Ude/sbGxMmzYMCkoKJCCggK/2FFX9HSybQci\nuQAABWdJREFUatWqShMdwsLCZPbs2QGzwRdNQ66ldO7cOU6cOAFA69at+eCDD9i6dWuldLNnzzZr\n+wrP9OzZk7NnzwKwefNmWrVqxf79+3nppZcoLS0lPz8fgFatWpm10LZt29KtWzdGjRoVLLNtyb//\n/W/uvfdeTpw4QVJSEitWrABQTyLwgn379jFlypRyExjOnDkDwMaNG81jTh8dM2YMM2fOpGnTpqFZ\n2w8hfve73/H73/8+2GZ4j7fRSwIU5T/55BPBeNOhWcN3bs5969evt9SGilAHa/aXLl2S6dOnm62i\nQFLX9CwuLpaNGzdK27ZtBZDFixdLUVFRwMq3Uk8JgKauU8A1TZOxY8eWmxJesYXk7TT8mmJ3Pd3h\nrqV07bXXyv79+2X//v2Wl++LpiHXUrr55pt5/vnnAWNM5OjRo3To0IHCwkJmzZpFVFQU/fr1C7KV\n9qdx48YcOnQo2GbYnsOHD5OcnMwPP/wAGK3MvLw89u7dS69evYJrnE0YO3YsBw8eNFtBq1evRkTK\njYm6tpDuvvvuYJhpa9wtps3Ozg7NxfPeRi8JYpQXEXnllVfMPuRAQx2r2TtJSkoSXdflySefDGi5\ndtczKytLsrKypGPHjhIeHi66rkt4eLh06tRJmjVrJrquS0REhJnOaqzUUyzWdMyYMeVaQb1795aV\nK1fKjz/+6DaNlYtmndhZT3fs3bvXbUspPDxcpkyZIlOmTPHbeJwnfNFUrVNSKBQKRcgQct137rhy\n5QrvvPMOAE888USQrakbFBcXs3//fkREdYf6yPHjxwE4e/Ys06ZN48477+SWW24hLi6OgwcP0q1b\nN4qLi83n4inc43y+naZpjB49mlmzZplPYIefJzo40yhqxvTp0z0ee+mllwDj9SCBeAK7N9giKF24\ncMF8UZVazOkfSktLOXfuHJqmERUVFWxzbEW3bt0AyM7OLrf/4sWLDBgwgOLiYpKSkmjdunUwzLMF\nOTk5zJ49m7KyMmJjY3n99dcrpTlw4ACAs8uLhx56qFzQUnjHxo0by63lCnVs033n7G8cPnx4sE2p\nE3zzzTfm54SEhCBaUjc4d+4cCQkJHD9+nEmTJvH2228TERFBREREsE0LSf70pz9x5swZYmNjzfcm\nVeTgwYPmBAhN08wp9grfuPrqq/nlL39JWVmZuZWWlpb7HkrYoqUEP8++ycjIMB+hoag5zic5KLzn\nzJkzfPXVV2YQ37lzJ82aNePw4cOkpqZy4sQJpkyZwsKFC1WLvhoSExPJyMhg48aNHtdxff7558DP\nLaVQ6V6yI5s3b+bGG28st0/XdfMBwU2bNg2GWW6xTVBykpOTE2wT6gSFhYXmxa7wjnfffbfcmKa4\nTFsOCwtj1apVPPzwwyogecGYMWPMp4B7IiUlBYAPP/wwECbVaWJiYnjiiSfMMSQnzucLhlJQskX3\nXUxMDHPmzAFg8ODB9O7dm8uXLwfZKnuzefNmtw9iVXjmtttu83hswIABjBo1SgUkP9KlSxe6dOnC\nggUL1GsqaklkZCR9+/YNthleYYuWkq7rTJ06lblz53Lp0iW6du2qnhCuCDiJiYns27ePDRs2AEZL\nKT09nV69erFo0aIgW6dQVM19991ni8q8LVpKCoVCoagf2KKlBNCkSZOQmyVSF5gxY4aaIeYlDRs2\npHPnznTu3DnYpigUdRbN18FuTdOkvg2Qa5qGiFgy+KL09Pu5lZ7+P3+90lTp6X980VR13ykUCoUi\nZKhR952aseVflJ7+Renpf5Sm/kXp6Rmfu+8UCoVCobAK1X2nUCgUipBBBSWFQqFQhAwqKCkUCoUi\nZFBBSaFQKBQhgwpKCoVCoQgZ/hfAHLPf42Z3yAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10d7a6978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "miscl_img = X_test[y_test != y_test_pred][:25]\n",
    "correct_lab = y_test[y_test != y_test_pred][:25]\n",
    "miscl_lab= y_test_pred[y_test != y_test_pred][:25]\n",
    "\n",
    "fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "for i in range(25):\n",
    "    img = miscl_img[i].reshape(28, 28)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "    ax[i].set_title('%d) t: %d p: %d' % (i+1, correct_lab[i], miscl_lab[i]))\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/mnist_miscl.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Debugging neural networks with gradient checking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.special import expit\n",
    "import sys\n",
    "\n",
    "\n",
    "class MLPGradientCheck(object):\n",
    "    \"\"\" Feedforward neural network / Multi-layer perceptron classifier.\n",
    "\n",
    "    Parameters\n",
    "    ------------\n",
    "    n_output : int\n",
    "      Number of output units, should be equal to the\n",
    "      number of unique class labels.\n",
    "\n",
    "    n_features : int\n",
    "      Number of features (dimensions) in the target dataset.\n",
    "      Should be equal to the number of columns in the X array.\n",
    "\n",
    "    n_hidden : int (default: 30)\n",
    "      Number of hidden units.\n",
    "\n",
    "    l1 : float (default: 0.0)\n",
    "      Lambda value for L1-regularization.\n",
    "      No regularization if l1=0.0 (default)\n",
    "\n",
    "    l2 : float (default: 0.0)\n",
    "      Lambda value for L2-regularization.\n",
    "      No regularization if l2=0.0 (default)\n",
    "\n",
    "    epochs : int (default: 500)\n",
    "      Number of passes over the training set.\n",
    "\n",
    "    eta : float (default: 0.001)\n",
    "      Learning rate.\n",
    "\n",
    "    alpha : float (default: 0.0)\n",
    "      Momentum constant. Factor multiplied with the\n",
    "      gradient of the previous epoch t-1 to improve\n",
    "      learning speed\n",
    "      w(t) := w(t) - (grad(t) + alpha*grad(t-1))\n",
    "    \n",
    "    decrease_const : float (default: 0.0)\n",
    "      Decrease constant. Shrinks the learning rate\n",
    "      after each epoch via eta / (1 + epoch*decrease_const)\n",
    "\n",
    "    shuffle : bool (default: False)\n",
    "      Shuffles training data every epoch if True to prevent circles.\n",
    "\n",
    "    minibatches : int (default: 1)\n",
    "      Divides training data into k minibatches for efficiency.\n",
    "      Normal gradient descent learning if k=1 (default).\n",
    "\n",
    "    random_state : int (default: None)\n",
    "      Set random state for shuffling and initializing the weights.\n",
    "\n",
    "    Attributes\n",
    "    -----------\n",
    "    cost_ : list\n",
    "      Sum of squared errors after each epoch.\n",
    "\n",
    "    \"\"\"\n",
    "    def __init__(self, n_output, n_features, n_hidden=30,\n",
    "                 l1=0.0, l2=0.0, epochs=500, eta=0.001, \n",
    "                 alpha=0.0, decrease_const=0.0, shuffle=True, \n",
    "                 minibatches=1, random_state=None):\n",
    "\n",
    "        np.random.seed(random_state)\n",
    "        self.n_output = n_output\n",
    "        self.n_features = n_features\n",
    "        self.n_hidden = n_hidden\n",
    "        self.w1, self.w2 = self._initialize_weights()\n",
    "        self.l1 = l1\n",
    "        self.l2 = l2\n",
    "        self.epochs = epochs\n",
    "        self.eta = eta\n",
    "        self.alpha = alpha\n",
    "        self.decrease_const = decrease_const\n",
    "        self.shuffle = shuffle\n",
    "        self.minibatches = minibatches\n",
    "\n",
    "    def _encode_labels(self, y, k):\n",
    "        \"\"\"Encode labels into one-hot representation\n",
    "\n",
    "        Parameters\n",
    "        ------------\n",
    "        y : array, shape = [n_samples]\n",
    "            Target values.\n",
    "\n",
    "        Returns\n",
    "        -----------\n",
    "        onehot : array, shape = (n_labels, n_samples)\n",
    "\n",
    "        \"\"\"\n",
    "        onehot = np.zeros((k, y.shape[0]))\n",
    "        for idx, val in enumerate(y):\n",
    "            onehot[val, idx] = 1.0\n",
    "        return onehot\n",
    "\n",
    "    def _initialize_weights(self):\n",
    "        \"\"\"Initialize weights with small random numbers.\"\"\"\n",
    "        w1 = np.random.uniform(-1.0, 1.0, size=self.n_hidden*(self.n_features + 1))\n",
    "        w1 = w1.reshape(self.n_hidden, self.n_features + 1)\n",
    "        w2 = np.random.uniform(-1.0, 1.0, size=self.n_output*(self.n_hidden + 1))\n",
    "        w2 = w2.reshape(self.n_output, self.n_hidden + 1)\n",
    "        return w1, w2\n",
    "\n",
    "    def _sigmoid(self, z):\n",
    "        \"\"\"Compute logistic function (sigmoid)\n",
    "\n",
    "        Uses scipy.special.expit to avoid overflow\n",
    "        error for very small input values z.\n",
    "\n",
    "        \"\"\"\n",
    "        # return 1.0 / (1.0 + np.exp(-z))\n",
    "        return expit(z)\n",
    "\n",
    "    def _sigmoid_gradient(self, z):\n",
    "        \"\"\"Compute gradient of the logistic function\"\"\"\n",
    "        sg = self._sigmoid(z)\n",
    "        return sg * (1 - sg)\n",
    "\n",
    "    def _add_bias_unit(self, X, how='column'):\n",
    "        \"\"\"Add bias unit (column or row of 1s) to array at index 0\"\"\"\n",
    "        if how == 'column':\n",
    "            X_new = np.ones((X.shape[0], X.shape[1]+1))\n",
    "            X_new[:, 1:] = X\n",
    "        elif how == 'row':\n",
    "            X_new = np.ones((X.shape[0]+1, X.shape[1]))\n",
    "            X_new[1:, :] = X\n",
    "        else:\n",
    "            raise AttributeError('`how` must be `column` or `row`')\n",
    "        return X_new\n",
    "\n",
    "    def _feedforward(self, X, w1, w2):\n",
    "        \"\"\"Compute feedforward step\n",
    "\n",
    "        Parameters\n",
    "        -----------\n",
    "        X : array, shape = [n_samples, n_features]\n",
    "          Input layer with original features.\n",
    "\n",
    "        w1 : array, shape = [n_hidden_units, n_features]\n",
    "          Weight matrix for input layer -> hidden layer.\n",
    "\n",
    "        w2 : array, shape = [n_output_units, n_hidden_units]\n",
    "          Weight matrix for hidden layer -> output layer.\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        a1 : array, shape = [n_samples, n_features+1]\n",
    "          Input values with bias unit.\n",
    "\n",
    "        z2 : array, shape = [n_hidden, n_samples]\n",
    "          Net input of hidden layer.\n",
    "\n",
    "        a2 : array, shape = [n_hidden+1, n_samples]\n",
    "          Activation of hidden layer.\n",
    "\n",
    "        z3 : array, shape = [n_output_units, n_samples]\n",
    "          Net input of output layer.\n",
    "\n",
    "        a3 : array, shape = [n_output_units, n_samples]\n",
    "          Activation of output layer.\n",
    "\n",
    "        \"\"\"\n",
    "        a1 = self._add_bias_unit(X, how='column')\n",
    "        z2 = w1.dot(a1.T)\n",
    "        a2 = self._sigmoid(z2)\n",
    "        a2 = self._add_bias_unit(a2, how='row')\n",
    "        z3 = w2.dot(a2)\n",
    "        a3 = self._sigmoid(z3)\n",
    "        return a1, z2, a2, z3, a3\n",
    "\n",
    "    def _L2_reg(self, lambda_, w1, w2):\n",
    "        \"\"\"Compute L2-regularization cost\"\"\"\n",
    "        return (lambda_/2.0) * (np.sum(w1[:, 1:] ** 2) + np.sum(w2[:, 1:] ** 2))\n",
    "\n",
    "    def _L1_reg(self, lambda_, w1, w2):\n",
    "        \"\"\"Compute L1-regularization cost\"\"\"\n",
    "        return (lambda_/2.0) * (np.abs(w1[:, 1:]).sum() + np.abs(w2[:, 1:]).sum())\n",
    "\n",
    "    def _get_cost(self, y_enc, output, w1, w2):\n",
    "        \"\"\"Compute cost function.\n",
    "\n",
    "        y_enc : array, shape = (n_labels, n_samples)\n",
    "          one-hot encoded class labels.\n",
    "\n",
    "        output : array, shape = [n_output_units, n_samples]\n",
    "          Activation of the output layer (feedforward)\n",
    "\n",
    "        w1 : array, shape = [n_hidden_units, n_features]\n",
    "          Weight matrix for input layer -> hidden layer.\n",
    "\n",
    "        w2 : array, shape = [n_output_units, n_hidden_units]\n",
    "          Weight matrix for hidden layer -> output layer.\n",
    "\n",
    "        Returns\n",
    "        ---------\n",
    "        cost : float\n",
    "          Regularized cost.\n",
    "\n",
    "        \"\"\"\n",
    "        term1 = -y_enc * (np.log(output))\n",
    "        term2 = (1 - y_enc) * np.log(1 - output)\n",
    "        cost = np.sum(term1 - term2)\n",
    "        L1_term = self._L1_reg(self.l1, w1, w2)\n",
    "        L2_term = self._L2_reg(self.l2, w1, w2)\n",
    "        cost = cost + L1_term + L2_term\n",
    "        return cost\n",
    "\n",
    "    def _get_gradient(self, a1, a2, a3, z2, y_enc, w1, w2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "\n",
    "        Parameters\n",
    "        ------------\n",
    "        a1 : array, shape = [n_samples, n_features+1]\n",
    "          Input values with bias unit.\n",
    "\n",
    "        a2 : array, shape = [n_hidden+1, n_samples]\n",
    "          Activation of hidden layer.\n",
    "\n",
    "        a3 : array, shape = [n_output_units, n_samples]\n",
    "          Activation of output layer.\n",
    "\n",
    "        z2 : array, shape = [n_hidden, n_samples]\n",
    "          Net input of hidden layer.\n",
    "\n",
    "        y_enc : array, shape = (n_labels, n_samples)\n",
    "          one-hot encoded class labels.\n",
    "\n",
    "        w1 : array, shape = [n_hidden_units, n_features]\n",
    "          Weight matrix for input layer -> hidden layer.\n",
    "\n",
    "        w2 : array, shape = [n_output_units, n_hidden_units]\n",
    "          Weight matrix for hidden layer -> output layer.\n",
    "\n",
    "        Returns\n",
    "        ---------\n",
    "\n",
    "        grad1 : array, shape = [n_hidden_units, n_features]\n",
    "          Gradient of the weight matrix w1.\n",
    "\n",
    "        grad2 : array, shape = [n_output_units, n_hidden_units]\n",
    "            Gradient of the weight matrix w2.\n",
    "\n",
    "        \"\"\"\n",
    "        # backpropagation\n",
    "        sigma3 = a3 - y_enc\n",
    "        z2 = self._add_bias_unit(z2, how='row')\n",
    "        sigma2 = w2.T.dot(sigma3) * self._sigmoid_gradient(z2)\n",
    "        sigma2 = sigma2[1:, :]\n",
    "        grad1 = sigma2.dot(a1)\n",
    "        grad2 = sigma3.dot(a2.T)\n",
    "\n",
    "        # regularize\n",
    "        grad1[:, 1:] += (w1[:, 1:] * (self.l1 + self.l2))\n",
    "        grad2[:, 1:] += (w2[:, 1:] * (self.l1 + self.l2))\n",
    "\n",
    "        return grad1, grad2\n",
    "\n",
    "    def _gradient_checking(self, X, y_enc, w1, w2, epsilon, grad1, grad2):\n",
    "        \"\"\" Apply gradient checking (for debugging only)\n",
    "\n",
    "        Returns\n",
    "        ---------\n",
    "        relative_error : float\n",
    "          Relative error between the numerically\n",
    "          approximated gradients and the backpropagated gradients.\n",
    "\n",
    "        \"\"\"\n",
    "        num_grad1 = np.zeros(np.shape(w1))\n",
    "        epsilon_ary1 = np.zeros(np.shape(w1))\n",
    "        for i in range(w1.shape[0]):\n",
    "            for j in range(w1.shape[1]):\n",
    "                epsilon_ary1[i, j] = epsilon\n",
    "                a1, z2, a2, z3, a3 = self._feedforward(X, w1 - epsilon_ary1, w2)\n",
    "                cost1 = self._get_cost(y_enc, a3, w1-epsilon_ary1, w2)\n",
    "                a1, z2, a2, z3, a3 = self._feedforward(X, w1 + epsilon_ary1, w2)\n",
    "                cost2 = self._get_cost(y_enc, a3, w1 + epsilon_ary1, w2)\n",
    "                num_grad1[i, j] = (cost2 - cost1) / (2 * epsilon)\n",
    "                epsilon_ary1[i, j] = 0\n",
    "\n",
    "        num_grad2 = np.zeros(np.shape(w2))\n",
    "        epsilon_ary2 = np.zeros(np.shape(w2))\n",
    "        for i in range(w2.shape[0]):\n",
    "            for j in range(w2.shape[1]):\n",
    "                epsilon_ary2[i, j] = epsilon\n",
    "                a1, z2, a2, z3, a3 = self._feedforward(X, w1, w2 - epsilon_ary2)\n",
    "                cost1 = self._get_cost(y_enc, a3, w1, w2 - epsilon_ary2)\n",
    "                a1, z2, a2, z3, a3 = self._feedforward(X, w1, w2 + epsilon_ary2)\n",
    "                cost2 = self._get_cost(y_enc, a3, w1, w2 + epsilon_ary2)\n",
    "                num_grad2[i, j] = (cost2 - cost1) / (2 * epsilon)\n",
    "                epsilon_ary2[i, j] = 0\n",
    "\n",
    "        num_grad = np.hstack((num_grad1.flatten(), num_grad2.flatten()))\n",
    "        grad = np.hstack((grad1.flatten(), grad2.flatten()))\n",
    "        norm1 = np.linalg.norm(num_grad - grad)\n",
    "        norm2 = np.linalg.norm(num_grad)\n",
    "        norm3 = np.linalg.norm(grad)\n",
    "        relative_error = norm1 / (norm2 + norm3)\n",
    "        return relative_error\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict class labels\n",
    "\n",
    "        Parameters\n",
    "        -----------\n",
    "        X : array, shape = [n_samples, n_features]\n",
    "          Input layer with original features.\n",
    "\n",
    "        Returns:\n",
    "        ----------\n",
    "        y_pred : array, shape = [n_samples]\n",
    "          Predicted class labels.\n",
    "\n",
    "        \"\"\"\n",
    "        if len(X.shape) != 2:\n",
    "            raise AttributeError('X must be a [n_samples, n_features] array.\\n'\n",
    "                                 'Use X[:,None] for 1-feature classification,'\n",
    "                                 '\\nor X[[i]] for 1-sample classification')\n",
    "\n",
    "        a1, z2, a2, z3, a3 = self._feedforward(X, self.w1, self.w2)\n",
    "        y_pred = np.argmax(z3, axis=0)\n",
    "        return y_pred\n",
    "\n",
    "    def fit(self, X, y, print_progress=False):\n",
    "        \"\"\" Learn weights from training data.\n",
    "\n",
    "        Parameters\n",
    "        -----------\n",
    "        X : array, shape = [n_samples, n_features]\n",
    "          Input layer with original features.\n",
    "\n",
    "        y : array, shape = [n_samples]\n",
    "          Target class labels.\n",
    "\n",
    "        print_progress : bool (default: False)\n",
    "          Prints progress as the number of epochs\n",
    "          to stderr.\n",
    "\n",
    "        Returns:\n",
    "        ----------\n",
    "        self\n",
    "\n",
    "        \"\"\"\n",
    "        self.cost_ = []\n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        y_enc = self._encode_labels(y, self.n_output)\n",
    "\n",
    "        delta_w1_prev = np.zeros(self.w1.shape)\n",
    "        delta_w2_prev = np.zeros(self.w2.shape)\n",
    "\n",
    "        for i in range(self.epochs):\n",
    "            \n",
    "            # adaptive learning rate\n",
    "            self.eta /= (1 + self.decrease_const*i)\n",
    "\n",
    "            if print_progress:\n",
    "                sys.stderr.write('\\rEpoch: %d/%d' % (i+1, self.epochs))\n",
    "                sys.stderr.flush()\n",
    "\n",
    "            if self.shuffle:\n",
    "                idx = np.random.permutation(y_data.shape[0])\n",
    "                X_data, y_data = X_data[idx], y_data[idx]\n",
    "\n",
    "            mini = np.array_split(range(y_data.shape[0]), self.minibatches)\n",
    "            for idx in mini:\n",
    "\n",
    "                # feedforward\n",
    "                a1, z2, a2, z3, a3 = self._feedforward(X[idx], self.w1, self.w2)\n",
    "                cost = self._get_cost(y_enc=y_enc[:, idx],\n",
    "                                      output=a3,\n",
    "                                      w1=self.w1,\n",
    "                                      w2=self.w2)\n",
    "                self.cost_.append(cost)\n",
    "\n",
    "                # compute gradient via backpropagation\n",
    "                grad1, grad2 = self._get_gradient(a1=a1, a2=a2,\n",
    "                                                  a3=a3, z2=z2,\n",
    "                                                  y_enc=y_enc[:, idx],\n",
    "                                                  w1=self.w1,\n",
    "                                                  w2=self.w2)\n",
    "                \n",
    "                ## start gradient checking\n",
    "                grad_diff = self._gradient_checking(X=X[idx], y_enc=y_enc[:, idx],\n",
    "                                                  w1=self.w1, w2=self.w2,\n",
    "                                                  epsilon=1e-5,\n",
    "                                                  grad1=grad1, grad2=grad2)\n",
    "                \n",
    "                if grad_diff <= 1e-7:\n",
    "                    print('Ok: %s' % grad_diff)\n",
    "                elif grad_diff <= 1e-4:\n",
    "                    print('Warning: %s' % grad_diff)\n",
    "                else:\n",
    "                    print('PROBLEM: %s' % grad_diff)\n",
    "\n",
    "                # update weights; [alpha * delta_w_prev] for momentum learning\n",
    "                delta_w1, delta_w2 = self.eta * grad1, self.eta * grad2\n",
    "                self.w1 -= (delta_w1 + (self.alpha * delta_w1_prev))\n",
    "                self.w2 -= (delta_w2 + (self.alpha * delta_w2_prev))\n",
    "                delta_w1_prev, delta_w2_prev = delta_w1, delta_w2\n",
    "\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn_check = MLPGradientCheck(n_output=10, \n",
    "                            n_features=X_train.shape[1], \n",
    "                            n_hidden=10, \n",
    "                            l2=0.0, \n",
    "                            l1=0.0, \n",
    "                            epochs=10, \n",
    "                            eta=0.001,\n",
    "                            alpha=0.0,\n",
    "                            decrease_const=0.0,\n",
    "                            minibatches=1, \n",
    "                            random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ok: 2.56712936241e-10\n",
      "Ok: 2.94603251069e-10\n",
      "Ok: 2.37615620231e-10\n",
      "Ok: 2.43469423226e-10\n",
      "Ok: 3.37872073158e-10\n",
      "Ok: 3.63466384861e-10\n",
      "Ok: 2.22472120785e-10\n",
      "Ok: 2.33163708438e-10\n",
      "Ok: 3.44653686551e-10\n",
      "Ok: 2.17161707211e-10\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<__main__.MLPGradientCheck at 0x10a13ab70>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn_check.fit(X_train[:5], y_train[:5], print_progress=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}