examples/supervised/backpropagation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backpropagation\n",
    "(In the context of fully-connected neural networks)  \n",
    "\n",
    "**Mathematical background:**  \n",
    "Following the most common literature notation, exponents denote layers and subscripts denote vector components. In case of weights they denote the weight connecting a node of the previous layer to a node of the next (written backwards), i.e. $w_{jk}^{l}$ is the weight connecting the node $k$ of layer $l-1$ to the node $j$ of layer $l$.\n",
    "\n",
    "<img src=\"./neural-network.png\" />\n",
    "\n",
    "Activation: $a_j^{l+1} = \\sigma(z_j^{l+1}) = \\sigma\\left(\\sum\\limits_{k=0}^{n_{l}} w_{jk}^{l+1} a_k^l\\right)$  \n",
    "where the bias is incorporated in the weights after augmenting $a^l$ by an extra component equal to 1.\n",
    "\n",
    "The cost function $C$ is a fuction (MSE, Cross-entropy, etc) of the final activation layer and the true labels: $C = C(a^L, y)$  \n",
    "What we are looking for is the derivative of the cost w.r.t. the weights:  \n",
    "\n",
    "$\n",
    "\\large \\frac{ \\partial C }{ \\partial w_{jk}^{l} } = \\frac{ \\partial C }{ \\partial a_{j}^{l} } \\cdot \\frac{ \\partial a_j^l }{ \\partial z_{j}^{l} } \\cdot \\frac{ \\partial z_j^l }{ \\partial w_{jk}^{l} } = \\frac{ \\partial C }{ \\partial a_{j}^{l} } \\cdot \\sigma^\\prime(z_{j}^{l}) \\cdot a_{k}^{l-1}\n",
    "$  \n",
    "\n",
    "All derivatives are easy to compute except $\\large \\frac{ \\partial C }{ \\partial a_{j}^{l} }$. For this one though we can find a recursive formula by applying the chain rule:  \n",
    "\n",
    "$\\large \\frac{ \\partial C }{ \\partial a_{k}^{l} } = \\sum\\limits_{j} \\frac{ \\partial C }{ \\partial a_{j}^{l+1} } \\cdot \\frac{ \\partial a_{j}^{l+1} }{ \\partial z_{j}^{l+1} } \\cdot \\frac{ \\partial z_{j}^{l+1} }{ \\partial a_{k}^{l} } = \\sum\\limits_{j} \\frac{ \\partial C }{ \\partial a_{j}^{l+1} } \\cdot \\sigma^\\prime(z_{j}^{l+1}) \\cdot w_{jk}^{l+1}\n",
    "$ \n",
    "\n",
    "\n",
    "The summation combines the effect of the node $k$ of layer $l$ to all the nodes of layer $l+1$.  \n",
    "\n",
    "This is it.  \n",
    "We expressed the derivative we are interested in, in terms of the derivative of the next layer. So by starting from the last layer and going backwards we can calculate all needed derivatives.  \n",
    "\n",
    "What we finally need is these equations in vector/matrix form so that our implementation is efficient.  \n",
    "Given the weight matrix:  \n",
    "$\n",
    "W^{l+1} = (w_{jk}^{l+1}) = \n",
    "\\begin{bmatrix}\n",
    "w_{11} & w_{12} & \\cdots w_{1n_{l}} \\\\\n",
    "w_{21} & w_{22} & \\cdots w_{1n_{l}} \\\\\n",
    "\\vdots & & \\vdots \\\\\n",
    "w_{n_{l+1}1} & w_{n_{l+1}2} & \\cdots  w_{n_{l+1}n_{l}} \\\\\n",
    "\\end{bmatrix}\n",
    "$\n",
    "\n",
    "we can write $\\nabla_{a^{l}} C = [\\frac{ \\partial C }{ \\partial a_{1}^{l} }, \\frac{ \\partial C }{ \\partial a_{2}^{l} }, \\cdots]^T$ as:\n",
    "\n",
    "$\n",
    "\\large \\nabla_{a^{l}} C = (W^{l+1})^T \\cdot (\\nabla_{a^{l+1}} C) \\odot \\sigma^\\prime(z^{l+1})\n",
    "$\n",
    "\n",
    "Finally, the scalar-by-matrix derivative of cost w.r.t. the weight matrix is:  \n",
    "$\n",
    "\\large \\nabla_{W^{l}} C = ( (\\nabla_{a^{l}} C) \\odot\\sigma^\\prime(z^{l}) ) \\cdot (a^{l-1})^T\n",
    "$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:36:02.367259Z",
     "start_time": "2020-04-12T18:36:02.338763Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container {width: 98%}</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>.container {width: 98%}</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:36:03.504258Z",
     "start_time": "2020-04-12T18:36:02.655760Z"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:39:38.838989Z",
     "start_time": "2020-04-12T18:39:37.570488Z"
    }
   },
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import datasets\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.base import BaseEstimator, ClassifierMixin\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "import sys\n",
    "sys.path.insert(0, \"../\")\n",
    "from fromscratch.supervised.neuralnetworks.nn import NN, Linear\n",
    "from fromscratch.supervised.neuralnetworks.loss import Loss, mse, crossentropy, softmax\n",
    "from fromscratch.supervised.neuralnetworks.activations import relu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create some synthetic data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:38:23.434893Z",
     "start_time": "2020-04-12T18:38:22.342889Z"
    }
   },
   "outputs": [],
   "source": [
    "N = 100\n",
    "dim = 1\n",
    "X = np.random.random((N, dim))\n",
    "X = np.sort(X, axis=0)\n",
    "y = 2*X*X + 1  # for regression\n",
    "y = y.T[0]\n",
    "\n",
    "y_c = ((y > y.mean()) & (y < 0.8*y.max())).astype(int)  # for classification\n",
    "y_c_oh = OneHotEncoder(sparse=False).fit_transform(y_c[:, None])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:38:25.269331Z",
     "start_time": "2020-04-12T18:38:24.305347Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fead-forward Neural Network\n",
      "---------------------------\n",
      "4 layers: [10, 10, 10, 1]\n",
      "\n",
      "       Inputs\n",
      "       \\ | /\n",
      " o o o o o o o o o o \n",
      "       \\ | /\n",
      " o o o o o o o o o o \n",
      "       \\ | /\n",
      " o o o o o o o o o o \n",
      "          | \n",
      "          o \n",
      "       Outputs\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net1 = NN(layers=[\n",
    "    Linear(10, input_size=2),\n",
    "    Linear(10),\n",
    "    Linear(10),\n",
    "    Linear(1)\n",
    "])\n",
    "net1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:38:32.653714Z",
     "start_time": "2020-04-12T18:38:31.713217Z"
    }
   },
   "outputs": [],
   "source": [
    "loss = Loss(net1, mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:38:37.823701Z",
     "start_time": "2020-04-12T18:38:34.573241Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1c146000cc0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd3xUVfrH8c8zQwoqxQXEAggqikhTQYm0aJCligKuWGBVNAriKlKU3bWBCsouoEuR/AARC8gCIii9RNqgBKWjLCpgAKUp0kKSmef3x52EEBIywCSTTJ7365VXZu69M/PcAN8czj33HFFVjDHGhC9XqAswxhiTvyzojTEmzFnQG2NMmLOgN8aYMGdBb4wxYa5EqAvISfny5bVq1aqhLsMYY4qMNWvW7FfVCjntK5RBX7VqVZKSkkJdhjHGFBkisiO3fdZ1Y4wxYc6C3hhjwlyeQS8i0SLytYisE5FNIvJqDseIiLwjIttEZL2I3JRlX0sR+d6/74Vgn4AxxpgzC6SP/gRwh6oeEZEIYLmIzFHVVVmOaQVU93/dCowGbhURNzASuBNIBlaLyExV3Xy2haalpZGcnExKSsrZvtSchejoaCpVqkRERESoSzHGBEmeQa/OZDhH/E8j/F/ZJ8hpD0z0H7tKRMqKyGVAVWCbqv4IICKT/ceeddAnJydTqlQpqlatioic7ctNAFSVAwcOkJycTLVq1UJdjjEmSALqoxcRt4isBfYCC1T1q2yHXAH8nOV5sn9bbttz+ox4EUkSkaR9+/adtj8lJYVy5cpZyOcjEaFcuXL2vyZjwkxAQa+qXlWtB1QCbhGRWtkOySl99Qzbc/qMBFWtr6r1K1TIcSiohXwBsJ+xMaHh8XgYNGgQHo8n6O99VqNuVPV3IBFomW1XMlA5y/NKwO4zbDfGGOOXkJBA38aNOfz3v9M/NjboYR/IqJsKIlLW/7gk0Bz4LtthM4Gu/tE3DYFDqroHWA1UF5FqIhIJdPYfa84gNjbWbhgzppjweDx80707iT4frwGzU1P538SJQf2MQEbdXAa87x9B4wKmqOrnIvIkgKq+C8wGWgPbgGPAI/596SLSE5gHuIHxqropqGdgjDFF2HfvvccInw83Tl93JNAsyJ+RZ4teVder6o2qWkdVa6nqAP/2d/0hjzqeUtWrVbW2qiZlef1sVb3Wv+/1INd/RsHs8zp69Cht2rShbt261KpVi08++YQBAwbQoEEDatWqRXx8PBmrdcXGxtKrVy+aNm3K9ddfz+rVq+nQoQPVq1fnn//8JwDbt2+nRo0a/PWvf6VOnTp06tSJY8eOnfa58+fPJyYmhptuuol7772XI0ecAVAvvPACNWvWpE6dOvTp0+e8z88YExpXzJ+fGfIKiMvFlV27BvUzCuVcN3l59tlnWbt27RmPOXToEOvXr8fn8+FyuahTpw5lypTJ9fh69eoxfPjwXPfPnTuXyy+/nC+++CLz/e+8805eeuklALp06cLnn39Ou3btAIiMjGTp0qW8/fbbtG/fnjVr1vCnP/2Jq6++ml69egHw/fffM27cOBo1asSjjz7KqFGjTgnt/fv389prr7Fw4UIuvPBC3nzzTYYOHUrPnj359NNP+e677xARfv/998B+cMaYQmXGyy9z544dKP5RKm437lGjICYmqJ8TtlMgHDp0CJ/PB4DP5+PQoUPn9X61a9dm4cKFPP/88yxbtowyZcqwZMkSbr31VmrXrs3ixYvZtOlkr9Rdd92V+bobbriByy67jKioKK666ip+/tkZcVq5cmUaNWoEwEMPPcTy5ctP+cxVq1axefNmGjVqRL169Xj//ffZsWMHpUuXJjo6mscee4zp06dzwQUXnNe5GWMK3rqpU4kZMIAj0dHotGm43ngD17JlEB8f9M8qki36M7W8M3g8HuLi4khNTSUyMpKPPvqImPP4LXnttdeyZs0aZs+eTf/+/WnRogUjR44kKSmJypUr88orr5wy/jwqKgoAl8uV+TjjeXp6OnD6UMbsz1WVO++8k0mTJp1Wz9dff82iRYuYPHkyI0aMYPHixed8bsaYgrX6pZe4ZuBAXIBvyRLcDRvm6+eFbYs+JiaGRYsWMXDgQBYtWnReIQ+we/duLrjgAh566CH69OnDN998A0D58uU5cuQIU6dOPev33LlzZ+b1g0mTJtG4ceNT9jds2JAVK1awbds2AI4dO8bWrVs5cuQIhw4donXr1gwfPjzPbixjTCHh8bCndWtuHjiQskAUkLx+fb5/bJFs0QcqJibmvAM+w4YNG+jbty8ul4uIiAhGjx7NjBkzqF27NlWrVqVBgwZn/Z7XX38977//Pk888QTVq1ene/fup+yvUKECEyZM4P777+fEiRMAvPbaa5QqVYr27duTkpKCqjJs2LCgnKMxJh95PHhvv51L/f+WBWco4oFp0/KluyYryRgpUpjUr19fs48j37JlC9dff32IKgq+7du307ZtWzZu3BjqUk4Tbj9rYwqDnU88QaWEBFw4F159ODNG/jBmDLWDEPQiskZV6+e0L2y7bowxptDw+Tg8Zw4uwIsT8HMqVQpayOfFgj5EqlatWihb88aYIFu5ktRbb+WGn39mDPCSCC0jIyk3ZUqBhDyEeR+9McaElMeDr2lTIr1e0oAqL77IwZIlGRQbG7Trh4GwoDfGmHxy4PnnKef1Zj6vuW8frUaPLvA6rOvGGGPyw9SpXLxsGV4gzf/1ZYhKsaA3xphgW7wY3wMP4MFZR/UVEVpHRlI9yHPYBMqCPgQSExNp27YtADNnzmTw4MG5Hvv7778zatSozOe7d++mU6dO+V6jMeYcjR+Pr2VLdni99LnuOv6+YAEXvf46gxITC7RfPivrow8ir9eL2+0+q9fcddddmfPi5CQj6Hv06AHA5Zdffk534RpjCsDkyWi3bghwKfDfgQOp1Lw5zZs3D2lZ4d2i93hg0CDn+3nKbVrhqlWrMmDAABo3bsx///vfXKcVnjt3LjVq1KBx48ZMnz49830nTJhAz549Afj111+55557qFu3LnXr1mXlypW88MIL/PDDD9SrV4++ffuyfft2atVyVnJMSUnhkUceoXbt2tx4440sWbIk8z07dOhAy5YtqV69Ov369QOcX0QPP/wwtWrVonbt2nZHrTHBlJxM2uOPA85dryUAbyGZg6potuiffRbymt/l0CFYvx58PnC5oE4dOMM0xdSrB3lMlpbTtMIA0dHRLF++nP3799OhQ4fTphXu168fjz/+OIsXL+aaa67hvvvuy/H9//a3v9GsWTM+/fRTvF4vR44cYfDgwWzcuDFzPpvt27dnHj9y5EjAmZ7hu+++o0WLFmzduhWAtWvX8u233xIVFcV1113H008/zd69e9m1a1fm+H2b3tiYIDlwAP78Z9KPHcOLM7VBxsXX0PTKnyqQpQQri8gSEdkiIptE5JkcjukrImv9XxtFxCsif/Lv2y4iG/z7Cm59vEOHnJAH5/t5TlMMuU8rnBHcuU0r/N1331GtWjWqV6+OiPDQQw/l+P6LFy/OnO/G7Xafcf58gOXLl9OlSxcAatSowZVXXpkZ9HFxcZQpU4bo6Ghq1qzJjh07uOqqq/jxxx95+umnmTt3LqVLlz7vn4kxxd7Ro9C2Ld6tW2nl89Hc5Qr5xdfsAmnRpwO9VfUbESkFrBGRBaq6OeMAVR0CDAEQkXZAL1U9mOU9blfV/UGrOoBpivF4IC4OUlMhMhI++ui8J/PPbVrhCy+8EMh9WuG1a9ee9tpgONM8RVmnRna73aSnp3PxxRezbt065s2bx8iRI5kyZQrjx48Pel3GFBupqdCxI/r119wnQom4OAa+/DLLly8v8JuiziSQpQT3qOo3/seHgS3AFWd4yf3A6ROoF7SYGFi0CAYOdL4H4Qd+rtMK16hRg59++okffvgh87U5iYuLY7T/Zgqv18sff/xBqVKlOHz4cI7HN23alI8++giArVu3snPnTq677rpc69+/fz8+n4+OHTsycODAzKmWjTHnwOeDhx+GefPodcEFbLj6av773//SpEkT+vfvX2hCHs7yYqyIVAVuBL7KZf8FQEtgWpbNCswXkTUikuvEDiISLyJJIpK0b9++sykrdzEx0L9/0JblyphWuE6dOhw8ePCM0wrXqVOHhg0b8t133xEdHU1CQgJt2rShcePGXHnllTm+/9tvv82SJUuoXbs2N998M5s2baJcuXI0atSIWrVq0bdv31OO79GjB16vl9q1a3PfffcxYcKEU1ry2e3atYvY2Fjq1avHww8/zKBBg87/h2JMcaQKzzwDkyYx9JJLeL9ECWbNmsXFF18c6spypqoBfQEXAWuADmc45j5gVrZtl/u/XwKsA5rm9Vk333yzZrd58+bTthWkn376SW+44YaQ1lBQQv2zNqbQe+UVVdBp1aqp2+XShQsXhroiBZI0l0wNqEUvIhE4rfSPVHX6GQ7tTLZuG1Xd7f++F/gUuCXQX0LGGFOoeDzQvj288gpf16xJx59+YsTIkcTFxYW6sjPK82KsOFcRxwFbVHXoGY4rAzQDHsqy7ULApaqH/Y9bAAPOu+oQsGmFjSnmPB6IjYXUVHwiPLd5M3/729948sknQ11ZngIZddMI6AJsEJGMwet/B6oAqOq7/m33APNV9WiW11YEPvWPOCkBfKyqc8+1WFXNl9Er5iQthCuOGVMovP46mpqKAF5VulWvTpd//zvUVQUkz6BX1eU4N3rlddwEYEK2bT8Cdc+xtlNER0dz4MABypUrZ2GfT1SVAwcOEB0dHepSjCk8PB4YMgT94gvAGV3iBW546ilKlCga95wWjSqBSpUqkZycTNBG5JgcRUdHU6lSpVCXYUzh4PHAHXdASgrgtHi9OC3a344dKzIXHItM0EdERFCtWrVQl2GMKS48HnjhhcyQh5PrvU6KiGBwbGyoKjtrRSbojTGmwGRpySvgw5m75j3gA6But26F6oaovFjQG2NMdmPHZrbkvcBC4FXga5eLqKgo/l1I5rAJVHhPU2yMMWfryy/hvfdQnAuv6ZwM+ebNm7No0aIi1ZoHC3pjjDnJ64WePUEVwemyeY+TLflXXnmlyIU8WNAbY4xj5Uq47TbYuJFUnD75Ezh98kW1JZ/B+uiNMcbjgWbNID0dr8vFUz4f5YFEYE1EBF8W0ZZ8Bgt6Y0zx5vHAo49CejoA6g/5N0Vwu92MHDGiSIc8WNAbY4qzLPPXZAyjTAWuf/JJXq9ShdhCtHjI+bCgN8YUX2PGOKtEcXIY5etuN2917UrXMAj4DHYx1hhTPHk8MGkSKkI6Tkv+VcADJCYmhrS0YLMWvTGm+Nm0Cdq04fgll9B5zx5qer0k4h9GGRlJbBGa3iAQ1qI3xhQv27dDixYQHc2rjRox0+tlMEX7hqi8WNAbY4qPL76A+vXhjz/YOmIEIz7/HPGPrinKN0TlJc+gF5HKIrJERLaIyCYReSaHY2JF5JCIrPV/vZRlX0sR+V5EtonIC8E+AWOMCcjChdCuHRw4gKam0ueJJyhTpgzTpk1j4MCBYdmSzxBIH3060FtVvxGRUsAaEVmgqpuzHbdMVdtm3SAibmAkcCeQDKwWkZk5vNYYY/LPyJHQpw/4V1DzpqZy8+HDvL56NbVr1w5xcfkvzxa9qu5R1W/8jw8DW4ArAnz/W4BtqvqjqqYCk4H251qsMcactdGjnflr/FMOe3FG2LT797+LRcjDWfbRi0hV4Ebgqxx2x4jIOhGZIyI3+LddAfyc5ZhkcvklISLxIpIkIkm2ipQxJihU4fXXT9n0A9CyRAlO3HRTaGoKgYCDXkQuAqYBz6rqH9l2fwNcqap1gf8AMzJelsNb5bj6tKomqGp9Va1foUKFQMsyxpjc/eMfsGtX5pTDAEOAlaphN1b+TAIKehGJwAn5j1R1evb9qvqHqh7xP54NRIhIeZwWfOUsh1YCdp931cYYk5dhw2DQILY0aUI8MA+IB8a7XESG4Vj5Mwlk1I0A44Atqjo0l2Mu9R+HiNzif98DwGqguohUE5FIoDMwM1jFG2NMjiZOhOeeg44d6bR3L2OBVsBY4KqrrgrrETY5CWTUTSOgC7BBRNb6t/0dqAKgqu8CnYDuIpIOHAc6q6oC6SLSE+eXqRsYr6qbgnwOxhhz0uefO7NRxsXxyV13sXnatFN29+3bt1iFPICo5thlHlL169fXpKSkUJdhjClqli+HO++EWrVY0L8/re+7j9tuu42//OUvzJw5k44dOxIfHx/qKvOFiKxR1fo57rOgN8aEhQ8+gMcfh4oVWT16NLH33su1115LYmIiZcqUCXV1+e5MQW+Tmhljir6pU6FrVwB8v/7KPzt35vLLL2fu3LnFIuTzYnPdGGOKtl9+gSeeyHzqPXGCWGDBggVUrFgxZGUVJhb0xpii69AhaNkSjh7FGxFBGs6i3n8ZNYqqVauGuLjCw4LeGFM0HT8Od90Fmzez5qWXaJKWxktAq4gI9l59dairK1Qs6I0xRU96OnTuDMuWkZKQQIcxY/AAg4EVPl+xuus1EHYx1hhTtKhCfDzMnEnasGG0/fBDfv75ZyIjI/F6vcXurtdAWNAbY4qW55+H997D+89/0nHxYhYtWsT7779P9erVSUxMJDY2ttjdEJUXC3pjTNExZAgMGYKvRw8e3LqVWbNmMXLkSLr6h1ZawOfMgt4YU/h5PPDvf8O0aehf/sITKSl8MmUKb775Jj169Ah1dYWeBb0xpnDzeCA2FlJTUZeL/3i9jB0/nhdffJF+/fqFuroiwUbdGGMKt/feg9RUAHyq7Jk2jWeffZZXX301xIUVHRb0xpjC69tv4aOPUBG8wAlVSrdrx9ChQ/HPjG4CYF03xpjCads2aNmSE6VK8cCJE1zr9bLM5WJwv34W8mfJWvTGmMJn5kxo0ABSUuhdqxbTvV4GA6tEWLZsWairK3Is6I0xhcuwYdC+Pfz+O95jx1izaBEulwu32203Q52jPLtuRKQyMBG4FPABCar6drZjHgSe9z89AnRX1XX+fduBw4AXSM9tvmRjjGHJEujdO/Oppqfz+DXXMOj//g+Px2M3Q52jQPro04HeqvqNiJQC1ojIAlXdnOWYn4BmqvqbiLQCEoBbs+y/XVX3B69sY0zYSUuD7t1RVQRQ/9cDCQlEx8ZaS/485Nl1o6p7VPUb/+PDwBbgimzHrFTV3/xPVwGVgl2oMSaM+XzQrRt8/z1pOK3LNOAZt5tvo6NDXFzRd1Z99CJSFbgR+OoMh3UD5mR5rsB8EVkjIrku1igi8SKSJCJJ+/btO5uyjDFFmSr06QMffMB/69ShGfAi0Ax412aiDIqAh1eKyEXANOBZVf0jl2Nuxwn6xlk2N1LV3SJyCbBARL5T1aXZX6uqCThdPtSvX7/wLWRrjMkfgwfDsGFsa9OG++fMwSfCKv9a1lF28TUoAgp6EYnACfmPVHV6LsfUAcYCrVT1QMZ2Vd3t/75XRD4FbgFOC3pjTDHj8TiTlH36KTubNqXm3Lk0uOUWBg4cyLRp0wDo2rWrXXwNgkBG3QgwDtiiqkNzOaYKMB3ooqpbs2y/EHCp6mH/4xbAgKBUbowpurLOXyNC1+XLufmWW5g3bx6lS5emefPmoa4wrATSom8EdAE2iMha/7a/A1UAVPVd4CWgHDDKf8daxjDKisCn/m0lgI9VdW5Qz8AYU/SMG4empiJAuir3VarEg/6QN8GXZ9Cr6nLgjPcbq+pjwGM5bP8RqHvO1Rljwk9SEt4PP3RCHmd0Tf0+fSzk85HNdWOMKTjffw+tWnHQ7aYbcAOwzOWizdGjNAh1bWHMgt4YUzCSk6FFC1JOnKDRsWP84HIxW4TIyEiG2MiafGVBb4zJfwcOwJ//TOrevTRKSaHyHXfw7j/+wVdffWXTGhQAC3pjTP46ehTatsW7dSt/Tk/nkpYtmT59OiVLluSOO+4IdXXFgs1eaYzJP6mp0KkTvq++olN6OqXatWPGjBmULFky1JUVKxb0xpj8sWKFM6f83LnEq1KiUyemTp1KVFRUqCsrdqzrxhgTfCtX4mvWDJfXSypwxZ138u6kSZQoYZETCtaiN8YE3e/PPIPL6wWcm3AeqVbNQj6ELOiNMUHlGzmSsklJmTdDpQFLXRY1oWQ/fWNM0Gx//HHo2ZPlQHOXi1dEaB0ZSfWuXUNdWrFm/5cyxpw/j4fdDz7IlT/9BMBNwLN9+rClbFkG2Tj5kLOgN8acH4+H9CZNuCxLn3wkUHbtWvrPmxfS0ozDum6MMedl+5AhuL1eBDLXevUB5Tp2DG1hJpMFvTHmnO1ds4YLZ8wATi7mnQ54HnyQ2vG5rhxqCpgFvTHmnCz/xz+QBg0oqcpjbjfvAgkifNGvH80+/DDU5ZksrI/eGHPWvmjVilZz5yLACaBd795sKVvWJigrpPJs0YtIZRFZIiJbRGSTiDyTwzEiIu+IyDYRWS8iN2XZ11JEvvfveyHYJ2CMKVif9e6dGfKC01osu3Yt/fv3t5AvpALpukkHeqvq9UBD4CkRqZntmFZAdf9XPDAaQETcwEj//prA/Tm81hhTBHg8Hh7u0oVqQ4fiwi68FiWBLCW4B9jjf3xYRLYAVwCbsxzWHpioqgqsEpGyInIZUBXY5l9SEBGZ7D8262uNMYWcx+Ph+SZNeMvrpQ6QitNK9OFceG1mF14LtbPqoxeRqsCNwFfZdl0B/JzlebJ/W07bb83lveNx/jdAlSpVzqYsY0w+UlU+efZZFnm9ROBMaTClaVMqRUdTrmNHC/kiIOBRNyJyETANeFZV/8i+O4eX6Bm2n75RNUFV66tq/QoVKgRaljEmHy1fvpzbbruNuK+/JiLL9iY1axI7b54NoSwiAmrRi0gETsh/pKrTczgkGaic5XklYDfODXI5bTfGFHKJiYk0b96cv3i9tAN8IijgiozkSpu7pkjJM+hFRIBxwBZVHZrLYTOBnv4++FuBQ6q6R0T2AdVFpBqwC+gMPBCc0o0x+eXgwYM88sgjPO318i/gW2BTt248dNVVEBsLNrqmSAmkRd8I6AJsEJG1/m1/B6oAqOq7wGygNbANOAY84t+XLiI9gXmAGxivqpuCegbGmKDxeDyMGDGC2bNn85dDh8ho2dUASjRoANZVUyQFMupmOTn3tWc9RoGnctk3G+cXgTGmEPN4PDRr1oy0tDRqAsP82wWIdrmofeBACKsz58OmQDDGADBmzBjS0tK4G1iJc8dreokS4HYjUVFOl40pkmwKBGOKOY/Hw1tvvcVnn33Gs5DZXXMC2Pncc1xdtqz1yxdxFvTGFGMrVqwgNjaW9PR04jgZ8gJEuVxOyPfvH8IKTTBY0BtTTB09epT4+Hjqp6fTHHjCvz3jgpy43dZdEyYs6I0phnbt2sVdd91Fmc2bWQhE45/SwO1GVMHlghEjrLsmTFjQG1PMfPvtt7Rr145Dhw6xumtXSk6c6ExQJoLr8cehShXrkw8zNurGmGJk1qxZNGnSBJfLxYoVK6iBv6tGBImOhq5dnT55C/mwYkFvTDGwcuVK2rRpw1133UXNmjX56quvqLN4MUycCO3bw2uvwaJFFvBhyrpujAlzo0aNomfPnqgqbrebN998k8sWLoRevaBDB5gyBdzuUJdp8pEFvTFhyuPxMGbMGN5///3MbarKvgkT4KOP4Pbbne8W8mHPgt6YMOTxeLjjjjtISUk5ZXtjETp+8gnUrQszZkB0dIgqNAXJ+uiNCUOjRo06JeQbAu+IsKBECdxXXglz5kDp0qEr0BQoa9EbEyY8Hg9Llizhp59+4sMPP0REcLlc3CbCQlUivF7kxAkYNAguuSTU5ZoCZEFvTBjweDzExcVx/PhxAFq2bMnzzz+Px+Oh27JlRM6Z4xzodsP334ewUhMKFvTGhIFp06ZlhryI0KRJE2Kjooj93/+cbpoMJUrYtAbFUCArTI0H2gJ7VbVWDvv7Ag9meb/rgQqqelBEtgOHAS+Qrqr1g1W4McYxZ84cxowZA4DL5SIqKop25ctDXBz4wx8AEXjkERsrXwwFcjF2AtAyt52qOkRV66lqPaA/8KWqHsxyyO3+/RbyxgTRihUraNGiBa1bt+aaa65h6tSpvPbaayxatIjau3adHvIZd76aYieQFaaWikjVAN/vfmDS+RRkjMnbwoULadmyJV6vF7fbzdChQ7n99tudnQcPwqOPOo9dLqe75tFHnZC31nyxFLQ+ehG5AKfl3zPLZgXmi4gCY1Q14QyvjwfiAapUqRKssowJO5s2beL+++/H6/Vmblu1ahW3R0fD55/D5MmwaxcMGQJpaTZBmQnqxdh2wIps3TaNVHW3iFwCLBCR71R1aU4v9v8SSACoX7++BrEuY8KCx+Ph7bffZsaMGVx44YVERUWRnp5OZGQkbcuVgzvugIyx8++8A08/HdqCTaERzKDvTLZuG1Xd7f++V0Q+BW4Bcgx6Y0zuli5dSlxcHOnp6bhcLiZMmED58uVJTEwkNjaW2sOHnwx5lwuOHAltwaZQCUrQi0gZoBnwUJZtFwIuVT3sf9wCGBCMzzOmuPB4PMyYMYMJEyaQnp4OOMMnN27cSP/+/YmpU8dpuU+Z4gS8CERG2hBKc4pAhldOAmKB8iKSDLwMRACo6rv+w+4B5qvq0SwvrQh8KiIZn/Oxqs4NXunGhLeEhAR69OiR2RcfERGBz+cjMjKS2NhY+PBDeO452LcP/vlPaNECli+3PnlzGlEtfN3h9evX16SkpFCXYUzILFu2jNjYWHw+H+CMj4+Pj6dKlSrENmtGzGefwVtvOQdHRcGSJRbuxZyIrMltGLvdGWtMIbNjxw4eeuihzJAHcLvddO3alZiaNSE+3umqyZCeDomJFvQmVzZ7pTGFyIwZM6hXrx779+8nMjISl8tFiRIlGDFiBDGbN0O1ajB1Kjz5JJQs6cxdY33yJg/WojemEEhJSaFv376MGDGCm2++mcmTJ7Nv3z5nVE2zZsRMngz/+Y9zcFSUc/NT165OS9765E0eLOiNCbGtW7dy3333sXbtWnr16sXgwYOJjIzkmn37iPn+e3j3Xdi58+QLMrpqbBFvEyALemNCxOPxMGzYMGbNmnIe+mAAABa4SURBVMUFF1zArFmzaNu2bcZOaNbMubMVnC6aEiWckLeuGnOWLOiNCYEFCxbQqlUrvF4vLpeLyZMnnwx5rxdeeulkyAP4fM7Mk1WqWFeNOWt2MdaYAjZv3jw6deqUOT5eRNi8ebOzc88eZzz8woXODVAZIiOdPnnrrjHnwFr0xhSQP/74g969ezN27FiqVq3KiRMnMueqiY2NhXnznDA/fBjGjYMaNeCDD5wX28yT5jxY0BtTABYsWEC3bt3YtWsXzz//PK+88grffvstiYmJ3N64MQ1nzoTBg+GGG2DxYuc7wG23hbZwExYs6I3JR4cPH6Zv376MGTOG6667jhUrVtCwYUMAYoCY336DHj1g40Z4/HEYPhwuuCC0RZuwY0FvTD5ZtGgR3bp1Y+fOnfTp04cBAwZQsmRJZ6fH41xUTU11ng8YAC++GLJaTXizi7HGBNmRI0d46qmnaN68OZGRkSxfvpwhQ4acDPkTJ6B375MhnzF00ph8YkFvTBAlJiZSp04dRo8eTa9evVi7di23Ze1n/9//nIuqHo8T7jaFgSkA1owwJgiOHj1K//79+c9//sM111zD0qVLady48akHffwxPPGEE+wzZ0L58jaFgSkQFvTGnKdly5bxyCOP8MMPP/DMM8/wxhtvcEHWC6pHj8Lf/gbjx0Pjxk7gV67s7LOANwXAum6MOUfHjh2jV69eNGvWDFUlMTGR4cOHnxryGzdCgwbw3nvO4iBLlpwMeWMKSJ5BLyLjRWSviGzMZX+siBwSkbX+r5ey7GspIt+LyDYReSGYhRsTSitWrKBevXoMHz6cHj16sG7dOpo1a3byAFX4v/9zQv7gQZg/HwYOtIuuJiQCadFPAFrmccwyVa3n/xoAICJuYCTQCqgJ3C8iNc+nWGNC7fjx4/Tu3ZsmTZqQlpbG4sWLGTFiBBdddNHJgw4dgvvvdxYIadIE1q2D5s1DV7Qp9vIMelVdChw8h/e+Bdimqj+qaiowGWh/Du9jTKHg8XioV68eQ4cO5YknnmD9+vXcfvvtpx6UlAQ33eQsDjJoEMydCxUrhqZgY/yC1UcfIyLrRGSOiPjv3eYK4OcsxyT7t+VIROJFJElEkvbt2xeksow5fykpKfTr14/GjRuTkpLCggULGD16NKVKlTp5kCoMG+ZMWZCWBl9+CS+8cOrEZMaESDA6DL8BrlTVIyLSGpgBVAckh2NzXYlcVROABHAWBw9CXcacF4/HwwcffMDs2bPZsWMH8fHxDBkyhNKlS5964IED8PDD8Pnn0L69M7rmT38KSc3G5OS8g15V/8jyeLaIjBKR8jgt+KzDCyoBu8/384zJbx6Ph3HjxjFhwgS8Xi8iwrBhw3j22WdPP3jZMnjgAdi7F955B3r2BMmpjWNM6Jx30IvIpcCvqqoicgtOd9AB4HeguohUA3YBnYEHzvfzjMlPCQkJ9OjRI3OueACXy8Xx48dPPdDrdfrgX34ZrrrKudP1ppsKuFpjApNn0IvIJCAWKC8iycDLQASAqr4LdAK6i0g6cBzorKoKpItIT2Ae4AbGq+qmfDkLY4Jg2rRpPPnkkzh/fR0icnK++Ax79kCXLrBokdOaHz0asnfnGFOISNa/1IVF/fr1NSkpKdRlmGLi+PHjvPXWW7z22mukp6dnbne73Tz++ON07dqVmIw7WOfPd0L+8GEYMcJZ3s+6akwhICJrVLV+Tvvs7g1TLHk8HiZOnMj27dtZt24de/bs4Y477mDlypWkpqbicrkYOXIk8fHxzgvS0px1XHNaHMSYQs6C3hQ72fvhRYR33nmHp59+Go/HQ2JiIrGxsSdb8Tt2OF00K1fa4iCmSLKgN8XKggUL6N69Oz6f75TtR44cASAmJuZkwHs8MGoUzJjhdM9MmgSdOxd0ycacNwt6Uyykp6czduxY+vTpc1rIn3axFZwbnpo3h/R0J+Q/+QTuvbfgCjYmiOy2PRP25s2bR7169ejevTvVq1cnKioKl8uF2+3m7rvvZsmSJae24u+/H1q3dkIenLtbt20L3QkYc56sRW/C1scff8yrr77K1q1bufrqq5k+fTp33303q1atOr0fHuDTT51We5Yx9LhctgKUKfIs6E3Y2bNnD927d+ezzz4DICIignHjxmVOIxwDxOzcCW++6XTLlC4Na9fC+vWnv1nz5vDKK7ZAiCnSLOhNkZcxVDI1NRWfz8eUKVNISUlBRFBVfD4fK1eudIJ+wYJTu2Uy3HgjPPWUM4d8xqLdUVEW8iYsWNCbIi2nKQvi4uJ44oknGPnQQzRKS2OHy8X9v/wCcXHOGq3ZLsYi4nTZ9O8PDz4IEyc627t2tZA3YcGC3hRJCQkJjB07lqSkJFSVhjjzdHwJ3Bkby71799LR60VUkbQ0Z8KxG25wxsN/8olzA1SGrH3wMTEW7ibsWNCbIsXj8dC9e3fWrVsHQEOgL3AXzhAyH8Abb8Dx4yeHlLlc8NxzMGSI87xHD6fV/ssvcOml1nI3Yc+C3hQJHo+H999/nw0JCbykSl0gCrgUZ8Y8cBZAEEBuuAE6doQBA5z+9shI6NDh5JtZq90UMxb0plBLSEjgx/79ufvgQR4AbuPUYM9OIiKcbpqYGGjWzOmTj421YDfFmgW9KXw8HkhMZMbvv3PxW28xKMuu7OGuWbe53c6Mkhmhbi13YwALelNYeDxOv/mePXhnzcLl82X2u2eVfVJtAWjaFGrWtL52Y3IRyMIj44G2wF5VrZXD/geB5/1PjwDdVXWdf9924DDgBdJzmyvZFGMeD4wbh2/CBMQ/RNKdZXdGiz0j4L0ilGjSxJlR8sIL4ZlnIGMqYWNMjgJp0U8ARgATc9n/E9BMVX8TkVY4C3zfmmX/7aq6/7yqNOFnxw74z3/QYcPA5zul5X5KdwwnQ35/pUpUmDLFWu3GnKU8g15Vl4pI1TPsX5nl6SqcRcCNOZXXC199BZ9/7nxt2JC5K2uLPfNwIA3Y0qIFNwJ07EgFa7kbc06C3UffDZiT5bkC80VEgTGqmhDkzzOF2e+/O0vvff45zJ4NBw6gJUrwR506fFa3Ll+sW8d7OAsQe4HZwK/AWhHqXnEFlz/wAHe/+WZIT8GYcBC0oBeR23GCvnGWzY1UdbeIXAIsEJHvVHVpLq+PB+IBqlSpEqyyTEHyeGDqVOeu040bYdkyZ06ZcuXw/vnPrLj4Yl5avpwvv/mGMmXK0Pr++2k9bRqN0tJY5nZTrm1bLr30Uh7OukarMea8BSXoRaQOMBZopaoHMrar6m7/970i8ilwC5Bj0Ptb+wngLA4ejLpMPkpIgGnToH17qFHDeT5lCmQsNn/VVdC3L/sbNmTE6tWMTkhg79691KhRg1GjRtGlSxcuuuiizKX73sw+ZbAxJmjOO+hFpAowHeiiqluzbL8QcKnqYf/jFsCA8/08E2IJCfDvf8NW/x/1/PnOd7cbVXX6210udrVsyQs7dzLlX/8iLS2NNm3a8Mwzz9C8eXNETl5qPWXpPmNMvghkeOUknPmiyotIMvAyTrcqqvou8BJQDhjl/wecMYyyIvCpf1sJ4GNVnZsP52Dyk8fjzNv+ww9QooQzb3t29eqx6dFHqfa3vxEBpPl83DtqFJtKlaJ79+707NmT6tWrF3jpxhhHIKNu7s9j/2PAYzls/xGoe+6lmZBKSXEWxu7b9/RpfbPZc889PDllCuk4LYJEoHzbtiR/9BGlS5fO/1qNMWdkd8aak3bvhi++cEbJLFwIx46ddkjWiyc/REbyXunSvPHyy7jdbnC7+VqVqKgoFv397xbyxhQSFvTFmc8Ha9acHNv+zTfO9iuvhEcfdS6o9uuHZlmN6QPgEmAaMDY1Ffbvp0OHDowaNYoff/wx57VYjTEhZUFfHPgnCSM21mmlv/ceHDoEq1fDr78687XHxMCgQdC2rbNAh/+C6YxffkHfeovLgHE4Q6uyO3LkCBUrVqRixYoW8MYUQhb04SpjkrBffnFuVspYUUmzdL7ExcG//gUtW0L58qe8fNeuXfzrX/9i+PDheX5Ux44dg1m5MSbILOjD0ejR0LPnmS+iut1O0D/0UOamXbt2MW3aNKZMmcKKFStyfFnTpk2Jjo6mQoUK7Nu3j44dOxJvUxMYU6hZ0IeL336DefOcbpmMse1ZiUBEhPM9PT1zndScwr1OnToMHDiQa665hr/+9a+kpqYCEBERweDBg617xpgixoK+qMm4I7VePWcGyB074MQJWL/emTjsggtOf01EBHTrBl27smHDBnZ9/DHfVazI1L59Twv3e++9l+uuuy7zpVdeeSUTJzoTl3a1qQmMKZJEtfDNNlC/fn1NSkoKdRmFR0Z/+8qVTqBnJwJdusCTTzph36IFnDiBD/hfjRpsaduWJSkp7Nixg1mzZuHzd+lcffXVPPzww6eFuzGm6BGRNbmt+WEt+sLuiy/g7rud7pYzqVEjc572DcOH882wYSR8/z0rN2+GzZtPO9ztdtOtWzf69++fH1UbYwoRC/rCZuVK+OQT587U9eth1aq8XxMZibdJE75ZvZpx48aRkJBATv9Tc7lcuN1ufD4fkZGRxMbGBr9+Y0yhY0FfGOzZ49yJ+vHHMDfLdEDXXw+PPQbvvw9paZl3pfqAb6+/nt9SUzl69CjzL72USe3a8fvvv+f6ES6Xi6ioKIYPH86BAwfspiZjihEL+lA4ehS+/BIWLHACfuNGZ3vWC6lut9Pv3r8/Gxo0IPmNNzi+Ywe/4qzpuGrLlpPH/vILbdu25cEHH6RUqVJ06NAhc6SM2+2md+/elC1b1sLdmGLKgj4/ZdyRevHFTjdMerozve/Klc4NTFFR0KSJE+h33gnHjuGNi4PUVChRgtXR0Yx97DHGjx+fY1dMBhHhtttuo3PnzgAkJibaSBljTCYL+vyQlgYTJsBTT528IzXDtddCr15OsDdqBCVL8ssvv7Bu3TpmzpzJurQ0mqiSeOIEq557LqCPy97fbnO8G2OysqAPhmPHnIWvly51vlatOmXmR4XMBTl+bdWKRXXqsG7+fNYNGcK6dev49ddfT3m7FTit9G6PPkrnzp1p06ZNZleMiHD99dfTtm1b/vjjD8Ba7caYM7Nx9IHKOjFYzZqwYsXJYE9KclruIlC3LicaNmTzvn3UmDaNCMCNs/h1KhAHrMJphd9www3UrVs38ys1NZV77rmH1NRUIiMjWbRoETExMXg8HuuKMcac0ZnG0ecZ9CIyHmgL7FXVWjnsF+BtoDVwDHhYVb/x72vp3+cGxqrq4EAKLixBvyEhgYPDh/OnI0eouWsXLp/PCXP/knk+t5tfq1RhS/nyeEqUYP7Ro2xMTubgwYMANMRZiGM/UB74Eqj8l7/w4osvct111xEREXHaZ2asoWoXTo0xZ+N8g74pcASYmEvQtwaexgn6W4G3VfVWEXEDW4E7gWRgNXC/qp5+90425xr0OYVkTtsSEhIYN24cl19+Of369ePCDRs4MHQoPp+PI9deS+TOnZRKSaHh//6HG6fbJYMPWAy8DnwFHAeio6OpWrXqKV9paWm89tprpKWl4fP5Moc3ZrTSjTEmmM7rzlhVXSoiVc9wSHucXwIKrBKRsiJyGVAV2OZfUhARmew/Ns+gPxcej4emTZvi9Xpxu920adMGr9fLnDlz8Hq9uFwuGjRowG+//caftm7lYf/rxs+YwQggMuON/ve/zGDP6FvPeOwDTgCf33ILT/TqxSB/qFesWPGUBa8zxMXFkZiYSLly5WzsujEmZIJxMfYK4Ocsz5P923LafmtubyIi8UA8QJUqVc66iMTERNL90wSkp6ezaNEiRASv1wuAz+cjOTmZG37/nVn4VzfP+OwsjzP+f5Puf5zxA0oF3gMmR0YyaPjwgALbRr8YYwqDYAT96U3ZUxvD2bfnSFUTgARwum7OtojY2FhKliyZeSFzvn+q3pdiY+mUlobL5aJT5cqU/uUX3NkKSodTtqXjBPvqBx90ltcT4be2bfmtbFkGWavcGFPEBCPok4HKWZ5XAnbj9IbktD1fxMTE8NXw4fz2ySdUrl2banPmwKxZzE9NdX7jeL3OsnnXXotu2eIMd8QZ8rijTx+SZ81yLrTeeCO6bx/lOnakWbYFNe7Or+KNMSYfBSPoZwI9/X3wtwKHVHWPiOwDqotINWAX0Bl4IAifl7PERGp37+6sqrR4sbMO6mWXOX3nqs6UAo8/DrGxSGwspKYibjcyahRXx8dz9Ztv5ltpxhgTSnkGvYhMwhklWF5EkoGX8Xdxq+q7wGycETfbcIZXPuLfly4iPYF5OD0j41V1Uz6cg8PjObl0nssFL74If/6zs1xeamrmikrExDjj4TPGxFs3jDEmzIXPDVMez6mhvmiRE+JZb3SyUDfGhKnisfBITIwT7tlDPSbGAt4YU6yFT9CDhboxxuTAFeoCjDHG5C8LemOMCXMW9MYYE+Ys6I0xJsxZ0BtjTJizoDfGmDBXKG+Y8k+fsOMcXloeZ52P4qY4nndxPGconuddHM8Zzv68r1TVCjntKJRBf65EJCm3O8PCWXE87+J4zlA8z7s4njME97yt68YYY8KcBb0xxoS5cAv6hFAXECLF8byL4zlD8Tzv4njOEMTzDqs+emOMMacLtxa9McaYbCzojTEmzBXJoBeRliLyvYhsE5EXctgvIvKOf/96EbkpFHUGUwDn/KD/XNeLyEoRqRuKOoMtr/POclwDEfGKSKeCrC8/BHLOIhIrImtFZJOIfFnQNeaHAP6OlxGRWSKyzn/ej4SizmASkfEisldENuayPzhZpqpF6gtnWcIfgKtwFiBfB9TMdkxrYA4gQEPgq1DXXQDnfBtwsf9xq6J+zoGed5bjFuMsa9kp1HUXwJ91WWAzUMX//JJQ111A5/134E3/4wrAQSAy1LWf53k3BW4CNuayPyhZVhRb9LcA21T1R1VNBSYD7bMd0x6YqI5VQFkRuaygCw2iPM9ZVVeq6m/+p6uASgVcY34I5M8a4GlgGrC3IIvLJ4Gc8wPAdFXdCaCqxeW8FSglIgJchBP06QVbZnCp6lKc88hNULKsKAb9FcDPWZ4n+7ed7TFFydmeTzecVkBRl+d5i8gVwD3AuwVYV34K5M/6WuBiEUkUkTUi0rXAqss/gZz3COB6YDewAXhGVX0FU17IBCXLiuJSgpLDtuxjRAM5pigJ+HxE5HacoG+crxUVjEDOezjwvKp6nYZekRfIOZcAbgbigJKAR0RWqerW/C4uHwVy3n8G1gJ3AFcDC0Rkmar+kd/FhVBQsqwoBn0yUDnL80o4v+HP9piiJKDzEZE6wFiglaoeKKDa8lMg510fmOwP+fJAaxFJV9UZBVNi0AX693u/qh4FjorIUqAuUJSDPpDzfgQYrE7n9TYR+QmoAXxdMCWGRFCyrCh23awGqotINRGJBDoDM7MdMxPo6r9i3RA4pKp7CrrQIMrznEWkCjAd6FLEW3ZZ5XneqlpNVauqalVgKtCjCIc8BPb3+zOgiYiUEJELgFuBLQVcZ7AFct47cf4Xg4hUBK4DfizQKgteULKsyLXoVTVdRHoC83Cu1I9X1U0i8qR//7s4oy9aA9uAYzgtgSIrwHN+CSgHjPK3btO1iM/4F+B5h5VAzllVt4jIXGA94APGqmqOw/OKigD/rAcCE0RkA06XxvOqWqSnLxaRSUAsUF5EkoGXgQgIbpbZFAjGGBPmimLXjTHGmLNgQW+MMWHOgt4YY8KcBb0xxoQ5C3pjjAlzFvTGGBPmLOiNMSbM/T93AZo4ADpuLwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 1e-5\n",
    "\n",
    "for epoch in range(5000):\n",
    "    loss.backward(X, y)\n",
    "    \n",
    "    for layer in range(net1.n_layers-1):\n",
    "        grad_w = net1.layers[layer].grad_w\n",
    "        net1.layers[layer].weights = net1.layers[layer].weights - lr * grad_w\n",
    "\n",
    "\n",
    "plt.plot(X.T[0], y, marker=\".\", c=\"k\", label=\"samples\")\n",
    "plt.plot(X.T[0], net1.forward(X), marker=\".\", c=\"r\", label=\"predictions\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-05T20:16:37.326116Z",
     "start_time": "2020-04-05T20:16:37.319123Z"
    }
   },
   "source": [
    "### Classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:39:10.815935Z",
     "start_time": "2020-04-12T18:39:09.092934Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fead-forward Neural Network\n",
      "---------------------------\n",
      "3 layers: [10, 10, 2]\n",
      "\n",
      "       Inputs\n",
      "       \\ | /\n",
      " o o o o o o o o o o \n",
      "       \\ | /\n",
      " o o o o o o o o o o \n",
      "       \\  /\n",
      "       o o \n",
      "       Outputs\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net2 = NN(layers=[\n",
    "    Linear(10, input_size=2),\n",
    "    Linear(10),\n",
    "    Linear(2)\n",
    "])\n",
    "net2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:39:45.250770Z",
     "start_time": "2020-04-12T18:39:44.108276Z"
    }
   },
   "outputs": [],
   "source": [
    "loss = Loss(net2, partial(crossentropy, y2_normalized=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T18:40:05.318350Z",
     "start_time": "2020-04-12T18:40:03.979363Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1c1473e15f8>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXSb9Z3v8fdXsrwpu+MsjrORfXEWYiAmgZhm2GlzaJi2FJKBLmlaaEunbJ3e06ZDB9rO3Bm6sCSncEMohduy9NJOCm0TDAU7QwIJCVntLFKc3U7iEDvepN/945EcWZZsyZb9yNb3dY6O/SySvo9sffzzT7/n94gxBqWUUr2fw+4ClFJKJYYGulJK9REa6Eop1UdooCulVB+hga6UUn1Eml1PPHToUDNu3Di7nl4ppXqlDz74oMoYkxtpm22BPm7cOLZs2WLX0yulVK8kIp5o27TLRSml+ggNdKWU6iM00JVSqo+wrQ89kqamJiorK6mvr7e7lD4vMzOT/Px8XC6X3aUopRIkqQK9srKS/v37M27cOETE7nL6LGMM1dXVVFZWMn78eLvLUUolSIddLiLyrIicFJGPo2wXEfmFiFSIyHYRubSzxdTX15OTk6Nh3s1EhJycHP1PSKk+JpYW+lrgV8C6KNtvBCYFblcATwW+doqGec/Q17lnlJWVUVJSQnFxMUVFRVH3WbfOenvNnTuX6urqlv0j3T98XXA5JyeHrVu3ArB8+fI2zxd6vx07dvDKK6+wdOlSCgoKYn68NWvW8Mwzz5CXl8eDDz4I0O7xRTv+0GMOrTWW10tFJ7FMnysi44A/GWNmRti2GigxxrwYWN4LFBtjjrX3mIWFhSZ8HPru3buZNm1azMWrrtHXu3uVlZWxePFi6uvrcTqdLFu2jPz8/Fb7VFZWsm7dOnw+X8s6EcHpdHL99ddT88YbXOXz8Xenk0nLlwNQ/txzXO33U+ZwMPjaazn/17+ywO/nDDAbEOAPIky47TbyRowAYzhx7Bje116j0O9nMzABuAF4AygHLgfedzjot2gR595+m0K/nypgJuAD1orgLypiemkpXwaOAv8BvO904vf7Ix5fZWUlzz//PD6fr9X28GN2Op0sDxzb888/j9/vJyMjgw0bNmioRyAiHxhjCiNuNMZ0eAPGAR9H2fYnYGHI8gagMMq+K4AtwJYxY8aYcLt27Wqzri9btGiR2bx5s23Pn2qvd0979NFHjcPhMEDLTURa3UK3AWY+mIcDX+8A0wDGB6YZTAWYk2D8YEwP3/xg6gJfg7cmMPeH1Bt+fOHHFtweaX34zel0mkcffdTuH2FSAraYKFmdiGGLkf53j9jsN8asMcYUGmMKc3MjnrmqVJ9RXFyM0+kErFFFpaWl+P3+VrfS0lLS09MBmA+8Bfwb8B7wGyAd64MuAUaOG0fm1Kktby4fcHbYMIJt+9A3nR+oXrQInn0W1q6l6pprWvbzh+xrQr73Aafz8iI+ngEI1CmBWxrw78Cjgbq3r17d5tiysrJwOp1kZWW1HH/oMQNkZGRQWlpKaWlpy+uVnp5OcXFxzK+1CoiW9KE32m+hrwZuD1neC4zs6DHnzZvX5i9PZ1qMpaWl5tFHHzWlpaVx3zeS8+fPm5tuusnMmjXLzJgxw7z00kvmRz/6kSksLDQzZswwX/3qV43f7zfGWC3s++67z1x11VVm6tSp5v333ze33nqrmThxovn+979vjDHm4MGDZsqUKWb58uWmoKDALF261NTW1rbcP9hCf/PNN838+fPN3LlzzW233WY++eQTY4wxDz30kJk2bZopKCgw3/3udxNyjEHaQu9+n/vc54zL5TLvvfde1H22r15tNl9yiTmfnt6qRWwWLTK+9HTTLGKaMzKMKS01prTUNGdkXFy3erW1HNZ69rlc1v5BofdzOlvvm5YW2+M9+GDrdSIt/y34wZhPf9qYwHvj4tNGfn+WlpaalStXmpUrV7batmTJEtOvX7+EvZ/7ItppoSdi2OLrwL0i8hLWh6E1poP+81jcd999bNu2rd19ampq2L59O36/H4fDwaxZsxg4cGDU/efMmcPjjz/e7mO+8cYb5OXl8d///d8tz3Httdfygx/8AIBly5bxpz/9iU9/+tOA1ZJ45513+PnPf86SJUv44IMPGDJkCBMmTOA73/kOAHv37uWZZ55hwYIFfOlLX+LJJ5/k/vvvb3nOqqoqfvzjH/O3v/0Nt9vNT3/6U/7zP/+Te++9l9dee409e/YgIpw9e7bjF04llaamJiZNmsSVV14ZeYeyMgq+8Q0I6UPH4UAyMuCxx6x/oUtKoLgYAv3JzrfearXOWVBgLefkQOBDTFm+vGV/wNov9H47dsArr8DSpUjw/jE8nkyYAM88A3l5yI03wn33QWMjYgz88Y9w113w9NOQlRV42qKI/eDR1l9yySXad94FHQa6iLwIFANDRaQS+CHgAjDGPA2sB24CKoA64O7uKjZcTU0Nfr8fAL/fT01NTbuBHouCggLuv/9+HnroIW655RauuuoqXnnlFX72s59RV1fH6dOnmTFjRkugf+Yzn2m534wZMxg5ciRg/WIePnyYQYMGMXr0aBYsWADAnXfeyS9+8YtWgb5p0yZ27drVsk9jYyNFRUUMGDCAzMxMvvKVr3DzzTdzyy23dOnYVM/zeDyMGTMm+g6/+U3rMAf4h3+AVasuBnJ4uBUVtQnrNvtEErpfURGsWNF6WyyPt2JF6/sFw//qq+Gvf4Uf/Qg+/hhefRXGju24pjBut5va2lqMMToSqxM6DHRjzO0dbDfAPQmrKKCjljRcHEXQ2NhIeno6L7zwQpf/sk+ePJkPPviA9evX873vfY/rrruOJ554gi1btjB69GhWrVrVavx2RkYGAA6Ho+X74HJzczPQdohg+LIxhmuvvZYXX3yxTT3vv/8+GzZs4KWXXuJXv/oVGzdu7NLxqZ7l8Xi47LLLou8Q/l9oRkbrME92oeG/YAEUFsIdd8C8efB//y8sXhzXw2VnZ2OMoaGhgczMzG4ouG9LqjNF41VUVMSGDRsSOm716NGjDBkyhDvvvJN+/fqxdu1aAIYOHcr58+d5+eWXue222+J6TK/XS1lZGUVFRbz44ossXLiw1fb58+dzzz33UFFRwcSJE6mrq6OyspK8vDzq6uq46aabmD9/PhMnTuzy8ameU1tbS3V1NWOjtVTffx9KS2HlyovrwrtKeptbboHNm+HWW+G66+DrX4e8PLjmmpiOy+12A9Zrp4Eev14d6BC9L66zduzYwQMPPIDD4cDlcvHUU0/xhz/8gYKCAsaNG9d+ayuKadOm8dxzz/G1r32NSZMm8fWvf73V9tzcXNauXcvtt99OQ0MDAD/+8Y/p378/S5Ysob6+HmMM//Vf/5WQY1Q9w+Oxpq2OGuirVln91D/7GfTv33OFdbfJk2HTJliyBJ54wlqXlQUbNnQY6qGBnpOT092V9j3RPi3t7luiRrkku4MHD5oZM2bYXUZEffH1Tibr1683gHn33Xfbbty0yRrR8thjPV9YT/m3fzNGxDpOh8OYGMaV//a3vzWA2b17dw8U2DvRzePQlVIRtNtCX7UKhg6Fe+/t2aJ60jXXQGi3SQzjykNb6Cp+GujdbNy4cXz8ccR5zVQf5/V6SUtLaxn51KKsDN54Ax54APr1s6e4nlBUZHWzXHONNVo9hmsIZ2dnA1BXV9fNxfVNGuhKdROPx8Po0aNbzn5ssWoV5ObCPQkfHJZ8ioqscenGwG9/2+Hu2kLvGg10pbpJxDHopaXwl7/Agw9CILz6vMmT4fLLYV20CVsv0kDvGg10pbqJx+Np23/+wx/CsGHWcL5UsmwZbN9u3dqhXS5do4GuVDdoamri6NGjrQP93Xfhb39LrdZ50Be+AGlp8Pzz7e6mLfSu0UDvRiUlJS2n67/++uv85Cc/ibrv2bNnefLJJ1uWjx49GvcJTCp5VFZW4vf7Wwf6D38Iw4enXuscrBE9N90EL7zQdqqDEBroXaOB3gm+dn4ho/nMZz7Dww8/HHV7eKDn5eXx8ssvd6o+Zb82QxbfeQc2boSHHoJAt0LKWbYMjh2zRr5EoV0uXdP7A72sDB57zPqaAIcOHWLq1Kn80z/9E7NmzeK2226jrq6OcePG8a//+q8sXLiQ3//+9/zlL3+hqKiISy+9lH/8x3/k/PnzgDVb49SpU1m4cCGvvvpqy+OuXbuWewNjjk+cOMGtt97K7NmzmT17NqWlpTz88MPs37+fOXPm8MADD3Do0CFmzrQuEFVfX8/dd99NQUEBc+fO5a233mp5zM9+9rPccMMNTJo0qeWSYD6fj7vuuouZM2dSUFCgZ5jaoE2gr1oFI0a0Ps0/1dxyCwwc2G63S1paGunp6dpC76TkPfX/vvvaTlwUrqbG+pDF7weHA2bNsn5hopkzB2KY9CvSdLdgXaTg3Xffpaqqis9+9rNtprt98MEH+epXv8rGjRuZOHEin//85yM+/re+9S0WLVrEa6+9hs/n4/z58/zkJz/h448/bpky+NChQy37PxE4fXrHjh3s2bOH6667jn379gGwbds2tm7dSkZGBlOmTOGb3/wmJ0+e5MiRIy3j33Xa3Z4XDPTRo0fD22/DW29Zv3uBaWVTUmYmfO5zVrfLU09FHYMfnHFRxa93t9BraqwwB+trTU1CHjZ8utt3330XoCWgQ6e7nTNnDs899xwej4c9e/Ywfvx4Jk2ahIhw5513Rnz8jRs3tszn4nQ6O5zy991332XZsmUATJ06lbFjx7YE+uLFixk4cCCZmZlMnz4dj8fDJZdcwoEDB/jmN7/JG2+8wYABA7r+oqi4eL1eRowYQebWrXD33dacLaHTzqaqZcugrg5eey3qLtnZ2drl0knJ20KPoSVNWZk1PWdjo3V5rBdeSMhMddGmuw1+YGOiTHe7bdu2bpnD2Zq+IbLQKXudTifNzc0MHjyYjz76iDfffJMnnniC3/3udzz77LMJr0tF5/F4uGnwYOssyYYGcLms/zh780yKibBgAYwfb3W7BBop4bSF3nm9u4UePLX4kUdimsktVsHpboGo092+9957VFRUANYHOPv27WPq1KkcPHiQ/fv3t9w3ksWLF/PUU08BVn/3uXPn6N+/P5988knE/a+++mpeeOEFAPbt24fX62XKlClR66+qqsLv97N06VIeeeQRPvzwwziOXiWCx+NhsdNpNTbA+g+ypMTWmpKCwwF33mkN3zxyJOIuGuid17sDHawQ/973EtryCU53O2vWLE6fPt3udLezZs1i/vz57Nmzh8zMTNasWcPNN9/MwoULo06b+vOf/5y33nqLgoIC5s2bx86dO8nJyWHBggXMnDmTBx54oNX+3/jGN/D5fBQUFPD5z3+etWvXtmqZhzty5AjFxcXMmTOHu+66i8cee6zrL4qKmd/vx+v1cnrWLAie9p+eHtPkVClh2bJ2pwLQLpfOk/b+ne9OhYWFZsuWLa3W7d69m2nTptlST9ChQ4e45ZZbUmJCrWR4vfui48ePM3LkSH75y19y7+9+B3v3wh/+oN0toWbMgOpqqy897HW5/vrrqampYdOmTTYVl9xE5ANjTGGkbb2/ha5Ukmk1ZLG6Gq68UsM8VFkZlJfDiRPwqU+1GXKsXS6dp4EeRqe7VV3VEuijR8P+/aCXDmytpOTi2aKNjW0+W9Aul85LukC3qwso1ejr3H2CgT4uLc0a4TJpks0VJZniYuti2AAibT5b0BZ65yVVoGdmZlJdXa1h082MMVRXV+tFeLuJ1+tl4MCBDDhxwlqhgd5acHTa2LERTy7SQO+8pBqHnp+fT2VlJadOnbK7lD4vMzOT/Px8u8vok1qmzS0vt1ZooEd29Cg0NVnnkoQMOw52uRhjuuW8jr4sqQLd5XIxfvx4u8tQqktaLmxRXm6d6p+XZ3dJySdSP3og0N1uN36/n4aGBv0vMk5J1eWiVF/QqoU+caJ1Mo1qrbjYGpsP1lj9kH50nUK38/Q3TakEqqmpoaam5mKga3dLZEVF8Oab1oeit9/ealinTqHbeRroSiVQywiX0aPhwAEN9PZcfTVMmGBN1hVCW+idp4GuVAIFA31SRobVN6xj0Ns3ZYp1Jm0IDfTO00BXKoGCgT4mOCmXttDbN3my1TUVnAYb7XLpCg10pRLI4/GQkZHBoKoqa4UGevumTIELF6CysmWVttA7TwNdqQTyer2MGTMGR0UFuN0wcqTdJSW3yZOtr4ELtoAGelfEFOgicoOI7BWRChFpc6VjERkoIn8UkY9EZKeI3J34UpVKfq3GoE+caI3iUNEF5/UP6UfXLpfO6zDQRcQJPAHcCEwHbheR6WG73QPsMsbMBoqB/y0i6QmuVamk12oMuna3dGzkSOv0f22hJ0QsLfTLgQpjzAFjTCPwErAkbB8D9BfrPN1+wGmgOaGVKpXk6uvrOX78OOPy83XIYqxErG6XkBa6BnrnxRLoo4DDIcuVgXWhfgVMA44CO4BvG2P8YfsgIitEZIuIbNH5WlRfc/iw9TaZ0a8fNDfrkMVYTZnSqoWuXS6dF0ugR+oEDJ8O8XpgG5AHzAF+JSJtLjVvjFljjCk0xhTm5ubGXaxSySw4ZHFicLZQbaHHZvJkOHQI6usBSEtLIz09XVvonRBLoFcCo0OW87Fa4qHuBl41lgrgIDA1MSUq1TsEAz3vwgVrhQZ6bKZMsa4xGri4OugUup0VS6BvBiaJyPjAB51fAF4P28cLLAYQkeHAFOBAIgtVKtl5PB4cDgeDq6qsD/qGD7e7pN4hOHQxbKSLdrnEr8Ppc40xzSJyL/Am4ASeNcbsFJGVge1PA48Aa0VkB1YXzUPGmKpurFuppOP1esnLy8O5f7/VOtchi7GJMhZdW+jxi2k+dGPMemB92LqnQ74/ClyX2NKU6l1ajUG/9FK7y+k9+ve3hi9qC73L9ExRpRLE4/FwyejRcPCg9p/HK2yki7bQO0cDXakE8Pl8HD58mNkDB1pX4tFAj0+Esega6PHTQFcqAY4dO0ZzczPTXS5rhY5Bj8+UKVBdbd3QLpfO0kBXKgFaLmzR1GSt0BZ6fMI+GNUWeudooCuVAMFAH3H+PAwYAHriXHyCk3RpoHeJBrpSCRAM9AEnT+qQxc4YNw7S0lr60bXLpXM00JVKAK/XS05ODmk6KVfnuFzW9UXDWujGhM8yotqjga5UAng8HiaMHm3NSaKB3jkhI13cbjd+v5+Ghgabi+pdNNCVSgCPx0NhTo51bUwN9M6ZMqXl+qI642LnaKAr1UXGGDweD7MDIaSB3klTpkBDA3i9Oid6J2mgK9VFp0+fpra2lsmOwNtJx6B3TsjQRQ30ztFAV6qLgiNcxtTXw6BBkJNjc0W9VMj1RbXLpXM00JXqomCgDz17VocsdsWwYZCdDb/9LXmB11Rb6PGJabZFpVR0wUB3HzsGCxfaXE0vtmkTXLgAmzZx6datzEcDPV7aQleqizweD4OysnBUVuoHol1RUmJduQhwNDdTjHa5xEsDXaku8nq9LBg5EtEhi11TXGydLQoYl4sStIUeLw10pbrI4/FQOHCgtaCB3nlFRfCNbwBw5tln2YQGerw00JXqIo/Hw4yMDGtBA71rAp9BpI8dC2iXS7w00JXqgtraWqqqqpjg88GQITB4sN0l9W6jRgGQfeYMoC30eGmgK9UFXq8XgJF1ddo6T4RAoDuPHyc9PV0DPU4a6Ep1QXDI4uCqKg30RBg50vp65IhOodsJGuhKdYHH4yEDyAjOg666Jj3dOsGoslIvctEJGuhKdYHH42GK04kYo4GeKPn5cOSIBnonaKAr1QVer5crhgyxFjTQE2PUKO1y6SQNdKW6wOPxMLd/f2tBZ1lMjECgaws9fhroSnWBx+NhqsMBQ4daMy2qrhs1CqqqGJSZqYEeJw10pTqpqamJI0eOMLapSbtbEik/3/ricGiXS5w00JXqpCNHjuD3+xl27pwGeiIFxqKPQk8sipcGulKd5PF4yAL6nTmjgZ5IgUAf4fNpoMdJA12pTvJ4PEwILmigJ04g0Ic1NWmXS5xiCnQRuUFE9opIhYg8HGWfYhHZJiI7ReTtxJapVPLxeDy0xLgGeuIMGABuN0MbGqitrcUE5khXHevwikUi4gSeAK4FKoHNIvK6MWZXyD6DgCeBG4wxXhEZ1l0FK5UsvF4vl/brB+fPa6Ankgjk5zOkrg6/309DQwOZmZl2V9UrxNJCvxyoMMYcMMY0Ai8BS8L2+SLwqjHGC2CMOZnYMpVKPh6Ph4KsLBg+HIJj0VVijBrFwED/uXa7xC6WQB8FHA5ZrgysCzUZGCwiJSLygYgsj/RAIrJCRLaIyJZTp051rmKlkoTH42GSMXpCUXcYNYr+584BOtIlHrFcJDrSJczDO7XSgHnAYiALKBORTcaYfa3uZMwaYA1AYWGhdoypXssYg9frZZTTqd0t3WHUKNxnzyJoCz0esQR6JTA6ZDkfOBphnypjTC1QKyLvALOBfSjVB508eRJHfT0DQQO9O+Tn4/D5yEVb6PGIpctlMzBJRMaLSDrwBeD1sH3+H3CViKSJSDZwBbA7saUqlTw8Hg8tHS0a6ImnJxd1SoctdGNMs4jcC7wJOIFnjTE7RWRlYPvTxpjdIvIGsB3wA782xnzcnYUrZScdstjNQgJdu1xiF0uXC8aY9cD6sHVPhy3/O/DviStNqeTVKtD1Q9HECwR6PtpCj4eeKapUJ3g8Hqa7XNYl0/r1s7ucvmf4cIzTqV0ucdJAV6oTvF4vM1wu7W7pLk4n/mHDtMslThroSnWCx+NhvM+n3S3dyIwapS30OGmgK9UJpw8dYnBDg7bQu5EjP1/70OOkga5UnM6dO0dOTY21oIHebRyjR2uXS5w00JWKkw5Z7CH5+QwEms6csbuSXkMDXak46ZDFHhIYupheVWVzIb2HBrpScQoGum/kSMjOtrucvisQ6NnV1TYX0ntooCsVJ4/Hw2QRHFOm2F1K3xYIdPfZszYX0ntooCsVJ6/XyxSHA9H+8+4VCPTgNLqqYxroSsWpav9+cnQMevfLzuZcWhqDdNhizDTQlYpT2sGD1jfaQu92Z7KzGXLhgt1l9Boa6ErFoaGhgUHBD+k00LtdTeBi0So2GuhKxeHw4cMXhyxOmGBnKSnhXP/+DGtqsruMXkMDXak4BIcs1g8bBllZdpfT59UOHswwvx801GOiga5UHIKBbrR13iMuDBmCAzDHjtldSq+gga5UHIKBnj5jht2lpISGoUMBaDp0yN5CegkNdKXicGrfPoYCTj2pqEc0Dx8OQMOBAzZX0jtooCsVj/Jy66uOcOkRJi8PgGaPx+ZKegcNdKXikOH1Wt9ooPcIx7BhNADm8GG7S+kVNNCVipHf72dwdTV+gEsusbuclODu148jAEeO2F1Kr6CBrlSMjh07xiV+P+eHDIHMTLvLSQlut5sjgOP4cbtL6RU00JWKUXCES+OYMXaXkjKys7M5ArhOnrS7lF5BA12pGAUDXafN7Tlut5tKIKOqCoyxu5ykp4GuVIxO7NrFEMA9Z47dpaSMYJdLWmMj6KXoOqSBrlSMGnfuBCBDTyrqMcEuF0A/GI2BBrpSMXIET27RIYs9JthCBzTQY6CBrlSM3EeP4gMdstiDsrOzqQwuaKB3SANdqRgYY8g5c4YzAwZAerrd5aQMl8tFVVqatVBZ2f7OSgNdqVicOXOG8c3NnB8xwu5SUk56v36cy8rSFnoMNNCVioHn0CEmAT7tbulx2dnZnMnO1kCPQUyBLiI3iMheEakQkYfb2e8yEfGJyG2JK1Ep+x3bsYNBgGv6dLtLSTlut5uqjAwN9Bh0GOgi4gSeAG4EpgO3i0ib3+rAfj8F3kx0kUrZ7fzWrQAMLCy0uZLU43a7OelyaR96DGJpoV8OVBhjDhhjGoGXgCUR9vsm8Aqg5+iqPqd5924ABlx6qc2VpJ7s7GyOO51QXQ319XaXk9RiCfRRQOjclZWBdS1EZBRwK/B0ew8kIitEZIuIbDl16lS8tSplm3Svl2ZAtA+9x7Uai370qJ2lJL1YAl0irAufVOFx4CFjjK+9BzLGrDHGFBpjCnNzc2OtUSnbDTxxgpNZWeBy2V1KynG73Rz2+60F7UdvV1oM+1QCo0OW84HwP5OFwEsiAjAUuElEmo0xf0hIlUrZbNgnn3B6xAjy7C4kBWVnZ+PxBdqK2o/erlgCfTMwSUTGA0eALwBfDN3BGDM++L2IrAX+pGGu+oq62lrGNzezNz/f7lJSktvtZnNDg7WgLfR2ddjlYoxpBu7FGr2yG/idMWaniKwUkZXdXaBSdjuydSsDQOdwsYnb7eZ4XR243RroHYilhY4xZj2wPmxdxA9AjTF3db0spZLH6f/5HwCyZs2yuZLUlJ2dTd2FC5gJExAN9HbpmaJKdeDC9u0ADLniCpsrSU1utxufz4cZOVL70Dugga5URyoqaAKGXXaZ3ZWkJLfbDUDT8OHa5dIBDXSlOpBVWcnhtDTS9MLQtsjOzgagITfXGoceHMKo2tBAV6oDQ6qrOTFggN1lpKxgC/3C4MHQ3Ax6UmJUGuhKtccY8urqqBk2zO5KUlYw0M8PHGit0H70qDTQlWpHc2UlbmNoGjvW7lJSVrDL5ZNgoGs/elQa6Eq1o6qsDADn1Kk2V5K6gi30s4GvGujRaaAr1Y5zH3wAQH+dZdE2wUA/7XKB06mB3g4NdKXa0bhzJ43AsHnz7C4lZQW7XOoaGmDECO1Db4cGulLtcB48yAFgjE6ba5tgC722thZGjdIWejs00JVqR/9jx/Cmp5OVlWV3KSmrVaBnZcFHH0Hgsw3Vmga6UtH4/QytqaFq8GC7K0lpwS6XwXv2wHvvWePQFy/WUI9AA12paI4dI9PvpzZPZ0G3k8vlwuVyMXLfPgjOi97YCCUlttaVjDTQlYrC7NsHgE/7z23ndrvZlZt78YpRaWlQXGxrTclIA12pKD758EMAMmbOtLkSlZ2dza6BA+HFF60V3/oWFBXZW1QSimk+dKVSUe22baQDQ2bPtruUlOd2u60PRW+9Ffr3h9pau0tKStpCVyoK/969HADGjm18j1IAABFuSURBVB/f4b6qe7UEugjMnAkff2x3SUlJA12pKDK8XsqBsTqPi+2ys7Opq6uzFgoKYMcOMMbeopKQBrpSkfj9DKiqwuNyMWjQILurSXktLXSwAv3MGTh2zN6ikpAGulKRVFaS7vNxNjcXEbG7mpTXKtCDH1Lv2GFfQUlKA12pSCoqAKgfPdrmQhSEdblooEelga5UJOXlAMjkyTYXoiCshT50qDVJl34w2oYOW1QqgsadO/EBA6ZNs7sURVigw8UPRlUr2kJXKoL6HTuoQIcsJotgl4sJjmwpKIBduy5OBaAADXSlIpL9+61A1yGLScHtduPz+WhsbLRWzJwJ9fWwf7+9hSUZDXSlwvl8ZB87pmPQk0jLRS5Cx6KDdruE0UBXKtzhwzibmznodDJixAi7q1GEzYkOMH26ddaofjDaiga6UuECI1zODR+Ow6FvkWTQJtCzs2HCBG2hh9HfVqXCBcagN+sHokmjTZcL6EiXCDTQlQpXXk6dCO5Jk+yuRAW0aaGD9cFoRQVcuGBTVcknpkAXkRtEZK+IVIjIwxG23yEi2wO3UhHR+UZVr+Xfu5dyYxijH4gmjYiBXlAAfj/s3m1TVcmnw0AXESfwBHAjMB24XUSmh+12EFhkjJkFPAKsSXShSvWU5j17dIRLkonY5aJTALQRSwv9cqDCGHPAGNMIvAQsCd3BGFNqjDkTWNwE5Ce2TKV6SHMzaV6vjkFPMhFb6JMmQUaGjnQJEUugjwIOhyxXBtZF82Xgz5E2iMgKEdkiIltOnToVe5VK9RSvF0dzs7bQk0zEQE9Lg2nTtIUeIpZAjzR3aMSZ5UXkGqxAfyjSdmPMGmNMoTGmMDc3N/YqleopgSGLFcBonWkxaUTscgG9elGYWAK9Egj9zc4HjobvJCKzgF8DS4wx1YkpT6keFhiyeG74cNLT020uRgVFbKGD9cHokSPWBS9UTIG+GZgkIuNFJB34AvB66A4iMgZ4FVhmjNmX+DKV6iHl5dQ5nWTpGPSk4nK5cLlckQMdtNsloMNAN8Y0A/cCbwK7gd8ZY3aKyEoRWRnY7QdADvCkiGwTkS3dVrFS3am8nENOJ2PHjbO7EhWm1UUugoIjXbTbBYhxPnRjzHpgfdi6p0O+/wrwlcSWplTPM+Xl7GpuZsyYMXaXosK0mRMdID8fBg7UFnqAnimqVFBzMxw8yF6/X0e4JKGIgS6iH4yG0EBXKujQIaS5WcegJ6mIXS5wcU4XE3HwXUrRQFcqKDBkUcegJ6eILXSwAr2mBiore76oJKOBrlSQBnpSixro+sFoCw10pYIqKrjgctE0aBD9+/e3uxoVJmqXi87p0kIDXamg8nKOZGXpkMUkFbWFPmQI5OVpoKOBrtRF5eWUi2h3S5KKGuhg9aNrl4sGulIANDVhDh1ix4ULOgY9SUXtcgEr0HfvtoaepjANdKUADh5EfD52NjZqCz1JBVvoJtLwxJkzoaGhZS6eVKWBrhToCJdewO124/P5aGxsbLtR53QBNNCVsmigJ72oU+iCNS+6w6GBbncBSiWF8nIasrKoQgM9WUWdQhcgKwsmTkz5D0ZjmpxLqT6vooITAwaQBejFV5JTu4EOVrfLRx/1YEXJR1voSgGUl+NJT2fMmDGIRLpIl7Jbu10uYH0wun8/RAv8FKCBrlRjI3g87PH5tLslicXUQjcGHnwQysp6sLLkoYGu1IED4Pez9fx5DfQk1mGgB8egP/kkFBenZKhroCsVGOHy4blzelJREuuwy2XjxovfNzbCunU9UFVy0UBXSocs9godttAdGmf6CihVXk5Tv36cRgM9mXUY6MuXg8tlfZ+WZi2nGA10pcrLOTN0KKCBnsw67HIpKoK33oLsbBg1qgcrSx4a6Ert3EnzhQsscDgYlaJB0Bt02EIHq9slMGqJT30q5T4Y1UBXqa2kBI4fZ8SJE/zVGNI2b7a7IhWFy+UiLS0tegsdrJ+n3299X19vtdhTiAa6Sm2rVwPWG8FljBUIKmm1Oyc6WMMVMzIufkB64UKP1JUsNNBV6mpqgnffBRGaAZ/TaQWCSlodBnpREWzYAI88Yp1o9MQTcPx4zxVoMw10lbr+1/+Cykp8X/saPxTh/9xxhxUIKmm1e5GLoKIi+Jd/gd//Hurq4J57eqa4JKCBrlLTxo3ws58BIGvXstEYHAsW2FyU6kiHLfRQU6bAqlXw6qvw8svdWley0EBXqeneey9+39hIMTpksTeIK9AB7r8fLr3UaqVXV3dfYUlCA12lluZm+OIXretPBhiHgxI00HuDmLpcQqWlwbPPwunT8J3vdF9hSUIDXaWOqiq4/np48UUITpErwra5c9kEOo9LLxB3Cx1g9mx4+GF4/nm4664+PTZdA12lhg8/hHnz4L334Pvfh8xMcDohM5O/jhxJbm5uy5mIKnl1KtAB/uEfrD/izz0HV10Fa9YkvrgkoFcsUn1TWZk1pvzsWVi/HvbsgZEjrWGKhYVw883W9uJi3lq1Srtbeom4u1yCSksvfu/zwcqV1hj1yy6Dt9+2hqv2gRFOMQW6iNwA/BxwAr82xvwkbLsEtt8E1AF3GWM+THCtSsXm73+H666zzhQM9e1vW2EO1ps38Ab2er3MmDGjh4tUndHpFnpxsfUfWXDOdGPgvvusE5CMsbb98z/DuXPW9rlzrQ9Re1nQdxjoIuIEngCuBSqBzSLyujFmV8huNwKTArcrgKcCXxNux5o1VL/yCjlLl1KwYkW7604//jgZZ87QMGQIQ779bQBOP/649a/X3LmYU6fIWbqU0++8g/vPf6b2xhsZcvXVbR4rlnokN7fl8eK5X3D/0Mfhww8ZcOQILp+PkzNm4PzkExBhyLe/3eqxw48x9JiAuI7j7TvvJOe11zg/cCANn/oUfPghGWfOALS8fu293rEeK9BSc/CxmTsXPvyQ3AMHGNjUxIHRo/FnZ7cs7y0sJPfLX265X9OgQQz90peY9pnPUPHcc1z485/JHTOGEdnZ1kWCd+1qOf3bABL8+pe/wHe/C0BZWRnr1q3DGENFRQU5OTmUlZVR1IvevKnozJkzfPLJJ5SWlnLllVe22b5mzRqeeeYZ8vLyePDBBy/+PIuK2P/P/8zY//gPxO+HjAycV1yBeecdBKC5GRMYxhpkAFwuHE8+aYV6v37Qvz/068eOtWvbzSG4mDXh751I6xNFjDHt7yBSBKwyxlwfWP4egDHmsZB9VgMlxpgXA8t7gWJjzLFoj1tYWGi2bNkSV7E71qxh8te+RjrWi13pdAKQ7/O1vGnD1wUFjzL8apEmbF1oAFQ6nTSmRf+bl97cHPF54rmfAU44HAz3+9vUFi70sSM9d+h+xHEc7qYmRgTnv+jguaH1633E6aQxsD6SdJ+PUSH7B+vqimj3P+F0sjsri9N+PzfX1eHE+pcy6IH+/fnz6NHU1dVx6NCh1o8pQmZmJhs2bNBQT1JlZWUsWrSIpqYmRISxY8e2+tzj9OnTHA87K3TcuHEt3TSHDh1iPlAMvA0MGTKEV0+fxkXIH/0YawlNzTMOB35gSOA9HP57boBjTicGyAt5zzYA5atXxx3qIvKBMaYw0rZYulxGAYdDlitp2/qOtM8ooFWgi8gKYAV0bkRB9Suv4OTii98QDKmQwAhfFy70RQ5/8YPrQh+rKjCtaiRDq6paPXdn7+cM/FGN9EsVvhx87NDHCN833uMYcPRoy/NHes7Q5wZa1X4hLY2q3Nzox3rqVJvXKNY/qqHLhK3zA7udTqb6fDiBJuDFoUN576qr2LNnD//+8ccUAwOAucArwB+zs7lq+nT27NnTpk5jDI2NjZSUlGigJ6mSkhJ8Ph9g/bz69evH1KlTW7b//e9/b3Of4D7Bn/mmwA1guMvFPVjdDw6sP/6hQe3H+r365YgROMeOJbO5mazmZqZUVDC/thZHYJ8j6ekIMKS+Purv+fkIueTCyjQS2Uo3xrR7A/4Rq988uLwM+GXYPv8NLAxZ3gDMa+9x582bZ+K1ffVqUwumEUwtmO2rV0dddwGMP+RWH7iFrmuKsK4h7LFiqacp5PHiuV/weUruuKPV40S7XQh57EjHGKzhQuAW63GU3HFHu88b+tyRXu9Yj/VChNc71lv4a3Mh5HULr6W0tNRkZWUZETGB95cBzOqQ7enp6a22ORwOk5WVZUpLS9s9HmWf4M/V6XRG/FmtXr261c/U5XK17BP+M8/IyDCrV6826enpZj6Yh8F8BcyTgdtXwHxPxCxKT2/zPB3lUPjveeh750KE9fECtpgouRpLC70SGB2ynA8c7cQ+XVawYgU7aNs3HG1dd/ehh9YTTx96+HEsWrGCHYHnjbUPPfgYiehDX/Sb3/A2xN6HHsdjhx8rdL0PPVhP6OsWWktRUREbNmygpKSEs2fPsm3bNpYuXcqKkO0lJSWsC1xzcu7cuVRXV1NcXKyt8yQW+nON9LMK/nwj9aGH/8yXL19OUVERBQUFrFu3jrPAZXPnsnXrVgh8X11dzWMRnieWHIIofehR1idKLH3oacA+YDFwBNgMfNEYszNkn5uBe7FGuVwB/MIYc3l7j9uZPnSllEp1XepDN8Y0i8i9wJtY3UzPGmN2isjKwPangfVYYV6BNWzx7kQVr5RSKjYxjUM3xqzHCu3QdU+HfG+A1JmjUimlkpCe+q+UUn2EBrpSSvURGuhKKdVHaKArpVQf0eGwxW57YpFTgKeTdx8KVCWwnN4gFY8ZUvO4U/GYITWPuzPHPNYYE/H0bNsCvStEZEu0cZh9VSoeM6TmcafiMUNqHneij1m7XJRSqo/QQFdKqT6itwZ637x+VPtS8ZghNY87FY8ZUvO4E3rMvbIPXSmlVFu9tYWulFIqjAa6Ukr1EUkd6CJyg4jsFZEKEXk4wnYRkV8Etm8XkUvtqDORYjjmOwLHul1ESkVkth11JlJHxxyy32Ui4hOR23qyvu4Sy3GLSLGIbBORnSLydk/XmGgx/H4PFJE/ishHgWPu9TO3isizInJSRD6Osj1xORbtyhd237Cm6t0PXAKkAx8B08P2uQn4M9bVnuYD/2N33T1wzFcCgwPf35gKxxyy30asWT9vs7vuHvpZDwJ2AWMCy8PsrrsHjvlfgJ8Gvs8FTgPpdtfexeO+GrgU+DjK9oTlWDK30C8HKowxB4wxjcBLwJKwfZYA64xlEzBIREb2dKEJ1OExG2NKjTFnAoubsK4O1ZvF8nMG+CbWpUFP9mRx3SiW4/4i8KoxxgtgjOntxx7LMRugv4gI0A8r0Jt7tszEMsa8g3Uc0SQsx5I50KNdeDrefXqTeI/ny1h/2XuzDo9ZREYBtwJP03fE8rOeDAwWkRIR+UBElvdYdd0jlmP+FTAN6xKWO4BvG2P8PVOebRKWYzFd4MImkS48Hz7GMpZ9epOYj0dErsEK9IXdWlH3i+WYHwceMsb4rIZbnxDLcacB87Au/5gFlInIJmPMvu4urpvEcszXA9uATwETgL+KyN+NMee6uzgbJSzHkjnQk+bi1D0opuMRkVnAr4EbjTHVPVRbd4nlmAuBlwJhPhS4SUSajTF/6JkSu0Wsv99VxphaoFZE3gFmY13jtzeK5ZjvBn5irM7lChE5CEwF3u+ZEm2RsBxL5i6XzcAkERkvIunAF4DXw/Z5HVge+JR4PlBjjDnW04UmUIfHLCJjgFeBZb24pRaqw2M2xow3xowzxowDXga+0cvDHGL7/f5/wFUikiYi2VgXYN/dw3UmUizH7MX6jwQRGQ5MAQ70aJU9L2E5lrQtdJOCF6eO8Zh/AOQATwZarM2mF89QF+Mx9zmxHLcxZreIvAFsB/zAr40xEYe+9QYx/qwfAdaKyA6sroiHjDG9ekpdEXkRKAaGikgl8EPABYnPMT31Xyml+ohk7nJRSikVBw10pZTqIzTQlVKqj9BAV0qpPkIDXSml+ggNdKWU6iM00JVSqo/4/xoFUA5kPskiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = 1e-3\n",
    "\n",
    "for epoch in range(1000):\n",
    "    loss.backward(X, y_c_oh)\n",
    "    \n",
    "    for layer in range(net2.n_layers-1):\n",
    "        grad_w = net2.layers[layer].grad_w\n",
    "        net2.layers[layer].weights = net2.layers[layer].weights - lr * grad_w\n",
    "\n",
    "\n",
    "plt.plot(X.T[0], y_c_oh[:, 1], marker=\".\", c=\"k\", label=\"samples\")\n",
    "plt.plot(X.T[0], softmax(net2.forward(X))[:, 1], marker=\".\", c=\"r\", label=\"predictions\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}