05b_LAB_Intro_NN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "marked": true
    }
   },
   "source": [
    "# Machine Learning Foundation\n",
    "\n",
    "## Course 5, Part b: Intro to Neural Networks LAB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: neurons as logic gates\n",
    "In this exercise we will experiment with neuron computations.  We will show how to represent basic logic functions like AND, OR, and XOR using single neurons (or more complicated structures).  Finally, at the end we will walk through how to represent neural networks as a chain of matrix computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoid function:\n",
    "\n",
    "$$\n",
    "\\sigma = \\frac{1}{1 + e^{-x}}\n",
    "$$\n",
    "\n",
    "$\\sigma$ ranges from (0, 1). When the input $x$ is negative, $\\sigma$ is close to 0. When $x$ is positive, $\\sigma$ is close to 1. At $x=0$, $\\sigma=0.5$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Quickly define the sigmoid function\n",
    "def sigmoid(x):\n",
    "    \"\"\"Sigmoid function\"\"\"\n",
    "    return 1.0 / (1.0 + np.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAAGQCAYAAABoJTxTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1ZklEQVR4nO3de3weZZ3//9cnSZMe0gOlpS09QJFyRk6hBdGl1UWRlcWzICKCFXFFF3X3J+664rrr/lxdd1VEWEBEECh8PfLFKqIS8AS0lFIK9AS0tLT0nLZpmza57+v7x30XQkjaTJPmvtO8no/H/bhn5rpm5pOrk+TdydwzkVJCkiRJUudVlLoASZIkqbcxREuSJEkZGaIlSZKkjAzRkiRJUkaGaEmSJCkjQ7QkSZKUkSFaktqIiAsj4jfltt+IqI+I6R20RUT8ICI2RsSj+67Kdvf9q4i4uCf3KUmlFt4nWlJfFBFvBL4OHAvkgGeAK1NKs0pa2G5ERD3wo5TSTe20vQm4EzgypbR1H9bwZeDwlNKH9tU+JKk3qCp1AZLU0yJiCHAv8AngbqAaeBOwo5R1ddEhwNJ9GaAlSa/wcg5JfdERACmlO1NKuZTS9pTSb1JK8wAi4iMR8cddnSPirRGxMCI2RcT3IuLBXZdVFPv+KSL+JyIaIuK5iHhDcfnyiFjT+lKHiBgaEbdGxNqIWBYRX4yIig72e1ZELCju97tAtPfFRMRHgZuA0yOiMSL+te22iv1SRBxenL4lIq6NiF9GxJaIeCQiXteq77ERcX9EbIiI1RHxTxFxNvBPwAeK+3mi2Pfly0wioqL4NS0rfu23RsTQYtuhxRoujogXImJdRPzzXv8rSlIJGaIl9UWLgFxE/DAi3h4RB3TUMSJGAD8GvgAcCCwE3tCm2xRgXrH9DmAGcCpwOPAh4LsRUVvsew0wFDgMOBP4MHBJB/v9CfBFYATwLHBGezWmlL4PXA78JaVUm1K6ek8DUHQB8K/AAcAS4KvFfQ8Gfgv8Gji4+HX8LqX0a+A/gLuK+zmhnW1+pPiaVvwaa4HvtunzRuBI4C3AlyLi6E7WK0llwxAtqc9JKW2mEOQScCOwNiLuiYhR7XQ/B3gqpfTTlFIL8B3gpTZ9nk8p/SCllAPuAsYDX0kp7Ugp/QbYCRweEZXAB4AvpJS2pJSWAt8ELupgv0+nlH6cUmoGvtXOfrvqpymlR4tf1+3AicXl7wBeSil9M6XUVKz1kU5u80Lgv1NKz6WUGin85+P8iGh9+eC/Fs/+PwE8AbQXxiWprBmiJfVJKaVnUkofSSmNA46jcMb1W+10PRhY3mq9BKxo02d1q+ntxX5tl9VSOKNcDSxr1bYMGNvJ/S5vp19XtA7l24o1QuE/Ac/u5TYP5rVfXxXQ+j8oHe1XknoNQ7SkPi+ltAC4hUKYbmsVMG7XTERE6/mM1gHNFD4EuMsE4MUO9ju+zX7Ht9OvI1uBga3WH51h3eXA6zpo29MtnVby2q+vhVf/R0OSej1DtKQ+JyKOiojPRcS44vx4CtcHP9xO918Cx0fEO4uXJHwSyBJIX1a83ONu4KsRMTgiDgE+C/yog/0eGxHvLu730xn3+0Rx/RMjoj/w5Qzr3guMjogrI6KmWOuUYttq4NBdH4Zsx53AZyJiYvE68F3XULdk2L8klT1DtKS+aAuFDwM+EhFbKYTn+cDn2nZMKa0D3kfhntLrgWOA2ez97fA+ReEs8XPAHyl8EPHm3ez3a8X9TgL+1NmdpJQWAV+h8AHBxcV9dXbdLcBZwLkULr1YTOGDggD/p/i+PiLmtLP6zcBtwEPA80ATha9ZkvYrPmxFkjIonoFdAVyYUnqg1PVIkkrDM9GStAcR8baIGBYRNRTukxy0f+mHJKmPMERL0p6dTuFuFesoXOLwzpTS9tKWJEkqJS/nkCRJkjLyTLQkSZKUkSFakiRJysgQLUmSJGVkiJYkSZIyMkRLkiRJGRmiJUmSpIwM0ZIkSVJGhmhJkiQpI0O0JEmSlJEhWpIkScrIEC1JkiRlZIiWJEmSMjJES5IkSRkZoiVJkqSMDNGSJElSRoZoSZIkKSNDtCRJkpSRIVqSJEnKyBAtSZIkZWSIliRJkjIyREuSJEkZGaIlSZKkjAzRkiRJUkaGaEmSJCkjQ7QkSZKUUbeE6Ii4OSLWRMT8DtqnRsSmiJhbfH2pVdvZEbEwIpZExFXdUY8kSZK0L0VKqesbifgroBG4NaV0XDvtU4F/SCm9o83ySmARcBawApgFXJBSerrLRUmSJEn7SLeciU4pPQRs2ItVJwNLUkrPpZR2AjOA87qjJkmSJGlf6clrok+PiCci4lcRcWxx2Vhgeas+K4rLJEmSpLJV1UP7mQMcklJqjIhzgJ8Dk4Bop2+715dExGXAZQADBgw4Zfz48fuo1I7l83kqKvwsZmc5Xtk5Ztk4Xp23fPlyUkpMmDCh1KX0Gh5f2The2The2ZRyvBYtWrQupTSy7fIeCdEppc2tpmdGxPciYgSFM8+t0/A4YGUH27gBuAGgrq4uzZ49ex9W3L76+nqmTp3a4/vtrRyv7ByzbByvzps6dSoNDQ3MnTu31KX0Gh5f2The2The2ZRyvCJiWXvLeyTSR8ToiIji9OTiftdT+CDhpIiYGBHVwPnAPT1RkyRJkrS3uuVMdETcCUwFRkTECuBqoB9ASul64L3AJyKiBdgOnJ8KtwVpiYgrgPuASuDmlNJT3VGTJEmStK90S4hOKV2wh/bvAt/toG0mMLM76pAkSZJ6gle0S5IkSRkZoiVJkqSMDNGSJElSRoZoSZIkKSNDtCRJkpSRIVqSJEnKyBAtSZIkZWSIliRJkjIyREuSJEkZGaIlSZKkjAzRkiRJUkaGaEmSJCkjQ7QkSZKUkSFakiRJysgQLUmSJGVkiJYkSZIyMkRLkiRJGRmiJUmSpIwM0ZIkSVJGhmhJkiQpI0O0JEmSlJEhWpIkScrIEC1JkiRlZIiWJEmSMjJES5IkSRkZoiVJkqSMDNGSJElSRoZoSZIkKSNDtCRJkpSRIVqSJEnKyBAtSZIkZWSIliRJkjLqlhAdETdHxJqImN9B+4URMa/4+nNEnNCqbWlEPBkRcyNidnfUI0mSJO1L3XUm+hbg7N20Pw+cmVJ6PfBvwA1t2qellE5MKdV1Uz2SJEnSPlPVHRtJKT0UEYfupv3PrWYfBsZ1x34lSZKkUoiUUvdsqBCi700pHbeHfv8AHJVSml6cfx7YCCTgf1NKbc9S71rvMuAygFGjRp0yY8aMbqk7i8bGRmpra3t8v72V45WdY5aN49V5V155JblcjmuuuabUpfQaHl/ZOF7ZOF7ZlHK8pk2b9lh7V0t0y5nozoqIacBHgTe2WnxGSmllRBwE3B8RC1JKD7VdtxiubwCoq6tLU6dO7YmSX6W+vp5S7Le3cryyc8yycbw6b9iwYTQ0NDheGXh8ZeN4ZeN4ZVOO49Vjd+eIiNcDNwHnpZTW71qeUlpZfF8D/AyY3FM1SZIkSXujR0J0REwAfgpclFJa1Gr5oIgYvGsaeCvQ7h0+JEmSpHLRLZdzRMSdwFRgRESsAK4G+gGklK4HvgQcCHwvIgBaiteWjAJ+VlxWBdyRUvp1d9QkSZIk7SvddXeOC/bQPh2Y3s7y54ATXruGJEmSVL58YqEkSZKUkSFakiRJysgQLUmSJGVkiJYkSZIyMkRLkiRJGRmiJUmSpIwM0ZIkSVJGhmhJkiQpI0O0JEmSlJEhWpIkScrIEC1JkiRlZIiWJEmSMjJES5IkSRkZoiVJkqSMDNGSJElSRoZoSZIkKSNDtCRJkpSRIVqSJEnKyBAtSZIkZWSIliRJkjIyREuSJEkZGaIlSZKkjAzRkiRJUkaGaEmSJCkjQ7QkSZKUkSFakiRJysgQLUmSJGVkiJYkSZIyMkRLkiRJGRmiJUmSpIwM0ZIkSVJGhmhJkiQpo24J0RFxc0SsiYj5HbRHRHwnIpZExLyIOLlV29kRsbDYdlV31CNJkiTtS911JvoW4OzdtL8dmFR8XQZcBxARlcC1xfZjgAsi4phuqkmSJEnaJ6q6YyMppYci4tDddDkPuDWllICHI2JYRIwBDgWWpJSeA4iIGcW+T3dHXZIkqXfK5RPNuTwt+UQul8ilREs+Ty6faMmlwns+kU+F+XwqLMulRD7/ynRKhW3lU/GVpzgNqfi+qy2lV7ftmk/wcnvaNZ8vvKdE8T3By/Op1fJX5uGV7T6/dCfzcotfWc4r6xfm2bXCy9O7ttWq6ZXpVmP36uUddHrNOu1vt6P+ndHxdrJuCRpW72Tq1Myr7VPdEqI7YSywvNX8iuKy9pZPaW8DEXEZhbPYjBo1ivr6+n1S6O40NjaWZL+9leOVnWOWjePVeQ0NDeRyOccrg75+fKWUaM5DUws05RJNLYmdOdiRgx25XdOF9535xNamndy98Dc05xPNOWjOU5jOQ0seWtpMtyTI7ZpPiZY85IrL8il7YOuVFi/qVLdoPR2vnu+oX+uZtv0706+jfWQV3bSh4TWp7L4feypEtzeEaTfLX7swpRuAGwDq6urS1BL8d6S+vp5S7Le3cryyc8yycbw6b9iwYTQ0NDheGewPx1cun9i0vZmN23bSsG0nG7fumm5m0/ZmNjc1s3l7M1uaWorTLWxpaqZxRwtbd+bI5bNE2aC6MkdNvwr696ukpqqCmqoKqqsqqa6pYHBlBdVVFfSrjOJ7BdWVhfeqyqBfZaGtqrKCfhVBZcWu5UFFFNorK4Kqiii8F5dXVVRQWQEVUVheWREvT7/yDhW75iOIKPSvKK4XvNJeERAU+rzcL16Z39W2a71dy4hXAm60adsVJFvPP/Tgg5x55pkv993VTnF9vVo5fj/2VIheAYxvNT8OWAlUd7BckiR1oCWXZ82WHazatJ1Vm5p4aVMTa7fsKLwaC+/rGneyYesOOsrBFQFDBvRjSP9+DBlQxeCafhw6YiCD+/ejtqaKgdWVDKqporamikE1VQyqrmRAdSUDq6sY0K8wPaC6sjDdr5K//Okh3jxtWs8ORC9W+I+AN0nrzXoqRN8DXFG85nkKsCmltCoi1gKTImIi8CJwPvDBHqpJkqSylM8nXtrcxAsbtvHC+m2F9+Jr1abtrN3y2nBcXVXByNoaRgyuYdwBAzhx/DBG1NYwfFA1wwdVM2xgPw4YWM0BA6sZNqgfg2uquvWMZ4VnT9XHdEuIjog7ganAiIhYAVwN9ANIKV0PzATOAZYA24BLim0tEXEFcB9QCdycUnqqO2qSJKnc7WjJ8fy6rSxZ08ji1Y0sWdvIktWNPL9+Kztb8i/3q6wIxg4bwPjhA/irSSMZM7Q/Y4YNYPTQ/hw8dACjh/RnyIDuDcWSdq+77s5xwR7aE/DJDtpmUgjZkiTtt7Y0NfP0ys3MX7mZ+S9uYv6Lm3h2bePLZ5QrAiYMH8jhB9Uy9ciRHHLgICYMH8iE4QM5eFh///QvlZmeupxDkqQ+I59PLFnbyKylG5j1/AaeWLGJ59dtfbl91JAajh87lLOPG82kUYM5fGQth40cRP9+lSWsWlIWhmhJkrool0/MW9HAI89vYPbSDcxetpGGbc0AjBxcw8kThvGek8dy7NihHHvwEA4a3L/EFUvqKkO0JEl7Ye2WHTy0aC0PLlrLHxavZWMxNB82YhBvO2Y0dYcewOSJw5kwfKDXKkv7IUO0JEmdkFJiwUtbmPnkKn6/YA1PrdwMwIjaaqYddRBnHjGSN7xuBCMH15S4Ukk9wRAtSdJuLFq9hXvnreLeeSt5bu1WKgLqDhnOP77tSM48YiTHjBlCRYVnmqW+xhAtSVIbKxu284slO/mPxx9k0epGKgKmTDyQj75xImcfO5oDaz3bLPV1hmhJkih8OLB+4RrueOQFHli4hpTg1ImD+cp5x3L2caP9MKCkVzFES5L6tFWbtnPXrOXcNWs5qzY1MXJwDX839XAOyb3I+845vdTlSSpThmhJUp/0zKrNXFf/LPfOW0kC3jRpJFefeyxvOfog+lVWUF+/qtQlSipjhmhJUp8ya+kGrqt/lt8vWMOg6kqmv+kwLjrtEMYPH1jq0iT1IoZoSdJ+L6VE/aK1XPfAszy6dAMHDOzH5846gg+ffihDB/YrdXmSeiFDtCRpv/bYso189ZdPM+eFBg4e2p+rzz2GD5w6noHV/gqUtPf8CSJJ2i8tW7+V//z1AmY++RIjB9fw/7/7eN5z8jiqqypKXZqk/YAhWpK0X2nYtpNrfr+EW/+ylKqKCq7860l87E2HMajGX3mSuo8/USRJ+4V8PnHbw8v45m8W0rijhffXjeczZx3BqCHe31lS9zNES5J6vcWrt/D5n8xjzgsNvPHwEXzxHUdz1OghpS5L0n7MEC1J6rV2tuS5rv5Zrn1gCQNrKvnv95/Au04aS0SUujRJ+zlDtCSpV3r8hY1c9ZMnWbh6C+eecDBXn3sMI2prSl2WpD7CEC1J6lV2tuT5r98s5MY/PMeowf35/sV1vOXoUaUuS1IfY4iWJPUayzds44o75vDEik1cMHkC/3TOUQzu78NSJPU8Q7QkqVeY+eQqPv+TeQBcd+HJvP34MSWuSFJfZoiWJJW1puYc//7Lp/nRwy9wwvhhfPeCkxg/fGCpy5LUxxmiJUll69m1jVxxx+M8s2ozl/3VYfzDW4/0iYOSyoIhWpJUlh5YuIZP3fE4/SqDmz9Sx5uP8sODksqHIVqSVFZSSvzwz0v5yr1Pc9ToIdx0cR0HDxtQ6rIk6VUM0ZKkstGSy/Ov//dpbnt4GWcdM4pvfeBEBtX4q0pS+fEnkySpLGza3swVd8zhD4vX8fEzD+PzbzuKigqfPCipPBmiJUklt2z9Vi69ZRbL1m/j6+95Pe8/dXypS5Kk3TJES5JKav6Lm/jwzY+ST4nbPjqF0193YKlLkqQ9MkRLkkpm1tINXPqDWQwZ0I8fTZ/CxBGDSl2SJHWKIVqSVBJ/WLyWj906m4OHDuBH06d4Bw5JvYohWpLU4+576iU+dcfjvO6gWm776GRG1NaUuiRJyqRbHvsUEWdHxMKIWBIRV7XT/o8RMbf4mh8RuYgYXmxbGhFPFttmd0c9kqTy9bPHV/B3t8/hmIOHMONjpxmgJfVKXT4THRGVwLXAWcAKYFZE3JNSenpXn5TSN4BvFPufC3wmpbSh1WampZTWdbUWSVJ5+9HDy/iXX8zntIkHcuPFddR6D2hJvVR3nImeDCxJKT2XUtoJzADO203/C4A7u2G/kqRe5EcPL+OLP5/Pm488iB9ccqoBWlKvFimlrm0g4r3A2Sml6cX5i4ApKaUr2uk7kMLZ6sN3nYmOiOeBjUAC/jeldEMH+7kMuAxg1KhRp8yYMaNLde+NxsZGamtre3y/vZXjlZ1jlo3j1XlXXnkluVyOa665piT7/9OLzdz45E5OGFnJp06qoaoXPETF4ysbxysbxyubUo7XtGnTHksp1bVd3h2nAdr7SdhRMj8X+FObSznOSCmtjIiDgPsjYkFK6aHXbLAQrm8AqKurS1OnTu1i2dnV19dTiv32Vo5Xdo5ZNo5X5w0bNoyGhoaSjNfMJ1fx/fvmcMbhB/L9i0+lf7/KHq9hb3h8ZeN4ZeN4ZVOO49Udl3OsAFo/WmocsLKDvufT5lKOlNLK4vsa4GcULg+RJO0Hfr9gNZ++83FOnnAAN364rtcEaEnak+4I0bOASRExMSKqKQTle9p2ioihwJnAL1otGxQRg3dNA28F5ndDTZKkEvvzknVc/qM5HD1mCDdfcioDq70GWtL+o8s/0VJKLRFxBXAfUAncnFJ6KiIuL7ZfX+z6LuA3KaWtrVYfBfwsInbVckdK6dddrUmSVFqPLdvI9FtnM/HAQdx66WSG9O9X6pIkqVt1y2mBlNJMYGabZde3mb8FuKXNsueAE7qjBklSeXh65WY+8oNHGTWkP7dNn8wBg6pLXZIkdbtuediKJEkAKxu2c8ktj1JbU8WPpk/hoMH9S12SJO0ThmhJUrfY3NTMJT+YxbYdOX5wyamMHTag1CVJ0j7jpzwkSV22syXPJ370GM+ubeSHl07mqNFDSl2SJO1ThmhJUpeklLjqp/P405L1fPN9J3DG4SNKXZIk7XNeziFJ6pL/uX8RP53zIp876wjec8q4UpcjST3CEC1J2mt3zXqB7/x+CR+oG88Vbz681OVIUo8xREuS9spDi9byTz+bz18dMZJ/f9dxFO/5L0l9giFakpTZc2sb+eQdc5h0UC3fu/Bk+lX660RS3+JPPUlSJpubmpl+62z6VVZw44frqK3xM+qS+h5DtCSp03L5xN/f+TgvrN/G9y48mfHDB5a6JEkqCU8fSJI67Rv3LeSBhWv593cex2mHHVjqciSpZDwTLUnqlJ8//iLXP/gsHzptAh867ZBSlyNJJWWIliTt0bwVDXz+J/OYMnE4V597bKnLkaSSM0RLknZrzeYmLrv1MUbU1ngnDkkq8ppoSVKHdrbk+cTtc9jc1MxPPvEGDqytKXVJklQWDNGSpA79x8xneGzZRq654CSOHjOk1OVIUtnwb3KSpHb9Yu6L3PLnpVx6xkTOPeHgUpcjSWXFEC1Jeo1Fq7dw1U+e5NRDD+AL5xxV6nIkqewYoiVJr7KlqZnLb3uMQTVVXPtBP0goSe3xmmhJ0stSSvzj/5nHsg3buGP6FA4a0r/UJUlSWfL0giTpZTf+4Tl+/dRLXHX2UUzxiYSS1CFDtCQJgL88u56v/WoB5xw/mulvmljqciSprBmiJUms2dLEp+58nENHDOLr7z2BiCh1SZJU1rwmWpL6uFw+8fd3zqVxRzO3T59CbY2/GiRpT/xJKUl93Ld/u4i/PLee/3rfCRw5enCpy5GkXsHLOSSpD3to0VqueWAJ7ztlHO89ZVypy5GkXsMQLUl91EubmrjyrrkccdBgvnLecaUuR5J6FUO0JPVBLbk8n77zcZqac1x74ckMqK4sdUmS1Kt4TbQk9UHfvH8Rjy7dwLc+cCKHH1Rb6nIkqdfxTLQk9TEPLFjDdfXPcsHkCbzzpLGlLkeSeiVDtCT1ISsbtvPZu+dy9JghXH3uMaUuR5J6rW4J0RFxdkQsjIglEXFVO+1TI2JTRMwtvr7U2XUlSd0jAZ++83F2tuS59oMn0b+f10FL0t7q8jXREVEJXAucBawAZkXEPSmlp9t0/UNK6R17ua4kqYvWbUtsWraRb59/IoeN9DpoSeqK7jgTPRlYklJ6LqW0E5gBnNcD60qSOqlhWzPrm/JcMHkC553oddCS1FXdEaLHAstbza8oLmvr9Ih4IiJ+FRHHZlxXkrSXVm3azrNrG6mpDK+DlqRu0h23uIt2lqU283OAQ1JKjRFxDvBzYFIn1y3sJOIy4DKAUaNGUV9fv7f17rXGxsaS7Le3cryyc8yycbz2LJdP/OesJnL5PAfXwsN/+kOpS+o1PL6ycbyycbyyKcfx6o4QvQIY32p+HLCydYeU0uZW0zMj4nsRMaIz67Za7wbgBoC6uro0derUbig9m/r6ekqx397K8crOMcvG8dqzb9y3gEUbn+WwkbVUtWx3vDLw+MrG8crG8cqmHMerOy7nmAVMioiJEVENnA/c07pDRIyOiChOTy7ud31n1pUk7Z0HF63l2gee5fxTxzOitqbU5UjSfqXLITql1AJcAdwHPAPcnVJ6KiIuj4jLi93eC8yPiCeA7wDnp4J21+1qTZLU1720qYnP3jWXI0cN5upzj93zCpKkTLrlsd8ppZnAzDbLrm81/V3gu51dV5K091pyeT595+Nsb85x7YUnMaDa+0FLUnfrlhAtSSof37x/EY8u3cC3PnAihx80uNTlSNJ+ycd+S9J+5IEFa7iu/lkumDyed57kHUMlaV8xREvSfmJlw3Y+c/dcjh4zxOugJWkfM0RL0n6gOZfnijvm0NyS59oPnkT/fl4HLUn7ktdES9J+4Ou/XsCcFxq45oKTOGxkbanLkaT9nmeiJamXu//p1dz4h+e56LRDOPeEg0tdjiT1CYZoSerFlm/YxufunstxY4fwxXccXepyJKnPMERLUi/V1JzjE7c/RgKu/eDJ1FR5HbQk9RSviZakXurqXzzF/Bc38/2L6zjkwEGlLkeS+hTPREtSL3TXrBe4a/Zyrph2OG85elSpy5GkPscQLUm9zJMrNvEvv3iKN00awWfOOqLU5UhSn2SIlqRepGHbTj5x+2OMGFTNt88/icqKKHVJktQneU20JPUS+XziyrvmsmbzDu6+/HSGD6oudUmS1Gd5JlqSeonv/H4x9QvX8qVzj+HE8cNKXY4k9WmGaEnqBR5YsIZv/24x7z55LBdOmVDqciSpzzNES1KZW7KmkU/f+TjHjBnCV995PBFeBy1JpWaIlqQytmlbMx+7dTY1/Sq44cN1DKj2gSqSVA78YKEklamWXJ5PzXicFRu3cefHTmPssAGlLkmSVGSIlqQy9bVfLeChRWv5z/ccT92hw0tdjiSpFS/nkKQy9OPHVnDTH5/nI284lA+c6gcJJancGKIlqczMeWEj//TTJ3nD6w7kn//m6FKXI0lqhyFaksrIS5ua+PhtjzF6aH+u/eDJ9Kv0x7QklSOviZakMtG4o4VLb5nFth0t3D59Cgf4REJJKluGaEkqA825PH93+xwWrt7C9y+u44hRg0tdkiRpN/w7oSSVWEqJf/n5fB5atJavvvM4ph55UKlLkiTtgSFakkrs2geWMGPWcj715sM5f7J34pCk3sAQLUkl9NM5K/iv3yzi3SeN5bNnHVHqciRJnWSIlqQS+dOSdfx/P57H6YcdyNfe83oiotQlSZI6yRAtSSWw8KUtXH7bYxw2chDXX3QK1VX+OJak3sSf2pLUw5at38pF33+EAdWV/OCSyQwd0K/UJUmSMjJES1IPWtmwnQ/e+AjNuTy3fXQKY4cNKHVJkqS9YIiWpB6yZksTF970CJu3N3PrpVM4crT3gpak3qpbQnREnB0RCyNiSURc1U77hRExr/j6c0Sc0KptaUQ8GRFzI2J2d9QjSeVm49adXHTTo7y0qYkfXHIqx48bWuqSJEld0OUnFkZEJXAtcBawApgVEfeklJ5u1e154MyU0saIeDtwAzClVfu0lNK6rtYiSeVoS1MzF//gUZ5fv5UffORU6g4dXuqSJEld1B1noicDS1JKz6WUdgIzgPNad0gp/TmltLE4+zAwrhv2K0llb9vOFi69ZRZPr9zMdReezBmHjyh1SZKkbhAppa5tIOK9wNkppenF+YuAKSmlKzro/w/AUa36Pw9sBBLwvymlGzpY7zLgMoBRo0adMmPGjC7VvTcaGxupra3t8f32Vo5Xdo5ZNuU+XjtaEt9+vIln1uf5xAk1TB7T5T/+7bUrr7ySXC7HNddcU7IaeptyP77KjeOVjeOVTSnHa9q0aY+llOraLu+On+jtPR2g3WQeEdOAjwJvbLX4jJTSyog4CLg/IhaklB56zQYL4foGgLq6ujR16tQuF55VfX09pdhvb+V4ZeeYZVPO47VpezOX3jKLBRu28c33n8C7Ty7tH+CGDRtGQ0ND2Y5XOSrn46scOV7ZOF7ZlON4dcflHCuA8a3mxwEr23aKiNcDNwHnpZTW71qeUlpZfF8D/IzC5SGS1Gutb9zBB298mHkrGvjuB08ueYCWJHW/7gjRs4BJETExIqqB84F7WneIiAnAT4GLUkqLWi0fFBGDd00DbwXmd0NNklQSL21q4v3/+xeWrGnkhg/Xcc7xY0pdkiRpH+jy5RwppZaIuAK4D6gEbk4pPRURlxfbrwe+BBwIfC8iAFqK15aMAn5WXFYF3JFS+nVXa5KkUnhh/TYu/P7DbNzazK2XTmbKYQeWuiRJ0j7SLZ9ySSnNBGa2WXZ9q+npwPR21nsOOKHtcknqbRav3sKHvv8ITc15bp8+hRPGDyt1SZKkfah0HxWXpP3Eo89v4OO3zaayooK7Pn4aR40eUuqSJEn7mI/9lqQu+PFjK7jwpoc5YGA1P778dAO0JPURnomWpL2Qzye+8ZuFXFf/LG943YFcd+EpDB3Yr9RlSZJ6iCFakjLatrOFz971BL9+6iUumDyBr5x3LP0q/cOeJPUlhmhJyuClTU1Mv3UWT63czBf/5mg++saJFO8wJEnqQwzRktRJs5Zu4Io75tDY1MJNH67jLUePKnVJkqQSMURL0h7k8onr6pfwP79dzLgDBnDLJZM5eowfIJSkvswQLUm7sWZLE5+96wn+uGQd555wMP/xruMY3N8PEEpSX2eIlqQO/HHxOq68ay5bmpr52ruP5wOnjvf6Z0kSYIiWpNdozuX5zu8W890HlnD4yFpunz6FI0cPLnVZkqQyYoiWpFaeXLGJz/9kHk+v2sz768bx5b89loHV/qiUJL2avxkkCdi+M8e3fruIG//wHCNqa7j+Q6dw9nGjS12WJKlMGaIl9Xl/fnYdX/jpkyxbv43zTx3PF845mqED/PCgJKljhmhJfVbDtp187VcLmDFrOYccOJA7PjaFN7xuRKnLkiT1AoZoSX3OjpYct/55Gdf8fjFbd+b4+JmH8Zm/PoL+/SpLXZokqZcwREvqM1JK/N95q/jGfQtYvmE7Zx4xki+ccxRHjfbBKZKkbAzRkvqER5/fwFdnPsMTyxs4avRgbvvoZN40aWSpy5Ik9VKGaEn7tVlLN/C9B5bwwMK1jBpSwzfe+3reffI4Kit8aIokae8ZoiXtd1JK1C9cy/fqlzBr6UaGD6rmH992JJeeMZEB1V73LEnqOkO0pP1GLp/45ZOruK7+WZ5ZtZmDh/bn6nOP4fxTJxieJUndyhAtqddbvbmJu2ctZ8as5bzYsJ3XjRzEN977es47cSzVVRWlLk+StB8yREvqlfL5xLy1Ldxx62x+t2ANuXzijMMP5F/ecTRvPWY0FV7zLEnahwzRknqV59Y2cu+8Vdw9ezkrNu7gwEEbmf6miVxw6gQOHTGo1OVJkvoIQ7Sksrd03VZ++eQq7p23imdWbSYCTj/sQP52Qp4r3/dmL9mQJPU4Q7SkspNS4plVW3hg4Rp+NX8V81/cDMAphxzAl95xDOccP4bRQ/tTX19vgJYklYQhWlJZ2LStmT8sWcuDC9fy4KK1rNmyA4ATxw/ji39zNG8/fgxjhw0ocZWSJBUYoiWVxKZtzTz2wgZmLd3II8+tZ+7yBvIJhvSv4k1HjOTM4mvUkP6lLlWSpNcwREva5/L5xLIN25i3ooFZSzcwe+lGFq7eQkpQVREcN3Yon5x2OFOPHMkJ44ZRVeklGpKk8maIltStWnJ5nl+3lfkrNzH/xc08+eImnl65mcYdLQAMqq7k5EMO4Jzjx3DqocM5cfwwH4QiSep1DNGS9sr2nTmWrt/K4jWNLFnTyLNrGlm8ZgvPr9tKcy4B0L9fBUePGcK7ThrLcWOHcOzBQzlq9GDPNEuSej1DtKR27WzJs3pzEy9tbmLFxm0sW7+NFzZs44Xi+64P/gFUBEwYPpDDD6rlzUeNYtJBtRw3diivGznIwCxJ2i8ZoqU+pqk5x7rGHaxr3MnaLTsK08X3lzY3sWpT4bWucQcpvbJeBIwe0p8Jwwdy5hEjmTB8IIeMGMSkg2qZOGIQ/ft5SYYkqe/olhAdEWcD3wYqgZtSSl9r0x7F9nOAbcBHUkpzOrOupFfL5RONO1rYWnw17mhhc1MLW5qa2by9+F6c3rhtJw3bmtmwdScN23aycVsz25tz7W53SP8qRg/tz+ihAzh69BDGDOvPmKH9GTN0AAcPG8C4AwYYlCVJKupyiI6ISuBa4CxgBTArIu5JKT3dqtvbgUnF1xTgOmBKJ9eVykY+n2jO58nlE825REuuOJ1PNLfkacnn2dmSaM7lac7l2ZnLs7Ol+CpO72jZ9Z5jR3OeppYcTc2F+aXLd/DjlXPYvjPH9uYc23bmWk0XAnNTc36PdVZWBIP7VzF8YDXDBvZjzND+HD1mCMMH9WPYwGpG1FYzoraGkYNrGFFbw4G11dRUGZAlSeqs7jgTPRlYklJ6DiAiZgDnAa2D8HnArSmlBDwcEcMiYgxwaCfWfY2FCxcyderUbii987btzLFy/Wb69++Oe9amDEs707g3e9vDOp1YKbXdcnr15M4dO6iuqXltc3r1Sm1mXzX/6nVe3Tel1/ZLxYWp1cK0a830yvJdvVJqvW6h367ttp5Prbbd3SKCioCKCEiJqsoKKiqCiuLyyuJ0ZUXhVdVqujKCygqorKigqqLV8ooAIA9sKL72Rw0NDQwbNqzUZfQKc+fOpaWlpcd/dvZmHl/ZOF7ZOF7ZlON4dUeIHgssbzW/gsLZ5j31GdvJdQGIiMuAywD69etHQ0NDl4rOatOOxLrtedi+rUf321rs4w3t7fZ3u15zU6v29nu2tzTaWbinfq3bo9VEe8srok17q34vv7+8LIr7CaLYvmu/7U2/6v1Vy+Ll+Ypdy9vUl8vlqayEXdG/QwlodVVG7tWzfUYul+vxnwW9VUtLCyklxysDj69sHK9sHK9synG8uiNEt5dt2v7276hPZ9YtLEzpBuAGgLq6ujR79uwsNXZZSokHH3zQszgZ1NfXO14ZOWbZOF6dN3XqVBoaGpg7d26pS+k1PL6ycbyycbyyKeV4RXtn9uieEL0CGN9qfhywspN9qjuxblnoaAAlSZLU93THDVxnAZMiYmJEVAPnA/e06XMP8OEoOA3YlFJa1cl1JUmSpLLS5TPRKaWWiLgCuI/CbepuTik9FRGXF9uvB2ZSuL3dEgq3uLtkd+t2tSZJkiRpX+qW+0SnlGZSCMqtl13fajoBn+zsupIkSVI583m8kiRJUkaGaEmSJCkjQ7QkSZKUkSFakiRJysgQLUmSJGVkiJYkSZIyMkRLkiRJGRmiJUmSpIwM0ZIkSVJGhmhJkiQpI0O0JEmSlJEhWpIkScrIEC1JkiRlZIiWJEmSMjJES5IkSRkZoiVJkqSMDNGSJElSRoZoSZIkKSNDtCRJkpSRIVqSJEnKyBAtSZIkZWSIliRJkjIyREuSJEkZGaIlSZKkjAzRkiRJUkaGaEmSJCkjQ7QkSZKUkSFakiRJysgQLUmSJGVkiJYkSZIyMkRLkiRJGRmiJUmSpIy6FKIjYnhE3B8Ri4vvB7TTZ3xEPBARz0TEUxHx963avhwRL0bE3OLrnK7UI0mSJPWErp6Jvgr4XUppEvC74nxbLcDnUkpHA6cBn4yIY1q1/09K6cTia2YX65EkSZL2ua6G6POAHxanfwi8s22HlNKqlNKc4vQW4BlgbBf3K0mSJJVMpJT2fuWIhpTSsFbzG1NKr7mko1X7ocBDwHEppc0R8WXgI8BmYDaFM9YbO1j3MuAygFGjRp0yY8aMva57bzU2NlJbW9vj++2tHK/sHLNsHK/Ou/LKK8nlclxzzTWlLqXX8PjKxvHKxvHKppTjNW3atMdSSnVtl+8xREfEb4HR7TT9M/DDzoboiKgFHgS+mlL6aXHZKGAdkIB/A8aklC7d0xdTV1eXZs+evadu3a6+vp6pU6f2+H57K8crO8csG8er86ZOnUpDQwNz584tdSm9hsdXNo5XNo5XNqUcr4hoN0RX7WnFlNJf72ajqyNiTEppVUSMAdZ00K8f8BPg9l0Burjt1a363Ajcu6d6JEmSpFLr6jXR9wAXF6cvBn7RtkNEBPB94JmU0n+3aRvTavZdwPwu1iNJkiTtc10N0V8DzoqIxcBZxXki4uCI2HWnjTOAi4A3t3Mru69HxJMRMQ+YBnymi/VIkiRJ+9weL+fYnZTSeuAt7SxfCZxTnP4jEB2sf1FX9i9JkiSVgk8slCRJkjIyREuSJEkZGaIlSZKkjAzRkiRJUkaGaEmSJCkjQ7QkSZKUkSFakiRJysgQLUmSJGVkiJYkSZIyMkRLkiRJGRmiJUmSpIwM0ZIkSVJGhmhJkiQpI0O0JEmSlJEhWpIkScrIEC1JkiRlZIiWJEmSMjJES5IkSRkZoiVJkqSMDNGSJElSRoZoSZIkKSNDtCRJkpSRIVqSJEnKyBAtSZIkZWSIliRJkjIyREuSJEkZGaIlSZKkjAzRkiRJUkaGaEmSJCkjQ7QkSZKUkSFakiRJysgQLUmSJGXUpRAdEcMj4v6IWFx8P6CDfksj4smImBsRs7OuL0mSJJWTrp6Jvgr4XUppEvC74nxHpqWUTkwp1e3l+pIkSVJZ6GqIPg/4YXH6h8A7e3h9SZIkqcdFSmnvV45oSCkNazW/MaX0mksyIuJ5YCOQgP9NKd2QZf1i22XAZcXZI4GFe1343hsBrCvBfnsrxys7xywbxysbxysbxysbxysbxyubUo7XISmlkW0XVu1prYj4LTC6naZ/zrDzM1JKKyPiIOD+iFiQUnoow/oUg/cNWdbpbhExu83lKNoNxys7xywbxysbxysbxysbxysbxyubchyvPYbolNJfd9QWEasjYkxKaVVEjAHWdLCNlcX3NRHxM2Ay8BDQqfUlSZKkctLVa6LvAS4uTl8M/KJth4gYFBGDd00DbwXmd3Z9SZIkqdx0NUR/DTgrIhYDZxXniYiDI2Jmsc8o4I8R8QTwKPDLlNKvd7d+GSvp5SS9kOOVnWOWjeOVjeOVjeOVjeOVjeOVTdmNV5c+WChJkiT1RT6xUJIkScrIEC1JkiRlZIhuIyLeFxFPRUQ+IuratH0hIpZExMKIeFsH6/fZR5lHxF3FR7vPLT7qfW4H/dp9DHxfExFfjogXW43ZOR30O7t4zC2JiD79VM+I+EZELIiIeRHxs4gY1kG/PnuM7el4iYLvFNvnRcTJpaizXETE+Ih4ICKeKf7s//t2+kyNiE2tvle/VIpay8Wevr88xl4REUe2Om7mRsTmiLiyTZ8+fXxFxM0RsSYi5rda1qksVfLfjyklX61ewNEUHuZSD9S1Wn4M8ARQA0wEngUq21n/68BVxemrgP8s9ddUonH8JvClDtqWAiNKXWOpX8CXgX/YQ5/K4rF2GFBdPAaPKXXtJRyztwJVxen/7Oj7q68eY505XoBzgF8BAZwGPFLquks8ZmOAk4vTg4FF7YzZVODeUtdaLq89fX95jHU4LpXASxQe3NF6eZ8+voC/Ak4G5rdatscsVQ6/Hz0T3UZK6ZmUUntPQzwPmJFS2pFSeh5YQuF+1+3169OPMo+IAN4P3FnqWvYDk4ElKaXnUko7gRkUjrE+KaX0m5RSS3H2YWBcKespQ505Xs4Dbk0FDwPDivfp75NSSqtSSnOK01uAZ4Cxpa2q1/MYa99bgGdTSstKXUg5SYWH721os7gzWarkvx8N0Z03Fljean4F7f+gHZVSWgWFH87AQT1QW7l5E7A6pbS4g/YE/CYiHovC49z7siuKf+68uYM/V3X2uOuLLqVwtqs9ffUY68zx4jHVgYg4FDgJeKSd5tMj4omI+FVEHNuzlZWdPX1/eYy173w6Prnk8fVqnclSJT/O9vjEwv1R7OZR5imljh74Eu0s63P3B+zk2F3A7s9Cd/kx8L3F7sYLuA74NwrH0b9RuATm0rabaGfd/fq468wxFhH/DLQAt3ewmT5zjLXRmeOlzx1TnRERtcBPgCtTSpvbNM+h8Cf4xuJnF34OTOrhEsvJnr6/PMbaiIhq4G+BL7TT7PG1d0p+nPXJEJ128yjz3VgBjG81Pw5Y2U6//fpR5nsau4ioAt4NnLKbbXT0GPj9TmePtYi4Ebi3nabOHnf7jU4cYxcD7wDekooXxrWzjT5zjLXRmeOlzx1TexIR/SgE6NtTSj9t2946VKeUZkbE9yJiREppXU/WWS468f3lMfZabwfmpJRWt23w+GpXZ7JUyY8zL+fovHuA8yOiJiImUvhf4qMd9OvLjzL/a2BBSmlFe42x+8fA9yltrhF8F+2PwyxgUkRMLJ7JOJ/CMdYnRcTZwOeBv00pbeugT18+xjpzvNwDfLh4B4XTgE27/mzaFxU/w/F94JmU0n930Gd0sR8RMZnC7871PVdl+ejk95fH2Gt1+Bdaj692dSZLlfz3Y588E707EfEu4BpgJPDLiJibUnpbSumpiLgbeJrCn5E/mVLKFde5Cbg+pTSbwqPL746IjwIvAO8ryRdSOq+55isiDgZuSimdQ+Ex8D8r/ryoAu5IrzwGvq/5ekScSOHPT0uBj8Orxyul1BIRVwD3Ufgk8s0ppadKVG85+C6FO+TcXzyGHk4pXe4xVtDR8RIRlxfbrwdmUrh7whJgG3BJqeotE2cAFwFPxiu35fwnYAK8PGbvBT4RES3AduD8jv4K0ge0+/3lMdaxiBgInEXxZ3xxWevx6tPHV0TcSeEOJSMiYgVwNR1kqXL7/ehjvyVJkqSMvJxDkiRJysgQLUmSJGVkiJYkSZIyMkRLkiRJGRmiJUmSpIwM0ZIkSVJGhmhJkiQpo/8Hhu47W+wgE+YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the sigmoid function\n",
    "vals = np.linspace(-10, 10, num=100, dtype=np.float32)\n",
    "activation = sigmoid(vals)\n",
    "fig = plt.figure(figsize=(12,6))\n",
    "fig.suptitle('Sigmoid function')\n",
    "plt.plot(vals, activation)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.axvline(x=0, color='k')\n",
    "plt.yticks()\n",
    "plt.ylim([-0.5, 1.5]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thinking of neurons as boolean logic gates\n",
    "\n",
    "A logic gate takes in two boolean (true/false or 1/0) inputs, and returns either a 0 or 1 depending on its rule. The truth table for a logic gate shows the outputs for each combination of inputs, (0, 0), (0, 1), (1,0), and (1, 1). For example, let's look at the truth table for an \"OR\" gate:\n",
    "\n",
    "### OR Gate\n",
    "\n",
    "<table>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"3\">OR gate truth table</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"2\">Input</th>\n",
    "<th>Output</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "</table>\n",
    "\n",
    "A neuron that uses the sigmoid activation function outputs a value between (0, 1). This naturally leads us to think about boolean values. Imagine a neuron that takes in two inputs, $x_1$ and $x_2$, and a bias term:\n",
    "\n",
    "![](data/logic01.png)\n",
    "\n",
    "By limiting the inputs of $x_1$ and $x_2$ to be in $\\left\\{0, 1\\right\\}$, we can simulate the effect of logic gates with our neuron. The goal is to find the weights (represented by ? marks above), such that it returns an output close to 0 or 1 depending on the inputs.\n",
    "\n",
    "What numbers for the weights would we need to fill in for this gate to output OR logic? Observe from the plot above that $\\sigma(z)$ is close to 0 when $z$ is largely negative (around -10 or less), and is close to 1 when $z$ is largely positive (around +10 or greater).\n",
    "\n",
    "$$\n",
    "z = w_1 x_1 + w_2 x_2 + b\n",
    "$$\n",
    "\n",
    "Let's think this through:\n",
    "\n",
    "* When $x_1$ and $x_2$ are both 0, the only value affecting $z$ is $b$. Because we want the result for (0, 0) to be close to zero, $b$ should be negative (at least -10)\n",
    "* If either $x_1$ or $x_2$ is 1, we want the output to be close to 1. That means the weights associated with $x_1$ and $x_2$ should be enough to offset $b$ to the point of causing $z$ to be at least 10.\n",
    "* Let's give $b$ a value of -10. How big do we need $w_1$ and $w_2$ to be? \n",
    "    * At least +20\n",
    "* So let's try out $w_1=20$, $w_2=20$, and $b=-10$!\n",
    "\n",
    "![](data/logic02.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logic_gate(w1, w2, b):\n",
    "    # Helper to create logic gate functions\n",
    "    # Plug in values for weight_a, weight_b, and bias\n",
    "    return lambda x1, x2: sigmoid(w1 * x1 + w2 * x2 + b)\n",
    "\n",
    "def test(gate):\n",
    "    # Helper function to test out our weight functions.\n",
    "    for a, b in (0, 0), (0, 1), (1, 0), (1, 1):\n",
    "        print(\"{}, {}: {}\".format(a, b, np.round(gate(a, b))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0, 0: 0.0\n",
      "0, 1: 1.0\n",
      "1, 0: 1.0\n",
      "1, 1: 1.0\n"
     ]
    }
   ],
   "source": [
    "or_gate = logic_gate(20, 20, -10)\n",
    "test(or_gate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"3\">OR gate truth table</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"2\">Input</th>\n",
    "<th>Output</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "</table>\n",
    "\n",
    "This matches! Great! Now you try finding the appropriate weight values for each truth table. Try not to guess and check- think through it logically and try to derive values that work.\n",
    "\n",
    "### AND Gate\n",
    "\n",
    "<table>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"3\">AND gate truth table</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"2\">Input</th>\n",
    "<th>Output</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "Determine what values for the neurons would make this function as an AND gate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0, 0: 0.0\n",
      "0, 1: 0.0\n",
      "1, 0: 0.0\n",
      "1, 1: 1.0\n"
     ]
    }
   ],
   "source": [
    "# TO DO: Fill in the w1, w2, and b parameters such that \n",
    "# the truth table matches\n",
    "### BEGIN SOLUTION\n",
    "w1 = 11\n",
    "w2 = 10\n",
    "b = -20\n",
    "and_gate = logic_gate(w1, w2, b)\n",
    "### END SOLUTION\n",
    "test(and_gate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "Do the same for the NOR gate and the NAND gate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NOR (Not Or) Gate\n",
    "\n",
    "<table>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"3\">NOR gate truth table</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"2\">Input</th>\n",
    "<th>Output</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0, 0: 1.0\n",
      "0, 1: 0.0\n",
      "1, 0: 0.0\n",
      "1, 1: 0.0\n"
     ]
    }
   ],
   "source": [
    "# TO DO: Fill in the w1, w2, and b parameters such that the \n",
    "# truth table matches\n",
    "### BEGIN SOLUTION\n",
    "w1 = -20\n",
    "w2 = -20\n",
    "b = 10\n",
    "nor_gate = logic_gate(w1, w2, b)\n",
    "### END SOLUTION\n",
    "\n",
    "test(nor_gate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NAND (Not And) Gate\n",
    "\n",
    "<table>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"3\">NAND gate truth table</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"2\">Input</th>\n",
    "<th>Output</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0, 0: 1.0\n",
      "0, 1: 1.0\n",
      "1, 0: 1.0\n",
      "1, 1: 0.0\n"
     ]
    }
   ],
   "source": [
    "### BEGIN SOLUTION\n",
    "# TO DO: Fill in the w1, w2, and b parameters such that the \n",
    "# truth table matches\n",
    "w1 = -11\n",
    "w2 = -10\n",
    "b = 20\n",
    "nand_gate = logic_gate(w1, w2, b)\n",
    "### END SOLUTION\n",
    "\n",
    "test(nand_gate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The limits of single neurons\n",
    "\n",
    "If you've taken computer science courses, you may know that the XOR gates are the basis of computation. They can be used as so-called \"half-adders\", the foundation of being able to add numbers together. Here's the truth table for XOR:\n",
    "\n",
    "### XOR (Exclusive Or) Gate\n",
    "\n",
    "<table>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"3\">XOR gate truth table</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<th colspan=\"2\">Input</th>\n",
    "<th>Output</th>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "<td>1</td>\n",
    "</tr>\n",
    "\n",
    "<tr>\n",
    "<td>1</td>\n",
    "<td>1</td>\n",
    "<td>0</td>\n",
    "</tr>\n",
    "\n",
    "</table>\n",
    "\n",
    "Can we create a set of weights such that a single neuron can output this property?\n",
    "\n",
    "It turns out we cannot, since single neurons can't correlate inputs. Can we still use neurons to somehow form an XOR gate?\n",
    "\n",
    "What if we tried something more complex:\n",
    "\n",
    "![](data/logic03.png)\n",
    "\n",
    "Here, we've got the inputs going to two separate gates: the top neuron is an OR gate, and the bottom is a NAND gate. The output of these gates then get passed to another neuron, which is an AND gate. If you work out the outputs at each combination of input values, you'll see that this is an XOR gate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0, 0: 0.0\n",
      "0, 1: 1.0\n",
      "1, 0: 1.0\n",
      "1, 1: 0.0\n"
     ]
    }
   ],
   "source": [
    "# Make sure we have or_gate, nand_gate, and and_gate working from above!\n",
    "def xor_gate(a, b):\n",
    "    c = or_gate(a, b)\n",
    "    d = nand_gate(a, b)\n",
    "    return and_gate(c, d)\n",
    "test(xor_gate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedforward Networks as Matrix Computations\n",
    "\n",
    "We discussed previously how the feed-forward computation of a neural network can be thought of as matrix calculations and activation functions.  We will do some actual computations with matrices to see this in action.\n",
    "\n",
    "![](data/FF_NN.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3\n",
    "Provided below are the following:\n",
    "\n",
    "- Three weight matrices `W_1`, `W_2` and `W_3` representing the weights in each layer.  The convention for these matrices is that each $W_{i,j}$ gives the weight from neuron $i$ in the previous (left) layer to neuron $j$ in the next (right) layer.  \n",
    "- A vector `x_in` representing a single input and a matrix `x_mat_in` representing 7 different inputs.\n",
    "- Two functions: `soft_max_vec` and `soft_max_mat` which apply the soft_max function to a single vector, and row-wise to a matrix.\n",
    "\n",
    "The goals for this exercise are:\n",
    "1. For input `x_in` calculate the inputs and outputs to each layer (assuming sigmoid activations for the middle two layers and soft_max output for the final layer.\n",
    "2. Write a function that does the entire neural network calculation for a single input\n",
    "3. Write a function that does the entire neural network calculation for a matrix of inputs, where each row is a single input.\n",
    "4. Test your functions on `x_in` and `x_mat_in`.\n",
    "\n",
    "This illustrates what happens in a NN during one single forward pass. Roughly speaking, after this forward pass, it remains to compare the output of the network to the known truth values, compute the gradient of the loss function and adjust the weight matrices `W_1`, `W_2` and `W_3` accordingly, and iterate. Hopefully this process will result in better weight matrices and our loss will be smaller afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the matrix W_1\n",
      "\n",
      "[[ 2 -1  1  4]\n",
      " [-1  2 -3  1]\n",
      " [ 3 -2 -1  5]]\n",
      "------------------------------\n",
      "vector input x_in\n",
      "\n",
      "[0.5 0.8 0.2]\n",
      "------------------------------\n",
      "matrix input x_mat_in -- starts with the vector `x_in`\n",
      "\n",
      "[[0.5 0.8 0.2]\n",
      " [0.1 0.9 0.6]\n",
      " [0.2 0.2 0.3]\n",
      " [0.6 0.1 0.9]\n",
      " [0.5 0.5 0.4]\n",
      " [0.9 0.1 0.9]\n",
      " [0.1 0.8 0.7]]\n"
     ]
    }
   ],
   "source": [
    "W_1 = np.array([[2,-1,1,4],[-1,2,-3,1],[3,-2,-1,5]])\n",
    "W_2 = np.array([[3,1,-2,1],[-2,4,1,-4],[-1,-3,2,-5],[3,1,1,1]])\n",
    "W_3 = np.array([[-1,3,-2],[1,-1,-3],[3,-2,2],[1,2,1]])\n",
    "x_in = np.array([.5,.8,.2])\n",
    "x_mat_in = np.array([[.5,.8,.2],[.1,.9,.6],[.2,.2,.3],\n",
    "                     [.6,.1,.9],[.5,.5,.4],[.9,.1,.9],[.1,.8,.7]])\n",
    "\n",
    "def soft_max_vec(vec):\n",
    "    return np.exp(vec)/(np.sum(np.exp(vec)))\n",
    "\n",
    "def soft_max_mat(mat):\n",
    "    return np.exp(mat)/(np.sum(np.exp(mat),axis=1).reshape(-1,1))\n",
    "\n",
    "print('the matrix W_1\\n')\n",
    "print(W_1)\n",
    "print('-'*30)\n",
    "print('vector input x_in\\n')\n",
    "print(x_in)\n",
    "print ('-'*30)\n",
    "print('matrix input x_mat_in -- starts with the vector `x_in`\\n')\n",
    "print(x_mat_in)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.8,  0.7, -2.1,  3.8])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### BEGIN SOLUTION\n",
    "z_2 = np.dot(x_in,W_1)\n",
    "z_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.68997448, 0.66818777, 0.10909682, 0.97811873])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_2 = sigmoid(z_2)\n",
    "a_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.55880727,  4.01355384,  0.48455118, -1.55014198])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z_3 = np.dot(a_2,W_2)\n",
    "z_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.97231549, 0.98225163, 0.61882199, 0.17506576])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_3 = sigmoid(z_3)\n",
    "a_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2.04146788,  1.04718238, -3.47867612])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z_4 = np.dot(a_3,W_3)\n",
    "z_4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.72780576, 0.26927918, 0.00291506])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_out = soft_max_vec(z_4)\n",
    "y_out\n",
    "### END SOLUTION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "## A one-line function to do the entire neural net computation\n",
    "\n",
    "def nn_comp_vec(x):\n",
    "    return soft_max_vec(sigmoid(sigmoid(np.dot(x,W_1)).dot(W_2)).dot(W_3))\n",
    "\n",
    "def nn_comp_mat(x):\n",
    "    return soft_max_mat(sigmoid(sigmoid(np.dot(x,W_1)).dot(W_2)).dot(W_3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.72780576, 0.26927918, 0.00291506])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn_comp_vec(x_in)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.72780576, 0.26927918, 0.00291506],\n",
       "       [0.62054212, 0.37682531, 0.00263257],\n",
       "       [0.69267581, 0.30361576, 0.00370844],\n",
       "       [0.36618794, 0.63016955, 0.00364252],\n",
       "       [0.57199769, 0.4251982 , 0.00280411],\n",
       "       [0.38373781, 0.61163804, 0.00462415],\n",
       "       [0.52510443, 0.4725011 , 0.00239447]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn_comp_mat(x_mat_in)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Machine Learning Foundation (C) 2020 IBM Corporation"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}