notebooks/Linear.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-Connected (Linear) Layer\n",
    "In this notebook, we will look into the forward and the backward the the ```nn.Linear``` layer. We will also manualy derive the expressions for the gradient of the loss respect to the input $\\frac{\\partial L}{\\partial I}$ of this layer and also the gradient of the loss with respect to the weights $\\frac{\\partial L}{\\partial W}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "require 'nn';\n",
    "n = torch.rand(5)\n",
    "lin = nn.Linear(5,4)\n",
    "m = lin:forward(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.2647\n",
       " 0.7477\n",
       " 0.5382\n",
       " 0.1914\n",
       " 0.0834\n",
       "[torch.DoubleTensor of size 5]\n",
       "\n"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.1242\n",
       " 0.2382\n",
       "-0.6489\n",
       "-0.2896\n",
       "[torch.DoubleTensor of size 4]\n",
       "\n"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Output (manually compute)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.1242\n",
       " 0.2382\n",
       "-0.6489\n",
       "-0.2896\n",
       "[torch.DoubleTensor of size 4]\n",
       "\n"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m_ = lin.weight*n + lin.bias\n",
    "print(m_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set gradient of the loss with respect of input of the next layer $\\frac{\\partial L}{\\partial I^{l+1}}$ of next layer to random values, and backward propagae the gradient from the next layer via this linear layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nextgrad = torch.rand(4)\n",
    "lin:backward(n, nextgrad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gradient of the loss with respect of input of this layer $\\frac{\\partial L}{\\partial I}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.1486\n",
       "-0.3457\n",
       "-0.0236\n",
       " 0.2292\n",
       " 0.3097\n",
       "[torch.DoubleTensor of size 5]\n",
       "\n"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin.gradInput"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Relation for calcuating the gradient of loss with respect to the input: $\\frac{\\partial L}{\\partial I^{l}} = \\frac{\\partial L}{\\partial I^{l+1}} \\times \\frac{\\partial O^{l}}{\\partial I^{l}}$. Note how the jacobian $\\frac{\\partial O^{l}}{\\partial I^{l}} = W^{l}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.1486 -0.3457 -0.0236  0.2292  0.3097\n",
       "[torch.DoubleTensor of size 1x5]\n",
       "\n"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nextgrad:reshape(1,4)*lin.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This layers gradient of Loss with respect to the weights: $\\frac{\\partial L}{\\partial W^{l}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.2135  0.6033  0.4343  0.1544  0.0673\n",
       " 0.0556  0.1572  0.1132  0.0402  0.0175\n",
       " 0.0850  0.2402  0.1729  0.0615  0.0268\n",
       " 0.1717  0.4850  0.3491  0.1242  0.0541\n",
       "[torch.DoubleTensor of size 4x5]\n",
       "\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin.gradWeight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Relation for calcuating the gradient of the loss with respect to the weights of this layer: $\\frac{\\partial L}{\\partial W^{l}} = \\frac{\\partial L}{\\partial O} \\frac{\\partial O}{\\partial W_{l}}$. <br/>\n",
    "Let us first calcuate $\\frac{\\partial O}{\\partial W_{l}}$ which is a jacobian of size $4\\times20$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dodw = torch.Tensor(4,20)\n",
    "st = 1\n",
    "for i = 1, 4 do\n",
    "    for j = 1, 5 do\n",
    "        dodw[i][st]=n[j]\n",
    "        st = st + 1\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finally, we can now calculate $\\frac{\\partial L}{\\partial W^{l}} = \\frac{\\partial L}{\\partial O} \\frac{\\partial O}{\\partial W_{l}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.2135  0.6033  0.4343  0.1544  0.0673\n",
       " 0.0556  0.1572  0.1132  0.0402  0.0175\n",
       " 0.0850  0.2402  0.1729  0.0615  0.0268\n",
       " 0.1717  0.4850  0.3491  0.1242  0.0541\n",
       "[torch.DoubleTensor of size 4x5]\n",
       "\n"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(nextgrad:reshape(1,4) * dodw):reshape(4,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This layers gradient of output with respect to the bias: $\\frac{\\partial L}{\\partial b^{l}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.8068\n",
       " 0.2102\n",
       " 0.3212\n",
       " 0.6487\n",
       "[torch.DoubleTensor of size 4]\n",
       "\n"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin.gradBias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Relation for calcuating this layers gradient of output with respect to the bias: $\\frac{\\partial L}{\\partial b^{l}} = \\frac{\\partial L}{\\partial I^{l+1}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.8068\n",
       " 0.2102\n",
       " 0.3212\n",
       " 0.6487\n",
       "[torch.DoubleTensor of size 4x1]\n",
       "\n"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nextgrad:reshape(1,4):t()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "iTorch",
   "language": "lua",
   "name": "itorch"
  },
  "language_info": {
   "name": "lua",
   "version": "5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}