PyTorch 101 Part 3 - Advance PyTorch Usage.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Advance PyTorch Usage.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"pI4KoQJMxXsm","colab_type":"text"},"source":["# PyTorch Adavnce Usage \n","\n","Hello readers, this is yet another post in a series we are doing PyTorch. This post is aimed for PyTorch users who are familiar with basics of PyTorch and would like to move to an intermediate level. While we have covered how to implement a basic classifier in an earlier post, in this post, we will be discussing how to implement more complex deep learning functionality using PyTorch. Some of the objectives of this posts are to make you understand. \n","\n","1. What is the difference between PyTorch classes like `nn.Module`, `nn.Functional` and when to use which\n","2. How to customise your training options such as different learning rates for different layers, weight initialisation etc. \n","3. How to use Tensorboard with PyTorch.\n","4. How to visualise the computation graph and print it's intermediate values for debugging.  \n","\n","So, let's get started. \n","\n","\n","## nn.Module vs nn.Functional\n","\n","This is something that comes quite a lot especially when you are reading open source code. In PyTorch, layers are often implemented as either one of `torch.nn.Module` objects or `torch.nn.Functional` functions. Which one to use? Which is better? \n","\n","As we had covered in Part 2, `torch.nn.Module` is basically the cornernstone of PyTorch. The way it works is you first define an `nn.Module` object,  and then invoke it's `forward` method to run it. This is a Object Oriented way of doing things. \n","\n","On the other hand, `nn.functional` provides some layers / activations in form of functions that can be directly called on the input rather than defining the an object. For example, in order to rescale an image tensor, you call `torch.nn.functional.interpolate` on an image tensor.\n","\n","\n","\n","\n","\n","\n"]},{"cell_type":"code","metadata":{"id":"TjQsU_s-P801","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"SMnMW1EBCKQ-","colab_type":"code","outputId":"b92b40f6-4c75-4d29-a74f-21b0034de0f4","executionInfo":{"status":"ok","timestamp":1556995002859,"user_tz":-330,"elapsed":4810,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["import torch\n","import torch.nn as nn\n","import torch.nn.functional as F\n","\n","inp = torch.randn(1,3,64,64)     # random input image\n","\n","# Same thing using two approaches\n","# ---------------------------------------\n","\n","# torch.nn\n","avg_pool = nn.AvgPool2d(4)     # create an object\n","nn_out = avg_pool(inp)         # invoke the forward method\n","\n","# torch.nn.Functional\n","f_out = F.avg_pool2d(inp, 4)\n","\n","\n","print (torch.equal(nn_out, f_out))        # check whether the same result is produced"],"execution_count":1,"outputs":[{"output_type":"stream","text":["True\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6TIx-ed-CShc","colab_type":"text"},"source":["So how do we choose what to use when? When the layer / activation / loss we are implementing has a loss. \n","\n","### Understanding Stateful-ness\n","\n","Normally, any layer can be seen as a function. For example, a convolutional operation is just a bunch of multiplication and addition operations. So, it makes sense for us to just implement it as a function right? But wait, the layer holds weights which need to be stored and updated while we are training. Therefore, from a programmatical angle, a layer is more than function. It also needs to hold data, which changes as we train our network. \n","\n","I now want to you to stress upon that fact that the data held by the convolutional layer **changes**. This means that the layer has a *state* which changes as we train. For us to implement a function that does the convolutional operation, we would also need to define a data structure to hold the weights of the layer seperately from the function itself. And then, make this external data structure an input to our function. \n","\n","Or just to beat the hassle, we could just define a class to hold the data structure, and make convolutional operation as an member function. This would really ease up our job, as we don't have to worry about stateful variables existing outside of the function. In these cases, we would prefer to use the `nn.Module` objects where we have weights or other states which might define the behaviour of the layer. For example, a dropout / Batch Norm layer behaves differently during training and inference. \n","\n","On the other hand, where no state or weights are required, one could use the `nn.functional`. Examples being, resizing (`nn.functional.interpolate`),  average pooling (`nn.functional.AvgPool2d`). \n","\n","Despite the above reasoning, most of the `nn.Module` classes have their `nn.functional` counterparts. However, the above line of reasoning is to be respected during practical work. \n","\n","\n","## nn.Parameter\n","\n","An important class in PyTorch is the `nn.Parameter` class, which to my surprise, has gotten little coverage in PyTorch introductory texts. Consider the following case. "]},{"cell_type":"code","metadata":{"id":"iEhpyfQoJWeN","colab_type":"code","outputId":"96ea708a-15bc-4cae-c573-1e37e19202c1","executionInfo":{"status":"ok","timestamp":1556995002863,"user_tz":-330,"elapsed":4794,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":221}},"source":["class net(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.conv = nn.Linear(10,5)\n","    \n","  def forward(self, x):\n","    return self.linear(x)\n","\n","\n","myNet = net()\n","print(list(myNet.parameters()))"],"execution_count":2,"outputs":[{"output_type":"stream","text":["[Parameter containing:\n","tensor([[ 0.1846,  0.0970,  0.2853, -0.0155,  0.0444,  0.0111, -0.2379,  0.3031,\n","         -0.0066, -0.3082],\n","        [-0.0597,  0.3029,  0.1465, -0.1953, -0.0772,  0.2787,  0.0993,  0.1490,\n","          0.1092,  0.2071],\n","        [-0.0020, -0.0827, -0.0210, -0.2498, -0.2665, -0.0401,  0.1341,  0.2261,\n","          0.1353, -0.0535],\n","        [ 0.2528, -0.2944,  0.2399,  0.0518, -0.1557,  0.3133,  0.2751,  0.1018,\n","          0.0560, -0.0129],\n","        [ 0.2143,  0.1001, -0.2193,  0.2649, -0.1005,  0.2521, -0.0974, -0.3137,\n","          0.3014, -0.0938]], requires_grad=True), Parameter containing:\n","tensor([-0.1840,  0.2230,  0.0539, -0.2962,  0.2762], requires_grad=True)]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"pAuFLAoOLI3V","colab_type":"text"},"source":["Each `nn.Module` has a `parameters()` function which returns, well, it's trainable parameters. We have to implicitly define what these parameters are. In definition of `nn.Conv2d`, the authors of PyTorch defined the parameters to that of a layer. However, notice on thing, that when we defined `net`, we didn't need to add the `parameters` of `nn.Conv2d` to `parameters` of `net`. It happened implicitly by virtue of setting `nn.Conv2d` object as a memeber of the `net` object. \n","\n","\n","This is internally faciliated by the `nn.Parameter` class, which subclasses the `Tensor` class. When we invoke `parameters()` function of a `nn.Module` object, it returns all it's members which are `nn.Parameter` objects. \n","\n","Infact, all the training weights of `nn.Module` classes are implemented as `nn.Parameter` objects. Whenever, a `nn.Module` (`nn.Conv2d` in our case) is assigned as a member of another `nn.Module`, the \"parameters\" of the assignee object (i.e. the weights of `nn.Conv2d`) are also added the \"parameters\" of the object which is beeng assigned to (parameters of `net` object). This is called registering \"parameters\" of a `nn.Module`\n","\n","If you try to assign a tensor to the `nn.Module` object, it won't show up in the `parameters()` unless you define it as `nn.Parameter` object. This has been done to facilitate scenarios where you might need to cache a non-differentiable tensor,  example in case,  caching previous output in case of RNNs. "]},{"cell_type":"code","metadata":{"id":"tx35QzM_K-rl","colab_type":"code","outputId":"6ca7fc6d-74ef-44b8-a737-8a3f2d242bbf","executionInfo":{"status":"ok","timestamp":1556995003648,"user_tz":-330,"elapsed":1026,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":221}},"source":["class net1(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.conv = nn.Linear(10,5)\n","    self.tens = torch.ones(3,4)                       # This won't show up in a parameter list \n","    \n","  def forward(self, x):\n","    return self.linear(x)\n","\n","\n","myNet = net1()\n","print(list(myNet.parameters()))"],"execution_count":3,"outputs":[{"output_type":"stream","text":["[Parameter containing:\n","tensor([[-0.0494, -0.1193, -0.2282, -0.1652, -0.0605, -0.0831,  0.0609, -0.0288,\n","         -0.1870,  0.1633],\n","        [ 0.0578, -0.1171, -0.1310,  0.3141,  0.2610, -0.0521,  0.2128,  0.0202,\n","          0.0022, -0.1664],\n","        [-0.2865,  0.0735, -0.0360,  0.1419,  0.2823,  0.1375, -0.1025,  0.0339,\n","          0.1438, -0.0067],\n","        [ 0.2777, -0.2733,  0.0601, -0.2115, -0.2909,  0.1398, -0.1229, -0.2810,\n","          0.2425,  0.2586],\n","        [ 0.2198, -0.2501, -0.1739, -0.1049, -0.0087,  0.0969, -0.0273, -0.2518,\n","         -0.2899,  0.2574]], requires_grad=True), Parameter containing:\n","tensor([ 0.2200, -0.2819, -0.2864,  0.1680,  0.1531], requires_grad=True)]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"6TdFyxh4Phbk","colab_type":"code","outputId":"bf163c06-c470-45d1-ed3c-28c440ee2f29","executionInfo":{"status":"ok","timestamp":1556995008462,"user_tz":-330,"elapsed":948,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":272}},"source":["class net2(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.conv = nn.Linear(10,5) \n","    self.tens = nn.Parameter(torch.ones(3,4))                       # This will show up in a parameter list \n","    \n","  def forward(self, x):\n","    return self.linear(x)\n","\n","\n","myNet = net2()\n","print(list(myNet.parameters()))"],"execution_count":4,"outputs":[{"output_type":"stream","text":["[Parameter containing:\n","tensor([[1., 1., 1., 1.],\n","        [1., 1., 1., 1.],\n","        [1., 1., 1., 1.]], requires_grad=True), Parameter containing:\n","tensor([[ 0.2822, -0.2529, -0.2258,  0.2095, -0.1306, -0.1231,  0.2388, -0.0766,\n","          0.1691,  0.1905],\n","        [ 0.0399, -0.2092, -0.1827,  0.2246,  0.2999, -0.2866,  0.2838,  0.0032,\n","         -0.0297,  0.0120],\n","        [ 0.0992, -0.2248, -0.1378, -0.2271, -0.1378, -0.1859, -0.1138,  0.0954,\n","         -0.2970,  0.1850],\n","        [ 0.0490,  0.0403, -0.1714,  0.3115,  0.0583, -0.0978,  0.0268,  0.2462,\n","         -0.0775, -0.3157],\n","        [-0.2351,  0.3097, -0.1333, -0.2439,  0.1744, -0.1694, -0.1920, -0.2379,\n","         -0.0764, -0.2017]], requires_grad=True), Parameter containing:\n","tensor([-0.2704,  0.1900, -0.1803,  0.2859,  0.1204], requires_grad=True)]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"04zpNbOvQRVT","colab_type":"code","outputId":"b195155f-be87-494b-93a7-69dfa790cc84","executionInfo":{"status":"ok","timestamp":1556995011743,"user_tz":-330,"elapsed":1088,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":459}},"source":["class net3(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.conv = nn.Linear(10,5) \n","    self.net  = net2()                      # Parameters of net2 will show up in list of parameters of net3\n","    \n","  def forward(self, x):\n","    return self.linear(x)\n","\n","\n","myNet = net3()\n","print(list(myNet.parameters()))"],"execution_count":5,"outputs":[{"output_type":"stream","text":["[Parameter containing:\n","tensor([[-0.1766, -0.2913,  0.0239, -0.2141, -0.1376, -0.0114, -0.1342, -0.1564,\n","          0.1679, -0.2835],\n","        [-0.2454,  0.2568, -0.0689,  0.1794,  0.1163, -0.2717,  0.2153, -0.1286,\n","         -0.3019,  0.0099],\n","        [-0.2485,  0.0370,  0.2365,  0.1740, -0.0854,  0.1632,  0.2836, -0.3125,\n","          0.1529,  0.2212],\n","        [ 0.1464,  0.3054, -0.1252,  0.0500, -0.2272, -0.1021, -0.0869, -0.0862,\n","         -0.0060,  0.0478],\n","        [-0.2726, -0.1273, -0.0293,  0.0069, -0.1914, -0.1478,  0.1430,  0.3047,\n","          0.2626,  0.2579]], requires_grad=True), Parameter containing:\n","tensor([-0.0170, -0.0150, -0.1105, -0.2139,  0.1391], requires_grad=True), Parameter containing:\n","tensor([[1., 1., 1., 1.],\n","        [1., 1., 1., 1.],\n","        [1., 1., 1., 1.]], requires_grad=True), Parameter containing:\n","tensor([[-0.2596,  0.0676,  0.0063,  0.3053,  0.1257, -0.0531,  0.2886,  0.3121,\n","          0.2968, -0.0409],\n","        [ 0.0509,  0.1206, -0.2226, -0.2056,  0.0236,  0.1018,  0.2560, -0.0791,\n","          0.1422, -0.2279],\n","        [ 0.3125,  0.2330,  0.3036,  0.1129,  0.1584,  0.2718,  0.2366,  0.0954,\n","          0.0224, -0.2901],\n","        [ 0.2803,  0.0581, -0.1077, -0.2281,  0.1905, -0.1933, -0.0240,  0.2780,\n","         -0.2991, -0.1184],\n","        [-0.1652, -0.0257, -0.2650, -0.0311, -0.1961, -0.2487, -0.2386, -0.1093,\n","         -0.0065, -0.1091]], requires_grad=True), Parameter containing:\n","tensor([-0.1839, -0.1933,  0.1486,  0.0720,  0.3085], requires_grad=True)]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ue5hWr_3q1Og","colab_type":"text"},"source":["### nn.ModuleList and nn.ParameterList()\n","\n","I remember I had to you a `nn.ModuleList` when I was implementing YOLO v3 in PyTorch. I had to create the network by parsing a text file which contained the architecture. I stored all the `nn.Module` objects corresponding in a Python list and then made the list a member of my `nn.Module` object representing the network.  \n","\n","To simplify it, something like this. "]},{"cell_type":"code","metadata":{"id":"WwMI5oQrtPHH","colab_type":"code","outputId":"63a2a0d7-1458-4412-c197-ffa6bfb74cb6","executionInfo":{"status":"ok","timestamp":1556995014811,"user_tz":-330,"elapsed":1204,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["layer_list = [nn.Conv2d(5,5,3), nn.BatchNorm2d(5), nn.Linear(5,2)]\n","\n","class myNet(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.layers = layer_list\n","  \n","  def forward(x):\n","    for layer in self.layers:\n","      x = layer(x)\n","\n","net = myNet()\n","\n","print(list(net.parameters())) "],"execution_count":6,"outputs":[{"output_type":"stream","text":["[]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"tWrK8Xpsu7cV","colab_type":"text"},"source":["As you see, unlike when we would register individual modules, assigning a Python List doesn't register the parameters of Modules inside the list. To fix this, we wrap our list with the `nn.ModuleList` class, and then assign it as a member of the network class. "]},{"cell_type":"code","metadata":{"id":"J4AhMFMCucZP","colab_type":"code","outputId":"51740c4a-5a37-431e-bac2-15e883fc16a5","executionInfo":{"status":"ok","timestamp":1556995019708,"user_tz":-330,"elapsed":1185,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":1887}},"source":["layer_list = [nn.Conv2d(5,5,3), nn.BatchNorm2d(5), nn.Linear(5,2)]\n","\n","class myNet(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.layers = nn.ModuleList(layer_list)\n","  \n","  def forward(x):\n","    for layer in self.layers:\n","      x = layer(x)\n","\n","net = myNet()\n","\n","print(list(net.parameters())) "],"execution_count":7,"outputs":[{"output_type":"stream","text":["[Parameter containing:\n","tensor([[[[-1.4344e-01,  1.3879e-01,  3.4137e-02],\n","          [-2.4901e-02, -1.1976e-01,  7.0103e-02],\n","          [ 8.4627e-02, -1.1385e-01, -1.1483e-01]],\n","\n","         [[-2.7231e-02, -1.0273e-01,  1.2866e-01],\n","          [ 4.9336e-04,  7.4797e-02,  1.4066e-01],\n","          [ 2.1796e-02, -1.0067e-01, -1.8363e-02]],\n","\n","         [[ 4.3644e-02, -3.5372e-02,  6.2309e-02],\n","          [ 6.4128e-02,  1.0099e-01,  6.6372e-02],\n","          [-1.2661e-02,  2.9149e-03,  1.0361e-01]],\n","\n","         [[ 1.2332e-01, -9.6792e-03,  1.4889e-01],\n","          [ 1.0564e-01,  1.8913e-02, -8.1076e-02],\n","          [ 1.2256e-01, -8.3358e-02, -1.0419e-01]],\n","\n","         [[ 1.1769e-01,  1.0935e-01,  1.2643e-01],\n","          [ 1.0030e-01, -3.0156e-02,  5.3058e-02],\n","          [-5.6571e-02,  1.0527e-01, -2.8197e-02]]],\n","\n","\n","        [[[-7.1571e-02,  1.0665e-01,  1.1227e-02],\n","          [-8.9246e-02,  1.3762e-01,  8.4198e-02],\n","          [-2.2030e-03, -1.4803e-01,  5.6313e-02]],\n","\n","         [[-1.4160e-01,  6.9753e-02, -9.0819e-02],\n","          [-1.1503e-01, -2.4388e-02,  1.1996e-01],\n","          [-7.7597e-02, -1.2044e-01, -4.2590e-02]],\n","\n","         [[ 4.1929e-02,  1.3535e-01, -1.6573e-02],\n","          [-1.6171e-02, -4.3973e-02,  1.9293e-02],\n","          [-8.8145e-02,  1.0897e-01, -5.7348e-02]],\n","\n","         [[-3.5718e-02,  1.4188e-01,  1.7953e-02],\n","          [-2.7124e-02,  9.6864e-02, -5.5249e-02],\n","          [-2.1549e-02, -2.8258e-02,  1.0488e-01]],\n","\n","         [[-3.5689e-02,  1.4829e-01, -4.1075e-02],\n","          [ 1.4898e-01, -9.6041e-02,  1.3397e-01],\n","          [ 2.4418e-02, -6.0716e-02, -1.4608e-01]]],\n","\n","\n","        [[[-4.5245e-02, -1.8057e-02,  9.9755e-02],\n","          [-1.2868e-01, -1.0389e-01,  3.0316e-03],\n","          [-1.4211e-01, -1.2536e-01, -4.0871e-02]],\n","\n","         [[-1.1863e-01, -4.5916e-03,  8.6010e-02],\n","          [-1.3670e-01, -6.8658e-02, -5.8052e-02],\n","          [-1.4634e-01, -1.1187e-01,  6.6570e-02]],\n","\n","         [[ 1.4498e-01,  1.7937e-02,  4.3052e-02],\n","          [-1.5224e-02, -3.2870e-02, -4.6386e-02],\n","          [ 1.2305e-02, -9.0238e-02,  9.3400e-02]],\n","\n","         [[-7.7733e-02, -8.1225e-02, -1.4324e-01],\n","          [ 3.4277e-02,  1.2761e-02,  8.6574e-02],\n","          [-1.0810e-02, -2.0431e-03, -6.3360e-03]],\n","\n","         [[-1.4672e-01,  2.3925e-02,  2.0265e-02],\n","          [-1.6099e-02, -1.6111e-02,  1.0867e-01],\n","          [-2.2614e-02,  6.0833e-02,  1.4179e-01]]],\n","\n","\n","        [[[-2.1235e-02,  8.6182e-02,  2.6255e-02],\n","          [ 5.6620e-02, -1.7150e-02,  7.0741e-02],\n","          [ 1.0051e-01, -4.2621e-03,  1.1838e-01]],\n","\n","         [[-4.9893e-02, -4.6008e-02, -1.2367e-02],\n","          [-1.3241e-01,  8.6646e-02,  1.3246e-01],\n","          [ 1.4485e-02, -7.2532e-02, -1.3045e-01]],\n","\n","         [[-6.8479e-03,  6.2704e-02,  1.3788e-02],\n","          [ 8.1182e-02,  1.1816e-01,  1.4023e-01],\n","          [ 2.1013e-02, -6.2293e-02,  9.7996e-02]],\n","\n","         [[ 1.5030e-02,  1.0053e-01, -1.3421e-01],\n","          [ 5.3346e-05, -4.6415e-02, -1.2296e-01],\n","          [ 6.8996e-02,  2.1112e-02, -3.2167e-02]],\n","\n","         [[ 4.6684e-02,  8.7871e-02,  4.7424e-02],\n","          [-5.2580e-02, -1.4412e-01,  1.0180e-01],\n","          [ 7.9472e-02, -2.7574e-02, -3.4423e-02]]],\n","\n","\n","        [[[ 7.3160e-03, -1.3664e-02,  5.2543e-02],\n","          [-5.9628e-02,  4.9382e-02,  8.1334e-02],\n","          [-8.5958e-02, -1.3012e-01, -2.5429e-02]],\n","\n","         [[-7.1023e-03,  1.1696e-01,  6.0417e-02],\n","          [-8.7534e-02, -6.7439e-02, -1.3679e-02],\n","          [-1.1923e-01, -4.1294e-02,  8.3034e-02]],\n","\n","         [[-6.2747e-02, -1.3231e-02, -2.2157e-02],\n","          [-7.0807e-02,  6.7693e-02,  7.5537e-03],\n","          [-1.3774e-01, -1.3412e-01,  7.9819e-03]],\n","\n","         [[ 9.1668e-03,  3.4620e-03,  6.2604e-02],\n","          [-1.1636e-01,  9.7796e-02,  1.1190e-01],\n","          [ 1.1515e-01, -9.9859e-02,  6.1661e-03]],\n","\n","         [[ 1.4358e-01, -8.3063e-02, -7.9493e-02],\n","          [-9.4298e-02, -1.4497e-01,  9.1234e-02],\n","          [ 3.0470e-02,  6.5455e-03,  1.2644e-01]]]], requires_grad=True), Parameter containing:\n","tensor([-0.1324, -0.0545, -0.1448, -0.1349, -0.0004], requires_grad=True), Parameter containing:\n","tensor([0.3889, 0.8522, 0.7708, 0.4505, 0.4050], requires_grad=True), Parameter containing:\n","tensor([0., 0., 0., 0., 0.], requires_grad=True), Parameter containing:\n","tensor([[ 0.3010, -0.3905, -0.1660, -0.3490,  0.3708],\n","        [-0.3473, -0.4291, -0.3264,  0.2176,  0.1121]], requires_grad=True), Parameter containing:\n","tensor([-0.4434,  0.1025], requires_grad=True)]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"WUv_-oSFvb5N","colab_type":"text"},"source":["Similarly, a list of tensors can be registered by wrapping the list inside a `nn.ParameterList` class. \n","\n","## Weight Initialisation\n","\n","Weight initialisation can influence the results of your training. What's more, you may require different weight initialisation schemes for different sort of layers. This can be accomplished by the `modules` and `apply functions. `modules` is a member function of `nn.Module` class which returns an iterator containing all the memeber `nn.Module` objects of a `nn.Module` function. Then `apply`  unction can be called on each  nn.Module`  to set it's initialisation. \n","\n"]},{"cell_type":"code","metadata":{"id":"iQeOdIed0cNG","colab_type":"code","outputId":"1f2a929f-c319-47fc-bcbd-b09713cb727f","executionInfo":{"status":"ok","timestamp":1556995024364,"user_tz":-330,"elapsed":996,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":303}},"source":["import matplotlib.pyplot as plt\n","%matplotlib inline\n","\n","class myNet(nn.Module):\n"," \n","  def __init__(self):\n","    super().__init__()\n","    self.conv = nn.Conv2d(10,10,3)\n","    self.bn = nn.BatchNorm2d(10)\n","  \n","  def weights_init(self):\n","    for module in self.modules():\n","      if isinstance(module, nn.Conv2d):\n","        nn.init.normal_(module.weight, mean = 0, std = 1)\n","        nn.init.constant_(module.bias, 0)\n","\n","Net = myNet()\n","Net.weights_init()\n","\n","for module in Net.modules():\n","  if isinstance(module, nn.Conv2d):\n","    weights = module.weight\n","    weights = weights.reshape(-1).detach().cpu().numpy()\n","    print(module.bias)                                       # Bias to zero\n","    plt.hist(weights)\n","    plt.show()\n","    \n","  "],"execution_count":8,"outputs":[{"output_type":"stream","text":["Parameter containing:\n","tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], requires_grad=True)\n"],"name":"stdout"},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEIxJREFUeJzt3Xus33V9x/HnS0C3oAYcZ00H7Q4u\n1QyNK+6EmXgJG164GJElYzTGoZBVEsgwc9GKiTgNCc7b4i64GhogQYStMEnADcaYzGSop9jVctPC\nSmhT2wJTYRi3wnt/nG/nj+6055zfpb+eT5+P5Jff9/v5Xj7vbwqvfvv5fS+pKiRJ7XrRuAuQJI2W\nQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklq3JHjLgDguOOOq8nJyXGXIUmLyoYN\nG56oqom51jskgn5ycpLp6elxlyFJi0qSx+aznkM3ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEG\nvSQ1bs6gT7Isyd1JHkhyf5JLu/ZXJLkzyQ+672O79iT5YpItSTYlef2oD0KStH/zOaPfA3yoqk4C\n3gBcnOQkYA1wV1WtAO7q5gHOAFZ0n9XAVUOvWpI0b3PeGVtVO4Ad3fTTSR4EjgfOBk7tVrsW+Bfg\nI137dTXz1vF7kxyTZGm3H6kvk2tuG1vfW688a2x9S8OwoDH6JJPAycC3gCU94f1DYEk3fTzweM9m\n27o2SdIYzDvok7wUWA98sKp+0rusO3uvhXScZHWS6STTu3fvXsimkqQFmFfQJzmKmZC/vqpu7pp3\nJlnaLV8K7OratwPLejY/oWt7gapaW1VTVTU1MTHnw9ckSX2ac4w+SYCrgQer6vM9i24Fzgeu7L6/\n1tN+SZKvAr8F/NjxeS1m4/p9wN8GNCzzeUzxG4H3At9LsrFru4yZgL8pyYXAY8C53bLbgTOBLcCz\nwPuHWrEkaUHmc9XNN4HsZ/Fps6xfwMUD1iVJGhLvjJWkxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN\nM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGzRn0SdYl\n2ZVkc0/bjUk2dp+te18xmGQyyU97ln1plMVLkuY2n3fGXgP8JXDd3oaq+v2900k+B/y4Z/1Hqmrl\nsAqUJA1mPu+MvSfJ5GzLkoSZl4L/znDLkiQNy6Bj9G8GdlbVD3raTkzy3STfSPLm/W2YZHWS6STT\nu3fvHrAMSdL+DBr0q4AbeuZ3AMur6mTgj4GvJHn5bBtW1dqqmqqqqYmJiQHLkCTtT99Bn+RI4HeB\nG/e2VdXPqurJbnoD8AjwqkGLlCT1b5Az+rcCD1XVtr0NSSaSHNFNvxJYATw6WImSpEHM5/LKG4B/\nA16dZFuSC7tF5/HCYRuAtwCbusst/w64qKqeGmbBkqSFmc9VN6v20/6+WdrWA+sHL0uSNCzeGStJ\njTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaN5+nV0oag8k1t42t761XnjW2vjV8ntFLUuMMeklqnEEv\nSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1Lj5vOGqXVJdiXZ3NP2iSTbk2zsPmf2LPtoki1J\nHk7yjlEVLkman/mc0V8DnD5L+xeqamX3uR0gyUnMvGLwNd02f733HbKSpPGYM+ir6h5gvu99PRv4\nalX9rKr+A9gCnDJAfZKkAQ0yRn9Jkk3d0M6xXdvxwOM962zr2iRJY9Lv0yuvAj4FVPf9OeCChewg\nyWpgNcDy5cv7LEMH2zifqCipP32d0VfVzqp6rqqeB77Mz4dntgPLelY9oWubbR9rq2qqqqYmJib6\nKUOSNA99BX2SpT2z5wB7r8i5FTgvyUuSnAisAL49WImSpEHMOXST5AbgVOC4JNuAy4FTk6xkZuhm\nK/ABgKq6P8lNwAPAHuDiqnpuNKVLkuZjzqCvqlWzNF99gPWvAK4YpChJ0vB4Z6wkNc6gl6TGGfSS\n1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN\nM+glqXEGvSQ1bs6gT7Iuya4km3vaPpPkoSSbktyS5JiufTLJT5Ns7D5fGmXxkqS5zeeM/hrg9H3a\n7gReW1WvA74PfLRn2SNVtbL7XDScMiVJ/Zoz6KvqHuCpfdruqKo93ey9wAkjqE2SNATDGKO/APh6\nz/yJSb6b5BtJ3jyE/UuSBnDkIBsn+RiwB7i+a9oBLK+qJ5P8JvD3SV5TVT+ZZdvVwGqA5cuXD1KG\nJOkA+j6jT/I+4J3Ae6qqAKrqZ1X1ZDe9AXgEeNVs21fV2qqaqqqpiYmJfsuQJM2hr6BPcjrwYeBd\nVfVsT/tEkiO66VcCK4BHh1GoJKk/cw7dJLkBOBU4Lsk24HJmrrJ5CXBnEoB7uyts3gJ8Msn/AM8D\nF1XVU7PuWJJ0UMwZ9FW1apbmq/ez7npg/aBFSZKGxztjJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BL\nUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXHzCvok\n65LsSrK5p+0VSe5M8oPu+9iuPUm+mGRLkk1JXj+q4iVJc5vvGf01wOn7tK0B7qqqFcBd3TzAGcy8\nFHwFsBq4avAyJUn9mlfQV9U9wL4v+T4buLabvhZ4d0/7dTXjXuCYJEuHUawkaeEGGaNfUlU7uukf\nAku66eOBx3vW29a1vUCS1Ummk0zv3r17gDIkSQcylB9jq6qAWuA2a6tqqqqmJiYmhlGGJGkWgwT9\nzr1DMt33rq59O7CsZ70TujZJ0hgMEvS3Aud30+cDX+tp/4Pu6ps3AD/uGeKRJB1kR85npSQ3AKcC\nxyXZBlwOXAnclORC4DHg3G7124EzgS3As8D7h1yzJGkB5hX0VbVqP4tOm2XdAi4epChJ0vB4Z6wk\nNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1Lj\nDHpJapxBL0mNM+glqXHzevHIbJK8Grixp+mVwMeBY4A/BHZ37ZdV1e19VyhJGkjfQV9VDwMrAZIc\nwcwLwG9h5tWBX6iqzw6lQknSQIY1dHMa8EhVPTak/UmShmRYQX8ecEPP/CVJNiVZl+TYIfUhSerD\nwEGf5MXAu4C/7ZquAn6NmWGdHcDn9rPd6iTTSaZ379492yqSpCEYxhn9GcB9VbUToKp2VtVzVfU8\n8GXglNk2qqq1VTVVVVMTExNDKEOSNJu+f4ztsYqeYZskS6tqRzd7DrB5CH1IOogm19w2ln63XnnW\nWPpt3UBBn+Ro4G3AB3qa/yzJSqCArfsskyQdZAMFfVX9F/BL+7S9d6CKJElD5Z2xktQ4g16SGmfQ\nS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0k\nNc6gl6TGGfSS1LiB3xmbZCvwNPAcsKeqppK8ArgRmGTmdYLnVtV/DtqXJGnhhvFycIDfrqoneubX\nAHdV1ZVJ1nTzHxlSX4e9cb24WdLiNKqhm7OBa7vpa4F3j6gfSdIchhH0BdyRZEOS1V3bkqra0U3/\nEFgyhH4kSX0YxtDNm6pqe5JfBu5M8lDvwqqqJLXvRt1fCqsBli9fPoQyJEmzGfiMvqq2d9+7gFuA\nU4CdSZYCdN+7ZtlubVVNVdXUxMTEoGVIkvZjoKBPcnSSl+2dBt4ObAZuBc7vVjsf+Nog/UiS+jfo\n0M0S4JYke/f1lar6hyTfAW5KciHwGHDugP1Ikvo0UNBX1aPAb8zS/iRw2iD7liQNh3fGSlLjDHpJ\napxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TG\nGfSS1DiDXpIaZ9BLUuP6Dvoky5LcneSBJPcnubRr/0SS7Uk2dp8zh1euJGmhBnmV4B7gQ1V1X/eC\n8A1J7uyWfaGqPjt4eZKkQfUd9FW1A9jRTT+d5EHg+GEVJkkajqGM0SeZBE4GvtU1XZJkU5J1SY7d\nzzark0wnmd69e/cwypAkzSJVNdgOkpcC3wCuqKqbkywBngAK+BSwtKouONA+pqamanp6eqA6DieT\na24bdwlSU7Zeeda4S+hLkg1VNTXXegOd0Sc5ClgPXF9VNwNU1c6qeq6qnge+DJwySB+SpMEMctVN\ngKuBB6vq8z3tS3tWOwfY3H95kqRBDXLVzRuB9wLfS7Kxa7sMWJVkJTNDN1uBDwxUoSRpIINcdfNN\nILMsur3/ciRJw+adsZLUOINekho3yBj9Yc/LHCUtBp7RS1LjPKOXdNgb57/OD8bNWp7RS1LjDHpJ\napxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWpcE49A8OFikrR/IzujT3J6koeT\nbEmyZlT9SJIObCRBn+QI4K+AM4CTmHm94Emj6EuSdGCjOqM/BdhSVY9W1X8DXwXOHlFfkqQDGFXQ\nHw883jO/rWuTJB1kY/sxNslqYHU3+0ySh0fc5XHAEyPu42DwOA4dLRwDtHEci/YY8ukXzC70OH51\nPiuNKui3A8t65k/o2v5PVa0F1o6o//8nyXRVTR2s/kbF4zh0tHAM0MZxtHAMMLrjGNXQzXeAFUlO\nTPJi4Dzg1hH1JUk6gJGc0VfVniSXAP8IHAGsq6r7R9GXJOnARjZGX1W3A7ePav99OGjDRCPmcRw6\nWjgGaOM4WjgGGNFxpKpGsV9J0iHCZ91IUuMOq6BP8qkkm5JsTHJHkl8Zd039SPKZJA91x3JLkmPG\nXdNCJfm9JPcneT7JortaooVHfCRZl2RXks3jrqVfSZYluTvJA91/T5eOu6aFSvILSb6d5N+7Y/jT\nofdxOA3dJHl5Vf2km/4j4KSqumjMZS1YkrcD/9z96P1pgKr6yJjLWpAkvw48D/wN8CdVNT3mkuat\ne8TH94G3MXMz4HeAVVX1wFgLW6AkbwGeAa6rqteOu55+JFkKLK2q+5K8DNgAvHsx/VkkCXB0VT2T\n5Cjgm8ClVXXvsPo4rM7o94Z852hgUf4tV1V3VNWebvZeZu5TWFSq6sGqGvVNcqPSxCM+quoe4Klx\n1zGIqtpRVfd1008DD7LI7sKvGc90s0d1n6Fm02EV9ABJrkjyOPAe4OPjrmcILgC+Pu4iDjM+4uMQ\nlGQSOBn41ngrWbgkRyTZCOwC7qyqoR5Dc0Gf5J+SbJ7lczZAVX2sqpYB1wOXjLfa/ZvrOLp1Pgbs\nYeZYDjnzOQZpGJK8FFgPfHCff7kvClX1XFWtZOZf56ckGepQWhMvHulVVW+d56rXM3Od/+UjLKdv\ncx1HkvcB7wROq0P0h5YF/FksNnM+4kMHTzeuvR64vqpuHnc9g6iqHyW5GzgdGNqP5M2d0R9IkhU9\ns2cDD42rlkEkOR34MPCuqnp23PUchnzExyGi+yHzauDBqvr8uOvpR5KJvVfOJflFZn7kH2o2HW5X\n3awHXs3M1R6PARdV1aI7E0uyBXgJ8GTXdO9iu3ooyTnAXwATwI+AjVX1jvFWNX9JzgT+nJ8/4uOK\nMZe0YEluAE5l5omJO4HLq+rqsRa1QEneBPwr8D1m/r8GuKy7M39RSPI64Fpm/lt6EXBTVX1yqH0c\nTkEvSYejw2roRpIORwa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN+1+GUooYvT/HoQAA\nAABJRU5ErkJggg==\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"HCNvqotz6zux","colab_type":"text"},"source":["There are a plethora of inplace initialisation functions to be found in the `torch..nn.init` module. \n","\n","### modules() vs children()\n","\n","A very similar function to `modules` is `children`. The difference is a slight but an important one. As we know, a `nn.Module` object can contain other `nn.Module` objects as it's data members. `children()` will only return an iterable of the `nn.Module` objects which are data members.\n","\n","On other hand, `nn.Modules` goes recursively inside each `nn.Module` object, printing each `nn.Module` object that comes along the way until there are no `nn.module` objects left. \n","\n","Note, that the above statement remains true for all objects / classes that subclass from `nn.Module` class."]},{"cell_type":"code","metadata":{"id":"RhgGc-O48f6H","colab_type":"code","outputId":"4b2cd18a-30c6-450f-cb7b-efb129648afc","executionInfo":{"status":"ok","timestamp":1556995029805,"user_tz":-330,"elapsed":995,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":357}},"source":["class myNet(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.convBN =  nn.Sequential(nn.Conv2d(10,10,3), nn.BatchNorm2d(10))\n","    self.linear =  nn.Linear(10,2)\n","    \n","  def forward(self, x):\n","    pass\n","  \n","\n","Net = myNet()\n","\n","print(\"Printing children\\n------------------------------\")\n","print(list(Net.children()))\n","print(\"\\n\\nPrinting Modules\\n------------------------------\")\n","print(list(Net.modules()))\n","\n","\n"],"execution_count":9,"outputs":[{"output_type":"stream","text":["Printing children\n","------------------------------\n","[Sequential(\n","  (0): Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1))\n","  (1): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","), Linear(in_features=10, out_features=2, bias=True)]\n","\n","\n","Printing Modules\n","------------------------------\n","[myNet(\n","  (convBN): Sequential(\n","    (0): Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1))\n","    (1): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","  )\n","  (linear): Linear(in_features=10, out_features=2, bias=True)\n","), Sequential(\n","  (0): Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1))\n","  (1): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","), Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1)), BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True), Linear(in_features=10, out_features=2, bias=True)]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"nKzXbbxCBq8_","colab_type":"text"},"source":["So, when we initiase the weights, we must use `modules()` function since we can't go inside the `nn.Sequential` object and initialise the weight for it's members.\n","\n","## Printing Information About the Network\n","\n","We may need to print information about the network, whether be it for the user or for debugging purposes. PyTorch provides a really neat way to print a lot of information about out network using it's `named_*` functions. There are 4 such functions. \n","\n","1. `named_parameters`. Returns an iterator which gives a tuple containing **name** of the parameters (if a convolutional layer is assigned as `self.conv1`, then it's parameters would be `conv1.weight` and `conv1.bias`) and the value returned by the `__repr__` function of the `nn.Parameter`\n","\n","2. `named_modules`. Same as above, but iterator returns modules like `modules()` function does. \n","3. `named_children` Same as above, but iterator return modules like `children()` returns\n","4. `named_buffers` Return buffer tensors such as running mean average of a Batch Norm layer. \n","\n"]},{"cell_type":"code","metadata":{"id":"S6-8s-dDGCJ2","colab_type":"code","outputId":"80c036d3-6504-48ea-b7e9-1ec8dc890587","executionInfo":{"status":"ok","timestamp":1556995032790,"user_tz":-330,"elapsed":1007,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":340}},"source":["for x in Net.named_modules():\n","  print(x[0], x[1], \"\\n-------------------------------\")"],"execution_count":10,"outputs":[{"output_type":"stream","text":[" myNet(\n","  (convBN): Sequential(\n","    (0): Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1))\n","    (1): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","  )\n","  (linear): Linear(in_features=10, out_features=2, bias=True)\n",") \n","-------------------------------\n","convBN Sequential(\n","  (0): Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1))\n","  (1): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",") \n","-------------------------------\n","convBN.0 Conv2d(10, 10, kernel_size=(3, 3), stride=(1, 1)) \n","-------------------------------\n","convBN.1 BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) \n","-------------------------------\n","linear Linear(in_features=10, out_features=2, bias=True) \n","-------------------------------\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ekZKznt6U4MA","colab_type":"text"},"source":["## Different Learning Rates for Different Layers\n","\n","In the last section, we will learn how to use different learning rates for different learning rates for our different layers. In general, we will cover how to have different hyperparameters for different groups of parameters, whether it be different learning rate for different layers, or different learning rate for biases and weights. \n","\n","The idea to implement such a thing is fairly simple. In our previous post, where we implemented a CIFAR classifier, we passed all the parameters of network as a whole to the optimiser object. \n"]},{"cell_type":"code","metadata":{"id":"GgJUMVfBo2ON","colab_type":"code","colab":{}},"source":["class myNet(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.fc1 = nn.Linear(10,5)\n","    self.fc2 = nn.Linear(5,2)\n","    \n","  def forward(self, x):\n","    return self.fc2(self.fc1(x))\n","\n","Net = myNet()\n","optimiser = torch.optim.SGD(Net.parameters(), lr = 0.5)\n","\n","\n","      "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"3vAf7-UzqMI2","colab_type":"text"},"source":["However, the `torch.optim` class allows us to provide different sets of parameters having different learning rates in form of a dictionary.\n","\n","\n"]},{"cell_type":"code","metadata":{"id":"rwN85XK6q7FB","colab_type":"code","colab":{}},"source":["optimiser = torch.optim.SGD([{\"params\": Net.fc1.parameters(), 'lr' : 0.001, \"momentum\" : 0.99},\n","                             {\"params\": Net.fc2.parameters()}], lr = 0.01, momentum = 0.9)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"NOL0N3C5sT6P","colab_type":"text"},"source":["In the above scenario, the parameters of `fc1` use a learning rate of 0.01 and momentum of 0.99. If a hyperparameter is not specified for a group of parameters (like `fc2`), they use the default value of that hyperparameter, given as input argument to the optimiser function. "]},{"cell_type":"code","metadata":{"id":"Tr22mwRWtWAa","colab_type":"code","colab":{}},"source":["params_bias = []\n","params_wts = []\n","\n","# seperate the bias and weights parameters\n","for name, parameter in Net.named_parameters():\n","  if \"bias\" in name:\n","    params_bias.append(parameter)\n","  elif \"weight\" in name:\n","    params_wts.append(parameter)\n","\n","# Set the optimiser to have different hyperparameters for bias and weights\n","optimiser = torch.optim.SGD([{\"params\": params_bias, 'lr' : 0.001, \"momentum\" : 0.99},\n","                             {\"params\": params_wts}], lr = 0.01, momentum = 0.9)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Db5lc4bSw7in","colab_type":"text"},"source":["## Scheduling Learning Rates\n","\n","The schedule your learning rate is going to follow is a major hyperparameter that you want to tune. PyTorch provides support for scheduling learning rates with it's `torch.optim.lr_scheduler` module which has a variety of learning rate schedules. The following example demonstrates one such example. "]},{"cell_type":"code","metadata":{"id":"FwzWQolRw62r","colab_type":"code","colab":{}},"source":["scheduler = torch.optim.lr_scheduler.MultiStepLR(optimiser, milestones = [10,20], gamma = 0.1)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"m-9R71hQxklV","colab_type":"text"},"source":["The above scheduler, multiplies the learning rate by `gamma` each time when we reach epochs contained in the `milestones`list. In our case, the learning rate is multiplied by 0.1 at the 10nth and the 20nth epoch. You will also have to write the line `scheduler.step` in the loop in your code that goes over the epochs. Generally, training loop is made of two nested loops, where one loop goes over the epochs, and the nested one goes over the batches in that epoch. Make sure you call `scheduler.step` at start of the epoch loop so your learning rate is updated. Be careful not to write it in the batch loop, otherwise your learning rate may be updated at the 10th batch rather than 10nth epoch. \n","\n","Also remember that `scheduler.step` is no replacement for `optim.step` and you'll have to call `optim.step` everytime you backprop backwards. (This would be in the \"batch\" loop)\n","\n","\n","## Saving your Model\n","\n","You might wanna save your model for later use for inference, or just might want to create training checkpoints. When it comes to saving models in PyTorch one has two options. \n","\n","First is to use `torch.save`. This is equivalent to serialising the entire `nn.Module` object using Pickle. This saves the entire model to disk. You can load this model later in the memory with `torch.load`."]},{"cell_type":"code","metadata":{"id":"LIEAePETyR-e","colab_type":"code","outputId":"705c46a6-d7b0-4118-9f9b-0a60db7a4d5a","executionInfo":{"status":"ok","timestamp":1556995046803,"user_tz":-330,"elapsed":970,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":119}},"source":["torch.save(Net, \"net.pth\")\n","\n","Net = torch.load(\"net.pth\")\n","\n","print(Net)"],"execution_count":15,"outputs":[{"output_type":"stream","text":["myNet(\n","  (fc1): Linear(in_features=10, out_features=5, bias=True)\n","  (fc2): Linear(in_features=5, out_features=2, bias=True)\n",")\n"],"name":"stdout"},{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/torch/serialization.py:256: UserWarning: Couldn't retrieve source code for container of type myNet. It won't be checked for correctness upon loading.\n","  \"type \" + obj.__name__ + \". It won't be checked \"\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"id":"c1V00Sce6kjo","colab_type":"text"},"source":["The above will save the entire model with weights and architecture. If you only need to save the weight, instead of saving the entire model, you can save just the `state_dict` of the model. The `state_dict` is basically a dictionary which maps the p`nn.Parameter` objects of a network to their values. "]},{"cell_type":"code","metadata":{"id":"b9f7a0lr60dy","colab_type":"code","outputId":"2ac8590f-3299-4dc5-f171-5aa45a4daa07","executionInfo":{"status":"ok","timestamp":1556404354362,"user_tz":-330,"elapsed":1242,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":255}},"source":["for key in Net.state_dict():\n","  print(key, Net.state_dict()[key])\n","  \n","torch.save(Net.state_dict(), \"net_state_dict.pth\")\n","\n","Net.load_state_dict(torch.load(\"net_state_dict.pth\"))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["fc1.weight tensor([[-0.0133, -0.2878,  0.0503,  0.0744,  0.1310,  0.0947, -0.3033,  0.1492,\n","          0.0490,  0.0941],\n","        [-0.0879,  0.2181,  0.0458, -0.2721, -0.0646, -0.2481,  0.1032,  0.2052,\n","         -0.0165, -0.0800],\n","        [-0.1940, -0.1277, -0.0104, -0.1482, -0.2348,  0.2273, -0.1213,  0.0237,\n","          0.1119,  0.3139],\n","        [ 0.1774,  0.1428,  0.1435,  0.0185,  0.1152, -0.2668, -0.2399, -0.2149,\n","         -0.3055,  0.2551],\n","        [ 0.3143, -0.2550, -0.2018, -0.0489,  0.1829,  0.0278,  0.0943, -0.0691,\n","         -0.0146,  0.2699]])\n","fc1.bias tensor([-0.1506,  0.2532,  0.2250, -0.1547, -0.2230])\n","fc2.weight tensor([[-0.2647, -0.2557, -0.2494,  0.0225, -0.3246],\n","        [-0.4338, -0.0281,  0.1650,  0.0791, -0.4264]])\n","fc2.bias tensor([-0.0396,  0.2155])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"BYOz20Ew7eFS","colab_type":"text"},"source":["As demonstrated above, one can load an exisiting `state_dict` into a `nn.Module` object. Note that this doesn't involve saving of entire model  but only the parameters. You will have to create the network with layers before you load the state dict. If the network architecture is not exactly the same as the one whose `state_dict` we saved, PyTorch will throw up an error. \n","\n","An optimiser object from `torch.optim` also has a `state_dict` object which is used to store the hyperparameters of optimisation algorithms. It can be saved and loaded in a similar way we did above."]},{"cell_type":"code","metadata":{"id":"4N1lhEbk6Xfx","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"JAXN3Aze6RrF","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}