PyTorch 101 Part 4 -Memory management and Multi-GPU Usage in PyTorch.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Memory management and Multi-GPU Usage in PyTorch.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"h6Rd9y6bMCmM","colab_type":"text"},"source":["# Memory management and Multi-GPU Usage in PyTorch"]},{"cell_type":"markdown","metadata":{"id":"CchmRRDdMCmO","colab_type":"text"},"source":["In this notebook, we will cover the following topics\n","\n","1. How to use multiple GPUs for your network, either using data parallelism or model parallelism. \n","2. How to automate selection of GPU while creating a new objects. \n","3. How to diagnose and analyse memory issues should they arise. "]},{"cell_type":"code","metadata":{"id":"elAr8oWHepcm","colab_type":"code","colab":{}},"source":["import torch\n","import torch.nn as nn"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hAhWtvK7MCmO","colab_type":"text"},"source":["\n","\n","\n","\n","## Moving tensors around CPU / GPUs\n","\n","\n","\n","\n","We use to `to` or `cuda` function to move the tensors around to CPU / GPUs. We pass the index of the GPU as the argument."]},{"cell_type":"code","metadata":{"id":"nvd-cs_hMCmP","colab_type":"code","colab":{}},"source":["if torch.cuda.is_available():\n","\tdev = \"cuda:0\"\n","else:\n","\tdev = \"cpu\"\n","\n","device = torch.device(dev)\n","\n","a = torch.zeros(4,3)   \n","a = a.to(0)       #alternatively, a.to(0)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qxbyt889ez6t","colab_type":"text"},"source":["In similar fashion as above, you can also move the `nn.Module` objects GPUs as well"]},{"cell_type":"code","metadata":{"id":"ix6Usmy0evlT","colab_type":"code","outputId":"800dbbb6-649e-40ed-9cf0-cf74048f36c3","executionInfo":{"status":"ok","timestamp":1556995101957,"user_tz":-330,"elapsed":1099,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["class myNetwork(nn.Module):\n","   def __init__(self):\n","      super().__init__()\n","      self.net = nn.Linear(5,1)\n","   \n","   def forward(self, x):\n","      return self.net(x)\n","\n","clf = myNetwork()\n","clf.to(0)"],"execution_count":4,"outputs":[{"output_type":"execute_result","data":{"text/plain":["myNetwork(\n","  (net): Linear(in_features=5, out_features=1, bias=True)\n",")"]},"metadata":{"tags":[]},"execution_count":4}]},{"cell_type":"markdown","metadata":{"id":"KujTXEX-fuKm","colab_type":"text"},"source":["We can get the device of a tensor by `get_device()`. Only supported for GPU Tensors"]},{"cell_type":"code","metadata":{"id":"UuCg1fLzewV8","colab_type":"code","colab":{}},"source":["dev = a.get_device() \n","b = torch.tensor(a.shape).to(dev)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"kbakI8zjgCSq","colab_type":"text"},"source":["We can also set the default device on which GPU tensors are created. "]},{"cell_type":"code","metadata":{"id":"1VFgoFisgaYJ","colab_type":"code","outputId":"0abf92ad-cdb2-460b-e309-da79183e3eda","executionInfo":{"status":"ok","timestamp":1556995106914,"user_tz":-330,"elapsed":784,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["torch.cuda.set_device(0)\n","\n","tens = torch.Tensor(3,4).cuda()\n","tens.get_device()"],"execution_count":6,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0"]},"metadata":{"tags":[]},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"w5RHuhUUzcCu","colab_type":"text"},"source":["## The new_* functions"]},{"cell_type":"markdown","metadata":{"id":"XStLVmfwzon1","colab_type":"text"},"source":["One can also make use of the bunch of new_ functions that made their way to PyTorch in version 1.0. When a function like new_ones is called on a Tensor it returns a new tensor of same data type, and on the same device as the tensor on which the new_ones function was invoked."]},{"cell_type":"code","metadata":{"id":"jlzuOSd4zurt","colab_type":"code","colab":{}},"source":["ones = torch.ones((2,)).cuda(0)\n","\n","# Create a tensor of ones of size (3,4) on same device as of \"ones\"\n","newOnes = ones.new_ones((3,4)) \n","\n","randTensor = torch.randn(2,4)\n","\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"mAgfpSZO0j6d","colab_type":"text"},"source":["A detailed list of new_ functions can be found in PyTorch docs the link of which I have provided below. "]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"bgO3nk8D1YMy"},"source":["## Using Multiple GPUs\n","\n","There are two ways how we could make use of multiple GPUs. \n","\n","1. **Data Parallelism**, where we divide batches into smaller batches, and process these smaller batches in parallel on multiple GPU.\n","2. **Model Parallelism**, where we break the neural network into smaller sub networks and then execute these sub networks on different GPUs.\n","\n","\n","### Data Parallelism\n","\n","Data Parallelism in PyTorch is achieved through the `nn.DataParallel` class. You initialize a `nn.DataParallel` object with a nn.Module object representing your network, and a list of GPU IDs, across which the batches have to be parallelised.\n","\n","There are a few things I want to shed light over. Despite the fact our data has to be parallelised over multiple GPUs, we have to **initially** store it on a single GPU.\n","\n","We also need to make sure the DataParallel object is on that particular GPU as well. The syntax remains similar to what we did earlier with nn.Module. \n","\n","DataParallel takes the input, splits it into smaller batches, replicates the neural network across all the devices, executes the pass and then collects the output back on the original GPU"]},{"cell_type":"code","metadata":{"id":"ihUCXAFT1aWu","colab_type":"code","colab":{}},"source":["myNet = myNetwork()\n","parallel_net = nn.DataParallel(myNet, device_ids = [0])\n","\n","inputs = torch.Tensor(5,)    #random inputs \n","\n","inputs = inputs.to(0)\n","myNet.to(0)\n","\n","predictions = parallel_net(inputs)\n","loss = (1 - predictions).mean()\n","loss.backward()\n","#optimiser.step()\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"LDQEMAnZVJmT","colab_type":"text"},"source":["One issue with DataParallel can be that it can put asymmetrical load on one GPU (the main node). There are generally two ways to circumvent these problem. \n","1. Compute the loss during the forward pass. This makes sure at least the loss calculation phase is parallelised. \n","2. Another way is to implement a parallel loss function layer. This is beyond the scope of this article. However, for those interested I have given a link to a medium article detailing implementation of such a layer at the end of this article. \n","\n","## Model Parallelism\n","\n","Model parallelism means that you break your network into smaller subnetworks that you then put on different GPUs. The main motivation for doing such a thing is that your network might be too large to fit inside a single GPU.\n","\n","Implementing Model parallelism is PyTorch is pretty easy as long as you remember 2 things.\n","1. The input and the network should always be on the same device. \n","1. to and cuda() functions have autograd support, so your gradients \n","can be copied from one GPU to another during backward pass. \n","\n","We will use the following piece of code to understand this better."]},{"cell_type":"code","metadata":{"id":"Czc7FvNyVo3J","colab_type":"code","outputId":"4106c42a-dc57-4580-cfd9-88dbe41cd657","executionInfo":{"status":"error","timestamp":1556995117006,"user_tz":-330,"elapsed":1230,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":375}},"source":["class model_parallel(nn.Module):\n","\tdef __init__(self):\n","\t\tsuper().__init__()\n","\t\tself.sub_network1 = nn.Linear(100,32)    #This part stays on GPU 1\n","\t\tself.sub_network2 = nn.Linear(32,10)     #This part stays on GPU 2\n","\n","\t\tself.sub_network1.cuda(0)\n","\t\tself.sub_network2.cuda(1)\n","\n","\tdef forward(x):\n","\t\tx = x.cuda(0)\n","\t\tx = self.sub_network1(x)\n","\t\tx = x.cuda(1)\n","\t\tx = self.sub_network2(x)\n","\t\treturn x\n","\n","\n","x = torch.Tensor(100,)        # Random Input\n","x = x.to(0)\n","\n","net = model_parallel()        # No need to put it on GPUs as that has been taken care of in the \n","                              # init function\n","\n","loss = (1 - net(x)).mean()\n","loss.backward()"],"execution_count":9,"outputs":[{"output_type":"error","ename":"RuntimeError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)","\u001b[0;32m<ipython-input-9-6da8482f906c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m \u001b[0mnet\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel_parallel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m        \u001b[0;31m# No need to put it on GPUs as that has been taken care of in the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     22\u001b[0m                               \u001b[0;31m# init function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m<ipython-input-9-6da8482f906c>\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msub_network1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msub_network2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36mcuda\u001b[0;34m(self, device)\u001b[0m\n\u001b[1;32m    263\u001b[0m             \u001b[0mModule\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    264\u001b[0m         \"\"\"\n\u001b[0;32m--> 265\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    266\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    267\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    197\u001b[0m                 \u001b[0;31m# Tensors stored in modules are graph leaves, and we don't\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    198\u001b[0m                 \u001b[0;31m# want to create copy nodes, so we have to unpack the data.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 199\u001b[0;31m                 \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    200\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_grad\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    201\u001b[0m                     \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_grad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_grad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(t)\u001b[0m\n\u001b[1;32m    263\u001b[0m             \u001b[0mModule\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    264\u001b[0m         \"\"\"\n\u001b[0;32m--> 265\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    266\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    267\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mRuntimeError\u001b[0m: CUDA error: invalid device ordinal"]}]},{"cell_type":"markdown","metadata":{"id":"v1PJbmuWWi5s","colab_type":"text"},"source":["## Troubleshooting Out of Memory Errors\n","\n","### Diagnosing GPU Usage\n","\n","One can diagnose the GPU usage using the GPUtil python library which can be installed using pip by typing `pip install gputil` in a terminal. The following piece of code illustrates use of the extension. \n"]},{"cell_type":"code","metadata":{"id":"1rtlegawcZII","colab_type":"code","outputId":"894ce2fa-3df1-4c20-c87a-9ee6bb3ad42c","executionInfo":{"status":"ok","timestamp":1556995133240,"user_tz":-330,"elapsed":11818,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":153}},"source":["!pip install gputil\n"],"execution_count":10,"outputs":[{"output_type":"stream","text":["Collecting gputil\n","  Downloading https://files.pythonhosted.org/packages/ed/0e/5c61eedde9f6c87713e89d794f01e378cfd9565847d4576fa627d758c554/GPUtil-1.4.0.tar.gz\n","Building wheels for collected packages: gputil\n","  Building wheel for gputil (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Stored in directory: /root/.cache/pip/wheels/3d/77/07/80562de4bb0786e5ea186911a2c831fdd0018bda69beab71fd\n","Successfully built gputil\n","Installing collected packages: gputil\n","Successfully installed gputil-1.4.0\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"hb7zqO6Cipt6","colab_type":"code","outputId":"b4baf93e-8a6f-4c51-89f6-ef940e6d50e0","executionInfo":{"status":"ok","timestamp":1556995133242,"user_tz":-330,"elapsed":9569,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["from GPUtil import showUtilization as gpu_usage\n","gpu_usage()"],"execution_count":11,"outputs":[{"output_type":"stream","text":["| ID | GPU | MEM |\n","------------------\n","|  0 |  0% |  5% |\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"YxfAwMTMcn3k","colab_type":"code","outputId":"c88b7227-d5a2-4b85-bfb2-9c4718dfef60","executionInfo":{"status":"error","timestamp":1556044553242,"user_tz":-330,"elapsed":1433,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":232}},"source":["\n"],"execution_count":0,"outputs":[{"output_type":"error","ename":"AttributeError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)","\u001b[0;32m<ipython-input-2-9f8b3c3683c1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m   \u001b[0mten\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1000000\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m   \u001b[0mtensorList\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mten\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgpu_usage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mAttributeError\u001b[0m: 'Tensor' object has no attribute 'append'"]}]},{"cell_type":"markdown","metadata":{"id":"wxsJW3UlfVK2","colab_type":"text"},"source":["### Dealing with Memory Losses using del keyword\n","\n","While PyTorch has a pretty aggresive garbage collector. But PyTorch will free up the variable only when there exist no pythonic reference to the object. \n","\n","It is to be kept in mind that Python doesn't enforce scoping rules as strongly as other languages such as C/C++. A variable is only freed when there exists no pointers to it. Consider the following case. "]},{"cell_type":"code","metadata":{"id":"JR2MVmFGgSfx","colab_type":"code","outputId":"769f69ef-1d13-4012-eb51-4bd3616ee137","executionInfo":{"status":"ok","timestamp":1556995139024,"user_tz":-330,"elapsed":1184,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["for x in range(10):\n","  tensor = torch.randn(1,4)\n","\n","print(tensor)   \n","\n"],"execution_count":12,"outputs":[{"output_type":"stream","text":["tensor([[-1.2010,  0.2314, -0.2265, -0.0234]])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"iLf4gDaMfQap","colab_type":"text"},"source":["We defined `tensor` inside the loop, however, it still existed after we exited the loop. Similarly, tensors holding the network's inputs, losses, output continue to take up memory even after we exit the training loop. \n","\n","A good practice to get rid of these variables is by using the `del` keyword."]},{"cell_type":"code","metadata":{"id":"rhQwwkIGgs8G","colab_type":"code","outputId":"86928406-24ec-481b-c634-935fd1d5c20d","executionInfo":{"status":"ok","timestamp":1556995143124,"user_tz":-330,"elapsed":1240,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["net = myNetwork()\n","opt = torch.optim.SGD(net.parameters(),lr = 0.01)\n","\n","inp = torch.randn(5,)\n","\n","for x in range(10):\n","  out = net(inp)\n","  loss = (1 - out).mean()\n","  opt.zero_grad()\n","  loss.backward()\n","  opt.step()\n","\n","print(out, loss)                      # these variables still exist\n","del out, loss                         # Free the memory taken by these variables\n"],"execution_count":13,"outputs":[{"output_type":"stream","text":["tensor([0.1282], grad_fn=<AddBackward0>) tensor(0.8718, grad_fn=<MeanBackward0>)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"r0vLRcKskPnq","colab_type":"text"},"source":["### Using Python Data Types Instead Of 1-D Tensors\n","\n","Often, we aggregate values in our training loop to compute some metrics. Biggest example of this is that we update the running loss  each iteration. However, if not done carefully in PyTorch, such a thing can lead to excess use of memory than what is required. \n","\n","Consider the following snippet of code. "]},{"cell_type":"code","metadata":{"id":"ZRTGFvtLlWni","colab_type":"code","colab":{}},"source":["total_loss = 0\n","\n","for x in range(10):\n","  # assume loss is computed \n","  iter_loss = torch.randn(3,4).mean()\n","  iter_loss.requires_grad = True     # losses are supposed to differentiable\n","  total_loss += iter_loss            # use total_loss += iter_loss.item) instead\n","  "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"BzYOaPlrltvA","colab_type":"text"},"source":[" We expect that in the subsequent iterations, the reference to `iter_loss` is reasigned to new `iter_loss`, and the object representing `iter_loss` from earlier representation will be freed. But this doesn't happen. Why?\n"," \n"," Since `iter_loss` is differentiable, the line `total_loss += iter_loss` creates a computation graph with one `AddBackward` function node. During subsequent iterations, `AddBackward` nodes are added to this graph and no object holding values of `iter_loss` is freed. \n"," \n"," The solution to this is to add a python data type, and not a tensor to `total_loss` which prevents creation of any computation graph.\n"," \n"," We merely replace the line `total_loss += iter_loss` with `total_loss += iter_loss.item()`. `item` returns the python data type from a tensor containing single values."]},{"cell_type":"markdown","metadata":{"id":"zzTcu22EoZ_1","colab_type":"text"},"source":["### Using torch.no_grad() for inference\n","\n","Whenever you are doing inference with your network, or any operation that doesn't require backpropagation of gradients, you should always put the code inside `torch.no_grad()` context manager. "]},{"cell_type":"code","metadata":{"id":"dt16P7KMpd86","colab_type":"code","colab":{}},"source":["net = myNetwork()\n","inp = torch.randn(5,)\n","\n","with torch.no_grad():\n","  out = net(inp)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"q8gR5tEC6dyt","colab_type":"text"},"source":["### Emptying CUDA cache\n","\n","While PyTorch aggressively frees up memory, a pytorch process may not give back the memory back to the OS even after you del your tensors. This memory is cached so that it can be quickly allocated to new tensors being allocated without requesting the OS new extra memory.\n","\n","This can be a problem when you are using more than two processes in your workflow.\n","\n","The first process can hold onto the GPU memory even if it's work is done causing OOM when the second process is launched. To remedy this, you can write the command at the end of your code. \n","\n"]},{"cell_type":"code","metadata":{"id":"dEymoRUv6lc6","colab_type":"code","outputId":"e764cf4b-2eac-4338-dfad-275f9758d433","executionInfo":{"status":"ok","timestamp":1556995179589,"user_tz":-330,"elapsed":12217,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":289}},"source":["\n","import torch\n","from GPUtil import showUtilization as gpu_usage\n","\n","print(\"Initial GPU Usage\")\n","gpu_usage()                             \n","\n","tensorList = []\n","for x in range(10):\n","  tensorList.append(torch.randn(10000000,10).cuda())   # reduce the size of tensor if you are getting OOM\n","  \n","  \n","\n","print(\"GPU Usage after allcoating a bunch of Tensors\")\n","gpu_usage()\n","\n","del tensorList\n","\n","print(\"GPU Usage after deleting the Tensors\")\n","gpu_usage()  \n","\n","print(\"GPU Usage after emptying the cache\")\n","torch.cuda.empty_cache()\n","gpu_usage()\n","\n"],"execution_count":16,"outputs":[{"output_type":"stream","text":["Initial GPU Usage\n","| ID | GPU | MEM |\n","------------------\n","|  0 |  0% |  5% |\n","GPU Usage after allcoating a bunch of Tensors\n","| ID | GPU | MEM |\n","------------------\n","|  0 |  3% | 30% |\n","GPU Usage after deleting the Tensors\n","| ID | GPU | MEM |\n","------------------\n","|  0 |  3% | 30% |\n","GPU Usage after emptying the cache\n","| ID | GPU | MEM |\n","------------------\n","|  0 |  3% |  5% |\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"wcfXIdvOppSh","colab_type":"text"},"source":["### Using CUDNN Backend\n","\n","One can use the `CUDNN` benchmark to have optimisations in the code. These are specially benificial if your input sized is fixed (You are not using RNNs).\n","\n"]},{"cell_type":"code","metadata":{"id":"g5VLMzW4plC9","colab_type":"code","colab":{}},"source":["torch.backends.cudnn.benchmark = True\n","torch.backends.cudnn.enabled = True"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"G2z8MjK_qco8","colab_type":"text"},"source":["### Using Half Precision Floats\n","\n","One can use half precision floats if the GPU has FP16 support. It's simple enough to convert a normal model to it's half precision variant. "]},{"cell_type":"code","metadata":{"id":"NUYXEfKkqtfF","colab_type":"code","outputId":"c149e141-9d80-4282-d2d7-3ef9c02745b3","executionInfo":{"status":"ok","timestamp":1556995188797,"user_tz":-330,"elapsed":1112,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["inp = torch.randn(5,).cuda().half()\n","\n","model = myNetwork().cuda().half()\n","\n","\n","model(inp)"],"execution_count":18,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([0.1536], device='cuda:0', dtype=torch.float16, grad_fn=<AddBackward0>)"]},"metadata":{"tags":[]},"execution_count":18}]},{"cell_type":"markdown","metadata":{"id":"TY8Q7hwju4wt","colab_type":"text"},"source":["Batch Norm layers have been reported to have convergence issues with half precision floats so it's better to use full precision for them."]},{"cell_type":"code","metadata":{"id":"2Wc6UCtDrPlL","colab_type":"code","outputId":"9c62905e-a7a8-41f3-a6a2-1124c8190afe","executionInfo":{"status":"ok","timestamp":1556995191984,"user_tz":-330,"elapsed":1119,"user":{"displayName":"Ayoosh Kathuria","photoUrl":"https://lh5.googleusercontent.com/-hC2hkjwNr9s/AAAAAAAAAAI/AAAAAAAACpo/DPqp1uUqR4E/s64/photo.jpg","userId":"11533138969683019189"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["import torch\n","import torch.nn as nn\n","class myNetworkBN(nn.Module):\n","  def __init__(self):\n","    super().__init__()\n","    self.l1 = nn.Linear(10,5)\n","    self.bn = nn.BatchNorm1d(5)\n","    self.l2 = nn.Linear(5,1)\n","     \n","  def forward(self,x):\n","    x = self.l1(x)\n","    x = self.bn(x)\n","    x = self.l2(x)\n","    return x \n","\n","inp = torch.randn(10,).cuda().half().unsqueeze(0)       # Unsquueze op to add mini-batch dimension\n","\n","model = myNetworkBN().cuda().half().eval()              # Eval mode = use population statistics in BN\n","\n","model(inp)"],"execution_count":19,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[-0.1593]], device='cuda:0', dtype=torch.float16,\n","       grad_fn=<AddmmBackward>)"]},"metadata":{"tags":[]},"execution_count":19}]},{"cell_type":"markdown","metadata":{"id":"RA2gow04yGh_","colab_type":"text"},"source":["One must always be careful about half precision floats when the value may get too large. It is recommended to use the Nvidia `apex` extension for using mixed precision training. "]},{"cell_type":"code","metadata":{"id":"iGcN0FJRv4pV","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}