Notebook_1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Notebook_1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyP2RlgUQViY14UebZilfome",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard",
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/leonardoLavagna/PyTorch-Notebooks/blob/main/Notebook_1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# PyTorch Notebook 1\n",
        "## Fundamentals\n",
        "In this first notebook we will start familiarizing with PyTorch and its fundamentals."
      ],
      "metadata": {
        "id": "kI4asVW14LZQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "To get PyThorch on your local machine go to https://pytorch.org. There you can also find usfeful materials, for example many tutorials: https://pytorch.org/tutorials. In the following Notebooks (this is the first of a series, see https://github.com/leonardoLavagna/PyTorch-Notebooks) we will use Google Colab (cfr. https://colab.research.google.com) that has all the libraries we will need immediately available and allow to use (even with the free plan) GPUs."
      ],
      "metadata": {
        "id": "rEuFkKPN5u5I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# GPU Info (in Google Colab)\n",
        "# !nvidia-smi"
      ],
      "metadata": {
        "id": "qkQQKKYx4QJd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "print(torch.__version__)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oevjohsa4fhc",
        "outputId": "6dbf20b8-0b8a-4afa-b8df-0fecb45c5cbc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.12.1+cu113\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tensors\n",
        "Tensors are the main building blocks of deep learning. In particular tensors are used to represent (numerical) data in PyTorch. There are many types of tensors with different dimensions (e.g. scalars, vectors, matrixes,...) and they are usually created using `torch.tensor()` (see https://pytorch.org/docs/stable/tensors.html)."
      ],
      "metadata": {
        "id": "hQxmUsj35mSW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create an empty tensor (scalar)\n",
        "# Remark . An empty tensor is usually not initialized.\n",
        "x = torch.empty(1)\n",
        "print(x)\n",
        "# For a scalar tensor x we can get the tensor value with x.item()\n",
        "print(x.item())\n",
        "# default type\n",
        "print(x.dtype)\n",
        "# dimension and shape\n",
        "print(x.ndim)\n",
        "print(x.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X8iWdyN65tET",
        "outputId": "73628048-7735-419f-dbff-ea0e6e781f8a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([9.8038e-35])\n",
            "9.80376982265173e-35\n",
            "torch.float32\n",
            "1\n",
            "torch.Size([1])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a specific tensor (scalar) with a different data type\n",
        "x = torch.tensor(1,dtype=torch.int32)\n",
        "print(x)\n",
        "print(x.item())\n",
        "print(x.dtype)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uks1A6Xs6cOm",
        "outputId": "5cee53cf-20c0-4747-9c83-04e721688809"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(1, dtype=torch.int32)\n",
            "1\n",
            "torch.int32\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Remark .** The default datatype of a tensor is `float32`. This can be changed in many other data types: `int32, float16, complex64`... see https://pytorch.org/docs/stable/tensors.html for all the data types available. Tensor's data types are of crucial importance since certain operations between tensors or certain functions applyied to tensors require specific data types and are one of the most common surce of code errors."
      ],
      "metadata": {
        "id": "YqdRChxlEYbH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a vector with all zeros\n",
        "x = torch.zeros(2)\n",
        "print(x)\n",
        "print(x.ndim)\n",
        "print(x.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NwywuXc_7VLE",
        "outputId": "b7b13427-96db-42ae-8de8-f90fdbbdaff2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0., 0.])\n",
            "1\n",
            "torch.Size([2])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a 2x2 matrix with all ones\n",
        "x = torch.ones(2,2)\n",
        "print(x)\n",
        "print(x.ndim)\n",
        "print(x.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SG8sifWZ9Zfi",
        "outputId": "1f33e579-ec5e-4baa-d79f-6abf9b71b51b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1., 1.],\n",
            "        [1., 1.]])\n",
            "2\n",
            "torch.Size([2, 2])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Access elements\n",
        "print(x[1])\n",
        "print(x[1][1])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8jWlwBNB9n1x",
        "outputId": "b072dfd7-2880-4baf-bf25-bf4a91bee7fe"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([1., 1.])\n",
            "tensor(1.)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor (1 tensor 2x3)\n",
        "x = torch.tensor([[[1,2,3],[4,5,6]]])\n",
        "print(x)\n",
        "print(x.ndim)\n",
        "print(x.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZKvYPUVM91bs",
        "outputId": "47eb3302-a098-43ec-e97d-aff5601547d2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[[1, 2, 3],\n",
            "         [4, 5, 6]]])\n",
            "3\n",
            "torch.Size([1, 2, 3])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a random tensor\n",
        "# Random seed (for this block)\n",
        "# Random seeds are necessary for reproducibility purposes.\n",
        "random_seed = 1234\n",
        "torch.manual_seed(random_seed)\n",
        "# Random tensor\n",
        "x = torch.rand(1,2,3,3)\n",
        "print(x)\n",
        "print(x.ndim)\n",
        "print(x.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x8HvP0pa-KWZ",
        "outputId": "016e7f96-1971-42b5-adbb-a1fcbd39d967"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[[[0.0290, 0.4019, 0.2598],\n",
            "          [0.3666, 0.0583, 0.7006],\n",
            "          [0.0518, 0.4681, 0.6738]],\n",
            "\n",
            "         [[0.3315, 0.7837, 0.5631],\n",
            "          [0.7749, 0.8208, 0.2793],\n",
            "          [0.6817, 0.2837, 0.6567]]]])\n",
            "4\n",
            "torch.Size([1, 2, 3, 3])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Access elements\n",
        "print(x[0])\n",
        "print(x[0][0])\n",
        "print(x[0][0][0])\n",
        "print(x[0][0][0][0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pjQjFUsv-2rL",
        "outputId": "29b1b28a-5718-4a72-85d2-0b1620e0f64c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[[0.0290, 0.4019, 0.2598],\n",
            "         [0.3666, 0.0583, 0.7006],\n",
            "         [0.0518, 0.4681, 0.6738]],\n",
            "\n",
            "        [[0.3315, 0.7837, 0.5631],\n",
            "         [0.7749, 0.8208, 0.2793],\n",
            "         [0.6817, 0.2837, 0.6567]]])\n",
            "tensor([[0.0290, 0.4019, 0.2598],\n",
            "        [0.3666, 0.0583, 0.7006],\n",
            "        [0.0518, 0.4681, 0.6738]])\n",
            "tensor([0.0290, 0.4019, 0.2598])\n",
            "tensor(0.0290)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Remark .** In many neural networks random tensors are very important because they often learn by strating with a random tensor and updating it adjusting the values to better represent or fit the data."
      ],
      "metadata": {
        "id": "BpAb_7v1_dsS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor-range (default start=0, end=length-1, step=1)\n",
        "range = torch.arange(start=0,end=10,step=2)\n",
        "print(range)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6_4Q2Z0iDL51",
        "outputId": "671a1c59-dc80-43a7-d112-400ffa4262d0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0, 2, 4, 6, 8])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating tensors like the previous one but with all zeros\n",
        "tensor_like = torch.zeros_like(input=range)\n",
        "print(tensor_like)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pqgKjx41Dwzb",
        "outputId": "c677d13c-5c6e-43d9-8bad-182047196a60"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0, 0, 0, 0, 0])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Slicing and reshaping\n",
        "Slicing is very similar to the same operation with python arrays, and reshaping can be done with the function `torch.tensor.view()`, see https://pytorch.org/docs/stable/generated/torch.Tensor.view.html."
      ],
      "metadata": {
        "id": "ekP6mwS7LGzX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Slicing\n",
        "x = torch.rand(4,4)\n",
        "print(x)\n",
        "print(x[:,0])\n",
        "print(x[0,0])\n",
        "print(x[1,1].item())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yzQjTtLHLJi4",
        "outputId": "a30cc70c-474c-4511-cf7d-9df47ee0a5ea"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0.2388, 0.7313, 0.6012, 0.3043],\n",
            "        [0.2548, 0.6294, 0.9665, 0.7399],\n",
            "        [0.4517, 0.4757, 0.7842, 0.1525],\n",
            "        [0.6662, 0.3343, 0.7893, 0.3216]])\n",
            "tensor([0.2388, 0.2548, 0.4517, 0.6662])\n",
            "tensor(0.2388)\n",
            "0.6293618679046631\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Reshaping\n",
        "print(x.size())\n",
        "\n",
        "y = x.view(16)\n",
        "print(y)\n",
        "print(y.size())\n",
        "\n",
        "y = x.view(-1,8)\n",
        "print(y)\n",
        "print(y.size())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k4Zm57CRLan1",
        "outputId": "af5ed024-63a2-4991-ce82-f552c3a580aa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([4, 4])\n",
            "tensor([0.2388, 0.7313, 0.6012, 0.3043, 0.2548, 0.6294, 0.9665, 0.7399, 0.4517,\n",
            "        0.4757, 0.7842, 0.1525, 0.6662, 0.3343, 0.7893, 0.3216])\n",
            "torch.Size([16])\n",
            "tensor([[0.2388, 0.7313, 0.6012, 0.3043, 0.2548, 0.6294, 0.9665, 0.7399],\n",
            "        [0.4517, 0.4757, 0.7842, 0.1525, 0.6662, 0.3343, 0.7893, 0.3216]])\n",
            "torch.Size([2, 8])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Aritmetic operations with tensors\n",
        "Aritmetic operations between tensors are very similar to aritmetic operations with arrays (e.g. in `numpy`, see: https://numpy.org)"
      ],
      "metadata": {
        "id": "CFuXC4IoAiiA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Elementwise addition of tensors with the same size\n",
        "a = torch.rand(2,2)\n",
        "b = torch.rand(2,2)\n",
        "c = a+b\n",
        "print(a)\n",
        "print(b)\n",
        "print(c)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ytfA5fuLAkwV",
        "outputId": "809f4e8d-1efc-4803-e46e-7f52d45fb859"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0.5247, 0.6688],\n",
            "        [0.8436, 0.4265]])\n",
            "tensor([[0.9561, 0.0770],\n",
            "        [0.4108, 0.0014]])\n",
            "tensor([[1.4809, 0.7458],\n",
            "        [1.2544, 0.4279]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# same as before\n",
        "print(torch.add(a,b))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YvGeTaibAwCX",
        "outputId": "a7d409a2-89e8-45dc-ef42-2dcd74a1bd8b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1.4809, 0.7458],\n",
            "        [1.2544, 0.4279]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# inplace addition, same as before\n",
        "# all pytorch function with _ at the end are usually inplace operations\n",
        "print(b.add_(a))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JTSOWWcgBeaH",
        "outputId": "fbceb885-5c4a-47dc-80f3-a70efd644538"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1.4809, 0.7458],\n",
            "        [1.2544, 0.4279]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "All the other aritmetic operations are carried out in a similar fashion. In particular we have:\n",
        "\n",
        "\n",
        "*   **elementwise subtraction** `a-b` or `torch.sub(a,b)` or `a.sub_(b)`;\n",
        "*   **elementwise multiplication** `a*b` or `torch.mul(a,b)` or `a.mul_(b)`;\n",
        "*   **elementwise division** ` a / b ` or `torch.div(a,b)` or `a.div_(b)`;\n",
        "*   **elementwise power** ` a**b ` or `torch.pow(a,b)` or `a.pow_(b)`;\n",
        "\n",
        "See https://pytorch.org/docs/stable/torch.html for all the other useful operations."
      ],
      "metadata": {
        "id": "oOxssJYlBko2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Remark .** The device where a tensor is saved is of crucial importance as the following code shows."
      ],
      "metadata": {
        "id": "QTXQHdu_G_Ov"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor \n",
        "# In order to see the importance of memory for tensor operations\n",
        "# a GPU must be available and the following block should be run on a GPU.\n",
        "# Google Colab provides GPU capabilities for free\n",
        "# To enable a GPU go to Runtime/Change runtime type\n",
        "x = torch.ones(5)\n",
        "print(x)\n",
        "# Move it in the GPU\n",
        "if torch.cuda.is_available():\n",
        "  device = torch.device(\"cuda\")\n",
        "  x = x.to(device)\n",
        "  print(x)\n",
        "\n",
        "# Move the tensor in the CPU\n",
        "x = x.to(\"cpu\")\n",
        "# Change the tensor in the CPU\n",
        "# Note that x+=1 is the same as x=x+1\n",
        "x += 1\n",
        "print(x)\n",
        "\n",
        "# Also the tensor in the GPU has been changed!\n",
        "if torch.cuda.is_available():\n",
        "  x = x.to(device)\n",
        "  print(x)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ponHvQxtHn3M",
        "outputId": "dbbfbb3f-b29f-4e4e-cac1-6f2bea480ad6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([1., 1., 1., 1., 1.])\n",
            "tensor([1., 1., 1., 1., 1.], device='cuda:0')\n",
            "tensor([2., 2., 2., 2., 2.])\n",
            "tensor([2., 2., 2., 2., 2.], device='cuda:0')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## From Numpy to PyThorch and back\n",
        "Many operations between tensors (e.g. matrix multiplications) are particularly easy to carry out with `numpy`. To convert a tensor to a numpy array, and vicerversa, we can use the following code. "
      ],
      "metadata": {
        "id": "w2hb3HekKIRS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np"
      ],
      "metadata": {
        "id": "WFY1WjBbKicT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "a = torch.ones(5)\n",
        "print(a)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gBQKgMR5KlkK",
        "outputId": "4c637275-722b-4f94-e487-ebec71fa1aa8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([1., 1., 1., 1., 1.])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# From PyThorch to Numpy\n",
        "b = a.numpy()\n",
        "print(b)\n",
        "print(type(b))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NcxD0LSUKqGU",
        "outputId": "80b5273a-93d9-4c51-b947-dd97a4b075b3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1. 1. 1. 1. 1.]\n",
            "<class 'numpy.ndarray'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# From Numpy to PyThorch\n",
        "a = np.ones(5)\n",
        "b = torch.from_numpy(a)\n",
        "print(b)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R_bzruukKtCu",
        "outputId": "74739767-258b-4994-a674-13fd30939d32"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([1., 1., 1., 1., 1.], dtype=torch.float64)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Remark .** At the time of writing `numpy` works only with CPUs. "
      ],
      "metadata": {
        "id": "wDeBLLGUK0Xc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercises\n",
        "\n",
        "\n",
        "1.   Create a random tensor `a` with shape (4,4) and perform a matrix multiplication with another random tensor `b` with shape (1, 4) (hint: you may have to transpose the second tensor).\n",
        "2.   Find the maximum and minimum values of the output of the matrix multiplication of `a` and `b` created in the previous exercise.\n",
        "3.   Speaking of random seeds, we saw how to set it with torch.manual_seed() but is there a GPU equivalent? (hint: look into the documentation for `torch.cuda` for this one). \n",
        "\n"
      ],
      "metadata": {
        "id": "MB7V_n0XO4iq"
      }
    }
  ]
}