day_1/sem_1.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mZOzGHuPwr4v"
      },
      "source": [
        "# `Занятие 1.1: Основы Pytorch`\n",
        "\n",
        "#### `Сириус, смена \"Алгоритмы и анализ данных\" 2024`\n",
        "\n",
        "#### `Алексеев Илья, ММП ВМК МГУ`\n",
        "\n",
        "#### `За основу взяты материалы Находнова Максима`\n",
        "\n",
        "- https://github.com/mmp-practicum-team/mmp_practicum_fall_2024/tree/main/Seminars/03-numpy\n",
        "- https://github.com/mmp-practicum-team/mmp_practicum_spring_2024/tree/main/Seminars/Seminar%2003"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GaGh9SZ_wr4w"
      },
      "source": [
        "О чём можно узнать из этого ноутбука:\n",
        "\n",
        "* операции при работе с массивами (тензорами)\n",
        "* многомерные массивы\n",
        "* работа с GPU\n",
        "\n",
        "Официальная документация:\n",
        "- https://pytorch.org/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:48.254252Z",
          "start_time": "2022-09-22T08:00:48.244253Z"
        },
        "id": "Uw8BOvbawr4x"
      },
      "outputs": [],
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Наивная работа с матрицами в Python"
      ],
      "metadata": {
        "id": "6r7C2ZBpB1Mw"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_vWCVV6Swr4x"
      },
      "source": [
        "### `Представление матрицы в Python`\n",
        "\n",
        "Простейший вариант - список списков:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:48.538438Z",
          "start_time": "2022-09-22T08:00:48.524440Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GVZ23sY-wr4x",
        "outputId": "f8374288-9061-4938-85d9-7322106ccfd3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1, 2, 3]\n",
            "[4, 5, 6]\n",
            "[7, 8, 9]\n",
            "\n",
            "[1, 0, 0]\n",
            "[0, 2, 0]\n",
            "[0, 0, 3]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "A = [\n",
        "    [1, 2, 3],\n",
        "    [4, 5, 6],\n",
        "    [7, 8, 9],\n",
        "]\n",
        "\n",
        "B = [\n",
        "    [1, 0, 0],\n",
        "    [0, 2, 0],\n",
        "    [0, 0, 3],\n",
        "]\n",
        "\n",
        "def print_matrix(A):\n",
        "    for row in A:\n",
        "        print(row)\n",
        "    print()\n",
        "\n",
        "print_matrix(A)\n",
        "print_matrix(B)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8FB9MFXOwr4z"
      },
      "source": [
        "### `Представление матрицы в Python`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e2Jit5kAwr4z"
      },
      "source": [
        "Одна из проблем списков — отсутствие поэлементных и матричных операций.\n",
        "\n",
        "Другой недостаток — работа со списками не позволяет использовать векторные инструкции в процессоре, которые на порядки ускоряют матричные вычисления"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ojyJOPxNwr4z"
      },
      "source": [
        "Попробуем самостоятельно реализовать несколько методов для матричных вычислений, работающих с представлением матриц в виде списков.\n",
        "\n",
        "Для оценки производительности будем использовать *декооратор* `timed`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:50.167484Z",
          "start_time": "2022-09-22T08:00:50.156486Z"
        },
        "id": "HTk6t_kCwr4z"
      },
      "outputs": [],
      "source": [
        "from time import perf_counter\n",
        "\n",
        "def timed(method):\n",
        "    def __timed(*args, **kw):\n",
        "        time_start = perf_counter()\n",
        "        result = method(*args, **kw)\n",
        "        time_end = perf_counter()\n",
        "        print('{}  {:.3f} ms\\n'.format(method.__name__,\n",
        "                                      (time_end - time_start) * 1000))\n",
        "        return result\n",
        "\n",
        "    return __timed"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gdqjbjMowr4z"
      },
      "source": [
        "### `Опишем базовые операции: транспонирование`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MAoiZkKp-b38"
      },
      "source": [
        "Для матрицы $A\\in\\text{Mat}(m\\times n)$:\n",
        "$$\n",
        "C_{ij} := A_{ji}\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xjzWcM57-b38",
        "outputId": "82b7443c-77ea-45fc-b66f-1cac1782cc0f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "matrix_transpose  0.009 ms\n",
            "\n",
            "[1, 2, 3]\n",
            "[4, 5, 6]\n",
            "[7, 8, 9]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "@timed\n",
        "def matrix_transpose(A):\n",
        "    m, n = len(A), len(A[0])\n",
        "    C = [[0] * n for _ in range(m)]\n",
        "    for i in range(m):\n",
        "      for j in range(n):\n",
        "        C[i][j] = A[i][j]\n",
        "    return C\n",
        "\n",
        "print_matrix(matrix_transpose(A))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "473o8z9Cwr4z"
      },
      "source": [
        "### `Опишем базовые операции: сложение`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tw_scF1Y-b39"
      },
      "source": [
        "Для матриц одинакового размера $A,B\\in\\text{Mat}(m\\times n)$:\n",
        "\n",
        "$$\n",
        "C_{ij}:=A_{ij}+B_{ij}\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:50.494747Z",
          "start_time": "2022-09-22T08:00:50.484748Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3cncedaAwr4z",
        "outputId": "49bdd051-5e34-487a-dfb8-2ca7faa94e07"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "matrix_add  0.013 ms\n",
            "\n",
            "[2, 2, 3]\n",
            "[4, 7, 6]\n",
            "[7, 8, 12]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "@timed\n",
        "def matrix_add(A, B):\n",
        "    # skip correctness of dimensions check\n",
        "\n",
        "    n, m = len(A), len(A[0])\n",
        "    C = [[0.0 for _ in range(m)] for _ in range(n)]\n",
        "\n",
        "    for i in range(n):\n",
        "        for j in range(m):\n",
        "            C[i][j] = A[i][j] + B[i][j]\n",
        "    return C\n",
        "\n",
        "print_matrix(matrix_add(A, B))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xowumGI7wr40"
      },
      "source": [
        "### `Опишем базовые операции: умножение`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AWPhH2lM-b39"
      },
      "source": [
        "Для матрицы $A\\in\\text{Mat}(m\\times n)$, $B\\in\\text{Mat}(n\\times m)$:\n",
        "$$\n",
        "C_{ij}=\\sum_{r=1}^n A_{ir}B_{rj}\n",
        "$$\n",
        "\n",
        "Анимированный пример:\n",
        "\n",
        "![mm_gif](https://notesbylex.com/_media/matrix-multiplication.gif)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:51.093622Z",
          "start_time": "2022-09-22T08:00:51.084622Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EQBK4OQEwr40",
        "outputId": "09306cc4-3773-4b62-9876-ebc2482f5378"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "matrix_transpose  0.008 ms\n",
            "\n",
            "matrix_mul  0.087 ms\n",
            "\n",
            "[1, 4, 9]\n",
            "[4, 10, 18]\n",
            "[7, 16, 27]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "def dot(a, b):\n",
        "    return sum(x * y for (x, y) in zip(a, b))\n",
        "\n",
        "@timed\n",
        "def matrix_mul(A, B):\n",
        "    # check correctness of dimensions\n",
        "    n, m = len(A), len(B[0])\n",
        "    C = [[0.0 for _ in range(n)] for _ in range(m)]\n",
        "\n",
        "    B_T = matrix_transpose(B)\n",
        "    for i in range(n):\n",
        "        for j in range(m):\n",
        "            C[i][j] = dot(A[i], B_T[j])\n",
        "    return C\n",
        "\n",
        "print_matrix(matrix_mul(A, B))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hESzT5BAwr40"
      },
      "source": [
        "## `Базовая работа с тензорвами в PyTorch`\n",
        "\n",
        "- Позволяет работать с многомерными массивами (**тензорами**)\n",
        "- Реализует множество базовых матричных операций\n",
        "- Работает на порядки быстрее самописных реализаций в Python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:51.493514Z",
          "start_time": "2022-09-22T08:00:51.428515Z"
        },
        "id": "yhd52Fo5wr40"
      },
      "outputs": [],
      "source": [
        "import torch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fXJ5nrGfwr40"
      },
      "source": [
        "### `Почему torch быстрый`\n",
        "\n",
        "- Значительная часть кода написана на C++ и CUDA\n",
        "\n",
        "- Базовым классом является `Tensor`, имеющий следующие отличия от списков:\n",
        "    1. Имеет фиксированную длину, задаваемую в момент его создания (списки в Python могут менять размер динамически)\n",
        "    2. Все элементы в NumPy array имеют один тип\n",
        "- PyTorch поддерживает ускорение с помощью GPU, что позволяет использовать возможности параллельной обработки. Это может привести к ускорению обучения и использования моделей глубокого обучения на порядок\n",
        "- И миллион других причин\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uSZUFyR0wr40"
      },
      "source": [
        "### `Способы создания torch.Tensor`\n",
        "\n",
        "1. Из списка списков\n",
        "2. Пустой\n",
        "3. Заполненный нулями\n",
        "4. Заполненный единицами\n",
        "5. Заполненный нужным значением"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xOb-It6W-b3-",
        "outputId": "c578a24e-fb7b-4e72-9692-6d32b7e1a867"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1, 2, 3],\n",
              "        [4, 5, 6]])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "a = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
        "a"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "a.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D_OCPIQgADB1",
        "outputId": "adc8d2ec-ecff-4b46-e92a-6ef48e77b1ef"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([2, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:52.259961Z",
          "start_time": "2022-09-22T08:00:52.252963Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1fR_01Yywr40",
        "outputId": "cb03ea2b-a5c4-4e73-e7c9-98d6391dd23a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 2.8026e-44,  3.2230e-44, -5.1851e-26],\n",
              "        [ 3.1689e-41,  1.5638e-42,  0.0000e+00]])"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "torch.empty([2, 3])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:52.445821Z",
          "start_time": "2022-09-22T08:00:52.436823Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9SOJ_CG2wr40",
        "outputId": "40c0e8a5-ab80-42ee-f6d0-1a944641acb2"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 0., 0.],\n",
              "        [0., 0., 0.]])"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "torch.zeros([2, 3])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:52.637411Z",
          "start_time": "2022-09-22T08:00:52.620413Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PfwcgUEzwr41",
        "outputId": "0847c7fa-da8e-484a-a8d0-f7789b43a0c4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 1., 1.],\n",
              "        [1., 1., 1.]])"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "torch.ones([2, 3])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:52.811524Z",
          "start_time": "2022-09-22T08:00:52.796522Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A-9ZRE2cwr41",
        "outputId": "11b52db3-166c-42f6-f934-c6239d05c905"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[3., 3., 3.],\n",
              "        [3., 3., 3.]])"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "torch.full([2, 3], fill_value=3.0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kjDsvKhmwr4_"
      },
      "source": [
        "Некорректные входные данные приведут к ошибке:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:55.940391Z",
          "start_time": "2022-09-22T08:00:55.924393Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 146
        },
        "id": "wbGe0lAxwr4_",
        "outputId": "f4148f65-5ad2-4b99-dc6c-96dec6d93faf"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "ValueError",
          "evalue": "expected sequence of length 3 at dim 1 (got 2)",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-14-f24a747d4613>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m6\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mValueError\u001b[0m: expected sequence of length 3 at dim 1 (got 2)"
          ]
        }
      ],
      "source": [
        "torch.tensor([[1, 2, 3], [4, 6]])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:56.462636Z",
          "start_time": "2022-09-22T08:00:56.444636Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gfIvNgVfwr5A",
        "outputId": "f7a0dd8d-6982-4c16-80b5-33efbf63b537"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([2, 3, 4, 5, 6, 7, 8, 9])"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "# Синтаксис аналогичен built-in функции `range`\n",
        "torch.arange(2, 10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PtdBWt4ewr5A"
      },
      "source": [
        "### `Способы создания torch.Tensor`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9yvPc7Pwr5A"
      },
      "source": [
        "**Задача.** Задайте массив `result` размером $50$ на $30$, состоящий из троек"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:00:57.341436Z",
          "start_time": "2022-09-22T08:00:57.333435Z"
        },
        "id": "MPyhcybPwr5A"
      },
      "outputs": [],
      "source": [
        "### your code here"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T06:50:02.269559Z",
          "start_time": "2022-09-22T06:50:02.253560Z"
        },
        "id": "xQR34U0Twr5A"
      },
      "source": [
        "**Задача.** Какая размерность будет у массива, полученного с помощью команды `torch.tensor([[1], [2]])`?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FV_fLkBVwr5D"
      },
      "source": [
        "### `Арифметические операции`\n",
        "\n",
        "- Арифметические операции в общем случае по-элементные и требуют одинакового размера операндов<br>\n",
        "\n",
        "- Но torch может применять их к операндам разного размера с помощью broadcasting, то есть правил обработки операндов разного размера<br><br>\n",
        "\n",
        "Примеры операций с массивами одного размера:"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "a = torch.tensor(A)\n",
        "a"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MdsAWOMOAWjZ",
        "outputId": "a431f224-2e12-4b56-d63d-f8ce76ee1717"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1, 2, 3],\n",
              "        [4, 5, 6],\n",
              "        [7, 8, 9]])"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "a * 2"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fBLsTK_NAdSv",
        "outputId": "69fe81b0-cc45-401d-c753-0fbe442d1aad"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 2,  4,  6],\n",
              "        [ 8, 10, 12],\n",
              "        [14, 16, 18]])"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "b = torch.tensor(B)\n",
        "b"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-95612uoAbET",
        "outputId": "ae8f5364-abce-435f-b003-fc09d8a10ea7"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1, 0, 0],\n",
              "        [0, 2, 0],\n",
              "        [0, 0, 3]])"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:07.483335Z",
          "start_time": "2022-09-22T08:01:07.468336Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lf_4cys_wr5E",
        "outputId": "1a9a8351-0ada-4c52-aa6a-26b142237e5a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 2,  2,  3],\n",
              "        [ 4,  7,  6],\n",
              "        [ 7,  8, 12]])"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "a + b"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T5Wamqumwr5E"
      },
      "source": [
        "### `Поэлементные арифметические операции`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:07.881483Z",
          "start_time": "2022-09-22T08:01:07.868481Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NLuiSggUwr5E",
        "outputId": "b66fd15e-80b1-4e88-e980-45a2c3ab4c37"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 0.8415,  0.9093,  0.1411],\n",
              "        [-0.7568, -0.9589, -0.2794],\n",
              "        [ 0.6570,  0.9894,  0.4121]])"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "torch.sin(a)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:08.099286Z",
          "start_time": "2022-09-22T08:01:08.093285Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Cb--ILeOwr5E",
        "outputId": "5e452dd3-9381-49e1-a656-7a16daccf998"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.7311, 0.8808, 0.9526],\n",
              "        [0.9820, 0.9933, 0.9975],\n",
              "        [0.9991, 0.9997, 0.9999]])"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ],
      "source": [
        "torch.sigmoid(a)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:08.303328Z",
          "start_time": "2022-09-22T08:01:08.293329Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AbdIrzASwr5E",
        "outputId": "d0d4e61b-ddb2-4db8-e68c-53a616c67154"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1.0000, 1.4142, 1.7321],\n",
              "        [2.0000, 2.2361, 2.4495],\n",
              "        [2.6458, 2.8284, 3.0000]])"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "a ** 0.5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T07:19:36.696313Z",
          "start_time": "2022-09-22T07:19:36.683314Z"
        },
        "id": "JnHmtvFBwr5E"
      },
      "source": [
        "Поэлементые операции сравнения:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:08.696455Z",
          "start_time": "2022-09-22T08:01:08.678456Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "59RrLNAMwr5E",
        "outputId": "15e5019a-6dd7-4eb7-cf1a-0290db24acc8"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[False, False, False],\n",
              "        [ True,  True,  True],\n",
              "        [ True,  True,  True]])"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "a > 3"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "c = torch.arange(12).reshape(3,4)\n",
        "c[c > 5] = 0\n",
        "c"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c1arVAVbFJzT",
        "outputId": "130a3140-4307-44e5-a13f-a768d534f99f"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0, 1, 2, 3],\n",
              "        [4, 5, 0, 0],\n",
              "        [0, 0, 0, 0]])"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wIWBy9Vlwr5E"
      },
      "source": [
        "### `Матричные арифметические операции`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:09.092838Z",
          "start_time": "2022-09-22T08:01:09.079837Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0cuAz2RYwr5E",
        "outputId": "af6af33c-d4f1-417d-89f8-9c5084f52755"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0, 2, 3],\n",
            "        [4, 3, 6],\n",
            "        [7, 8, 6]])\n"
          ]
        }
      ],
      "source": [
        "print(a - b)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:09.312868Z",
          "start_time": "2022-09-22T08:01:09.300868Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PbqVDUWJwr5E",
        "outputId": "810607b9-02d8-4a7d-c607-67c43e359a83"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 1,  0,  0],\n",
            "        [ 0, 10,  0],\n",
            "        [ 0,  0, 27]])\n"
          ]
        }
      ],
      "source": [
        "print(a * b)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:09.504270Z",
          "start_time": "2022-09-22T08:01:09.501269Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sNG4QwuIwr5E",
        "outputId": "79d3a305-dbbd-47b3-aea5-c6daa1a3f434"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1.0000, 0.0000, 0.0000],\n",
            "        [0.0000, 0.4000, 0.0000],\n",
            "        [0.0000, 0.0000, 0.3333]])\n"
          ]
        }
      ],
      "source": [
        "print(b / a)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LL9IMT3Owr5G"
      },
      "source": [
        "### `Матричное умножение`\n",
        "\n",
        "Расмотрим для случая двумерных матриц"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:17.673826Z",
          "start_time": "2022-09-22T08:01:17.659828Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZWCMRVeWwr5G",
        "outputId": "3bbf4a81-3b3c-485e-c280-cd86abeac82f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 1,  4,  9],\n",
            "        [ 4, 10, 18],\n",
            "        [ 7, 16, 27]]) \n",
            "\n",
            "tensor([[ 1,  4,  9],\n",
            "        [ 4, 10, 18],\n",
            "        [ 7, 16, 27]])\n",
            "tensor([[ 1,  4,  9],\n",
            "        [ 4, 10, 18],\n",
            "        [ 7, 16, 27]])\n"
          ]
        }
      ],
      "source": [
        "print(a @ b, '\\n')\n",
        "print(torch.mm(a, b))\n",
        "print(torch.matmul(a, b))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ruKEwUpBwr5G"
      },
      "source": [
        "### `Транспонирование`\n",
        "\n",
        "- Расмотрим для случая двумерных матриц, кому интересны многомерные, изучайте\n",
        "https://stackoverflow.com/questions/32034237/how-does-numpys-transpose-method-permute-the-axes-of-an-array\n",
        "- При транспонировании (как и `reshape`) возвращается ссылка на те же данные"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:18.346450Z",
          "start_time": "2022-09-22T08:01:18.340451Z"
        },
        "id": "Sb5c3LQ3wr5H",
        "outputId": "fbd7c7ed-9949-4e7e-c2e4-1e82709bc994",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1, 2, 3],\n",
            "        [4, 5, 6],\n",
            "        [7, 8, 9]]) \n",
            "\n",
            " tensor([[1, 4, 7],\n",
            "        [2, 5, 8],\n",
            "        [3, 6, 9]])\n"
          ]
        }
      ],
      "source": [
        "print(a, '\\n\\n', a.T)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ld63YyY4wr5H"
      },
      "source": [
        "### `Сравнение скорости`\n",
        "\n",
        "Воспользуемся декоратором `timed` для сравнения скорости матричных операций, реализованных в `torch` и самостоятельно:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:18.711053Z",
          "start_time": "2022-09-22T08:01:18.708055Z"
        },
        "id": "BtZObWGWwr5H"
      },
      "outputs": [],
      "source": [
        "@timed\n",
        "def matrix_add_np(A, B): return A + B\n",
        "\n",
        "@timed\n",
        "def matrix_mul_np(A, B): return A @ B"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2022-09-22T08:01:20.085052Z",
          "start_time": "2022-09-22T08:01:18.909053Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "23AocmUnwr5H",
        "outputId": "704ab49d-f2ff-4c22-a85b-7bc5a847a8b7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "matrix_add  14355.076 ms\n",
            "\n",
            "matrix_add_np  5.409 ms\n",
            "\n",
            "matrix_transpose  139.585 ms\n",
            "\n",
            "matrix_mul  49526.197 ms\n",
            "\n",
            "matrix_mul_np  15.752 ms\n",
            "\n"
          ]
        }
      ],
      "source": [
        "tmp = [range(1000) for _ in range(1000)]\n",
        "X, Y = torch.tensor(tmp), torch.tensor(tmp)\n",
        "\n",
        "_ = matrix_add(X, Y)\n",
        "_ = matrix_add_np(X, Y)\n",
        "\n",
        "tmp = [range(200) for _ in range(200)]\n",
        "X, Y = torch.tensor(tmp), torch.tensor(tmp)\n",
        "\n",
        "_ = matrix_mul(X, Y)\n",
        "_ = matrix_mul_np(X, Y)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## `Работа с GPU`"
      ],
      "metadata": {
        "id": "dDkDlqplB7Yy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Копирование на девайс:"
      ],
      "metadata": {
        "id": "3wHjYGnODY-h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "a = torch.tensor([range(1000) for _ in range(1000)])\n",
        "a_gpu = a.cuda()"
      ],
      "metadata": {
        "id": "zpD3xKbdCER4"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "a.device"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GWzmG5x_CiIe",
        "outputId": "f0be94fe-65b1-4a2e-d90e-3528f6130522"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "device(type='cpu')"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "a_gpu.device"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pl1BmozuCkf8",
        "outputId": "3931a896-adbb-4064-e506-8e0544e28d8d"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "device(type='cuda', index=0)"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Огромный прирост скорости:"
      ],
      "metadata": {
        "id": "nOpCQPWtDa8S"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tmp = [range(1000) for _ in range(1000)]\n",
        "X, Y = torch.tensor(tmp), torch.tensor(tmp)\n",
        "\n",
        "_ = matrix_add_np(X, Y)\n",
        "\n",
        "X_gpu, Y_gpu = X.cuda(), Y.cuda()\n",
        "_ = matrix_add_np(X_gpu, Y_gpu)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kkGbqNBRCroV",
        "outputId": "7489c2e6-1856-481c-b928-668f2a9b3ab6"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "matrix_add_np  2.495 ms\n",
            "\n",
            "matrix_add_np  43.926 ms\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Правила работы:\n",
        "- тензоры, участвующие в одной операции, должны находиться на одном устройстве (оба на CPU / оба на GPU)\n",
        "- при вызове `.cuda()` данные копируются, причем не мгновенно"
      ],
      "metadata": {
        "id": "VGUBe5z4DhqN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X + X_gpu"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 146
        },
        "id": "FYnt_YXxCzFV",
        "outputId": "3c4222b0-e9e0-4483-b705-d65017478f18"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-37-836e3385f0d0>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mX\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mX_gpu\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "X = torch.arange(1280000).reshape(10,-1)\n",
        "\n",
        "start = perf_counter()\n",
        "X.cuda().cpu().cuda().cpu()\n",
        "print(perf_counter() - start)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9WeUZ0uPDzip",
        "outputId": "b34ed37c-46dd-49ab-a6ce-b73d09851296"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.012777309000000514\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TisntbEl-gYx"
      },
      "source": [
        "## `Autograd`"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### `Интро`"
      ],
      "metadata": {
        "id": "4sLeVVYiHkZf"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-OLSx6YQ-gYy"
      },
      "source": [
        "Все тензоры содержат атрибут `.grad`, который может хранить градиент по этому тензору."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:32:18.734804Z",
          "start_time": "2023-02-28T12:32:18.709812Z"
        },
        "id": "TVvRHtiG-gYy"
      },
      "outputs": [],
      "source": [
        "X.grad"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JFQBWkcp-gYy"
      },
      "source": [
        "По умолчанию, тензоры, создаваемые в pytorch, не будут требовать, чтобы для них посчитали градиент. Для этого надо добавить дополнительный аргумент `requires_grad=True`, либо вызвав метод `.requires_grad_()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:32:37.982633Z",
          "start_time": "2023-02-28T12:32:37.955461Z"
        },
        "id": "4iLRIKC3-gYy"
      },
      "outputs": [],
      "source": [
        "x = torch.tensor([1., 2., 3.], requires_grad=True)\n",
        "y = torch.tensor([1., 2., 3.], requires_grad=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T09:50:22.820474Z",
          "start_time": "2023-02-28T09:50:22.793992Z"
        },
        "id": "-yqmqAGp-gYy"
      },
      "source": [
        "Производя операции с переменными, по которым нужно считать градиенты, мы конструируем граф вычислений:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:33:38.274017Z",
          "start_time": "2023-02-28T12:33:38.247800Z"
        },
        "id": "yIBXk3Ij-gYy",
        "outputId": "4ba3eb60-244d-4c99-f279-941e4b597030",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ],
      "source": [
        "z = (3 * x**3 - y**2).sum()\n",
        "z.requires_grad"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T09:50:37.275708Z",
          "start_time": "2023-02-28T09:50:37.248285Z"
        },
        "id": "E9jDluKX-gYy"
      },
      "source": [
        "В каждой переменной есть информация о том, как именно она была получена при проходе вперёд. Исходя из этой информации у тензоров в графе вычислений хранятся функции, которые должны быть вызваны на обратном проходе для расчёта градиента."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:34:15.874921Z",
          "start_time": "2023-02-28T12:34:15.848470Z"
        },
        "id": "epd-GVQO-gYy",
        "outputId": "a4b680bf-27d9-43e2-c1cb-1c5d2f09415c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<SumBackward0 at 0x7b1016a57310>"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ],
      "source": [
        "z.grad_fn"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T09:50:49.361799Z",
          "start_time": "2023-02-28T09:50:49.334010Z"
        },
        "id": "wr8lR4iR-gYz"
      },
      "source": [
        "Для примера сделаем из тензора `z` скаляр, сложив все его элементы, и посчитаем градиенты с помощью функции .backward()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:36:35.186266Z",
          "start_time": "2023-02-28T12:36:35.134969Z"
        },
        "id": "z5RE1022-gYz"
      },
      "outputs": [],
      "source": [
        "z.backward()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:36:36.720780Z",
          "start_time": "2023-02-28T12:36:36.698582Z"
        },
        "id": "7fAQsxY5-gYz",
        "outputId": "9c149fcc-1aa7-4f2a-e057-7bd1b2212710",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([ 9., 36., 81.]), tensor([-2., -4., -6.]))"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ],
      "source": [
        "x.grad, y.grad"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T09:50:49.361799Z",
          "start_time": "2023-02-28T09:50:49.334010Z"
        },
        "id": "CvXYkksQ-gYz"
      },
      "source": [
        "Сравним с посчитанным вручную градиентом:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:36:55.067720Z",
          "start_time": "2023-02-28T12:36:55.038990Z"
        },
        "id": "LkDyzLjP-gYz",
        "outputId": "041f7c7d-d82a-4b11-ce5a-11da58521d93",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ],
      "source": [
        "torch.allclose(x.grad, 9 * x**2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-dh0Bzu0-gY0"
      },
      "source": [
        "### `Визуализация графа вычислений`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3B68Wwio-gY0"
      },
      "source": [
        "Для примера визуализируем, как бы выглядел граф вычислений для линейной регресси.\n",
        "\n",
        "Заметим, что пакет визуализации `pytorchviz` предназначен в первую очередь для визуализации нейронных сетей, поэтому нам необходимо будет использовать класс `torch.nn.Parameter`, который является обёрткой над тензором и несколько расширяет возможности аргумента `requires_grad=True`\n",
        "\n",
        "Графы визуализируются библиотекой `graphviz`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T09:53:47.806077Z",
          "start_time": "2023-02-28T09:53:20.107297Z"
        },
        "id": "02s-Y8PB-gY0",
        "outputId": "cc34968c-f197-4c3b-8eb8-4f44feee63b1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting torchviz\n",
            "  Downloading torchviz-0.0.3-py3-none-any.whl.metadata (2.1 kB)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from torchviz) (2.5.1+cu121)\n",
            "Requirement already satisfied: graphviz in /usr/local/lib/python3.10/dist-packages (from torchviz) (0.20.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (4.12.2)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (2024.10.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch->torchviz) (1.3.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->torchviz) (3.0.2)\n",
            "Downloading torchviz-0.0.3-py3-none-any.whl (5.7 kB)\n",
            "Installing collected packages: torchviz\n",
            "Successfully installed torchviz-0.0.3\n"
          ]
        }
      ],
      "source": [
        "! pip3 install torchviz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:39:02.053088Z",
          "start_time": "2023-02-28T12:39:02.027188Z"
        },
        "id": "Sl0PK4G8-gY0"
      },
      "outputs": [],
      "source": [
        "from torchviz import make_dot"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([1., 2., 3.], requires_grad=True)\n",
        "y = torch.tensor([1., 2., 3.], requires_grad=True)\n",
        "z = (3 * x**3 - y**2).sum()\n",
        "make_dot(z, params={'x': x, 'y': y, 'z': z})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "ABxwOsfUH0UX",
        "outputId": "b1187e7d-f7ca-4fd3-824e-dc8299317075"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.43.0 (0)\n -->\n<!-- Title: %3 Pages: 1 -->\n<svg width=\"222pt\" height=\"390pt\"\n viewBox=\"0.00 0.00 222.00 390.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 386)\">\n<title>%3</title>\n<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-386 218,-386 218,4 -4,4\"/>\n<!-- 135308999412688 -->\n<g id=\"node1\" class=\"node\">\n<title>135308999412688</title>\n<polygon fill=\"#caff70\" stroke=\"black\" points=\"133.5,-30 79.5,-30 79.5,0 133.5,0 133.5,-30\"/>\n<text text-anchor=\"middle\" x=\"106.5\" y=\"-18\" font-family=\"monospace\" font-size=\"10.00\">z</text>\n<text text-anchor=\"middle\" x=\"106.5\" y=\"-7\" font-family=\"monospace\" font-size=\"10.00\"> ()</text>\n</g>\n<!-- 135309029435344 -->\n<g id=\"node2\" class=\"node\">\n<title>135309029435344</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"151,-85 62,-85 62,-66 151,-66 151,-85\"/>\n<text text-anchor=\"middle\" x=\"106.5\" y=\"-73\" font-family=\"monospace\" font-size=\"10.00\">SumBackward0</text>\n</g>\n<!-- 135309029435344&#45;&gt;135308999412688 -->\n<g id=\"edge9\" class=\"edge\">\n<title>135309029435344&#45;&gt;135308999412688</title>\n<path fill=\"none\" stroke=\"black\" d=\"M106.5,-65.87C106.5,-59.11 106.5,-49.35 106.5,-40.26\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"110,-40.11 106.5,-30.11 103,-40.11 110,-40.11\"/>\n</g>\n<!-- 135309029435632 -->\n<g id=\"node3\" class=\"node\">\n<title>135309029435632</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"151,-140 62,-140 62,-121 151,-121 151,-140\"/>\n<text text-anchor=\"middle\" x=\"106.5\" y=\"-128\" font-family=\"monospace\" font-size=\"10.00\">SubBackward0</text>\n</g>\n<!-- 135309029435632&#45;&gt;135309029435344 -->\n<g id=\"edge1\" class=\"edge\">\n<title>135309029435632&#45;&gt;135309029435344</title>\n<path fill=\"none\" stroke=\"black\" d=\"M106.5,-120.75C106.5,-113.8 106.5,-103.85 106.5,-95.13\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"110,-95.09 106.5,-85.09 103,-95.09 110,-95.09\"/>\n</g>\n<!-- 135309029428528 -->\n<g id=\"node4\" class=\"node\">\n<title>135309029428528</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"98,-195 9,-195 9,-176 98,-176 98,-195\"/>\n<text text-anchor=\"middle\" x=\"53.5\" y=\"-183\" font-family=\"monospace\" font-size=\"10.00\">MulBackward0</text>\n</g>\n<!-- 135309029428528&#45;&gt;135309029435632 -->\n<g id=\"edge2\" class=\"edge\">\n<title>135309029428528&#45;&gt;135309029435632</title>\n<path fill=\"none\" stroke=\"black\" d=\"M62.25,-175.75C69.97,-168.03 81.4,-156.6 90.72,-147.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"93.31,-149.64 97.91,-140.09 88.36,-144.69 93.31,-149.64\"/>\n</g>\n<!-- 135309029440048 -->\n<g id=\"node5\" class=\"node\">\n<title>135309029440048</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"95,-250 6,-250 6,-231 95,-231 95,-250\"/>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-238\" font-family=\"monospace\" font-size=\"10.00\">PowBackward0</text>\n</g>\n<!-- 135309029440048&#45;&gt;135309029428528 -->\n<g id=\"edge3\" class=\"edge\">\n<title>135309029440048&#45;&gt;135309029428528</title>\n<path fill=\"none\" stroke=\"black\" d=\"M51,-230.75C51.39,-223.8 51.95,-213.85 52.45,-205.13\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"55.94,-205.27 53.01,-195.09 48.95,-204.88 55.94,-205.27\"/>\n</g>\n<!-- 135309029427520 -->\n<g id=\"node6\" class=\"node\">\n<title>135309029427520</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"101,-310.5 0,-310.5 0,-291.5 101,-291.5 101,-310.5\"/>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-298.5\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n</g>\n<!-- 135309029427520&#45;&gt;135309029440048 -->\n<g id=\"edge4\" class=\"edge\">\n<title>135309029427520&#45;&gt;135309029440048</title>\n<path fill=\"none\" stroke=\"black\" d=\"M50.5,-291.37C50.5,-283.25 50.5,-270.81 50.5,-260.39\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"54,-260.17 50.5,-250.17 47,-260.17 54,-260.17\"/>\n</g>\n<!-- 135308999130096 -->\n<g id=\"node7\" class=\"node\">\n<title>135308999130096</title>\n<polygon fill=\"lightblue\" stroke=\"black\" points=\"77.5,-382 23.5,-382 23.5,-352 77.5,-352 77.5,-382\"/>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-370\" font-family=\"monospace\" font-size=\"10.00\">x</text>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-359\" font-family=\"monospace\" font-size=\"10.00\"> (3)</text>\n</g>\n<!-- 135308999130096&#45;&gt;135309029427520 -->\n<g id=\"edge5\" class=\"edge\">\n<title>135308999130096&#45;&gt;135309029427520</title>\n<path fill=\"none\" stroke=\"black\" d=\"M50.5,-351.8C50.5,-342.7 50.5,-330.79 50.5,-320.9\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"54,-320.84 50.5,-310.84 47,-320.84 54,-320.84\"/>\n</g>\n<!-- 135309029430592 -->\n<g id=\"node8\" class=\"node\">\n<title>135309029430592</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"206,-195 117,-195 117,-176 206,-176 206,-195\"/>\n<text text-anchor=\"middle\" x=\"161.5\" y=\"-183\" font-family=\"monospace\" font-size=\"10.00\">PowBackward0</text>\n</g>\n<!-- 135309029430592&#45;&gt;135309029435632 -->\n<g id=\"edge6\" class=\"edge\">\n<title>135309029430592&#45;&gt;135309029435632</title>\n<path fill=\"none\" stroke=\"black\" d=\"M152.42,-175.75C144.41,-168.03 132.54,-156.6 122.88,-147.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"125.04,-144.51 115.41,-140.09 120.19,-149.55 125.04,-144.51\"/>\n</g>\n<!-- 135309029430640 -->\n<g id=\"node9\" class=\"node\">\n<title>135309029430640</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"214,-250 113,-250 113,-231 214,-231 214,-250\"/>\n<text text-anchor=\"middle\" x=\"163.5\" y=\"-238\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n</g>\n<!-- 135309029430640&#45;&gt;135309029430592 -->\n<g id=\"edge7\" class=\"edge\">\n<title>135309029430640&#45;&gt;135309029430592</title>\n<path fill=\"none\" stroke=\"black\" d=\"M163.17,-230.75C162.91,-223.8 162.53,-213.85 162.2,-205.13\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"165.7,-204.95 161.82,-195.09 158.7,-205.21 165.7,-204.95\"/>\n</g>\n<!-- 135308999116496 -->\n<g id=\"node10\" class=\"node\">\n<title>135308999116496</title>\n<polygon fill=\"lightblue\" stroke=\"black\" points=\"190.5,-316 136.5,-316 136.5,-286 190.5,-286 190.5,-316\"/>\n<text text-anchor=\"middle\" x=\"163.5\" y=\"-304\" font-family=\"monospace\" font-size=\"10.00\">y</text>\n<text text-anchor=\"middle\" x=\"163.5\" y=\"-293\" font-family=\"monospace\" font-size=\"10.00\"> (3)</text>\n</g>\n<!-- 135308999116496&#45;&gt;135309029430640 -->\n<g id=\"edge8\" class=\"edge\">\n<title>135308999116496&#45;&gt;135309029430640</title>\n<path fill=\"none\" stroke=\"black\" d=\"M163.5,-285.84C163.5,-278.21 163.5,-268.7 163.5,-260.45\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"167,-260.27 163.5,-250.27 160,-260.27 167,-260.27\"/>\n</g>\n</g>\n</svg>\n",
            "text/plain": [
              "<graphviz.graphs.Digraph at 0x7b1016a24c70>"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T12:42:47.149224Z",
          "start_time": "2023-02-28T12:42:46.981157Z"
        },
        "id": "P28uRDbo-gY0",
        "outputId": "a99e31c5-711d-4e44-f5a2-1a95e264e457",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 615
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.43.0 (0)\n -->\n<!-- Title: %3 Pages: 1 -->\n<svg width=\"219pt\" height=\"445pt\"\n viewBox=\"0.00 0.00 219.00 445.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 441)\">\n<title>%3</title>\n<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-441 215,-441 215,4 -4,4\"/>\n<!-- 135308999610496 -->\n<g id=\"node1\" class=\"node\">\n<title>135308999610496</title>\n<polygon fill=\"#caff70\" stroke=\"black\" points=\"132.5,-30 78.5,-30 78.5,0 132.5,0 132.5,-30\"/>\n<text text-anchor=\"middle\" x=\"105.5\" y=\"-18\" font-family=\"monospace\" font-size=\"10.00\">Loss</text>\n<text text-anchor=\"middle\" x=\"105.5\" y=\"-7\" font-family=\"monospace\" font-size=\"10.00\"> ()</text>\n</g>\n<!-- 135309029435440 -->\n<g id=\"node2\" class=\"node\">\n<title>135309029435440</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"153,-85 58,-85 58,-66 153,-66 153,-85\"/>\n<text text-anchor=\"middle\" x=\"105.5\" y=\"-73\" font-family=\"monospace\" font-size=\"10.00\">MeanBackward0</text>\n</g>\n<!-- 135309029435440&#45;&gt;135308999610496 -->\n<g id=\"edge9\" class=\"edge\">\n<title>135309029435440&#45;&gt;135308999610496</title>\n<path fill=\"none\" stroke=\"black\" d=\"M105.5,-65.87C105.5,-59.11 105.5,-49.35 105.5,-40.26\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"109,-40.11 105.5,-30.11 102,-40.11 109,-40.11\"/>\n</g>\n<!-- 135309029441200 -->\n<g id=\"node3\" class=\"node\">\n<title>135309029441200</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"150,-140 61,-140 61,-121 150,-121 150,-140\"/>\n<text text-anchor=\"middle\" x=\"105.5\" y=\"-128\" font-family=\"monospace\" font-size=\"10.00\">PowBackward0</text>\n</g>\n<!-- 135309029441200&#45;&gt;135309029435440 -->\n<g id=\"edge1\" class=\"edge\">\n<title>135309029441200&#45;&gt;135309029435440</title>\n<path fill=\"none\" stroke=\"black\" d=\"M105.5,-120.75C105.5,-113.8 105.5,-103.85 105.5,-95.13\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"109,-95.09 105.5,-85.09 102,-95.09 109,-95.09\"/>\n</g>\n<!-- 135309029430160 -->\n<g id=\"node4\" class=\"node\">\n<title>135309029430160</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"150,-195 61,-195 61,-176 150,-176 150,-195\"/>\n<text text-anchor=\"middle\" x=\"105.5\" y=\"-183\" font-family=\"monospace\" font-size=\"10.00\">SubBackward0</text>\n</g>\n<!-- 135309029430160&#45;&gt;135309029441200 -->\n<g id=\"edge2\" class=\"edge\">\n<title>135309029430160&#45;&gt;135309029441200</title>\n<path fill=\"none\" stroke=\"black\" d=\"M105.5,-175.75C105.5,-168.8 105.5,-158.85 105.5,-150.13\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"109,-150.09 105.5,-140.09 102,-150.09 109,-150.09\"/>\n</g>\n<!-- 135309029430400 -->\n<g id=\"node5\" class=\"node\">\n<title>135309029430400</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"150,-250 61,-250 61,-231 150,-231 150,-250\"/>\n<text text-anchor=\"middle\" x=\"105.5\" y=\"-238\" font-family=\"monospace\" font-size=\"10.00\">AddBackward0</text>\n</g>\n<!-- 135309029430400&#45;&gt;135309029430160 -->\n<g id=\"edge3\" class=\"edge\">\n<title>135309029430400&#45;&gt;135309029430160</title>\n<path fill=\"none\" stroke=\"black\" d=\"M105.5,-230.75C105.5,-223.8 105.5,-213.85 105.5,-205.13\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"109,-205.09 105.5,-195.09 102,-205.09 109,-205.09\"/>\n</g>\n<!-- 135309029432896 -->\n<g id=\"node6\" class=\"node\">\n<title>135309029432896</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"92,-305 9,-305 9,-286 92,-286 92,-305\"/>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-293\" font-family=\"monospace\" font-size=\"10.00\">MvBackward0</text>\n</g>\n<!-- 135309029432896&#45;&gt;135309029430400 -->\n<g id=\"edge4\" class=\"edge\">\n<title>135309029432896&#45;&gt;135309029430400</title>\n<path fill=\"none\" stroke=\"black\" d=\"M59.58,-285.75C67.59,-278.03 79.46,-266.6 89.12,-257.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"91.81,-259.55 96.59,-250.09 86.96,-254.51 91.81,-259.55\"/>\n</g>\n<!-- 135309029433280 -->\n<g id=\"node7\" class=\"node\">\n<title>135309029433280</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"101,-365.5 0,-365.5 0,-346.5 101,-346.5 101,-365.5\"/>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-353.5\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n</g>\n<!-- 135309029433280&#45;&gt;135309029432896 -->\n<g id=\"edge5\" class=\"edge\">\n<title>135309029433280&#45;&gt;135309029432896</title>\n<path fill=\"none\" stroke=\"black\" d=\"M50.5,-346.37C50.5,-338.25 50.5,-325.81 50.5,-315.39\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"54,-315.17 50.5,-305.17 47,-315.17 54,-315.17\"/>\n</g>\n<!-- 135308999610816 -->\n<g id=\"node8\" class=\"node\">\n<title>135308999610816</title>\n<polygon fill=\"lightblue\" stroke=\"black\" points=\"77.5,-437 23.5,-437 23.5,-407 77.5,-407 77.5,-437\"/>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-425\" font-family=\"monospace\" font-size=\"10.00\">Weight</text>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-414\" font-family=\"monospace\" font-size=\"10.00\"> (10)</text>\n</g>\n<!-- 135308999610816&#45;&gt;135309029433280 -->\n<g id=\"edge6\" class=\"edge\">\n<title>135308999610816&#45;&gt;135309029433280</title>\n<path fill=\"none\" stroke=\"black\" d=\"M50.5,-406.8C50.5,-397.7 50.5,-385.79 50.5,-375.9\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"54,-375.84 50.5,-365.84 47,-375.84 54,-375.84\"/>\n</g>\n<!-- 135309029436544 -->\n<g id=\"node9\" class=\"node\">\n<title>135309029436544</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"211,-305 110,-305 110,-286 211,-286 211,-305\"/>\n<text text-anchor=\"middle\" x=\"160.5\" y=\"-293\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n</g>\n<!-- 135309029436544&#45;&gt;135309029430400 -->\n<g id=\"edge7\" class=\"edge\">\n<title>135309029436544&#45;&gt;135309029430400</title>\n<path fill=\"none\" stroke=\"black\" d=\"M151.42,-285.75C143.41,-278.03 131.54,-266.6 121.88,-257.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"124.04,-254.51 114.41,-250.09 119.19,-259.55 124.04,-254.51\"/>\n</g>\n<!-- 135308999610736 -->\n<g id=\"node10\" class=\"node\">\n<title>135308999610736</title>\n<polygon fill=\"lightblue\" stroke=\"black\" points=\"187.5,-371 133.5,-371 133.5,-341 187.5,-341 187.5,-371\"/>\n<text text-anchor=\"middle\" x=\"160.5\" y=\"-359\" font-family=\"monospace\" font-size=\"10.00\">Bias</text>\n<text text-anchor=\"middle\" x=\"160.5\" y=\"-348\" font-family=\"monospace\" font-size=\"10.00\"> (1)</text>\n</g>\n<!-- 135308999610736&#45;&gt;135309029436544 -->\n<g id=\"edge8\" class=\"edge\">\n<title>135308999610736&#45;&gt;135309029436544</title>\n<path fill=\"none\" stroke=\"black\" d=\"M160.5,-340.84C160.5,-333.21 160.5,-323.7 160.5,-315.45\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"164,-315.27 160.5,-305.27 157,-315.27 164,-315.27\"/>\n</g>\n</g>\n</svg>\n",
            "text/plain": [
              "<graphviz.graphs.Digraph at 0x7b1016a26710>"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ],
      "source": [
        "X = torch.rand(5, 10)\n",
        "y = torch.rand(5)\n",
        "\n",
        "w = torch.rand(10, requires_grad=True)\n",
        "b = torch.rand(1, requires_grad=True)\n",
        "\n",
        "y_hat = X @ w + b\n",
        "\n",
        "loss = torch.mean((y - y_hat)**2)\n",
        "make_dot(loss, params={'Weight': w, 'Bias': b, 'Loss': loss})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XZjcg0s1-gY0"
      },
      "source": [
        "Что можно сказать об этом графе вычислений?\n",
        "\n",
        "* В листьях графа мы не видим тензора `X`, так кка нам требуется расчёт градиента только по параметрам модели.\n",
        "* `MvBackward0` соответствует матрично-векторному умножению (отсюда и первые буквы `Mv`) c матрицей `X`.\n",
        "* `AddBackward0` соответствует добавлению смещения `b`, остальная часть графа — вычисление MSE."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PjCf14ZH-gY5"
      },
      "source": [
        "### `Менеджеры контекста`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j-pGDssA-gY5"
      },
      "source": [
        "Поведением градиента сразу группы тензоров можно управлять с помощью специальных функций, которые вызываются через стандартную семантику питона `with foo():`, где `foo` — одна из трёх функций ниже:\n",
        "\n",
        "**Default Mode**\n",
        "\n",
        "Стандартный режим работы pytorch, в котором управление градиентами происходит через requires_grad. Явно его нужно вызывать только внутри других контекстных менеджеров, чтобы временно снова активировать расчёт градиентов (что случается крайне редко) вызовом `torch.enable_grad()`.\n",
        "\n",
        "**No grad mode**\n",
        "\n",
        "Данный режим используется когда блока кода нет необходимости вычислять градиенты, что занимает как вычислительные ресурсы, так и дополнительную память. Реализуется вызовом `torch.no_grad()`.\n",
        "\n",
        "**Inference mode**\n",
        "\n",
        "Аналогично No grad mode отключает расчёт градиентов, но кроме того проводит дополнительные оптимизации, что делает вычисления внутри блока кода ещё быстрее. Однако, тензоры, созданные в таком блоке будет невозможно использовать совместно с тензорами, для которых расчёт градиента необходим. Реализуется вызовом `torch.inference_mode()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T13:12:02.050827Z",
          "start_time": "2023-02-28T13:12:02.022463Z"
        },
        "id": "vXVmGmSI-gY5",
        "outputId": "307bee38-1ee8-4f6d-f63e-08b6492ea849",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {},
          "execution_count": 50
        }
      ],
      "source": [
        "x = torch.rand(3, requires_grad=True)\n",
        "\n",
        "with torch.no_grad():\n",
        "    y = x + x\n",
        "\n",
        "y.requires_grad"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-02-28T13:12:09.802900Z",
          "start_time": "2023-02-28T13:12:09.774068Z"
        },
        "id": "IpviFT2O-gY5",
        "outputId": "399d27ff-e0cd-4fb1-dee0-06c91f4945f3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {},
          "execution_count": 51
        }
      ],
      "source": [
        "@torch.no_grad()\n",
        "def foo(x):\n",
        "    return x + 2\n",
        "\n",
        "y = foo(x)\n",
        "y.requires_grad"
      ]
    }
  ],
  "metadata": {
    "celltoolbar": "Slideshow",
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "gpuType": "T4"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.15"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}