Lab3_GeometricDeepLearning_GCN_On_Cora_PyTorch.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lab3_GeometricDeepLearning_GCN_On_Cora_PyTorch.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.7"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OlrxKhI3WFo0"
      },
      "source": [
        "# ACM Summer School 2021\n",
        "\n",
        "## Lab : Geometric DeepLearning - Graph Convolution Network on Cora Dataset with PyTorch\n",
        "\n",
        "## Presenter : Aditya Intwala\n",
        "\n",
        "![Status](https://img.shields.io/static/v1.svg?label=Status&message=Finished&color=green)\n",
        "\n",
        "**Exercise notebook:** \n",
        "[![View on Github](https://img.shields.io/static/v1.svg?logo=github&label=Repo&message=View%20On%20Github&color=lightgrey)](https://github.com/adityaintwala/ACM-GeometricDeepLearning-2021/blob/main/Lab3_GeometricDeepLearning_GCN_On_Cora_PyTorch.ipynb)\n",
        "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/adityaintwala/ACM-GeometricDeepLearning-2021/blob/main/Lab3_GeometricDeepLearning_GCN_On_Cora_PyTorch.ipynb)\n",
        "\n",
        "**Solution notebook:** \n",
        "[![View on Github](https://img.shields.io/static/v1.svg?logo=github&label=Repo&message=View%20On%20Github&color=lightgrey)](https://github.com/adityaintwala/ACM-GeometricDeepLearning-2021/blob/main/Solution/Lab3_GeometricDeepLearning_GCN_On_Cora_PyTorch.ipynb)\n",
        "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/adityaintwala/ACM-GeometricDeepLearning-2021/blob/main/Solution/Lab3_GeometricDeepLearning_GCN_On_Cora_PyTorch.ipynb)  \n",
        "\n",
        "**Solution python file:** \n",
        "[![View on Github](https://img.shields.io/static/v1.svg?logo=github&label=Repo&message=View%20On%20Github&color=lightgrey)](https://github.com/adityaintwala/ACM-GeometricDeepLearning-2021/blob/main/Lab3_GeometricDeepLearning_GCN_On_Cora_PyTorch.py)\n",
        "\n",
        "*This is an empty version of the notebook for the Lab of Geometric Deep Learning session at ACM Summer School 2021.*\n",
        "\n",
        "*Try to fill in the blanks indicated by `## TODO` comments and `...` parts in the code*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V37TmbtnWFo3"
      },
      "source": [
        "In this tutorial, we will implement custom Graph Convolution Network along with our custom GCNConv layer and use it for classification task on Cora dataset. We will also use the GCNConv layer pre-implemented in Pytorch for the same dataset and compare the results.  \n",
        "\n",
        "Below, we will start by importing our standard libraries."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7-9gfBveWFo3"
      },
      "source": [
        "# Install required packages.\n",
        "!pip install -q torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cu102.html\n",
        "!pip install -q torch-sparse -f https://pytorch-geometric.com/whl/torch-1.9.0+cu102.html\n",
        "!pip install -q torch-geometric\n",
        "## Standard libraries\n",
        "import os\n",
        "import math\n",
        "import numpy as np \n",
        "import time\n",
        "\n",
        "## Imports for plotting\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline \n",
        "from matplotlib.colors import to_rgba\n",
        "import seaborn as sns\n",
        "sns.set()\n",
        "\n",
        "import networkx as nx\n",
        "\n",
        "## PyTorch\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "\n",
        "# torch geometric\n",
        "try: \n",
        "    import torch_geometric\n",
        "except ModuleNotFoundError:\n",
        "    # You might need to install those packages with specific CUDA+PyTorch version. \n",
        "    # See https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html for details \n",
        "    !pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cu102.html\n",
        "    !pip install torch-sparse -f https://pytorch-geometric.com/whl/torch-1.9.0+cu102.html\n",
        "    !pip install torch-cluster -f https://pytorch-geometric.com/whl/torch-1.9.0+cu102.html\n",
        "    !pip install torch-spline-conv -f https://pytorch-geometric.com/whl/torch-1.9.0+cu102.html\n",
        "    !pip install torch-geometric\n",
        "    import torch_geometric\n",
        "from torch_geometric.nn import MessagePassing, GCNConv\n",
        "from torch_geometric.utils import add_self_loops, degree\n",
        "from torch_geometric.datasets import Planetoid #for Cora dataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gb3ofAYJhZl7"
      },
      "source": [
        "## Cora Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VfK6l2urhZl7"
      },
      "source": [
        "<center width=\"100%\" style=\"padding:10px\"><img src=\"https://relational.fit.cvut.cz/assets/img/datasets-generated/CORA.svg?raw=1\" width=\"500px\"></center>\n",
        "The Cora dataset consists of 2708 scientific publications classified into one of seven classes. The citation network consists of 5429 links. Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary. The dictionary consists of 1433 unique words. More information about the dataset is available at [Cora](https://relational.fit.cvut.cz/dataset/CORA)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mMW1-6UvWFo5"
      },
      "source": [
        "## Custom Graph Convolution Networks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vELS057lWFo6"
      },
      "source": [
        "### Custom Graph Convolution Layer\n",
        "\n",
        "Graph Convolutional Networks have been introduced by [Kipf et al.](https://openreview.net/pdf?id=SJU4ayYgl) in 2016 at the University of Amsterdam. He also wrote a great [blog post](https://tkipf.github.io/graph-convolutional-networks/) about this topic, which is recommended if you want to read about GCNs from a different perspective. GCNs are similar to convolutions in images in the sense that the \"filter\" parameters are typically shared over all locations in the graph. At the same time, GCNs rely on message passing methods, which means that vertices exchange information with the neighbors, and send \"messages\" to each other. Before looking at the math, we can try to visually understand how GCNs work. The first step is that each node creates a feature vector that represents the message it wants to send to all its neighbors. In the second step, the messages are sent to the neighbors, so that a node receives one message per adjacent node. Below we have visualized the two steps for our example graph. \n",
        "\n",
        "<center width=\"100%\" style=\"padding:10px\"><img src=\"https://github.com/phlippe/uvadlc_notebooks/blob/master/docs/tutorial_notebooks/tutorial7/graph_message_passing.svg?raw=1\" width=\"700px\"></center>\n",
        "\n",
        "If we want to formulate that in more mathematical terms, we need to first decide how to combine all the messages a node receives. As the number of messages vary across nodes, we need an operation that works for any number. Hence, the usual way to go is to sum or take the mean. Given the previous features of nodes $H^{(l)}$, the GCN layer is defined as follows:\n",
        "\n",
        "$$H^{(l+1)} = \\sigma\\left(\\hat{D}^{-1/2}\\hat{A}\\hat{D}^{-1/2}H^{(l)}W^{(l)}\\right)$$\n",
        "\n",
        "$W^{(l)}$ is the weight parameters with which we transform the input features into messages ($H^{(l)}W^{(l)}$). To the adjacency matrix $A$ we add the identity matrix so that each node sends its own message also to itself: $\\hat{A}=A+I$. Finally, to take the average instead of summing, we calculate the matrix $\\hat{D}$ which is a diagonal matrix with $D_{ii}$ denoting the number of neighbors node $i$ has. $\\sigma$ represents an arbitrary activation function, and not necessarily the sigmoid (usually a ReLU-based activation function is used in GNNs)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SdfJ5aE2WFo7"
      },
      "source": [
        "class GCNLayer(MessagePassing):\n",
        "    def __init__(self, in_channels, out_channels):\n",
        "        super(GCNLayer, self).__init__(aggr='add')  # \"Add\" aggregation\n",
        "        ## TODO: Add a linear layer with in_channels as input dim and out_channels as output dim\n",
        "\n",
        "    def forward(self, x, edge_index):\n",
        "        ## TODO: Step 1: Add self-loops\n",
        "        edge_index, _ = ...\n",
        "\n",
        "        ## TODO: Step 2: Multiply with weights\n",
        "        x = ...\n",
        "\n",
        "        ## TODO: Step 3: Calculate the normalization\n",
        "        \n",
        "        \n",
        "        \n",
        "        \n",
        "        norm = ...\n",
        "\n",
        "        # Step 4: Propagate the embeddings to the next layer\n",
        "        return self.propagate(edge_index, size=(x.size(0), x.size(0)), x=x, norm=norm)\n",
        "\n",
        "    def message(self, x_j, norm):\n",
        "        # Normalize node features.\n",
        "        return norm.view(-1, 1) * x_j"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N0HoPONUWFo7"
      },
      "source": [
        "Create a GCN network using Pytorch nn.Module and our custom GCNLayer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RHtAp1g2WFo8",
        "outputId": "03218f90-fa1a-44c6-8c85-2672c90b3a55"
      },
      "source": [
        "class GCNNetwork(torch.nn.Module):\n",
        "    def __init__(self, dataset):\n",
        "        super(GCNNetwork, self).__init__()\n",
        "        ## TODO: Use two GCNLayer, output of first should be 16\n",
        "        self.conv1 = ...\n",
        "        self.conv2 = ...\n",
        "\n",
        "    def forward(self, data):\n",
        "        x, edge_index = data.x, data.edge_index\n",
        "        ## TODO: Stack layers Conv1, ReLU,Dropout,Conv2\n",
        "        x = ...\n",
        "        x = ...\n",
        "        x = ...\n",
        "        x = ...\n",
        "\n",
        "        return F.log_softmax(x, dim=1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Node features:\n",
            " tensor([[[0., 1.],\n",
            "         [2., 3.],\n",
            "         [4., 5.],\n",
            "         [6., 7.]]])\n",
            "\n",
            "Adjacency matrix:\n",
            " tensor([[[1., 1., 0., 0.],\n",
            "         [1., 1., 1., 1.],\n",
            "         [0., 1., 1., 1.],\n",
            "         [0., 1., 1., 1.]]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "blSIfQoFWFo8"
      },
      "source": [
        "## Pytorch Graph Convolution Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eP9y0QVhWFo9"
      },
      "source": [
        "class GCNNetworkPytorch(torch.nn.Module):\n",
        "    def __init__(self,dataset):\n",
        "        super(GCNNetworkPytorch, self).__init__()\n",
        "        ## TODO: Use two GCNConv layers, output of first should be 16\n",
        "        self.conv1 = ...\n",
        "        self.conv2 = ...\n",
        "        \n",
        "\n",
        "    def forward(self,data):\n",
        "        x, edge_index, edge_weight = data.x, data.edge_index, data.edge_attr\n",
        "        ## TODO: Stack layers Conv1, ReLU,Dropout,Conv2\n",
        "        x = ...\n",
        "        x = ...\n",
        "        x = ...\n",
        "        return F.log_softmax(x, dim=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IiRsfDjzWFo9"
      },
      "source": [
        "## Training and Testing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6JtB6TuDWFo9",
        "outputId": "d83d7391-47a3-4bce-9d74-8e7cc734bc4d"
      },
      "source": [
        "def plot_dataset(dataset):\n",
        "    edges_raw = dataset.data.edge_index.numpy()\n",
        "    edges = [(x, y) for x, y in zip(edges_raw[0, :], edges_raw[1, :])]\n",
        "    labels = dataset.data.y.numpy()\n",
        "\n",
        "    G = nx.Graph()\n",
        "    G.add_nodes_from(list(range(np.max(edges_raw))))\n",
        "    G.add_edges_from(edges)\n",
        "    plt.subplot(111)\n",
        "    options = { 'node_size': 1, 'width': 0.2 }\n",
        "    nx.draw(G, with_labels=False, node_color=labels.tolist(), cmap=plt.cm.tab10, **options)\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Attention probs\n",
            " tensor([[[[0.3543, 0.6457, 0.0000, 0.0000],\n",
            "          [0.1096, 0.1450, 0.2642, 0.4813],\n",
            "          [0.0000, 0.1858, 0.2885, 0.5257],\n",
            "          [0.0000, 0.2391, 0.2696, 0.4913]],\n",
            "\n",
            "         [[0.5100, 0.4900, 0.0000, 0.0000],\n",
            "          [0.2975, 0.2436, 0.2340, 0.2249],\n",
            "          [0.0000, 0.3838, 0.3142, 0.3019],\n",
            "          [0.0000, 0.4018, 0.3289, 0.2693]]]])\n",
            "Adjacency matrix tensor([[[1., 1., 0., 0.],\n",
            "         [1., 1., 1., 1.],\n",
            "         [0., 1., 1., 1.],\n",
            "         [0., 1., 1., 1.]]])\n",
            "Input features tensor([[[0., 1.],\n",
            "         [2., 3.],\n",
            "         [4., 5.],\n",
            "         [6., 7.]]])\n",
            "Output features tensor([[[1.2913, 1.9800],\n",
            "         [4.2344, 3.7725],\n",
            "         [4.6798, 4.8362],\n",
            "         [4.5043, 4.7351]]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sKfdbazRhZmA"
      },
      "source": [
        "def train(model, optimizer, data):\n",
        "    train_accuracies, test_accuracies = list(), list()\n",
        "    for epoch in range(100):\n",
        "            model.train()\n",
        "            optimizer.zero_grad()\n",
        "            out = model(data)\n",
        "            loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "            train_acc = test(model, data)\n",
        "            test_acc = test(model, data, train=False)\n",
        "\n",
        "            train_accuracies.append(train_acc)\n",
        "            test_accuracies.append(test_acc)\n",
        "            print('Epoch: {:03d}, Loss: {:.5f}, Train Acc: {:.5f}, Test Acc: {:.5f}'.\n",
        "                  format(epoch, loss, train_acc, test_acc))\n",
        "\n",
        "    return train_accuracies, test_accuracies"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Chy_GRrhZmB"
      },
      "source": [
        "def test(model, data, train=True):\n",
        "    model.eval()\n",
        "\n",
        "    correct = 0\n",
        "    pred = model(data).max(dim=1)[1]\n",
        "\n",
        "    if train:\n",
        "        correct += pred[data.train_mask].eq(data.y[data.train_mask]).sum().item()\n",
        "        return correct / (len(data.y[data.train_mask]))\n",
        "    else:\n",
        "        correct += pred[data.test_mask].eq(data.y[data.test_mask]).sum().item()\n",
        "        return correct / (len(data.y[data.test_mask]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ypjj-zu7hZmB"
      },
      "source": [
        "## Cora Example"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A8tOLT04hZmB"
      },
      "source": [
        "Download / Load the Cora dataset using Planetoid dataset from torch_geometric"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n256G73yhZmB"
      },
      "source": [
        "print('========================== Downloading / Loading Cora Dataset ==========================')\n",
        "## TODO: Load Cora dataset using Planetoid\n",
        "dataset = ...\n",
        "\n",
        "#plot the dataset\n",
        "plot_dataset(dataset)\n",
        "\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0aUYCbYFhZmC"
      },
      "source": [
        "print('========================== GCNNetwork with our custom GCNLayer ==========================')\n",
        "## TODO: Create GCNNetwork\n",
        "customGCN = ...\n",
        "#assign data to device for training\n",
        "data = dataset[0].to(device)\n",
        "#create optimizer for training\n",
        "optimizer = torch.optim.Adam(customGCN.parameters(), lr=0.01, weight_decay=5e-4)\n",
        "print('============= Training & Evaluation =============')\n",
        "customGCN_train_accuracies, customGCN_test_accuracies = train(customGCN, optimizer, data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ybvm8Pg2hZmC"
      },
      "source": [
        "print('========================== GCNNetworkPytorch with pre implemented GraphConv Layer ==========================')\n",
        "## TODO: Create GCNNetworkPytorch\n",
        "pytorchGCN = ...\n",
        "#assign data to device for training\n",
        "data = dataset[0].to(device)\n",
        "#create optimizer for training\n",
        "optimizer = torch.optim.Adam([ dict(params=pytorchGCN.conv1.parameters(), weight_decay=5e-4), dict(params=pytorchGCN.conv2.parameters(), weight_decay=0)], lr=0.01)  # Only perform weight-decay on first convolution.\n",
        "print('============= Training & Evaluation =============')\n",
        "pytorchGCN_train_accuracies, pytorchGCN_test_accuracies = train(pytorchGCN, optimizer, data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d_9aJd4yhZmC"
      },
      "source": [
        "Plot the training and testing accuracies for both the models for comparision."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RsNQW47GhZmC"
      },
      "source": [
        "plt.plot(customGCN_train_accuracies, label=\"customGCN Train accuracy\")\n",
        "plt.plot(customGCN_test_accuracies, label=\"customGCN Validation accuracy\")\n",
        "plt.plot(pytorchGCN_train_accuracies, label=\"pytorchGCN Train accuracy\")\n",
        "plt.plot(pytorchGCN_test_accuracies, label=\"pytorchGCN Validation accuracy\")\n",
        "plt.xlabel(\"# Epoch\")\n",
        "plt.ylabel(\"Accuracy\")\n",
        "plt.legend(loc='lower right')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WNoVgSbfWFpH"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "In this tutorial, we have seen the custom implementation of Graph Convolution layer and compared the performance of it with the pre implemented GraphConv layer in pytorch geometric on Cora dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EfTzNcY4hZmD"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}