Sage_Jupy.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Sage-Jupy",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPAJ8oDRGCRYO9+JN1BVN/f",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/momo54/Sage-Jupy/blob/main/Sage_Jupy.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a20DWAV9TW46"
      },
      "source": [
        "# Running [SaGe](https://sage.univ-nantes.fr) in a Jupyter Notebook\n",
        "\n",
        "Sage is a SPARQL query engine for public Linked Data providers that implements Web preemption. The SPARQL engine includes a smart Sage client and a Sage SPARQL query server hosting RDF datasets (hosted using PostgreSQL or HDT). SPARQL queries are suspended by the web server after a fixed quantum of time and resumed upon client request. Using Web preemption, Sage ensures stable response times for query execution and completeness of results under high load.\n",
        "\n",
        "The complete approach and experimental results are available in a Research paper accepted at The Web Conference 2019, available here. Thomas Minier, Hala Skaf-Molli and Pascal Molli. \"SaGe: Web Preemption for Public SPARQL Query services\" in Proceedings of the 2019 World Wide Web Conference (WWW'19), San Francisco, USA, May 13-17, 2019.\n",
        "\n",
        "We appreciate your feedback/comments/questions to be sent to our mailing list or our issue tracker on [github](https://github.com/sage-org)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1GibuoDtTnsC"
      },
      "source": [
        "## Installation\n",
        "\n",
        "We install SaGe just with the HDT backend. There are other backend to store and update data, but not supported directly in Jupyter Notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fZ352p9F2jri",
        "outputId": "86b1f5ba-637d-4522-d3f9-73d04d3d76da"
      },
      "source": [
        "!pip install sage-engine\n",
        "!pip install pybind11\n",
        "!pip install hdt\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: sage-engine in /usr/local/lib/python3.7/dist-packages (2.3.0)\n",
            "Requirement already satisfied: PyYAML==5.4.1 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (5.4.1)\n",
            "Requirement already satisfied: rdflib-jsonld==0.5.0 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (0.5.0)\n",
            "Requirement already satisfied: uvicorn==0.13.4 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (0.13.4)\n",
            "Requirement already satisfied: click==7.1.2 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (7.1.2)\n",
            "Requirement already satisfied: protobuf==3.15.7 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (3.15.7)\n",
            "Requirement already satisfied: pylru<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (1.2.0)\n",
            "Requirement already satisfied: fastapi==0.63.0 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (0.63.0)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.36 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (1.38.0)\n",
            "Requirement already satisfied: rdflib==5.0.0 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (5.0.0)\n",
            "Requirement already satisfied: coloredlogs==15.0 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (15.0)\n",
            "Requirement already satisfied: uvloop==0.15.2 in /usr/local/lib/python3.7/dist-packages (from sage-engine) (0.15.2)\n",
            "Requirement already satisfied: typing-extensions; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from uvicorn==0.13.4->sage-engine) (3.7.4.3)\n",
            "Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.7/dist-packages (from uvicorn==0.13.4->sage-engine) (0.12.0)\n",
            "Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.7/dist-packages (from protobuf==3.15.7->sage-engine) (1.15.0)\n",
            "Requirement already satisfied: starlette==0.13.6 in /usr/local/lib/python3.7/dist-packages (from fastapi==0.63.0->sage-engine) (0.13.6)\n",
            "Requirement already satisfied: pydantic<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from fastapi==0.63.0->sage-engine) (1.8.2)\n",
            "Requirement already satisfied: isodate in /usr/local/lib/python3.7/dist-packages (from rdflib==5.0.0->sage-engine) (0.6.0)\n",
            "Requirement already satisfied: pyparsing in /usr/local/lib/python3.7/dist-packages (from rdflib==5.0.0->sage-engine) (2.4.7)\n",
            "Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.7/dist-packages (from coloredlogs==15.0->sage-engine) (9.1)\n",
            "Requirement already satisfied: pybind11 in /usr/local/lib/python3.7/dist-packages (2.6.2)\n",
            "Collecting hdt\n",
            "  Using cached https://files.pythonhosted.org/packages/51/82/41f1e4a131881da64a1ab2c4675dd93020a1a7109be08a2eb790cb6b92c6/hdt-2.3.tar.gz\n",
            "Collecting pybind11==2.2.4\n",
            "  Using cached https://files.pythonhosted.org/packages/f2/7c/e71995e59e108799800cb0fce6c4b4927914d7eada0723dd20bae3b51786/pybind11-2.2.4-py2.py3-none-any.whl\n",
            "Building wheels for collected packages: hdt\n",
            "  Building wheel for hdt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for hdt: filename=hdt-2.3-cp37-cp37m-linux_x86_64.whl size=5278053 sha256=4832150c39f2a3363e0aec2197d4cadebd3725697e68e34ab93b38c57ed52eb7\n",
            "  Stored in directory: /root/.cache/pip/wheels/c6/64/28/ee2f54a78b64368f3e633637a0707549ba7a6e1c30079d966b\n",
            "Successfully built hdt\n",
            "Installing collected packages: pybind11, hdt\n",
            "  Found existing installation: pybind11 2.6.2\n",
            "    Uninstalling pybind11-2.6.2:\n",
            "      Successfully uninstalled pybind11-2.6.2\n",
            "Successfully installed hdt-2.3 pybind11-2.2.4\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Wn_QCavT0AX"
      },
      "source": [
        "## Configuration\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-F9BXjJa71AC"
      },
      "source": [
        "We need a dataset and to configure the server to use this dataset.\n",
        "\n",
        "\n",
        "*   config.yaml is a simple configuration file for SaGe\n",
        "\n",
        "\n",
        "1.   Quantum is fixed to 75ms \n",
        "2.   max_results=2000\n",
        "\n",
        "\n",
        "*   swdf.hdt is the 'semantic web dog foord ' dataset in the HDT format. SaGe can use HDT file, or PostGres Backend or a SQLlite backend... HDT is nice when running in a Jupyter Netbooks.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xmowqyylh4kT",
        "outputId": "96674d8a-b68f-4139-91ac-37fac02ef8a1"
      },
      "source": [
        "!wget http://gaia.infor.uva.es/hdt/swdf-2012-11-28.hdt.gz\n",
        "!gunzip -f swdf-2012-11-28.hdt.gz\n",
        "## just a config.yaml.\n",
        "!wget -q \"https://raw.githubusercontent.com/momo54/Sage-Jupy/main/config.yaml\" -O config.yaml\n",
        "!cat config.yaml"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-06-06 08:43:39--  http://gaia.infor.uva.es/hdt/swdf-2012-11-28.hdt.gz\n",
            "Resolving gaia.infor.uva.es (gaia.infor.uva.es)... 157.88.123.104\n",
            "Connecting to gaia.infor.uva.es (gaia.infor.uva.es)|157.88.123.104|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2403825 (2.3M) [application/x-gzip]\n",
            "Saving to: ‘swdf-2012-11-28.hdt.gz’\n",
            "\n",
            "swdf-2012-11-28.hdt 100%[===================>]   2.29M  1.72MB/s    in 1.3s    \n",
            "\n",
            "2021-06-06 08:43:41 (1.72 MB/s) - ‘swdf-2012-11-28.hdt.gz’ saved [2403825/2403825]\n",
            "\n",
            "name: SaGe Test server\n",
            "maintainer: Chuck Norris\n",
            "quota: 75\n",
            "max_results: 2000\n",
            "default_graph_uri: http://localhost:8000/sparql/part\n",
            "graphs:\n",
            "-\n",
            "  name: swdf\n",
            "  uri: http://example.org/swdf\n",
            "  description: Semantic Web Dog Food in HDT\n",
            "  backend: hdt-file\n",
            "  file: swdf-2012-11-28.hdt\n",
            "-\n",
            "  name: swdfsq\n",
            "  uri: http://example.org/swdf-sq\n",
            "  description: Semantic Web Dog Food in SQlite\n",
            "  backend: sqlite\n",
            "  database: data-sq.db\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "521LLJh0UhiK"
      },
      "source": [
        "## Starting the server\n",
        "\n",
        "The SaGe server is started with 2 workers, a quantum of 75ms and maxpage size of 2000 results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D6GG0STh-eqS",
        "outputId": "5a3fb4a4-545d-44ac-84db-eded94a5ba87"
      },
      "source": [
        "%%bash --bg --out script_out\n",
        "sage config.yaml -p 8000 -w 2 -h \"0.0.0.0\" > server_out"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Starting job # 0 in a separate thread.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Maj46K9Knurr"
      },
      "source": [
        "## print server output\n",
        "!tail server_out"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rnu7coTdmoYt"
      },
      "source": [
        "Test if the SaGe Server is running. You should see \"\"The SaGe SPARQL query server is running!\"\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aUKX2cCo_P91",
        "outputId": "0ad8e2e3-0720-449b-9812-73edcf647170"
      },
      "source": [
        "## just testing the server is running...\n",
        "!curl http://0.0.0.0:8000"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "curl: (7) Failed to connect to 0.0.0.0 port 8000: Connection refused\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JgC4kRQRUpeV"
      },
      "source": [
        "## Running queries\n",
        "\n",
        "As a web server, SaGe can be queryied in any language. \n",
        "Below, we show how to do that in Python (as we are in Jupyter Notebook). We also provide a JS client and Java client.\n",
        "\n",
        "Just Call the SaGe server for only one quantum. The server interupt the query after a quantum exhausted or the max results reached."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vErANUp6F6Jh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "45580829-7715-402e-cdc7-977bde1971ce"
      },
      "source": [
        "import requests\n",
        "from json import dumps     \n",
        "\n",
        "###\n",
        "query='select * where {?s a ?o . ?o a ?s}'\n",
        "####\n",
        "\n",
        "entrypoint='http://0.0.0.0:8000/sparql'\n",
        "default_graph_uri='http://example.org/swdf'\n",
        "headers = {\"accept\": \"text/html\",\n",
        "        \"content-type\": \"application/json\",\n",
        "        \"next\": None}\n",
        "payload = {\"query\": query,\n",
        "        \"defaultGraph\": default_graph_uri}\n",
        "has_next = True                                                                                                                                         \n",
        "count = 0                                                                                                                                               \n",
        "nbResults = 0                                                                                                                                           \n",
        "nbCalls = 0  \n",
        "limit = 10\n",
        "\n",
        "## call the server\n",
        "response = requests.post(entrypoint, headers=headers, data=dumps(payload))\n",
        "\n",
        "## the results\n",
        "json_response = response.json() \n",
        "nbResults += len(json_response['bindings'])\n",
        "print(f'got:{nbResults}')\n",
        "\n",
        "## print some results\n",
        "for bindings in json_response['bindings']:\n",
        "    print(str(bindings))\n",
        "    count += 1\n",
        "    if count >= limit:\n",
        "      break\n",
        "\n",
        "## the link to continue the execution\n",
        "has_next = json_response['next']                                                                                                                    \n",
        "payload[\"next\"] = json_response[\"next\"]\n",
        "nbCalls += 1\n",
        "\n",
        "print(f'and the next link is {json_response[\"next\"]}')"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "got:0\n",
            "and the next link is EucEGuQECvcBClIKAj9vEi9odHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjdHlwZRoCP3MiF2h0dHA6Ly9leGFtcGxlLm9yZy9zd2RmGkkKAj9vEkNodHRwOi8vZGF0YS5zZW1hbnRpY3dlYi5vcmcvY29uZmVyZW5jZS9scmVjLzIwMDgvcGFwZXJzLzEwMS9hdXRob3JzGjQKAj9zEi5odHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjU2VxIgQ0NTA0KhoyMDIxLTA2LTA2VDA4OjQ1OjM0LjE1ODgxMTLtAQpSCgI/cxIvaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zI3R5cGUaAj9vIhdodHRwOi8vZXhhbXBsZS5vcmcvc3dkZhJBCgI/bxI7aHR0cDovL2RhdGEuc2VtYW50aWN3ZWIub3JnL2NvbmZlcmVuY2UvbHJlYy8yMDA4L3BhcGVycy8xMDESNQoCP3MSL2h0dHA6Ly9zd3JjLm9udG93YXJlLm9yZy9vbnRvbG9neSNJblByb2NlZWRpbmdzIgEwKhoyMDIxLTA2LTA2VDA4OjQ1OjM0LjE1ODgxMVpBCgI/bxI7aHR0cDovL2RhdGEuc2VtYW50aWN3ZWIub3JnL2NvbmZlcmVuY2UvbHJlYy8yMDA4L3BhcGVycy8xMDFaNQoCP3MSL2h0dHA6Ly9zd3JjLm9udG93YXJlLm9yZy9vbnRvbG9neSNJblByb2NlZWRpbmdz\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ivwJg3_A4KS"
      },
      "source": [
        "## Inspecting the Next Link\n",
        "\n",
        "We can decode the value of the next link\n",
        "As you can see. The next link contain the state of the suspended query"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9RlWApj_-r03",
        "outputId": "7b548fff-5a40-48c0-a6eb-ab9a630ae984"
      },
      "source": [
        "from sage.http_server.utils import decode_saved_plan, encode_saved_plan\n",
        "from sage.query_engine.protobuf.iterators_pb2 import (RootTree,SavedProjectionIterator,SavedScanIterator)\n",
        "import sys\n",
        "\n",
        "next_link=json_response[\"next\"]\n",
        "\n",
        "print(f'next link size:{sys.getsizeof(next_link)} bytes')\n",
        "print(f'next link:{next_link}')\n",
        "if next_link is not None:\n",
        "  saved_plan = next_link\n",
        "  plan = decode_saved_plan(saved_plan)\n",
        "  root = RootTree()\n",
        "  root.ParseFromString(plan)\n",
        "  print(root)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "next link size:873 bytes\n",
            "next link:EucEGuQECvcBClIKAj9vEi9odHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjdHlwZRoCP3MiF2h0dHA6Ly9leGFtcGxlLm9yZy9zd2RmGkkKAj9vEkNodHRwOi8vZGF0YS5zZW1hbnRpY3dlYi5vcmcvY29uZmVyZW5jZS9scmVjLzIwMDgvcGFwZXJzLzEwMS9hdXRob3JzGjQKAj9zEi5odHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjU2VxIgQ0NTA0KhoyMDIxLTA2LTA2VDA4OjQ1OjM0LjE1ODgxMTLtAQpSCgI/cxIvaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zI3R5cGUaAj9vIhdodHRwOi8vZXhhbXBsZS5vcmcvc3dkZhJBCgI/bxI7aHR0cDovL2RhdGEuc2VtYW50aWN3ZWIub3JnL2NvbmZlcmVuY2UvbHJlYy8yMDA4L3BhcGVycy8xMDESNQoCP3MSL2h0dHA6Ly9zd3JjLm9udG93YXJlLm9yZy9vbnRvbG9neSNJblByb2NlZWRpbmdzIgEwKhoyMDIxLTA2LTA2VDA4OjQ1OjM0LjE1ODgxMVpBCgI/bxI7aHR0cDovL2RhdGEuc2VtYW50aWN3ZWIub3JnL2NvbmZlcmVuY2UvbHJlYy8yMDA4L3BhcGVycy8xMDFaNQoCP3MSL2h0dHA6Ly9zd3JjLm9udG93YXJlLm9yZy9vbnRvbG9neSNJblByb2NlZWRpbmdz\n",
            "proj_source {\n",
            "  join_source {\n",
            "    scan_left {\n",
            "      pattern {\n",
            "        subject: \"?o\"\n",
            "        predicate: \"http://www.w3.org/1999/02/22-rdf-syntax-ns#type\"\n",
            "        object: \"?s\"\n",
            "        graph: \"http://example.org/swdf\"\n",
            "      }\n",
            "      mu {\n",
            "        key: \"?o\"\n",
            "        value: \"http://data.semanticweb.org/conference/lrec/2008/papers/101/authors\"\n",
            "      }\n",
            "      mu {\n",
            "        key: \"?s\"\n",
            "        value: \"http://www.w3.org/1999/02/22-rdf-syntax-ns#Seq\"\n",
            "      }\n",
            "      last_read: \"4504\"\n",
            "      timestamp: \"2021-06-06T08:45:34.158811\"\n",
            "    }\n",
            "    scan_right {\n",
            "      pattern {\n",
            "        subject: \"?s\"\n",
            "        predicate: \"http://www.w3.org/1999/02/22-rdf-syntax-ns#type\"\n",
            "        object: \"?o\"\n",
            "        graph: \"http://example.org/swdf\"\n",
            "      }\n",
            "      muc {\n",
            "        key: \"?o\"\n",
            "        value: \"http://data.semanticweb.org/conference/lrec/2008/papers/101\"\n",
            "      }\n",
            "      muc {\n",
            "        key: \"?s\"\n",
            "        value: \"http://swrc.ontoware.org/ontology#InProceedings\"\n",
            "      }\n",
            "      last_read: \"0\"\n",
            "      timestamp: \"2021-06-06T08:45:34.158811\"\n",
            "    }\n",
            "    muc {\n",
            "      key: \"?o\"\n",
            "      value: \"http://data.semanticweb.org/conference/lrec/2008/papers/101\"\n",
            "    }\n",
            "    muc {\n",
            "      key: \"?s\"\n",
            "      value: \"http://swrc.ontoware.org/ontology#InProceedings\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XVHIHznefAO5"
      },
      "source": [
        "Sending the next link back to server allow to restart the query from where it has been stopped. Basically, it works as next/next/next until no more results are available..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NGjSXh5he5kx"
      },
      "source": [
        "if has_next :\n",
        "  response = requests.post(entrypoint, headers=headers, data=dumps(payload))\n",
        "  json_response = response.json()                                                                                                                     \n",
        "  has_next = json_response['next']                                                                                                                    \n",
        "  payload[\"next\"] = json_response[\"next\"]\n",
        "  nbResults += len(json_response['bindings'])\n",
        "  nbCalls += 1\n",
        "  count=0\n",
        "  for bindings in json_response['bindings']:\n",
        "    print(str(bindings))\n",
        "    count += 1\n",
        "    if count >= limit:\n",
        "      break\n"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6K9mLhkhBI6_"
      },
      "source": [
        "## Observing progression\n",
        "\n",
        "If we decode the next link again, we can see that scan progressed (compare 'last_read' fields)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dYxtATPKBMlY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e3df8147-6f38-4e11-d906-8d2cf6a8ec09"
      },
      "source": [
        "from sage.http_server.utils import decode_saved_plan, encode_saved_plan\n",
        "from sage.query_engine.protobuf.iterators_pb2 import (RootTree,SavedProjectionIterator,SavedScanIterator)\n",
        "next_link=json_response[\"next\"]\n",
        "print(f'the next link {next_link} contains')\n",
        "if next_link is not None:\n",
        "  saved_plan = next_link\n",
        "  plan = decode_saved_plan(saved_plan)\n",
        "  root = RootTree()\n",
        "  root.ParseFromString(plan)\n",
        "  print(root)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "the next link EpIEGo8ECtwBClIKAj9vEi9odHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjdHlwZRoCP3MiF2h0dHA6Ly9leGFtcGxlLm9yZy9zd2RmGjsKAj9vEjVodHRwOi8vZGF0YS5zZW1hbnRpY3dlYi5vcmcvcGVyc29uL2VyaWthLWRlLWZyYW5jZXNjbxomCgI/cxIgaHR0cDovL3htbG5zLmNvbS9mb2FmLzAuMS9QZXJzb24iBTE0MjkyKhoyMDIxLTA2LTA2VDA4OjQ1OjM0LjE1ODgxMTLQAQpSCgI/cxIvaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zI3R5cGUaAj9vIhdodHRwOi8vZXhhbXBsZS5vcmcvc3dkZhImCgI/cxIgaHR0cDovL3htbG5zLmNvbS9mb2FmLzAuMS9QZXJzb24SMwoCP28SLWh0dHA6Ly9kYXRhLnNlbWFudGljd2ViLm9yZy9wZXJzb24vZXJpay13aWxkZSIBMCoaMjAyMS0wNi0wNlQwODo0NTozNC4xNTg4MTFaJgoCP3MSIGh0dHA6Ly94bWxucy5jb20vZm9hZi8wLjEvUGVyc29uWjMKAj9vEi1odHRwOi8vZGF0YS5zZW1hbnRpY3dlYi5vcmcvcGVyc29uL2VyaWstd2lsZGU= contains\n",
            "proj_source {\n",
            "  join_source {\n",
            "    scan_left {\n",
            "      pattern {\n",
            "        subject: \"?o\"\n",
            "        predicate: \"http://www.w3.org/1999/02/22-rdf-syntax-ns#type\"\n",
            "        object: \"?s\"\n",
            "        graph: \"http://example.org/swdf\"\n",
            "      }\n",
            "      mu {\n",
            "        key: \"?o\"\n",
            "        value: \"http://data.semanticweb.org/person/erika-de-francesco\"\n",
            "      }\n",
            "      mu {\n",
            "        key: \"?s\"\n",
            "        value: \"http://xmlns.com/foaf/0.1/Person\"\n",
            "      }\n",
            "      last_read: \"14292\"\n",
            "      timestamp: \"2021-06-06T08:45:34.158811\"\n",
            "    }\n",
            "    scan_right {\n",
            "      pattern {\n",
            "        subject: \"?s\"\n",
            "        predicate: \"http://www.w3.org/1999/02/22-rdf-syntax-ns#type\"\n",
            "        object: \"?o\"\n",
            "        graph: \"http://example.org/swdf\"\n",
            "      }\n",
            "      muc {\n",
            "        key: \"?o\"\n",
            "        value: \"http://data.semanticweb.org/person/erik-wilde\"\n",
            "      }\n",
            "      muc {\n",
            "        key: \"?s\"\n",
            "        value: \"http://xmlns.com/foaf/0.1/Person\"\n",
            "      }\n",
            "      last_read: \"0\"\n",
            "      timestamp: \"2021-06-06T08:45:34.158811\"\n",
            "    }\n",
            "    muc {\n",
            "      key: \"?o\"\n",
            "      value: \"http://data.semanticweb.org/person/erik-wilde\"\n",
            "    }\n",
            "    muc {\n",
            "      key: \"?s\"\n",
            "      value: \"http://xmlns.com/foaf/0.1/Person\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XQQPtqJ33nAG"
      },
      "source": [
        "## Iterate until termination\n",
        "\n",
        "Well, now we iterate until the end"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8NwnAC5e3rP_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a207e85c-6f8d-4763-ad6d-bc4826a685de"
      },
      "source": [
        "while has_next :\n",
        "  response = requests.post(entrypoint, headers=headers, data=dumps(payload))\n",
        "  json_response = response.json()                                                                                                                     \n",
        "  has_next = json_response['next']                                                                                                                    \n",
        "  payload[\"next\"] = json_response[\"next\"]\n",
        "  nbResults += len(json_response['bindings'])\n",
        "  nbCalls += 1\n",
        "\n",
        "## print some bindings...\n",
        "count=0\n",
        "for bindings in json_response['bindings']:\n",
        "  print(str(bindings))\n",
        "  count += 1\n",
        "  if count >= limit:\n",
        "    break\n",
        "\n",
        "print(f'got {nbResults} results')\n",
        "print(f'made {nbCalls} calls')"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "got 0 results\n",
            "made 5 calls\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}