examples/01_Introducing_paperetl.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Introducing paperetl\n",
        "\n",
        "[paperetl](https://github.com/neuml/paperetl) is an ETL library for processing medical and scientific papers. paperetl transforms XML, CSV and PDF articles into a structured dataset, enabling downstream processing by machine learning applications.\n",
        "\n",
        "This notebook gives a brief overview of paperetl."
      ],
      "metadata": {
        "id": "uSRZj8RWSBVh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install dependencies\n",
        "\n",
        "Install `paperetl` and all dependencies. This step also downloads input data to process."
      ],
      "metadata": {
        "id": "0sqt7nEFSmO4"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i-DF6MYkR7zX"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install git+https://github.com/neuml/paperetl\n",
        " \n",
        "# Download NLTK data\n",
        "!python -c \"import nltk; nltk.download('punkt')\"\n",
        "\n",
        "# Download data\n",
        "!mkdir -p paperetl\n",
        "!wget -N https://github.com/neuml/paperetl/releases/download/v1.6.0/tests.tar.gz\n",
        "!tar -xvzf tests.tar.gz"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Review data\n",
        "\n",
        "Now let's take a look at the input data, which is a list of files in a directory."
      ],
      "metadata": {
        "id": "xLekHfEAUBH6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ls -l paperetl/file/data"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SQR2B_KVTlGL",
        "outputId": "1328a2f2-770c-4b0c-cd68-5a5ee1fa5c0e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "total 1692\n",
            "-rw-rw-r-- 1 1000 1000  95375 Nov  4  2020 0.xml\n",
            "-rw-rw-r-- 1 1000 1000    353 Dec  5  2021 10.csv\n",
            "-rw-rw-r-- 1 1000 1000 310066 Nov  4  2020 1.xml\n",
            "-rw-rw-r-- 1 1000 1000 349016 Nov  4  2020 2.xml\n",
            "-rw-rw-r-- 1 1000 1000 232888 Nov  4  2020 3.xml\n",
            "-rw-rw-r-- 1 1000 1000 235276 Nov  4  2020 4.xml\n",
            "-rw-rw-r-- 1 1000 1000  50414 Nov  4  2020 5.xml\n",
            "-rw-rw-r-- 1 1000 1000  92683 Nov  4  2020 6.xml\n",
            "-rw-rw-r-- 1 1000 1000 139379 Nov  4  2020 7.xml\n",
            "-rw-rw-r-- 1 1000 1000  41640 Nov  4  2020 8.xml\n",
            "-rw-rw-r-- 1 1000 1000  77557 Nov  4  2020 9.xml\n",
            "-rw-r--r-- 1 1000 1000   5364 Dec  5  2021 arxiv.xml\n",
            "-rw-r--r-- 1 1000 1000  70272 Oct  5  2021 pubmed.xml\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this example, we're only covering XML and CSV files. Processing PDF articles requires [installing GROBID](https://github.com/neuml/paperetl#additional-dependencies)."
      ],
      "metadata": {
        "id": "b3O72owsUI6K"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Process data\n",
        "\n",
        "Next, we'll run the ETL process to load the files into a SQLite articles database."
      ],
      "metadata": {
        "id": "G5bDFFCXUhGJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python -m paperetl.file paperetl/file/data paperetl/models"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1R0mP1VcUpeP",
        "outputId": "e7d6863e-92a1-4e5f-c221-34e114af02c2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Processing: paperetl/file/data/0.xml\n",
            "Processing: paperetl/file/data/1.xml\n",
            "Processing: paperetl/file/data/10.csv\n",
            "Processing: paperetl/file/data/2.xml\n",
            "Processing: paperetl/file/data/3.xml\n",
            "Processing: paperetl/file/data/4.xml\n",
            "Processing: paperetl/file/data/5.xml\n",
            "Processing: paperetl/file/data/6.xml\n",
            "Processing: paperetl/file/data/7.xml\n",
            "Processing: paperetl/file/data/8.xml\n",
            "Processing: paperetl/file/data/9.xml\n",
            "Processing: paperetl/file/data/arxiv.xml\n",
            "Processing: paperetl/file/data/pubmed.xml\n",
            "Total articles inserted: 21\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ls -l paperetl/models"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ER8QFxTZhQst",
        "outputId": "7d0189fc-8f02-468c-a090-6b2527d0b18d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "total 940\n",
            "-rw-r--r-- 1 root root 962560 Jan 23 16:29 articles.sqlite\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This ETL process took the XML and CSV files, parsed the metadata/content and loaded it all into `articles.sqlite`. "
      ],
      "metadata": {
        "id": "t-ra9JbhhZl8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Review parsed data\n",
        "\n",
        "The two main tables in `articles.sqlite` are articles and sections. \n",
        "\n",
        "- The articles table stores metadata (date, authors, publication, title...)\n",
        "- The sections table stores the article text split into sections and sentences\n",
        "\n",
        "Now let's take a look at what was loaded. "
      ],
      "metadata": {
        "id": "Bhlm_ZAgdxkW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import sqlite3\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "from IPython.display import display, HTML\n",
        "\n",
        "def execute(sql):\n",
        "  db = sqlite3.connect(\"paperetl/models/articles.sqlite\")\n",
        "  cursor = db.cursor()\n",
        "  cursor.execute(sql)\n",
        "\n",
        "  df = pd.DataFrame([list(x) for x in cursor], columns=[c[0] for c in cursor.description])\n",
        "  display(HTML(df.to_html(index=False)))\n",
        "\n",
        "# Show articles\n",
        "execute(\"SELECT * FROM articles LIMIT 5\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 432
        },
        "id": "DDEYJNQhd1Cw",
        "outputId": "d5f5503f-e2e1-4315-ce9b-43bfb166df71"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Id</th>\n",
              "      <th>Source</th>\n",
              "      <th>Published</th>\n",
              "      <th>Publication</th>\n",
              "      <th>Authors</th>\n",
              "      <th>Affiliations</th>\n",
              "      <th>Affiliation</th>\n",
              "      <th>Title</th>\n",
              "      <th>Tags</th>\n",
              "      <th>Reference</th>\n",
              "      <th>Entry</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>00398e4c637f5e5447e35e63669187f0239c0357</td>\n",
              "      <td>0.xml</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>Gibbs, Hamish; Liu, Yang; Pearson, Carl; Jarvis, Christopher; Grundy, Chris; Quilty, Billy; Diamond, Charlie; Cmmid, Lshtm; Eggo, Rosalind</td>\n",
              "      <td>Department of Infectious Disease Epidemiology, School of Hygiene and Tropical Medicine; Centre for Mathematical Modelling of Infectious Diseases, School of Hygiene and Tropical Medicine</td>\n",
              "      <td>Centre for Mathematical Modelling of Infectious Diseases, School of Hygiene and Tropical Medicine</td>\n",
              "      <td>Changing travel patterns in China during the early stages of the COVID-19 pandemic</td>\n",
              "      <td>PDF</td>\n",
              "      <td>https://doi.org/10.1038/s41467-020-18783-0</td>\n",
              "      <td>2023-01-23 00:00:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1001</td>\n",
              "      <td>datasource2</td>\n",
              "      <td>None</td>\n",
              "      <td>Test Journal2</td>\n",
              "      <td>Test Author2</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>Test Article2</td>\n",
              "      <td>None</td>\n",
              "      <td>test url2</td>\n",
              "      <td>2021-04-01 00:00:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1000</td>\n",
              "      <td>datasource</td>\n",
              "      <td>None</td>\n",
              "      <td>Test Journal</td>\n",
              "      <td>Test Author</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>Test Article</td>\n",
              "      <td>None</td>\n",
              "      <td>test url</td>\n",
              "      <td>2021-05-01 00:00:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>00c4c8c42473d25ebb38c4a8a14200c6900be2e9</td>\n",
              "      <td>1.xml</td>\n",
              "      <td>2020-01-23 00:00:00</td>\n",
              "      <td>Abouk and Heydari (2020)</td>\n",
              "      <td>Chernozhukov, Victor; Kasahara, Hiroyuki; Schrimpf, Paul; Chernozhukov, V; Kasahara, H; Schrimpf, P</td>\n",
              "      <td>Department of Economics and Center for Statistics and Data Science, MIT; School of Economics, UBC</td>\n",
              "      <td>School of Economics, UBC</td>\n",
              "      <td>1.xml</td>\n",
              "      <td>PDF</td>\n",
              "      <td>https://doi.org/10.1016/j.jeconom.2020.09.003</td>\n",
              "      <td>2023-01-23 00:00:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3d2fb136bbd9bd95f86fc49bdcf5ad08ada6913b</td>\n",
              "      <td>3.xml</td>\n",
              "      <td>2021-01-23 00:00:00</td>\n",
              "      <td>Biosensors and Bioelectronics</td>\n",
              "      <td>Yüce, Meral; Filiztekin, Elif; Gasia, Korin; Zkaya, Ö</td>\n",
              "      <td>SUNUM Nanotechnology Research and Application Centre, Sabanci University; Faculty of Engineering and Natural Sciences, Sabanci University</td>\n",
              "      <td>Faculty of Engineering and Natural Sciences, Sabanci University</td>\n",
              "      <td>COVID-19 diagnosis -A review of current methods</td>\n",
              "      <td>PDF</td>\n",
              "      <td>https://doi.org/10.1016/j.bios.2020.112752</td>\n",
              "      <td>2023-01-23 00:00:00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Show sections\n",
        "execute(\"SELECT * FROM sections LIMIT 5\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        },
        "id": "sexEVsPVf62F",
        "outputId": "2a69f83d-0aec-45f0-9292-f2c36cae04c7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Id</th>\n",
              "      <th>Article</th>\n",
              "      <th>Name</th>\n",
              "      <th>Text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>00398e4c637f5e5447e35e63669187f0239c0357</td>\n",
              "      <td>TITLE</td>\n",
              "      <td>Changing travel patterns in China during the early stages of the COVID-19 pandemic</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>00398e4c637f5e5447e35e63669187f0239c0357</td>\n",
              "      <td>None</td>\n",
              "      <td>T he COVID-19 pandemic was first identified in Wuhan, China, in late 2019, and came to prominence in January 2020, and quickly spread within the country.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>00398e4c637f5e5447e35e63669187f0239c0357</td>\n",
              "      <td>None</td>\n",
              "      <td>January is also a major holiday period in China, and the 40-day period around Lunar New Year (LNY), or Chunyun, marks the largest annual human movement in the world, with major travel flows out of large cities 1 .</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>00398e4c637f5e5447e35e63669187f0239c0357</td>\n",
              "      <td>None</td>\n",
              "      <td>The purpose of this holiday travel is often to visit family members.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>00398e4c637f5e5447e35e63669187f0239c0357</td>\n",
              "      <td>None</td>\n",
              "      <td>The temporary displacement from residential addresses as a result of this holiday travel could last one to two weeks, up to a month.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The results above show a sample of the metadata and content. "
      ],
      "metadata": {
        "id": "THtBWd5hgweB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Wrapping up\n",
        "\n",
        "This notebook gave a brief overview of paperetl. The processed data can be used for a simple query and display application. It can also feed machine learning models for more advanced use cases (see [paperai](https://github.com/neuml/paperai)). "
      ],
      "metadata": {
        "id": "SiAVyvKihD_r"
      }
    }
  ]
}