diff --git a/docs/tutorials/99_other/OEP_Research_Data_Publishing_Guidebook.ipynb b/docs/tutorials/99_other/OEP_Research_Data_Publishing_Guidebook.ipynb index 08004f0e..c0fd8aa9 100644 --- a/docs/tutorials/99_other/OEP_Research_Data_Publishing_Guidebook.ipynb +++ b/docs/tutorials/99_other/OEP_Research_Data_Publishing_Guidebook.ipynb @@ -1,225 +1,229 @@ { - "cells": [ - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\"OpenEnergy\n", - "\n", - "# Guidebook - How to Publish Your Data on the OEP\n", - "\n", - "The development of tutorials for the Open Energy Family takes place publicly in a dedicated [tutorial repository](https://github.com/OpenEnergyPlatform/tutorial).
\n", - "Please report bugs and suggestions as [new issues](https://github.com/OpenEnergyPlatform/academy/issues).
\n", - "\n", - "license: [**GNU Affero General Public License Version 3 (AGPL-3.0)**](https://github.com/openego/data_processing/blob/master/LICENSE)
\n", - "copyright: **Reiner Lemoine Institut**
\n", - "authors: **christian-rli, jh-RLI, Ludee**
\n", - "\n", - "If Jupyter Notebooks are new to you and you'd like to get an introduction, have a look at this less than 10 minute [introduction video](https://www.youtube.com/watch?v=q_BzsPxwLOE). Official installation instructions are available on [jupyter's readthedocs page](https://jupyter.readthedocs.io/en/latest/install.html)." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Introduction\n", - "\n", - "If your goal is to publish your energy system research data and you're wondering just about what that entails, this is the right resource for you. This tutorial will guide you through the steps needed for publishing your data on the OEDB and supplying it with proper metadata. The process has the following steps:\n", - "\n", - "1 Creation of metadata
\n", - "2 Initiation of metadata review
\n", - "3 Uploading data and metadata
\n", - "\n", - "This document describes each of these steps, refers to resources relevant for the tasks and ends by introducing the tools needed to upload your data. If your system is already set up, you can jump to an [upload tutorial](https://openenergyplatform.github.io/academy/tutorials/02_oep_client/02_client_python_upload/). If all of this is new to you, we recommend reading on. By uploading your data in this process described below, you will ensure that your data is:
\n", - "\n", - "- public\n", - "- findable\n", - "- structured\n", - "- licensed openly\n", - "- supplied with sources\n", - "- reviewed" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1 Creation of metadata\n", - "\n", - "Metadata provide information about other data. By creating metadata for your dataset, you are making sure, that it is easier to find and to understand, it's supplied with its sources and its structure is well described. The OEP uses [oemetadata](https://github.com/OpenEnergyPlatform/oemetadata) as a standard, which was developed specifically for energy research modelling. It is based on json and compatible to the [datapackage standard](https://specs.frictionlessdata.io/data-package/#language). An example oemetadata file in the current version is available in the [oemetadata repository](https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/latest/example.json). \n", - "\n", - "Filling out a string for the first time might take a while. Once you're familiar with what the different fields stand for, it'll become much faster. You can start with an empty [template string](https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/latest/template.json) and just work your way from top to bottom. A [description for every key](https://github.com/OpenEnergyPlatform/oemetadata/blob/develop/metadata/latest/metadata_key_description.md) of the string with example entries is available on the oemetadata repository.\n", - "\n", - "Take care when creating your resources that your data types correspond to [postgresql datatypes](https://www.postgresql.org/docs/14/datatype.html), because these definitions will be used when uploading your data. Most likely, you will mostly use these types:\n", - "\n", - "- For strings or any other kind of textual data, use \"text\"\n", - "- For numbers without fractional component, use \"integer\"\n", - "- For number _with_ a fractional component, use \"float\"\n", - "\n", - "
\n", - "When selecting a license for your data, make sure that it is an open license! \n", - "
\n", - "\n", - "Proprietary data is not currently allowed on the OEP. We recommend using [ODbL-1.0](https://opendatacommons.org/licenses/odbl/1.0/)." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2 Initiation of metadata review: \n", - "\n", - "Once your metadata string is ready, send it in for a review! The review process takes place publicly on GitHub. Create a [new issue](https://github.com/OpenEnergyPlatform/data-preprocessing/issues/new/choose) in the data-preprocessing repository and follow the workflow described in the issue. \n", - "\n", - "This includes uploading your metadata string to the folder [data-review](https://github.com/OpenEnergyPlatform/data-preprocessing/tree/master/data-review) in a new branch and referencing the issue number. If you're unable or don't know how to do this, just attach your string to a comment in the issue you created and get in contact with a reviewer. The reviewer can help you with this. \n", - "\n", - "There is a [reference manual for reviewers](https://github.com/OpenEnergyPlatform/data-preprocessing/blob/master/data-review/manual/review_manual.md) which will be followed to complete the review process. Once the review is done, the new branch will be merged with the master branch and if your data is on the OEP, a reviewed badge will eventually show next to it (this still needs to be implemented).\n", - "\n", - "
\n", - "You don't need to wait for the review to be done in order to upload your data or metadata to the OEDB!\n", - "
\n", - "\n", - "So you may just continue with the next step." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3 Uploading data\n", - "\n", - "### 3.1 Requirements\n", - "\n", - "The OEDB is a [PostgreSQL](https://www.postgresql.org/) database with a public, RESTful API. The OEP functions as an interface to it. Downloads from the OEDB can be carried out by anyone without registration. However, in order to upload, you will need a user account on the OpenEnergyPlatform. There you will be presented with a token, which allows you to upload data via the public API. If you haven't got one already, [create a new account](https://openenergyplatform.org/user/register). To retrieve your token, do the following:\n", - "\n", - "\n", - "#### 3.1.1. Login\n", - "\n", - "Click on the login button and sign in on the [OEP](https://openenergyplatform.org/user/login/?next=/).\n", - "\n", - "\"OpenEnergy

\n", - "\n", - "\n", - "#### 3.1.2. Copy your token\n", - "\n", - "Click on your Profile Name to see your information. To view your token, click on \"Show token\" \n", - "\n", - "\"OpenEnergy " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.2. Using the API\n", - "\n", - "You can access the API with any tool that can send HTTP-requests. That means you can technically use the address bar of your browser to access data on the OEDB. \n", - "\n", - "For example the following links will return a json string of the columns and the rows respectively of the wind_turbine_library, a dataset published on the OEP:\n", - "\n", - "https://openenergyplatform.org/api/v0/schema/supply/tables/wind_turbine_library/columns/
\n", - "https://openenergyplatform.org/api/v0/schema/supply/tables/wind_turbine_library/rows/\n", - "\n", - "Extending on this browser based approach to more complex tasks would be unfeasible however. You would also need to configure your browser to send your token in the header of every request in order to upload any data. \n", - "\n", - "Using the [API documentation](https://oep-data-interface.readthedocs.io/en/latest/api/how_to.html#authenticate) for working with the OEDB, users in the community can build their own tools to access the OEP. There is already a suite of small tools, mainly written in python, that aim to facilitate access. There is a hands on tutorial that will guide you through the process of uploading example data and metadata from files on your computer all the way into a publicly visible place on the OEP. The structure of the same tutorial is available as a template for you to fill in with your own data. These tutorials are going to use a range of tools. A brief description of these tools and how to set them up on your system will make up the rest of this guidebook." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "##### oedialect\n", - "\n", - "[SQLAlchemy](https://www.sqlalchemy.org/) is the python toolkit to work with PostgreSQL databases. SQLAlchemy internally uses so called \"dialects\" to provide a consistent interface to different database drivers. The [oedialect](https://github.com/OpenEnergyPlatform/oedialect) supplies your SQLAlchemy installation with a dialect using the REST-API of the Open Energy Platform (OEP). In short, the oedialect allows you to use SQLAlchemy to down- and upload data to the OEP, which is helpful if you're using python as a means to handy your data. [Installation instructions for SQLAlchemy](https://www.liquidweb.com/kb/how-to-install-sqlalchemy/) for different operating systems are available on liquidweb. To install the oedialect, you can use pip:\n", - "\n", - "`pip install oedialect`" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "##### oem2orm\n", - "\n", - "[oem2orm](https://github.com/OpenEnergyPlatform/data-preprocessing/tree/feature/oep-upload-oem2orm/data-import/oep-upload) stands for oemetadata to object relational mapping. It's a convenience tool written in python. It can create an engine to connect you to the OEP with your username and token and has a function, which reads all the metadata metadata strings in a folder and based on the contained information creates tables on the OEP using the [oedialect](https://github.com/OpenEnergyPlatform/oedialect). You can install it using pip:
\n", - "\n", - "`pip install oem2orm`" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "##### oep-client\n", - "The [oep-client](https://github.com/wingechr/oep-client) is a command line tool written in python. It offers a range of functionalities, including, downloading data and metadata, creating a table, uploading data, updating a table's metadata, and deleting tables that you created. As it is written in python it's also possible to import its functions to your python project. You can install the oep-client using pip:\n", - "\n", - "`python3 -m pip install --upgrade oep-client`\n", - " " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "##### saio\n", - "[saio](https://github.com/coroa/saio) is another convenience tool. It allows for easy access to the data on the OEDB by allowing you to autoload table definitions. It also provides a helper function to read an sqlalchemy.orm.query.Query into a (Geo)Pandas dataframe. Install saio via pip:\n", - "\n", - "`pip install saio`\n", - " " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "##### pandas\n", - "[pandas](https://pandas.pydata.org/pandas-docs/stable/index.html) is an open source library providing easy-to-use data structures and data analysis tools for the python. Pandas is often used in research to handle data and it is also used in several descriptions to read data into python before uploading it to the OEDB. Use pip to install pandas:\n", - "\n", - "`pip install pandas`\n", - " " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "##### requests\n", - "\n", - "[Requests](https://pypi.org/project/requests/) allows you to send HTTP requests easily. It is used in tutorials that directly use the API of the OEP to make a connection. Use pip to install requests:\n", - "\n", - "`pip install requests`\n", - " " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.5" - } + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"OpenEnergy\n", + "\n", + "# Guidebook - How to Publish Your Data on the OEP\n", + "\n", + "The development of tutorials for the Open Energy Family takes place publicly in a dedicated [tutorial repository](https://github.com/OpenEnergyPlatform/tutorial).
\n", + "Please report bugs and suggestions as [new issues](https://github.com/OpenEnergyPlatform/academy/issues).
\n", + "\n", + "license: [**GNU Affero General Public License Version 3 (AGPL-3.0)**](https://github.com/openego/data_processing/blob/master/LICENSE)
\n", + "copyright: **Reiner Lemoine Institut**
\n", + "authors: **christian-rli, jh-RLI, Ludee**
\n", + "\n", + "If Jupyter Notebooks are new to you and you'd like to get an introduction, have a look at this less than 10 minute [introduction video](https://www.youtube.com/watch?v=q_BzsPxwLOE). Official installation instructions are available on [jupyter's readthedocs page](https://jupyter.readthedocs.io/en/latest/install.html)." + ] }, - "nbformat": 4, - "nbformat_minor": 1 + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "If your goal is to publish your energy system research data and you're wondering just about what that entails, this is the right resource for you. This tutorial will guide you through the steps needed for publishing your data on the OEDB and supplying it with proper metadata. The process has the following steps:\n", + "\n", + "1 Creation of metadata
\n", + "2 Initiation of metadata review
\n", + "3 Uploading data and metadata
\n", + "\n", + "This document describes each of these steps, refers to resources relevant for the tasks and ends by introducing the tools needed to upload your data. If your system is already set up, you can jump to an [upload tutorial](https://openenergyplatform.github.io/academy/tutorials/02_oep_client/02_client_python_upload/). If all of this is new to you, we recommend reading on. By uploading your data in this process described below, you will ensure that your data is:
\n", + "\n", + "- public\n", + "- findable\n", + "- structured\n", + "- licensed openly\n", + "- supplied with sources\n", + "- reviewed" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1 Creation of metadata\n", + "\n", + "Metadata provide information about other data. By creating metadata for your dataset, you are making sure, that it is easier to find and to understand, it's supplied with its sources and its structure is well described. The OEP uses [oemetadata](https://github.com/OpenEnergyPlatform/oemetadata) as a standard, which was developed specifically for energy research modelling. It is based on json and compatible to the [datapackage standard](https://specs.frictionlessdata.io/data-package/#language). An example oemetadata file in the current version is available in the [oemetadata repository](https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/latest/example.json). \n", + "\n", + "Filling out a string for the first time might take a while. Once you're familiar with what the different fields stand for, it'll become much faster. You can start with an empty [template string](https://github.com/OpenEnergyPlatform/oemetadata/blob/master/metadata/latest/template.json) and just work your way from top to bottom. A [description for every key](https://github.com/OpenEnergyPlatform/oemetadata/blob/develop/metadata/latest/metadata_key_description.md) of the string with example entries is available on the oemetadata repository.\n", + "\n", + "Take care when creating your resources that your data types correspond to [postgresql datatypes](https://www.postgresql.org/docs/14/datatype.html), because these definitions will be used when uploading your data. Most likely, you will mostly use these types:\n", + "\n", + "- For strings or any other kind of textual data, use \"text\"\n", + "- For numbers without fractional component, use \"integer\"\n", + "- For number _with_ a fractional component, use \"float\"\n", + "\n", + "
\n", + "When selecting a license for your data, make sure that it is an open license! \n", + "
\n", + "\n", + "Proprietary data is not currently allowed on the OEP. We recommend using [ODbL-1.0](https://opendatacommons.org/licenses/odbl/1.0/)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2 Initiation of metadata review: \n", + "\n", + "Once your metadata string is ready, send it in for a review! The review process takes place publicly on GitHub. Create a [new issue](https://github.com/OpenEnergyPlatform/data-preprocessing/issues/new/choose) in the data-preprocessing repository and follow the workflow described in the issue. \n", + "\n", + "This includes uploading your metadata string to the folder [data-review](https://github.com/OpenEnergyPlatform/data-preprocessing/tree/master/data-review) in a new branch and referencing the issue number. If you're unable or don't know how to do this, just attach your string to a comment in the issue you created and get in contact with a reviewer. The reviewer can help you with this. \n", + "\n", + "There is a [reference manual for reviewers](https://github.com/OpenEnergyPlatform/data-preprocessing/blob/master/data-review/manual/review_manual.md) which will be followed to complete the review process. Once the review is done, the new branch will be merged with the master branch and if your data is on the OEP, a reviewed badge will eventually show next to it (this still needs to be implemented).\n", + "\n", + "
\n", + "You don't need to wait for the review to be done in order to upload your data or metadata to the OEDB!\n", + "
\n", + "\n", + "So you may just continue with the next step." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3 Uploading data\n", + "\n", + "### 3.1 Requirements\n", + "\n", + "The OEDB is a [PostgreSQL](https://www.postgresql.org/) database with a public, RESTful API. The OEP functions as an interface to it. Downloads from the OEDB can be carried out by anyone without registration. However, in order to upload, you will need a user account on the OpenEnergyPlatform. There you will be presented with a token, which allows you to upload data via the public API. If you haven't got one already, [create a new account](https://openenergyplatform.org/user/register). To retrieve your token, do the following:\n", + "\n", + "\n", + "#### 3.1.1. Login\n", + "\n", + "Click on the login button and sign in on the [OEP](https://openenergyplatform.org/user/login/?next=/).\n", + "\n", + "\"OpenEnergy

" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### 3.1.2. Copy your token\n", + "\n", + "After logging in, click on your profile name to see your information. To view your token, click on \"Show token\" \n", + "\n", + "\"OpenEnergy " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.2. Using the API\n", + "\n", + "You can access the API with any tool that can send HTTP-requests. That means you can technically use the address bar of your browser to access data on the OEDB. \n", + "\n", + "For example the following links will return a json string of the columns and the rows respectively of the wind_turbine_library, a dataset published on the OEP:\n", + "\n", + "https://openenergyplatform.org/api/v0/schema/supply/tables/wind_turbine_library/columns/
\n", + "https://openenergyplatform.org/api/v0/schema/supply/tables/wind_turbine_library/rows/\n", + "\n", + "Extending on this browser based approach to more complex tasks would be unfeasible however. You would also need to configure your browser to send your token in the header of every request in order to upload any data. \n", + "\n", + "Using the [API documentation](https://oep-data-interface.readthedocs.io/en/latest/api/how_to.html#authenticate) for working with the OEDB, users in the community can build their own tools to access the OEP. There is already a suite of small tools, mainly written in python, that aim to facilitate access. There is a hands on tutorial that will guide you through the process of uploading example data and metadata from files on your computer all the way into a publicly visible place on the OEP. The structure of the same tutorial is available as a template for you to fill in with your own data. These tutorials are going to use a range of tools. A brief description of these tools and how to set them up on your system will make up the rest of this guidebook." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##### oedialect\n", + "\n", + "[SQLAlchemy](https://www.sqlalchemy.org/) is the python toolkit to work with PostgreSQL databases. SQLAlchemy internally uses so called \"dialects\" to provide a consistent interface to different database drivers. The [oedialect](https://github.com/OpenEnergyPlatform/oedialect) supplies your SQLAlchemy installation with a dialect using the REST-API of the Open Energy Platform (OEP). In short, the oedialect allows you to use SQLAlchemy to down- and upload data to the OEP, which is helpful if you're using python as a means to handy your data. [Installation instructions for SQLAlchemy](https://www.liquidweb.com/kb/how-to-install-sqlalchemy/) for different operating systems are available on liquidweb. To install the oedialect, you can use pip:\n", + "\n", + "`pip install oedialect`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##### oem2orm\n", + "\n", + "[oem2orm](https://github.com/OpenEnergyPlatform/data-preprocessing/tree/feature/oep-upload-oem2orm/data-import/oep-upload) stands for oemetadata to object relational mapping. It's a convenience tool written in python. It can create an engine to connect you to the OEP with your username and token and has a function, which reads all the metadata metadata strings in a folder and based on the contained information creates tables on the OEP using the [oedialect](https://github.com/OpenEnergyPlatform/oedialect). You can install it using pip:
\n", + "\n", + "`pip install oem2orm`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##### oep-client\n", + "The [oep-client](https://github.com/wingechr/oep-client) is a command line tool written in python. It offers a range of functionalities, including, downloading data and metadata, creating a table, uploading data, updating a table's metadata, and deleting tables that you created. As it is written in python it's also possible to import its functions to your python project. You can install the oep-client using pip:\n", + "\n", + "`python3 -m pip install --upgrade oep-client`\n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##### saio\n", + "[saio](https://github.com/coroa/saio) is another convenience tool. It allows for easy access to the data on the OEDB by allowing you to autoload table definitions. It also provides a helper function to read an sqlalchemy.orm.query.Query into a (Geo)Pandas dataframe. Install saio via pip:\n", + "\n", + "`pip install saio`\n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##### pandas\n", + "[pandas](https://pandas.pydata.org/pandas-docs/stable/index.html) is an open source library providing easy-to-use data structures and data analysis tools for the python. Pandas is often used in research to handle data and it is also used in several descriptions to read data into python before uploading it to the OEDB. Use pip to install pandas:\n", + "\n", + "`pip install pandas`\n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##### requests\n", + "\n", + "[Requests](https://pypi.org/project/requests/) allows you to send HTTP requests easily. It is used in tutorials that directly use the API of the OEP to make a connection. Use pip to install requests:\n", + "\n", + "`pip install requests`\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.12" + } + }, + "nbformat": 4, + "nbformat_minor": 4 } diff --git a/docs/tutorials/99_other/images/login.png b/docs/tutorials/99_other/images/login.png new file mode 100644 index 00000000..2b98b6c8 Binary files /dev/null and b/docs/tutorials/99_other/images/login.png differ diff --git a/docs/tutorials/99_other/images/token.png b/docs/tutorials/99_other/images/token.png new file mode 100644 index 00000000..dd596d25 Binary files /dev/null and b/docs/tutorials/99_other/images/token.png differ