Merge branch 'dev' into features/#392-include-powerplants

CHANGELOG.rst

@@ -137,6 +137,12 @@ Added
 * Extend zensus by a combined table with all cells where
   there's either building, apartment or population data
   `#359 <https://github.com/openego/eGon-data/issues/359>`_
+* Add example metadata for OSM, VG250 and Zensus VG250.
+  Add metadata templates for licences, context and some helper
+  functions. Extend docs on how to create metadata for tables.
+  `#139 <https://github.com/openego/eGon-data/issues/139>`_
+* Integrate DSM potentials for CTS and industry
+  `#259 <https://github.com/openego/eGon-data/issues/259>`_
 * Include allocation of conventional (non CHP) power plants
   `#392 <https://github.com/openego/eGon-data/issues/392>`_
 
@@ -206,15 +212,20 @@ Changed
   `#397 <https://github.com/openego/eGon-data/issues/397>`_
 * Rename columns gid to id
   `#169 <https://github.com/openego/eGon-data/issues/169>`_
+* Remove upper version limit of pandas
+  `#383 <https://github.com/openego/eGon-data/issues/383>`_
 * Use random seed from CLI parameters for CHP and society prognosis functions
   `#351 <https://github.com/openego/eGon-data/issues/351>`_
 * Changed demand.egon_schmidt_industrial_sites - table and merged table (industrial_sites)
   `#423 <https://github.com/openego/eGon-data/issues/423>`_
+* Replace 'gas' carrier with 'CH4' and 'H2' carriers
+  `#436 <https://github.com/openego/eGon-data/issues/436>`_
 * Adjust file path for industrial sites import
   `#397 <https://github.com/openego/eGon-data/issues/418>`_
 * Rename columns subst_id to bus_id
   `#335 <https://github.com/openego/eGon-data/issues/335>`_
 
+
 Bug fixes
 ---------
 * Heat demand data import
@@ -272,3 +283,8 @@ Bug fixes
   `#432 <https://github.com/openego/eGon-data/issues/432>`_
 * Add curl to prerequisites in the docs
   `#440 <https://github.com/openego/eGon-data/issues/440>`_
+* Replace NAN by 0 to avoid empty p_set column in DB
+  `#414 <https://github.com/openego/eGon-data/issues/414>`_
+* Exchange bus 0 and bus 1 in Power-to-H2 links
+  `#458 <https://github.com/openego/eGon-data/issues/458>`_
+

CONTRIBUTING.rst

@@ -275,6 +275,105 @@ be saved locally, please use `CWD` to store the data. This is achieved by using
   filepath = Path(".") / "filename.csv"
   urlretrieve("https://url/to/file", filepath)
 
+Add metadata
+------------
+
+Add a metadata for every dataset you create for describing data with
+machine-readable information. Adhere to the OEP Metadata v1.4.1, you can
+follow
+`the example <https://github.com/OpenEnergyPlatform/oemetadata/blob/develop/metadata/latest/example.json>`_
+to understand how the fields are used. Field are described in detail in the
+`Open Energy Metadata Description`_.
+
+You can obtain the metadata string from a table you created in SQL via
+
+.. code-block:: sql
+
+  SELECT obj_description('<SCHEMA>.<TABLE>'::regclass);
+
+Alternatively, you can write the table comment directly to a JSON file by
+
+.. code-block:: bash
+
+  psql -h <HOST> -p <PORT> -d <DB> -U <USER> -c "\COPY (SELECT obj_description('<SCHEMA>.<TABLE>'::regclass)) TO '/PATH/TO/FILE.json';"
+
+For bulk export of all DB's table comments you can use `this script
+<https://gist.github.com/nesnoj/86145999eca8182f43c2bca36bcc984f>`_.
+Please verify that your metadata string is in compliance with the OEP Metadata
+standard version 1.4.1 using the `OMI tool
+<https://github.com/OpenEnergyPlatform/omi>`_ (tool is shipped with eGon-data):
+
+.. code-block:: bash
+
+  omi translate -f oep-v1.4 metadata_file.json
+
+If your metadata string is correct, OMI puts the keys in the correct order and
+prints the full string (use `-o` option for export).
+
+You may omit the fields `id` and `publicationDate` in your string as it will be
+automatically set at the end of the pipeline but you're required to set them to
+some value for a complete validation with OMI. For datasets published on the
+OEP `id` will be the URL which points to the table, it will follow the pattern
+`https://openenergy-platform.org/dataedit/view/SCHEMA/TABLE`.
+
+For previous discussions on metadata, you may want to check
+`PR 176 <https://github.com/openego/eGon-data/pull/176>`_.
+
+Helpers
+^^^^^^^
+
+There are some **licence templates** provided in :py:mod:`egon.data.metadata`
+you can make use of for fields 11.4 and 12 of the
+`Open Energy Metadata Description`_. Also, there's a template for the
+**metaMetadata** (field 16).
+
+There are some functions to quickly generate a template for the
+**resource fields** (field 14.6.1 in `Open Energy Metadata Description`_) from
+a SQLA table class or a DB table. This might be especially helpful if your
+table has plenty of columns.
+
+* From SQLA table class:
+  :py:func:`egon.data.metadata.generate_resource_fields_from_sqla_model`
+* From database table:
+  :py:func:`egon.data.metadata.generate_resource_fields_from_db_table`
+
+Sources
+^^^^^^^
+
+The **sources** (field 11) are the most important parts of the metadata which
+need to be filled manually. You may also add references to tables in eGon-data
+(e.g. from an upstream task) so you don't have to list all original sources
+again. Make sure you include all upstream attribution requirements.
+
+The following example uses various input datasets whose attribution must be
+retained:
+
+.. code-block:: python
+
+  "sources": [
+      {
+          "title": "eGo^n - Medium voltage grid districts",
+          "description": (
+              "Medium-voltage grid districts describe the area supplied by "
+              "one MV grid. Medium-voltage grid districts are defined by one "
+              "polygon that represents the supply area. Each MV grid district "
+              "is connected to the HV grid via a single substation."
+          ),
+          "path": "https://openenergy-platform.org/dataedit/view/"
+                  "grid/egon_mv_grid_district", # "id" in the source dataset
+          "licenses": [
+              license_odbl(attribution=
+                  "© OpenStreetMap contributors, 2021; "
+                  "© Statistische Ämter des Bundes und der Länder, 2014; "
+                  "© Statistisches Bundesamt, Wiesbaden 2015; "
+                  "(Daten verändert)"
+              )
+          ]
+      },
+      # more sources...
+  ]
+
+.. _Open Energy Metadata Description: https://github.com/OpenEnergyPlatform/oemetadata/blob/develop/metadata/v141/metadata_key_description.md
 
 Adjusting test mode data
 ------------------------
@@ -301,7 +400,7 @@ How to document Python scripts
 
 Use docstrings to document your Python code. Note that PEP 8 also
 contains a `section <PEP8-docstrings_>`_ on docstrings and that there is
-a whole `PEP <PEP257_>`_ dedicated to docstring convetions. Try to
+a whole `PEP <PEP257_>`_ dedicated to docstring conventions. Try to
 adhere to both of them.
 Additionally every Python script needs to contain a header describing
 the general functionality and objective and including information on

setup.py

@@ -93,7 +93,7 @@ def read(*names, **kwargs):
         "netcdf4",
         "oedialect==0.0.8",
         "openpyxl",
-        "pandas>1.2.0,<1.3.0",
+        "pandas>1.2.0",
         "psycopg2",
         "pyaml",
         "rasterio",
@@ -102,6 +102,7 @@ def read(*names, **kwargs):
         "xarray",
         "xlrd",
         "rioxarray",
+        "omi"
     ],
     extras_require={
         "dev": ["black", "flake8", "isort>=5", "pre-commit", "pytest", "tox"]