Merge branch 'dev' into features/#148-drop-python36-support

CHANGELOG.rst

@@ -45,6 +45,11 @@ Added
   `#9 <https://github.com/openego/eGon-data/issues/9>`_
 * RE potential areas data import
   `#124 <https://github.com/openego/eGon-data/issues/124>`_
+* Heat demand data import
+  `#101 <https://github.com/openego/eGon-data/issues/101>`_
+* Demographic change integration
+  `#47 <https://github.com/openego/eGon-data/issues/47>`_
+
 
 Changed
 -------
@@ -53,8 +58,11 @@ Changed
   `#20 <https://github.com/openego/eGon-data/issues/20>`_
 * Switch from Travis to GitHub actions for CI jobs
   `#92 <https://github.com/openego/eGon-data/issues/92>`_
-  * Rename columns to id and zensus_population_id in zensus tables
+* Rename columns to id and zensus_population_id in zensus tables
   `#140 <https://github.com/openego/eGon-data/issues/140>`_
+* Revise docs CONTRIBUTING section and in particular PR guidelines
+  `#88 <https://github.com/openego/eGon-data/issues/88>`_ and
+  `#145 <https://github.com/openego/eGon-data/issues/145>`_
 
 Bug fixes
 ---------

CONTRIBUTING.rst

@@ -6,50 +6,12 @@ Contributions are welcome, and they are greatly appreciated! Every
 little bit helps, and credit will always be given.
 
 
-Extending the data workflow
-===========================
-
-
-Adjusting test mode data
-------------------------
-
-When integrating new data or data processing scripts, make sure the
-:ref:`Test mode` still works correctly on a limited subset of data.
-In particular, if a new external data sources gets integrated make sure the
-data gets cut to the region of the test mode.
-
-
-Bug reports
-===========
-
-When `reporting a bug <https://github.com/openego/eGon-data/issues>`_ please include:
-
-    * Your operating system name and version.
-    * Any details about your local setup that might be helpful in troubleshooting.
-    * Detailed steps to reproduce the bug.
-
-Documentation improvements
-==========================
-
-eGo^N Data could always use more documentation, whether as part of the
-official eGo^N Data docs, in docstrings, or even on the web in blog posts,
-articles, and such.
-
-Feature requests and feedback
-=============================
-
-The best way to send feedback is to file an issue at https://github.com/openego/eGon-data/issues.
-
-If you are proposing a feature:
-
-* Explain in detail how it would work.
-* Keep the scope as narrow as possible, to make it easier to implement.
-* Remember that this is a volunteer-driven project, and that code contributions are welcome :)
-
 Development
 ===========
 
-To set up `eGon-data` for local development:
+To set up `eGon-data` for local development (points (1) and (2) are only
+relevant for people who are not members of the organization
+`openego <https://github.com/openego>`_):
 
 1. Fork `eGon-data <https://github.com/openego/eGon-data>`_
    (look for the "Fork" button).
@@ -61,11 +23,16 @@ To set up `eGon-data` for local development:
 
     git checkout -b name-of-your-bugfix-or-feature
 
+   Consider to use the following pattern for naming branches
+   `<features/fixes>/#<issue-number>-<minimal-description-as-name>`.
    Now you can make your changes locally.
 
-4. When you're done making changes run all the checks and docs builder with `tox <https://tox.readthedocs.io/en/latest/install.html>`_ one command::
+4. When you're done making changes run all the checks and docs builder with
+`tox <https://tox.readthedocs.io/en/latest/install.html>`_ one command:
+
+   .. code-block:: bash
 
-    tox
+      tox
 
 5. Commit your changes and push your branch to GitHub::
 
@@ -75,31 +42,125 @@ To set up `eGon-data` for local development:
 
 6. Submit a pull request through the GitHub website.
 
+Tips
+^^^^
+
+To run a subset of tests::
+
+    tox -e envname -- pytest -k test_myfeature
+
+To run all the test environments in *parallel*::
+
+    tox -p auto
+
+
 Pull Request Guidelines
 -----------------------
 
-If you need some code review or feedback while you're developing the code just make the pull request.
+We use pull requests (PR) to integrate code changes from branches.
+PRs always need to be reviewed (exception proves the rule!). Therefore, ask
+one of the other developers for reviewing your changes. Once approved, the PR
+can be merged. Please delete the branch after merging.
 
-For merging, you should:
+Before requesting a review, please
 
 1. Include passing tests (run ``tox``) [1]_.
-2. Update documentation when there's new API, functionality etc.
-3. Add a note to ``CHANGELOG.rst`` about the changes.
-4. Add yourself to ``AUTHORS.rst``.
+2. Let the workflow run in :ref:`Test mode` once from scratch to verify
+   successful execution
+3. Update documentation when there's new API, functionality etc.
+4. Add a note to ``CHANGELOG.rst`` about the changes.
+5. Add yourself to ``AUTHORS.rst``.
+
+When requesting reviews, please keep in mind it might be a significant effort
+to review the PR. Try to make it easier for them and keep the overall effort
+as low as possible. Therefore,
+
+* asking for reviewing specific aspects helps reviewers a lot to focus on the
+  relevant parts
+* when multiple people are asked for a review it should be avoided that they
+  check/test the same things. Be even more specific what you expect from
+  someone in particular.
+
+
+What needs to be reviewed?
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Things that definitely should be checked during a review of a PR:
+
+* *Is the code working?* The contributor should already have made sure that
+  this is the case. Either by automated test or manual execution.
+* *Is the data correct?* Verifying that newly integrated and processed data
+  is correct is usually not possible during reviewing a PR. If it is necessary,
+  please ask the reviewer specifically for this.
+* *Do tests pass?* See automatic checks.
+* *Is the documentation up-to-date?* Please check this.
+* *Was* ``CHANGELOG.rst`` *updated accordingly?* Should be the case, please
+  verify.
+* *Is metadata complete and correct (in case of data integration)?* Please
+  verify. In case of a pending metadata creation make sure an appropriate issue is filed. 
+
+
+Additional case-dependent aspects make sense.
+
+
 
 .. [1] If you don't have all the necessary python versions available locally you can rely on
        `CI on GitHub actions <https://github.com/openego/eGon-data/actions?query=workflow%3A%22Tests%2C+code+style+%26+coverage%22>`_
        - it will run the tests for each change you add in the pull request.
 
        It will be slower though ...
 
-Tips
-----
 
-To run a subset of tests::
+Extending the data workflow
+---------------------------
 
-    tox -e envname -- pytest -k test_myfeature
 
-To run all the test environments in *parallel*::
+Adjusting test mode data
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+When integrating new data or data processing scripts, make sure the
+:ref:`Test mode` still works correctly on a limited subset of data.
+In particular, if a new external data sources gets integrated make sure the
+data gets cut to the region of the test mode.
+
+
+Bug reports and feature requests
+================================
+
+When `reporting a bug <https://github.com/openego/eGon-data/issues>`_ please include:
+
+* Your operating system name and version.
+* Any details about your local setup that might be helpful in troubleshooting.
+* Detailed steps to reproduce the bug.
+
+If you are proposing a feature:
+
+* Explain in detail how it would work.
+* Keep the scope as narrow as possible, to make it easier to implement.
+* Remember that this is a volunteer-driven project, and that code contributions are welcome :)
+
+Documentation improvements
+==========================
+
+eGo^N Data could always use more documentation, whether as part of the
+official eGo^N Data docs, in docstrings, or even on the web in blog posts,
+articles, and such.
+Please make sure you change the documentation along with code changes.
+
+The changes of the documentation in a feature branch get visible once a pull
+request is opened.
+
+You can build the documentation locally with (executed in the repos root
+directory)
+
+.. code-block:: bash
+
+   sphinx-build -E -a docs docs/_build/
+
+Eventually, you might need to install additional dependencies for building the
+documenmtation:
+
+.. code-block:: bash
+
+   pip install -r docs/requirements.txt
 
-    tox -p auto

README.rst

@@ -106,6 +106,10 @@ packages are required too. Right now these are:
   On recent Ubuntu version you can install it via
   :code:`sudo apt install osm2pgsql`.
 
+* `postgis <https://postgis.net/>`_
+  On recent Ubuntu version you can install it via
+  :code:`sudo apt install postgis`.
+
 
 Installation
 ============

setup.py

@@ -81,16 +81,18 @@ def read(*names, **kwargs):
     ],
     python_requires=">=3.7.*",
     install_requires=[
+        # eg: 'aspectlib==1.1.1', 'six>=1.7',
         "apache-airflow<2.0",
         "click",
+        "disaggregator @ git+https://github.com/openego/disaggregator.git@features/pip_install",
+        "geopandas",
         "importlib-resources",
         "oedialect==0.0.8",
         "pyaml",
         "psycopg2",
-        "sqlalchemy",
-        "geopandas",
-        "disaggregator @ git+https://github.com/openego/disaggregator.git@features/pip_install"
-        # eg: 'aspectlib==1.1.1', 'six>=1.7',
+        "rasterio",
+        "rtree",
+        "sqlalchemy"
     ],
     extras_require={
         "dev": ["black", "flake8", "isort>=5", "pre-commit", "pytest", "tox"]

src/egon/data/airflow/dags/pipeline.py

@@ -13,12 +13,14 @@
 import egon.data.importing.demandregio as import_dr
 import egon.data.processing.openstreetmap as process_osm
 import egon.data.importing.zensus as import_zs
+import egon.data.processing.zensus as process_zs
 import egon.data.processing.power_plants as power_plants
 import egon.data.importing.nep_input_data as nep_input
 import egon.data.importing.etrago as etrago
 import egon.data.importing.mastr as mastr
 import egon.data.processing.substation as substation
 import egon.data.importing.re_potential_areas as re_potential_areas
+import egon.data.importing.heat_demand_data as import_hd
 
 # Prepare connection to db for operators
 airflow_db_connection()
@@ -129,6 +131,38 @@
     )
     vg250_clean_and_prepare >> demandregio_import
 
+    # Society prognosis
+    prognosis_tables = PythonOperator(
+        task_id="create-prognosis-tables",
+        python_callable=process_zs.create_tables
+    )
+
+    map_zensus_nuts3 = PythonOperator(
+        task_id="map-zensus-to-nuts3",
+        python_callable=process_zs.map_zensus_nuts3
+    )
+
+    setup >> prognosis_tables >> map_zensus_nuts3
+    vg250_clean_and_prepare >> map_zensus_nuts3
+    population_import >> map_zensus_nuts3
+
+    population_prognosis = PythonOperator(
+        task_id="zensus-population-prognosis",
+        python_callable=process_zs.population_prognosis_to_zensus
+    )
+
+    map_zensus_nuts3 >> population_prognosis
+    demandregio_import >> population_prognosis
+
+    household_prognosis = PythonOperator(
+        task_id="zensus-household-prognosis",
+        python_callable=process_zs.household_prognosis_to_zensus
+    )
+
+    map_zensus_nuts3 >> household_prognosis
+    demandregio_import >> household_prognosis
+    zensus_misc_import >> household_prognosis
+
     # Power plant setup
     power_plant_tables = PythonOperator(
         task_id="create-power-plant-tables",
@@ -193,7 +227,6 @@
     substation_functions >> hvmv_substation_extraction
     substation_functions >> ehv_substation_extraction
 
-
     # Import potential areas for wind onshore and ground-mounted PV
     download_re_potential_areas = PythonOperator(
         task_id="download_re_potential_area_data",
@@ -210,3 +243,11 @@
     )
     setup >> download_re_potential_areas >> create_re_potential_areas_tables
     create_re_potential_areas_tables >> insert_re_potential_areas
+
+    # Future heat demand calculation based on Peta5_0_1 data
+    heat_demand_import = PythonOperator(
+        task_id="import-heat-demand",
+        python_callable=import_hd.future_heat_demand_data_import
+    )
+    vg250_clean_and_prepare >> heat_demand_import
+    population_import >> heat_demand_import

src/egon/data/airflow/entrypoints/create_postgis_extension.sql

@@ -1,2 +1,3 @@
 CREATE EXTENSION hstore;
 CREATE EXTENSION postgis;
+CREATE EXTENSION postgis_raster;