Merge branch 'dev' into features/#127-add-hydro-biomass-plants-eGon2035

.github/workflows/build.yml

@@ -26,7 +26,6 @@ jobs:
     strategy:
       matrix:
         python-version:
-          - "3.6"
           - "3.7"
           - "3.8"
         platform: [ubuntu-latest, macos-latest]

CHANGELOG.rst

@@ -37,6 +37,20 @@ Added
   `#5 <https://github.com/openego/eGon-data/issues/5>`_
 * Download cleaned open-MaStR data from Zenodo
   `#14 <https://github.com/openego/eGon-data/issues/14>`_
+* NEP 2021 input data import
+  `#45 <https://github.com/openego/eGon-data/issues/45>`_
+* Option for running workflow in test mode
+  `#112 <https://github.com/openego/eGon-data/issues/112>`_
+* Abstraction of hvmv and ehv substations
+  `#9 <https://github.com/openego/eGon-data/issues/9>`_
+* Filter zensus being inside Germany and assign population to municipalities
+  `#7 <https://github.com/openego/eGon-data/issues/7>`_
+* RE potential areas data import
+  `#124 <https://github.com/openego/eGon-data/issues/124>`_
+* Heat demand data import
+  `#101 <https://github.com/openego/eGon-data/issues/101>`_
+* Demographic change integration
+  `#47 <https://github.com/openego/eGon-data/issues/47>`_
 
 Changed
 -------
@@ -45,6 +59,17 @@ Changed
   `#20 <https://github.com/openego/eGon-data/issues/20>`_
 * Switch from Travis to GitHub actions for CI jobs
   `#92 <https://github.com/openego/eGon-data/issues/92>`_
+* Rename columns to id and zensus_population_id in zensus tables
+  `#140 <https://github.com/openego/eGon-data/issues/140>`_
+* Revise docs CONTRIBUTING section and in particular PR guidelines
+  `#88 <https://github.com/openego/eGon-data/issues/88>`_ and
+  `#145 <https://github.com/openego/eGon-data/issues/145>`_
+* Drop support for Python3.6
+  `#148 <https://github.com/openego/eGon-data/issues/148>`_
+* Improve selection of zensus data in test mode
+  `#151 <https://github.com/openego/eGon-data/issues/151>`_
 
 Bug fixes
 ---------
+* Heat demand data import
+  `#157 <https://github.com/openego/eGon-data/issues/157>`_

CONTRIBUTING.rst

@@ -5,37 +5,13 @@ Contributing
 Contributions are welcome, and they are greatly appreciated! Every
 little bit helps, and credit will always be given.
 
-Bug reports
-===========
-
-When `reporting a bug <https://github.com/openego/eGon-data/issues>`_ please include:
-
-    * Your operating system name and version.
-    * Any details about your local setup that might be helpful in troubleshooting.
-    * Detailed steps to reproduce the bug.
-
-Documentation improvements
-==========================
-
-eGo^N Data could always use more documentation, whether as part of the
-official eGo^N Data docs, in docstrings, or even on the web in blog posts,
-articles, and such.
-
-Feature requests and feedback
-=============================
-
-The best way to send feedback is to file an issue at https://github.com/openego/eGon-data/issues.
-
-If you are proposing a feature:
-
-* Explain in detail how it would work.
-* Keep the scope as narrow as possible, to make it easier to implement.
-* Remember that this is a volunteer-driven project, and that code contributions are welcome :)
 
 Development
 ===========
 
-To set up `eGon-data` for local development:
+To set up `eGon-data` for local development (points (1) and (2) are only
+relevant for people who are not members of the organization
+`openego <https://github.com/openego>`_):
 
 1. Fork `eGon-data <https://github.com/openego/eGon-data>`_
    (look for the "Fork" button).
@@ -47,11 +23,16 @@ To set up `eGon-data` for local development:
 
     git checkout -b name-of-your-bugfix-or-feature
 
+   Consider to use the following pattern for naming branches
+   `<features/fixes>/#<issue-number>-<minimal-description-as-name>`.
    Now you can make your changes locally.
 
-4. When you're done making changes run all the checks and docs builder with `tox <https://tox.readthedocs.io/en/latest/install.html>`_ one command::
+4. When you're done making changes run all the checks and docs builder with
+`tox <https://tox.readthedocs.io/en/latest/install.html>`_ one command:
+
+   .. code-block:: bash
 
-    tox
+      tox
 
 5. Commit your changes and push your branch to GitHub::
 
@@ -61,31 +42,125 @@ To set up `eGon-data` for local development:
 
 6. Submit a pull request through the GitHub website.
 
+Tips
+^^^^
+
+To run a subset of tests::
+
+    tox -e envname -- pytest -k test_myfeature
+
+To run all the test environments in *parallel*::
+
+    tox -p auto
+
+
 Pull Request Guidelines
 -----------------------
 
-If you need some code review or feedback while you're developing the code just make the pull request.
+We use pull requests (PR) to integrate code changes from branches.
+PRs always need to be reviewed (exception proves the rule!). Therefore, ask
+one of the other developers for reviewing your changes. Once approved, the PR
+can be merged. Please delete the branch after merging.
 
-For merging, you should:
+Before requesting a review, please
 
 1. Include passing tests (run ``tox``) [1]_.
-2. Update documentation when there's new API, functionality etc.
-3. Add a note to ``CHANGELOG.rst`` about the changes.
-4. Add yourself to ``AUTHORS.rst``.
+2. Let the workflow run in :ref:`Test mode` once from scratch to verify
+   successful execution
+3. Update documentation when there's new API, functionality etc.
+4. Add a note to ``CHANGELOG.rst`` about the changes.
+5. Add yourself to ``AUTHORS.rst``.
+
+When requesting reviews, please keep in mind it might be a significant effort
+to review the PR. Try to make it easier for them and keep the overall effort
+as low as possible. Therefore,
+
+* asking for reviewing specific aspects helps reviewers a lot to focus on the
+  relevant parts
+* when multiple people are asked for a review it should be avoided that they
+  check/test the same things. Be even more specific what you expect from
+  someone in particular.
+
+
+What needs to be reviewed?
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Things that definitely should be checked during a review of a PR:
+
+* *Is the code working?* The contributor should already have made sure that
+  this is the case. Either by automated test or manual execution.
+* *Is the data correct?* Verifying that newly integrated and processed data
+  is correct is usually not possible during reviewing a PR. If it is necessary,
+  please ask the reviewer specifically for this.
+* *Do tests pass?* See automatic checks.
+* *Is the documentation up-to-date?* Please check this.
+* *Was* ``CHANGELOG.rst`` *updated accordingly?* Should be the case, please
+  verify.
+* *Is metadata complete and correct (in case of data integration)?* Please
+  verify. In case of a pending metadata creation make sure an appropriate issue is filed. 
+
+
+Additional case-dependent aspects make sense.
+
+
 
 .. [1] If you don't have all the necessary python versions available locally you can rely on
        `CI on GitHub actions <https://github.com/openego/eGon-data/actions?query=workflow%3A%22Tests%2C+code+style+%26+coverage%22>`_
        - it will run the tests for each change you add in the pull request.
 
        It will be slower though ...
 
-Tips
-----
 
-To run a subset of tests::
+Extending the data workflow
+---------------------------
 
-    tox -e envname -- pytest -k test_myfeature
 
-To run all the test environments in *parallel*::
+Adjusting test mode data
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+When integrating new data or data processing scripts, make sure the
+:ref:`Test mode` still works correctly on a limited subset of data.
+In particular, if a new external data sources gets integrated make sure the
+data gets cut to the region of the test mode.
+
+
+Bug reports and feature requests
+================================
+
+When `reporting a bug <https://github.com/openego/eGon-data/issues>`_ please include:
+
+* Your operating system name and version.
+* Any details about your local setup that might be helpful in troubleshooting.
+* Detailed steps to reproduce the bug.
+
+If you are proposing a feature:
+
+* Explain in detail how it would work.
+* Keep the scope as narrow as possible, to make it easier to implement.
+* Remember that this is a volunteer-driven project, and that code contributions are welcome :)
+
+Documentation improvements
+==========================
+
+eGo^N Data could always use more documentation, whether as part of the
+official eGo^N Data docs, in docstrings, or even on the web in blog posts,
+articles, and such.
+Please make sure you change the documentation along with code changes.
+
+The changes of the documentation in a feature branch get visible once a pull
+request is opened.
+
+You can build the documentation locally with (executed in the repos root
+directory)
+
+.. code-block:: bash
+
+   sphinx-build -E -a docs docs/_build/
+
+Eventually, you might need to install additional dependencies for building the
+documenmtation:
+
+.. code-block:: bash
+
+   pip install -r docs/requirements.txt
 
-    tox -p auto

README.rst

@@ -106,6 +106,10 @@ packages are required too. Right now these are:
   On recent Ubuntu version you can install it via
   :code:`sudo apt install osm2pgsql`.
 
+* `postgis <https://postgis.net/>`_
+  On recent Ubuntu version you can install it via
+  :code:`sudo apt install postgis`.
+
 
 Installation
 ============
@@ -175,12 +179,12 @@ Test mode
 ---------
 
 The workflow can be tested on a smaller subset of data on example of the
-federal state of Bremen.
+federal state of Schleswig-Holstein.
+Data is reduced during execution of the workflow to represent only this area.
 
 .. warning::
 
-   Right now, only OSM data for Bremen get's imported. This is hard-wired in
-   `egon.data/data_sets.yml`.
+   Right now, the test mode is set in `egon.data/airflow/pipeline.py`.
 
 
 .. end-getting-started-information

setup.py

@@ -59,7 +59,6 @@ def read(*names, **kwargs):
         "Operating System :: Microsoft :: Windows",
         "Programming Language :: Python",
         "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: Implementation :: CPython",
@@ -80,17 +79,20 @@ def read(*names, **kwargs):
     keywords=[
         # eg: 'keyword1', 'keyword2', 'keyword3',
     ],
-    python_requires=">=3.6.*",
+    python_requires=">=3.7.*",
     install_requires=[
+        # eg: 'aspectlib==1.1.1', 'six>=1.7',
         "apache-airflow<2.0",
         "click",
+        "disaggregator @ git+https://github.com/openego/disaggregator.git@features/pip_install",
+        "geopandas",
+        "importlib-resources",
         "oedialect==0.0.8",
         "pyaml",
         "psycopg2",
-        "sqlalchemy",
-        "geopandas",
-        "disaggregator @ git+https://github.com/openego/disaggregator.git@features/pip_install"
-        # eg: 'aspectlib==1.1.1', 'six>=1.7',
+        "rasterio",
+        "rtree",
+        "sqlalchemy"
     ],
     extras_require={
         "dev": ["black", "flake8", "isort>=5", "pre-commit", "pytest", "tox"]