Merge branch 'dev' into documentation/rli_datasets

src/egon/data/datasets/emobility/heavy_duty_transport/h2_demand_distribution.py

docs/about.rst

@@ -44,6 +44,7 @@ The figure below visualizes the interdependencies between the different tools.
   :width: 800
   :alt: eGon-data tool chain
 
+.. _concept-and-scenarios-ref:
 
 Modeling concept and scenarios
 ===============================

docs/data.rst

@@ -1,10 +1,23 @@
 ****
 Data
 ****
+The description of the methods, input data and results of the eGon-data pipeline is given in the following section. 
+References to datasets and functions are integrated if more detailed information is required.
+
+Main input data and their processing
+====================================
+
+All methods in the eGon-data workflow rely on public and freely available data from different external sources. The most important data sources
+and their processing within the eGon-data pipeline are described here. 
+
+.. include:: data/input_data.rst
 
 Grid models
 ===========
 
+Power grid models of different voltage levels form a central part of the eGon data model, which is required for cross-grid-level optimization.
+In addition, sector coupling necessitates the representation of the gas grid infrastructure, which is also described in this section. 
+
 Electricity grid
 ----------------
 
@@ -18,6 +31,11 @@ Gas grid
 Demand
 ======
 
+Electricity, heat and gas demands from different consumption sectors are taken into account in eGon-data. The related methods to distribute and 
+process the demand data are described in the following chapters for the different consumption sectors separately. 
+
+.. _electricity-demand-ref:
+
 Electricity
 -----------
 
@@ -42,6 +60,9 @@ Mobility
 Supply
 ======
 
+The distribution and assignment of supply capacities or potentials are carried out technology-specific. The different methods are described in the 
+following chapters. 
+
 Electricity
 -----------
 
@@ -60,6 +81,10 @@ Gas
 Flexibility options
 ===================
 
+Different flexibility options are part of the model and can be utilized in the optimization of the energy system. Therefore detailed information about 
+flexibility potentials and their distribution are needed. The considered technologies described in the following chapters range from different storage units, 
+through dynamic line rating to Demand-Side-Management measures. 
+
 Demand-Side-Management
 ----------------------
 
@@ -94,14 +119,4 @@ Heat stores
 Published data
 ==============
 
-Data bundle
------------
 
-The data bundle is published on
-`zenodo <https://sandbox.zenodo.org/record/1167119>`_. It contains several data
-sets, which serve as a basis for egon-data. One such data set is the geocoding
-for the `MaStR data set <https://sandbox.zenodo.org/record/1132987>`_ which is
-used for eGon-data as well. Whenever the MaStR data set is updated it is
-necessary to redo the geocoding with the new data set and update the data
-bundle accordingly. The geocoding can be done based on the
-`mastr-geocoding repository <https://github.com/RLI-sandbox/mastr-geocoding>`_.

docs/data/DLR.rst

@@ -1 +1,47 @@
-Methods to include dynamic line rating in our model. 
+====================================================
+Methods to include dynamic line rating in our model
+====================================================
+
+To calculate the transmission capacity of each transmission line in the model, 
+the procedure suggested in the **Principles for the Expansion Planning of the
+German Transmission Network** [NEP2021] where used:
+
+1. Import the temperature and wind temporal raster layers from ERA-5. Hourly
+resolution data from the year 2011 was used. Raster resolution
+latitude-longitude grids at 0.25° x 0.25°.
+
+2. Import shape file for the 9 regions proposed by the Principles for
+the Expansion Planning. See Figure 1.
+
+.. image:: images/regions_DLR.png
+  :width: 400
+  :alt: regions DLR
+
+Figure 1: Representative regions in Germany for DLR analysis [NEP2021]
+
+3. Find the lowest wind speed in each region. To perform this, for each
+independent region, the wind speed of every cell in the raster layer should be
+extracted and compared. This procedure is repeated for each hour in the
+year 2011. The results are the 8760 lowest wind speed per region.
+
+4. Find the highest temperature in each region. To perform this, for each
+independent region, the temperature of every cell in the raster layer should
+be extracted and compared. This procedure is repeated for each hour in the
+year 2011. The results are the 8760 maximum temperature per region.
+
+5. Calculate the maximum capacity for each region using the parameters shown in
+Figure 2.
+
+.. image:: images/table_max_capacity_DLR.png
+  :width: 400
+  :alt: table_max_capacity_DLR
+
+Figure 2: transmission capacity based on max temperature and min wind speed [NEP2021]
+
+6. Assign the maximum capacity of the corresponding region to each transmission
+line inside each one of them. Crossborder lines and underground lines receive
+no values. It means that their capacities are static and equal to their nominal
+values. Lines that cross borders between regions receive the lowest 
+capacity per hour of the regions containing the line.
+
+.. [NEP2021] Principles for the Expansion Planning of the German Transmission Network https://www.netzentwicklungsplan.de/

docs/data/batteries.rst

@@ -1 +1,37 @@
-Information about batteries in our system. 
+Battery storage units comprise home batteries and larger, grid-supportive batteries. National capacities for home batteries arise from external sources, e.g. the Grid Development Plan for the scenario ``eGon2035``, whereas the capacities of large-scale batteries are a result of the grid optimization tool `eTraGo <https://github.com/openego/eTraGo>`_.
+
+Home battery capacities are first distributed to medium-voltage grid districts (MVGD) and based on that further disaggregated to single buildings. The distribution on MVGD level is done proportional to the installed capacities of solar rooftop power plants, assuming that they are used as solar home storage. 
+
+Potential large-scale batteries are included in the data model at every substation. The data model includes technical and economic parameters, such as efficiencies and investment costs. The energy-to-power ratio is set to a fixed value of 6 hours. Other central parameters are given in the following table
+
+.. list-table:: Parameters of batteries for scenario eGon2035
+   :widths: 40 30 30
+   :header-rows: 1
+
+   * - 
+     - Value
+     - Sources
+
+   * - Efficiency store
+     - 98 % 
+     - [DAE_store]_
+
+   * - Efficiency dispatch
+     - 98 %
+     - [DAE_store]_
+
+   * - Standing loss
+     - 0 %
+     - [DAE_store]_
+
+   * - Investment costs
+     - 838 €/kW
+     - [DAE_store]_
+
+   * - Home storage units
+     - 16.8 GW
+     - [NEP2021]_
+
+
+On transmission grid level, distinguishing between home batteries and large-scale batteries was not possible. Therefore, the capacities of home batteries were set as a lower boundary of the large-scale battery capacities. 
+This is implemented in the dataset :py:class:`StorageEtrago <egon.data.datasets.storages_etrago.StorageEtrago>`, the data for batteries in the transmission grid is stored in the database table :py:class:`grid.egon_etrago_storage <egon.data.datasets.etrago_setup.EgonPfHvStorage>`.

docs/data/electricity_grids.rst

@@ -1,7 +1,125 @@
-Information about our electricity grids and how they were created
+.. _ehv-hv-grids:
 
-High and extra-high voltage grids 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+High and extra-high voltage grids
+++++++++++++++++++++++++++++++++++
+
+The model of the German extra-high (eHV) and high voltage (HV) grid is based
+on data retrieved from OpenStreetMap (status January 2021) [OSM]_ and additional
+parameters for standard transmission lines from [Brakelmann2004]_. To gather all
+required information, such as line topology, voltage level, substation locations,
+and electrical parameters, to create a calculable power system model, the `osmTGmod
+tool <https://github.com/openego/osmTGmod>`_ was used. The corresponding dataset
+:py:class:`Osmtgmod <egon.data.datasets..osmtgmod.Osmtgmod>` executes osmTGmod
+and writes the resulting data to the database.
+
+The resulting grid model includes the voltage levels 380, 220 and 110 kV and
+all substations interconnecting these grid levels. For further information on the
+generation of the grid topology please refer to [Mueller2018]_.
+
+.. _ding0-grids:
 
 Medium and low-voltage grids
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+++++++++++++++++++++++++++++
+
+Medium (MV) and low (LV) voltage grid topologies for entire Germany are generated using
+the python tool ding0 `ding0 <https://github.com/openego/ding0>`_.
+ding0 generates synthetic grid topologies based on high-resolution geodata and routing
+algorithms as well as typical network planning principles.
+The generation of the
+grid topologies is not part of eGon_data, but ding0 solely uses data generated with eGon_data,
+such as locations of HV/MV stations (see :ref:`ehv-hv-grids`), locations and peak demands
+of buildings in the grid (see :ref:`building-data-ref` respectively :ref:`electricity-demand-ref`),
+as well as locations of generators from MaStR (see :ref:`mastr-ref`). A full list
+of tables used in ding0 can be found in its `config <https://github.com/openego/ding0/blob/dev/ding0/config/config_db_tables.cfg>`_.
+An exemplary MV grid with one underlying LV grid is shown in figure :ref:`ding0-mv-grid`.
+The grid data of all over 3.800 MV grids is published on `zenodo <https://zenodo.org/record/890479>`_.
+
+.. figure:: /images/ding0_mv_lv_grid.png
+  :name: ding0-mv-grid
+  :width: 600
+
+  Exemplary synthetic medium-voltage grid with underlying low-voltage grid generated with ding0
+
+Besides data on buildings and generators, ding0 requires data on the supplied areas
+by each grid. This is as well done in eGon_data and described in the following.
+
+.. _mv-grid-districts:
+
+MV grid districts
+~~~~~~~~~~~~~~~~~~
+
+Medium-voltage (MV) grid districts describe the area supplied by one MV grid.
+They are defined by one polygon that represents the
+supply area. Each MV grid district is connected to the HV grid via a single
+substation. An exemplary MV grid district is shown in figure :ref:`ding0-mv-grid` (orange line).
+
+The MV grid districts are generated in the dataset
+:class:`MvGridDistricts<egon.data.datasets.mv_grid_districts.MvGridDistricts>`.
+The methods used for identifying the MV grid districts are heavily inspired
+by Hülk et al. (2017) [Huelk2017]_
+(section 2.3), but the implementation differs in detail.
+The main difference is that direct adjacency is preferred over proximity.
+For polygons of municipalities
+without a substation inside, it is iteratively checked for direct adjacent
+other polygons that have a substation inside. Speaking visually, a MV grid
+district grows around a polygon with a substation inside.
+
+The grid districts are identified using three data sources
+
+1. Polygons of municipalities (:class:`Vg250GemClean<egon.data.datasets.mv_grid_districts.Vg250GemClean>`)
+2. Locations of HV-MV substations (:class:`EgonHvmvSubstation<egon.data.datasets.osmtgmod.substation.EgonHvmvSubstation>`)
+3. HV-MV substation voronoi polygons (:class:`EgonHvmvSubstationVoronoi<egon.data.datasets.substation_voronoi.EgonHvmvSubstationVoronoi>`)
+
+Fundamentally, it is assumed that grid districts (supply areas) often go
+along borders of administrative units, in particular along the borders of
+municipalities due to the concession levy.
+Furthermore, it is assumed that one grid district is supplied via a single
+substation and that locations of substations and grid districts are designed
+for aiming least lengths of grid line and cables.
+
+With these assumptions, the three data sources from above are processed as
+follows:
+
+* Find the number of substations inside each municipality
+* Split municipalities with more than one substation inside
+
+  * Cut polygons of municipalities with voronoi polygons of respective
+    substations
+  * Assign resulting municipality polygon fragments to nearest substation
+* Assign municipalities without a single substation to nearest substation in
+  the neighborhood
+* Merge all municipality polygons and parts of municipality polygons to a
+  single polygon grouped by the assigned substation
+
+For finding the nearest substation, as already said, direct adjacency is
+preferred over closest distance. This means, the nearest substation does not
+necessarily have to be the closest substation in the sense of beeline distance.
+But it is the substation definitely located in a neighboring polygon. This
+prevents the algorithm to find solutions where a MV grid districts consists of
+multi-polygons with some space in between.
+Nevertheless, beeline distance still plays an important role, as the algorithm
+acts in two steps
+
+1. Iteratively look for neighboring polygons until there are no further
+   polygons
+2. Find a polygon to assign to by minimum beeline distance
+
+The second step is required in order to cover edge cases, such as islands.
+
+For understanding how this is implemented into separate functions, please
+see :func:`define_mv_grid_districts<egon.data.datasets.mv_grid_districts.define_mv_grid_districts>`.
+
+.. _load-areas-ref:
+
+Load areas
+~~~~~~~~~~~~
+
+Load areas (LAs) are defined as geographic clusters where electricity is consumed.
+They are used in ding0 to determine the extent and number of LV grids. Thus, within
+each LA there are one or multiple MV-LV substations, each supplying one LV grid.
+Exemplary load areas are shown in figure :ref:`ding0-mv-grid` (grey and orange areas).
+
+The load areas are set up in the
+:class:`LoadArea<egon.data.datasets.loadarea.LoadArea>` dataset.
+The methods used for identifying the load areas are heavily inspired
+by Hülk et al. (2017) [Huelk2017]_ (section 2.4).

docs/data/electricity_supply.rst

@@ -1 +1,86 @@
-Information on electricity supply by carrier etc. 
+The electrical power plants park, including data on geolocations, installed capacities, etc.
+for the different scenarios is set up in the dataset
+:class:`PowerPlants<egon.data.datasets.power_plants.PowerPlants>`.
+
+Main inputs into the dataset are target capacities per technology and federal state
+in each scenario (see :ref:`concept-and-scenarios-ref`) as well as the MaStR (see :ref:`mastr-ref`),
+OpenStreetMap (see :ref:`osm-ref`) and potential areas (provided through the data bundle,
+see :ref:`data-bundle-ref`) to distribute the generator capacities within each federal state region.
+The approach taken to distribute the target capacities within each federal state differs for
+the different technologies and is described in the following.
+The final distribution in the eGon2035 scenario is shown in figure :ref:`generator-park-egon-2035`.
+
+.. figure:: /images/Erzeugerpark.png
+  :name: generator-park-egon-2035
+  :width: 400
+
+  Generator park in the eGon2035 scenario
+
+Onshore wind
++++++++++++++
+
+Offshore wind
+++++++++++++++
+
+PV ground mounted
+++++++++++++++++++
+
+PV rooftop
++++++++++++
+
+In a first step, the target capacity in the eGon2035 and eGon100RE scenarios is distributed
+to all MV grid districts linear to the residential and CTS electricity demands in the
+grid district (done in function
+:func:`pv_rooftop_per_mv_grid<egon.data.datasets.power_plants.pv_rooftop.pv_rooftop_per_mv_grid>`).
+
+Afterwards, the PV rooftop capacity per MV grid district is disaggregated
+to individual buildings inside the grid district (done in function
+:func:`pv_rooftop_to_buildings<egon.data.datasets.power_plants.pv_rooftop_buildings.pv_rooftop_to_buildings>`).
+The basis for this is data from the MaStR, which is first cleaned and missing information
+inferred, and then allocated to specific buildings. New PV plants are in a last step
+added based on the capacity distribution from MaStR.
+These steps are in more detail described in the following.
+
+MaStR data cleaning and inference:
+
+* Drop duplicates and entries with missing critical data.
+* Determine most plausible capacity from multiple values given in MaStR data.
+* Drop generators that don't have a plausible capacity (23.5 MW > P > 0.1 kW).
+* Randomly and weighted add a start-up date if it is missing.
+* Extract zip and municipality from 'site' given in MaStR data.
+* Geocode unique zip and municipality combinations with Nominatim (1 sec
+  delay). Drop generators for which geocoding failed or which are located
+  outside the municipalities of Germany.
+* Add some visual sanity checks for cleaned data.
+
+Allocation of MaStR plants to buildings:
+
+* Allocate each generator to an existing building from OSM or a synthetic building
+  (see :ref:`building-data-ref`).
+* Determine the quantile each generator and building is in depending on the
+  capacity of the generator and the area of the polygon of the building.
+* Randomly distribute generators within each municipality preferably within
+  the same building area quantile as the generators are capacity wise.
+* If not enough buildings exist within a municipality and quantile additional
+  buildings from other quantiles are chosen randomly.
+
+Disaggregation of PV rooftop scenario capacities:
+
+* The scenario data per federal state is linearly distributed to the MV grid
+  districts according to the PV rooftop potential per MV grid district.
+* The rooftop potential is estimated from the building area given from the OSM
+  buildings.
+* Grid districts, which are located in several federal states, are allocated
+  PV capacity according to their respective roof potential in the individual
+  federal states.
+* The disaggregation of PV plants within a grid district respects existing
+  plants from MaStR, which did not reach their end of life.
+* New PV plants are randomly and weighted generated using the capacity distribution of
+  PV rooftop plants from MaStR.
+* Plant metadata (e.g. plant orientation) is also added randomly and weighted
+  using MaStR data as basis.
+
+Hydro
+++++++
+
+

docs/data/gas_grids.rst

@@ -1,7 +1,7 @@
 Information about the gas grids and how they were created
 
-Methane grid 
-~~~~~~~~~~~~
+Methane grid
+++++++++++++++
 
 Hydrogen grid
-~~~~~~~~~~~~~
+++++++++++++++