Convert PyPSA Network object to Grid object and profiles

[rouille]

Pull Request doc

Purpose

Convert a PyPSA Network object to a Grid object and times series for demand, hydro, solar and wind

What the code is doing

Use existing code in #608 to convert a in memory PyPSA Network object into objects that can be handled by the scenario framework. The feature is incomplete:

• storage is not yet implemented
• plant, branch, etc data frames of the Grid object have missing/new columns and different types than what is expected (see powersimdata.input.const module)
• hydro and wind profiles cannot be extracted from a PyPSA-Eur network

For example:

Reading a PyPSA-Eur netCDF file from Zenodo:

>>> from powersimdata.network.europe_tub.model import TUB
>>> from powersimdata.input.converter.pypsa_to_grid import FromPyPSA
>>> tub = TUB("Europe", reduction=512)
Title: PyPSA-Eur: An Open Optimisation Model of the European Transmission System (Dataset)
Keywords: power system model, capacity expansion model, energy system model, electricity system model, python
Publication date: 2021-09-22
DOI: 10.5281/zenodo.5521712
Total size: 1767.1 MB

Link: https://zenodo.org/api/files/6e58c6dc-083b-46c8-8255-8516e3eb0ed8/networks.zip   size: 1767.1 MB
networks.zip is already downloaded correctly.
All files have been downloaded.
>>> tub.build()
WARNING:pypsa.io:
Importing PyPSA from older version of PyPSA than current version.
Please read the release notes at https://pypsa.readthedocs.io/en/latest/release_notes.html
carefully to prepare your network for import.
Currently used PyPSA version [0, 19, 3], imported network file PyPSA version [0, 18, 0].

INFO:pypsa.io:Imported network elec_s_512.nc has buses, carriers, generators, lines, links, loads, storage_units
>>> grid = FromPyPSA(tub.network)
/Users/brdo/CEM/PowerSimData/powersimdata/input/converter/pypsa_to_grid.py:195: UserWarning: Substations could not be parsed.
  warnings.warn("Substations could not be parsed.")
/Users/brdo/CEM/PowerSimData/powersimdata/input/converter/pypsa_to_grid.py:247: UserWarning: The export of storages are not implemented yet.
  warnings.warn("The export of storages are not implemented yet.")
>>> grid.plant
                       weight        Pmax bus_id        type  Pmin  ramp_30  status
plant_id                                                                           
AL0 0 offwind-ac   698.367965    0.000000  AL0 0  offwind-ac   0.0      0.0     NaN
AL0 0 onwind      1762.449543    0.000000  AL0 0      onwind   0.0      0.0     NaN
AL0 0 solar       2347.489775    0.000000  AL0 0       solar   0.0      0.0     NaN
AT0 0 onwind       171.690009    0.000000  AT0 0      onwind   0.0      0.0     NaN
AT0 0 ror          297.580368  297.580368  AT0 0         ror   0.0      0.0     NaN
...                       ...         ...    ...         ...   ...      ...     ...
SK0 2 ror          188.960000  188.960000  SK0 2         ror   0.0      0.0     NaN
SK0 2 solar        993.396581    0.000000  SK0 2       solar   0.0      0.0     NaN
SK0 3 onwind       642.421167    0.000000  SK0 3      onwind   0.0      0.0     NaN
SK0 3 ror           27.850000   27.850000  SK0 3         ror   0.0      0.0     NaN
SK0 3 solar        501.830654    0.000000  SK0 3       solar   0.0      0.0     NaN

[2305 rows x 7 columns]

Extracting demand and solar profiles work well:

>>> from powersimdata.input.converter.pypsa_to_profiles import (
...     get_pypsa_demand_profile,
...     get_pypsa_gen_profile,
... )
>>> demand = get_pypsa_demand_profile(tub.network)
>>> demand
                         AL0 0       AT0 0       AT0 1       AT0 2       AT0 3   ...      SI0 2       SK0 0       SK0 1        SK0 2       SK0 3 
UTC                                                                              ...                                                             
2013-01-01 00:00:00  683.702703  537.098973  695.257525  325.506959  570.104515  ...  206.150601  600.840434  699.647401  1142.382463  342.129702
2013-01-01 01:00:00  634.945946  514.860245  666.470200  312.029256  546.499183  ...  197.094890  573.441247  667.742474  1090.288182  326.528096
2013-01-01 02:00:00  593.391892  487.314548  630.813172  295.335282  517.260762  ...  193.543630  554.671726  645.886344  1054.601549  315.840382
2013-01-01 03:00:00  554.054054  485.545558  628.523271  294.263192  515.383065  ...  190.702623  551.219860  641.866825  1048.038490  313.874826
2013-01-01 04:00:00  555.162162  491.694904  636.483402  297.989981  521.910297  ...  192.300689  552.082826  642.871705  1049.679254  314.366215
...                         ...         ...         ...         ...         ...  ...         ...         ...         ...          ...         ...
2013-12-31 19:00:00  710.297297  625.211508  809.316398  378.907254  663.631698  ...  217.514630  638.163736  743.108442  1213.345539  363.382283
2013-12-31 20:00:00  665.418919  594.549020  769.624782  360.324360  631.084954  ...  199.758334  615.079381  716.227913  1169.455082  350.237623
2013-12-31 21:00:00  683.148649  619.651826  802.119566  375.537828  657.730365  ...  189.282119  608.607133  708.691317  1157.149346  346.552205
2013-12-31 22:00:00  690.351351  588.483912  761.773694  356.648623  624.647136  ...  180.226408  598.683018  697.135202  1138.280551  340.901229
2013-12-31 23:00:00  671.513514  568.098412  735.385312  344.294061  603.008916  ...  192.300689  586.385745  682.815668  1114.899654  333.898934

[8760 rows x 509 columns]
>>> solar = get_pypsa_gen_profile(tub.network, "solar")
>>> solar
                     AL0 0 solar  AT0 0 solar  AT0 1 solar  AT0 2 solar  AT0 3 solar  ...  SI0 2 solar  SK0 0 solar  SK0 1 solar  SK0 2 solar  SK0 3 solar
UTC                                                                                   ...                                                                 
2013-01-01 00:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 01:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 02:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 03:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 04:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
...                          ...          ...          ...          ...          ...  ...          ...          ...          ...          ...          ...
2013-12-31 19:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 20:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 21:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 22:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 23:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0

[8760 rows x 502 columns]

But not wind and hydro:

>>> wind = get_pypsa_gen_profile(tub.network, "wind")
>>> wind
Empty DataFrame
Columns: []
Index: [2013-01-01 00:00:00, 2013-01-01 01:00:00, 2013-01-01 02:00:00, 2013-01-01 03:00:00, 2013-01-01 04:00:00, 2013-01-01 05:00:00, 2013-01-01 06:00:00, 2013-01-01 07:00:00, 2013-01-01 08:00:00, 2013-01-01 09:00:00, 2013-01-01 10:00:00, 2013-01-01 11:00:00, 2013-01-01 12:00:00, 2013-01-01 13:00:00, 2013-01-01 14:00:00, 2013-01-01 15:00:00, 2013-01-01 16:00:00, 2013-01-01 17:00:00, 2013-01-01 18:00:00, 2013-01-01 19:00:00, 2013-01-01 20:00:00, 2013-01-01 21:00:00, 2013-01-01 22:00:00, 2013-01-01 23:00:00, 2013-01-02 00:00:00, 2013-01-02 01:00:00, 2013-01-02 02:00:00, 2013-01-02 03:00:00, 2013-01-02 04:00:00, 2013-01-02 05:00:00, 2013-01-02 06:00:00, 2013-01-02 07:00:00, 2013-01-02 08:00:00, 2013-01-02 09:00:00, 2013-01-02 10:00:00, 2013-01-02 11:00:00, 2013-01-02 12:00:00, 2013-01-02 13:00:00, 2013-01-02 14:00:00, 2013-01-02 15:00:00, 2013-01-02 16:00:00, 2013-01-02 17:00:00, 2013-01-02 18:00:00, 2013-01-02 19:00:00, 2013-01-02 20:00:00, 2013-01-02 21:00:00, 2013-01-02 22:00:00, 2013-01-02 23:00:00, 2013-01-03 00:00:00, 2013-01-03 01:00:00, 2013-01-03 02:00:00, 2013-01-03 03:00:00, 2013-01-03 04:00:00, 2013-01-03 05:00:00, 2013-01-03 06:00:00, 2013-01-03 07:00:00, 2013-01-03 08:00:00, 2013-01-03 09:00:00, 2013-01-03 10:00:00, 2013-01-03 11:00:00, 2013-01-03 12:00:00, 2013-01-03 13:00:00, 2013-01-03 14:00:00, 2013-01-03 15:00:00, 2013-01-03 16:00:00, 2013-01-03 17:00:00, 2013-01-03 18:00:00, 2013-01-03 19:00:00, 2013-01-03 20:00:00, 2013-01-03 21:00:00, 2013-01-03 22:00:00, 2013-01-03 23:00:00, 2013-01-04 00:00:00, 2013-01-04 01:00:00, 2013-01-04 02:00:00, 2013-01-04 03:00:00, 2013-01-04 04:00:00, 2013-01-04 05:00:00, 2013-01-04 06:00:00, 2013-01-04 07:00:00, 2013-01-04 08:00:00, 2013-01-04 09:00:00, 2013-01-04 10:00:00, 2013-01-04 11:00:00, 2013-01-04 12:00:00, 2013-01-04 13:00:00, 2013-01-04 14:00:00, 2013-01-04 15:00:00, 2013-01-04 16:00:00, 2013-01-04 17:00:00, 2013-01-04 18:00:00, 2013-01-04 19:00:00, 2013-01-04 20:00:00, 2013-01-04 21:00:00, 2013-01-04 22:00:00, 2013-01-04 23:00:00, 2013-01-05 00:00:00, 2013-01-05 01:00:00, 2013-01-05 02:00:00, 2013-01-05 03:00:00, ...]

[8760 rows x 0 columns]
>>> hydro = get_pypsa_gen_profile(tub.network, "hydro")
>>> hydro
Empty DataFrame
Columns: []
Index: [2013-01-01 00:00:00, 2013-01-01 01:00:00, 2013-01-01 02:00:00, 2013-01-01 03:00:00, 2013-01-01 04:00:00, 2013-01-01 05:00:00, 2013-01-01 06:00:00, 2013-01-01 07:00:00, 2013-01-01 08:00:00, 2013-01-01 09:00:00, 2013-01-01 10:00:00, 2013-01-01 11:00:00, 2013-01-01 12:00:00, 2013-01-01 13:00:00, 2013-01-01 14:00:00, 2013-01-01 15:00:00, 2013-01-01 16:00:00, 2013-01-01 17:00:00, 2013-01-01 18:00:00, 2013-01-01 19:00:00, 2013-01-01 20:00:00, 2013-01-01 21:00:00, 2013-01-01 22:00:00, 2013-01-01 23:00:00, 2013-01-02 00:00:00, 2013-01-02 01:00:00, 2013-01-02 02:00:00, 2013-01-02 03:00:00, 2013-01-02 04:00:00, 2013-01-02 05:00:00, 2013-01-02 06:00:00, 2013-01-02 07:00:00, 2013-01-02 08:00:00, 2013-01-02 09:00:00, 2013-01-02 10:00:00, 2013-01-02 11:00:00, 2013-01-02 12:00:00, 2013-01-02 13:00:00, 2013-01-02 14:00:00, 2013-01-02 15:00:00, 2013-01-02 16:00:00, 2013-01-02 17:00:00, 2013-01-02 18:00:00, 2013-01-02 19:00:00, 2013-01-02 20:00:00, 2013-01-02 21:00:00, 2013-01-02 22:00:00, 2013-01-02 23:00:00, 2013-01-03 00:00:00, 2013-01-03 01:00:00, 2013-01-03 02:00:00, 2013-01-03 03:00:00, 2013-01-03 04:00:00, 2013-01-03 05:00:00, 2013-01-03 06:00:00, 2013-01-03 07:00:00, 2013-01-03 08:00:00, 2013-01-03 09:00:00, 2013-01-03 10:00:00, 2013-01-03 11:00:00, 2013-01-03 12:00:00, 2013-01-03 13:00:00, 2013-01-03 14:00:00, 2013-01-03 15:00:00, 2013-01-03 16:00:00, 2013-01-03 17:00:00, 2013-01-03 18:00:00, 2013-01-03 19:00:00, 2013-01-03 20:00:00, 2013-01-03 21:00:00, 2013-01-03 22:00:00, 2013-01-03 23:00:00, 2013-01-04 00:00:00, 2013-01-04 01:00:00, 2013-01-04 02:00:00, 2013-01-04 03:00:00, 2013-01-04 04:00:00, 2013-01-04 05:00:00, 2013-01-04 06:00:00, 2013-01-04 07:00:00, 2013-01-04 08:00:00, 2013-01-04 09:00:00, 2013-01-04 10:00:00, 2013-01-04 11:00:00, 2013-01-04 12:00:00, 2013-01-04 13:00:00, 2013-01-04 14:00:00, 2013-01-04 15:00:00, 2013-01-04 16:00:00, 2013-01-04 17:00:00, 2013-01-04 18:00:00, 2013-01-04 19:00:00, 2013-01-04 20:00:00, 2013-01-04 21:00:00, 2013-01-04 22:00:00, 2013-01-04 23:00:00, 2013-01-05 00:00:00, 2013-01-05 01:00:00, 2013-01-05 02:00:00, 2013-01-05 03:00:00, ...]

[8760 rows x 0 columns]

Testing

Add new unit tests

Where to look

Two new modules:

• pypsa_to_grid that creates a Grid object from the Network object
• pypsa_to_profiles that extracts the profiles from the Network object

Usage Example/Visuals

How the code can be used and/or images of any graphs, tables or other visuals (not always applicable).

Time estimate

How long will it take for reviewers and observers to understand this code change?

[rouille]

@BainanXia, @jenhagg, this is ready to be merged into develop. We can successfully convert a PyPSA Network object to a Grid object and extract profiles. The next step is to set the model immutables after conversion (or during, to be discussed) but this can be done off the develop branch with potential refactoring of the TUB and Grid class.

[BainanXia]

Very nice! Thanks 🚀