Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract profiles from PyPSA's Network objects #682

Merged
merged 5 commits into from
Nov 4, 2022
Merged

Conversation

rouille
Copy link
Collaborator

@rouille rouille commented Sep 16, 2022

Pull Request doc

Purpose

Extract generator and demand profiles from a PyPSA Network object

What the code is doing

Loop through the Generator and StorageUnit components of a Network object and build user specified profiles through a dictionary mapping profile type (hydro, solar, wind) to carriers (onwind, solar, ror, etc). Inflow profiles are normalized such that the maximum inflow is 1

Testing

New unit tests

Where to look

The get_pypsa_gen_profile function

Usage Example/Visuals

>>> from powersimdata.input.converter.pypsa_to_profiles import get_pypsa_gen_profile
>>> from powersimdata.network.europe_tub.model import TUB
>>> profile2carrier = {'hydro': {'hydro', 'ror', 'PHS'}, 'solar': {'solar'}, 'wind': {'onwind', 'offwind-ac', 'offwind-dc'}}
>>> tub = TUB("Europe", reduction=128)
Title: PyPSA-Eur: An Open Optimisation Model of the European Transmission System (Dataset)
Keywords: power system model, capacity expansion model, energy system model, electricity system model, python
Publication date: 2022-09-20
DOI: 10.5281/zenodo.7251657
Total size: 1702.1 MB

Link: https://zenodo.org/api/files/44276d06-8d1e-4dd7-bed7-57f8fbf9f4d5/networks.zip   size: 1702.1 MB
networks.zip is already downloaded correctly.
All files have been downloaded.
>>> tub.build()
INFO:pypsa.io:Imported network elec_s_128_ec.nc has buses, carriers, generators, lines, links, loads, storage_units, stores
>>> profile = get_pypsa_gen_profile(tub.network, profile2carrier)
>>> profile["hydro"]
                     AL1 0 ror  AT1 0 ror  AT1 1 ror  BE1 0 ror  ...  SE2 3 hydro inflow  SE2 4 hydro inflow  SI1 0 hydro inflow  SK1 0 hydro inflow
UTC                                                              ...                                                                                
2013-01-01 00:00:00   0.274540   0.300145   0.300145   0.995894  ...            0.656168            0.656168            0.265311            0.087945
2013-01-01 01:00:00   0.274515   0.300029   0.300029   1.000000  ...            0.654647            0.654647            0.265273            0.087792
2013-01-01 02:00:00   0.274505   0.299936   0.299936   1.000000  ...            0.653922            0.653922            0.265214            0.087755
2013-01-01 03:00:00   0.274627   0.299868   0.299868   1.000000  ...            0.652859            0.652859            0.265212            0.087960
2013-01-01 04:00:00   0.274700   0.299820   0.299820   1.000000  ...            0.652117            0.652117            0.265189            0.088083
...                        ...        ...        ...        ...  ...                 ...                 ...                 ...                 ...
2013-12-31 19:00:00   0.250162   0.368205   0.368205   0.579924  ...            0.491466            0.491466            0.470877            0.130620
2013-12-31 20:00:00   0.250062   0.367945   0.367945   0.577514  ...            0.491266            0.491266            0.470428            0.130693
2013-12-31 21:00:00   0.250067   0.367699   0.367699   0.575495  ...            0.491070            0.491070            0.469969            0.130761
2013-12-31 22:00:00   0.250018   0.367444   0.367444   0.572038  ...            0.490882            0.490882            0.469443            0.130799
2013-12-31 23:00:00   0.250010   0.367188   0.367188   0.567667  ...            0.490702            0.490702            0.468825            0.130814

[8760 rows x 152 columns]
>>> profile["hydro"].max()
AL1 0 ror             1.0
AT1 0 ror             1.0
AT1 1 ror             1.0
BE1 0 ror             1.0
BE1 2 ror             1.0
                     ... 
SE2 2 hydro inflow    1.0
SE2 3 hydro inflow    1.0
SE2 4 hydro inflow    1.0
SI1 0 hydro inflow    1.0
SK1 0 hydro inflow    1.0
Length: 152, dtype: float64
>>> profile["wind"]
                     AL1 0 offwind-ac  AL1 0 onwind  AT1 0 onwind  AT1 1 onwind  ...  SE2 4 onwind  SI1 0 offwind-ac  SI1 0 onwind  SK1 0 onwind
UTC                                                                              ...                                                            
2013-01-01 00:00:00          0.002930      0.001468      0.024682      0.180255  ...      0.648024          0.000000      0.055151      0.361028
2013-01-01 01:00:00          0.001872      0.000000      0.016699      0.190325  ...      0.654533          0.000000      0.052604      0.368898
2013-01-01 02:00:00          0.000000      0.000000      0.012459      0.190493  ...      0.659420          0.000000      0.052227      0.382934
2013-01-01 03:00:00          0.000000      0.000000      0.012660      0.188959  ...      0.680052          0.000000      0.050760      0.388341
2013-01-01 04:00:00          0.000000      0.000000      0.013591      0.177692  ...      0.719678          0.000000      0.047294      0.409304
...                               ...           ...           ...           ...  ...           ...               ...           ...           ...
2013-12-31 19:00:00          0.026572      0.024328      0.034577      0.118275  ...      0.670652          0.025049      0.001707      0.125728
2013-12-31 20:00:00          0.029891      0.031360      0.042418      0.130074  ...      0.584858          0.021100      0.000000      0.142435
2013-12-31 21:00:00          0.032750      0.034518      0.036139      0.139478  ...      0.489739          0.000000      0.004894      0.153933
2013-12-31 22:00:00          0.028040      0.027423      0.036657      0.135560  ...      0.438377          0.000000      0.009091      0.163441
2013-12-31 23:00:00          0.018277      0.020763      0.012115      0.140614  ...      0.368146          0.000000      0.013550      0.183303

[8760 rows x 266 columns]
>>> profile["wind"].max()
AL1 0 offwind-ac    0.885396
AL1 0 onwind        0.640752
AT1 0 onwind        0.612467
AT1 1 onwind        0.921961
BA1 0 onwind        0.863360
                      ...   
SE2 4 offwind-dc    0.885500
SE2 4 onwind        0.999895
SI1 0 offwind-ac    0.885500
SI1 0 onwind        0.926825
SK1 0 onwind        0.941593
Length: 266, dtype: float64
>>> profile["solar"]
                     AL1 0 solar  AT1 0 solar  AT1 1 solar  BA1 0 solar  BE1 0 solar  ...  SE2 1 solar  SE2 2 solar  SE2 4 solar  SI1 0 solar  SK1 0 solar
UTC                                                                                   ...                                                                 
2013-01-01 00:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 01:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 02:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 03:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-01-01 04:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
...                          ...          ...          ...          ...          ...  ...          ...          ...          ...          ...          ...
2013-12-31 19:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 20:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 21:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 22:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0
2013-12-31 23:00:00          0.0          0.0          0.0          0.0          0.0  ...          0.0          0.0          0.0          0.0          0.0

[8760 rows x 126 columns]
>>> profile["solar"].max()
AL1 0 solar    0.808040
AT1 0 solar    0.815206
AT1 1 solar    0.766280
BA1 0 solar    0.786220
BE1 0 solar    0.827806
                 ...   
SE2 1 solar    0.800421
SE2 2 solar    0.782425
SE2 4 solar    0.793002
SI1 0 solar    0.815808
SK1 0 solar    0.782587
Length: 126, dtype: float64

Note also that PHS does not have inflow profile.

>>> sum(tub.network.storage_units_t["inflow"].columns.str.contains("PHS"))
0
>>> sum(tub.network.storage_units_t["inflow"].columns.str.contains("hydro"))
73

Time estimate

15min

@BainanXia
Copy link
Collaborator

Moving the conversation here: I've been discussing with @rouille that it might be cleaner to deal with all inflow profiles in this way -- always normalize the profile based on profile max and assign the Pmax of the added plant to be the profile max (we could transform the normalized profile into MWh via such Pmax later in the workflow). Then assign the pnorm to be the capacity of the added dcline in the subsystem. What do you think @FabianHofmann ?

@rouille
Copy link
Collaborator Author

rouille commented Sep 16, 2022

Moving the conversation here: I've been discussing with @rouille that it might be cleaner to deal with all inflow profiles in this way -- always normalize the profile based on profile max and assign the Pmax of the added plant to be the profile max (we could transform the normalized profile into MWh via such Pmax later in the workflow). Then assign the pnorm to be the capacity of the added dcline in the subsystem. What do you think @FabianHofmann ?

Done in last commit:

>>> from powersimdata.input.converter.pypsa_to_profiles import get_pypsa_gen_profile
>>> from powersimdata.network.constants.resource import get_resource
>>> from powersimdata.network.europe_tub.model import TUB
>>> profile2carrier = get_resource("europe_tub")["group_profile_resources"]
>>> tub = TUB("Europe", reduction=1024)
Title: PyPSA-Eur: An Open Optimisation Model of the European Transmission System (Dataset)
Keywords: power system model, capacity expansion model, energy system model, electricity system model, python
Publication date: 2022-09-10
DOI: 10.5281/zenodo.6913032
Total size: 2075.6 MB

Link: https://zenodo.org/api/files/7528030f-2993-4f54-83e4-9fdd223b9ba5/networks.zip   size: 2075.6 MB
networks.zip is already downloaded correctly.
All files have been downloaded.
>>> tub.build()
INFO:pypsa.io:Imported network elec_s_1024_ec.nc has buses, carriers, generators, lines, links, loads, storage_units, stores
>>> profile = get_pypsa_gen_profile(tub.network, profile2carrier)
>>> profile["hydro"].max().loc[profile["hydro"].columns.str.contains("hydro")]
ES1 25 hydro    1.0
BG1 10 hydro    1.0
SE2 44 hydro    1.0
NO2 0 hydro     1.0
NO2 12 hydro    1.0
               ... 
SE2 15 hydro    1.0
ES1 37 hydro    1.0
BG1 5 hydro     1.0
ME1 0 hydro     1.0
NO2 11 hydro    1.0
Length: 267, dtype: float64
>>> profile["hydro"].max().loc[profile["hydro"].columns.str.contains("hydro")].max()
1.0

@FabianHofmann
Copy link
Contributor

@BainanXia yes, I think this makes sense. Whether to normalize or not does not really make a difference as long as it is consistent. (Just a comment, the profile is rather added to the generator in the subsystem than to the dcline which connects to the system)

Comment on lines 32 to 52
profile = {}
for p, c in profile2carrier.items():
profile[p] = pd.DataFrame()
for component in ["generators", "storage_units", "stores"]:
if hasattr(network, component):
carrier_in_component = getattr(network, component)
id_carrier = set(carrier_in_component.query("carrier==list(@c)").index)
for t in ["inflow", "p_max_pu"]:
if t in getattr(network, component + "_t"):
ts = getattr(network, component + "_t")[t]
if not ts.empty:
id_ts = set(ts.columns)
idx = list(id_carrier.intersection(id_ts))
norm = ts[idx].max() if t == "inflow" else 1
profile[p] = pd.concat([profile[p], ts[idx] / norm], axis=1)
if len(set(c) - set(carrier_in_component.carrier.unique())):
continue
else:
break
profile[p].rename_axis(index="UTC", columns=None, inplace=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure whether this code is a bit bloated.
Pypsa's get_switchable_as_dense can also be used with 'Store' and 'StorageUnit' and the attribute 'inflow'. Does that make this easier here?

Copy link
Collaborator Author

@rouille rouille Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I could use get_switchable_as_dense to access the profiles. I believe I will have then to use a try/except logic since depending on the Network object, hydro/PHS could be located in the SorageUnit component or Store component. Is that correct? If we focus on PyPSA-Eur only, the code could be greatly simplified. I was trying to be general.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, a hydro/PHS facility could theoretically be a Store component. But it is quite unlikely, since the Store component does not have an "inflow" profile, only the StorageUnit component has. So I'd say for the inflow profile at least, it is not necessary to access stores.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation. I will simplify the logic then.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Can you take a look @FabianHofmann? My understanding is that we will never find an hydro, solar and wind profile in the Store component and hence there is no need to look there.

Copy link
Contributor

@FabianHofmann FabianHofmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! This looks good @rouille

@rouille
Copy link
Collaborator Author

rouille commented Oct 28, 2022

I believe this is good to go. See latest commit where column name of carriers with inflow profiles have been renamed:

>>> profile2carrier = {'hydro': {'hydro', 'ror', 'PHS'}, 'solar': {'solar'}, 'wind': {'onwind', 'offwind-ac', 'offwind-dc'}}
>>> from powersimdata.input.converter.pypsa_to_profiles import get_pypsa_gen_profile
>>> from powersimdata.network.europe_tub.model import TUB
>>> tub = TUB("Europe", reduction=128)
Title: PyPSA-Eur: An Open Optimisation Model of the European Transmission System (Dataset)
Keywords: power system model, capacity expansion model, energy system model, electricity system model, python
Publication date: 2022-09-20
DOI: 10.5281/zenodo.7251657
Total size: 1702.1 MB

Link: https://zenodo.org/api/files/44276d06-8d1e-4dd7-bed7-57f8fbf9f4d5/networks.zip   size: 1702.1 MB
networks.zip is already downloaded correctly.
All files have been downloaded.
>>> tub.build()
INFO:pypsa.io:Imported network elec_s_128_ec.nc has buses, carriers, generators, lines, links, loads, storage_units, stores
>>> profile = get_pypsa_gen_profile(tub.network, profile2carrier)
>>> profile["hydro"]
                     AL1 0 ror  AT1 0 ror  AT1 1 ror  BE1 0 ror  ...  SE2 3 hydro inflow  SE2 4 hydro inflow  SI1 0 hydro inflow  SK1 0 hydro inflow
UTC                                                              ...                                                                                
2013-01-01 00:00:00   0.274540   0.300145   0.300145   0.995894  ...            0.656168            0.656168            0.265311            0.087945
2013-01-01 01:00:00   0.274515   0.300029   0.300029   1.000000  ...            0.654647            0.654647            0.265273            0.087792
2013-01-01 02:00:00   0.274505   0.299936   0.299936   1.000000  ...            0.653922            0.653922            0.265214            0.087755
2013-01-01 03:00:00   0.274627   0.299868   0.299868   1.000000  ...            0.652859            0.652859            0.265212            0.087960
2013-01-01 04:00:00   0.274700   0.299820   0.299820   1.000000  ...            0.652117            0.652117            0.265189            0.088083
...                        ...        ...        ...        ...  ...                 ...                 ...                 ...                 ...
2013-12-31 19:00:00   0.250162   0.368205   0.368205   0.579924  ...            0.491466            0.491466            0.470877            0.130620
2013-12-31 20:00:00   0.250062   0.367945   0.367945   0.577514  ...            0.491266            0.491266            0.470428            0.130693
2013-12-31 21:00:00   0.250067   0.367699   0.367699   0.575495  ...            0.491070            0.491070            0.469969            0.130761
2013-12-31 22:00:00   0.250018   0.367444   0.367444   0.572038  ...            0.490882            0.490882            0.469443            0.130799
2013-12-31 23:00:00   0.250010   0.367188   0.367188   0.567667  ...            0.490702            0.490702            0.468825            0.130814

[8760 rows x 152 columns]

Copy link
Collaborator

@BainanXia BainanXia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@rouille rouille merged commit 2003a93 into ben/import Nov 4, 2022
@rouille rouille deleted the ben/profile branch November 4, 2022 21:33
rouille added a commit that referenced this pull request Nov 5, 2022
Extract profiles from PyPSA's Network objects
@rouille rouille restored the ben/profile branch November 5, 2022 04:06
rouille added a commit that referenced this pull request Nov 10, 2022
Extract profiles from PyPSA's Network objects
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants