Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix / clean up EIA-930 timestamp corrections #301

Open
grgmiller opened this issue May 18, 2023 · 0 comments
Open

Fix / clean up EIA-930 timestamp corrections #301

grgmiller opened this issue May 18, 2023 · 0 comments
Labels
code hygene code readability and structure consumed emissions Improve consumed emission calculation data cleaning Cleaning and standardizing data generation data accuracy/completeness of generation data hourly profiles Accuracy of hourly profile imputation

Comments

@grgmiller
Copy link
Collaborator

The eia930.manual_930_adjust() function adjusts incorrect timestamps in the raw EIA-930 data:

def manual_930_adjust(raw: pd.DataFrame):

However, while poking around in this function as part of #300, it became clear that there are some potential issues with this function that need to be addressed:

  1. We currently are repeating adjustment functions and the code is quite messy and unorganized. It would be great if we could write a standardized functions to do these adjustments based on a specific start_datetime, end_datetime, and adjustment, and define a dictionary of all of these adjustment parameters, like in https://github.com/singularity-energy/scrapers/blob/master/scrapers/eia_genfuelmix.py#L87.
  2. When shifting the data to correct the timestamps, it matters whether you are shifting the data columns, or shifting the timestamp. For example, if you want to convert from a start of hour to end of hour convention, if you are shifting the data you need to shift by +1, but if you are shifting the timestamp you need to shift by -1 hour. It looks like generally we are shifting the data, but for the PJM interchange adjustment, it appears that we are shifting the datetime index instead and may be using the incorrect sign convention.
  3. The correction for TEPC data is out of date and needs to be updated (ie the issue has been corrected in the raw EIA-930 data).
  4. We should check that the timestamp corrections we are applying here are consistent with the timestamp corrections we are applying in the grid carbon API.
@grgmiller grgmiller added data cleaning Cleaning and standardizing data code hygene code readability and structure hourly profiles Accuracy of hourly profile imputation consumed emissions Improve consumed emission calculation generation data accuracy/completeness of generation data labels May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code hygene code readability and structure consumed emissions Improve consumed emission calculation data cleaning Cleaning and standardizing data generation data accuracy/completeness of generation data hourly profiles Accuracy of hourly profile imputation
Projects
None yet
Development

No branches or pull requests

1 participant