Create a data_maturity label for EIA data
#1855
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cmgosnell
commented
Aug 22, 2022
these columns being anywhere in the non-m eia860 enables the columns to exist in the tables without the eia860m being loaded. Moving them from the generators -> generators_existing is slightly aspirational. We hope they'll add them in their next non-monthly data updates so this just adds the empties here where we hope they'll show up later.
Codecov Report
@@ Coverage Diff @@
## dev #1855 +/- ##
=======================================
+ Coverage 83.0% 83.2% +0.1%
=======================================
Files 65 65
Lines 7327 7518 +191
=======================================
+ Hits 6088 6255 +167
- Misses 1239 1263 +24
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
|
This looks like some hackish stuff happening. Can you explain more in the PR comment what all you had to do to make it work? |
zaneselvans
requested changes
Aug 23, 2022
zaneselvans
reviewed
Aug 23, 2022
| annual release and should be used with caution. :pr:`1834` | ||
| annual release and should be used with caution. We also integrated a ``data_maturity`` | ||
| column and related ``data_maturities`` table into most of the EIA data tables in | ||
| order to alter users to the level of finality of the data. :pr:`1834` :pr:`1855` |
zaneselvans
requested changes
Aug 23, 2022
zaneselvans
approved these changes
Aug 24, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We added the early release eia data in #1834. This PR adds a
data_maturitycolumn into the EIA data tables to communicate with users about where the data comes from in the hope of communicating how much trust they should place in the permanence of the data.Things that happened in here:
add_data_maturitymethod into the standard excel extractor that 🥁 adds adata_maturitycolumn. This is deployed inprocess_raw(Question: the generic version ofprocess_rawis customized in all of the datasets... shouldadd_data_maturitybe called directly inGenericExtractor.extractinstead?)utility_id_eia. This PR actually sets this up in a non-hacky way (imo). For each entity inENTITIESwe add anot_to_drop_colslist which is employed in the harvesting process to effectively avoid dropping those columns during harvesting.boiler_fuel_eia923without some additional work (see enable non-data columns in aggregated boiler_fuel_eia923 table #1847)