Transform and harvesting the eia860 Energy Storage table

[aesharpe]

Overview

Closes #3506 and #3536

What problem does this address?

• Adds a transformer for the EIA860 energy storage table
• Puts this table through the harvesting process

What did you change?

• Added a transformer for the storage table
• Added new fields for the storage table
• Added coding metadata for core_eia__codes_storage_technology_types and core_eia__codes_storage_enclosure_types
• Updated some column names to fix typos, better reflect column contents, naming protocols, and existing column names
• Add the core_eia860__yearly_generators_energy_storage to the list of finished EIA assets and resource metadata

Notes

• I noticed that some of the columns that get dropped from the specific generator tables have values that aren't in the general generators table. We should probably do a comparison before we drop them, but that is OOS.
• Looks like the boolean columns are coming out as 1/0 values in the core tables, need to figure out where to add the dtype fix to stop this.

Testing

How did you make sure this worked? How can a reviewer verify this?

• Still need to run all the tests

To-do list

Beta Give feedback

1. If updating analyses or data processing functions: make sure to update or write data validation tests (e.g., test_minmax_rows())
2. Update the release notes: reference the PR and related issues.
3. Ensure docs build, unit & integration tests, and test coverage pass locally with make pytest-coverage (otherwise the merge queue may reject your PR)
4. Review the PR yourself and call out any questions or issues you have
5. For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
6. For significant ETL, data coverage or analysis changes, once make pytest-coverage passes, ensure the full ETL runs locally and run data validation tests using make pytest-validate (a ~10 hour run). If you can't run this locally, run the build-deploy-pudl GitHub Action (or ask someone with permissions to). Then, check the logs on the #pudl-deployments Slack channel or gs://builds.catalyst.coop.

[cmgosnell]

It looks like you missed one code step, which is to define those tables as db tables. there are many examples of that in metadata.resources.eia. This could be the cause of your pydantic error.

You also need to add the _core table in the tables to be harvested (see comment in transform.eia

I noticed that some of the columns that get dropped from the specific generator tables have values that aren't in the general generators table. We should probably do a comparison before we drop them, but that is OOS.

I'm not sure what you mean here. This maybe could be related to the previous harvest comment.

Looks like the boolean columns are coming out as 1/0 values in the core tables, need to figure out where to add the dtype fix to stop this.

Are you seeing this in the _core table or in the core table? the core table is definitely getting dtyped with the dypes defined in the schema.

(i haven't tried to build these assets yet)

[aesharpe]

Are you seeing this in the _core table or in the core table? the core table is definitely getting dtyped with the dypes defined in the schema.

Yeah, both tables just have 0 or 1 values instead of True/False

[aesharpe]

Yeah, both tables just have 0 or 1 values instead of True/False

Realized that this is because I was viewing them in SQL where the dtypes were dropped

[cmgosnell]

I second zane's suggestion to remove the _application suffix but that is not blocking for me. also one docs comment that is also not blocking.

[zaneselvans]

Yeah, SQLite stores booleans as integers. It's generally kinda lazy about dtypes.

If you want to see rich dtypes, you can always read from the parquet files, which is what you'll get if you use something like

raw_weather = defs.load_asset_value(AssetKey("raw_gridpathratoolkit__daily_weather"))

[aesharpe]

I second zane's suggestion to remove the _application suffix but that is not blocking for me. also one docs comment that is also not blocking.

Happy to change if it's 2:1

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB