Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reformat qlat inputs #595

Merged
merged 4 commits into from
Dec 6, 2022

Conversation

shorvath-noaa
Copy link
Contributor

Added capability to convert nexus qlat csv files output by ngen (single file per nexus with all timesteps) into hourly parquet files (one file for every hour containing all nexus points) that conforms to t-route's current method of reading qlat input data.

Additions

  • If file filter for qlat input data is "nex-", these csv files will be read in and converted into hourly parquet files. These files are saved in a directory designated by new "binary_nexus_file_folder" parameter under "forcing_parameters" in the config file.
  • Updates the "nexus_input_folder" and "nexus_file_pattern_filter" parameters to reflect the new parquet directory/files
  • Runtime errors are raised if the binary_nexus_file_folder is not provided, does not exist, or is not empty.
  • Update method of reading qlat files to be able to read either csv or parquet files.

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Visually tested in supported browsers and devices (see checklist below 👇)
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

  • Windows
  • Linux
  • Browser

Accessibility

  • Keyboard friendly
  • Screen reader friendly

Other

  • Is useable without CSS
  • Is useable without JS
  • Flexible from small to large screens
  • No linting errors or warnings
  • JavaScript tests are passing

kumdonoaa
kumdonoaa previously approved these changes Dec 5, 2022
raise(RuntimeError("Output binary qlat folder supplied in config does not exist"))
elif len(os.listdir(binary_folder)) != 0:
raise(RuntimeError("Output binary qlat folder supplied in config is not empty"))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All files in binary_folder should be cleanly deleted for a new simulation. When Delete option is used, one hidden file, i.e., .ipynb_checkpoints, still remains to give the error of line 53. It is recommended to use "rm -r * .*" to cleanly delete all files. Other than this, this PR successfully ran with nextgen-generated csv files.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new change at line 52 solved my initial issue. Well done!!

raise(RuntimeError("Output binary qlat folder supplied in config does not exist"))
elif len(os.listdir(binary_folder)) != 0:
raise(RuntimeError("Output binary qlat folder supplied in config is not empty"))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new change at line 52 solved my initial issue. Well done!!

@shorvath-noaa shorvath-noaa merged commit 7571e1e into NOAA-OWP:master Dec 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants