Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop]: Update python docstrings and generate preliminary technical documentation #1131

Merged
merged 94 commits into from
Oct 2, 2024

Conversation

gspetro-NOAA
Copy link
Collaborator

@gspetro-NOAA gspetro-NOAA commented Sep 11, 2024

DESCRIPTION OF CHANGES:

Initial implementation of SRW App technical documentation. Adds a section called "Technical Documentation" to the SRW App docs.

This PR also removes outdated/redundant text files in doc (i.e., RUNTIME and INSTALL).

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

TESTS CONDUCTED:

Fundamental tests pass on Hera.

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              11.82
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               8.87
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              24.91
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024091  COMPLETE              42.96
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240916200  COMPLETE              26.50
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024091620042  COMPLETE              44.27
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             159.33
  • derecho.intel
  • gaea.intel
  • hera.gnu
  • hera.intel
  • hercules.intel
  • jet.intel
  • orion.intel
  • wcoss2.intel
  • NOAA Cloud (indicate which platform)
  • Jenkins
  • fundamental test suite
  • comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

N/A

DOCUMENTATION:

All documentation

ISSUE:

Resolves Issue #1130
Resolves Issue #340

CHECKLIST

  • My code follows the style guidelines in the Contributor's Guide
  • I have performed a self-review of my own code using the Code Reviewer's Guide
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes do not require updates to the documentation (explain).
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published N/A

Copy link
Collaborator

@MichaelLueken MichaelLueken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gspetro-NOAA -

Thanks for adding a preliminary update to add technical documentation to the majority of the ush and tests/WE2E python scripts! Some of the input arguments and return definitions are lower case instead of capitalized, so I noted the ones that I saw during my initial review. A misspelt documenation was also noted during my review, as well as the modification of run_WE2E_tests.sh to run_WE2E_tests.py (the old shell script was removed after run_WE2E_tests.py was complete).

ush/create_aqm_rc_file.py Outdated Show resolved Hide resolved
ush/create_aqm_rc_file.py Outdated Show resolved Hide resolved
ush/create_diag_table_file.py Outdated Show resolved Hide resolved
ush/python_utils/environment.py Outdated Show resolved Hide resolved
ush/python_utils/environment.py Outdated Show resolved Hide resolved
ush/python_utils/environment.py Outdated Show resolved Hide resolved
ush/python_utils/filesys_cmds_vrfy.py Outdated Show resolved Hide resolved
ush/python_utils/run_command.py Outdated Show resolved Hide resolved
ush/retrieve_data.py Outdated Show resolved Hide resolved
ush/run_srw_tests.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@MichaelLueken MichaelLueken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gspetro-NOAA -

Thank you very much for making my requested changes! This work looks good to me.

The AQM and fundamental tests were run and successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
aqm_grid_AQM_NA13km_suite_GFS_v16_20240918090657                   COMPLETE            7843.59
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            7843.59
----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              11.81
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               8.91
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              25.19
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024091  COMPLETE              42.46
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240918162  COMPLETE              26.63
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024091816270  COMPLETE              45.41
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             160.41

Approving PR now.

@EdwardSnyder-NOAA
Copy link
Collaborator

Changes look good to me.

Calculating core-hour usage and printing final summary
----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              26.71
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE              19.77
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              50.03
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024092  COMPLETE              55.93
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240926073  COMPLETE              41.41
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024092607313  COMPLETE              91.24
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             285.09

Detailed summary written to /glade/work/esnyder/pr-1131/expt_dirs/WE2E_summary_20240927090333.txt

@MichaelLueken MichaelLueken added the run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests label Sep 27, 2024
@MichaelLueken
Copy link
Collaborator

The Jenkins tests successfully passed on Gaea, Hercules, and Orion. The Derecho pipeline was down and the tests on Hera GNU, Hera Intel, and Jet were aborted after running over 8 hours.

I'm in the process of running the coverage WE2E tests on these machines now.

@MichaelLueken
Copy link
Collaborator

Moving into October, the NRAL0032 account is no longer useable. Will run the Derecho coverage tests on Hercules. Hera is down for maintenance. Will update PR with Hera Intel coverage test results. Jet WE2E coverage tests are still running this morning.

@MichaelLueken
Copy link
Collaborator

The Derecho coverage WE2E tests were run on Hercules and all tests successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20241001075245                      COMPLETE              29.30
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              72.38
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024100107524  COMPLETE              52.26
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20241001  COMPLETE              55.30
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              14.08
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024100107524  COMPLETE              26.15
pregen_grid_orog_sfc_climo_20241001075250                          COMPLETE              11.52
specify_template_filenames_20241001075252                          COMPLETE              12.39
2019_hurricane_barry_20241001075252                                COMPLETE              46.69
2019_memorial_day_heat_wave_20241001075253                         COMPLETE              46.03
2020_denver_radiation_inversion_20241001075254                     COMPLETE              47.32
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             413.42

@MichaelLueken
Copy link
Collaborator

Both the fundamental and coverage WE2E tests are just sitting in queue on Jet, so I decided to run the Jet WE2E coverage tests on Hera. All tests successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
community_20241002133730                                           COMPLETE              22.63
custom_ESGgrid_20241002133731                                      COMPLETE              25.66
custom_ESGgrid_Great_Lakes_snow_8km_20241002133732                 COMPLETE              20.51
custom_GFDLgrid_20241002133733                                     COMPLETE              11.42
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018_202410  COMPLETE              11.40
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h_20  COMPLETE              83.08
get_from_HPSS_ics_RAP_lbcs_RAP_20241002133735                      COMPLETE              20.04
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20241002133736  COMPLETE             475.19
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              60.26
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE               7.11
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             731.84
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1469.14

The Hera Intel WE2E coverage tests have successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Peru_12km_20240930200120                            COMPLETE              36.51
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2024093  COMPLETE               9.35
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE            1532.58
get_from_HPSS_ics_HRRR_lbcs_RAP_20240930200122                     COMPLETE              17.89
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               7.45
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              24.31
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240930200125  COMPLETE              13.72
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE               9.59
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202409  COMPLETE             456.01
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240930  COMPLETE             606.59
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202409302  COMPLETE             906.17
pregen_grid_orog_sfc_climo_20240930200129                          COMPLETE              10.08
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            3630.25

as well as the Hera GNU WE2E coverage tests:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Central_Asia_3km_20241002122754                     COMPLETE              71.24
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200_202410  COMPLETE              14.17
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20241002122756              COMPLETE              20.32
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024100212  COMPLETE              76.71
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              30.66
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20241002122  COMPLETE              37.37
long_fcst_20241002122759                                           COMPLETE             136.52
MET_verification_only_vx_20241002122800                            COMPLETE               0.39
MET_ensemble_verification_only_vx_time_lag_20241002122802          COMPLETE               9.90
2019_halloween_storm_20241002122804                                COMPLETE              87.04
2020_jan_cold_blast_20241002122805                                 COMPLETE              85.39
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             569.71

Moving forward with merging this work now.

@MichaelLueken MichaelLueken merged commit 13d3ce5 into ufs-community:develop Oct 2, 2024
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Technical Documentation to SRW App docs Text files in doc/ are outdated and redundant
4 participants