rrfs_ci: Updates from develop #403
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…hysics seed generation script (ufs-community#704) ## DESCRIPTION OF CHANGES: Add missing user-defined options for tendency-based stochastic physics and fix the ensemble-based seed generation script to work regardless of whether stochastic physics is turned on or not. ## TESTS CONDUCTED: Tested on Hera using the following WE2E configurations with and without stochastic physics: config.grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh config.community_ensemble_2mems.sh ## ISSUE (optional): [Issue ufs-community#702](ufs-community#702) ## CONTRIBUTORS (optional): Thanks to @mkavulich and @chan-hoo for finding this problem.
… update NAM HPSS settings and WE2E tests (ufs-community#707) * Change to netcdf4 when using the NA 3-km domain * Update HPSS paths for NAM data * Update NAM HPSS locations and dates for WE2E tests. * Remove lines from merge.
…WE2E_test.sh script (ufs-community#711)
…nco we2e tests (ufs-community#713) * Update machine script for orion * Update machine script for wcoss_dell_p3 * Update we2e run script for wcoss and orion * Reorganize nco we2e tests * remove machine based logic * Add symlink for nco inline post test
…X) (ufs-community#726) * Grid-stat and point-stat run scripts. * Stand-alone scripts for verification. * Added comments to gridvx scripts. * Added qsub_job.sh and added comments to provide context on running Vx.
…#717) * Allow multiple template names * parameterize file_TMPL_FN and add a we2e test * Increase maxtries_task for make_grid/orog/sfc_climo * Modify file name and description
…_ics task modification, and tweaks to stochastic physics namelist settings (ufs-community#721) * Modify RRFS North America 3- and 13-km domain configuration and WE2E test. * Change sotyp_from_climo to "true" based on operational RAP grib2 files. * Update for changes to stochastic physics namelist options. * Check for DO_ENSEMBLE="TRUE" when running ensemble verification and turn of VX when running in NCO mode. * Revert to 3-km domain. * Remove commented-out GFDL grid for the RRFS_NA_13km domain * Add RRFS_NA_13km WE2E test * Changes to comments.
…mmunity#725) * Add 3km grid over Indianapolis. This is about 600km x 600km in extent (200 x 200 grid points). It is intended for use in the WE2E tests. * Edit comments.
These changes hook in the Python-based data ingest tool, replacing the previous scripts that handled this work as part of the get_extrn_mdl_file task. No attempt was made in this PR to replace the NOMADS fetching script with the Python utility, but the NOMADS data location has been added to the data_locations.yml file. The functionality to write the data summary file has also been added to the Python tool to match the capabilities of the existing workflow tools.
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a1.ncep.noaa.gov> Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v72a1.ncep.noaa.gov> Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a3.ncep.noaa.gov> Co-authored-by: chan-hoo <chan-hoo.jeon@noaa.gov>
…nity#734) * Updates to port regional workflow to gaea * Temp change with -v as batch option * new fixes for gaea/slurm * Updated time for make lbcs * added TEST data directory path * Update gaea.sh * fixes for PR
…ity#740) ## DESCRIPTION OF CHANGES: The script/function `get_WE2Etest_names_subdirs_descs.sh` (which is called from `run_WE2E_tests.sh` if needed) creates a CSV (Comma-Separated Value) file named `WE2E_test_info.csv` that contains information about the WE2E tests. Currently, this CSV file contains only 3 columns: the test name, the names of any alternate names for the test, and the test description. In order to have a more complete summary of the WE2E tests, this PR modifies `get_WE2Etest_names_subdirs_descs.sh` so that additional information is included in the CSV file. This additional information consists of the values of the following experiment variables for each test: ``` PREDEF_GRID_NAME CCPP_PHYS_SUITE EXTRN_MDL_NAME_ICS EXTRN_MDL_NAME_LBCS DATE_FIRST_CYCL DATE_LAST_CYCL CYCL_HRS INCR_CYCL_FREQ FCST_LEN_HRS LBC_SPEC_INTVL_HRS NUM_ENS_MEMBERS ``` In addition, the script uses this information to calculate the number of times each test calls the forecast model (e.g. if the test uses 3 different cycle dates, then the forecast model will be called 3 times; if it is an ensemble test for a single cycle, the test will call the forecast model as many times as the number of ensemble members). ## TESTS CONDUCTED: The script `run_WE2E_tests.sh` was called that in turn calls `get_WE2Etest_names_subdirs_descs.sh`. This created a new CSV file that contained the new fields (columns). The CSV file was imported into Google Sheets (using "|" as the field/column separator) and looked correct. ## DOCUMENTATION: The documentation is for the most part already within the `get_WE2Etest_names_subdirs_descs.sh`. This PR slightly modifies that documentation to update it.
* update vertical structure of NCO mode * update sample script for nco * Fix typo on write component of new RRFS CONUS
* Updated the default CCPP physics option to FV3_GFS_v16 * Updated the default CCPP physics option to FV3_GFS_v16 in config_defaults.sh Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>
* Workflow in python starting to work. * Use new python_utils package structure. * Some bug fixes. * Use uppercase TRUE/FALSE in var_dfns * Use config.sh by default. * Minor bug fixes. * Remove config.yaml * Update to the latest develop * Remove quotes from numbers in predef grid. * Minor bug fix. * Move validity checker to the bottom of setup * Add more unit tests. * Update with python_utils changes. * Update to latest develop additions (Need to re-run regression test) * Use set_namelist and fill_jinja_template as python functions. * Replace sed regex searches with python re. * Use python realpath. * Construct settings as dictionary before passing to fill_jinja and set_namelist * Use yaml for setting predefined grid parameters. * Use xml parser for ccpp phys suite definition file. * Remove more run_command calls. * Simplify some func argument processing. * Move different config format parsers to same file. * Use os.path.join for the sake of macosx * Remove remaining func argument processing via os.environ. * Minor bug fix in set_extrn_mdl_params.sh * Add suite defn in test_data. * Minor fixes on unittest on jet. * Simplify boolean condition checks. * Include old in renaming of old directories * Fix conflicting yaml !join tag for paths and strings. * Bug fix with setting sfcperst dict. * Imitate "readlink -m" with os.path.realpath instead of os.readlink * Don't use /tmp as that is shared by multiple users. * Bug fix with cron line, maintain quotes around TRUE/FALSE. * Update to latest develop (untested) * Bug fix with existing cron line and quotes. * Bug fix with case-sensitive MACHINE name, and empty EXPT_DIR. * Update to latest develop * More updates. * Bug fix thanks to @willmayfield! Check both starting/ending characters are brackets for shell variable to be considered an array. * Make empty EXPT_BASEDIR workable. * Update to latest develop * Update in predef grid. * Check f90nml as well. Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>
…fs-community#750) * Fix typo and failure on wcoss * fix new line issue on wcoss dell * remove capture_output * Get USER from environment Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>
## DESCRIPTION OF CHANGES: Added two new WE2E config files for the Sub-CONUS Indianapolis domain to support the upcoming SRW release. In addition, modified the external data used in the `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` to match more common datasets used in the WE2E testing process. ## TESTS CONDUCTED: Successfully ran the new WE2E tests (`config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh`, `config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh`) and `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` on NOAA Parallel Works AWS instance. ## DEPENDENCIES: None. ## DOCUMENTATION: No documentation changes are required.
…ponent parameters (ufs-community#733) * Added a fixed WoF grid and the python tool to determine the write component parameters * Update set_predef_grid_params.sh * Renamed file as recommended and removed unused lines * Modified comment Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com> Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>
* change env to mod * update we2e script
## DESCRIPTION OF CHANGES: * Modifications to `run_WE2E_tests.sh`: * Add examples to help/usage statement * Modifications to `check_expts_status.sh`: * Add arguments list that can be processed by `process_args` * Add new optional arguments: `num_log_lines`, `verbose` * Include a help/usage message ## TESTS CONDUCTED: * Ran `run_WE2E_tests.sh --help` from the command line and got the expected help message. * Ran `check_expts_status.sh --help` from the command line and got the expected help message. * Used `run_WE2E_tests.sh` to run a set of 2 WE2E tests -- works as expected. * Used `check_expts_status` to check on the status of the 2 tests run above and got the expected status message. ## DEPENDENCIES: PR #[241](ufs-community/ufs-srweather-app#241) ## DOCUMENTATION: A lot of this PR is documentation in the scripts. There is an accompanying documentation PR #[241](ufs-community/ufs-srweather-app#241) into ufs-srweather-app.
…nd LBC data retrieval (ufs-community#744)
…community#756) ## DESCRIPTION OF CHANGES: 1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR ufs-community#725 ). 2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context). ## TESTS CONDUCTED: 1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does). 2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).
…lues (ufs-community#759) * Modify RRFS North America 3- and 13-km domain configuration and WE2E test. * Modify default ISEED values for SPP * Fix grid in WE2E test
* update python scripts
…fs-community#775) * update input namelist of chgres_cube * update diag_table templates * update scripts * back to original * specify miniconda version on Jet
…odel (ufs-community#796) * update new variables introduced in uwm * update python script * update template for HRRR suite
…mmunity#779) * remove unused module loads for MET * update met version in machine files * changes to pointstat/pb2nc bufr mapping * add NA to retain bufr variables without qc markers * updated bufr variable list * change bufr var list * Change versions to MET 10.1.1 and METplus 4.1.1
…munity#803) Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>
…ity#787) ## DESCRIPTION OF CHANGES: For clarity, rename variables related to GFDLgrid-type grids: 1) Rename `GFDLgrid_RES` to `GFDLgrid_NUM_CELLS`. 2) Rename `GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES` to `GFDLgrid_USE_NUM_CELLS_IN_FILENAMES`. 3) Rename WE2E tests on GFDLgrid-type custom grids according to variable renaming above. Related improvements: * In `link_fix.sh`, remove creation of unnecessary symlink when using a GFDLgrid-type grid with `GFDLgrid_USE_NUM_CELLS_IN_FILENAMES` is set to `"FALSE"`. * In WE2E test configuration files that use custom grids, move the definition of the variable `POST_OUTPUT_DOMAIN_NAME` to the beginning of the custom grid definition because it is really the name of the custom native grid, and it's most appropriate to place that at the beginning of the grid definition section. * Reduce `WTIME_MAKE_OROG` from 1 hour to 20 minutes since on Hera, 1 hour causes the task to wait quite a while in the queue. This used to be 20 minutes but was changed to 1 hour because 20 minutes was not sufficient to complete the task on `WCOSS_DELL_P3`. Thus, reset it to 1 hour only in the machine file for `WCOSS_DELL_P3`. * Minor fixes to comments in python plotting scripts. * Edits to other comments. ## TESTS CONDUCTED: Ran the four WE2E tests that use custom grids (listed below) on Hera. All were successful. * `custom_ESGgrid` * `custom_GFDLgrid` * `custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE` (previously `custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE`) * `custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_TRUE` (previously `custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE`) ## DEPENDENCIES: PR #[310](ufs-community/ufs-srweather-app#310) in `ufs-srweather-app`. ## DOCUMENTATION: Updates to documentation are in PR #[310](ufs-community/ufs-srweather-app#310) in `ufs-srweather-app`.
christinaholtNOAA
requested review from
robgpita,
christopherwharrop-noaa and
venitahagerty
as code owners
venitahagerty
added
ci-hera-intel-WE
|
Machine: hera
|
|
Found this message in build.out on Hera: |
venitahagerty
added
ci-jet-intel-WE
|
Machine: jet
|
This is a bug fix for running the WE2E tests on Hera. The non-existence of this path is causing failures after the path has become unreadable.
* add metplus paths * add run_vx.local for jet Co-authored-by: Edward Snyder <Edward.Snyder@noaa.com>
robgpita
added
the
ci-aws-intel-build
|
The build test passed on PCluster. @venitahagerty, I wonder why it couldn't find |
christinaholtNOAA
added
ci-hera-intel-WE
venitahagerty
removed
ci-hera-intel-WE
|
Machine: hera |
|
Machine: jet |
|
This PR has gotten a bit convoluted after I squashed a previous merge from ufs/develop. Opening a new one to start fresh. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DESCRIPTION OF CHANGES:
Update to the top of ufs-community develop.
There are some wonky looking things in this PR because I squashed the last merge accidentally. This PR should not be squashed.
TESTS CONDUCTED:
None
DEPENDENCIES:
PR #165 in NOAA-GSL/ufs-srweather-app