Skip to content

Conversation

RussTreadon-NOAA
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Aug 28, 2024

Description

This PR updates g-w components which handle JEDI radiance bias correction files. Currently, JEDI radiance bias correction files are processed at the satellite_sensor. This PR replaces these multiple files with a single radiance bias correction tarball.

Resolves #2862

Refs:

Type of change

  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES

How has this been tested?

  • Clone, build, and cycled tests on WCOSS2 (Cactus)
  • Clone, build, and cycled tests on Hera
  • Clone, build, and cycled tests on Hercules

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • New and existing tests pass with my changes

@RussTreadon-NOAA
Copy link
Contributor Author

This PR is opened as a draft to allow others to comment on the changes while we wait for g-w issue #2869 and GDASApp PR #1267 to be closed.

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Aug 28, 2024
@RussTreadon-NOAA
Copy link
Contributor Author

Hera tests

Install GDASApp PR #1267 with feature/radbcor at 554c55a inside g-w feature/radbcor at 5985e9f.

All 48 test_gdasapp tests Passed (see comment here). All jobs in g-w C96C48_ufs_hybatmDA CI successfully run to completion.

Hera(hfe05):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prtest$ rocotostat -d prtest.db -w prtest.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Sep 07 2024 13:35:14    Sep 07 2024 13:55:14
202402240000        Done    Sep 07 2024 13:35:14    Sep 09 2024 17:25:13

The changes in this PR are ready for review.

@RussTreadon-NOAA RussTreadon-NOAA marked this pull request as ready for review September 9, 2024 17:29
@RussTreadon-NOAA
Copy link
Contributor Author

g-w CI tests

g-w feature/radbcor at 386313b along with GDASApp feature/radbcor at 554c55a have been installed on Hera, Hercules, and WCOSS2 (Dogwood).

The following g-w CI is being run on each machine.

  • Hera: C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA, C48mx500_3DVarAOWCDA
  • Hercules: C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA
  • WCOSS2 (Dogwood): C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA

The Hera and Hercules tests are complete. The indicated g-w CI configurations successfully ran to completion on these machines.

The Dogwood g-w CI configurations are still running.

@RussTreadon-NOAA
Copy link
Contributor Author

g-w CI tests on WCOSS2 (Dogwood)

The C96C48_hybatmDA, C96C48_ufs_hybatmDA, and C96C48_hybatmaerosnowDA g-w CI tests successfully ran to completion on Dogwood.

@aerorahul aerorahul added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Sep 30, 2024
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Sep 30, 2024
@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C48_ATM FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C48_ATM_5f7ceebe/logs/2021032312/gfsmetpg2o1.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Sep 30, 2024
@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C48_ATM FAILED on Hercules in Build# 3 in
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/EXPDIR/C48_ATM_5f7ceebe

@RussTreadon-NOAA
Copy link
Contributor Author

I do not run C48_ATM because this test does not exercise DA. Log file gfsmetpg2o1.log contains the message

09/30 10:50:15.619 metplus (met_util.py:247) INFO: Check the log file for more information: /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48_ATM_5f7ceebe/gfs.2021032312/met\
pg2o1.716680/grid2obs_step1/metplus_output/logs/C48_ATM_5f7ceebe/master_metplus_grid2obs_step1_upper_air_gatherbyVSDB_for20210323_runon20240930105013.log
srun: error: hercules-01-01: task 0: Exited with exit code 1

The indicated log file has

DEBUG 1: Default Config File: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/intel/2021.9.0/met-9.1.3-76sujnp/share/met/co\
nfig/STATAnalysisConfig_default
DEBUG 1: User Config File: /work2/noaa/stmp/CI/HERCULES/2875/gfs/sorc/verif-global.fd/parm/metplus_config/metplus_use_cases/METplusV3.1/grid2obs/met_config/METV9.1/STA\
TAnalysisConfig
ERROR  :
ERROR  : process_search_dirs() -> no STAT files found in the directories specified!
ERROR  :
ERROR  :
ERROR  : main() -> encountered an error value of 1.  Calling clean_up() and usage() before exiting.
ERROR  :

We need someone from VPPGB to examine the log files to figure out what went wrong.

@aerorahul
Copy link
Contributor

I do not run C48_ATM because this test does not exercise DA. Log file gfsmetpg2o1.log contains the message

09/30 10:50:15.619 metplus (met_util.py:247) INFO: Check the log file for more information: /work/noaa/stmp/mterry/HERCULES/RUNDIRS/C48_ATM_5f7ceebe/gfs.2021032312/met\
pg2o1.716680/grid2obs_step1/metplus_output/logs/C48_ATM_5f7ceebe/master_metplus_grid2obs_step1_upper_air_gatherbyVSDB_for20210323_runon20240930105013.log
srun: error: hercules-01-01: task 0: Exited with exit code 1

The indicated log file has

DEBUG 1: Default Config File: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/intel/2021.9.0/met-9.1.3-76sujnp/share/met/co\
nfig/STATAnalysisConfig_default
DEBUG 1: User Config File: /work2/noaa/stmp/CI/HERCULES/2875/gfs/sorc/verif-global.fd/parm/metplus_config/metplus_use_cases/METplusV3.1/grid2obs/met_config/METV9.1/STA\
TAnalysisConfig
ERROR  :
ERROR  : process_search_dirs() -> no STAT files found in the directories specified!
ERROR  :
ERROR  :
ERROR  : main() -> encountered an error value of 1.  Calling clean_up() and usage() before exiting.
ERROR  :

We need someone from VPPGB to examine the log files to figure out what went wrong.

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

@WalterKolczynski-NOAA
Copy link
Contributor

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

Are you talking about @DavidHuber-NOAA's PR that removed a race condition (#2907) or mine? Because mine hasn't been merged yet.

@aerorahul
Copy link
Contributor

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

Are you talking about @DavidHuber-NOAA's PR that removed a race condition (#2907) or mine? Because mine hasn't been merged yet.

That is the one. Is this failure related to that feature? I think there was work being done in the verif-global repo. If that is the case, we should acknowledge and move ahead.

@WalterKolczynski-NOAA
Copy link
Contributor

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

Are you talking about @DavidHuber-NOAA's PR that removed a race condition (#2907) or mine? Because mine hasn't been merged yet.

That is the one. Is this failure related to that feature? I think there was work being done in the verif-global repo. If that is the case, we should acknowledge and move ahead.

Agreed, since the failure is unrelated to this PR, I say we just merge.

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit e2f56f5 into NOAA-EMC:develop Sep 30, 2024
8 of 11 checks passed
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/radbcor branch September 30, 2024 17:02
@malloryprow
Copy link
Contributor

@malloryprow @WalterKolczynski-NOAA Is this related to a change to delay met processing?

Are you talking about @DavidHuber-NOAA's PR that removed a race condition (#2907) or mine? Because mine hasn't been merged yet.

That is the one. Is this failure related to that feature? I think there was work being done in the verif-global repo. If that is the case, we should acknowledge and move ahead.

It does appear to be related

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96C48_hybatmDA_5f7ceebe/logs/2021122106/gdasanaldiag.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C96_atm3DVar FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gdasanaldiag.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f102.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f108.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f111.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f114.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gdasanaldiag.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 3 in
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/EXPDIR/C96C48_hybatmDA_5f7ceebe

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C96_atm3DVar FAILED on Hercules in Build# 3 in
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/EXPDIR/C96_atm3DVar_5f7ceebe

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 3 with error logs:

/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C48_S2SWA_gefs_5f7ceebe/logs/2021032312/wave_post_pnt_mem000.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Sep 30, 2024

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 3 in
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/EXPDIR/C48_S2SWA_gefs_5f7ceebe

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Sep 30, 2024
@emcbot
Copy link

emcbot commented Sep 30, 2024

CI Failed on Hercules in Build# 3
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2875


Experiment C48_ATM_5f7ceebe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Sep 30 10:54:04 CDT 2024
Experiment C48_ATM_5f7ceebe Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C48_ATM_5f7ceebe/logs/2021032312/gfsmetpg2o1.log
Experiment C96_atm3DVar_5f7ceebe Terminated with 0
FAIL
FAIL tasks failed and 14 dead at Mon Sep 30 12:18:41 CDT 2024
Experiment C96_atm3DVar_5f7ceebe Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gdasanaldiag.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f084.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f087.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f090.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f093.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f096.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f099.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f102.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f105.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f108.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f111.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f114.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f117.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122100/gfsatmos_prod_f120.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gdasanaldiag.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f084.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f087.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f090.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f093.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f096.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f099.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f102.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f105.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f108.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f111.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f114.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f117.log
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96_atm3DVar_5f7ceebe/logs/2021122106/gfsatmos_prod_f120.log
Experiment C96C48_hybatmDA_5f7ceebe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Sep 30 12:18:48 CDT 2024
Experiment C96C48_hybatmDA_5f7ceebe Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C96C48_hybatmDA_5f7ceebe/logs/2021122106/gdasanaldiag.log
Experiment C48_S2SWA_gefs_5f7ceebe Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Sep 30 12:25:28 CDT 2024
Experiment C48_S2SWA_gefs_5f7ceebe Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2875/RUNTESTS/COMROOT/C48_S2SWA_gefs_5f7ceebe/logs/2021032312/wave_post_pnt_mem000.log
Experiment C48_S2SW_5f7ceebe Completed 1 Cycles: *SUCCESS* at Mon Sep 30 12:48:57 CDT 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Process JEDI radiance files as a single tarball