Use submodules rather than ecbuild_bundle git #811

CoryMartin-NOAA · 2023-12-18T21:34:30Z

This PR creates git submodules for each external JEDI library rather than using ecbuild_bundle to get the library source code through git.

There is also a new set of scripts to automate the update process of the submodules (both an update and a git add, in separate scripts).

Peppered through here are also some minor changes for CI testing that are needed due to this change + the global-workflow using submodules.

This reverts commit 99a6e88.

… into feature/submodule

Update soca paths to be consistent with use of submodules

RussTreadon-NOAA · 2024-01-12T13:59:26Z

FYI: users need to load git-lfs (which defaults to git-lfs/3.1.2) on Hercules in order to exercise git-lfs.

…R path (#811)

RussTreadon-NOAA · 2024-01-12T19:30:00Z

Install GDASApp develop and feature/submodule on Hercules inside g-w. Run ctests for both installations. All tests pass in both installations except test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY.

The develop run of the VRFY job failed with

INFO EvaSetup: Running figure driver
Traceback (most recent call last):
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/bin/eva", line 8, in <module>
    sys.exit(main())
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py", line 120, in main
    eva(config_file)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py", line 85, in eva
    figure_driver(eva_dict, data_collections, timing, logger)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/plotting/batch/base/plot_tools/figure_driver.py", line \
43, in figure_driver
    graphics = graphics_section.get('figure_list')
AttributeError: 'list' object has no attribute 'get'
running eva on evayamls/eva_icec_amsr2_south_2018041512.yaml
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 230, in <mo\
dule>
    subprocess.run(['eva', infile], check=True)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['eva', 'evayamls/eva_icec_amsr2_south_2018041512.yaml']' returned non-zero exit status 1.
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY 1705068162 1

The feature/submodule run of the VRFY job didn't get this far. It died with

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[36]: /work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/e\
xgdas_global_marine_analysis_vrfy.py
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrf\
y.py", line 22, in <module>
    import gen_eva_obs_yaml
ModuleNotFoundError: No module named 'gen_eva_obs_yaml'
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY 1705086750 1

RussTreadon-NOAA · 2024-01-12T19:50:00Z

FYI

The update to global-workflow develop at c041968 is likely to cause GDASApp jobs to fail. c041968 . resdet and resens have been replaced with resdetatmos and resensatmos. This was easy to update in feature/submodule. Changes to COMROT and ROTDIR may take more time to sort out.

Tagging @CoryMartin-NOAA , @andytangborn , @guillaumevernieres , @AndrewEichmann-NOAA , and @DavidNew-NOAA

CoryMartin-NOAA · 2024-01-12T20:00:51Z

@RussTreadon-NOAA the develop VRFY issue I think is being handled by @apchoiCMD . The feature branch issue looks familiar, is this something that got 'unresolved' by a merge commit? Looks like it may be a PYTHONPATH or cmake related problem?

RussTreadon-NOAA · 2024-01-12T20:07:44Z

@RussTreadon-NOAA the develop VRFY issue I think is being handled by @apchoiCMD . The feature branch issue looks familiar, is this something that got 'unresolved' by a merge commit? Looks like it may be a PYTHONPATH or cmake related problem?

Agreed regarding a path issue for feature/submodule. I need to first update feature/submodule to handle g-w c041968. The update causes many test_gdasapp to fail.

RussTreadon-NOAA · 2024-01-12T21:07:25Z

@RussTreadon-NOAA the develop VRFY issue I think is being handled by @apchoiCMD . The feature branch issue looks familiar, is this something that got 'unresolved' by a merge commit? Looks like it may be a PYTHONPATH or cmake related problem?

Agreed regarding a path issue for feature/submodule. I need to first update feature/submodule to handle g-w c041968. The update causes many test_gdasapp to fail.

The vrfy jobs runs up to the AttributeError: 'list' object has no attribute 'get' failure observed in develop if I explicitly set the path to ush/eva and ush/soca in PYTHONPATH via the TEST section manually added to run_jjobs.sh

##TEST
PYTHONPATH=/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/ush/eva:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/ush/soca:${PYTHONPATH}
##TEST
PYTHONPATH=${HOMEgfs}/ush/python/wxflow/src:${PYTHONPATH}

The PYTHONPATH generated by the ctest is

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[30]: PYTHONPATH=/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/soca:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/:/work2/noaa/da/rtreadon/git/global-\
workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/eva:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../ush/python/wxflow/src:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/ush:/work2/noaa/da/rtreadon/git/global-wor\
kflow/hercules/sorc/gdas.cd/build/gdas/../lib/python3.7:/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/intel/2021.9.0/py-pybind11-2.8.1-2cs7v5h/lib/python3.10/site-packages:/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/intel/2021.9.0/bufr-12.0.1-6pgbiv5/lib64/python3.10/site-packages:/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/intel/2021.9.0/bufr-12.0.1-6pgbiv5/lib/python3.10/site-packages

Note that eva and soca are in the above PTYHTONPATH as

/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/soca
/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/eva

It seems the sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush piece causes problems. The direct path /sorc/gdas.cd/ush works. Why this sensitivity?

apchoiCMD · 2024-01-12T23:32:45Z

@RussTreadon-NOAA the develop VRFY issue I think is being handled by @apchoiCMD . The feature branch issue looks familiar, is this something that got 'unresolved' by a merge commit? Looks like it may be a PYTHONPATH or cmake related problem?

@RussTreadon-NOAA I am handling "vrfy" task on hera and it looks like resolved, so I will submit the PR along with checking it on Orion side, too. @CoryMartin-NOAA just wonder have you updated eva on Orion, too?

RussTreadon-NOAA · 2024-01-13T00:41:41Z

@apchoiCMD , if you have change(s) you would like me to test on Orion, let me know what to change and I'll give it a try.

RussTreadon-NOAA · 2024-01-13T14:18:14Z

Modify HOMEgfs written to config.base via the following change to ush/soca/run_jjobs.py

         # swap a few variables in config.base
-        var2replace = {'HOMEgfs': self.homegfs,
+        self.homegfs_real = os.path.realpath(self.homegfs)
+        var2replace = {'HOMEgfs': self.homegfs_real,
                        'STMP': self.stmp,

With this change the sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush piece of PYTHONPATH is replaced with sorc/gdas.cd/ush. As a result, test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY runs up to the same error which occurs in develop, namely

INFO DataCollections:   hofxQC::seaIceFraction       (float32) | Min=+0.0000e+00, Max=+7.1078e-01, RMS=+1.6852e-01
INFO DataCollections: --------------------------------------------------------------------------------
INFO EvaSetup: Running figure driver
Traceback (most recent call last):
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/bin/eva", line 8, in <module>
    sys.exit(main())
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py", line 120, in main
    eva(config_file)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py", line 85, in eva
    figure_driver(eva_dict, data_collections, timing, logger)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/plotting/batch/base/plot_tools/figure_driver.py", lin\
e 43, in figure_driver
    graphics = graphics_section.get('figure_list')
AttributeError: 'list' object has no attribute 'get'
running eva on evayamls/eva_icec_amsr2_south_2018041512.yaml
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 230, in <module>
    subprocess.run(['eva', infile], check=True)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['eva', 'evayamls/eva_icec_amsr2_south_2018041512.yaml']' returned non-zero exit status 1.
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY 1705154009 1

@apchoiCMD is working on a fix for this error.

The above change to ush/soca/run_jjobs.py was committed to feature/submodule at 695447b

All ctest_gdasapp run within global-workflow with this change in place. 51 out of 52 tests pass on Hercules.

(gdasapp) hercules-login-2:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/build$ ctest -R test_gdasapp
Test project /work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/build
      Start 1325: test_gdasapp_util_coding_norms
 1/52 Test #1325: test_gdasapp_util_coding_norms ........................   Passed    0.81 sec
      Start 1326: test_gdasapp_util_ioda_example
 2/52 Test #1326: test_gdasapp_util_ioda_example ........................   Passed    0.23 sec
      Start 1327: test_gdasapp_util_prepdata
 3/52 Test #1327: test_gdasapp_util_prepdata ............................   Passed    0.79 sec
      Start 1328: test_gdasapp_util_rads2ioda
 4/52 Test #1328: test_gdasapp_util_rads2ioda ...........................   Passed    0.22 sec
      Start 1329: test_gdasapp_util_ghrsst2ioda
 5/52 Test #1329: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.12 sec
      Start 1330: test_gdasapp_util_smap2ioda
 6/52 Test #1330: test_gdasapp_util_smap2ioda ...........................   Passed    0.19 sec
      Start 1331: test_gdasapp_util_smos2ioda
 7/52 Test #1331: test_gdasapp_util_smos2ioda ...........................   Passed    0.13 sec
      Start 1332: test_gdasapp_util_viirsaod2ioda
 8/52 Test #1332: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.22 sec
      Start 1333: test_gdasapp_util_icecamsr2ioda
 9/52 Test #1333: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.21 sec
      Start 1665: test_gdasapp_check_python_norms
10/52 Test #1665: test_gdasapp_check_python_norms .......................   Passed    0.92 sec
      Start 1666: test_gdasapp_check_yaml_keys
11/52 Test #1666: test_gdasapp_check_yaml_keys ..........................   Passed    0.07 sec
      Start 1667: test_gdasapp_jedi_increment_to_fv3
12/52 Test #1667: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    0.45 sec
      Start 1668: test_gdasapp_convert_ewok_yaml
13/52 Test #1668: test_gdasapp_convert_ewok_yaml ........................   Passed    0.25 sec
      Start 1669: test_gdasapp_setup_cycled_exp
14/52 Test #1669: test_gdasapp_setup_cycled_exp .........................   Passed    1.01 sec
      Start 1670: test_gdasapp_convert_bufr_temp_dbuoy
15/52 Test #1670: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    0.39 sec
      Start 1671: test_gdasapp_convert_bufr_salt_dbuoy
16/52 Test #1671: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.40 sec
      Start 1672: test_gdasapp_convert_bufr_temp_mbuoyb
17/52 Test #1672: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.42 sec
      Start 1673: test_gdasapp_convert_bufr_salt_mbuoyb
18/52 Test #1673: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.39 sec
      Start 1674: test_gdasapp_convert_bufr_tesacprof
19/52 Test #1674: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.41 sec
      Start 1675: test_gdasapp_convert_bufr_trkobprof
20/52 Test #1675: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.38 sec
      Start 1676: test_gdasapp_convert_bufr_sfcships
21/52 Test #1676: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.39 sec
      Start 1677: test_gdasapp_convert_bufr_sfcshipsu
22/52 Test #1677: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.39 sec
      Start 1678: test_gdasapp_soca_nsst_increment_to_mom6
23/52 Test #1678: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   45.03 sec
      Start 1679: test_gdasapp_soca_prep
24/52 Test #1679: test_gdasapp_soca_prep ................................   Passed    7.55 sec
      Start 1680: test_gdasapp_soca_run_clean
25/52 Test #1680: test_gdasapp_soca_run_clean ...........................   Passed    0.25 sec
      Start 1681: test_gdasapp_soca_setup_obsproc
26/52 Test #1681: test_gdasapp_soca_setup_obsproc .......................   Passed    0.83 sec
      Start 1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/52 Test #1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   43.60 sec
      Start 1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/52 Test #1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   42.26 sec
      Start 1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/52 Test #1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   42.27 sec
      Start 1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/52 Test #1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.27 sec
      Start 1686: test_gdasapp_soca_copy_scratch
31/52 Test #1686: test_gdasapp_soca_copy_scratch ........................   Passed    2.86 sec
      Start 1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
32/52 Test #1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.26 sec
      Start 1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
33/52 Test #1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   42.48 sec
      Start 1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
34/52 Test #1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....***Failed  367.41 sec
      Start 1690: test_gdasapp_soca_socahybridweights
35/52 Test #1690: test_gdasapp_soca_socahybridweights ...................   Passed  138.30 sec
      Start 1691: test_gdasapp_soca_incr_handler
36/52 Test #1691: test_gdasapp_soca_incr_handler ........................   Passed  138.24 sec
      Start 1692: test_gdasapp_soca_ens_handler
37/52 Test #1692: test_gdasapp_soca_ens_handler .........................   Passed   42.24 sec
      Start 1693: test_gdasapp_land_create_ens
38/52 Test #1693: test_gdasapp_land_create_ens ..........................   Passed    1.97 sec
      Start 1694: test_gdasapp_land_imsproc
39/52 Test #1694: test_gdasapp_land_imsproc .............................   Passed    7.40 sec
      Start 1695: test_gdasapp_land_apply_jediincr
40/52 Test #1695: test_gdasapp_land_apply_jediincr ......................   Passed    2.92 sec
      Start 1696: test_gdasapp_land_letkfoi_snowda
41/52 Test #1696: test_gdasapp_land_letkfoi_snowda ......................   Passed    8.90 sec
      Start 1697: test_gdasapp_convert_bufr_adpsfc_snow
42/52 Test #1697: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    3.02 sec
      Start 1698: test_gdasapp_convert_bufr_adpsfc
43/52 Test #1698: test_gdasapp_convert_bufr_adpsfc ......................   Passed    2.76 sec
      Start 1699: test_gdasapp_convert_gsi_satbias
44/52 Test #1699: test_gdasapp_convert_gsi_satbias ......................   Passed    1.93 sec
      Start 1700: test_gdasapp_setup_atm_cycled_exp
45/52 Test #1700: test_gdasapp_setup_atm_cycled_exp .....................   Passed    1.85 sec
      Start 1701: test_gdasapp_atm_jjob_var_init
46/52 Test #1701: test_gdasapp_atm_jjob_var_init ........................   Passed   49.95 sec
      Start 1702: test_gdasapp_atm_jjob_var_run
47/52 Test #1702: test_gdasapp_atm_jjob_var_run .........................   Passed   74.34 sec
      Start 1703: test_gdasapp_atm_jjob_var_final
48/52 Test #1703: test_gdasapp_atm_jjob_var_final .......................   Passed   42.39 sec
      Start 1704: test_gdasapp_atm_jjob_ens_init
49/52 Test #1704: test_gdasapp_atm_jjob_ens_init ........................   Passed   22.95 sec
      Start 1705: test_gdasapp_atm_jjob_ens_run
50/52 Test #1705: test_gdasapp_atm_jjob_ens_run .........................   Passed  266.39 sec
      Start 1706: test_gdasapp_atm_jjob_ens_final
51/52 Test #1706: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.39 sec
      Start 1707: test_gdasapp_aero_gen_3dvar_yaml
52/52 Test #1707: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.63 sec

98% tests passed, 1 tests failed out of 52

Label Time Summary:
gdas-utils    =   2.92 sec*proc (9 tests)
script        =   2.92 sec*proc (9 tests)

Total Test time (real) = 1536.89 sec

The following tests FAILED:
        1689 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Errors while running CTest

RussTreadon-NOAA · 2024-01-13T14:19:22Z

I have no more planned work on this PR apart from changes requested or suggested by reviewers.

RussTreadon-NOAA · 2024-01-13T18:48:00Z

Orion gw ctests
Install feature/submodule at 695447b in g-w develop at c590476. All ctests pass except the VRFY job

(gdasapp) Orion-login-3:/work2/noaa/da/rtreadon/git/global-workflow/orion/sorc/gdas.cd/build$ ctest -R test_gdasapp
Test project /work2/noaa/da/rtreadon/git/global-workflow/orion/sorc/gdas.cd/build
      Start 1325: test_gdasapp_util_coding_norms
 1/52 Test #1325: test_gdasapp_util_coding_norms ........................   Passed    1.14 sec
      Start 1326: test_gdasapp_util_ioda_example
 2/52 Test #1326: test_gdasapp_util_ioda_example ........................   Passed    2.33 sec
      Start 1327: test_gdasapp_util_prepdata
 3/52 Test #1327: test_gdasapp_util_prepdata ............................   Passed    0.70 sec
      Start 1328: test_gdasapp_util_rads2ioda
 4/52 Test #1328: test_gdasapp_util_rads2ioda ...........................   Passed    0.17 sec
      Start 1329: test_gdasapp_util_ghrsst2ioda
 5/52 Test #1329: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.16 sec
      Start 1330: test_gdasapp_util_smap2ioda
 6/52 Test #1330: test_gdasapp_util_smap2ioda ...........................   Passed    0.16 sec
      Start 1331: test_gdasapp_util_smos2ioda
 7/52 Test #1331: test_gdasapp_util_smos2ioda ...........................   Passed    0.16 sec
      Start 1332: test_gdasapp_util_viirsaod2ioda
 8/52 Test #1332: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.17 sec
      Start 1333: test_gdasapp_util_icecamsr2ioda
 9/52 Test #1333: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.16 sec
      Start 1665: test_gdasapp_check_python_norms
10/52 Test #1665: test_gdasapp_check_python_norms .......................   Passed    1.10 sec
      Start 1666: test_gdasapp_check_yaml_keys
11/52 Test #1666: test_gdasapp_check_yaml_keys ..........................   Passed    0.09 sec
      Start 1667: test_gdasapp_jedi_increment_to_fv3
12/52 Test #1667: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    0.52 sec
      Start 1668: test_gdasapp_convert_ewok_yaml
13/52 Test #1668: test_gdasapp_convert_ewok_yaml ........................   Passed    0.19 sec
      Start 1669: test_gdasapp_setup_cycled_exp
14/52 Test #1669: test_gdasapp_setup_cycled_exp .........................   Passed    0.43 sec
      Start 1670: test_gdasapp_convert_bufr_temp_dbuoy
15/52 Test #1670: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    1.96 sec
      Start 1671: test_gdasapp_convert_bufr_salt_dbuoy
16/52 Test #1671: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.25 sec
      Start 1672: test_gdasapp_convert_bufr_temp_mbuoyb
17/52 Test #1672: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.25 sec
      Start 1673: test_gdasapp_convert_bufr_salt_mbuoyb
18/52 Test #1673: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.25 sec
      Start 1674: test_gdasapp_convert_bufr_tesacprof
19/52 Test #1674: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.26 sec
      Start 1675: test_gdasapp_convert_bufr_trkobprof
20/52 Test #1675: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.25 sec
      Start 1676: test_gdasapp_convert_bufr_sfcships
21/52 Test #1676: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.26 sec
      Start 1677: test_gdasapp_convert_bufr_sfcshipsu
22/52 Test #1677: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.26 sec
      Start 1678: test_gdasapp_soca_nsst_increment_to_mom6
23/52 Test #1678: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed    1.16 sec
      Start 1679: test_gdasapp_soca_prep
24/52 Test #1679: test_gdasapp_soca_prep ................................   Passed    3.21 sec
      Start 1680: test_gdasapp_soca_run_clean
25/52 Test #1680: test_gdasapp_soca_run_clean ...........................   Passed    0.25 sec
      Start 1681: test_gdasapp_soca_setup_obsproc
26/52 Test #1681: test_gdasapp_soca_setup_obsproc .......................   Passed    0.75 sec
      Start 1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/52 Test #1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed  139.26 sec
      Start 1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/52 Test #1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed  298.73 sec
      Start 1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/52 Test #1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed  332.22 sec
      Start 1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/52 Test #1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed  299.96 sec
      Start 1686: test_gdasapp_soca_copy_scratch
31/52 Test #1686: test_gdasapp_soca_copy_scratch ........................   Passed    0.92 sec
      Start 1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
32/52 Test #1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed  267.32 sec
      Start 1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
33/52 Test #1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed  298.30 sec
      Start 1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
34/52 Test #1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....***Failed  683.07 sec
      Start 1690: test_gdasapp_soca_socahybridweights
35/52 Test #1690: test_gdasapp_soca_socahybridweights ...................   Passed  298.80 sec
      Start 1691: test_gdasapp_soca_incr_handler
36/52 Test #1691: test_gdasapp_soca_incr_handler ........................   Passed  268.11 sec
      Start 1692: test_gdasapp_soca_ens_handler
37/52 Test #1692: test_gdasapp_soca_ens_handler .........................   Passed  333.18 sec
      Start 1693: test_gdasapp_land_create_ens
38/52 Test #1693: test_gdasapp_land_create_ens ..........................   Passed   13.21 sec
      Start 1694: test_gdasapp_land_imsproc
39/52 Test #1694: test_gdasapp_land_imsproc .............................   Passed   18.26 sec
      Start 1695: test_gdasapp_land_apply_jediincr
40/52 Test #1695: test_gdasapp_land_apply_jediincr ......................   Passed   10.03 sec
      Start 1696: test_gdasapp_land_letkfoi_snowda
41/52 Test #1696: test_gdasapp_land_letkfoi_snowda ......................   Passed   31.83 sec
      Start 1697: test_gdasapp_convert_bufr_adpsfc_snow
42/52 Test #1697: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    6.47 sec
      Start 1698: test_gdasapp_convert_bufr_adpsfc
43/52 Test #1698: test_gdasapp_convert_bufr_adpsfc ......................   Passed    4.42 sec
      Start 1699: test_gdasapp_convert_gsi_satbias
44/52 Test #1699: test_gdasapp_convert_gsi_satbias ......................   Passed    2.15 sec
      Start 1700: test_gdasapp_setup_atm_cycled_exp
45/52 Test #1700: test_gdasapp_setup_atm_cycled_exp .....................   Passed    1.39 sec
      Start 1701: test_gdasapp_atm_jjob_var_init
46/52 Test #1701: test_gdasapp_atm_jjob_var_init ........................   Passed  237.90 sec
      Start 1702: test_gdasapp_atm_jjob_var_run
47/52 Test #1702: test_gdasapp_atm_jjob_var_run .........................   Passed  395.72 sec
      Start 1703: test_gdasapp_atm_jjob_var_final
48/52 Test #1703: test_gdasapp_atm_jjob_var_final .......................   Passed  170.48 sec
      Start 1704: test_gdasapp_atm_jjob_ens_init
49/52 Test #1704: test_gdasapp_atm_jjob_ens_init ........................   Passed   47.50 sec
      Start 1705: test_gdasapp_atm_jjob_ens_run
50/52 Test #1705: test_gdasapp_atm_jjob_ens_run .........................   Passed  938.73 sec
      Start 1706: test_gdasapp_atm_jjob_ens_final
51/52 Test #1706: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.23 sec
      Start 1707: test_gdasapp_aero_gen_3dvar_yaml
52/52 Test #1707: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.57 sec

98% tests passed, 1 tests failed out of 52

Label Time Summary:
gdas-utils    =   5.15 sec*proc (9 tests)
script        =   5.15 sec*proc (9 tests)

Total Test time (real) = 5158.59 sec

The following tests FAILED:
        1689 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Errors while running CTest
Output from these tests are in: /work2/noaa/da/rtreadon/git/global-workflow/orion/sorc/gdas.cd/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

RussTreadon-NOAA · 2024-01-13T19:41:23Z

Hera gw ctests
Install feature/submodule at 695447b in g-w develop at c590476. All ctests pass except the VRFY job.

(gdasapp) Hera(hfe12):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/submodule/sorc/gdas.cd/build$ ctest -R test_gdasapp
Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/submodule/sorc/gdas.cd/build
      Start 1325: test_gdasapp_util_coding_norms
 1/52 Test #1325: test_gdasapp_util_coding_norms ........................   Passed    1.38 sec
      Start 1326: test_gdasapp_util_ioda_example
 2/52 Test #1326: test_gdasapp_util_ioda_example ........................   Passed    3.87 sec
      Start 1327: test_gdasapp_util_prepdata
 3/52 Test #1327: test_gdasapp_util_prepdata ............................   Passed    1.41 sec
      Start 1328: test_gdasapp_util_rads2ioda
 4/52 Test #1328: test_gdasapp_util_rads2ioda ...........................   Passed    1.00 sec
      Start 1329: test_gdasapp_util_ghrsst2ioda
 5/52 Test #1329: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.36 sec
      Start 1330: test_gdasapp_util_smap2ioda
 6/52 Test #1330: test_gdasapp_util_smap2ioda ...........................   Passed    0.27 sec
      Start 1331: test_gdasapp_util_smos2ioda
 7/52 Test #1331: test_gdasapp_util_smos2ioda ...........................   Passed    0.15 sec
      Start 1332: test_gdasapp_util_viirsaod2ioda
 8/52 Test #1332: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.17 sec
      Start 1333: test_gdasapp_util_icecamsr2ioda
 9/52 Test #1333: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.19 sec
      Start 1665: test_gdasapp_check_python_norms
10/52 Test #1665: test_gdasapp_check_python_norms .......................   Passed    1.22 sec
      Start 1666: test_gdasapp_check_yaml_keys
11/52 Test #1666: test_gdasapp_check_yaml_keys ..........................   Passed    0.20 sec
      Start 1667: test_gdasapp_jedi_increment_to_fv3
12/52 Test #1667: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    1.82 sec
      Start 1668: test_gdasapp_convert_ewok_yaml
13/52 Test #1668: test_gdasapp_convert_ewok_yaml ........................   Passed    0.69 sec
      Start 1669: test_gdasapp_setup_cycled_exp
14/52 Test #1669: test_gdasapp_setup_cycled_exp .........................   Passed    1.16 sec
      Start 1670: test_gdasapp_convert_bufr_temp_dbuoy
15/52 Test #1670: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    2.43 sec
      Start 1671: test_gdasapp_convert_bufr_salt_dbuoy
16/52 Test #1671: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.33 sec
      Start 1672: test_gdasapp_convert_bufr_temp_mbuoyb
17/52 Test #1672: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.19 sec
      Start 1673: test_gdasapp_convert_bufr_salt_mbuoyb
18/52 Test #1673: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.19 sec
      Start 1674: test_gdasapp_convert_bufr_tesacprof
19/52 Test #1674: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.20 sec
      Start 1675: test_gdasapp_convert_bufr_trkobprof
20/52 Test #1675: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.19 sec
      Start 1676: test_gdasapp_convert_bufr_sfcships
21/52 Test #1676: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.24 sec
      Start 1677: test_gdasapp_convert_bufr_sfcshipsu
22/52 Test #1677: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.28 sec
      Start 1678: test_gdasapp_soca_nsst_increment_to_mom6
23/52 Test #1678: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   11.44 sec
      Start 1679: test_gdasapp_soca_prep
24/52 Test #1679: test_gdasapp_soca_prep ................................   Passed    3.86 sec
      Start 1680: test_gdasapp_soca_run_clean
25/52 Test #1680: test_gdasapp_soca_run_clean ...........................   Passed    0.55 sec
      Start 1681: test_gdasapp_soca_setup_obsproc
26/52 Test #1681: test_gdasapp_soca_setup_obsproc .......................   Passed    0.64 sec
      Start 1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/52 Test #1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed  427.15 sec
      Start 1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/52 Test #1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   42.51 sec
      Start 1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/52 Test #1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   42.26 sec
      Start 1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/52 Test #1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.29 sec
      Start 1686: test_gdasapp_soca_copy_scratch
31/52 Test #1686: test_gdasapp_soca_copy_scratch ........................   Passed    1.36 sec
      Start 1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
32/52 Test #1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.81 sec
      Start 1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
33/52 Test #1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   44.22 sec
      Start 1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
34/52 Test #1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....***Failed  300.87 sec
      Start 1690: test_gdasapp_soca_socahybridweights
35/52 Test #1690: test_gdasapp_soca_socahybridweights ...................   Passed  107.88 sec
      Start 1691: test_gdasapp_soca_incr_handler
36/52 Test #1691: test_gdasapp_soca_incr_handler ........................   Passed  139.10 sec
      Start 1692: test_gdasapp_soca_ens_handler
37/52 Test #1692: test_gdasapp_soca_ens_handler .........................   Passed   74.25 sec
      Start 1693: test_gdasapp_land_create_ens
38/52 Test #1693: test_gdasapp_land_create_ens ..........................   Passed    2.70 sec
      Start 1694: test_gdasapp_land_imsproc
39/52 Test #1694: test_gdasapp_land_imsproc .............................   Passed    6.20 sec
      Start 1695: test_gdasapp_land_apply_jediincr
40/52 Test #1695: test_gdasapp_land_apply_jediincr ......................   Passed    5.06 sec
      Start 1696: test_gdasapp_land_letkfoi_snowda
41/52 Test #1696: test_gdasapp_land_letkfoi_snowda ......................   Passed   13.45 sec
      Start 1697: test_gdasapp_convert_bufr_adpsfc_snow
42/52 Test #1697: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    4.59 sec
      Start 1698: test_gdasapp_convert_bufr_adpsfc
43/52 Test #1698: test_gdasapp_convert_bufr_adpsfc ......................   Passed    4.34 sec
      Start 1699: test_gdasapp_convert_gsi_satbias
44/52 Test #1699: test_gdasapp_convert_gsi_satbias ......................   Passed    1.45 sec
      Start 1700: test_gdasapp_setup_atm_cycled_exp
45/52 Test #1700: test_gdasapp_setup_atm_cycled_exp .....................   Passed    1.03 sec
      Start 1701: test_gdasapp_atm_jjob_var_init
46/52 Test #1701: test_gdasapp_atm_jjob_var_init ........................   Passed  109.07 sec
      Start 1702: test_gdasapp_atm_jjob_var_run
47/52 Test #1702: test_gdasapp_atm_jjob_var_run .........................   Passed  395.71 sec
      Start 1703: test_gdasapp_atm_jjob_var_final
48/52 Test #1703: test_gdasapp_atm_jjob_var_final .......................   Passed   74.15 sec
      Start 1704: test_gdasapp_atm_jjob_ens_init
49/52 Test #1704: test_gdasapp_atm_jjob_ens_init ........................   Passed   43.34 sec
      Start 1705: test_gdasapp_atm_jjob_ens_run
50/52 Test #1705: test_gdasapp_atm_jjob_ens_run .........................   Passed  266.65 sec
      Start 1706: test_gdasapp_atm_jjob_ens_final
51/52 Test #1706: test_gdasapp_atm_jjob_ens_final .......................   Passed  106.16 sec
      Start 1707: test_gdasapp_aero_gen_3dvar_yaml
52/52 Test #1707: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    1.02 sec

98% tests passed, 1 tests failed out of 52

Label Time Summary:
gdas-utils    =   8.80 sec*proc (9 tests)
script        =   8.80 sec*proc (9 tests)

Total Test time (real) = 2336.14 sec

The following tests FAILED:
        1689 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Errors while running CTest
Output from these tests are in: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/submodule/sorc/gdas.cd/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
(gdasapp) Hera(hfe12):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/submodule/sorc/gdas.cd/build$

As mentioned before the Hecules, Hera, and Orion g-w snapshots contain the following local modifications

jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT - replace ocnanalrun, a typo, with ocnanalbmat
parm/config/gfs/config.resources - set nth_ocnanalrun to one thread
sorc/build_all.sh - allow GDASApp build on hercules and hera.intel

Note: build_all.sh executes detect_machine.sh to set MACHINE. Script detect_machine.sh includes the compiler with the machine id for hera

# Append compiler (only on machines that have multiple compilers)
if [ $MACHINE_ID = hera ] || [ $MACHINE_ID = cheyenne ]; then
    MACHINE_ID=${MACHINE_ID}.${COMPILER}
fi

As such, detect_machine.sh returns hera.intel when build_all.sh is executed on Hera. The existing logic in build_all.sh did not allow for hera.intel to be a valid machine.

RussTreadon-NOAA · 2024-01-16T11:29:19Z

global-workflow branch RussTreadon-NOAA:feature/gdasapp has been created to bring the changes in this PR into the g-w UFS-DA build. Once this PR is merged into GDASApp develop, the hash for gdas.cd in feature/gdasapp will be updated to the new GDASApp develop hash

See g-w issue #2222 for details.

RussTreadon-NOAA

feature/submodule installed on Hera, Hercules, and Orion within g-w develop. g-w ctest_gdasapp run on each machine. All tests pass except test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY . A fix for this ctest is under development.

Approve.

guillaumevernieres

👍

CoryMartin-NOAA · 2024-01-16T14:12:47Z

I will merge this later today!

AndrewEichmann-NOAA · 2024-01-16T22:12:15Z

also resolves #835

CoryMartin-NOAA added 26 commits December 4, 2023 13:08

Changes to gitignore and cmakelists

1f4d600

Add submodules

f980d48

Changes to gitignore and ufo submodule

cde0f28

update gitignore

cf89d5d

Check out branches

4df2e8e

Merge branch 'develop' into feature/submodule

8c50486

try to move bundle into subdirectory

99a6e88

Revert "try to move bundle into subdirectory"

77c47db

This reverts commit 99a6e88.

Update modules

ee412a8

Start to have scripts to update submodules

f3edb41

Actually cd

2d30810

Add JCSDA test data to submodules

b615448

Update data repos too

509126e

Merge branch 'develop' into feature/submodule

4910733

Add CRTM as a submodule

337e3b4

Remove data submodules

a1fc812

Try cmake magic

a99a29a

Bugfix

3e6045a

More stuff

1877dc2

this is not working but I want to save before the weekend

6ba88ff

add recursive clones everywhere for CI

f0feb8f

Working maybe

64c05de

Working script to update hashes

3ba469d

Everything working but stable nightly

54fea31

Merge branch 'feature/submodule' of https://github.com/NOAA-EMC/GDASApp…

11b57b5

… into feature/submodule

detabulate

c38bdaa

CoryMartin-NOAA requested review from aerorahul, guillaumevernieres and RussTreadon-NOAA December 18, 2023 21:34

CoryMartin-NOAA self-assigned this Dec 18, 2023

update paths in soca scripts and tests (#845)

f19bdd9

Update soca paths to be consistent with use of submodules

CoryMartin-NOAA dismissed guillaumevernieres’s stale review via f19bdd9 January 10, 2024 21:46

RussTreadon-NOAA and others added 4 commits January 11, 2024 14:07

Merge branch 'develop' into feature/submodule

b7a7569

update hashes for jedi libraries (#811)

b699294

Merge branch 'develop' into feature/submodule

4846a7f

Merge branch 'develop' into feature/submodule

a958ea8

update submodule hashes for land-imsproc and vader, add gdas to DMPDI…

2671b13

…R path (#811)

replace resdet and resens with resdetatmos and resensatmos (#811)

54ff2af

update comrot to be consistent with g-w develop (#811)

b9379ba

use realpath for HOMEgfs in config.base (#811)

695447b

RussTreadon-NOAA self-requested a review January 16, 2024 11:31

RussTreadon-NOAA approved these changes Jan 16, 2024

View reviewed changes

guillaumevernieres approved these changes Jan 16, 2024

View reviewed changes

CoryMartin-NOAA merged commit 639ef16 into develop Jan 16, 2024
5 checks passed

CoryMartin-NOAA deleted the feature/submodule branch January 16, 2024 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use submodules rather than ecbuild_bundle git #811

Use submodules rather than ecbuild_bundle git #811

CoryMartin-NOAA commented Dec 18, 2023

RussTreadon-NOAA commented Jan 12, 2024

RussTreadon-NOAA commented Jan 12, 2024

RussTreadon-NOAA commented Jan 12, 2024

CoryMartin-NOAA commented Jan 12, 2024

RussTreadon-NOAA commented Jan 12, 2024

RussTreadon-NOAA commented Jan 12, 2024

apchoiCMD commented Jan 12, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 16, 2024

RussTreadon-NOAA left a comment

guillaumevernieres left a comment

CoryMartin-NOAA commented Jan 16, 2024

AndrewEichmann-NOAA commented Jan 16, 2024

Use submodules rather than ecbuild_bundle git #811

Use submodules rather than ecbuild_bundle git #811

Conversation

CoryMartin-NOAA commented Dec 18, 2023

RussTreadon-NOAA commented Jan 12, 2024

RussTreadon-NOAA commented Jan 12, 2024

RussTreadon-NOAA commented Jan 12, 2024

CoryMartin-NOAA commented Jan 12, 2024

RussTreadon-NOAA commented Jan 12, 2024

RussTreadon-NOAA commented Jan 12, 2024

apchoiCMD commented Jan 12, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 13, 2024

RussTreadon-NOAA commented Jan 16, 2024

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

guillaumevernieres left a comment

Choose a reason for hiding this comment

CoryMartin-NOAA commented Jan 16, 2024

AndrewEichmann-NOAA commented Jan 16, 2024