Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch snow DA to use 2DVar for deterministic and ensemble mean #3163

Merged
merged 250 commits into from
Dec 20, 2024
Merged

Switch snow DA to use 2DVar for deterministic and ensemble mean #3163

merged 250 commits into from
Dec 20, 2024

Conversation

CoryMartin-NOAA
Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA commented Dec 12, 2024 •
edited by RussTreadon-NOAA
Loading

Description

This PR moves the snow DA from a LETKF-OI approach with a fake ensemble to a 2DVar approach for the deterministic and the ensemble mean for GDAS/GFS. This PR also adds support for JCB and refactoring of the jobs to leverage the Jedi class in the workflow.

Resolves #3002

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? YES
  • Does this change require a documentation update? MAYBE
  • Does this change require an update to any of the following submodules? YES

How has this been tested?

  • Cycled test on Hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

RussTreadon-NOAA and others added 30 commits August 28, 2024 17:01
@RussTreadon-NOAA
replace separate JEDI radiance bias correction files with tarball (NO…
20fb88c 
…AA-EMC#2862)
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
8a219cd
@RussTreadon-NOAA
rename JEDI radiance bias correction tarball file to be more self-des…
73aa233 
…cribing (NOAA-EMC#2862)
@RussTreadon-NOAA
Merge remote-tracking branch 'upstream/develop' into feature/radbcor
00a013f
@RussTreadon-NOAA
Merge branch 'NOAA-EMC:develop' into feature/radbcor
3215d9b
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
04b4347
@RussTreadon-NOAA
update gdas.cd hash to include recent commits to GDASApp develop (NOA…
1203866 
…A-EMC#2862)
@RussTreadon-NOAA
update pygfs jedi class and atmospheric tasks to extract JEDI radianc…
5985e9f 
…e bias correction files from tarball (NOAA-EMC#2862)
@jiaruidong2017
Remove the GTS BUFR2IODA part of the snow obs prep job and
4098f10 
Replace it with a direct read from BUFR at runtime.
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
009659c
@jiaruidong2017
Change HOMEgfs to PARMgfs.
8d1bc87
@RussTreadon-NOAA
move radiance bias correction staging to jedi class (NOAA-EMC#2868)
73e56a7
@jiaruidong2017
Merge branch 'develop' into feature/snow_gtsobs
6ff65bc
@RussTreadon-NOAA
initial attempt to generalize processing of variational bias correcti…
666d7d6 
…on files using jedi class (NOAA-EMC#2862)
@RussTreadon-NOAA
move VarBcDir inside initialize_analysis method (NOAA-EMC#2862)
1e4a284
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
386313b
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
dd9099e
@RussTreadon-NOAA
update sorc/gdas.cd hash to bring in radiance tarball ctest changes (N…
b48e126 
…OAA-EMC#2862)
@CoryMartin-NOAA
Merge branch 'develop' into feature/snow_gtsobs
2295cab
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
27434bb
@jiaruidong2017
Made changes to stage the static GTS mapping files.
5074d11
@jiaruidong2017
Change the name to be more descriptive.
f122edf
@RussTreadon-NOAA
add extract_tar method to jedi class, use extract_tar in atm and atme…
ba77d4e 
…ns analysis scripts (NOAA-EMC#2862)
@RussTreadon-NOAA
correct typo in atmens_analysis.py (2862)
bc3a6dd
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
7ba33c1
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
7182a2a
@RussTreadon-NOAA
Merge branch 'develop' into feature/radbcor
8016852
@jiaruidong2017
Merge branch 'develop' into feature/snow_gtsobs
73b4ec0
@jiaruidong2017
Update IMS preprocessing job.
086158d
@RussTreadon-NOAA
Merge branch 'NOAA-EMC:develop' into feature/radbcor
4fb8f81
@WalterKolczynski-NOAA
Copy link
Contributor

Going to let @RussTreadon-NOAA set the labels to confirm they work correctly for him.

@RussTreadon-NOAA RussTreadon-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Dec 19, 2024
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Dec 19, 2024
@RussTreadon-NOAA
Copy link
Contributor

CI locations

  • Hera: /scratch1/NCEPDEV/global/CI/3163
  • Hera: /scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/test/stable/20241219/global-workflow - reference checks on
  • Hercules: /work2/noaa/global/CI/HERCULES/3163

@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Dec 19, 2024
@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Dec 19, 2024
@emcbot
Copy link

emcbot commented Dec 20, 2024

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3163


Experiment C48mx500_3DVarAOWCDA_f549a6b4 Completed 2 Cycles: *SUCCESS* at Thu Dec 19 16:11:18 CST 2024
Experiment C48_ATM_f549a6b4 Completed 2 Cycles: *SUCCESS* at Thu Dec 19 16:11:18 CST 2024
Experiment C48mx500_hybAOWCDA_f549a6b4 Completed 2 Cycles: *SUCCESS* at Thu Dec 19 16:23:24 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_f549a6b4 Completed 1 Cycles: *SUCCESS* at Thu Dec 19 17:06:28 CST 2024
Experiment C96_atm3DVar_f549a6b4 Completed 3 Cycles: *SUCCESS* at Thu Dec 19 17:12:09 CST 2024
Experiment C96C48_hybatmDA_f549a6b4 Completed 3 Cycles: *SUCCESS* at Thu Dec 19 17:18:13 CST 2024
Experiment C48_S2SW_f549a6b4 Completed 2 Cycles: *SUCCESS* at Thu Dec 19 17:49:20 CST 2024
Experiment C48_S2SWA_gefs_f549a6b4 Completed 1 Cycles: *SUCCESS* at Thu Dec 19 18:56:28 CST 2024

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 20, 2024
@emcbot
Copy link

emcbot commented Dec 20, 2024

CI Passed on Hera in Build# 2
Built and ran in directory /scratch1/NCEPDEV/global/CI/3163


Experiment C48mx500_3DVarAOWCDA_f549a6b4 Completed 2 Cycles: *SUCCESS* at Thu Dec 19 22:45:30 UTC 2024
Experiment C48_ATM_f549a6b4 Completed 2 Cycles: *SUCCESS* at Thu Dec 19 22:45:30 UTC 2024
Experiment C48mx500_hybAOWCDA_f549a6b4 Completed 2 Cycles: *SUCCESS* at Thu Dec 19 22:57:39 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_f549a6b4 Completed 1 Cycles: *SUCCESS* at Thu Dec 19 23:04:05 UTC 2024
Experiment C96C48_hybatmaerosnowDA_f549a6b4 Completed 3 Cycles: *SUCCESS* at Fri Dec 20 00:04:47 UTC 2024
Experiment C96_atm3DVar_f549a6b4 Completed 3 Cycles: *SUCCESS* at Fri Dec 20 00:10:43 UTC 2024
Experiment C96C48_hybatmDA_f549a6b4 Completed 3 Cycles: *SUCCESS* at Fri Dec 20 00:10:52 UTC 2024
Experiment C96C48_ufs_hybatmDA_f549a6b4 Completed 3 Cycles: *SUCCESS* at Fri Dec 20 00:53:18 UTC 2024
Experiment C48_S2SW_f549a6b4 Completed 2 Cycles: *SUCCESS* at Fri Dec 20 00:59:57 UTC 2024
Experiment C48_S2SWA_gefs_f549a6b4 Completed 1 Cycles: *SUCCESS* at Fri Dec 20 01:30:26 UTC 2024

RussTreadon-NOAA
RussTreadon-NOAA approved these changes Dec 20, 2024
View reviewed changes
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated CI passes on Hera and Hercules.

Approve.

@RussTreadon-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA : is Passed on Hera and Hercules sufficient or do we need to also run on WCOSS2 (Dogwood) after it returns to service?

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit d479111 into NOAA-EMC:develop Dec 20, 2024
10 of 11 checks passed
@RussTreadon-NOAA
Copy link
Contributor

Thank you @WalterKolczynski-NOAA

DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Dec 20, 2024
@DavidHuber-NOAA
Merge remote-tracking branch 'emc/develop' into fix/awips
4b7d39e 
* emc/develop:
  Change orog gravity wave drag scheme for grid sizes less than 10km (NOAA-EMC#3175)
  Switch snow DA to use 2DVar for deterministic and ensemble mean (NOAA-EMC#3163)
  Update compression options for GEFS history files (NOAA-EMC#3184)
tsga added a commit to tsga/global-workflow that referenced this pull request Dec 22, 2024
@tsga
Merge branch 'feature/land_iau' into feature/lnd_regrid_iau
850264f 
* feature/land_iau:
  modify for Clara's gauss-fv3 regridding
  Change orog gravity wave drag scheme for grid sizes less than 10km (NOAA-EMC#3175)
  Switch snow DA to use 2DVar for deterministic and ensemble mean (NOAA-EMC#3163)
  Update compression options for GEFS history files (NOAA-EMC#3184)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TravisElless-NOAA TravisElless-NOAA TravisElless-NOAA approved these changes

@aerorahul aerorahul aerorahul left review comments

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@jiaruidong2017 jiaruidong2017 jiaruidong2017 approved these changes

@RussTreadon-NOAA RussTreadon-NOAA RussTreadon-NOAA approved these changes

@KateFriedman-NOAA KateFriedman-NOAA Awaiting requested review from KateFriedman-NOAA KateFriedman-NOAA is a code owner

@DavidHuber-NOAA DavidHuber-NOAA Awaiting requested review from DavidHuber-NOAA DavidHuber-NOAA is a code owner

@CatherineThomas-NOAA CatherineThomas-NOAA Awaiting requested review from CatherineThomas-NOAA CatherineThomas-NOAA is a code owner

Assignees

@CoryMartin-NOAA CoryMartin-NOAA

Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Switch to 2DVar for snow DA
9 participants
@CoryMartin-NOAA @RussTreadon-NOAA @jiaruidong2017 @TravisElless-NOAA @WalterKolczynski-NOAA @emcbot @aerorahul @DavidNew-NOAA @DavidHuber-NOAA