Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use ESMF 8.2.0 library #621

Closed
MinsukJi-NOAA opened this issue Jan 25, 2022 · 17 comments · Fixed by #622
Closed

Use ESMF 8.2.0 library #621

MinsukJi-NOAA opened this issue Jan 25, 2022 · 17 comments · Fixed by #622
Assignees
@MinsukJi-NOAA
Labels
enhancement New feature or request

Comments

@MinsukJi-NOAA
Copy link
Contributor

MinsukJi-NOAA commented Jan 25, 2022 •
edited by GeorgeGayno-NOAA
Loading

Update the ESMF library from 8.1.0 to 8.2.0. This change is needed to utilize latest fixes in the ESMF; see, for example, #143.

A corresponding PR will be made, in which regression (consistency) tests for those utilities affected by the ESMF library (chgres_cube and grid_gen) on all 5 platforms will be run and comparison of results be posted.

Part of #143.
@GeorgeGayno-NOAA GeorgeGayno-NOAA added the enhancement New feature or request label Jan 25, 2022
@MinsukJi-NOAA
Copy link
Contributor Author

@GeorgeGayno-NOAA These are some of the differences from using ESMF 8.2.0. There are LOTS of these statistics to consider since there are many failed consistency tests on all 5 platforms. Wanted to check with you if you have any suggestions on how to go about analyzing these systematically.

+ nccmp -dmfqS out.atm.tile1.nc /scratch1/NCEPDEV/nems/role.ufsutils/ufs_utils/reg_tests/chgres_cube/baseline_data/c96_fv3_restart/out.atm.tile1.nc
Variable Group Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
u_w      /         2 -2.23517e-07  2.5332e-07 -2.38419e-07 1.49012e-08  2.5332e-07 -1.11759e-07 1.79124e-07
v_w      /        16  8.19564e-08 1.37836e-06 -2.38419e-07 4.76837e-07 7.15256e-07  5.12227e-09 1.56207e-07

+ nccmp -dmfqS out.atm.tile7.nc /lfs4/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/reg_tests/chgres_cube/baseline_data/3km_conus_hrrr_gfssdf_grib2/out.atm.tile7.nc
Variable Group Count      Sum  AbsSum       Min      Max    Range        Mean    StdDev
u_w      /      1152  1.15363 50.6469 -0.190329 0.216066 0.406396  0.00100141 0.0529021
v_w      /      1152  1.18029 27.8806 -0.379808 0.261978 0.641786  0.00102456 0.0378907
u_s      /      1088 -20.8364 37.2089 -0.177918 0.108623 0.286541  -0.0191511 0.0372238
v_s      /      1088 -6.41019 33.7436 -0.127974 0.139483 0.267457 -0.00589172 0.0372765

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA These are some of the differences from using ESMF 8.2.0. There are LOTS of these statistics to consider since there are many failed consistency tests on all 5 platforms. Wanted to check with you if you have any suggestions on how to go about analyzing these systematically.

+ nccmp -dmfqS out.atm.tile1.nc /scratch1/NCEPDEV/nems/role.ufsutils/ufs_utils/reg_tests/chgres_cube/baseline_data/c96_fv3_restart/out.atm.tile1.nc
Variable Group Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
u_w      /         2 -2.23517e-07  2.5332e-07 -2.38419e-07 1.49012e-08  2.5332e-07 -1.11759e-07 1.79124e-07
v_w      /        16  8.19564e-08 1.37836e-06 -2.38419e-07 4.76837e-07 7.15256e-07  5.12227e-09 1.56207e-07

+ nccmp -dmfqS out.atm.tile7.nc /lfs4/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/reg_tests/chgres_cube/baseline_data/3km_conus_hrrr_gfssdf_grib2/out.atm.tile7.nc
Variable Group Count      Sum  AbsSum       Min      Max    Range        Mean    StdDev
u_w      /      1152  1.15363 50.6469 -0.190329 0.216066 0.406396  0.00100141 0.0529021
v_w      /      1152  1.18029 27.8806 -0.379808 0.261978 0.641786  0.00102456 0.0378907
u_s      /      1088 -20.8364 37.2089 -0.177918 0.108623 0.286541  -0.0191511 0.0372238
v_s      /      1088 -6.41019 33.7436 -0.127974 0.139483 0.267457 -0.00589172 0.0372765

@MinsukJi-NOAA The differences from the first test look small. But the differences from the second test seem bigger than I would expect. I would visualize the cases with the big differences and make some difference plots with the baseline data. Also, the grid generation procedure depends on ESMF. So you will need to run the 'grid_gen' tests as well.

@MinsukJi-NOAA
Copy link
Contributor Author

On Hera:

  • grid_gen tests all passed.
  • For chgres_cube tests, some of the failed cases with large-value differences are shown below. Left: ESMF 8.2.0; Middle: Difference; Right: ESMF 8.1.0.
    • c192_fv3_history

02_c192_fv3_history_out_atm_tile3_v_s

  • c192_gfs_grib2

08_c192_gfs_grib2_out_atm_tile3_u_w

  • 3km_conus_hrrr_gfsdf_grib2

10_3km_conus_hrrr_gfsdf_grib2_out_atm_tile7_u_s

  • 3km_conus_hrrr_newsfc_grib2

11_3km_conus_hrrr_newsfc_grib2_out_atm_tile7_u_s

@GeorgeGayno-NOAA GeorgeGayno-NOAA mentioned this issue Jan 27, 2022
Closed
@GeorgeGayno-NOAA
Copy link
Collaborator

@MinsukJi-NOAA Thanks for making these plots. I was looking at the ESMF 8.2.0 release notes:

https://github.com/esmf-org/esmf/releases/tag/ESMF_8_2_0

There were some changes to the regridding which result in small differences in the regridding weights. So that probably explains what we are seeing. I may look at the C192 case more closely myself. But I am inclined to make the upgrade to 8.2.0.

@GeorgeGayno-NOAA
Copy link
Collaborator

@MinsukJi-NOAA Where are your test data? I want to take a look.

@GeorgeGayno-NOAA
Copy link
Collaborator

@MinsukJi-NOAA What branch are you working from?

@MinsukJi-NOAA
Copy link
Contributor Author

MinsukJi-NOAA commented Jan 27, 2022 •
edited
Loading

@GeorgeGayno-NOAA I am working from the latest develop branch: 04700f9
My test data on Hera is here: /scratch2/NCEPDEV/stmp1/Minsuk.Ji/reg-tests/chgres-cube
Summary log is located here: /scratch1/NCEPDEV/stmp4/Minsuk.Ji/UFS_UTILS_04700f9/reg_tests/chgres_cube

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA I am working from the latest develop branch: 04700f9 My test data on Hera is here: /scratch2/NCEPDEV/stmp1/Minsuk.Ji/reg-tests/chgres-cube Summary log is located here: /scratch1/NCEPDEV/stmp4/Minsuk.Ji/UFS_UTILS_04700f9/reg_tests/chgres_cube

@MinsukJi-NOAA Please create a branch off 'develop' in your fork for any work under this issue. See our wiki page for more details: https://github.com/ufs-community/UFS_UTILS/wiki/3.-Checking-out-code-and-making-changes

MinsukJi-NOAA added a commit to MinsukJi-NOAA/UFS_UTILS that referenced this issue Jan 27, 2022
@MinsukJi-NOAA
Change hera, jet, orion modulefiles for esmf 8.2.0. ufs-community#621 u…
222bfa7 
…fs-community#143
@MinsukJi-NOAA
Copy link
Contributor Author

MinsukJi-NOAA commented Jan 27, 2022 •
edited
Loading

@GeorgeGayno-NOAA I am working from the latest develop branch: 04700f9 My test data on Hera is here: /scratch2/NCEPDEV/stmp1/Minsuk.Ji/reg-tests/chgres-cube Summary log is located here: /scratch1/NCEPDEV/stmp4/Minsuk.Ji/UFS_UTILS_04700f9/reg_tests/chgres_cube

@MinsukJi-NOAA Please create a branch off 'develop' in your fork for any work under this issue. See our wiki page for more details: https://github.com/ufs-community/UFS_UTILS/wiki/3.-Checking-out-code-and-making-changes

@GeorgeGayno-NOAA my working branch is here: https://github.com/MinsukJi-NOAA/UFS_UTILS/tree/feature/esmf_8_2_0.
So far, I have run regression tests on hera, jet, and orion. I will also report results from jet and orion.

@MinsukJi-NOAA
Copy link
Contributor Author

On Jet:

  • grid_gen tests all passed

  • For chgres_cube tests, some of the failed cases with large-value differences are shown below. Left: ESMF 8.2.0; Middle: Difference; Right: ESMF 8.1.0.

    • c192_fv3_history

02_c192_fv3_history_out_atm_tile3_v_s

  • c192_gfs_grib2

08_c192_gfs_grib2_out_atm_tile5_u_w

  • 3km_conus_hrrr_gfsdf_grib2

10_3km_conus_hrrr_gfsdf_grib2_out_atm_tile7_u_s

  • 3km_conus_hrrr_newsfc_grib2

11_3km_conus_hrrr_newsfc_grib2_out_atm_tile7_u_s

@MinsukJi-NOAA
Copy link
Contributor Author

On Orion:

  • grid_gen tests all passed

  • For chgres_cube tests, some of the failed cases with large-value differences are shown below. Left: ESMF 8.2.0; Middle: Difference; Right: ESMF 8.1.0.

    • c192_fv3_history

02_c192_fv3_history_out_atm_tile3_u_s

  • c192_gfs_grib2

07_c192_gfs_grib2_out_atm_tile5_u_w

  • 3km_conus_hrrr_gfsdf_grib2 and 3km_conus_hrrr_newsfc_grib2 have the usual high-value difference points in the center of tile7 (similar to those shown above for Hera and Jet)

@GeorgeGayno-NOAA
Copy link
Collaborator

To be complete, the workflow files should be updated to use ESMF v8.2.0:

https://github.com/ufs-community/UFS_UTILS/tree/develop/.github/workflows

@MinsukJi-NOAA I can help with that.

@MinsukJi-NOAA
Copy link
Contributor Author

To be complete, the workflow files should be updated to use ESMF v8.2.0:

https://github.com/ufs-community/UFS_UTILS/tree/develop/.github/workflows

@MinsukJi-NOAA I can help with that.

@GeorgeGayno-NOAA thanks. I will look into it.

@MinsukJi-NOAA
Copy link
Contributor Author

MinsukJi-NOAA commented Jan 28, 2022 •
edited
Loading

On WCOSS Cray:

  • All grid_gen tests failed. But an examination of statistics in consistency.log indicates the differences are ~ 10^-7.

  • All chgres_cube tests failed. Mostly, the differences are small. Case with significant differences are plotted below.

    • c192_fv3_history

08_c192_fv3_history_out_atm_tile5_u_w

  • c192_gfs_grib2

14_c192_gfs_grib2_out_atm_tile5_u_w

  • 3km_conus_hrrr_gfsdf_grib2 and 3km_conus_hrrr_newsfc_grib2 have the usual high-value difference points in the center of tile7 (similar to those shown above for Hera and Jet)

@MinsukJi-NOAA
Copy link
Contributor Author

On WCOSS Dell:

  • All grid_gen tests passed

  • For chgres_cube tests, some of the failed cases with large-value differences are shown below. Left: ESMF 8.2.0; Middle: Difference; Right: ESMF 8.1.0.

    • c192_fv3_history
      08_c192_fv3_history_out_atm_tile5_u_w

    • c192_gfs_gribs
      14_c192_gfs_grib2_out_atm_tile5_u_w

    • 3km_conus_hrrr_gfssdf_grib2 and 3km_conus_hrrr_newsfc_grib2 have the usual high-value difference points in the center of tile7 (similar to those shown above for Hera and Jet)

@MinsukJi-NOAA
Copy link
Contributor Author

@GeorgeGayno-NOAA please note that a few other minor module changes had to be made for wcoss dell: MinsukJi-NOAA@12fe8a0
I am not sure if the g2 library version change from 3.4.1 to 3.4.2 requires testing ufs utilities other than chgres_cube and grid_gen

@MinsukJi-NOAA MinsukJi-NOAA mentioned this issue Jan 28, 2022
Merged
GeorgeGayno-NOAA added a commit to MinsukJi-NOAA/UFS_UTILS that referenced this issue Jan 31, 2022
@GeorgeGayno-NOAA
Update chgres and sfc_climo_gen suppression files to
3a92691 
ignore memory leaks associated with ESMF v8.2.0.

Fixes ufs-community#621.
@GeorgeGayno-NOAA
Copy link
Collaborator

@MinsukJi-NOAA The differences from the baseline are small enough that I will not stop the merge.

GeorgeGayno-NOAA pushed a commit that referenced this issue Jan 31, 2022
@MinsukJi-NOAA
Use ESMF 8.2.0 library
26cd024 
Upgrade to version 8.2.0. 

Part of #143
Fixes #621.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MinsukJi-NOAA MinsukJi-NOAA

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use ESMF 8.2.0 library (feature/esmf_8_2_0) MinsukJi-NOAA/UFS_UTILS
2 participants
@GeorgeGayno-NOAA @MinsukJi-NOAA