Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrapper for ccpp-physics #812 #972

Closed
wants to merge 3 commits into from

Conversation

grantfirl
Copy link
Collaborator

@grantfirl grantfirl commented Dec 22, 2021

ALL CREDIT TO @rmontuoro FOR THIS PR. This is only a wrapper for testing/merging purposes. All text below is from #964

PR Checklist

  • Ths PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

  • This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

  • An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
    are specified below.

  • Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.

  • New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Description

This PR implements a vertically-integrated formula to properly compute tracer column mass and ensure mass conservation for all tracers except TKE. See issue NCAR/ccpp-physics#811 for details.

This PR will change the baseline for all regression tests using Scale-Aware Mass-Flux (SAMF) shallow or deep convective schemes, as well as the Scale-aware TKE-based Moist Eddy-Diffusivity Mass-Flux (SATMEDMF) PBL scheme.

Issue(s) addressed

Testing

  • hera.intel
  • hera.gnu
  • orion.intel
  • cheyenne.intel
  • cheyenne.gnu
  • gaea.intel
  • jet.intel
  • wcoss_cray
  • wcoss_dell_p3
  • opnReqTest for newly added/changed feature
  • CI

Dependencies

@grantfirl grantfirl marked this pull request as ready for review December 22, 2021 03:25
@grantfirl
Copy link
Collaborator Author

FYI, the following RTs failed on hera.intel. I'm not sure if this represents only those that include the SAMF deep or shallow and/or SATMEDMF schemes or not since I'm not as familiar with the tests as others.

cpld_control_p7 001 failed in check_result
cpld_control_p7 001 failed in run_test
cpld_control_p7_rrtmgp 002 failed in check_result
cpld_control_p7_rrtmgp 002 failed in run_test
cpld_2threads_p7 003 failed in check_result
cpld_2threads_p7 003 failed in run_test
cpld_decomp_p7 004 failed in check_result
cpld_decomp_p7 004 failed in run_test
cpld_mpi_p7 005 failed in check_result
cpld_mpi_p7 005 failed in run_test
cpld_bmark_p7 006 failed in check_result
cpld_bmark_p7 006 failed in run_test
cpld_bmark_mpi_p7 007 failed in check_result
cpld_bmark_mpi_p7 007 failed in run_test
cpld_control_c96_p7 008 failed in check_result
cpld_control_c96_p7 008 failed in run_test
cpld_control_c192_p7 010 failed in check_result
cpld_control_c192_p7 010 failed in run_test
cpld_control_c384_p7 012 failed in check_result
cpld_control_c384_p7 012 failed in run_test
cpld_debug_p7 014 failed in check_result
cpld_debug_p7 014 failed in run_test
control 015 failed in check_result
control 015 failed in run_test
control_decomp 016 failed in check_result
control_decomp 016 failed in run_test
control_2threads 017 failed in check_result
control_2threads 017 failed in run_test
control_fhzero 019 failed in check_result
control_fhzero 019 failed in run_test
control_CubedSphereGrid 020 failed in check_result
control_CubedSphereGrid 020 failed in run_test
control_latlon 021 failed in check_result
control_latlon 021 failed in run_test
control_wrtGauss_netcdf_parallel 022 failed in check_result
control_wrtGauss_netcdf_parallel 022 failed in run_test
control_c48 023 failed in check_result
control_c48 023 failed in run_test
control_c192 024 failed in check_result
control_c192 024 failed in run_test
control_c384 025 failed in check_result
control_c384 025 failed in run_test
control_c384gdas 026 failed in check_result
control_c384gdas 026 failed in run_test
control_stochy 027 failed in check_result
control_stochy 027 failed in run_test
control_lndp 029 failed in check_result
control_lndp 029 failed in run_test
control_p7 030 failed in check_result
control_p7 030 failed in run_test
control_p7_rrtmgp 031 failed in check_result
control_p7_rrtmgp 031 failed in run_test
control_decomp_p7 033 failed in check_result
control_decomp_p7 033 failed in run_test
control_2threads_p7 034 failed in check_result
control_2threads_p7 034 failed in run_test
regional_control 035 failed in check_result
regional_control 035 failed in run_test
regional_noquilt 037 failed in check_result
regional_noquilt 037 failed in run_test
regional_2threads 038 failed in check_result
regional_2threads 038 failed in run_test
regional_hafs 039 failed in check_result
regional_hafs 039 failed in run_test
regional_netcdf_parallel 040 failed in check_result
regional_netcdf_parallel 040 failed in run_test
regional_RRTMGP 041 failed in check_result
regional_RRTMGP 041 failed in run_test
control_rrtmgp 048 failed in check_result
control_rrtmgp 048 failed in run_test
control_rrtmgp_c192 049 failed in check_result
control_rrtmgp_c192 049 failed in run_test
control_csawmg 050 failed in check_result
control_csawmg 050 failed in run_test
control_csawmgt 051 failed in check_result
control_csawmgt 051 failed in run_test
control_flake 052 failed in check_result
control_flake 052 failed in run_test
control_ras 053 failed in check_result
control_ras 053 failed in run_test
control_thompson 054 failed in check_result
control_thompson 054 failed in run_test
control_thompson_no_aero 055 failed in check_result
control_thompson_no_aero 055 failed in run_test
fv3_HAFS_v0_hwrf_thompson 056 failed in check_result
fv3_HAFS_v0_hwrf_thompson 056 failed in run_test
fv3_esg_HAFS_v0_hwrf_thompson 057 failed in check_result
fv3_esg_HAFS_v0_hwrf_thompson 057 failed in run_test
control_wam 058 failed in check_result
control_wam 058 failed in run_test
control_debug 059 failed in check_result
control_debug 059 failed in run_test
control_2threads_debug 060 failed in check_result
control_2threads_debug 060 failed in run_test
control_CubedSphereGrid_debug 061 failed in check_result
control_CubedSphereGrid_debug 061 failed in run_test
control_wrtGauss_netcdf_parallel_debug 062 failed in check_result
control_wrtGauss_netcdf_parallel_debug 062 failed in run_test
control_stochy_debug 063 failed in check_result
control_stochy_debug 063 failed in run_test
control_lndp_debug 064 failed in check_result
control_lndp_debug 064 failed in run_test
control_rrtmgp_debug 065 failed in check_result
control_rrtmgp_debug 065 failed in run_test
control_csawmg_debug 066 failed in check_result
control_csawmg_debug 066 failed in run_test
control_csawmgt_debug 067 failed in check_result
control_csawmgt_debug 067 failed in run_test
control_ras_debug 068 failed in check_result
control_ras_debug 068 failed in run_test
cpld_control_p7 001 failed in check_result
cpld_control_p7 001 failed in run_test
cpld_control_p7_rrtmgp 002 failed in check_result
cpld_control_p7_rrtmgp 002 failed in run_test
cpld_2threads_p7 003 failed in check_result
cpld_2threads_p7 003 failed in run_test
cpld_decomp_p7 004 failed in check_result
cpld_decomp_p7 004 failed in run_test
cpld_mpi_p7 005 failed in check_result
cpld_mpi_p7 005 failed in run_test
cpld_bmark_p7 006 failed in check_result
cpld_bmark_p7 006 failed in run_test
cpld_bmark_mpi_p7 007 failed in check_result
cpld_bmark_mpi_p7 007 failed in run_test
cpld_control_c96_p7 008 failed in check_result
cpld_control_c96_p7 008 failed in run_test
cpld_control_c192_p7 010 failed in check_result
cpld_control_c192_p7 010 failed in run_test
cpld_control_c384_p7 012 failed in check_result
cpld_control_c384_p7 012 failed in run_test
cpld_debug_p7 014 failed in check_result
cpld_debug_p7 014 failed in run_test
control 015 failed in check_result
control 015 failed in run_test
control_decomp 016 failed in check_result
control_decomp 016 failed in run_test
control_2threads 017 failed in check_result
control_2threads 017 failed in run_test
control_fhzero 019 failed in check_result
control_fhzero 019 failed in run_test
control_CubedSphereGrid 020 failed in check_result
control_CubedSphereGrid 020 failed in run_test
control_latlon 021 failed in check_result
control_latlon 021 failed in run_test
control_wrtGauss_netcdf_parallel 022 failed in check_result
control_wrtGauss_netcdf_parallel 022 failed in run_test
control_c48 023 failed in check_result
control_c48 023 failed in run_test
control_c192 024 failed in check_result
control_c192 024 failed in run_test
control_c384 025 failed in check_result
control_c384 025 failed in run_test
control_c384gdas 026 failed in check_result
control_c384gdas 026 failed in run_test
control_stochy 027 failed in check_result
control_stochy 027 failed in run_test
control_lndp 029 failed in check_result
control_lndp 029 failed in run_test
control_p7 030 failed in check_result
control_p7 030 failed in run_test
control_p7_rrtmgp 031 failed in check_result
control_p7_rrtmgp 031 failed in run_test
control_decomp_p7 033 failed in check_result
control_decomp_p7 033 failed in run_test
control_2threads_p7 034 failed in check_result
control_2threads_p7 034 failed in run_test
regional_control 035 failed in check_result
regional_control 035 failed in run_test
regional_noquilt 037 failed in check_result
regional_noquilt 037 failed in run_test
regional_2threads 038 failed in check_result
regional_2threads 038 failed in run_test
regional_hafs 039 failed in check_result
regional_hafs 039 failed in run_test
regional_netcdf_parallel 040 failed in check_result
regional_netcdf_parallel 040 failed in run_test
regional_RRTMGP 041 failed in check_result
regional_RRTMGP 041 failed in run_test
control_rrtmgp 048 failed in check_result
control_rrtmgp 048 failed in run_test
control_rrtmgp_c192 049 failed in check_result
control_rrtmgp_c192 049 failed in run_test
control_csawmg 050 failed in check_result
control_csawmg 050 failed in run_test
control_csawmgt 051 failed in check_result
control_csawmgt 051 failed in run_test
control_flake 052 failed in check_result
control_flake 052 failed in run_test
control_ras 053 failed in check_result
control_ras 053 failed in run_test
control_thompson 054 failed in check_result
control_thompson 054 failed in run_test
control_thompson_no_aero 055 failed in check_result
control_thompson_no_aero 055 failed in run_test
fv3_HAFS_v0_hwrf_thompson 056 failed in check_result
fv3_HAFS_v0_hwrf_thompson 056 failed in run_test
fv3_esg_HAFS_v0_hwrf_thompson 057 failed in check_result
fv3_esg_HAFS_v0_hwrf_thompson 057 failed in run_test
control_wam 058 failed in check_result
control_wam 058 failed in run_test
control_debug 059 failed in check_result
control_debug 059 failed in run_test
control_2threads_debug 060 failed in check_result
control_2threads_debug 060 failed in run_test
control_CubedSphereGrid_debug 061 failed in check_result
control_CubedSphereGrid_debug 061 failed in run_test
control_wrtGauss_netcdf_parallel_debug 062 failed in check_result
control_wrtGauss_netcdf_parallel_debug 062 failed in run_test
control_stochy_debug 063 failed in check_result
control_stochy_debug 063 failed in run_test
control_lndp_debug 064 failed in check_result
control_lndp_debug 064 failed in run_test
control_rrtmgp_debug 065 failed in check_result
control_rrtmgp_debug 065 failed in run_test
control_csawmg_debug 066 failed in check_result
control_csawmg_debug 066 failed in run_test
control_csawmgt_debug 067 failed in check_result
control_csawmgt_debug 067 failed in run_test
control_ras_debug 068 failed in check_result
control_ras_debug 068 failed in run_test
control_diag_debug 069 failed in check_result
control_diag_debug 069 failed in run_test
control_debug_p7 070 failed in check_result
control_debug_p7 070 failed in run_test
control_thompson_debug 071 failed in check_result
control_thompson_debug 071 failed in run_test
control_thompson_no_aero_debug 072 failed in check_result
control_thompson_no_aero_debug 072 failed in run_test
control_thompson_extdiag_debug 073 failed in check_result
control_thompson_extdiag_debug 073 failed in run_test
regional_debug 074 failed in check_result
regional_debug 074 failed in run_test
fv3_HAFS_v0_hwrf_thompson_debug 087 failed in check_result
fv3_HAFS_v0_hwrf_thompson_debug 087 failed in run_test
fv3_esg_HAFS_v0_hwrf_thompson_debug 088 failed in check_result
fv3_esg_HAFS_v0_hwrf_thompson_debug 088 failed in run_test
hafs_regional_atm 089 failed in check_result
hafs_regional_atm 089 failed in run_test
hafs_regional_atm_ocn 090 failed in check_result
hafs_regional_atm_ocn 090 failed in run_test
hafs_regional_atm_wav 091 failed in check_result
hafs_regional_atm_wav 091 failed in run_test
hafs_regional_atm_ocn_wav 092 failed in check_result
hafs_regional_atm_ocn_wav 092 failed in run_test
hafs_regional_1nest_atm 093 failed in check_result
hafs_regional_1nest_atm 093 failed in run_test
hafs_regional_telescopic_2nests_atm 094 failed in check_result
hafs_regional_telescopic_2nests_atm 094 failed in run_test
hafs_global_1nest_atm 095 failed in check_result
hafs_global_1nest_atm 095 failed in run_test
hafs_regional_docn 097 failed in check_result
hafs_regional_docn 097 failed in run_test
hafs_regional_docn_oisst 098 failed in check_result
hafs_regional_docn_oisst 098 failed in run_test
control_atmwav 110 failed in check_result
control_atmwav 110 failed in run_test
control_c384gdas_wav 111 failed in run_test
control_atm_aerosols 112 failed in check_result
control_atm_aerosols 112 failed in run_test

@grantfirl
Copy link
Collaborator Author

It looks like the only test that failed the run_test only was control_c384gdas_wav which had the following error during run:

forrtl: severe (38): error during write, unit 50, file /scratch1/BMC/gmtb/Grant.Firl/FV3_RT/stmp2/Grant.Firl/FV3_RT/rt_15495/control_c384gdas_wav/PET473.ESMF_LogFile

@climbfuji
Copy link
Collaborator

climbfuji commented Dec 22, 2021 via email

@grantfirl
Copy link
Collaborator Author

Looks like a disk quota issue. We got an email this morning, once again, that stmp2 is over quota.

OK, thanks. I'll clear some space and try that test again to make sure.

@grantfirl
Copy link
Collaborator Author

Confirmed that the fail run test was a disk quota issue. Add control_c384gdas_wav to the list of failed check_results.

@SamuelTrahanNOAA
Copy link
Collaborator

The control_flake froze for a half hour without even starting to execute. I didn't even get the initial NEMS message. On a second try, it finished in four minutes. I'm going to assume that was due to a hardware or OS issue unless it happens again.

@SamuelTrahanNOAA
Copy link
Collaborator

Two more tests froze. I attached to a running process stuck in this state, and it appears the PMPI_Init_thread is freezing. That's part of the MPI initialization. This failure is definitely not caused by our code.

@junwang-noaa
Copy link
Collaborator

Code was committed in PR#981.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ensure column mass conservation for tracers in TVD schemes
4 participants