convection_permitting suite does not run

[barlage]

@haiqinli [ @jderrico-noaa @joeolson42 @tanyasmirnova @mdtoyNOAA @AndersJensen-NOAA ] (can we tag Georg?)

I'm trying to establish a baseline for the convection_permitting suite and I'm not getting the current model to run with the same namelist that I use for the NCAR code. I've run three test cases with different compiler flags. The base directory is here (jet:/lfs5/BMC/wrfruc/Michael.Barlage/mpas/mpas_testcase/run_case):

gsl-26ef9904f.convection_permitting.ncar.ncar.conus.120km.gfs.2023031015
gsl-26ef9904f-gsltest.convection_permitting.ncar.ncar.conus.120km.gfs.2023031015
gsl-26ef9904f-debug.convection_permitting.ncar.ncar.conus.120km.gfs.2023031015

you'll get the most information from the debug case, basically this is the error (I put the slurm file in that directory)

################################################################
# Run the model
################################################################
forrtl: warning (406): fort: (1): In call to GET_ZU_ZD_PDF_FIM, an array temporary was created for a
rgument #2

Image              PC                Routine            Line        Source             
atmosphere_model.  00000000033FAAF8  module_cu_gf_deep        4939  module_cu_gf_deep.F
atmosphere_model.  0000000003288CFE  module_cu_gf_deep         904  module_cu_gf_deep.F
atmosphere_model.  00000000017593B3  module_cu_gf_mp_c         516  module_cu_gf_mpas.F
atmosphere_model.  0000000000B9F30D  mpas_atmphys_driv         494  mpas_atmphys_driver_convection.F
atmosphere_model.  0000000000B5FC08  mpas_atmphys_driv         343  mpas_atmphys_driver.F
atmosphere_model.  0000000000944888  atm_core_mp_atm_d        1004  mpas_atm_core.F
atmosphere_model.  000000000093FC72  atm_core_mp_atm_c         787  mpas_atm_core.F
atmosphere_model.  000000000041BA0C  mpas_subdriver_mp         417  mpas_subdriver.F
atmosphere_model.  000000000041595D  MAIN__                     20  mpas.F

[haiqinli]

Thanks Mike. We also noticed this, and I will fix this in the pending PR (#71).

[barlage]

@haiqinli can you tag Georg? I can't find his handle.

[barlage]

@haiqinli is this error related to #71 or is it a separate issue? I think optimally if it is a separate issue then it should be created in a separate, simpler PR, but that's just my opinion.

[haiqinli]

@barlage @GeorgGrell The fix has been included in the updates of PR #71. We can further discuss how to proceed it during this afternoon's meeting. Thanks.

[barlage]

@haiqinli @GeorgGrell I was incorrect that this was a convection error and as Georg mentioned it is just bunch of warnings. The crash is actually occuring in the microphysics.

@AndersJensen-NOAA I think there are some issues here with the nested case statements. Some of these thompson variables, like ni_p, are not being allocated when the scheme is active since they are nested under the tempo and nssl config cases.

[AndersJensen-NOAA]

Good catch @haiqinli and @barlage. I'm testing a fix now: https://github.com/AndersJensen-NOAA/MPAS-Model/tree/bugfix/nest_mp_cases

[barlage]

@AndersJensen-NOAA these nested case statements are confusing and I don't see why they are necessary but I guess that's inherited from the ncar code

[AndersJensen-NOAA]

@barlage and @haiqinli This fixed code compiles, but I won't have time to test the code until next week. Nested case statements are new to me, but we are stuck with them until NCAR makes the change.

[louiswicker]

Hey all, I talked to Clark today about some problems here using 8.2.2-11 woth the convective suite. It does not work for YSU and MYJ, and NSSL-2M microphysics.

Was this issue ever resolved?

[barlage]

@louiswicker I don't think @AndersJensen-NOAA completed his final testing on his code fix for this nested case statement issue. @clark-evans reported to us on your conversation and I suspect this is the same issue that you are having. There is a mess of a nested case statement that got overly complicated when more MP schemes were added.

[barlage]

@louiswicker FWIW, I don't think testing earlier versions is worth the effort since I believe this has been broken since the v8.2.2 merge.

[louiswicker]

Absolutely agree - I was trying figure out where the break happened. Thank you for your time responding.

[clark-evans]

Thank you both for continuing the discussion here. I didn't recall this issue when I talked with @louiswicker yesterday as we haven't really touched it in months, and I got stuck on my prior of the 8.2.2-2.11 testing issues @barlage started to encounter yesterday.

I'm adding the table of working/not working configurations from Lou's email to this thread for future reference. This is based on our 8.2.2-2.9 code.

[barlage]

PR #116 opened to fix convection_permitting suite. Because there are changes to the GF code, this suite will not be reproducible compared to NCAR, but should be reproducible for future GSL comparisons. We could propose a cu_scheme="none" test for NCAR comparison.

FYI @louiswicker