GSL atmosphere_model will not reproduce NCAR atmosphere_model

[barlage]

For out-of-the-box mesoscale_reference and convection_permitting suites, the GSL code does not reproduce NCAR code. For example, CONUS mesoscale_reference testcase differences (using code fixing #65) after 1 hour (5 timesteps):

nccmp -dsSqf bar-455a5a9fa.mesoscale_reference.ncar.ncar.conus.120km.gfs.2023031015/history.2023-03-10_16.00.00.nc ncar-v8.2.2.mesoscale_reference.ncar.ncar.conus.120km.gfs.2023031015/history.2023-03-10_16.00.00.nc

Variable               Group Count          Sum      AbsSum          Min          Max       Range         Mean      StdDev
qv                     /     10347 -5.82311e-06 0.000208967 -1.82134e-05  2.44682e-05 4.26816e-05 -5.62783e-10 4.40722e-07
qc                     /       370  1.03647e-05 1.86199e-05 -1.05407e-06  6.84692e-06 7.90099e-06  2.80127e-08 4.23858e-07
qr                     /       856  6.41402e-06 0.000107672 -1.28703e-05  1.21038e-05 2.49742e-05  7.49302e-09 9.11512e-07
qi                     /      4993 -4.43827e-06 1.42143e-05 -2.46453e-06  6.26966e-07  3.0915e-06 -8.88899e-10 5.44309e-08
qs                     /      3897   5.7315e-06 1.21774e-05 -4.81627e-07  4.72372e-06 5.20535e-06  1.47075e-09 8.52991e-08
qg                     /      3406  2.76559e-08 1.63107e-07 -6.19158e-09  1.77972e-08 2.39888e-08  8.11976e-12 4.73913e-10
u                      /     63447  0.000180574    0.116743 -0.000180483  0.000187874 0.000368357  2.84606e-09 5.37565e-06
w                      /     46769  0.000399901  0.00722084 -7.26432e-06  6.59935e-06 1.38637e-05  8.55055e-09  3.8066e-07
pressure               /      2368      7.90759     196.563     -20.7344      15.1797     35.9141   0.00333936    0.810607
surface_pressure       /        32      5.74219     23.2109     -5.53125      9.98438     15.5156     0.179443     2.16991
rho                    /      2690  -9.0152e-07 0.000141688 -1.07288e-06  1.66893e-06 2.74181e-06 -3.35138e-10 8.91583e-08
theta                  /      2485    0.0183105    0.660706    -0.063324    0.0474854    0.110809  7.36843e-06  0.00236695
relhum                 /     10460    -0.627489     10.8649    -0.677376     0.561337     1.23871 -5.99894e-05   0.0136334
divergence             /     30253 -2.11224e-09 7.54819e-07 -1.15688e-09   1.1014e-09 2.25828e-09 -6.98191e-14  5.1891e-11
vorticity              /     46096 -1.67385e-09 1.90474e-06 -3.60251e-09  3.56295e-09 7.16545e-09 -3.63123e-14 1.15045e-10
ke                     /     20974    0.0046308    0.840786  -0.00610352   0.00390625   0.0100098  2.20788e-07 0.000134643
uReconstructZonal      /     18379  -3.7672e-05   0.0385466 -0.000185966  0.000179291 0.000365257 -2.04973e-09 5.69106e-06
uReconstructMeridional /     22521 -0.000259013   0.0263623 -0.000149727  0.000125885 0.000275612 -1.15009e-08 3.19913e-06
ertel_pv               /     27142   -0.0163986    0.591087   -0.0217799    0.0208461    0.042626 -6.04177e-07 0.000354364
u_pv                   /       368  0.000276625  0.00286525 -7.24792e-05    0.0001297 0.000202179  7.51699e-07 1.45387e-05
v_pv                   /       506  8.92244e-05  0.00188054 -3.24249e-05  6.86646e-05 0.000101089  1.76333e-07 7.37711e-06
theta_pv               /       145  6.10352e-05  0.00549316  -0.00012207  9.15527e-05 0.000213623  4.20932e-07 4.14751e-05
vort_pv                /       609  -9.4181e-10 1.76564e-08 -4.07454e-10  4.65661e-10 8.73115e-10 -1.54649e-12 6.01499e-11
depv_dt_lw             /     24734 -1.07915e-09 4.36438e-08 -1.00408e-09  1.00408e-09 2.00816e-09   -4.363e-14 1.63292e-11
depv_dt_sw             /     25137 -8.64733e-11 2.74772e-08 -4.94765e-10  3.38332e-10 8.33097e-10 -3.44008e-15 6.39317e-12
depv_dt_bl             /     25677 -2.67592e-07 5.87958e-06 -2.67306e-07  1.22556e-07 3.89862e-07 -1.04215e-11 3.49661e-09
depv_dt_cu             /     10439  2.14703e-07 7.74505e-06 -2.43279e-07  2.45578e-07 4.88857e-07  2.05674e-11 7.40202e-09
depv_dt_mix            /     35708 -6.43592e-10 5.01075e-07 -1.38334e-09  1.74805e-09 3.13139e-09 -1.80237e-14 4.88766e-11
dtheta_dt_mp           /      1095  2.22524e-05 0.000805791   -8.795e-05  6.59519e-05 0.000153902  2.03218e-08 4.95232e-06
depv_dt_mp             /     28514 -2.07761e-05 0.000592738 -3.02505e-05  2.89525e-05  5.9203e-05 -7.28629e-10 4.78172e-07
depv_dt_diab           /     33253 -2.08312e-05 0.000599537 -3.02505e-05  2.89525e-05  5.9203e-05 -6.26446e-10 4.42804e-07
depv_dt_fric           /     37552 -2.25347e-06 5.07487e-05   -6.519e-06  3.02762e-06 9.54661e-06 -6.00094e-11 4.35107e-08
depv_dt_diab_pv        /       688  6.55136e-09   3.939e-07 -1.61867e-08    1.321e-08 2.93967e-08  9.52233e-12 1.84542e-09
depv_dt_fric_pv        /       721 -1.27339e-08 5.37006e-08 -9.77889e-09  9.24047e-10 1.07029e-08 -1.76614e-11 3.85797e-10
rainnc                 /       276  -0.00226518  0.00670537  -0.00353809   0.00143323  0.00497132 -8.20717e-06 0.000236026
precipw                /       245  -0.00163984  0.00830603  -0.00219202   0.00242233  0.00461435 -6.69324e-06 0.000244621
cuprec                 /        26   8.3805e-10 1.45554e-08 -2.38833e-09  2.14641e-09 4.53474e-09  3.22327e-11 9.76563e-10
rainc                  /        24  6.09372e-07 1.04721e-05 -1.72015e-06  1.54227e-06 3.26242e-06  2.53905e-08 7.32668e-07
hpbl                   /        21    0.0842514    0.193611   -0.0121536     0.100037     0.11219   0.00401197   0.0228747
hfx                    /        21  -0.00104582  0.00382769  -0.00105286  0.000692368  0.00174522 -4.98011e-05 0.000363773
qfx                    /        29 -1.80762e-11 3.23894e-10 -1.05501e-10  4.72937e-11 1.52795e-10 -6.23317e-13 2.34702e-11
cd                     /         9  7.45058e-09 2.39816e-08 -5.58794e-09  5.12227e-09 1.07102e-08  8.27842e-10  3.3369e-09
cda                    /        11  6.40284e-09 2.66591e-08 -3.72529e-09   6.0536e-09 9.77889e-09  5.82077e-10 2.96025e-09
ck                     /        11  7.21775e-09 1.16415e-08 -9.31323e-10  2.32831e-09 3.25963e-09  6.56159e-10 1.21125e-09
cka                    /        15  7.91624e-09  1.3737e-08 -9.31323e-10   3.0268e-09 3.95812e-09  5.27749e-10 1.31699e-09
lh                     /        30 -9.01222e-05  0.00079298 -0.000267029  0.000118256 0.000385284 -3.00407e-06 5.81692e-05
u10                    /        54  1.04308e-07 2.16812e-05 -2.38419e-06  1.90735e-06 4.29153e-06  1.93163e-09 6.00568e-07
v10                    /        47  2.53553e-06 2.12169e-05 -1.19209e-06  1.43051e-06  2.6226e-06  5.39474e-08 5.85776e-07
q2                     /        19   -0.0110567    0.017515   -0.0142858   0.00118274   0.0154686  -0.00058193  0.00334108
t2m                    /         9     -37.3928     37.3929     -20.4113  6.10352e-05     20.4114     -4.15475     7.51508
th2m                   /         9     -37.4258     37.4258      -20.416  3.05176e-05      20.416     -4.15842     7.52054
acsnow                 /        14 -1.35231e-07 1.53334e-07  -1.2666e-07  3.72529e-09 1.30385e-07 -9.65935e-09 3.38508e-08
skintemp               /         3 -9.15527e-05 9.15527e-05 -3.05176e-05 -3.05176e-05           0 -3.05176e-05           0
snow                   /         2 -4.05312e-06 4.05312e-06 -2.14577e-06 -1.90735e-06 2.38419e-07 -2.02656e-06 1.68587e-07
snowh                  /         2 -2.14204e-08 2.14204e-08 -1.11759e-08 -1.02445e-08 9.31323e-10 -1.07102e-08 6.58545e-10

after one timestep:

Variable Group Count        Sum     AbsSum         Min        Max       Range        Mean      StdDev
q2       /         5 0.00390335 0.00390335 0.000355857 0.00104086 0.000685005 0.000780671 0.000263849
t2m      /         5    11.7124    11.7124    0.702576    3.71896     3.01639     2.34247     1.22392
th2m     /         5    11.7211    11.7211    0.704285    3.72086     3.01657     2.34422     1.22515

[barlage]

There are 5 seaice points in the domain, so exploring the diffs with seaice.

[barlage]

Does anyone know why we are deallocating these four variables?

Compared to the NCAR code?

Seems like that would mean that we aren't carrying the sfclayer calculated values to the downstream calls. Is that intentional? It's causing my current test code to crash (I believe).

[tanyasmirnova]

Does anyone know why we are deallocating these four variables?
These are the variables, correct:
if(allocated(chs_sea) ) deallocate(chs_sea )
if(allocated(chs2_sea) ) deallocate(chs2_sea )
if(allocated(cqs2_sea) ) deallocate(cqs2_sea )
if(allocated(cqs_sea) ) deallocate(cqs_sea )

Compared to the NCAR code?

I think it is a bug. These variables are not sent to MPAS in sfclayer_to_MPAS, thus they will be not defined when they are needed in mpas_atmphys_driver_seaice.F90

Seems like that would mean that we aren't carrying the sfclayer calculated values to the downstream calls. Is that intentional? It's causing my current test code to crash (I believe).

[barlage]

@tanyasmirnova yes, those are the four that stand out, I haven't looked too closely to see if there are others. I will make some changes to not deallocate these. This may or may not change the hrrrv5 baseline.

[joeolson42]

Tanya, is it an oversight that they are NOT sent to the MPAS in sfclayer_to_MPAS? Maybe the seaice coupling hasn't been completed. In any case, I think you're on the right track to figure out this non-reproducibility issue.. Also, in the NCAR code, these lines seem like a bug, right: if(allocated(flhc_p) ) deallocate(flhc_sea ) if(allocated(flqc_p) ) deallocate(flqc_sea ) Is there a reason why the variables are different in each line? The logic is different from all the other lines around it.

…

On Fri, Nov 8, 2024 at 7:29 AM Michael Barlage ***@***.***> wrote: @tanyasmirnova <https://github.com/tanyasmirnova> yes, those are the four that stand out, I haven't looked too closely to see if there are others. I will make some changes to not deallocate these. This may or may not change the hrrrv5 baseline. — Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLRR3WBPFF6T754J2PJV4DZ7TDDDAVCNFSM6AAAAABRJWUVJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRUHEYDENRVGM> . You are receiving this because you were assigned.Message ID: ***@***.***>

-- Joseph Olson Numerical Weather Prediction Model Developer Environmental Prediction Advancement Division NOAA-Global Systems Laboratory Boulder, Colorado

[barlage]

Although it has been a bit tedious, it has generally been a positive thing for me to diagnose these things since I think I understand the code a little better each time. I couldn't understand how some of these variables are being used across schemes. Now I see that pretty much all this memory passing is done through mpas_atmphys_vars. I am going to look at the entire sfclayer driver to try and reduce changes with ncar code. I'm encouraged that this is the issue with reproducibility. I'll post the branch here shortly for you all to see.

[tanyasmirnova]

@joeolson42 Joe, I think that it is confusing that some variables are sent back to MPAS state, and others are not. When Laura moved the 2-m diagnostics to seaice driver, I think it would be cleaner to get all variables used in there from MPAS grid. I guess she had some reasoning behind not doing it for these four *_sea variables, but I have no idea what it was.

[barlage]

I think I understand the logic. Any vars that you want to persist forever are sent to the MPAS state variables. But any vars you only want to persist for the physics timestep are put into these _p and _sea variables. For these phys timestep vars that you only want to persist for your scheme's call, they are allocated before the scheme and deallocated after. For those that are going to be used downstream (but still in the physics timestep), those should not be deallocated after the scheme, but deallocated after the last scheme that uses it. That's why several of these surface layer variables are not deallocated after the sfclayer driver, but deallocated either after the lsm driver or seaice driver. Overall, I think the purpose is to optimize memory management.

[joeolson42]

My understanding is similar to yours, but I am a bit confused as to why some of those *_p variables persist from driver to driver when they could be sent "back to MPAS" by setting their permanently allocated counterparts equal to the *_p variables. Then it would be straightforward to have each of the *_p variables deallocated before leaving the driver. I think this is how it's done for most variables held in memory, even if they aren't really "state" variables. There just happens to be a few exceptions and I can't help to wonder if they may be unintentional bugs. I think it would be cleaner to send any variable that needs to be kept in memory to the non-*_p counterpart. Does that make sense? We are all trying to learn MPAS by reading the current code which may be buggy. I think a lot of us are confused about how it is actually meant to be coded.

…

On Fri, Nov 8, 2024 at 11:12 AM Michael Barlage ***@***.***> wrote: I think I understand the logic. Any vars that you want to persist forever are sent to the MPAS state variables. But any vars you only want to persist for the physics timestep are put into these _p and _sea variables. For these phys timestep vars that you only want to persist for your scheme's call, they are allocated before the scheme and deallocated after. For those that are going to be used downstream (but still in the physics timestep), those should not be deallocated after the scheme, but deallocated after the last scheme that uses it. That's why several of these surface layer variables are not deallocated after the sfclayer driver, but deallocated either after the lsm driver or seaice driver. Overall, I think the purpose is to optimize memory management. — Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLRR3RCNVSTUTFRLIIYO6DZ7T5JRAVCNFSM6AAAAABRJWUVJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRVGQ3TAMZSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Joseph Olson Numerical Weather Prediction Model Developer Environmental Prediction Advancement Division NOAA-Global Systems Laboratory Boulder, Colorado

[barlage]

It certainly makes the current deallocation process messy since it's not clear what has been deallocated after a scheme driver call and what hasn't. It seems like it would be better to check/deallocate everything in mpas_atmphys_vars at the end of the physics driver calls but I guess that would have a larger memory footprint.

But what you're saying definitely makes sense for the vars that have both state and _p counterparts, otherwise what does the state contain??