Regional run transient noise when fresh lateral boundary information is read

[MatthewPyle-NOAA]

Description

Using the relatively new pressure tendency diagnostic, we’ve seen a slight, transient increase in domain-averaged surface pressure tendency at the frequency of lateral boundary updates in limited-area runs.

To Reproduce:

What compilers/machines are you seeing this with? Has been seen with Intel built code on various WCOSS platforms, but in theory should be compiler/machine independent.

Give explicit steps to reproduce the behavior.

1. Make a regional run of sufficient length so fresh boundary information is read during the model integration. Also, need nrows_blend > 0 for the behavior to be triggered in a noticeable way. Also need print_diff_pgr = .true. to see the pressure tendency information.
2. Given a regional run of this kind, should see an increase in domain-averaged surface pressure tendency after a boundary update time. Example below in the "output logs" portion.

Additional context

Originally opened this issue in the GFDL_atmos_cubed_sphere repository as NOAA-GFDL/GFDL_atmos_cubed_sphere#169 The actual code changes will made to the dev/emc branch of GFDL_atmos_cubed_sphere under this PR: NOAA-GFDL/GFDL_atmos_cubed_sphere#173

Output

output logs

An example using the hafs_regional_atm regression test, but with print_diff_pgr=.true., showing the domain-average pressure changes around the time of lateral boundary information being ingested 3 h into the forecast. This example with the current UFS code:

At forecast hour     2.950 mean abs pgr change is       0.54056187 hPa/hr
At forecast hour     3.000 mean abs pgr change is       0.56172339 hPa/hr
At forecast hour     3.050 mean abs pgr change is       0.60422858 hPa/hr
At forecast hour     3.100 mean abs pgr change is       0.60683374 hPa/hr
At forecast hour     3.150 mean abs pgr change is       0.59024507 hPa/hr

Same case, but with a potential fix in place:

At forecast hour     2.950 mean abs pgr change is       0.54056187 hPa/hr
At forecast hour     3.000 mean abs pgr change is       0.56172339 hPa/hr
At forecast hour     3.050 mean abs pgr change is       0.57903745 hPa/hr
At forecast hour     3.100 mean abs pgr change is       0.57276973 hPa/hr
At forecast hour     3.150 mean abs pgr change is       0.56487195 hPa/hr