DATM-MOM6-CICE6 is not restart reproducible

[binli2337]

Description

The following three tests have been done:

1. datm_12h_cfsr (The model is run for 12 hours using CFSR forcing.)
2. datm_6h_cfsr (The model is run for 6 hours using CFSR forcing.)
3. datm_restart_cfsr (The model is run from hour 6 to hour 12 using restart files generated from test 2.)

Results from test 1 and test 3 are different at the end of the runs.

To Reproduce:

What compilers/machines are you seeing this with?
Tests are performed on Hera using Intel compiler.

Give explicit steps to reproduce the behavior.

1. git clone https://github.com/binli2337/ufs-weather-model
2. cd ufs-weather-model
3. git checkout feature/restart_test
4. git submodule update --init --recursive
5. cd tests
6. ./rt.sh -k -l rt.conf1

Output from the tests can be found on Hera at /scratch1/NCEPDEV/stmp2/Bin.Li/FV3_RT/RESTART_TESTS.

[DeniseWorthen]

Bin, Could you please try using this branch of MOM6? It contains the vertex shear halo bug fix which has been shown to be needed for restart repro using the coupled model 1deg restart test (#227 (comment))

[binli2337]

@DeniseWorthen ,Thanks.
The updated MOM6 has been used in new tests and the model is still not restart reproducible.

In a constant forcing test, where the CFSR forcing does not change over time, the model is still not restart reproducible.

[DeniseWorthen]

In debug mode, I was able to get this test branch of ufs-datm to restart-repro.

In non-debug mode the model, coupler history files placed in the run sequence after ice->med showed that CICE was not reproducing on the first timestep. This was despite the fact that all fields being sent by the Atm and Ocn to CICE were identical at this point in the run sequence.

By adding the compile option -fp model precise to the CMakeLists.txt in CICE-interface, restart repro was possible. The compiler option -fp-model precise was previously used in the pre-cmake cice6 build for intel compilers on hera,orion, wcoss-dell-p3 and cheyenne but this setting was not carried over in the cmake build. Adding the flag to the CMakeLists.txt may not be the optimal solution. I have created Issue #302 since a solution may also be required for the MOM6 build in ufs-datm.

Additional notes:

1. During the debugging of this issue, a bug was discovered in the time-interpolation of the DATM fields. I had made this change earlier in a personal DATM branch but it was mistakenly not committed. The issue has no impact on restart-reproducibility. The bug is described in Issue #20.

2. The 1-deg ufs-datm is running at dt-atmos=900s. This is a legacy setting from the C96 ATM. During the debugging of this issue, I used a dt-atmos (=dt-ice) of 3600s. We need to evaluate the correct timestep required for the 1-deg ufs-datm. I was also testing using a sequential run sequence since the coupler history files are easier to interpret in this case. I have created issue #301 to track evaluation of these alternatives.

3. The restart test implemented in this branch uses only 2 tests, with checkpoint restarts written at hour6 during the 12h run used to run the restart test.