C-grid coupled validation

[apcraig]

We are trying to validate the C-grid code in UFS, CESM, and RASM. Some initial discussion is here, #717. In summary, UFS is not bit-for-bit when updating to the C-grid version. We suspect this is due to changes in the coupling/forcing averaging/handling. We tested changing the uocn/vocn/ss_tltx/ss_tlty from an "S" averaging to and "F" averaging in UFS and this did not fix the problem, although it may play a role. Additional debugging needs to be done. @DeniseWorthen is working on UFS, @dabail10 is working on CESM, @apcraig is working on RASM. The UFS results show largest differences after one coupling period on the trigrid seam, so tripole vs displaced pole could play and role, and we need to consider that.

As a next step, the CICE in RASM will be updated to pre and post C-grid versions and run side-by-side to check results. UFS will be run with ndte=1 to more cleanly assess differences after one timestep. CESM is planning to create pre and post C-grid branches for validation as well.

[DeniseWorthen]

I wasn't able to run with ndte=1 (bad departure point error) but with ndte=2, the anomaly along the seam appears

[DeniseWorthen]

I believe I have a set of changes for my cgrid branch which reproduce our pre-cgrid baselines. The changes are shown in this comparison. I'm not sure that these are the minimum required changes however, so I'll do some more testing.

[apcraig]

@DeniseWorthen, WOW! That's great. Looks pretty reasonable to me. I think I understand these changes. It looks like it's mostly making the uocn/vocn/ss_tltx/ss_tlty consistent with what was there before. But then there are a few missing halo updates.

We will have to decide how to proceed. Obviously, you can do whatever you want in the cap. The changes in ice_grid should be fine, although I wonder why testing in standalone didn't pick up that issue. The halo updates there and in ice_dyn_evp should affect all cases to some degree.

The main thing we'll have to discuss is the change of "F" to "S" mapping in ice_dyn_evp.F90. I think standalone will want to stick with "S" and I think "S" is the correct mapping for those state fields. But the mapping was "F" for coupled systems. Do we need to add a namelist to allow the user to control the mapping type? Do we all agree "S" is the right thing to do and so coupled results will change?

Again, many thanks for your efforts!!

[dabail10]

Thanks so much for this. I am still working on getting the C-grid version working in CESM. I hope to have that before the end of the week.

[DeniseWorthen]

I've tested and found I can remove the added HaloUpdates in ice_dyn_evp (for tmass and aice_init), which I anticipated. However, I cannot remove the added HaloUpdates in ice_grid for tarea, tarear, uarea, uarear and retain B4B. It does seem curious that those aren't getting picked up as required in the standalone.

I agree the main issue is the F/S mapping and that S may be correct so we need to accept B4B changes. I'm trying to understand why they would have been mapped as fluxes to begin with.

[apcraig]

Thanks for the update @DeniseWorthen. Sounds good.

Part of the reason the "F" was used before is because that's what existed for averaging T to U (and vv). There really wasn't an "S" except for regular rectangular grids. Going thru the C grid implementation highlighted a number of small issues in handling of the grid and forcing. The grid discretization is actually quite poor, and we plan to provide an option to improve this, #653. We also found issues in the locations of a few fields (T vs U) largely related to forcing/coupling, and we added several new options for handling grid averaging to improve that.

As we migrated code that lived in the standalone forcing or in the coupling layer to a more central location with more general capability to handle A/B/C/CD coupling/forcing fields vs internal ice fields , it's probably not too surprising that we're running into a few of these issues.

Ultimately, it's up to you and @dabail10 to decide what you need/want for your coupled systems. I do think "S" is the better interpolation for the ocean velocity and surface tilt terms, but happy to discuss further and ultimately support multiple options for these issues.

[apcraig]

FYI, I have the current version of CICE (with c-grid) running in RASM. Answers are different, but haven't look beyond that yet. Will do some more testing, mainly to confirm the cap is OK. I think with @DeniseWorthen efforts, we have a good handle where answers changes are coming in.

I did find the problem noted by @dabail10 in terms of "USE_NETCDF" in the io_pio2 query var routine. We will want to fix that at some point in one of the PRs. I may create a PR for "RASM fixes", still TBD.

[dabail10]

Just an update here. I am having a hard time getting the C-grid code to run with CESM. Actually, I can't even get the version before the C-grid merge to work. It could be a CAP issue, but I wonder if it also might be the OMP updates. I will try to make some progress on it in the next couple days.

[apcraig]

More info on RASM. If I run an f19_g16 G case, I get identical answers for a 10 day run before and after the cgrid merge if I use @DeniseWorthen's mods. Without those mods, answers are different. So that is consistent with what @DeniseWorthen got.

Running our 9km regional Arctic grid, the results are somewhat different. I am not seeing bit-for-bit before and after the cgrid merge even with @DeniseWorthen's mods. But, it is bit-for-bit after the cgrid merge with and without @DeniseWorthen's mods. So that is almost the opposite! I've tried to do some additional testing and still don't have a good sense why the results are turning out this way. It could be that the mods don't play a role in the 9km regional Arctic grid, it's a regular 9km x 9km regional grid. It could be that the "S" and "F" averaging is the bit-for-bit on a regular grid and that the haloupdates play no role. The slightly more concerning thing is that I haven't figure out yet why the results are different before and after the cgrid update. I'll either have to get into a debugging exercise or just accept it. My main worry is that something is not quite right for the 9km configuration in particular. I will probably spend a bit more time debugging this.

But separately, I think it's good news the f19_g16 case in RASM is bit-for-bit once we include @DeniseWorthen's mods. Further confirmation that we understand the differences in general.