ACCESS-OM3 NUOPC driver choices

[aekiss]

Having adopted NUOPC and the CMEPS mediator, we now need a driver. This discussion is a place to share notes and thoughts about this.

[aekiss]

ESMF 8.4.0 introduced the Earth System Model eXecutable (ESMX) layer - see Daniel Rosen's CW2023 talk and slides for an overview. This provides a driver, and also an executable and a CMake-based build system.

The ESMF 8.4.0 release notes have this caveat:

Exploration and experimentation with ESMX is strongly encouraged. However, please recognize that it is a new development effort that comes with the typical rough edges. For example, no formal reference manual is available yet. A good starting point to learn about the layer is the ESMX/README.md. We expect development of ESMX to continue throughout the next several release cycles.

[micaeljtoliveira]

The alternative to using ESMX is to write the driver ourselves. The work does not need to be done from scratch though, as there are some examples available out there:

• CESM
• UFS

[aekiss]

If possible I prefer the idea of using a shared driver rather than writing our own, so we can reap the benefits of bugfixes and features that others contribute.

[aekiss]

@dougiesquire - were you using the CESM driver in the coupled configurations you were working on?

@kieranricardo, @MartinDix - what driver did you use for your UM-CDEPS configuration? (and did you use the CMEPS mediator?)

[kieranricardo]

@aekiss yep we're using CMEPS along with the driver in the CMEPS repo: https://github.com/ESCOMP/CMEPS/tree/main/cesm/driver. That driver is all set up to be compiled with arbitrary nuopc caps (but the caps can't be changed once compiled) and it's been pretty good to work with

[dougiesquire]

Same for me, though I haven't done anything withe NUOPC since last year

[aekiss]

Thanks. Did either of you use the CMEPS mediator?

[dougiesquire]

Yes, we both use the CMEPS mediator I believe

[aekiss]

I've been chatting with Mariana (@mvertens), who has a deep knowledge of the CESM driver. This seems to be very mature and capable. She commented (emphasis mine):

The CESM driver is closely tied to the CIME Case Control System (CCS) (see https://esmci.github.io/cime/versions/master/html/index.html) - and is the most general driver in terms of the number of components it supports. If you look at the driver you will see that it tries to ingest a set of configuration variables out of the nuopc.config file in your executable directory It also ties in nicely to all of the other workflow capabilities and regression testing framework that comes with the CCS. In addition, both the CIME CCS uses the concept of 'compsets' which is a very flexible way to swap out data components (e.g. DATM for prognostic ATM) when configuring different model experiments

[micaeljtoliveira]

Thanks @aekiss for the information. That confirms impression I got from reading the code of the different drivers.

[aekiss]

In terms of maturity,

• the CESM driver was originally part of CIME (before CMEPS was apparently factored out of CIME), with git history extending back at least to 2018 and (for what it's worth) has the most code (~3100 lines)
• the UFS driver originates from NEMS, and its history stretches back to 2007. It's about 1300 Fortran lines.
• ESMX began in Sep 2022, and seems like an initial prototype: ~600 lines of Fortran, 21 commits, 1 author (the ESMF/NUOPC architect and technical lead)

[aekiss]

@kieranricardo and @dougiesquire - are your configurations on github somewhere?

[micaeljtoliveira]

@dougiesquire has two configurations on github:

https://github.com/dougiesquire/gmom_jra
https://github.com/dougiesquire/gmom_jra_wd

[micaeljtoliveira]

And this one:
https://github.com/dougiesquire/d_jra_wd

[kieranricardo]

I can tidy up my UM-CDEPS config and chuck it on GitHub if that'll be helpful :)

[aekiss]

@kieranricardo that would be great if it's not too much hassle

[dougiesquire]

The repos @micaeljtoliveira linked above are payu configurations to run particular CESM "compsets". GMOM_JRA and GMOM_JRA_WD are existing CESM compsets (though they are officially "unsupported"). D_JRA_WD is a compset I hacked together for CICE6-WW3-DOCN-DATM-DROF-SLND-SGLC. My bodge to build the executable for this compset is all in https://github.com/dougiesquire/d_jra_wd/tree/main/build

[aekiss]

I'm starting to feel like the CESM coupler could be the best choice, but how much of the CIME workflow will we need to adopt? CIME seems to have a lot of nice features, but some of those overlap with solutions we already have, e.g. payu.

[micaeljtoliveira]

My understanding is that we don't really need to use CIME at all to use the CESM coupler. The test configurations above are actually run with payu.

[aekiss]

@dougiesquire - how did you build /scratch/tm70/ds0092/cime/scratch/GMOM_JRA_WD/bld/cesm.exe? I'm wondering if we can merge what you did with the build process we have in this repo.

[kieranricardo]

Looks like we already have a version on GitHub, it's pretty clunky but gets the job done!

https://github.com/ACCESS-NRI/dev_coupling/blob/main/um-cdeps-build/build_um_cdeps.sh

(this should probably be called build_um_cmeps.sh instead)

[kieranricardo]

CMEPS expects each cap to be a module called atm_comp_nuopc, ocn_comp_nuopc etc. and after compiling all the model components and the mediator you just link them all into a single executable.

I'm not sure whether you actually need -I$cdeps_dir/fox/include -I$cdeps_dir/dshr in the compiler flags for the mediator, and if you're adding/removing model components (replacing them with a stub, data model components count as components here!) you need to add/remove -DATM_PRESENT etc.

Edit: turns out you don't any of CDEPS -I$cdeps_dir/fox/include -I$cdeps_dir/dshr to build CMEPS

[aekiss]

great, thanks @kieranricardo!

[kieranricardo]

no worries! I removed some unnecessary flags (no guarantees this is the minimal set)

[micaeljtoliveira]

I wouldn't worry too much about the flags, as CMake will take care of setting all of those for us, as long as we set the correct dependencies for the executable.

[aekiss]

Just linking here to Dougie's Hive posts on his configs: https://forum.access-hive.org.au/t/cesm-configurations-on-gadi-using-cime/115

[aekiss]

Just noting here that we've settled on using the CESM driver from CMEPS - see commits 87097e7 and 83a8083

[MartinDix]

As I understand it, the CESM driver itself calculates the ocean surface fluxes from atmospheric wind, temperature etc. For coupling with the UM we need the fluxes to be calculated in the UM and passed directly. It didn't seem that the CESM driver could do this so we were looking at developing a modified version.

[kieranricardo]

From what I've seen, the CMEPS driver should be general enough to suit our needs, but we will have add to and slightly modify the CMEPS mediator (which is where the flux calculations happen).

It looks like the CMEPS mediator currently has 3 different coupling modes – CESM, NEMS, and HAFS – that can be switched by specifying “coupling_mode” in nuopc.runconfig. I’m playing around with adding a third ACCESS option that will suit our needs. This will involve creating an “esmFldsExchange_access_mod.F90” file and adding support for “coupling_mode == access” in a few places in the mediator. The modifications to the existing code should be pretty minimal, and because they already support three different coupling options we may even be able to merge our mediator changes into the main CMEPS.