Rearrange allocation of submodels across nodes

[aekiss]

Back in the May TWG meeting we noted that YATM is allocated to the first PE on the first node, followed by the root PE of MOM on the same node. We discussed whether another placement of accessom2 components on nodes would be preferable for IO etc.

Should the order be CICE, MOM, YATM so the 3 root PEs are on separate nodes?

Is this simply a matter of changing the order of the submodels in config.yaml?

[russfiedler]

I think that originally (long time ago) that was the order but it got changed for some reason. OASIS related I would guess but maybe YATM relies on being the root PE of MPI_COMM_WORLD. I'd hope not but I suppose it could be tested pretty quickly with the 1 degree config to see if there are any potential problems.

[aekiss]

Unfortunately your hunch was right @russfiedler.

With submodules rearranged as MOM, CICE, YATM it exited with

assertion failed: matmxx does not have global PE == 0
1
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
assertion failed: accessom2_sync_config: Unsupported calendar type
1

with the last 2 lines repeated 240 times (which is the total number of MOM+CICE cpus).
see
/home/156/aek156/payu/testing/all-configs/v2.0.0rc6/1deg_jra55_iaf_v2.0.0rc6_iss212

The first assertion that fails is in accessom2.F90. @nichannah - I guess this is needed for a good reason?

[aekiss]

slack discussion: https://arccss.slack.com/archives/CC78YM9EE/p1594782404008100