Use ClimaCore 0.11 and MPITrampoline

[Sbozzolo]

No description provided.

[Sbozzolo]

Closes #2138

[Sbozzolo]

I am trying to get MPI to work.

I managed to have it working here: https://buildkite.com/clima/climaatmos-ci/builds/15017

Then, I pushed again the same commit, and it is failing:
https://buildkite.com/clima/climaatmos-ci/builds/15020

The main difference is that in one case the depot was cold, so everything had to be compiled.

[Sbozzolo]

In this build:

https://buildkite.com/clima/climaatmos-ci/builds/15043#018bf91f-8c1f-4f82-8a8e-a5b4cd410de8

CA.solve_atmos!(CA.get_integrator(CA.AtmosConfig()) run on 2 processes, but not the MPI example

      - label: "ClimaAtmos"
        command: "julia --color=yes --project=examples -e 'import ClimaAtmos; ClimaAtmos.solve_atmos!(ClimaAtmos.get_integrator(ClimaAtmos.AtmosConfig()))'"
        agents:
          slurm_ntasks: 2
          slurm_mem: 16G
        env:
          CLIMACOMMS_CONTEXT: "MPI"

[szy21]

Maybe change the CLIMACORE_DISTRIBUTED: "MPI" in MPI example to CLIMACOMMS_CONTEXT: "MPI"?

[Sbozzolo]

Maybe change the CLIMACORE_DISTRIBUTED: "MPI" in MPI example to CLIMACOMMS_CONTEXT: "MPI"?

Failing https://buildkite.com/clima/climaatmos-ci/builds/15045

[simonbyrne]

For some reason it is using the wrong preferences:
https://buildkite.com/clima/climaatmos-ci/builds/15080#018bfd3b-6aed-4bf1-8208-d1fecdc8dc1b/160-167

[Sbozzolo]

For some reason it is using the wrong preferences: https://buildkite.com/clima/climaatmos-ci/builds/15080#018bfd3b-6aed-4bf1-8208-d1fecdc8dc1b/160-167

Thanks for the lead, let me investigate this.

[simonbyrne]

Ah, I think it's because it's picking up the preferences from the shared depot (which was set previously).

├ cat /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/environments/v1.9/LocalPreferences.toml
[MPIPreferences]
__clear__ = ["preloads_env_switch"]
_format = "1.0"
abi = "OpenMPI"
binary = "system"
cclibs = []
libmpi = "libmpi"
mpiexec = "mpiexec"
preloads = []

either delete that file, or just disable the shared depot for now.

[Sbozzolo]

Ah, I think it's because it's picking up the preferences from the shared depot (which was set previously).

├ cat /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/environments/v1.9/LocalPreferences.toml
[MPIPreferences]
__clear__ = ["preloads_env_switch"]
_format = "1.0"
abi = "OpenMPI"
binary = "system"
cclibs = []
libmpi = "libmpi"
mpiexec = "mpiexec"
preloads = []

either delete that file, or just disable the shared depot for now.

Yes, I just arrived at the same conclusion, which also explains why it was working when the depot was empty.

I'll clean up this PR, clean the depot, and try again.