Distributed ensemble simulations

[glwagner]

This PR implements MPI-based DistributedInverseProblem, and supercedes #291 and #294.

[navidcy]

Should this

ParameterEstimocean.jl/test/test_ensemble_kalman_inversion.jl

Line 99 in e658ac9

@test length(eki.iteration_summaries) == iterations + 3

be

@test length(eki.iteration_summaries) == iterations + 1

?

[navidcy]

Here,

ParameterEstimocean.jl/test/test_ensemble_kalman_inversion.jl

Line 160 in e658ac9

for field_name in keys(fields(model))

the loop is over all model fields (u, v, w, b) but obs don't have w:

ParameterEstimocean.jl/test/test_ensemble_kalman_inversion.jl

Line 28 in e658ac9

observation = SyntheticObservations(data_path, field_names=(:u, :v, :b))

[navidcy]

@glwagner have a look at 936f2c3

[navidcy]

https://github.com/CliMA/ParameterEstimocean.jl/actions/runs/3372540837/jobs/5596093372#step:6:282

Seems like the try-catch for using NCDatasets doesn't do the job?

Was there always a problem with netcdf + linux x86 or is it only after Julia v1.8 that problems started? Because this looks like it wasn't dealing with any probs...

[glwagner]

Should this

ParameterEstimocean.jl/test/test_ensemble_kalman_inversion.jl

Line 99 in e658ac9

@test length(eki.iteration_summaries) == iterations + 3

be

@test length(eki.iteration_summaries) == iterations + 1

?

Yeah.

[glwagner]

https://github.com/CliMA/ParameterEstimocean.jl/actions/runs/3372540837/jobs/5596093372#step:6:282

Seems like the try-catch for using NCDatasets doesn't do the job?

Was there always a problem with netcdf + linux x86 or is it only after Julia v1.8 that problems started? Because this looks like it wasn't dealing with any probs...

I also found that it wasn't sufficient.

[glwagner]

Was there always a problem with netcdf + linux x86 or is it only after Julia v1.8 that problems started? Because this looks like it wasn't dealing with any probs...

My impression is that problems start with julia 1.8. But I don't know for sure.

[navidcy]

Alexander-Barth/NCDatasets.jl#150

[navidcy]

I think this is ready. I'd like to merge CliMA/Oceananigans.jl#2786 first and then use Oceananigans v0.78.1 here.