Add handling of spatial simulations

[ccarouge]

Tests for the spatial/global configuration for CABLE offline should be the next configuration we add to the test suite (at least before CASA).

Sub-tasks for this issue include:

• Defining some data structure or class describe a spatial test. This will probably contain information such as the branch, the science configuration used for testing, and the met forcing.
• Organise model outputs produced from the spatial tests into a suitable directory structure.
• Running tests with payu. We will want to run with payu because of the following reasons:
  1. Encourage uptake of payu amongst users of CABLE.
  2. We will want to test CABLE online in benchcab and payu is routinely used for running an online configuration of CABLE.
  3. Using payu allows us to test longer running simulations easily (payu makes it easy to run a model multiple times and have state persist in the model via restart files).
• Merge in Add CABLE model driver payu-org/payu#314
• Add ability to build CABLE with MPI at runtime.
• Add payu in the benchcab conda environment #235

Spatial met forcing files and an example run directory are available here: /g/data/tm70/ccc561/CABLE/CABLE-as-ACCESS_Yingping. This can be used as inspiration for our run directories for the spatial tests. Note, if we use the same configurations we should:

• remove CASA from the run.
• make the namelist file as similar to the site runs. This isn't completely necessary but prevents us from eventual complications due to CABLE options.

[ccarouge]

For the spatial simulations, could we design it so that we have a base work directory (like bench_example) for each met. forcing source we want to use. This is the equivalent of the flux site met. forcing. And then, a list of namelist options is used to run various science configurations with each met. forcing.

Design questions to settle:

• how to run for several years for one met. forcing? Traditionally, met. forcing files are annual files which sets the length of the CABLE simulation. But we will want benchcab to run for several years for each task.
• how to deal with spin-up for the slow parts such as carbon stocks? Is it ok to provide a common starting point for all tasks?

[SeanBryan51]

For the directory structure, this is what I have as a plan so far:

runs/
├── spatial
│   └── tasks
│	├── <spatial-task-name> (a payu control / experiment directory)
│	└── ...
├── payu-laboratory
│   └── ...
└── fluxsite
    └── ...

The runs/spatial/tasks directory contains a bunch of payu control directories (or experiments) which are configured for each spatial task. Similar to the fluxsite case, each spatial task will have a unique realisation, science configuration and met forcing.

Note we have a separate payu-laboratory directory (this is so we keep all CABLE outputs produced by benchcab under the bench_example work directory). A custom laboratory path can be specified using the laboratory key in the payu config.yaml.

Also, the custom laboratory directory is in runs/payu-laboratory (i.e. outside the spatial directory). This is because we may want to have another configuration that runs with payu, e.g. online cable.

[SeanBryan51]

• how to run for several years for one met. forcing? Traditionally, met. forcing files are annual files which sets the length of the CABLE simulation. But we will want benchcab to run for several years for each task.

Running with payu allows us to use a different met forcing file for consecutive years/restarts (see https://github.com/CABLE-LSM/cable_example). However, we only have met forcing files up from 1901 to 1909 so the run will fail if we go beyond 1909. Should we add the ability to repeat the forcing every N years/restarts?

• how to deal with spin-up for the slow parts such as carbon stocks? Is it ok to provide a common starting point for all tasks?

I don't think I can answer this but it sounds reasonable to initialise CABLE with the same restart file. Would it beneficial to the user if they could use their own restart file?

It would also be great to have a test in benchcab that can verify the spinup is working correctly.

[SeanBryan51]

Some other design related questions:

1. Should we have some non-payu spatial tests?
   • There isn't much support for debugging models using the payu framework, removing a layer of abstraction will make things easier for the user to see how CABLE is being run.
   • We can still verify that CABLE is running correctly by running for a short time (e.g. 1 year). This would be enough to do bitwise comparisons between output files for example.
2. There is the issue of building with MPI using a custom build script. Should we add another build_script option specifically for building CABLE with MPI?
3. More of a comment, but since we are going with 1 PBS job per spatial task, we will probably have to use PBS job dependencies to run the bitwise comparison tasks on spatial outputs.

[ccarouge]

Note we have a separate payu-laboratory directory (this is so we keep all CABLE outputs produced by benchcab under the bench_example work directory). A custom laboratory path can be specified using the laboratory key in the payu config.yaml.

Also, the custom laboratory directory is

So why do we need to be able to set up a custom path for the laboratory? Do you have a use case in mind or it is just to give us the option if the use case appears afterwards?

In the last paragraph, I'm guessing you meant "standard laboratory directory" and not "custom". I am just mentioning it to make sure I understand correctly what you mean.

[SeanBryan51]

The main idea is to keep the "benchcab-payu" laboratory separate from the "standalone payu" laboratory. By default the laboratory is /scratch/${PROJECT}/${USER}/${MODEL}. A use case could be if we just want to reset the bench_example directory, we can rm -rf the runs directory without having to worry about any side effects that might have occurred outside of the bench_example directory.

[ccarouge]

2. There is the issue of building with MPI using a custom build script. Should we add another build_script option specifically for building CABLE with MPI?

Would the option be to get the command to run to build with MPI support?
An issue here is whether all historical scripts use the same way to choose an MPI compilation. Are all scripts using a command-line option or would some require hard-wired changes within the script? I don't know the answer. Should we only develop for the scripts we know of, which are using a command-line option (if I recall correctly)? In which case, we simply need an additional option to have the command that needs to be run.

[ccarouge]

The main idea is to keep the "benchcab-payu" laboratory separate from the "standalone payu" laboratory. By default the laboratory is /scratch/${PROJECT}/${USER}/${MODEL}. A use case could be if we just want to reset the bench_example directory, we can rm -rf the runs directory without having to worry about any side effects that might have occurred outside of the bench_example directory.

Sorry, it's my bad. I read "benchcab config.yaml" instead of "payu config.yaml", so I was wondering why we would want to let users specify their own path. I totally get the idea to set everything within the benchcab work directory. And that's much better that way.