Continuous integration testing - specific to iHAMOCC

[jmaerz]

@TomasTorsvik and @JorgSchwinger , in response to todays NorESM meeting (and linked to #281 (comment)), I would suggest that we could think a bit about sharing a wrapper for continuous integration testing dedicated to iHAMOCC - so that we can run such common wrapper script whenever needed. It's probably not much of a script, but makes it easier to communicate about what was tested before a PR. This also reminds us on integrating such tests for potential new components/model features.

[JorgSchwinger]

I don't really understand this. What is the relation to the testing that Mariana is developing?

[TomasTorsvik]

BLOM regression test functionality was added in PR #262 :

This PR has changes to buildnml and to the testdefs directory to enable BLOM to have its first set of regression tests.
This PR resolves issue #258.

To see how the regression test is working - do the following:
(Note: the BLOM external points to my repository until this PR is merged)

> git clone https://github.com/mvertens/NorESM.git
> cd NorESM
> git checkout noresm_develop_v7_proto
> cd cime/scripts
> ./query_testlists --xml-category aux_blom_noresm

will give the following output

aux_blom_noresm: SMS_D_Ld1.T62_tn14.NOINYOC.betzy_intel
aux_blom_noresm: ERS_Ld5.T62_tn14.NOINYOC.betzy_intel
aux_blom_noresm: ERS_Ld5.T62_tn14.NOINY.betzy_intel
aux_blom_noresm: ERS_Ld5.T62_tn14_wtn14nw.NOINY_WW3.betzy_intel
aux_blom_noresm: ERS_Ld5.T62_tn14_wtn14nw.NOINY_WW3.betzy_intel.blom-wavice

To run a test suite and also generate baselines:

./create_test 
    --xml-category aux_blom_noresm  
    --xml-machine betzy --xml-compiler intel 
    --generate <your_baseline_directory_name>
    --baseline-root /cluster/shared/noresm/noresm_baselines 
    --project <your_project>

Baselines for the above aux_blom_noresm tests will be created in
/cluster/shared/noresm/noresm_baselines/<your_baseline_directory_name>

More documentation on regression testing in NorESM can be found at:
https://noresm-docs.readthedocs.io/en/noresm2/testing/systemtests.html

[TomasTorsvik]

@mvertens , @gold2718
maybe you could advice about this?

I think what we want is something like blom/cime_config/testdefs/testlist_hamocc.xml, or to add a category "aux_hamocc_noresm" to the existing testlist_blom.xml file. Not sure what is best.

We probably want NOINYOC compset with T62_tn21 grid (and NOIIAOC?, NOIIAJRAOC?, maybe optional for later) and some different settings for turning components on/off. I suppose we could include this by making blom/cime_config/testdefs/testmods_dirs/blom/hamocc-all, and adding a new user_nl_blom file.

[gold2718]

@mvertens , @gold2718 maybe you could advice about this?

I think what we want is something like blom/cime_config/testdefs/testlist_hamocc.xml, or to add a category "aux_hamocc_noresm" to the existing testlist_blom.xml file. Not sure what is best.

I would recommend adding new tests to the existing testlist_blom.xml file. Also, keep in mind that a single test entry can be on more than one test list (by adding a new machine entry to an existing test. Adding tests to a "aux_hamocc_noresm" sounds like a good idea for BLOM.

We probably want NOINYOC compset with T62_tn21 grid (and NOIIAOC?, NOIIAJRAOC?, maybe optional for later) and some different settings for turning components on/off. I suppose we could include this by making blom/cime_config/testdefs/testmods_dirs/blom/hamocc-all, and adding a new user_nl_blom file.

I am not qualified to comment about specific tests, however, I am happy to look over any proposed testlist changes.

[TomasTorsvik]

I started putting together a summary of test options currently available for BLOM/iHAMOCC on the BLOM.wiki

[jmaerz]

Hi @TomasTorsvik and @gold2718 , many thanks! - this is really helpful! - and can also lead to a shared testing (wrapping) script.

[mvertens]

@TomasTorsvik - thanks for putting the documentation together. I've edited the wiki page a bit to clarify things. You never want to add the --xml-* options if you are just creating a single test. I've also added HAMOCC specific tests via new files in
$SRCROOT/components/blom/cime_config/testdefs/testmods_dir/blom/hamocc.
And added new tests to testlist_blom.xml to reference those fiels. This is in #281.

[jmaerz]

Dear @mvertens , as briefly mentioned in our telecon: this is not urgent, but could you have a short look at: #294 (comment) - I seem to miss a point in setting up a baseline test and compare to it afterward. (I still need to adapt the script to cases, where a real testsuite is used and not only a single case - I am aware of this...)