Move regression test input data outside baseline directories

[DusanJovic-NOAA]

Description

Update regression test scripts to point to separate input data directory, which is now outside baseline directories.

Testing

Passed full regression test on Hera. Will run full tests on other Tier-1 platforms.

[BinLiu-NOAA]

Look good to me. And it's also a good idea to move the regression test input data outside the baseline dir. Thanks!

[climbfuji]

This is great, so much simpler. I hope this gets merged asap, in particular before any updates are made to the input data!

[DeniseWorthen]

This looks pretty straightforward. I would agree that getting it committed asap would be worthwhile.

[junwang-noaa]

Jiande started to run RT. @DeniseWorthen do you think we can combine this PR with your CICE6 commit after MOM6 commit?

[JessicaMeixner-NOAA]

mod_def files change more frequently than other input changes, I think it would be good to keep the mod_defs in the baseline directory while the other files live in the INPUTDATA_ROOT directory. This could be done as a second commit for follow up if desired.

[junwang-noaa]

May I ask what makes mod_def files changed so frequently when ww3 source code is updated? Is there a way to make ww3 grid def not tied up with source code updates?

[JessicaMeixner-NOAA]

The mod_def files pre-process grid and physics information. The fix file they process does not change and should remain in the INPUTDATA_ROOT directory. However, if a new feature is added, this will change the binary processed file. While not every update of WW3 updates the mod_defs, they will certainly be updated more than any other fixed input file, which is why I would recommend putting this in the baseline directory. A major rewrite/restructuring of WW3 would be required for these files to not need to occasionally be updated including optimization of the pre-processing that occurs in the generation of these mod_def files.

[DusanJovic-NOAA]

This PR separates all input files (now located in $INPUTDATA_ROOT) from all output (history) files which we save and call 'baselines'. When the rt.sh is executed with '-c' command line flag (create baseline mode) tests are executed and all output files that we need to save as 'a new baseline' are saved in $NEW_BASELINE which is then moved to develop-YYYYMMDD.

Moving purely input data ( mod_def.* ) from $INPUTDATA_ROOT to a baseline (output) directory will break this clear separation of input vs output, and so rt.sh will need to know that some input data are in $INPUTDATA_ROOT and some are in develop-YYYYMMDD ($RTPWD). It will also need some mechanism of syncing (some) input data from baseline directory over to the new baseline directory, which will be the special case just for these files.

[DusanJovic-NOAA]

All mod_def.* files are binary but I see they all start with header like:

^@^@^AÏWAVEWATCH III MODEL DEFINITION FILE2020-06-18

What's the meaning of this date 2020-06-18? Is this some sort of version number of the file? If it is will the 'new' version of the mod_def files have different (version) date? If they will, maybe we can store multiple 'versions' of the mod_def files in same input directory and have rt.sh use the one that specific version of WW3 submoudule needs.
Then when WW3 is updated and it needs new mod_def file we can just add them (as a new version) to the current $INPUTDATA_ROOT, without creating entire new input data directory. Adding new files should not require new input-data-YYYYMMDD.

[JessicaMeixner-NOAA]

@DusanJovic-NOAA the 2020-06-18 is coming from: https://github.com/NOAA-EMC/WW3/blob/develop/model/ftn/w3iogrmd.ftn#L70 which is the "version" number of that file. I like your idea of just having WW3 version numbers so that we can add more files without having more and more directories. That is also clean in terms of keeping the input files separate and all together. The only wrinkle with that, is then my ambition of some day adding an automatic generator of that input file when creating the baseline directory doesn't really work in the way I originally was thinking about, but I'm sure something could be worked out for that too.

@junwang-noaa the mod_def files contain a lot of information and sometimes you are just storing that a new feature is "false" and you are not using it but because it's now an option this file will change. Sometimes it changes every commit, many times it does not. You can see a description of all of the inputs that go into this file here: https://github.com/NOAA-EMC/WW3/blob/develop/model/inp/ww3_grid.inp the mod_def will also change if you use a different switch file but are still using the same ww3_grid.inp input file.

[BinLiu-NOAA]

@JessicaMeixner-NOAA @junwang-noaa @DusanJovic-NOAA, If I understand correctly, the INPUTDATA_ROOT dir and its content will also change (slowly, not as frequently as the baseline), right? For example, adding new input files for new regression test cases, replacing old/obsolete files, etc. If that's the case, the WW3 mod_def.* files (typically change less frequently) can still sit in the INPUTDATA_ROOT dir. Whenever, they need to be updated, they can be updated in the INPUTDATA_ROOT dir as well. I assume the INPUTDATA_ROOT dirs will also have different versions (saying INPUTDATA_yyyymmdd). Just a thought. May or may not be useful.

[DeniseWorthen]

Jiande started to run RT. @DeniseWorthen do you think we can combine this PR with your CICE6 commit after MOM6 commit?

Yes, that would work. Depending on whether cice or the bulk flux PR was going in first, I was going to add the compile fix for CICE and MOM6 when DATM=true also. Would that be ok, or is bundling the cice update, the rt input update and the compile fix too much?

[junwang-noaa]

Can you elaborate more on the binary processed file? So if a new feature is added, mod_def files need to be changed, it seems that the mod_def has some information to connect which features can be used for certain grids? I thought mod_def file are the grid definition file, but it seems it also contains some configurations related to ww3 model, is that correct?

…

On Wed, Dec 2, 2020 at 4:22 PM Jessica Meixner ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tests/fv3_conf/ccpp_control_run.IN <#312 (comment)> : > if [ $CPLWAV = .T. ]; then -cp @[RTPWD]/WW3_input_data/mod_def.* . -cp @***@***.***@***@***.***/ww3_multi.inp . -cp @***@***.***@***@***.***/rmp_src* . +cp @[INPUTDATA_ROOT]/WW3_input_data/mod_def.* . The mod_def files pre-process grid and physics information. The fix file they process does not change and should remain in the INPUTDATA_ROOT directory. However, if a new feature is added, this will change the binary processed file. While not every update of WW3 updates the mod_defs, they will certainly be updated more than any other fixed input file, which is why I would recommend putting this in the baseline directory. A major rewrite/restructuring of WW3 would be required for these files to not need to occasionally be updated including optimization of the pre-processing that occurs in the generation of these mod_def files. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#312 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TLTRHYG2SVSEU7KNFTSS2VZ3ANCNFSM4UK6LKHQ> .

[MinsukJi-NOAA]

This is great. Thank you @DusanJovic-NOAA.

[junwang-noaa]

@BinLiu_NOAA, that is correct. As you know, data in INPUTDATA_ROOT is quite large, and we don't expect it to change often, while the mod_def files are actually small and will change more often, that is causing issues. As Dusan mentioned, we have to treat those mod_def files differently. I feel this preprocessing files are tied with model configuration (with namelist variables defined in it), I am not sure if we can break it in a static part and a part related to model configuration. But that will require additional work.

…

On Wed, Dec 2, 2020 at 5:53 PM Bin Liu ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tests/fv3_conf/ccpp_control_run.IN <#312 (comment)> : > if [ $CPLWAV = .T. ]; then -cp @[RTPWD]/WW3_input_data/mod_def.* . -cp @***@***.***@***@***.***/ww3_multi.inp . -cp @***@***.***@***@***.***/rmp_src* . +cp @[INPUTDATA_ROOT]/WW3_input_data/mod_def.* . @JessicaMeixner-NOAA <https://github.com/JessicaMeixner-NOAA> @junwang-noaa <https://github.com/junwang-noaa> @DusanJovic-NOAA <https://github.com/DusanJovic-NOAA>, If I understand correctly, the INPUTDATA_ROOT dir and its content will also change (slowly, not as frequently as the baseline), right? For example, adding new input files for new regression test cases, replacing old/obsolete files, etc. If that's the case, the WW3 mod_def.* files (typically change less frequently) can still sit in the INPUTDATA_ROOT dir. Whenever, they need to be updated, they can be updated in the INPUTDATA_ROOT dir as well. I assume the INPUTDATA_ROOT dirs will also have different versions (saying INPUTDATA_yyyymmdd). Just a thought. May or may not be useful. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#312 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TLTPOIKVQZWRYAB26DSS3ANJANCNFSM4UK6LKHQ> .

[MinsukJi-NOAA]

@DusanJovic-NOAA, please make these changes to utest to address the failing ci test

diff --git a/tests/utest b/tests/utest
old mode 100755
new mode 100644
index fb0c410..c60ac25
--- a/tests/utest
+++ b/tests/utest
@@ -137,6 +137,7 @@ run_utests() {
       std_base)
         CREATE_BASELINE=true
         RTPWD=$baseline_location
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_std_base
         cat <<-EOF > ${RUNDIR_ROOT}/unit_test${RT_SUFFIX}.env
                                        export UNIT_TEST=true
@@ -147,6 +148,7 @@ run_utests() {
       std)
         CREATE_BASELINE=false
         RTPWD=$NEW_BASELINE
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_std_base
         cat <<-EOF > ${RUNDIR_ROOT}/unit_test${RT_SUFFIX}.env
                                        export UNIT_TEST=true
@@ -156,6 +158,7 @@ run_utests() {
       thr)
         CREATE_BASELINE=false
         RTPWD=$NEW_BASELINE
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_std_base
         THRD=2
         # INPES is sometimes odd, so use JNPES. Make sure JNPES is divisible by THRD
@@ -176,6 +179,7 @@ run_utests() {
       mpi)
         CREATE_BASELINE=false
         RTPWD=$NEW_BASELINE
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_std_base
         JNPES=$(( JNPES/2 ))
         WRITE_GROUP=2
@@ -195,6 +199,7 @@ run_utests() {
       dcp)
         CREATE_BASELINE=false
         RTPWD=$NEW_BASELINE
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_std_base
         temp=$INPES
         INPES=$JNPES
@@ -209,6 +214,7 @@ run_utests() {
       rst) # this is not going to work for regional model
         CREATE_BASELINE=false
         RTPWD=$NEW_BASELINE
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_std_base
         # Set up date and time of restart files for restart run
         RESTART_FILE_PREFIX="${SYEAR}${SMONTH}${SDAY}.$(printf "%02d" $(( SHOUR + FHMAX/2  )))0000"
@@ -272,6 +278,7 @@ run_utests() {
       bit_base)
         CREATE_BASELINE=true
         RTPWD=$baseline_location
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_bit_base
         comp_nm=bit
         cat <<-EOF >${RUNDIR_ROOT}/unit_test${RT_SUFFIX}.env
@@ -282,6 +289,7 @@ run_utests() {
       bit)
         CREATE_BASELINE=false
         RTPWD=$NEW_BASELINE
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_bit_base
         comp_nm=bit
         cat <<-EOF >${RUNDIR_ROOT}/unit_test${RT_SUFFIX}.env
@@ -292,6 +300,7 @@ run_utests() {
       dbg_base)
         CREATE_BASELINE=true
         RTPWD=$baseline_location
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_dbg_base
         comp_nm=dbg
         WLCLK=30
@@ -304,6 +313,7 @@ run_utests() {
       dbg)
         CREATE_BASELINE=false
         RTPWD=$NEW_BASELINE
+       INPUTDATA_ROOT=$RTPWD
         BL_SUFFIX=_dbg_base
         comp_nm=dbg
         WLCLK=30
@@ -318,6 +328,7 @@ run_utests() {
     cat <<- EOF > ${RUNDIR_ROOT}/run_test${RT_SUFFIX}.env
                        export MACHINE_ID=${MACHINE_ID}
                        export RTPWD=${RTPWD}
+                       export INPUTDATA_ROOT=${INPUTDATA_ROOT}
                        export PATHRT=${PATHRT}
                        export PATHTR=${PATHTR}
                        export NEW_BASELINE=${NEW_BASELINE}
@@ -716,6 +727,7 @@ else
   baseline_location=$DISKNM/NEMSfv3gfs/develop-20201118
 fi
 RTPWD=$baseline_location
+INPUTDATA_ROOT=$RTPWD
 
 rm -rf $NEW_BASELINE
 mkdir -p $NEW_BASELINE

[DeniseWorthen]

Code was committed with PR #270