UFS Development with ESMF

Download, Build, Run UFS Regression Tests

A good way to work on the UFS code is to use the "RT" regression test framework.

• Download UFS code
  • By default the head of the develop branch is checked out.
• Run the regression tests
  • It is not necessary to build the UFS executable before using the regression test script (rt.sh). The script will automatically build the required executable from the source code, with options as specified in the rt.conf file.
  • To see all the options supported by the regression test script, inside the test directory issue ./rt.sh without arguments.
  • In order to run only a specific regression test, e.g. cpld_bmark_p7, use ./rt.sh -n cpld_bmark_p7.
  • By default, the regression test script removes the RUNDIR's at the end of the execution. In order to hold on to the RUNDIR, e.g. for manual submissions of the job_card, supply the -k options: ./rt.sh -n cpld_bmark_p7 -k.
• Re-running a sub-set of RT cases without rebuilding the executable.
  • The following assumes that the cases to be re-run all use the same executable, i.e. are located under the same COMPILE line in the rt.conf.
  • The following procedure will setup new RUNDIRs for the cases, and re-run them, using a previously built exectuable.
  • Assuming the executable to be used is ./path/to/somewhere/ufs.exe use the following steps to re-run cases with this exectuable:
    • Under ./tests cp rt.conf rt.conf1. Edit rt.conf1 removing (or commenting out) all lines, except the RUN lines for the configurations wanting to test. In particular make sure the COMPILE lines are gone or commented out.
    • cp ./path/to/somewhere/ufs.exe fv3_0.exe and cp modules.fv3_001 modules.fv3_0
    • Now run via ./rt.sh -l rt.conf1 -k
• Manually building and re-building the UFS executable.
  • Set up the appropriate build environment. On Hera this is easily done via the following two commands, executed from the top level ufs-weather-model directory:
    • module use modulefiles
    • module load ufs_hera.intel
  • Determine the CMAKE_FLAGS needed to build the specific test executable by examining the ./test/rt.conf file. E.g. for the 'cpld_bmark_p7' test, the preceding COMPILE specification shows flags -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v16_coupled_nsstNoahmpUGWPv1.
  • Use the build.sh script to build. This is executed from the top level ufs-weather-model directory.
    • E.g. to build the executable for the 'cpld_bmark_p7' test, the build command is:
      env CMAKE_FLAGS='-DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v16_coupled_nsstNoahmpUGWPv1' ./build.sh
    • This creates a ./build subdirectory, where the actual build artifacts are kept. This includes the executable ufs_model.
  • Subsequent re-builds of the executable, e.g. after changing any of the UFS source files, can be done using the same command. It automatically only rebuilds the minimum required files.
  • Alternatively, a re-build can also be done by changing into the ./build directory (which was created by the ./build.sh execution), and calling make directly. For more verbose output, i.e. to see the actual compile and link commands and output, use make VERBOSE=1.
  • Another alternative is to use cmake --build . --verbose, which under the hood calls Unix Make anyway.
• Manually re-running a test case within its RUNDIR
  • The manually built executable is named ufs_model, and is located under the ./build sub-directory. In order to use it from the RUNDIR, either copy it, renaming it to the expected executable name (e.g. fv3.exe), or modify the job_card to reference the appropriate executable.
  • Then, from within the RUNDIR, submit the job via sbatch job_card.

Build against your own ESMF library

It is often necessary to build your own ESMF library and run the UFS against it. To do this, your ESMF build must be consistent with the hpc-stack currently used by the UFS. Use the following script to assist in building and linking against a custom build of ESMF:

• Testing UFS with new ESMF snapshots