Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor some build scripts #794

Merged
merged 6 commits into from
May 18, 2022
Merged

Refactor some build scripts #794

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ede3ebe
Refactor some build scripts
WalterKolczynski-NOAA May 17, 2022
198f501
Fix indentation in build usage messages
WalterKolczynski-NOAA May 17, 2022
b6243ee
Merge branch 'develop' into feature/build_app
WalterKolczynski-NOAA May 17, 2022
e661b57
Remove debug print and ensure bash
WalterKolczynski-NOAA May 17, 2022
1afd787
Restore missing indentation in partial_build
WalterKolczynski-NOAA May 17, 2022
a3ad198
Remove option to set CCPP suites from command line
WalterKolczynski-NOAA May 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Refactor some build scripts 
Refactors the build_ufs script to allow the build of any UFS app. Additionally,
the user may now supply a list of CCPP suites from the command line. The new
format of the command is now:

build_ufs.sh [-a UFS_app][-s CCPP_suites][-v]
  -a UFS_app:
    Specify the UFS application to build. The default if none is provided is
      S2SWA
  -s CCPP_suites:
     Build UFS with given comma-separated list of CCPP suites instead of default
      list
  -v:
    Turn on verbose mode (BUILD_VERBOSE)

build_all.sh is similarly refactored to include the same three options. Also, a
new '-c build_config' option is added to specify an alternative list of programs
to build.

build_all.sh [-a UFS_app][-c build_config][-h][-s CCPP_SUITES][-v]
  -a UFS_app:
    Build a specific UFS app instead of the default
  -c build_config:
    Selectively build based on the provided config instead of the default config
  -h:
    Print usage message and exit
  -s CCPP_suites:
    Build UFS with given comma-separated list of CCPP suites instead of default
      list
  -v:
    Run all scripts in verbose mode

partial_build.sh is also updated to take the new -c option to specify a build
config.

partial_build.sh [-c config_file][-h][-v]
  -c config_file:
    Selectively build based on the provided config. The default if none is
      specified is gfs_build.cfg
  -h:
    Print usage message and exit
  -v:
    Run in verbose mode

In addition to the above, build_all and partial_build had their indentation
redone and their usage/help statement updated/added. Also, the build config-
uration file was renamed to from fv3gfs_build.cfg to gfs_build.cfg.

Fixes #745
Fixes #751
  • Loading branch information
@WalterKolczynski-NOAA
WalterKolczynski-NOAA committed May 17, 2022
commit ede3ebe1e865cb61ab4c5ad052e27e555db56ff7
225 changes: 124 additions & 101 deletions sorc/build_all.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/bin/sh

set +x
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does set -eu work? If so, please add.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, missed this before I merged. But adding set -eu to all scripts is part of my long-term plans (see Issue #397).

#------------------------------------
# Exception handling is now included.
Expand All @@ -9,42 +10,65 @@ set +x
# Anything other than "true" will use libraries locally.
#------------------------------------

_build_ufs_options=""

while getopts "ac" option;do
case $option in
a)
echo "Received -a flag, building ufs-weather-model for ATMAERO app"
echo "skipping builds not needed for prototype runs"
_build_ufs_options=-a
break
;;
c)
echo "Received -c flag, building ufs-weather-model for S2SW app"
echo "skipping builds not needed for prototype runs"
_build_ufs_options=-c
break
;;
esac
done

export USE_PREINST_LIBS="true"

#------------------------------------
# END USER DEFINED STUFF
#------------------------------------
function _usage() {
cat <<-EOF
Builds all of the global-workflow components by calling the individual build
scripts in sequence.

Usage: $BASH_SOURCE [-a UFS_app][-c build_config][-h][-s CCPP_SUITES][-v]
-a UFS_app:
Build a specific UFS app instead of the default
-c build_config:
Selectively build based on the provided config instead of the default config
-h:
print this help message and exit
-s CCPP_SUITES:
Build UFS with given comma-separated list of CCPP suites instead of default list
-v:
Execute all build scripts with -v option to turn on verbose where supported

EOF
exit 1
}

_build_ufs_opt=""
_verbose_opt=""
# Reset option counter in case this script is sourced
OPTIND=1
while getopts ":a:c:s:hv" option; do
case "${option}" in
a) _build_ufs_opt+="-a ${OPTARG} ";;
c) _partial_opt+="-c ${OPTARG} ";;
h) _usage;;
s) _build_ufs_opt+="-s ${OPTARG} ";;
v) _verbose_opt="-v";;
\?)
echo "Unrecognized option: ${option}"
usage
;;
:)
echo "${option} requires an argument"
usage
;;
esac
done

build_dir=$(pwd)
logs_dir=$build_dir/logs
if [ ! -d $logs_dir ]; then
echo "Creating logs folder"
mkdir $logs_dir
echo "Creating logs folder"
mkdir $logs_dir
fi

# Check final exec folder exists
if [ ! -d "../exec" ]; then
echo "Creating ../exec folder"
mkdir ../exec
echo "Creating ../exec folder"
mkdir ../exec
fi

#------------------------------------
Expand All @@ -56,12 +80,12 @@ source ./machine-setup.sh > /dev/null 2>&1
#------------------------------------
# INCLUDE PARTIAL BUILD
#------------------------------------
. ./partial_build.sh $@
source ./partial_build.sh $_verbose_opt $_partial_opt

if [ $target = jet ]; then
Build_gldas=false
Build_gfs_util=false
Build_ww3_prepost=false
Build_gldas=false
Build_gfs_util=false
Build_ww3_prepost=false
fi

#------------------------------------
Expand All @@ -70,133 +94,132 @@ fi
ERRSCRIPT=${ERRSCRIPT:-'eval [[ $err = 0 ]]'}
err=0


#------------------------------------
# build WW3 pre & post execs
#------------------------------------
$Build_ww3_prepost && {
echo " .... Building WW3 pre and post execs .... "
./build_ww3prepost.sh > $logs_dir/build_ww3_prepost.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building WW3 pre/post processing."
echo "The log file is in $logs_dir/build_ww3_prepost.log"
fi
((err+=$rc))
echo " .... Building WW3 pre and post execs .... "
./build_ww3prepost.sh ${_verbose_opt} > $logs_dir/build_ww3_prepost.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building WW3 pre/post processing."
echo "The log file is in $logs_dir/build_ww3_prepost.log"
fi
((err+=$rc))
}

#------------------------------------
# build forecast model
#------------------------------------
$Build_ufs_model && {
echo " .... Building forecast model .... "
./build_ufs.sh ${_build_ufs_options} > $logs_dir/build_ufs.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building UFS model."
echo "The log file is in $logs_dir/build_ufs.log"
fi
((err+=$rc))
echo " .... Building forecast model .... "
./build_ufs.sh $_verbose_opt ${_build_ufs_opt} > $logs_dir/build_ufs.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building UFS model."
echo "The log file is in $logs_dir/build_ufs.log"
fi
((err+=$rc))
}

#------------------------------------
# build gsi
#------------------------------------
$Build_gsi && {
echo " .... Building gsi .... "
./build_gsi.sh > $logs_dir/build_gsi.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building gsi."
echo "The log file is in $logs_dir/build_gsi.log"
fi
((err+=$rc))
echo " .... Building gsi .... "
./build_gsi.sh $_verbose_opt > $logs_dir/build_gsi.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building gsi."
echo "The log file is in $logs_dir/build_gsi.log"
fi
((err+=$rc))
}

#------------------------------------
# build ncep_post
#------------------------------------
$Build_ncep_post && {
echo " .... Building ncep_post .... "
./build_ncep_post.sh > $logs_dir/build_ncep_post.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building ncep_post."
echo "The log file is in $logs_dir/build_ncep_post.log"
fi
((err+=$rc))
echo " .... Building ncep_post .... "
./build_ncep_post.sh $_verbose_opt > $logs_dir/build_ncep_post.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building ncep_post."
echo "The log file is in $logs_dir/build_ncep_post.log"
fi
((err+=$rc))
}

#------------------------------------
# build ufs_utils
#------------------------------------
$Build_ufs_utils && {
echo " .... Building ufs_utils .... "
./build_ufs_utils.sh > $logs_dir/build_ufs_utils.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building ufs_utils."
echo "The log file is in $logs_dir/build_ufs_utils.log"
fi
((err+=$rc))
echo " .... Building ufs_utils .... "
./build_ufs_utils.sh $_verbose_opt > $logs_dir/build_ufs_utils.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building ufs_utils."
echo "The log file is in $logs_dir/build_ufs_utils.log"
fi
((err+=$rc))
}

#------------------------------------
# build gldas
#------------------------------------
$Build_gldas && {
echo " .... Building gldas .... "
./build_gldas.sh > $logs_dir/build_gldas.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building gldas."
echo "The log file is in $logs_dir/build_gldas.log"
fi
((err+=$rc))
echo " .... Building gldas .... "
./build_gldas.sh $_verbose_opt > $logs_dir/build_gldas.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building gldas."
echo "The log file is in $logs_dir/build_gldas.log"
fi
((err+=$rc))
}

#------------------------------------
# build gfs_wafs - optional checkout
#------------------------------------
if [ -d gfs_wafs.fd ]; then
$Build_gfs_wafs && {
echo " .... Building gfs_wafs .... "
./build_gfs_wafs.sh > $logs_dir/build_gfs_wafs.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building gfs_wafs."
echo "The log file is in $logs_dir/build_gfs_wafs.log"
fi
((err+=$rc))
$Build_gfs_wafs && {
echo " .... Building gfs_wafs .... "
./build_gfs_wafs.sh $_verbose_opt > $logs_dir/build_gfs_wafs.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building gfs_wafs."
echo "The log file is in $logs_dir/build_gfs_wafs.log"
fi
((err+=$rc))
}
fi

#------------------------------------
# build workflow_utils
#------------------------------------
$Build_workflow_utils && {
echo " .... Building workflow_utils .... "
target=$target ./build_workflow_utils.sh > $logs_dir/build_workflow_utils.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building workflow_utils."
echo "The log file is in $logs_dir/build_workflow_utils.log"
fi
((err+=$rc))
echo " .... Building workflow_utils .... "
target=$target ./build_workflow_utils.sh $_verbose_opt > $logs_dir/build_workflow_utils.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building workflow_utils."
echo "The log file is in $logs_dir/build_workflow_utils.log"
fi
((err+=$rc))
}

#------------------------------------
# build gfs_util
#------------------------------------
$Build_gfs_util && {
echo " .... Building gfs_util .... "
./build_gfs_util.sh > $logs_dir/build_gfs_util.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building gfs_util."
echo "The log file is in $logs_dir/build_gfs_util.log"
fi
((err+=$rc))
echo " .... Building gfs_util .... "
./build_gfs_util.sh $_verbose_opt > $logs_dir/build_gfs_util.log 2>&1
rc=$?
if [[ $rc -ne 0 ]] ; then
echo "Fatal error in building gfs_util."
echo "The log file is in $logs_dir/build_gfs_util.log"
fi
((err+=$rc))
}

#------------------------------------
Expand Down
18 changes: 8 additions & 10 deletions sorc/build_ufs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,15 @@
#! /usr/bin/env bash
set -eux

# Build ATMW by default
APP="ATMW"
CCPP_SUITES="FV3_GFS_v16,FV3_GFS_v16_RRTMGP,FV3_GFS_v16_ugwpv1,FV3_GFS_v17_p8"
# Default settings
APP="S2SWA"
CCPP_SUITES="FV3_GFS_v16,FV3_GFS_v16_RRTMGP,FV3_GFS_v16_ugwpv1,FV3_GFS_v17_p8,FV3_GFS_v16_coupled_nsstNoahmpUGWPv1,FV3_GFS_v16_coupled_p7_rrtmgp,FV3_GFS_v17_coupled_p8"

while getopts "c" option; do
while getopts "a:s:v" option; do
case "${option}" in
c)
APP="S2SW"
CCPP_SUITES="FV3_GFS_v16_coupled_nsstNoahmpUGWPv1,FV3_GFS_v16_coupled_p7_rrtmgp,FV3_GFS_v17_coupled_p8"
;;
a) APP="${OPTARG}" ;;
s) CCPP_SUITES="${OPTARG}";;
v) BUILD_VERBOSE="YES";;
*)
echo "Unrecognized option: ${1}"
exit 1
Expand All @@ -30,7 +29,6 @@ esac

MOD_PATH=$cwd/ufs_model.fd/modulefiles


cd ufs_model.fd/
set +x
module purge
Expand All @@ -43,5 +41,5 @@ if [ -d build ]; then
rm -R build
fi
mkdir -p build && cd build
cmake -DAPP=${APP} -DCCPP_SUITES=${CCPP_SUITES} -DUFS_GOCART="ON" ..
cmake -DAPP=${APP} -DCCPP_SUITES=${CCPP_SUITES} ..
OMP_NUM_THREADS=1 make -j ${BUILD_JOBS:-8} VERBOSE=${BUILD_VERBOSE:-}
0 sorc/fv3gfs_build.cfg → sorc/gfs_build.cfg
View file
Open in desktop
File renamed without changes.
Loading