Add POSTAMBLE_CMD into preamble postamble (NOAA-EMC#2235)

This PR adds the ability to run a command at the end of jobs via the
preamble's postamble function. A new command can be set via
`POSTAMBLE_CMD` and will be invoked at the end of jobs. Users can
add the command to the top of an env file to have every job run it or
it can be placed within a job if-block in the env file to run for just that
job.

Resolves NOAA-EMC#1145

Author: KateFriedman-NOAA

env/HERA.env

@@ -18,6 +18,8 @@ export npe_node_max=40
 export launcher="srun -l --export=ALL"
 export mpmd_opt="--multi-prog --output=mpmd.%j.%t.out"
 
+#export POSTAMBLE_CMD='report-mem'
+
 # Configure MPI environment
 #export I_MPI_ADJUST_ALLREDUCE=5
 #export MPI_BUFS_PER_PROC=2048

ush/preamble.sh

@@ -16,6 +16,8 @@
 #   TRACE (YES/NO): Whether to echo every command (set -x) [default: "YES"]
 #   STRICT (YES/NO): Whether to exit immediately on error or undefined variable
 #     (set -eu) [default: "YES"]
+#   POSTAMBLE_CMD (empty/set): A command to run at the end of the job
+#     [default: empty]
 #
 #######
 set +x
@@ -70,6 +72,24 @@ postamble() {
     start_time="${2}"
     rc="${3}"
 
+    # Execute postamble command
+    #
+    # Commands can be added to the postamble by appending them to $POSTAMBLE_CMD:
+    #    POSTAMBLE_CMD="new_thing; ${POSTAMBLE_CMD:-}" # (before existing commands)
+    #    POSTAMBLE_CMD="${POSTAMBLE_CMD:-}; new_thing" # (after existing commands)
+    #
+    # Always use this form so previous POSTAMBLE_CMD are not overwritten. This should
+    #   only be used for commands that execute conditionally (i.e. on certain machines
+    #   or jobs). Global changes should just be added to this function.
+    # These commands will be called when EACH SCRIPT terminates, so be mindful. Please
+    #   consult with global-workflow CMs about permanent changes to $POSTAMBLE_CMD or
+    #   this postamble function.
+    #
+
+    if [[ -v 'POSTAMBLE_CMD' ]]; then
+      ${POSTAMBLE_CMD}
+    fi
+
     # Calculate the elapsed time
     end_time=$(date +%s)
     end_time_human=$(date -d@"${end_time}" -u +%H:%M:%S)