Accumulated changes from develop

Moving accumulated changes from branch develop onto master.
*) Bug fixes: rescaling LW calculation to treat scattering
*) Sensitivity (Jacobian) of LW fluxes to surface temperature
*) NRLSSI2 model of solar variability including the ability to specify total solar irradiance (earth-system-radiation#43)
*) Parameterization for optimal single angle at which to do LW no-scattering calculations

Argument lists for ty_gas_optics_rrtmgp%load() are changed for both LW and SW versions.

The optimal angle works but has no effect; updated data to come.

Testing and continuous integration has also been expanded, including a new set of regression tests in tests/ 

Co-authored-by: Valentin Clement (バレンタイン クレメン) <clementval@gmail.com>
Co-authored-by: Benjamin R. Hillman <bhillma@sandia.gov>
Co-authored-by: Dmitry Alexeev <53860485+alexeedm@users.noreply.github.com>
Co-authored-by: inpolonsky <51209434+inpolonsky@users.noreply.github.com>
Co-authored-by: jdelamere <jdelamere@users.noreply.github.com>

Author: 6 people

.gitignore

@@ -14,4 +14,11 @@
 Makefile.libs
 Makefile.conf
 
-src/
+# Intel code coverage
+*.dyn
+*.dpi
+*.spi
+*.spl
+*.HTML
+*.html
+*.gif

.travis.yml

@@ -7,7 +7,8 @@ language: cpp
 #  - RTE_KERNELS="openacc"
 matrix:
   include:
-  - compiler: gcc # Using
+  - name: "PGI community edition "
+    compiler: gcc
     env:
     - FC=pgfortran
     - FCFLAGS="-Mallocatable=03 -Mstandard -Mbounds -Mchkptr -Kieee -Mchkstk"
@@ -20,10 +21,11 @@ matrix:
           - gcc-8
           - gfortran-8
           - libnetcdf-dev
-  - compiler: gcc
+  - name: "gcc-9"
+    compiler: gcc
     env:
     - FC=gfortran-9
-    - FCFLAGS="-ffree-line-length-none -m64 -std=f2003 -march=native -fbounds-check -finit-real=nan -DUSE_CBOOL"
+    - FCFLAGS="-ffree-line-length-none -m64 -std=f2008 -march=native -fbounds-check -finit-real=nan -DUSE_CBOOL"
     - TEST_PGI=no
     - NCHOME=/usr/
     addons:
@@ -34,10 +36,11 @@ matrix:
           - gcc-9
           - gfortran-9
           - libnetcdf-dev
-  - compiler: gcc
+  - name: "gcc-8"
+    compiler: gcc
     env:
     - FC=gfortran-8
-    - FCFLAGS="-ffree-line-length-none -m64 -std=f2003 -march=native -fbounds-check -finit-real=nan -DUSE_CBOOL"
+    - FCFLAGS="-ffree-line-length-none -m64 -std=f2008 -march=native -fbounds-check -finit-real=nan -DUSE_CBOOL"
     - TEST_PGI=no
     - NCHOME=/usr/
     addons:
@@ -64,6 +67,7 @@ before_install:
   fi
 
 # Install netcdf-fortran for PGI
+# git clone https://github.com/Unidata/netcdf-fortran.git
 - git clone https://github.com/Unidata/netcdf-fortran.git --branch v4.4.4
 - cd netcdf-fortran
 - FC=$FC ./configure --prefix=${HOME}/netcdff
@@ -82,57 +86,91 @@ install:
 # conda create --yes -n test python=$TRAVIS_PYTHON_VERSION
 - conda create --yes -n test python=3.7
 - source activate test
-- conda install --yes urllib3 netcdf4 xarray dask
+- conda install --yes urllib3 netcdf4 xarray dask scipy
 
 before_script:
 - export RRTMGP_ROOT=$PWD
-- export RRTMGP_DIR=$PWD/build
+- export RRTMGP_BUILD=$PWD/build
 - export NFHOME=${HOME}/netcdff
 - export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$NFHOME/lib
 - cd examples/rfmip-clear-sky
 - python3 ./stage_files.py
 - cd ../..
 script:
-# Build library - default kernels
-- cd build || exit 1
-- make || exit 1
-- cd .. || exit 1
+#
+# Default kernels
+#
+- cd ${RRTMGP_ROOT}
+#
+# Build library
+#
+- make -C build clean || exit 1
+- make -C build       || exit 1
+#
 # Build and run RFMIP examples
-- cd examples/rfmip-clear-sky || exit 1
+#
+- cd ${RRTMGP_ROOT}/examples/rfmip-clear-sky || exit 1
 - make clean
 - make || exit 1
 - python3 ./run-rfmip-examples.py
-- python3 ./compare-to-reference.py --fail=1.e-4
-- cd ../..
+- python3 ./compare-to-reference.py --fail=7.e-4
+- make clean
+#
 # Build and run all-sky example
-- cd examples/all-sky || exit 1
+#
+- cd ${RRTMGP_ROOT}/examples/all-sky || exit 1
 - make clean
 - make || exit 1
 - python3 ./run-allsky-example.py
 - python3 ./compare-to-reference.py
-- cd ../..
+- make clean
 #
-# Build library - Open-acc kernels
+# Build and regression tests
 #
-- export RTE_KERNELS="openacc"
-- cd build || exit 1
+- cd ${RRTMGP_ROOT}/tests|| exit 1
 - make clean
 - make || exit 1
-- cd .. || exit 1
+- cp ${RRTMGP_ROOT}/examples/rfmip-clear-sky/multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-1-2_none.nc test_atmospheres.nc
+- ./clear_sky_regression test_atmospheres.nc ${RRTMGP_ROOT}/rrtmgp/data/rrtmgp-data-lw-g256-2018-12-04.nc
+- ./clear_sky_regression test_atmospheres.nc ${RRTMGP_ROOT}/rrtmgp/data/rrtmgp-data-sw-g224-2018-12-04.nc
+- python3 verification.py
+#
+# Open-acc kernels
+#
+- cd ${RRTMGP_ROOT}
+#
+# Build library
+#
+- make -C build clean || exit 1
+- make -C build       || exit 1
+#
 # Build and run RFMIP examples
-- cd examples/rfmip-clear-sky || exit 1
+#
+- cd ${RRTMGP_ROOT}/examples/rfmip-clear-sky || exit 1
 - make clean
 - make || exit 1
 - python3 ./run-rfmip-examples.py
-- python3 ./compare-to-reference.py --fail=1.e-4
-- cd ../..
+- python3 ./compare-to-reference.py --fail=7.e-4
+- make clean
+#
 # Build and run all-sky example
-- cd examples/all-sky || exit 1
+#
+- cd ${RRTMGP_ROOT}/examples/all-sky || exit 1
 - make clean
 - make || exit 1
 - python3 ./run-allsky-example.py
 - python3 ./compare-to-reference.py
-- cd ../..
+- make clean
+#
+# Build and regression tests
+#
+- cd ${RRTMGP_ROOT}/tests|| exit 1
+- make clean
+- make || exit 1
+- cp ${RRTMGP_ROOT}/examples/rfmip-clear-sky/multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-1-2_none.nc test_atmospheres.nc
+- ./clear_sky_regression test_atmospheres.nc ${RRTMGP_ROOT}/rrtmgp/data/rrtmgp-data-lw-g256-2018-12-04.nc
+- ./clear_sky_regression test_atmospheres.nc ${RRTMGP_ROOT}/rrtmgp/data/rrtmgp-data-sw-g224-2018-12-04.nc
+- python3 verification.py
 notifications:
   email:
     on_success: never

Contributing.md

@@ -0,0 +1,19 @@
+### Contributing to RTE+RRTMGP
+
+Thanks for considering making a contribution to RTE+RRTMGP.
+
+The code in this repository is intended to work with compilers supporting the Fortran 2008 standard. It is also expected to run end-to-end on GPUs when compiled with OpenACC. Commits are tested with [Travis](https://travis-ci.com) against gfortran versions 8 and 9 and against various versions > 19.9 of the PGI compiler using resources provided by the [Swiss Supercomputing Center](https://cscs.ch). The testing uses two general codes in `examples/`for which results are compared against existing implemetations,  and custom codes in tests/ intended to excercise all code options.
+
+##### Did you find a bug? 
+
+Please file an issue on the [Github page](https://github.com/RobertPincus/rte-rrtmgp/issues). 
+
+##### Did you write a patch that fixes a bug?
+
+Please fork this repository, branch from `develop`, make your changes, and open a Github [pull request](https://github.com/RobertPincus/rte-rrtmgp/pulls) against branch `develop`. 
+
+##### Did you add functionality? 
+
+Please fork this repository, branch from `develop`, make your changes, and open a Github [pull request](https://github.com/RobertPincus/rte-rrtmgp/pulls) against branch `develop`,  adding a new regression test or comparison against the reference in `tests/verification.py` or `tests/validation-plots.py` as appropriate.  
+
+RTE+RRTMGP is intended to be a core that users can extend with custom code to suit their own needs. 

README.md

@@ -6,9 +6,16 @@ RRTMGP uses a k-distribution to provide an optical description (absorption and p
 
 RTE computes fluxes given spectrally-resolved optical descriptions and source functions. The fluxes are normally summarized or reduced via a user extensible class.
 
-Example programs and documenation are evolving - please see examples/ in the repo and Wiki on the project's Github page. Suggestions are welcome. Meanwhile for questions please contact Robert Pincus and Eli Mlawer at rrtmgp@aer.com.
+Example programs and documentation are evolving - please see examples/ in the repo and Wiki on the project's Github page. Suggestions are welcome. Meanwhile for questions please contact Robert Pincus and Eli Mlawer at rrtmgp@aer.com.
+
+## Recent changes
+
+1. The default method for solution for longwave problems that include scattering has been changed from 2-stream methods to a re-scaled and refined no-scattering calculation following [Tang et al. 2018](https://doi.org/10.1175/JAS-D-18-0014.1).
+2. In RRTMGP gas optics, the spectrally-resolved solar source function in can be adjusted by specifying the total solar irradiance (`gas_optics%set_tsi(tsi)`) and/or the facular and sunspot indicies (`gas_optics%set_solar_variability(mg_index, sb_index, tsi)`)from the [NRLSSI2 model of solar variability](http://doi.org/10.1175/BAMS-D-14-00265.1).  
+3. `rte_lw()` now includes optional arguments for computing the Jacobian (derivative) of broadband flux with respect to changes in surface temperature. In calculations neglecting scattering only the Jacobian of upwelling flux is computed. When using re-scaling to account for scattering the Jacobians of both up- and downwelling flux are computed.
+
+Relative to commit `69d36c9` to `master` on Apr 20, 2020, the required arguments to both the longwave and shortwave versions of `ty_gas_optics_rrtmgp%load()`have changed.
 
-In the most recent revision, the default method for solution for longwave problems that include scattering has been changed from 2-stream methods to a re-scaled and refined no-scattering calculation following [Tang et al. 2018](https://doi.org/10.1175/JAS-D-18-0014.1).
 
 ## Building the libraries.
 

azure-pipelines.yml

@@ -4,23 +4,35 @@ jobs:
   strategy:
     matrix:
       pgi_19_9_gpu:
-        compiler_module: PGI/19.9.0
+        compiler_module: PGI/19.9
+        PrgEnv: PrgEnv-pgi
         accel_module: cudatoolkit
-        FCFLAGS: "-O3 -ta=tesla:cc60,cuda10.1 -Minfo -Mallocatable=03 -gopt -Minline,reshape,maxsize:40"
+        FCFLAGS: "-O3 -ta=tesla:cc60,cuda10.1 -Mallocatable=03 -gopt -Minline,reshape,maxsize:40"
         RTE_KERNELS: openacc
       pgi_default_gpu:
         compiler_module: pgi
+        PrgEnv: PrgEnv-pgi
         accel_module: craype-accel-nvidia60
         # Generic accelerator flag
-        FCFLAGS: "-O3 -acc                   -Minfo -Mallocatable=03 -gopt"
+        FCFLAGS: "-O3 -acc                   -Mallocatable=03 -gopt"
         RTE_KERNELS: openacc
+      pgi_19_10_cpu:
+        compiler_module: PGI/19.10
+        PrgEnv: PrgEnv-pgi
+        accel_module:
+        # Error checking flags
+        FCFLAGS: "-Mallocatable=03 -Mstandard -Mbounds -Mchkptr -Kieee -Mchkstk"
+        RUN_CMD:
       pgi_19_9_cpu:
-        compiler_module: PGI/19.9.0
+        compiler_module: PGI/19.9
+        PrgEnv: PrgEnv-pgi
         accel_module:
         # Error checking flags
         FCFLAGS: "-Mallocatable=03 -Mstandard -Mbounds -Mchkptr -Kieee -Mchkstk"
+        RUN_CMD:
       pgi_default_cpu:
         compiler_module: pgi
+        PrgEnv: PrgEnv-pgi
         accel_module:
         # Error checking flags
         FCFLAGS: "-Mallocatable=03 -Mstandard -Mbounds -Mchkptr -Kieee -Mchkstk"
@@ -32,21 +44,21 @@ jobs:
   steps:
   - script: |
       set -e
-      
+
       echo "
       export PATH=$CRAY_BINUTILS_BIN:$PATH
       module load daint-gpu
-      module swap PrgEnv-cray PrgEnv-pgi
+      module swap PrgEnv-cray $(PrgEnv)
       module load cray-netcdf cray-hdf5
       module swap pgi $(compiler_module)
       module load $(accel_module)
       module load cray-python/3.6.5.7
       export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
-      
+
       echo Environment:
       module list
       " > modules
-      
+
     displayName: 'Create module environment'
   - script: |
       set -e
@@ -60,29 +72,41 @@ jobs:
       set -e
       source modules
       export RRTMGP_ROOT=$PWD
-      export RRTMGP_DIR=$PWD/build
+      export RRTMGP_BUILD=$PWD/build
       export FC=ftn
+      make -C build/ clean
+
       make -C build/ -j 8
+      make -C tests clean
+      make -C tests -j 1
+      make -C examples/all-sky clean
       make -C examples/all-sky -j 8
+      make -C examples/rfmip-clear-sky clean
       make -C examples/rfmip-clear-sky -j 8
     displayName: 'Make'
   - script: |
       set -e
       source modules
-      cd examples/rfmip-clear-sky
+      export RRTMGP_ROOT=$PWD
+      cd ${RRTMGP_ROOT}/examples/rfmip-clear-sky
       srun -C gpu -A c15 -p cscsci python ./run-rfmip-examples.py --block_size 1800
-      cd ../..
-      cd examples/all-sky
+      cd  ${RRTMGP_ROOT}/examples/all-sky
       srun -C gpu -A c15 -p cscsci python ./run-allsky-example.py
+      cd  ${RRTMGP_ROOT}/tests
+      cp ${RRTMGP_ROOT}/examples/rfmip-clear-sky/multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-1-2_none.nc test_atmospheres.nc
+      srun -C gpu -A c15 -p cscsci ./clear_sky_regression test_atmospheres.nc ${RRTMGP_ROOT}/rrtmgp/data/rrtmgp-data-lw-g256-2018-12-04.nc
+      srun -C gpu -A c15 -p cscsci ./clear_sky_regression test_atmospheres.nc ${RRTMGP_ROOT}/rrtmgp/data/rrtmgp-data-sw-g224-2018-12-04.nc
     displayName: 'Run'
   - script: |
       set -e
       source modules
+      export RRTMGP_ROOT=$PWD
       # This module will unload some of the build modules, so do the checks separately
       module load netcdf-python
-      cd examples/rfmip-clear-sky
-      python ./compare-to-reference.py --fail=1.e-4
-      cd ../..
-      cd examples/all-sky
+      cd ${RRTMGP_ROOT}/examples/rfmip-clear-sky
+      python ./compare-to-reference.py --fail=7.e-4
+      cd ${RRTMGP_ROOT}/examples/all-sky
       python ./compare-to-reference.py
+      cd ${RRTMGP_ROOT}/tests
+      python verification.py
     displayName: 'Check results'

build/Makefile.conf.ifort

@@ -8,11 +8,11 @@ export FC = ifort
 #
 # Optimized
 #
-export FCFLAGS  += -m64 -O3 -g -traceback -assume realloc_lhs -extend-source 132
+export FCFLAGS  += -m64 -O3 -g -traceback -heap-arrays -assume realloc_lhs -extend-source 132
 export F77FLAGS += -m64 -O3 -g -traceback
 # can add  -qopt-report-phase=vec
 #
 # Debugging
 #
-# export FCFLAGS  = -m64 -g  -traceback -assume realloc_lhs -extend-source 132 -check bounds,uninit,pointers,stack -stand f03
+# export FCFLAGS  = -m64 -g  -traceback -heap-arrays -assume realloc_lhs -extend-source 132 -check bounds,uninit,pointers,stack -stand f08
 # export F77FLAGS = -m64 -g  -traceback                                        -check bounds,uninit,pointers,stack

examples/all-sky/compare-to-reference.py

@@ -31,7 +31,7 @@
     # If a version of the file exists in the reference directory, no need to download (can be over-ridden)
     if (args.download_reference or not os.path.exists(ref_file)):
         os.makedirs(args.ref_dir, exist_ok=True)
-        urllib.request.urlretrieve("https://owncloud.gwdg.de/index.php/s/wgCL7imbA0QRCEf/download", ref_file)
+        urllib.request.urlretrieve("https://owncloud.gwdg.de/index.php/s/OjbNzRTlXUk0G5w/download", ref_file)
     tst = xr.open_dataset(tst_file)
     ref = xr.open_dataset(ref_file)
 
@@ -42,8 +42,10 @@
         if np.any(np.isnan(tst.variables[v].values)):
             raise Exception("Some test values are missing. Now that is strange.")
 
-        diff = abs((tst-ref).variables[v].values)
-        avg  = 0.5*(tst+ref).variables[v].values
+        # express as (tst-ref).variables[v].values when replacing reference file
+        #   to have same number of columns
+        diff = abs((tst.variables[v]-ref.variables[v]).values)
+        avg  = 0.5*(tst.variables[v]+ref.variables[v]).values
         # Division raises a runtime warning when we divide by zero even if the
         #   values in those locations will be ignored.
         with np.errstate(divide='ignore', invalid='ignore'):

examples/all-sky/garand-atmos-1.nc

@@ -2,11 +2,14 @@
 #
 # This script runs runs the RTE+RRTMGP all-sky examples
 #
-import os, subprocess, glob
+import os
+import shutil
+import subprocess
+import glob
 import argparse
 
 rte_rrtmgp_dir   = os.path.join("..", "..")
-# This files lives in $RRTMGP_DIR/examples/all-sky/
+# This files lives in $RRTMGP_ROOT/examples/all-sky/
 all_sky_dir      = "."
 # Code should be run in the all_sky_dir directory
 
@@ -18,8 +21,10 @@
 
 # In the local directory
 all_sky_exe_name = os.path.join(all_sky_dir, "rrtmgp_allsky")
+input_file       = os.path.join(all_sky_dir, "garand-atmos-1.nc")
 atmos_file       = os.path.join(all_sky_dir, "rrtmgp-allsky.nc")
 
+
 if __name__ == '__main__':
     parser = argparse.ArgumentParser(description="Runs all-sky examples, resetting output.")
     parser.add_argument("--run_command", type=str, default="",
@@ -38,7 +43,7 @@
 
     os.chdir(all_sky_dir)
     # Remove cloudy-sky fluxes from the file containing the atmospheric profiles
-    subprocess.run(["git", "checkout", atmos_file])
+    shutil.copyfile(input_file, atmos_file)
     subprocess.run([all_sky_exe_name, atmos_file, lw_gas_coeffs_file, lw_clouds_coeff_file, ncol_str, nloops_str])
     subprocess.run([all_sky_exe_name, atmos_file, sw_gas_coeffs_file, sw_clouds_coeff_file, ncol_str, nloops_str])