Compiler instructions (aka the include.mk files)

Compiler instruction file

These files are located in directory ED2/ED/build/make.
Once you've cloned the mainline and set up the compilers, HDF5 libraries, and optionally the MPI libraries, navigate to ED2/ED/build/make. There are a few templates of files based on the operational system and family of compilers:

• include.mk.gnu. Linux using gfortran and gcc
• include.mk.intel. Linux using ifx and icx
• include.mk.macos. MacOS using one version of gfortran and gcc

Make a copy of the file that most closely fits your system specifications to a unique to a unique file (e.g. include.mk.mylab). Below is a list of variables that must be set. You can add other variables for convenience (for example, create variables that define paths to HDF5 and other libraries).

Name	Description
MAKE	The location of the make command. In Linux systems, this is typically /usr/bin/make or /usr/bin/gmake, whereas for GNU Make installed with HomeBrew, set this to MAKE=$(HOMEBREW_PREFIX)/bin/gmake```. Consider changing this only if you get messages on make` command not being found.
BASE	The base path where ED2 compiling instructions are located. The default value $(ED_ROOT)/build should work for virtually all cases.
HDF5_PATH	The base path where the HDF5 installation is located. Make sure that the path points to an HDF5 version that is fully compatible with the intended compiler. If your libhdf5_fortran.so or libhdf5_fortran.dylib file is at /path/to/hdf5/lib, set HDF5_PATH=/path/to/hdf5.
HDF5_INCS	If HDF5_PATH is correctly set, then this should be HDF5_INCS=$(HDF5_PATH)/include. If the include files are located in a standard directory (e.g., /usr/include), it is fine to leave this variable empty (i.e., HDF5_INCS=).
HDF5_LIBS	The path where to find HDF5 libraries and dependencies. A few examples: All libraries (zlib and HDF5) are located in standard directories (i.e., /usr/lib). This variable should be set as: HDF5_LIBS= -lz -lm -lhdf5 -lhdf5_fortran -lhdf5_hl HDF5 libraries are located in a non-standard directories (set in HDF5_HOME), but zlib is installed in a standard directory: HDF5_LIBS= -lz -lm -L$(HDF5_HOME)/lib -lhdf5 -lhdf5_fortran -lhdf5_hl Both zlib and HDF5 libraries are located in a non-standard directories. For example zlib is installed in /path/to/zlib/1.2.11_intel and HDF5 is installed in the path defined in HDF5_HOME: ZLIB_HOME=/path/to/zlib/1.2.11_intel HDF5_LIBS= -lm -L$(ZLIB_HOME)/lib -lz -L$(HDF5_HOME)/lib -lhdf5 -lhdf5_fortran -lhdf5_hl
LAPACK_PATH	If you will be compiling ED2 with ifx and your intel compiler has the Math Kernel Libraries, leave this empty (LAPACK_PATH=). Otherwise, provide the path where the Linear Algebra Package (lapack) is installed. If your liblapack.a or liblapack.dylib file is at /path/to/lapack/lib, set LAPACK_HOME=/path/to/labapack.
LAPACK_INCS	If LAPACK_PATH is correctly set, then this should be LAPACK_INCS=$(LAPACK_HOME)/include. If the include files are located in a standard directory (e.g., /usr/include, it is fine to leave this variable empty (i.e., LAPACK_INCS=).
LAPACK_LIBS	The path where to find lapack libraries and dependencies. A few examples: Lapack libraries are located in standard directories (i.e., /usr/lib). this variable should be set as: LAPACK_LIBS= -llapack -lblas Lapack libraries are located in a non-standard directories (set in LAPACK_HOME): LAPACK_LIBS= -L$(LAPACK_HOME)/lib -llapack -lblas
USE_COLLECTIVE_MPIO	If you have a version of HDF5 compiled in parallel, and you are planning to use parallel runs (i.e., a gridded, multi-polygon simulation, or a coupled ED2-BRAMS simulation), then you may benefit from collective I/O. In this case, you may set this variable to 1. For single-site runs, or in case your HDF5 file is not compiled with parallel, you can set this variable to 0.
CMACH	This informs the compiler of which system you are using. Possible options include: CMACH=LINUX. Option for Linux systems. CMACH=MACOS. In case you are using MacOS Note. These are currently the only systems supported. If you have a different system (e.g., Windows or some other type of Unix system), we accept contributions. You will likely need to edit rsys.F90, utils_c.c, and utils_sub_names.h and provide new, system-specific, settings.
FC_TYPE	The type of Fortran/C compilers being used. Options are: GNU, if using gfortran and gcc compilers INTEL, if using ifx and icc compilers (or ifort and icc in older systems) PGI, if using pgfortran and pgcc compilers Note. In case you are using mpi compilers (mpif90 and mpicc), you must set FC_TYPE consistently to the compiler type used to build mpif90 and mpicc. In case you do not know, you can search for the location of mpif90 in your system (for example, in Linux or Mac you can use command which mpif90), then open the file in any text editor (mpif90 is a script) and look for variable F90BASE.
F_COMP	The Fortran compiler. Typical options are: gfortran, if using GNU Fortran compilers without MPI capabilities. ifx, if using Intel Fortran compilers without MPI capabilities (or ifort in older systems). pgfortran, if using PGI Fortran compilers without MPI capabilities. mpif90, if using MPI capabilities, regardless of the Fortran compiler.
C_COMP	The C compiler. Typical options are: gcc, if using GNU C compilers without MPI capabilities. icx, if using Intel C compilers without MPI capabilities (or icc in older systems). pgcc, if using PGI C compilers without MPI capabilities. mpicc, if using MPI capabilities, regardless of the C compiler. Note.Do not mix different families of Fortran and C compilers. For example, if F_COMP=ifx, then ensure that C_COMP=icx, otherwise the code may have unexpected behaviour.
LOADER	The Fortran compiler used to generate the executable. Unless otherwise instructed by system administrators, use the default option (LOADER=$(F_COMP)).
MOD_EXT	The extension of module files compiled with Fortran. Unless otherwise instructed by system administrators, use the default option (MOD_EXT=mod).
KIND_COMP	This sets compiler flags in case your include.mk file is derived from include.mk.gnu or include.mk.intel. Intel-based options. A. Full debugging options. This is strongly recommended whenever you change the source code. This option will check for uninitialised variables, matrix bound checks, mismatches in argument types between subroutine/function declaration and calls, floating point exceptions, and unused variables and functions. You will need to compile the code two times. If using the script install.sh, use option -s 1 for the first compilation, -s 2 for preparing the second compilation, and -s 3 for the second compilation. E. Use this option for research simulations. This option will skip all the checks and use the highest possible optimisation level for maximum simulation speed. GNU-based options. A. Full debugging options. This is strongly recommended whenever you change the source code. This option will check for uninitialised variables, matrix bound checks, floating point exceptions, and unused variables and functions. E. Use this option for research simulations. This option will skip all the checks and use the highest possible optimisation level for maximum simulation speed. Note. Options B, C, and D are currently deprecated. They were formerly used to provide intermediate levels of debugging.
F_OPTS	The list of compilation options for Fortran. Check the sections on Fortran compilation flags for ifx and gfortran for suggestions. Note>/br>If using include.mk based on include.mk.gnu, include.mk.intel or include.mk.macos, we recommend keeping the default options and set the type of compilation through KIND_COMP.
C_OPTS	The list of compilation options for C. Note>/br>If using include.mk based on include.mk.gnu, include.mk.intel or include.mk.macos, we recommend keeping the default options and set the type of compilation through KIND_COMP.
LOADER_OPTS	The list of compilation options for the step that generates the executable file. This is almost always the same options as F_OPTS, except that LOADER_OPTS rarely has the -static option. Note>/br>If using include.mk based on include.mk.gnu, include.mk.intel or include.mk.macos, we recommend keeping the default options and set the type of compilation through KIND_COMP.
F_LOWO_OPTS	Optional variable, that allows compiling a few Fortran subroutines with slightly lower optimisation level. This is useful when compiling the code with Intel compiles (ifx or ifort). Note>/br>If using include.mk based on include.mk.gnu, include.mk.intel or include.mk.macos, we recommend keeping the default options and set the type of compilation through KIND_COMP.
MPI_PATH	This sets the path where to find the OpenMPI or MPICH installation. Leave this blank (i.e., MPI_PATH=) unless for some reason your system does not have mpif90 and mpicc and you were advised to set this variable by your system administrator.
PAR_INCS	This sets the path where to find the include files for MPI. If MPI_PATH is blank, leave this blank too (i.e., PAR_INCS=). Otherwise, set it to PAR_INCS=$(MPI_PATH)/include.
PAR_LIBS	This sets the path where to find the library files for MPI. If MPI_PATH is blank, leave this blank too (i.e., PAR_LIBS=). Otherwise, set it to PAR_LIBS=-L$(MPI_PATH)/lib -lmpi or PAR_LIBS=-L$(MPI_PATH)/lib -lmpich -lpmpich depending on the MPI distribution.
ARCHIVE	Command to create the ED2 library. Unless otherwise instructed by system administrators, this variable should be set to ARCHIVE=ar rs.

Useful Fortran compilation flags for Intel (ifx).

This provides a short summary of compilation options. Most of the descriptions were taken directly from the Intel compiler manual. For additional information, type man ifx in the computer you plan to run ED2.

Required/strongly recommended flags for all compilations

Flag	Description
-assume byterecl	Specifies that the units for the OPEN statement RECL specifier (record length) value are in bytes for unformatted data files, not longwords (four-byte units). This is the case in ED2.
-FR	Specifies source files are in free format (as opposed to F77 format). All code in ED2 uses free format.
-qopenmp	Generates multithreaded code based on OpenMP directives. This is very useful to speed up simulations as it allows patches to be solved in parallel. In some ifort systems, the option may be -openmp instead of -qopenmp.
-qmkl	This loads the Math Kernel Library, which has the lapack libraries embedded.
-recursive	Ensures that subroutines can call themselves recursively (ED2 contains a few recursive subroutines).
-static	This prevents linking with shared libraries. Use this in F_OPTS only (the ED2 library compilation), not LOADER_OPTS (the ED2 executable compilation).
-traceback	Displays the source file traceback information (e.g., line numbers and the tree of calls) at run time when a severe error occurs.

Debugging flags (KIND_COMP=A)

Flag	Description
-check all,noarg_temp_created	Enables all check options, with the exception of whether arguments were copied to temporary storage. This option will check the following: Bounds. Checks on whether array sub-script and substring references are within declared bounds. Format. This will ensure data formatted for output is consistent with variable type, and if output variable is too large to fit in the designated format description field. Uninitialised variables. Checks that the code does not attempt to use variables that have not been initialized (i.e., assigned a valid initial value). Also, see -ftrapuv
-debug full	This will produce enhanced debug information useful for breakpoints and stepping, and information useful in finding scalar local variables.
-debug-parameters all	Generates debug information for all parameters defined in the program.
-fpe0	This will issue errors whenever a floating point exception occurs. This includes: invalid values, division by zero, overflow, and underflow (the latter only if -no-ftz is set).
-ftrapuv	This assigns non-sensical initial values to all local variables, which helps identifying variables that were not properly initialised.
-g	Produces symbolic debug information in the object file. Useful when running a debugger with the model (idb, gdb, TotalView, ddd, lldb)
-gen-interface	Tells the compiler to generate an interface block for each routine (that is, for each SUBROUTINE and FUNCTION statement) defined in the source file, even if they are not inside modules.
-implicitnone	Sets the default type of a variable to undefined (IMPLICIT NONE).
-O0	Disables all optimisations. Debugger works much better when the code optimisation is disabled
-no-ftz	Prevents the code to flush very small values to zero. This can be useful to spot incorrectly set variables.
-warn declarations	Enables warnings about any undeclared names. The compiler will use the default implicit data typing rules for such undeclared names. The IMPLICIT and IMPLICIT NONE statements override this option.
-warn errors	Tells the compiler to change all warning-level messages to error-level messages; this includes warnings about Fortran standards violations.
-warn interfaces	Tells the compiler to check the interfaces of all subroutines called and functions invoked in your compilation against a set of interface blocks stored separately from the source being compiled.
-warn uncalled	Enables warnings when a statement function is never called.
-warn unused	Enables warnings about variables that are declared but never used.
-warn usage	Enables warnings about questionable programming practices.

Performance flags (KIND_COMP=E)

Flag	Description
-O2	This option is the default for optimisations. The main distribution of ED2 is routinely tested for bugs that would prevent using the aggressive optimisation; therefore, you can take advantage of option -O3 (see below) for most sub-routines.
-O3	Enables -O2 optimisations plus more aggressive optimisations, such as prefetching, scalar replacement, and loop transformations. Enables optimisations for maximum speed, but does not guarantee higher performance. Note. In case the compilation is taking extraordinarily long (several hours), you may want to set an additional variable F_LOWO_OPTS in your include.mk file and use -O2 instead of -O3 (plus the other options used in F_OPTS). This will compile the files known to take notoriously long to compile with -O3 at lower optimisation level, whilst compiling most subroutines with the most aggressive optimisation.

Useful Fortran compilation flags for GNU (gfortran).

This provides a short summary of compilation options. Most of the descriptions were taken directly from the GNU compiler manual. For additional information, type man gfortran in the computer you plan to run ED2.

Required/strongly recommended flags for all compilations

Flag	Description
-fbacktrace	Displays the source file traceback information (e.g., line numbers and the tree of calls) at run time when a severe error occurs.
-ffree-line-length-none	Specifies source files are in free format (as opposed to F77 format), with no line length limit. All code in ED2 uses free format.
-fopenmp	Generates multithreaded code based on OpenMP directives. This is very useful to speed up simulations as it allows patches to be solved in parallel.
-frecursive	Ensures that subroutines can call themselves recursively (ED2 contains a few recursive subroutines).
-static	This prevents linking with shared libraries. Use this in F_OPTS only (the ED2 library compilation), not LOADER_OPTS (the ED2 executable compilation).

Debugging flags (KIND_COMP=A)

Flag	Description
-fcheck=all,no-array-temps	Enables all check options, with the exception of whether arguments were copied to temporary storage. This option will check the following: Bounds. Checks on whether array sub-script and substring references are within declared bounds. Do. Checks for invalid modifications of loop iteration variables Memory. Checks for valid memory allocation during run-time, including pointers and allocatable variables.
-ffpe-trap=invalid,zero,overflow,underflow	This will issue errors whenever a floating point exception occurs. This includes: invalid values, division by zero, overflow, and underflow.
-fimplicit-none	Sets the default type of a variable to undefined (IMPLICIT NONE).
-finit-integer=-2147483648	This assigns nearly NaN initial values to all integer variables, which helps identify variables that were not properly initialised.
-finit-real=snan	This assigns a signaling NaN initial value to all real variables, which helps identify variables that were not properly initialised.
-Wall	Enables commonly used warning options pertaining to questionable programming practices that we believe are easy to avoid. This currently includes (but is not limited to): Missing ampersand in continued character constants Type, rank, and other mismatches between formal parameters and actual arguments to functions and subroutines. Character assignment that would truncate the assigned string. Implicit conversions that are likely to change the value of the expression after conversion, and implicit conversions between different types and kinds Procedures that may be called without an explicit interface. Constant integer division that would truncate its result (e.g., 3/5 evaluates to 0). "Suspicious" code constructs, which are technically legal but usually indicate coding error. DO loop with step either 1 or -1 that would yield an underflow or an overflow during iteration of an induction variable of the loop. User-defined procedure or module procedures that have the same name as an intrinsic. Unused arguments, parameters, or variables.
-Werror	Turns all warnings into errors.

Performance flags (KIND_COMP=E)

Flag	Description
-O2	This option is the default for optimisations. The main distribution of ED2 is routinely tested for bugs that would prevent using the aggressive optimisation; therefore, you can take advantage of option -O3 (see below) for most sub-routines.
-O3	Enables -O2 optimisations plus more aggressive optimisations, such as prefetching, scalar replacement, and loop transformations. Enables optimisations for maximum speed, but does not guarantee higher performance.

Useful C compilation flags for Intel (icc).

Flag	Description
-DLITTLE	Not a native icc option, but this must be provided to compilation instructions, to ensure the code assumes little endian variables.
-g	Produces symbolic debug information in the object file. Useful only if compiling the code for debugging, especially if you plan to use debuggers (idb, gdb, TotalView, ddd, lldb).
-O0 or -O3	Optimisation level: the recommended option depends on the intent. -O0. Disables all optimisations. Debugger works much better when the code optimisation is disabled. -O3. Enables -O2 optimisations plus more aggressive optimisations, such as prefetching, scalar replacement, and loop transformations. Enables optimisations for maximum speed, but does not guarantee higher performance. Recommended option for science and high-performance simulations.
-qopenmp	Generates multithreaded code based on OpenMP directives. This is very useful to speed up simulations as it allows patches to be solved in parallel.
-static	This prevents linking with shared libraries.
-traceback	Displays the source file traceback information (e.g., line numbers and the tree of calls) at run time when a severe error occurs.

Useful C compilation flags for GNU (gcc).

Flag	Description
-DLITTLE	Not a native gcc option, but this must be provided to compilation instructions, to ensure the code assumes little endian variables.
-g	Produces symbolic debug information in the object file. Useful only if compiling the code for debugging, especially if you plan to use debuggers (idb, gdb, TotalView, ddd, lldb).
-fopenmp	Generates multithreaded code based on OpenMP directives. This is very useful to speed up simulations as it allows patches to be solved in parallel.
-mtraceback=full	Displays the source file traceback information (e.g., line numbers and the tree of calls) at run time when a severe error occurs.
-O0 or -O3	Optimisation level: the recommended option depends on the intent. -O0. Disables all optimisations. Debugger works much better when the code optimisation is disabled. -O3. Enables -O2 optimisations plus more aggressive optimisations, such as prefetching, scalar replacement, and loop transformations. Enables optimisations for maximum speed, but does not guarantee higher performance. Recommended option for science and high-performance simulations.
-static	This prevents linking with shared libraries.

A note about compilers

It is possible that your system has multiple compilers available. This list has a few points to consider when choosing which compiler to use based on previous experience. Please mind that things can change over time and some of these remarks may need to be revisited.

• Speed. From former comparisons at the Harvard Cluster, the model takes considerably longer to compile when using ifx and icc, but the simulation runtime is generally faster compared to gfortran/gcc.
• Debugging. None of the compiler settings catch all the bugs or warnings. When developing the code, it is advisable to compile the code with both ifx/icx and gfortran/gcc using the debugging flags.
  • One feature that is unique to ifx is the possibility of generating interfaces. This capability is very useful when the code development involves changing the list of arguments in functions and subroutines, especially when these functions and subroutine are not contained inside a module.
• Portability. The GNU compilers (gfortran/gcc) are free, and therefore it is a good practice to ensure that the model can compile and run using these compilers, even when your preferred compilers are ifx/icx. Whenever a pull request is submitted, GitHub will try to compile the pull request with gfortran, so it is advisable to run tests in gfortran before submitting the pull request. If this is not possible, then make sure to mention that the code was not tested with gfortran.