Disable link time optimization

[spencerkclark]

Link time optimization is turned on by default in the MPI-wrapped compilers under MPICH 4.0, the version of MPICH available from the package manager in Ubuntu LTS image version 22.04:

$ mpichversion
MPICH Version:    	4.0
MPICH Release date:	Fri Jan 21 10:42:29 CST 2022
MPICH Device:    	ch4:ofi
MPICH configure: 	--build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --with-libfabric=/usr --with-slurm=/usr --with-device=ch4:ofi --with-pm=hydra --with-hwloc-prefix=/usr --with-wrapper-dl-type=none --enable-shared --without-yaksa --prefix=/usr --enable-fortran=all --disable-rpath --disable-wrapper-rpath --sysconfdir=/etc/mpich --libdir=/usr/lib/x86_64-linux-gnu --includedir=/usr/include/x86_64-linux-gnu/mpich --docdir=/usr/share/doc/mpich CPPFLAGS= CFLAGS= CXXFLAGS= FFLAGS=-O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -fallow-invalid-boz -fallow-argument-mismatch FCFLAGS=-O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -fallow-invalid-boz -fallow-argument-mismatch BASH_SHELL=/bin/bash
MPICH CC: 	gcc  -g -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security  -O2
MPICH CXX: 	g++  -g -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -O2
MPICH F77: 	gfortran -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong  -fallow-invalid-boz -fallow-argument-mismatch -g -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -cpp  -fallow-invalid-boz -fallow-argument-mismatch -O2
MPICH FC: 	gfortran -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong  -fallow-invalid-boz -fallow-argument-mismatch -g -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -cpp  -fallow-invalid-boz -fallow-argument-mismatch -O2
MPICH Custom Information:

$ mpif90 -show
gfortran -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -fallow-invalid-boz -fallow-argument-mismatch -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -I/usr/include/x86_64-linux-gnu/mpich -I/usr/include/x86_64-linux-gnu/mpich -L/usr/lib/x86_64-linux-gnu -lmpichfort -lmpich

$ mpicc -show
gcc -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -I/usr/include/x86_64-linux-gnu/mpich -L/usr/lib/x86_64-linux-gnu -lmpich

$ mpicxx -show
g++ -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -I/usr/include/x86_64-linux-gnu/mpich -L/usr/lib/x86_64-linux-gnu -lmpichcxx -lmpich

In separate experiments it has been determined that this optimization appears to lead to differences between the fortran executable and a Python-wrapped version of SHiELD (since a separate linking step is required to produce each, giving the optimizer the opportunity to change results). This PR is just an experiment to see if disabling link time optimization also leads to reproducible builds with version 11.4 GNU compilers (in a quick hacky way it just adds the -fno-lto flag to the FFLAGS and CFLAGS used in the GNU Makefile template in SHiELD_build). We'll see if repeatedly triggered builds succeed (update: through one build locally to generate the checksums and four consecutive successful builds in CI, this experiment appears to have been successful).

Note that with MPICH installed in our other environments, link time optimization is not enabled by default, which I believe is why we have not encountered issues like this with FV3GFS, either in our nix development environment in the fv3gfs-fortran repository or with the version of MPICH available in Ubuntu LTS image 20.04, used for the fv3net prognostic run.

[spencerkclark]

We ultimately decided to address this in ai2cm/SHiELD-wrapper#1 by using the identical base image to that in the fv3net prognostic run, i.e. a version 20.04 Ubuntu LTS image. This PR provides a path forward for future images, however.