Skip to content

Rhel6 using Intel

Jump to bottom
Glen Hansen edited this page Apr 19, 2017 · 9 revisions

This set of build instructions are for the Albany suite on Rhel6 linux.

The compilers are Intel 17.0, Intel MPI 5.1, and RHEL 6 linux.

Modules on CEE:

	module load sierra-devel/intel-17.0.1-intelmpi-5.1
	module load sierra-mkl/17.0-2017.2.174

Build supporting packages

Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.

Notes: In this example

  • The TPLs are installed in /projects/albany

Building ZLib

  • Version 1.2.11

      CC=mpiicc CFLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" ./configure --64 --prefix=/projects/albany/intel5.1
  make -j 4
  make install

Building Yaml

  • git clone git@github.com:jbeder/yaml-cpp.git

      INTEL_MPI_DIR=/sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64

  cmake \
  	-DCMAKE_CXX_COMPILER:STRING=${INTEL_MPI_DIR}/bin/mpiicpc \
  	-DCMAKE_CXX_FLAGS:STRING='-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential' \
  	-DCMAKE_C_COMPILER:STRING=${INTEL_MPI_DIR}/bin/mpiicc \
  	-DCMAKE_C_FLAGS:STRING='-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential' \
  	-DCMAKE_INSTALL_PREFIX:PATH=/projects/albany/intel5.1 \
  	-DCMAKE_BUILD_TYPE:STRING=Release \
  	-DBUILD_SHARED_LIBS:BOOL=ON \
  	-DYAML_CPP_BUILD_TOOLS:BOOL=OFF \
  ..

  make -j 4
  make install

Building HDF5

  • Version 1.10.0 patch 1

      ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
   CFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
   FCFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" --enable-parallel \
   --with-zlib=/projects/albany/intel5.1 --prefix=/projects/albany/intel5.1

  make -j 4
  make install

Building parallel-netcdf

  • Version 1.8.1

  • Configure and build:

      ./configure --prefix=/projects/albany/intel5.1 CC=mpiicc F77=mpiifort F90=mpiifort CXX=mpiicpc \
  CXXFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
  CFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
  FFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
  FCFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
  --with-mpi=/sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64

  make -j 8 
  make install

Building Netcdf

  • Version 4.4.1.1

  • Edit the file include/netcdf.h. Make the following edits near line 265:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
  #define NC_MAX_ATTRS    8192
  #define NC_MAX_VARS   524288    /* max variables per file */
  #define NC_MAX_NAME   256
  #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */

  • Add the install directory of the hdf5 shared libs and the libpnetcdf.so file to your LD_LIBRARY_PATH or the config below will fail:

      export LD_LIBRARY_PATH=/projects/albany/intel5.1/lib:$LD_LIBRARY_PATH

  • Now, configure using CMake and build:

      ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc \
      CXXFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
      CFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
      LDFLAGS="-fPIC -L/projects/albany/intel5.1/lib -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
      FCFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
      --prefix=/projects/albany/intel5.1 --enable-netcdf-4 --enable-pnetcdf

  make -j 8
  make install

Building boost

  • Version 1.63.0

  • Build boost with the commands

      echo 'using intel-linux : : /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/bin/intel64/icpc : <cxxflags>"-O3 -DNDEBUG -diag-disable=cpu-dispatch" ;' > ~/user-config.jam
  echo 'using mpi : /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64/bin/mpiicpc : <cxxflags>"-I/sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64/include -O3 -DNDEBUG -diag-disable=cpu-dispatch" ;' >> ~/user-config.jam
  ./bootstrap.sh --with-toolset=intel-linux --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/projects/albany/intel5.1
  ./b2 toolset=intel-linux -j 4
  ./b2 toolset=intel-linux -j 4 install

  • The above bjam command gave me an error, so I removed the following lines from project-config.jam:

      if ! intel-linux in [ feature.values <toolset> ]
  {
      using intel-linux ;
  }

Building ParMetis

  • Version 4.0.3

      wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
  tar -xvf parmetis-4.0.3.tar.gz

  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd parmetis-4.0.3/metis/include
  vi metis.h
  #define IDXTYPEWIDTH 64
  #define REALTYPEWIDTH 64

  • Run CMake to build parmetis

      cd /projects/albany/src/parmetis-4.0.3 
  cd build
  export metis=/projects/albany/src/parmetis-4.0.3/metis/  
  cmake \
  -DCMAKE_INSTALL_PREFIX=/projects/albany/intel5.1 \
  -DMETIS_PATH=$metis \
  -DGKLIB_PATH=$metis/GKlib \
  -DCMAKE_C_COMPILER=mpiicc \
  -DCMAKE_CXX_COMPILER=mpiicpc \
  -DCMAKE_C_FLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -fPIC" \
  -DCMAKE_CXX_FLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -fPIC" \
  ..
  make -j 4
  make install

  cp libmetis/libmetis.a /projects/albany/intel5.1/lib
  cp $metis/include/metis.h /projects/albany/intel5.1/include

SuperLU (optional)

  • Version 4.3

  • Build paths

      mkdir /projects/albany/intel5.1/SuperLU_4.3
  mkdir /projects/albany/intel5.1/SuperLU_4.3/include
  mkdir /projects/albany/intel5.1/SuperLU_4.3/lib

  • Edit make.inc for your machine and environment

      PLAT = _linux
  SuperLUroot = /projects/albany/intel5.1/SuperLU_4.3
  SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
  TMGLIB          = libtmglib.a
  MKLHOME         = /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64
  BLASLIB     = -L${MKLHOME} -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
  LIBS        = $(SUPERLULIB) $(BLASLIB)
  ARCH         = xiar
  ARCHFLAGS    = cr
  RANLIB       = ranlib
  CC           = mpiicc 
  CFLAGS       = -O3 -DNDEBUG -DPRNTlevel=0 -DAdd_ -mkl=sequential
  NOOPTS       =
  FORTRAN      = mpiifort
  FFLAGS       = -O3 -mkl=sequential
  LOADER       = $(CC)
  LOADOPTS     =
  MATLAB       = /usr/sww/matlab

  • Build SuperLU

      make
  cd SRC
  cp *.h /projects/albany/intel5.1/SuperLU_4.3/include

  • Note: Do not make SuperLU in parallel (-j * make argument) as the build will be incomplete.

Build HWLOC

  • HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

  • Version 1.11.6

      ./configure CC=mpiicc CXX=mpiicpc --prefix=/projects/albany/intel5.1
  make -j 4
  make install

Docker Image

SPACK Package

ToDo List

Albany FAQs

Capabilities

Sub-Projects

Compiling

Nightly Tests

Code HowTo

User Group Meetings

PAALS Tutorials

Wiki HowTo

Visualization Utilities

Good Commits

Python and Albany

Clone this wiki locally