-
Notifications
You must be signed in to change notification settings - Fork 35
Graham's notes on CX1
This page is created for notes/potential improvements to the guidelines without disturbing Fabio's original notes.
These are not guidelines to follow specifically - please don't remove info from this page either.
Need to compile things with icc.
Download and compile python 2.7.3, install to home dir.
Needs bzip2 available:
wget http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz
tar xpzf bzip2-1.0.6.tar.gz
cd bzip2-1.0.6
make
make -f Makefile_libbz2_so
Make sure the .so and .h files are found by Python configure.
In order to install pip on CX1, you can use:
curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
python get-pip.py
cd build/pip
python setup.py build
python setup.py install
Install virtualenv (if you don't have it already):
git clone git://github.com/pypa/virtualenv.git
cd virtualenv/
python setup.py install --user
Create a virtualenv in your PyOP2 tree:
python -m virtualenv --distribute .venv
and activate it:
. .venv/bin/activate
You can now use pip to install packages in your virtualenv!
To leave the virutalenv you can use deactivate.
Installing dependencies:
pip install cython decorator pyyaml pytest codepy ply jinja2 mako
pip install https://launchpad.net/instant/1.0.x/1.0.0/+download/instant-1.0.0.tar.gz
pip install hg+https://bitbucket.org/fr710/pycparser#egg=pycparser-2.08
Used guidelines at: http://software.intel.com/en-us/articles/numpy-scipy-with-mkl/
Used NP 1.6.2 and Scipy 0.9.0. Needed this fix: http://mail.scipy.org/pipermail/scipy-user/2011-July/029908.html
you need to
module load opencl
Clone the PyOpenCL repo and edit setup.py lines:
IncludeDir("CL", []),
LibraryDir("CL", []),
to
IncludeDir("CL", ['/apps/opencl/2.0/usr/include']),
LibraryDir("CL", ['/apps/opencl/2.0/usr/lib64']),
Then install with
python setup.py build
python setup.py install
Installation as per the README
git clone http://git.tiker.net/trees/pycuda.git
cd pycuda
git submodule init
git submodule update
python configure.py --no-use-shipped-boost
python setup.py build
python setup.py install
Use the codepy from the OP2 repository. Do not use the CX1 branch, this was just for experimentation. Use the master branch.
In order to get it to see CUSP, I added this at line 457 of pyop2/cuda.py:
nvcc_toolchain.add_library('cusp',['<path to cusp>'],[],[])
As per the usual guidelines, but add --download-f-blas-lapack=1
Petsc4py as per usual guidelines.
Problem: Need libbz2 and rebuild Python with bz2 module. Argh.
Get from http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz and build.
pip install bzr+http://bazaar.launchpad.net/~florian-rathgeber/ufc/python-setup#egg=ufc_utils
pip install https://sourcesup.renater.fr/frs/download.php/2309/ScientificPython-2.8.tar.gz
pip install bzr+http://bazaar.launchpad.net/~fiat-core/fiat/main#egg=fiat
Codepy does not support the Intel compiler. Use the OP2/codepy fork.
#!/bin/bash
# Modules
module load intel-suite/11.1
module load mpi
module load cmake/2.8.9
module load swig/2.0.2
module load gmsh/2.5.0
module load bzr/2.5.0
module load cuda/5.0.35
module load boost/1.52.0
# Fluidity modules
module load vtk/5.8.0
module load netcdf
module load udunits
module load parmetis
module load zoltan
module load libscotch
# Base dir
export BASE=/home/grm08/ics-v2
# In lieu of module load Python/2.7.3
export PYTHON_HOME=$BASE/env
export PYTHON_VERSION=2.7.3
export PATH=$PYTHON_HOME/bin:$PATH
export LD_LIBRARY_PATH=/apps/python/2.7.3/lib:$LD_LIBRARY_PATH
export PYTHONPATH=$PYTHON_HOME/lib/python2.7/site-packages:$PYTHONPATH
export LD_PRELOAD=/usr/lib64/libz.so:/usr/lib64/libxml2.so:$LD_PRELOAD
# Use virtualenv
source $BASE/env/bin/activate
# PETSc
export PETSC_DIR=$BASE/env/lib/python2.7/site-packages/petsc
# OP2
export OP2_DIR=$BASE/OP2-Common/op2
# PyOP2
export PYTHONPATH=$BASE/PyOP2:$PYTHONPATH
# UFL
export PYTHONPATH=$BASE/ufl/trunk:$PYTHONPATH
# FFC
export PYTHONPATH=$BASE/ffc/pyop2:$PYTHONPATH
# Instant
export PYTHONPATH=$BASE/instant/trunk:$PYTHONPATH
# FIAT
export PYTHONPATH=$BASE/fiat/trunk:$PYTHONPATH
# UFC
export PYTHONPATH=$BASE/ufc/trunk/src/utils/python:$PYTHONPATH
# BLAS/LAPACK for Scipy
export BLAS=/apps/intel/cc/11.1.073/mkl/lib/intel64/libblas.so
export LAPACK=/apps/intel/cc/11.1.073/mkl/lib/intel64/liblapack.so
# Fluidity
export PYTHONPATH=$BASE/fluidity/python:$PYTHONPATH
export LD_LIBRARY_PATH=/home/grm08/ics-v2/fluidity/lib:$LD_LIBRARY_PATH
# PyCUDA
export PYTHONPATH=$BASE/pycuda/build/lib.linux-x86_64-2.7:$PYTHONPATH
# Seems to be required for numpy to work
export LD_PRELOAD=/apps/intel/cc/11.1.073/lib/intel64/libiomp5.so:$LD_PRELOAD
#PBS -N buildfluidity
#PBS -l walltime=1:00:00
#PBS -l select=1:ncpus=1:mem=1800mb
source /home/grm08/ics-v2/e.sh
export LIBS="-L${ZOLTAN_HOME}/lib -L${VTK_LIBS} -L${NETCDF_HOME}/lib -L${UDUNITS_HOME}/lib -L${PARMETIS_HOME}/lib -L/usr/X11R6/lib64/ -L${MKL_HOME}/lib/64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core"
export CPPFLAGS="-I${ZOLTAN_HOME}/include -I${VTK_INCLUDE} -I${NETCDF_HOME}/include -I${UDUNITS_HOME}/include -I${PARMETIS_HOME}/include"
export ZOLTAN_DEPS="-L${ZOLTAN_HOME}/lib -L${LIBSCOTCH_HOME}/lib -lpthread -lptscotch -lpatoh -lptscotcherr -lptscotcherrexit -lscotchmetis -lscotch"
export FLUIDITY_DIR=${BASE}/fluidity
export PYTHONPATH=${FLUIDITY_DIR}/python:$PYTHONPATH
cd ${FLUIDITY_DIR}
./configure --disable-petsc-fortran-modules --enable-2d-adaptivity > compile.log 2>&1 && make clean >> compile.log 2>&1 && make >> compile.log 2>&1 && make fltools >> compile.log 2>&1
BOOST_INC_DIR = ['/apps/boost/1.52.0/build/boost_1_52_0']
BOOST_LIB_DIR = ['/apps/boost/1.52.0/build/boost_1_52_0/stage/lib']
BOOST_COMPILER = 'gcc43'
USE_SHIPPED_BOOST = False
BOOST_PYTHON_LIBNAME = ['boost_python']
BOOST_THREAD_LIBNAME = ['boost_thread']
CUDA_TRACE = False
CUDA_ROOT = '/apps/cuda/5.0.35'
CUDA_ENABLE_GL = False
CUDA_ENABLE_CURAND = True
CUDADRV_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CUDADRV_LIBNAME = ['cuda']
CUDART_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CUDART_LIBNAME = ['cudart']
CURAND_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CURAND_LIBNAME = ['curand']
CXXFLAGS = []
LDFLAGS = []