KNL planning

[andreasnoack]

UPDATED: I'll try to maintain KNL builds. You can use

/global/cscratch1/sd/noack/julia/julia      # built with gcc 4.8.5
/global/cscratch1/sd/noack/juliaintel/julia # built with icc 17.0.0 20160721

Some adjustments are necessary before Celeste can run on KNL. My thoughts were that this issue could be a source for the relevant information related to KNL, i.e. updated build information for Julia and a list issue related to KNL runs.

The main issue is that we'll need LLVM's development version to run on KNL. To use LLVM-svn, as the development version is called, we'll need the development version of Julia. Hence, Celeste will be having two moving targets which might cause some headaches because LLVM changes might break Julia. In order to make this easier to manage, I'll suggest that we keep commits SHAs for a pair of Julia and LLVM-svn that work together. I'll try to keep them updated.

So the build info right now are

Build instructions

In Make.user put

override LLVM_VER=svn
override USE_INTEL_MKL=1

and execute make -C deps get-llvm. Then checkout

Julia df62c9922c320bff0f6d32bdf3faf87337925e2f
LLVM  584a5d174f2d0776a9390b12799abe8262ffc2ba
Updated 6 March 2017

Intel sepcific

If building with the Intel compilers, it is also necessary to apply this patch to LLVM.
https://reviews.llvm.org/D27610

Right now, Julia's master version is not much different from Julia 0.5 but it might change. If that happens, we'll need to decide what to do. We might need fixes from master so the best thing might be to keep Celeste up to date with master but that could be demanding. Alternatively, we could branch off and only cherry-pick the fixes we need but I'll suggest that we take that discussion only when it becomes relevant.

Finally, there are some performance considerations for KNL. To do well on KNL we'd need threading and vectorization to work well. Some of the code base might not vectorize in the form it's written now (I don't know yet) but we should be aware that some refactoring for vectorization might be needed.

[andreasnoack]

Celeste is running on KNL. See screenshot below. It seems to work fine except that much of the code is single threaded so the runtime is pretty slow for benchmark_infer.

[andreasnoack]

@kpamnany I've updated the SHAs. Yichao's patch to LLVM has been merged so latest svn version works with Julia master.

[Keno]

I've ported rr to Cori/KNL (rr-debugger/rr#1904). For any segfaults, etc. capturing such a problem in rr and keeping the trace would allow me to easily diagnose. I suspect it won't work with MPI out of the box, but looking at that is on my todo list for the end of next week.

[andreasnoack]

I've updated the SHAs to versions with Keno's fixes included. I've also rebuilt /global/cscratch1/sd/noack/julia/julia such that it includes the fixes. I'll recommend that you wait about 24 hours before you try anything with this binary since many packages need new tags after some changes to in Base's macro handling.

[andreasnoack]

@kpamnany I've now managed to built with icc. See the top post for a link to the binary.

[andreasnoack]

On KNL, the memory allocation required by Julia's threads and LAPACK (eigenvalue problem in Newton's method) through OMP can collide such that Julia crashes. We should probably set OMP_NUM_THREADS=1 when running on KNL.

[jeff-regier]

I always set OMP_NUM_THREADS=1 in the slurm scripts: https://github.com/jeff-regier/Celeste.jl/blob/master/nersc/infer.sl#L16 . I think @kpamnany does this too for runs on the supercomputer.

[Keno]

Linking JuliaLang/julia#19640

[andreasnoack]

I've updated the top comment with the extra info for building with the Intel compilers.

[kpamnany]

Building with Intel compilers, I get:

    JULIA usr/lib/julia/inference.ji
essentials.jl
ctypes.jl
generator.jl
reflection.jl
options.jl
promotion.jl
tuple.jl
range.jl
expr.jl
error.jl
bool.jl
number.jl
int.jl
A method error occurred before the base MethodError type was defined. Aborting...
Core.Inference.#convert() world 1007
(UInt128, 0)
while loading int.jl, in expression starting on line 415
rec_backtrace at /scratch/kpamnany/julia.intel/src/stackwalk.c:84
...

Any ideas?

[andreasnoack]

This must be a version of Julia with JuliaLang/julia#17057. I don't know why it wouldn't work on KNL but we are still adjusting to that change. If you want something running now, I'd try to build 374c3d6b3c3e15c00f7d3df8b5cb7c8a763aa746 instead of latest master.

[andreasnoack]

I've updated and tagged most packages to work and not show warnings on Julia master. The main exception is StaticArrays where my PR is still under review. Therefore you should use the master branch of my fork https://github.com/andreasnoack/StaticArrays.jl. Hopefully, we'll be able to get it reviewed and merged today but I don't have commit access to that repo.

A minor problem is DataFrames and DataArrays. I don't think they are critical in the computations but they still throw a lot of warnings and it is not likely that DataArrays will be fixed. We should check that we don't call any code that uses DataArrays or DataFrames when we benchmark because the deprecation warnings are slowing things down a lot.

Finally, the Intel build in the juliaintel directory segfaults in the benchmark_infer.jl tests but the gcc build doesn't. This only happens on KNL and I only realized this yesterday after all the packages had been fixed. @Keno is looking into this.

[hsseung]

Regarding the build info of 26 January 2017,
The Julia checkout returns the error message "fatal: reference is not a tree:"
The LLVM checkout works fine.