Compiler search uses a pool of workers

[alalazo]

closes #10127

Following the request in #10127 (review) I started refactoring the logic underneath:

$ spack compiler add

The version in this PR shows much better performance than the version in develop. The main modifications are:

1. spack.compilers.find_compilers employs a multiprocess.pool.ThreadPool to execute system commands for the detection of compiler versions.
2. A few memoized functions have been introduced to avoid poking the filesystem multiple times for the same results.

Timing tests will be posted in the discussion below.

[alalazo]

Local workstation (Ubuntu 18.04 - SSD local filesystem)

With a1565f1:

<this branch>$ time spack compiler add
==> Added 3 new compilers to /home/mculpo/.spack/linux/compilers.yaml
    gcc@8.2.0  gcc@7.3.0  clang@6.0.1-svn334776-1~exp1~20181018152737.116
==> Compilers are defined in the following files:
    /home/mculpo/.spack/linux/compilers.yaml

real    0m0,808s
user    0m0,910s
sys     0m0,289s

while using develop at ec2c5e5:

<develop>$ time spack compiler add
==> Added 3 new compilers to /home/mculpo/.spack/linux/compilers.yaml
    gcc@8.2.0  gcc@7.3.0  clang@6.0.1-svn334776-1~exp1~20181018152737.116
==> Compilers are defined in the following files:
    /home/mculpo/.spack/linux/compilers.yaml

real    0m1,508s
user    0m3,755s
sys     0m0,718s

Fidis front-end (rhel 7.3 - GPFS filesystem)
With a1565f1:

<this-branch>$ time spack compiler add
==> Added 1 new compiler to /home/culpo/.spack/linux/compilers.yaml
    gcc@4.8.5
==> Compilers are defined in the following files:
    /home/culpo/.spack/linux/compilers.yaml

real	0m0.611s
user	0m0.432s
sys	0m0.180s

while using develop at ec2c5e5:

<develop>$ time spack compiler add
==> Added 1 new compiler to /home/culpo/.spack/linux/compilers.yaml
    gcc@4.8.5
==> Compilers are defined in the following files:
    /home/culpo/.spack/linux/compilers.yaml

real	0m1.908s
user	0m1.449s
sys	0m0.463s

[alalazo]

@tgamblin @becker33 I need to check what's going on with Travis (I get an error similar to #10186 but I can't reproduce it locally with python 2.7). Anyhow, the numbers above seem interesting - if you could give me an early feedback whether you like this approach or not I'll proceed fixing the Cray environments.

[alalazo]

The tests now pass, even though I had to skip logging some errors for Python 2.6 (see da56be6). The command:

$ spack compiler add

is 2x-3x faster than develop depending on the machine. Looking forward for some early feedback.

[alalazo]

FYI, while working on this PR I noticed that Python 2.6 and Python 2.7, on develop, throw exceptions in the spawned processes due to unicode strings not being handled correctly. The offending part is this output captured by one of the command we use to detect the version of compilers:

Command exited with status 1:
    '/usr/bin/cc' '-V'
cc: error: unrecognized command line option ‘-V’
cc: fatal error: no input files

where the non ascii characters '\u2018' and '\u2019' are used to report the error. With fd4e9aa I was able to solve the issue for Python 2.7, while I couldn't find a proper solution for Python 2.6 (hence da56be6).

The problem can be trimmed down to the different behavior sys.stdout.write exhibits between the two Python versions:

$ python2.6
Python 2.6.9 (default, Apr 29 2018, 12:56:59) 
[GCC 7.3.0] on linux4
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.write(u'\u2018\n')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 0: ordinal not in range(128)

$ python2.7
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.write(u'\u2018\n')
‘

This essentially means that llnl.util.tty currently can't handle unicode for Python 2.X. I'm not sure if this is worth a bug report or not...

[tgamblin]

@alalazo: this looks awesome to me. Especially the performance results.

The only major piece of feedback I have is that it would be nice to use ThreadPool.map() directly, without the execute/invoke stuff. I just think it's easier to read and more familiar to people if you do that. For example, why not do this (or similar with a function):
python tp.map(lambda args: f(*args), arguments)

instead of flipping it around with execute()?

2. As kind of indicated above, I think it would be better if we made it so that the mapped functions don't take callback arguments, because I see that as more portable to a multiprocess pool if it ever makes sense to use one.

More details in comments. See what you think.

[alalazo]

@tgamblin If the code in spack/compilers/__init__.py looks good enough I'll proceed restoring the two unit tests under FIXME and porting the Cray part. In case, let me know if the code needs further explanations in the comments or docstrings.

[alalazo]

Real time reported for compiler detection as of 397cdf3:

PR	Ubuntu 18.04 (SSD ext4)	CentOS 7.3 (GPFS)	Cray (Piz Daint)
Before this PR	0m1,442s	0m1.862s	0m1.577s
After this PR	0m0,680s	0m0.650s	0m1.520s

Summary: the time to detect compilers for usual linux systems improved substantially, Cray remained quite the same (not surprising as the main optimizations were on avoiding useless os.stat and the Cray detection system is based modules instead of on file searching).

[alalazo]

Ready for further review!

[scheibelp]

@alalazo I think this check might be overly strict. I'm seeing cases where path_hints is [] for spack compiler find (which I expect would check PATH) and so this check is not triggered.