Add option to limit number of concurrent calls to linker when building with -j

[byorgey]

On my eight-core machine, running cabal install -j8 renders the machine virtually unusable, presumably due to trying to run many invocations of the linker in parallel (?). It would be nice to be able to do something like cabal install -j8 --max-linkers=3 so I can compile, download, etc. up to 8 packages in parallel but only have 3 linking phases running at a time.

However, it's also possible that I have misdiagnosed the problem. The real issue, of course, is that I don't want cabal install -j8 to make my machine grind to a halt.

[tibbe]

I'd like to first make sure that the linking is actually the problem. Why wouldn't running 8 linkers in parallel on an 8-core machine work?

[23Skidoo]

Can't you just use -j7, for example?

[byorgey]

I got this idea from the shake documentation here: http://hackage.haskell.org/package/shake-0.10.7/docs/Development-Shake.html#v:newResource . To quote, "calls to compilers are usually CPU bound but calls to linkers are usually disk bound. Running 8 linkers will often cause an 8 CPU system to grid to a halt." Though I can't say that I fully understand why running 8 disk bound processes would cause the system to grind to a halt.

Re: just using -j7, empirically anything above -j3 or -j4 is just as bad as -j8. The difference between 7 and 8 would not be that bad, but only getting to use 3 of my 8 cores when building packages makes me a sad panda.

[23Skidoo]

OK, it should be possible to implement --max-linkers using a semaphore.

[byorgey]

I will take a shot at implementing it (just wanted to get some feedback before I attempted it), and see if it helps. Is there a canonical semaphore library/abstraction that we should use nowadays?

[23Skidoo]

@byorgey

I have an initial implementation of a minimal semaphore module on this branch: https://github.com/23Skidoo/cabal/commits/ghc-parmake
Doesn't work on Windows yet.

[byorgey]

Ah, cool. So I should wait for that to get merged in?

[23Skidoo]

You can use System.Posix.Semaphore in the meanwhile.

[bennofs]

Maybe the reason that running 8 linkers in parallel causes the system to crash is that linking often requires a lot of RAM when using GHC? So if that's the case (out of memory + lots of swapping?), it would be better to add an option to not start the linker if there is not much free RAM left?

[byorgey]

Yes, that could certainly be the case. I will watch the memory usage next time to see if that's what is happening. However, I am not sure your suggestion would work very well --- it seems one could easily get in a situation where a bunch of linkers fire up all at once (because RAM usage is not too high) but then once they are running they exhaust the RAM.

[byorgey]

Incidentally, after more experience and investigation, I'm pretty sure my inherent problem is not linkers per se but running out of memory, causing my system to start swapping. But if running the linker uses a lot of memory (?) this could still help.

[23Skidoo]

@byorgey Have you tried my patches (#1572)?

[byorgey]

@23Skidoo not yet. I'll give them a try soon.