Run multiple cargo tasks in parallel

[sourcefrog]

Although cargo and the test framework try to use many cores, they often go through phases of being single-threaded.

The overall mutation tests would go faster if we ran multiple cargo builds in parallel.

Conceivably up to NCPUs builds could be run in parallel. But on some machines that might cause thrashing or too much memory pressure. Possibly we should also look at available memory. There should at least be an option to limit it. As a starting point perhaps NCPUs/2 is reasonable.

This requires making multiple scratch directories. However, we should not make more than the number of discovered mutants.

---

• Factor out a CargoTask with a poll method
• Launch a thread for each worker
• Add a command-line option --jobs, -j and thread it through.
• For now, still default to 1 job?
• Make multiple build directories
• Give the progress model a concept of multiple cargo tasks being underway
• In the lab code:
  • While there's a build dir free and still work to be done, start a new job in it
  • Sleep a fraction of a second
  • Update the progress bar for all running jobs
  • Poll all jobs to see if any have finished
• Update readme docs
• Add at least a smoke test that -j2 works.
• Test Mutant::unapply.
• Clean up self-mutants.

Deferred until later:

• Consider increasing the auto-set timeout when there are multiple jobs?
• Draw progress while copying build trees? #94

[sourcefrog]

@dmoka the code in my process branch is probably ready to try out if you like, with say -j4 in some tree.

[dmoka]

Sorry, I had a hectic last week and could not find time for this! I just see that it is merged, if you still need to test it, let me know, happy to do that.

[sourcefrog]

No problem! It's in the new 1.1.0 release, give it a try and let me know.

[dmoka]

Just tested, works like charm, great job! Also read the corresponding docs.

It will help a lot performance-wise to produce more mutants and running tests against them.

In the doc you say that we can set the jobs up to the number of CPU cores. It is not up to the number of threads, so cpu_cores * 2?

[sourcefrog]

Just tested, works like charm, great job! Also read the corresponding docs.

It will help a lot performance-wise to produce more mutants and running tests against them.

In the doc you say that we can set the jobs up to the number of CPU cores. It is not up to the number of threads, so cpu_cores * 2?

It's more of an indicative number than a rigorous limit. You could set it to the number of threads or even higher. But given that most of the jobs it runs will at least some of the time produce some internal concurrency, I suspect there will usually be diminishing returns in going higher than the number of cores, and more risk that you run out of memory or start thrashing...

I suppose you can imagine a project with a dominant very-long-running single-threaded memory-intensive test that uses exactly one CPU thread for most of the time and that would scale very well up to N_THREADS, on a machine that is unconstrained by RAM. It seems possible but unusual.

But, I'd be interested to hear what you find, on your tree and your machine...