Feature proposal: Launch ipcluster from python

[twiecki]

For simple parallelization requirements on a single host it is often an extra burden (especially when designing APIs) to require the user to launch ipcluster. If instead I could also launch these from within my Python library it could be a very nice replacement for multiprocessing which supports this mode. Specifically, if my library is using multiprocessing the user doesn't have to care about anything and get single-machine parallelization for free. The problem is that multiprocessing sucks and I'd much rather use ipyparallel for this as well.

I suppose there might be an unsupported way to do this already by importing ipcluster somehow.

Anyway, I think that would be useful for certain problems.

[minrk]

Agreed. For a summary of the plans I have in my head:

• make ipcluster a service, rather than a script
• ipcluster would manage many instances of controllers and engine groups, not just one at a time
• talk to it via REST API
• add a Python wrapper for the service API, so you can start/stop clusters

This would alleviate a lot of the annoyances of ipcluster, I think.

In the meantime, there's ipython-cluster-helper, which provides much of what you describe, I think.

[twiecki]

Thanks. I think ipython-cluster-helper really only works with cluster schedulers, rather than locally.

[twiecki]

@minrk just wondering if perhaps there's a WIP version I could try.

[minrk]

Sorry, not yet.

[jakirkham]

Also, in favor of this proposal, currently I find myself having to shell out the launch and shutdown the cluster. It would be great not to have to do that. Plus, am a big fan of being able to run multiple instances of the cluster at once.