Parallel downloads

[cool-RR]

How about having pip download all the packages in parallel instead of waiting for each one to finish before downloading the next?

After that is implemented, how about having pip start installing one package while it's downloading the next ones?

[cool-RR]

Does anyone care?

[dstufft]

Yes we do care. However it's largely a problem of there being bigger issues to tackle first. Sorry that nobody responded to your ticket though!

[cool-RR]

If we're looking at parallelizing just the download part, is it much more
complex than sticking a concurrent.futures.ThreadPoolExecutor on it and
adding a configuration option to turn it off?

On Mon, Jan 20, 2014 at 7:23 PM, Donald Stufft notifications@github.comwrote:

Yes we do care. However it's largely a problem of there being bigger
issues to tackle first. Sorry that nobody responded to your ticket though!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/825#issuecomment-32779319
.

[jquast]

I disagree. Most downloads are far slower than a disk can write, even if writing many files. As an optional --feature, I think it is a trival choice -- people who know they are downloading over a LAN and onto a slow disk would chose not to --enable such feature. People like myself downloading several-hundred megabyte packages over broadband would gain much to chose to use such --feature.

[sholsapp]

+1 to parallel download (and install if possible).

[mattvonrocketstein]

+1, I frequently work on projects that have hundreds of dependencies and are built from scratch repeatedly throughout the average workday. This stuff crawls even with a pypi mirror on the LAN so having this would be great! http://stackoverflow.com/questions/11021130/parallel-pip-install Options presented are not great, but it does show there is (and has been) interest in a solution to the issue

[jamesob]

+1, this would be huge. Willing to take a crack at implementing.

[mattvonrocketstein]

If anyone is curious why this should be a core feature and not something an external tool provides I can summarize the problem. Using something like "xargs --max-args=1 --max-procs=4 sudo pip install < requires.txt" in general can actually increase the total downloads quite a bit. This happens when for example "django-foo" and "django-bar" both require Django, but specify different constraints for which version is acceptable. I think pip itself has to compute the unified requirement set (probably including requirements-of-requirements) to avoid redundant downloads.

[sholsapp]

Do knowledgeable people know if parallel install is an option? When I tried implementing a build system that executed pip in parallel, I hit problems specifically related to installing modules that had native extensions, but didn't dig further into it. If people know what might be wrong there, I'd love to know.

[fillest]

+1 to parallelization. Not just for downloading but also for things like "Collecting ... File was already downloaded" with pip wheel. It is painfully slow for projects with huge dependency list and multiple hosts, it sometimes becomes one of deployment bottlenecks. I had to add a cache layer on our project - run it only if requirements.txt has changed.

[RonnyPfannschmidt]

note that enabling complete "parallelism" amounts to a relatively complete re-factoring of many internals
this needs an idea for a starting point and quite many re-factoring steps over time

also there is ux issues, while downloading/unpacking is simple, wheel generation and setup.py invocation are full of error potentials/conditions (however they are needed until everybody uses wheels whenever possible and they are needed right in the middle of execution)

[mattvonrocketstein]

so if the UX complication is sidestepped with only parallel downloads/unpacking, doesn't it make sense for that to be the first goal? full parallelism does seem like a huge undertaking.. i'm no expert but as far as that implementation here's two ideas that came to mind

a) in the set of all the requirements and the requirements of requirements, subsets must be discovered wherein individual requirements must be installed sequentially. all such subsets can then be installed in parallel at least. as far as ux, errors here can be shown whenever they occur because any error encountered is truly an error.

b) in the set of all the requirements and the requirements of requirements, a parallel worker pops something out and tries installing it. if the req installs successfully, great. if it fails, we assume another requirement must be installed first and put it back in. as far as ux, errors here must be suppressed unless they persist until the end of this procedure, because any error encountered might be something we can recover from later in the process.

[RonnyPfannschmidt]

the ux issue cannot be side-stepped if any of the downloaded packages is a sdist

the requirement graph is nonlinear and changes whenever more about the dependencies and constrains of a just freshly downloaded package gets known

as soon as a sdist is downloaded, the build/egg-info/wheel process of it has to be triggered

[pradyunsg]

Here's what I think...

To be able to have parallel downloads, pip would need to determine which packages are to be installed and where they need to be fetched from. As of today, it is not possible since pip cannot determine the dependencies of a package without downloading it. While there's a bunch of interesting p

That said, this is definitely something that'll be pretty awesome to have. :)