pip install of a directory is super slow

[msabramo]

See #2195 (comment), for a summary of this issue.

---

I am dubious of why pip needs 17 seconds to process a local directory that is not on NFS (in fact, it's on an SSD drive) for pip, which has no dependencies, since everything is vendored.

$ time pip install --no-install ~/dev/git-repos/pip
DEPRECATION: --no-install and --no-download are deprecated. See https://github.com/pypa/pip/issues/906.
Processing /Users/marca/dev/git-repos/pip
  Requirement already satisfied (use --upgrade to upgrade): pip==6.0.dev1 from file:///Users/marca/dev/git-repos/pip in /Users/marca/dev/git-repos/pip
pip install --no-install ~/dev/git-repos/pip  2.80s user 5.86s system 50% cpu 17.205 total

It should probably at least be logging whatever is taking that long, but maybe it shouldn't even be doing whatever it's doing.

Note that the "Processing" line appears right away and pretty much the whole delay seems to be between that line and the next one.

[tomprince]

It is making a copy of the entire directory, including .git. It probably shouldn't be doing that, no.

[msabramo]

$ du -sh pip
263M    pip

$ du -sk * .cache .git .tox .travis | sort -nr | head -n 5
181860  .tox
34836   tests
31700   .git
9212    pip
2852    build

[msabramo]

I tried passing 3 -v's (time pip install -vvv --no-install ~/dev/git-repos/pip) -- that didn't yield any more info.

[msabramo]

Stepping through it with pdb, things slow down when I get to:

> /Users/marca/dev/git-repos/pip/pip/req/req_set.py(365)prepare_files()
-> unpack_url(

[msabramo]

And yep, @tomprince is right - it slows down when it does a copy of the whole tree:

> /Users/marca/dev/git-repos/pip/pip/download.py(635)unpack_file_url()
-> shutil.copytree(link_path, location, symlinks=True)

[msabramo]

$ time pip install --no-install ~/dev/git-repos/pip
DEPRECATION: --no-install and --no-download are deprecated. See https://github.com/pypa/pip/issues/906.
Processing /Users/marca/dev/git-repos/pip
  2014-12-15 15:23:34.630794: Copying tree; link_path = '/Users/marca/dev/git-repos/pip'; location = '/var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/pip-D6etc4-build'
  2014-12-15 15:23:57.418679: DONE copying tree; link_path = '/Users/marca/dev/git-repos/pip'; location = '/var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/pip-D6etc4-build'
  Requirement already satisfied (use --upgrade to upgrade): pip==6.0.dev1 from file:///Users/marca/dev/git-repos/pip in /Users/marca/dev/git-repos/pip
pip install --no-install ~/dev/git-repos/pip  2.75s user 5.03s system 32% cpu 24.168 total
>>> elapsed time 24s

[msabramo]

Some discussion in #2196

[msabramo]

It's much faster now that #2196 is merged.

[msabramo]

This should be reopened since #2196 was reverted. I'd like to come with an alternative PR that builds an sdist instead of using heuristics to figure out what to copy. See the comments on that PR for details.