Pipenv install fails to install sub dependencies from private repos

[siddartha1992]

Describe you environment

1. OSX (MAC)
2. Python version: 2.7.10
3. Pipenv version: 8.2.6

Issue

1. setup.py inside invocations repo -> installing CaaS package :

install_requires=[
    'CaaS>=1.0'
  ],

  dependency_links=[
    'https://<private_repo>#egg=CaaS-1.0'
  ],

2. Installing invocations in a virtualenv
   python setup.py install

3. Valdiating CaaS installed correctly.

(test_1) c4b301cf5d25:invocations quj291$ pip freeze
CaaS==1.0

So far everything works.

4. Created a Pipfile

[requires]
python_version = '2.7'

[packages]
invocations = { git = 'git://<private-repo>/invocations',  ref = 'master' }

5. Fails to install invocations because CaaS package cannot be found: pipenv install

Collecting CaaS>=1.0 (from invocations)

  Could not find a version that satisfies the requirement CaaS>=1.0 (from invocations) (from versions: )
No matching distribution found for CaaS>=1.0 (from invocations)

6. Tried pipenv install --verbose

Collecting CaaS>=1.0 (from invocations)
  1 location(s) to search for versions of CaaS:
  * https://pypi.python.org/simple/caas/
  Getting page https://pypi.python.org/simple/caas/
  Looking up "https://pypi.python.org/simple/caas/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/caas/ HTTP/1.1" 404 33
  Status code 404 not in [200, 203, 300, 301]
  Could not fetch URL https://pypi.python.org/simple/caas/: 404 Client Error: Not Found (caas does not exist) for url: https://pypi.python.org/simple/caas/ - skipping
Cleaning up...

Tries to get CaaS from pypi instead of private github repo's inside dependency_links of invocation's setup.py

Is this expected? How can I install CaaS package?

Thanks!

[techalchemy]

You need to include the editable flag for VCS repositories to resolve sub dependencies I believe

I.e. invocations = { git = 'git://<private-repo>/invocations', ref = 'master', editable = true}

Note that pipenv even issues a warning when you do this:

[siddartha1992]

1. Updated Pipfile

[requires]
python_version = '2.7'

[packages]
invocations = { git = '<private_repo>/invocations', ref = 'master', editable = true }

2. Output of pipenv install --verbose

c4b301cf5d25:docker-server-10-20 quj291$ pipenv install  --verbose
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches CaaS>=1.0
Tried: (no version found at all)

3. Ran with skip-lock option --skip-lock: pipenv install --skip-lock --verbose

Collecting CaaS>=1.0 (from invocations)
  1 location(s) to search for versions of CaaS:
  * https://pypi.python.org/simple/caas/
  Getting page https://pypi.python.org/simple/caas/
  Looking up "https://pypi.python.org/simple/caas/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/caas/ HTTP/1.1" 404 33
  Status code 404 not in [200, 203, 300, 301]
  Could not fetch URL https://pypi.python.org/simple/caas/: 404 Client Error: Not Found (caas does not exist) for url: https://pypi.python.org/simple/caas/ - skipping
Cleaning up...

and

  Could not find a version that satisfies the requirement CaaS>=1.0 (from invocations) (from versions: )
No matching distribution found for CaaS>=1.0 (from invocations)

4. pipenv graph

invocations==0.1.5
  - boto [required: >=2.46, installed: 2.48.0]
  - CaaS [required: >=1.0, installed: ?]
  - invoke [required: >=0.15, installed: 0.21.0]
  - PyYAML [required: >=3.12, installed: 3.12]
  - simplejson [required: >=3.10, installed: 3.11.1]

Its still downloading CaaS from https://pypi.python.org/simple/caas/ instead of dependency_links in setup.py of invocations package.

Thanks!

[siddartha1992]

Updated the title to reflect the problem.

• boto, invoke, PyYAML, and simplejson (public repos) are all dependencies of invocations and they were installed without any issues.

• CaaS (private repo) wasn't able to install. pipenv tried to install this package from https://pypi.python.org/simple/caas/ instead of what's set in dependency_links in setup.py

[techalchemy]

Oh I see what you're saying, I either misunderstood or we solved one problem but uncovered another. I have an idea about where this might be happening but it will be messy to untangle since I don't have an example repository. Essentially, we aren't currently resolving anything in setup.py since we rely on pip to do that resolution. Out of curiosity, what is the output of pipenv run pip freeze?

[techalchemy]

P.S. Thanks for the update and the thorough report, it makes things much easier

[siddartha1992]

I have abstracted some additonal packages from Pipfile and setup.py, but here is the complete output of pipenv run pip freeze:

c4b301cf5d25:docker-server-10-20 quj291$ pipenv run pip freeze
ansible==2.4.0.0
asn1crypto==0.23.0
bcrypt==3.1.4
boto==2.48.0
cffi==1.11.2
cryptography==2.1.1
enum34==1.1.6
idna==2.6
invoke==0.21.0
ipaddress==1.0.18
Jinja2==2.9.6
MarkupSafe==1.0
paramiko==2.3.1
pyasn1==0.3.7
pycparser==2.18
PyNaCl==1.1.2
PyYAML==3.12
simplejson==3.11.1
six==1.11.0

Just a hunch, but might be related to : https://github.com/kennethreitz/pipenv/issues/571

[techalchemy]

Do you have underscores in your repo name?

[siddartha1992]

No, but it has hyphens. (referring to private repo)

[erinxocon]

Last check, hyphens through errors in repo links I think. Marking as a bug for fixing. Thanks for the through report @siddartha1992! And thanks for the ongoing help @techalchemy

[techalchemy]

@erinxocon the original issue was related to pep 440 normalization where we were replacing underscores with dashes. That has been an ongoing issue and has come up in a few places, but we don't do any manipulation that I know of with dashes. We have had a number of issues with VCS urls related to pip-tools resolving and may need /cc @vphilippon on this but not sure if this is in his wheelhouse. I've done a bunch of work on that code, just not sure how far to take the design decisions there vs resolving them in piptools

[techalchemy]

@siddartha1992 as a point of clarification on this issue, did pipenv fail to install the dependencies or did it fail to resolve the dependencies? The output you provided doesn't make it completely clear but there is a big difference as to the cause of the problem