Open
Description
Issue description
Our team publishes our python libraries as .whl files to our private artifactory repository. I am trying to package one of our libraries with extras. Some of the extras include dependencies from our private repository. When I try to pipenv install my package with extras from a private repository, pipenv seems to only be looking in pypi, even though the whl metadata accurately specifies the private repository as the source of the dependency.
Expected result
My setup looks like this:
setuptools.setup(
name="my-private-library",
author="",
description="Common Module",
url=[redacted],
packages=setuptools.find_packages(),
include_package_data=True,
package_data={"my-private-library": ["*.yml"]},
extras_require={
extra: get_locked_reqs(extra)
for extra in ["ingestor", "utils", "great_expectations", "spark", "snowflake"]
},
classifiers=["Programming Language :: Python :: 3"],
setup_requires=["setuptools_scm"],
use_scm_version={"write_to": "my-private-library/version.py"},
)
Installing extras that only come from pypi works fine:
snowflake % pipenv shell --python 3.9.5
Creating a virtualenv for this project...
Pipfile: /Users/Olivia.Dalglish/sandbox/snowflake/Pipfile
Using /Users/Olivia.Dalglish/.pyenv/versions/3.9.5/bin/python3.9 (3.9.5) to create virtualenv...
⠋ Creating virtual environment...created virtual environment CPython3.9.5.final.0-64 in 648ms
creator CPython3Posix(dest=/Users/Olivia.Dalglish/.local/share/virtualenvs/snowflake-fhNP0dGh, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/Olivia.Dalglish/Library/Application Support/virtualenv)
added seed packages: pip==23.2.1, setuptools==68.2.2, wheel==0.41.2
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
Virtualenv location: /Users/Olivia.Dalglish/.local/share/virtualenvs/snowflake-fhNP0dGh
Creating a Pipfile for this project...
Launching subshell in virtual environment...
Restored session: Thu Oct 5 13:03:38 EDT 2023
. /Users/Olivia.Dalglish/.local/share/virtualenvs/snowflake-fhNP0dGh/bin/activate
snowflake % . /Users/Olivia.Dalglish/.local/share/virtualenvs/snowflake-fhNP0dGh/bin/activate
(snowflake) snowflake % pipenv install "my-private-library[snowflake]==8.2.4.dev3+g2e01559.d20231005" -i https://my-private-artifactory-url
Installing my-private-library==8.2.4.dev3+g2e01559.d20231005...
Resolving my-private-library[snowflake]==8.2.4.dev3+g2e01559.d20231005...
Added my-private-library to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (7d9c202e65d26cc990fcffd2f301c31e3d539f29a1d374aaba9c281d179a0154)!
Installing dependencies from Pipfile.lock (9a0154)...
(snowflake) snowflake % pip list
Package Version
-------------------------- -----------------------------
asn1crypto 1.5.1
certifi 2023.7.22
cffi 1.16.0
charset-normalizer 2.1.1
cryptography 40.0.2
my-private-library 8.2.4.dev3+g2e01559.d20231005
filelock 3.12.4
idna 3.4
oscrypto 1.3.0
packaging 23.2
pip 23.2.1
pycparser 2.21
pycryptodomex 3.19.0
PyJWT 2.8.0
pyOpenSSL 23.2.0
pytz 2023.3.post1
requests 2.31.0
setuptools 68.2.2
snowflake-connector-python 3.0.3
typing_extensions 4.5.0
urllib3 1.26.17
wheel 0.41.2
(snowflake) snowflake %
Actual result
Installing the ingestor extras includes an extra I'm calling my-private-dependency that is hosted on our artifactory. Theres a package on PyPi with an equivalent name that only has version 0.1 released as you can see in the error below
(utils) utils % pipenv install "my-private-library[ingestor]==8.2.4.dev3+g2e01559.d20231005" -i https://my-private-artifactory-url
Installing my-private-library==8.2.4.dev3+g2e01559.d20231005...
Resolving my-private-library[ingestor]==8.2.4.dev3+g2e01559.d20231005...
✔ Installation Succeeded
Pipfile.lock (9c4a63) out of date, updating to (092dee)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (395d37477b27522e268a40d96a99d5ff595c4918e9762b7335db0f3e91092dee)!
Installing dependencies from Pipfile.lock (092dee)...
[pipenv.exceptions.InstallError]: ERROR: Could not find a version that satisfies the requirement my-private-dependency== (from versions: 0.1)
[pipenv.exceptions.InstallError]: ERROR: No matching distribution found for my-private-dependency==
ERROR: Couldn't install package: {}
Package installation failed...
(utils) utils %
If I start up a normal venv and install using pip it works fine
~ % cd sandbox/dsacq_common_extras/test
test % python3 -m venv venv
test % source venv/bin/activate
(venv) test % pip3 install "my-private-library[ingestor]==8.2.4.dev3+g2e01559.d20231005" -i https://my-private-artifactory-url/api/pypi/sandbox-pypi/simple --no-cache-dir
Looking in indexes: https://my-private-artifactory-url/api/pypi/sandbox-pypi/simple
Collecting my-private-library[ingestor]==8.2.4.dev3+g2e01559.d20231005
Downloading https://my-private-artifactory-url/api/pypi/sandbox-pypi/my-private-library/8.2.4.dev3%2Bg2e01559.d20231005/dsacq_common-8.2.4.dev3%2Bg2e01559.d20231005-py3-none-any.whl (133 kB)
|████████████████████████████████| 133 kB 1.8 MB/s
Collecting my-private-dependency@
https://my-private-artifactory-url/api/pypi/dataengineering-pypi/my-private-dependency/13.6.0/my-private-dependency-13.6.0-py3-none-any.whl#sha256=[redacted]
Downloading https://my-private-artifactory-url/api/pypi/dataengineering-pypi/my-private-dependency/13.6.0/my-private-dependency-13.6.0-py3-none-any.whl (114 kB)
|████████████████████████████████| 114 kB 54.7 MB/s
Collecting snowflake-connector-python==3.0.3
Downloading https://my-private-artifactory-url/api/pypi/sandbox-pypi/packages/packages/06/b1/e4ccf79ebf54fbdd51d207c871f60ad673a634bf7a6607d164ede626ccfc/snowflake_connector_python-3.0.3-cp310-cp310-macosx_10_14_x86_64.whl (15.4 MB)
|████████████████████████████████| 15.4 MB 3.4 MB/s
[redacting the rest]
Additionally, the metadata of the whl file looks correct
>>> import subprocess
>>> cmd = ["pkginfo", "-f", "requires_dist", "Downloads/my-private-library-8.2.4.dev3+g2e01559.d20231005-py3-none-any.whl"]
>>> process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
>>> stdout, stderr = process.communicate()
>>> for dependency in stdout.split(","):
... if "my-private-dependency" in dependency:
... print(dependency)
...
"my-private-dependency @ https://my-private-artifactory-url/api/pypi/dataengineering-pypi/my-private-dependency/13.6.0/my-private-dependency-13.6.0-py3-none-any.whl#sha256=[redacted] ; extra == 'ingestor'"
>>>
Steps to replicate
I'm not sure how much information I can give here since I am working with proprietary artifacts. But in summary,
pipenv install my-private-library[extra-with-private-dep] # fails because it only looks in pypi
pipenv install my-private-library[extra-with-only-pypi-deps] # works fine