Private repos and --skip-lock strange behavior

[scastria]

Issue description

There seems to be some issue with private repos and the usage of --skip-lock. I have a private repo in AWS CodeArttifact. If I setup my Pipfile with the private repo as a [[source]] listed at the top and specify my private package with the index pointing to the private repo, pipenv install works but pipenv install --skip-lock does not. However, if I remove the [[source]] at the top and remove the index setting on the private package, but setup PIP_EXTRA_INDEX_URL as env variable in my shell, then pipenv install --skip-lock DOES work. It behaves as if using --skip-lock does not pass the credentials to the private repo to allow it to resolve.

Expected result

pipenv install --skip-lock to resolve private package

Actual result

stuff > pipenv install --skip-lock --verbose
Using python: 3.10
Path to python: /opt/homebrew/bin/python3
Creating a virtualenv for this project...
Pipfile: /Users/myusername/stuff/Pipfile
Using /opt/homebrew/bin/python3 (3.10.8) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.10.8.final.0-64 in 112ms
  creator CPython3Posix(dest=/Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/myusername/Library/Application Support/virtualenv)
    added seed packages: pip==23.0.1, setuptools==67.6.1, wheel==0.40.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M
Installing dependencies from Pipfile...
Writing supplied requirement line to temporary file: 'mycompany-utils==0.0.203'
Install Phase: Standard Requirements
Preparing Installation of 'mycompany-utils'
$ /Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M/bin/python /opt/homebrew/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://aws:MYTOKEN@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/ --no-input --upgrade --exists-action=i -r /var/folders/qd/sndx6g1d447d2vx_rcmp2yzr0000gn/T/pipenv-_i4qbgs4-requirements/pipenv-zg2b9_x3-hashed-reqs.txt
Using source directory: '/Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M/src'
Looking in indexes: https://aws:****@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/

Collecting mycompany-utils==0.0.203

  Using cached https://data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/mycompany-utils/0.0.203/mycompany_utils-0.0.203-py3-none-any.whl (50 kB)

ERROR: Could not find a version that satisfies the requirement PyYAML (from mycompany-utils) (from versions: none)
ERROR: No matching distribution found for PyYAML
An error occurred while installing mycompany-utils==0.0.203! Will try again.
Installing initially failed dependencies...
Writing supplied requirement line to temporary file: 'mycompany-utils==0.0.203'
Install Phase: Standard Requirements
Preparing Installation of 'mycompany-utils'
$ /Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M/bin/python /opt/homebrew/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://aws:MYTOKEN@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/ --no-input --upgrade --no-use-pep517 --exists-action=i -r /var/folders/qd/sndx6g1d447d2vx_rcmp2yzr0000gn/T/pipenv-_i4qbgs4-requirements/pipenv-eqcwadvi-hashed-reqs.txt
Using source directory: '/Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M/src'
Looking in indexes: https://aws:****@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/

Collecting mycompany-utils==0.0.203

  Using cached https://data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/mycompany-utils/0.0.203/mycompany_utils-0.0.203-py3-none-any.whl (50 kB)

ERROR: Could not find a version that satisfies the requirement boto3 (from mycompany-utils) (from versions: none)
ERROR: No matching distribution found for boto3
[pipenv.exceptions.InstallError]: ERROR: Could not find a version that satisfies the requirement boto3 (from mycompany-utils) (from versions: none)
[pipenv.exceptions.InstallError]: ERROR: No matching distribution found for boto3
ERROR: Couldn't install package: [Requirement(_name='mycompany-utils', vcs=None, req=NamedRequirement(name='mycompany-utils', version='==0.0.203', req=Requirement.parse('mycompany-utils==0.0.203'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=mycompany-utils, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===0.0.203, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=mycompany-utils==0.0.203)>), markers=None, _specifiers='==0.0.203', index='data_engineering', editable=False, hashes=frozenset(), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=mycompany-utils, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===0.0.203, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=mycompany-utils==0.0.203)>, _ireq=None)]
 Package installation failed...

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

1. Create Pipfile with [[source]]:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-XXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
mycompany_utils = {version="==0.0.203", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

2. export CODEARTIFACT_AUTH_TOKEN with valid AWS codeartifact token
3. pipenv install --skip-lock
4. FAILURE
5. Edit Pipfile to NOT use [[source]], but rely on a PIP_EXTRA_INDEX_URL:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
greenstreet_utils = "==0.0.203"

[dev-packages]

[requires]
python_version = "3.10"

6. pipenv --rm
7. export PIP_EXTRA_INDEX_URL with valid AWS codeartifact URL with valid token
8. pipenv install --skip-lock
9. SUCCESS

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2023.3.20'

Pipenv location: '/opt/homebrew/lib/python3.10/site-packages/pipenv'

Python location: '/opt/homebrew/opt/python@3.10/bin/python3.10'

OS Name: 'posix'

User pip version: '23.0.1'

user Python installations found:

• 3.10.8: /opt/homebrew/bin/python3
• 3.10.8: /usr/local/bin/python
• 3.9.15: /opt/homebrew/bin/python3.9
• 3.9.6: /usr/bin/python3
• 3.8.15: /opt/homebrew/bin/python3.8

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.8',
 'os_name': 'posix',
 'platform_machine': 'arm64',
 'platform_python_implementation': 'CPython',
 'platform_release': '22.4.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST '
                     '2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000',
 'python_full_version': '3.10.8',
 'python_version': '3.10',
 'sys_platform': 'darwin'}

System environment variables:

• MANPATH
• TERM_PROGRAM
• SHELL
• TERM
• HOMEBREW_REPOSITORY
• TMPDIR
• GOBIN
• TERM_PROGRAM_VERSION
• ADOPS_PAT
• OLDPWD
• TERM_SESSION_ID
• USER
• LDAP_USERNAME
• CODEARTIFACT_AUTH_TOKEN
• SSH_AUTH_SOCK
• HOMEBREW_NO_AUTO_UPDATE
• PATH
• __CFBundleIdentifier
• PWD
• JAVA_HOME
• LANG
• LDAP_PASSWORD
• XPC_FLAGS
• PS1
• XPC_SERVICE_NAME
• SHLVL
• HOME
• SHELL_SESSION_HISTORY
• HOMEBREW_PREFIX
• LOGNAME
• GOPATH
• DEV_CLIENT_ID
• INFOPATH
• HOMEBREW_CELLAR
• BITBUCKET_PAT
• DEV_CLIENT_SECRET
• _
• __CF_USER_TEXT_ENCODING
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/Users/myusername/.go/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /Users/myusername/stuff

[matteius]

@scastria Can you try adding to your Pipfile:

[pipenv]
install_search_all_sources = true

More details: https://pipenv.pypa.io/en/latest/indexes/

[scastria]

@matteius Unfortunately, that does not make a difference. Also, just as a reminder, running WITHOUT --skip-lock works fine even without the above install_search_all suggestion

[matteius]

@scastria side question -- why even use --skip-lock? It bypasses most of the benefits and security features of pipenv.

What if you leave the install_search_all suggestion and swap the order of your sources so your private source is the default? You would have to specify anything that needs to come from pypi though. What I am wondering is if your private package is requiring another private package and then failing to resolve it.

Additionally, --skip-lock is broken in general, have you tried the patch branch for this issue? #5653

[scastria]

@matteius That is a good question. My company started using pipenv last year around May and back then, it would sometimes take 10 minutes to run a pipenv install. We discovered that if we added --skip-lock, it only took 30 seconds or less. Therefore, we all got in the habit of avoiding the lock file and have never looked back. But I do agree that we are bypassing a lot of the great features of pipenv by doing so.

Our private package only depends on public packages.

Changing the default repo/index and specifying public packages to come from pypi would be too big of an impact on all of our repos. Let me try the patch branch...

[matteius]

@scastria Yeah, I personally would like to remove the --skip-lock but I think there would be some push back so I am going to hold on that for now. There have been some big performance improvements in pipenv since the time your company started using pipenv. I suspect based on the timeline of my development, that the issue here is --skip-lock never properly accounted for source specifications in the Pipfile (I am speculating) and I'll take that a step further and say pip 23.x made some changes that maybe compounded this fact.

When I originally did my work on index restricted packages, I mostly ignored the code paths for skip-lock because as a feature it didn't make sense to me, and working from the Pipfile is a whole different beast than working from the lockfile in so many ways. Let me know though if that skip lock branch doesn't resolve it and I can tag this issue appropriately.

[scastria]

@matteius Not sure I did it right, but I installed the patch branch with this??

pip install git+https://github.com/pypa/pipenv.git@issue-5367-fix

That gave me version: 2023.3.19.dev0

Using this version had no effect with my original Pipfile.

However, it DID work after adding the install_search_all setting!

[matteius]

@scastria Cool, so the take away there is that pip 23.x basically caused users to required the install_search_all_sources = true setting, I've seen a number of reports resolved with that after we vendor'd in pip 23 this year. The main take-away is skip-lock was broken since at least September when I did the named package categories work and refactor.

[matteius]

@scastria Also would love to implore you to see if the performance of pipenv sync is better for you in newer versions and just run with the lockfile. All of the python package managers have been improving their performance over the last year: https://lincolnloop.github.io/python-package-manager-shootout/

[scastria]

Thanks, I will try.

[matteius]

Expecting a release in the next week.