Replacing the URL of a source (e.g. PyPI) at the global level

[JacobHenner]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Similar to one of the proposals in #1070 (which was recently marked stale), Poetry should allow the user to override the default repository URL (PyPI). The user should be able to do this without modifying pyproject.toml.

In certain environments (e.g. corporate networks) PyPI is unavailable, but a mirror exists. These users should be able to specify the address of the mirror without modifying project files, as the mirror settings are irrelevant to contributors in different environments. Similarly, if a mirror user adds a dependency, the generated lock file should not list the user's mirror as the source. The source should remain the default (which in most cases would refer to standard PyPI).

This feature exists in pipenv, see pypa/pipenv#2075 (where the need for this functionality is described in greater detail) and pypa/pipenv#2281.

[countergram]

This is essential for many business uses, not simply when PyPI is unavailable but also in any case where the organization has its own libraries (not uncommon). Note that since some private repo tools (e.g. Nexus) use basic auth URLs, putting the repo URL into a project config file is absolutely inappropriate and a global config or environment variable (e.g. pip.conf, PIP_INDEX_URL) is necessary.

[cjw296]

#625 also seems related.

Something I tried, which might be nice to make work:

poetry config repositories.pypi https://.../+simple/

[vlcinsky]

@sdispater, I wonder, if #1070 elaboration of requested feature is usable as is or it needs some update. If so, I volunteer to join effort with one or few others, have a telco and try to move this request on as this is one of two showstoppers for our usage of poetry (the other is managing versions of the resulting package - but this I definitely do not want to discuss here).

[lovepocky]

a simple patch for ci/cd:

# install dependencies from lock file
COPY pyproject.toml poetry.lock /opt/app/

RUN sed -i "s/${origin_pypi_url}/${private_pypi_cache_url}/g" poetry.lock
RUN sed -i "s/${origin_pypi_url}/${private_pypi_cache_url}/g" pyproject.toml

RUN poetry install -vvv

[bjoernpollex]

@lovepocky Wouldn't that break the content-hash in poetry.lock? I think this might cause poetry to refresh the lock file.

[sdispater]

Poetry needs the url information of a dependency for a private repository. Otherwise, it cannot guarantee the determinism of the lock file since two files, even with the same name, may not have the same information.

And if it's a question of not storing the private index credentials in the pyproject.toml, only the base url should be put in the pyproject.toml file. The credentials should be configured separately vie the config command or via environment variables, see https://python-poetry.org/docs/repositories/#configuring-credentials

[JacobHenner]

Poetry needs the url information of a dependency for a private repository. Otherwise, it cannot guarantee the determinism of the lock file since two files, even with the same name, may not have the same information.

The idea here is the private repo specified as the override will be a PyPI mirror. The packages served by the mirror will be exact copies of the ones from https://pypi.org/, without any modifications. Anything else belongs in a separate repo, with URLs included explicitly.

[vlcinsky]

Poetry needs the url information of a dependency for a private repository.

I agree. This contributes to usability of poetry as it provides complete information where to install from.

it cannot guarantee the determinism of the lock file since two files, even with the same name, may not have the same information.

{file = "yarl-1.4.2.tar.gz", hash = "sha256:58cd9c469eced558cd81aa3f484b2924e8897049e06889e8ff2510435b7ef74b"}

I thought, that the hash above is calculated from the package file content and does not depend on filename and url and thus it allows to check, that two files (even from different urls) provide exactly the same information.

Treat source url the way git treats remote configuration

The analogy is not perfect, but it is very close to the use case.

git allows to clone a repository, have initial remote configured, but it is easy to change the remote to another git server (e.g. from Github to GitLab or alternative repo name) and all will still work. If I configure the remote badly, git will complain immediately at the first command dealing with remote server, because the commit hashes will not match.

I hope, poetry will once allow me to keep existing pyproject.toml and poetry.lock untouched and accept alternative url (e.g. configured via env variable) of my private pypi for given source (name) to do sort of "temporary git remote reconfiguration".

If my alternative private pypi url serves exactly the same packages for my installation (checked by comparing hashes), all shall run as usually, if alternative url provides different package content, it shall fail.

Such level of determinism would still provide all the service I appreciate from poetry today and would provide enough flexibility to fit common CI/CD processes.

[cjw296]

As above, my use case is a private pypi mirror. At some stage, the public pypi may even be firewalled off, and it doesn't feel right to have to have a different pyproject.toml for use behind a firewall as for in front of it, for the same code.

[jhbuhrman]

I fully agree with #1632 (comment).

IIRC poetry is using pip already under the hood for a certain part of its functionality. Wouldn't it be sufficient if poetry would simply adhere to the pip.conf (Unix-derived) or pip.ini (Windows) [global] configuration items index, index-url, and trusted-host?
(see https://pip.pypa.io/en/stable/user_guide/#config-file)

[NateScarlet]

@jhbuhrman
#1554 (comment) said poetry will not going to respect pip.ini