Un-deprecate --process-dependency-links until an alternative is implemented

[multimeric]

• Pip version: 8.1.1
• Python version: 3.5.2
• Operating System: Ubuntu 16.04

Description:

The --process-dependency-links was re-added to pip a while ago because there are a number of valid use-cases for the flag: #1955

For this reason, it should probably be un-deprecated until an implementation of PEP 508/440 is in pip. The dependency_links argument to setuptools's setup is also still documented and not deprecated: http://setuptools.readthedocs.io/en/latest/setuptools.html#dependencies-that-aren-t-in-pypi

What I've run:

$ pip install request --process-dependency-links
Collecting request
  Downloading request-0.0.12.tar.gz
  DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
... 

[Kxrr]

Agreed. Current behaviour of pip for dependency-links really sucks.
#4295
#3610

[jleclanche]

[Wrote in #3939, moving my comment here]

So, here's the main issue: If I don't want to use a requirements.txt file (because, say, I want declarative dependencies all specified in setup.cfg), how am I supposed to specify a URL as dependency?

This doesn't work:

[options]
install_requires = 
  requests==2.18.4
  https://github.com/example/example_lib/archive/master.zip

I also think dependency links are weird and fine to drop, but canonically how is this use case served if not with those?

[jleclanche]

Another issue is that dependency_links apparently cannot be specified in setup.cfg, only as setup.py arguments. If they are to be undeprecated, that should be fixed.

[robertour]

Is there any news regarding this? Is there any alternative in progress? dependency_links_ seems to be the only option for internal/private distribution of libraries.

Instead of deprecating it, it should be fixed, a lot of people forget or get confuse because they forget to put both the name-version of the library in the the install_requires and the dependency link that contains the tarbal together with the egg. As an alternative, I would love to see that @jleclanche has suggested, a simple link to the tarball in the install_requires (specifying the egg could be optional).

I don't think we are alone: https://stackoverflow.com/questions/3472430/how-can-i-make-setuptools-install-a-package-thats-not-on-pypi

[RonnyPfannschmidt]

@robertour tools like devpi support a more clear approach since years now

[RonnyPfannschmidt]

@dstufft btw - i would like to argue that the existene of devpi demonstrates a more clean solution to the problem making dependency links unnecessary

IMHO index inheritance + white listing/blacklisting is more consistent and more comprehensible at a ops level

[robertour]

@RonnyPfannschmidt, thanks for the suggestion, it seems I am missing something important (discussions about this deprecation have been going for too long) but generally speaking I cannot agree more with this comment from reddit:

devpi could be a solution, but bringing another service into the game that needs maintenance and having to wait for some administrator to setup, will not get me into working very soon. Also we still would have the need to build packages and then put them onto devip (or has devpi a feature to track git repositories??)

Ideally, I would like an alternative that won't take me more than adding a file in my library, or few lines in the setup.py.

I basically don't understand the underlying decision of removing it, I can see how it can lead to bad practices and insecurities, but as long as it is keep for internal use, it should not be a major issue.

[pradyunsg]

#4175 means that PEP 508 URL support will be there in pip 10. I think this can be closed.

Thoughts @pfmoore @dstufft?

[pfmoore]

I thought 86b0779 meant that the @ syntax cannot be used for dependencies, so this means #4175 isn't a solution for this issue. The comments over there were that we shouldn't implement @ support for dependencies until PyPI was able to block uploads that used it (because we don't want installing something from PyPI to be able to grab stuff from arbitrary URLs, presumably).

[rgerkin]

It shouldn't be deprecated until there is an alternative in place, i.e. PEP 508 support. Until then, too many people are using it.

[ghost]

What's the proper workflow for this?

Let 's say there's a library that I need to add a feature to. I fork it on Github. Feature isn't going to be merged any time soon, so I have a tool that I set to use my Github fork instead of PyPi.

Now all of my users have to add --process-dependency-links when installing my tool with pip, which is a deprecated and even once removed flag?

Is there some option in setup.py that I'm missing or is there really no way around this? It seems like the only viable way to do this is to push up a forked pypi package. That is just going to add to user's confusion once your pull request does get merged, anyway.

[pradyunsg]

Let's come around to this.

--process-dependency-links does not really have an alternative today. So, I'll suggest that we go ahead and un-deprecate it.

However, the flag does mean that pip would reach out to arbitrary URLs -- it should have a warning attached to it about this.

[ziima]

Hi guys, I just like to note that the deprecation pathway in this case was way too short. I know that dependency links have been marked as deprecated for quite some time, but PEP 508 URLs, which can be used to replace them, haven't been implemented until 18.1. As a result there was only a 3 month long window to switch from dependency links to URL requirements, which is very short time for large projects.

[KevinMars]

@rgerkin Hi, I am trying to follow your instructions to no avail,

Searching for PACKAGE@ git+ssh://git@bitbucket.org:OWNER/PACKAGE.git@BRANCH
Reading https://pypi.org/simple/PACKAGE/
Couldn't find index page for 'PACKAGE' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/

PACKAGE@ git+ssh://git@bitbucket.org:OWNER/PACKAGE.git@BRANCH, this is in install_requires.

Would you have an idea why I am getting the above?

[rgerkin]

@KevinMars There are a few differences between my example and what you have, including the use of git_ssh, bitbucket, a.git suffix, a named branch, and no version tag. Maybe one or more of those things is leading pip to search on PyPI instead of at your URL. What version of pip are you using?

[osjerick]

To remark something that I've found: Using setup.py to install the package with python setup.py install still requires declarations of external dependencies in dependency_links.

In your setup.py file:

import os, sys
from setuptools import setup, find_packages

def read_requirements():
    """Parse requirements from requirements.txt."""
    reqs_path = os.path.join('.', 'requirements.txt')
    with open(reqs_path, 'r') as f:
        requirements = [line.rstrip() for line in f]
    return requirements

setup(
    ..., # Other stuff here
    install_requires=read_requirements(),
    )

[jw-cpnet]

@rgerkin Thanks for sharing this solution. But what if I'm using pbr to setup my Python package? How to adapt this to fit pbr?

[justinsantoro]

@KevinMars I have the same exact issue. Did you ever figure out the fix? I am trying to require a specific branch of a private bitbucket repo over SSH.

[PhilippeFerreiraDeSousa]

I just realized --process-dependency-links didn't existed anymore. I'm grateful for all the work the community does. Trying to justify the decision in never ending discussions and a maze of issue closing and redirection was the chosen solution but I still think leaving this option would have harmed nobody.