API to parse requirements.txt for setup.py

[techtonik]

requirements.txt could a convenient single declarative point for specifying dependencies if it could be pasted into setup.py install_requires as-is. But requirements.txt contains comments and references to other requirements files, which seem to be incompatible with install_requires (reference needed).

It would be nice to have official API that parses requirements.txt file into a list suitable for install_requires section.

https://stackoverflow.com/questions/14399534/reference-requirements-txt-for-the-install-requires-kwarg-in-setuptools-setup-py

[benoit-pierre]

There's also no support for environment markers in install_requires requirements (e.g. python-xlib==0.19; "linux" in sys_platform will not work), something that is handled by extras_require.

Note that install_requires can just be a multi-lines string, so if you stick to a subset of pip's requirements format that is supported by setuptools, you should be able to use something as simple as:

with open('requirements.txt') as fp:
    install_requires = fp.read()

setup(install_requires=install_requires, ...)

in your setup.py.

Also looks like a duplicate of #1074.

[techtonik]

Note that install_requires can just be a multi-lines string

Nice. Save me a lot of time. If only I could detect when requirements.txt becomes incompatible without waiting for setup.py to fail.

so if you stick to a subset of pip's requirements format that is supported by setuptools

Are those differences documented somewhere?

[benoit-pierre]

Well, at the very least, setuptools should error out if environment markers are used:

 setuptools/dist.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git i/setuptools/dist.py w/setuptools/dist.py
index 6b97ed33..eb9e4442 100644
--- i/setuptools/dist.py
+++ w/setuptools/dist.py
@@ -153,7 +153,11 @@ def assert_bool(dist, attr, value):
 def check_requirements(dist, attr, value):
     """Verify that install_requires is a valid requirements list"""
     try:
-        list(pkg_resources.parse_requirements(value))
+        reqs = []
+        for r in pkg_resources.parse_requirements(value):
+            if r.marker:
+                raise ValueError("environment markers are not supported, in '%s'" % r)
+            reqs.append(r)
     except (TypeError, ValueError) as error:
         tmpl = (
             "{attr!r} must be a string or list of strings "

And the documentation should be fixed, as both direct install with setup.py and from the generated wheel may fail.

[benoit-pierre]

After inspection, environment markers do work when installing from source, but not with the generated wheels, see #1081.

[jaraco]

Agreed this is a duplicate of #1074.

[tuukkamustonen]

Might be a stupid question, but does pip's requirements.txt format even support named extras_requires dependencies? E.g. parsing/converting following from a requirements.txt file:

extras_requires={
    'extra_scope': 'somepackage >= 1.0.0'
}

From like:

somepackage >= 1.0.0; [extra_scope]

?

[techtonik]

#1074 needs a better title.

Also, there are two scenarios in parsing requirements.txt which add the complexity into resolution of this issue:

1. parsing requirements.txt just to get structured data about all package dependencies (including optional and platform specific)
2. parsing requirements.txt to match requirements with specific current system

2 is actually two step process, which includes step 1. But because 2 is more often needed than 1, it is mixed up together and makes API hard.

[techtonik]

Might be a stupid question, but does pip's requirements.txt format even support named extras_requires dependencies?

@tuukkamustonen https://stackoverflow.com/a/43090648/239247 - is that it?

[tuukkamustonen]

https://stackoverflow.com/a/43090648/239247 - is that it?

@techtonik That is for depending on named extras in other packages. I'm after being able to declare them in mine, so to say something like pip install -r requirements.txt [extra1, extra2] (like in pip install .[extra1, extra2].

So some format that I can parse from requirements.txt into setup.pys:

extras_requires={
    'extra_scope': 'somepackage >= 1.0.0'
}

[techtonik]

@tuukkamustonen so, you want a mechanism to specify different sets of optional dependencies? I believe it is done with tags specified in square brackets.

somepackage >= 1.0.0 [extra_scope]

https://www.python.org/dev/peps/pep-0508/#extras
https://stackoverflow.com/questions/3664478/optional-dependencies-in-a-pip-requirements-file

[tuukkamustonen]

@techtonik That syntax just gives a parsing error (in pip 9.0.1).

[techtonik]

@tuukkamustonen then it is a bug.

[jaraco]

@tuukkamustonen You use setup.py to declare your package's dependencies (and extras). You use requirements.txt to install any number of packages (and their dependencies).

So if you want to declare an extra scope for your package, you do that in setup.py:

extras_require={
  'extra_scope': [
    'somepackage >= 1.0.0',
  ],
}

Then, if you want to install your package with the extra scope, you can pip install .[extra_scope] or put .[extra_scope] in your requirements.txt or if you've uploaded your package to pypi, you can pip install your_package[extra_scope] or pip install your_package[extra_scope] >= 9.0.

Does that help?

[tuukkamustonen]

@jaraco Thanks for the write-up, but my question was if I can do that (extras_require declaration) in requirements.txt. So to define named extras in requirements.txt.

The reason I asked this in the context of this ticket is that if I wanted to parse named extras from requirements.txt into extras_require programmatically (even though everyone is recommending against it, but this ticket is about supporting that use case anyway), the API that this ticket suggests, should also support that. But that leads to a question if requirements.txt syntax even supports named extras.

Maybe an easier to understand use case would be this: Let's forget setup.py altogether and say that I have dev-requirements.txt which define requirements for being able to build and upload my project. In addition to that I could have test-requirements.txt and doc-requirements.txt for also being able to run tests and compile documentation.

Why would I split the dev requirements into several files? In CI (e.g. Jenkins), not every job needs all the dependencies, and maybe I want to optimize the time it takes to setup a virtualenv (let's say I'm spinning up a new Docker container for each build).

In order to avoid cluttering my repository with *-requirements.txt files, it would be fun to describe them all in one dev-requirements.txt, as in:

# Base
setuptools
twine >= 1.0.0

# Tests
pytest >= 2.0.0 [tests]

# Doc
sphinx [tests]

And then being able to run something like:

pip install -r dev-requirements.txt
pip install -r dev-requirements.txt [tests]  # yeah I don't think this is supported?
pip install -r dev-requirements.txt [doc]

I followed the syntax suggested by #1080 (comment) here (but note that it gives a parsing error). According to @techtonik that's a bug. There may be some confusion here, so is it really a bug, or just feature that is not specified/supported?