What should we do with unparseable requirements?

[antonymilne]

(NB Copied over the issue & comments with minor edits from the private-kedro repository that will soon be archived. Original ticket was written by @AntonyMilneQB)

We're running into some problems with "unparseable" requirements in requirements.txt. By this I mean requirements which are valid in a requirements file (i.e. when you do pip install -r requirements.txt) but are not valid in install_requires in setup.py. These are requirements like:

--extra-index-url https://this.wont.work
-r other_requirements.txt
./path/to/package.whl
http://some.website.com/package.whl

The common cause of bug reports (KED-2897, KED-2346) is the local whl file case, but the other things above will cause exactly the same sort of error. In the case of a local wheel file there is a workaround (use package @ file:///path/to/package.whl) but in the other cases there isn't. Packaging a local wheel or the other options doesn't make sense anyway, so the error message is fair enough.

Now there's currently three places where this is relevant:

• kedro pipeline package. Following Cloud native credentials storage #1280, this gives an error when a user tries packaging a pipeline with unparseable requirements in requirements.txt.
• kedro package. Currently this gives an error for some and silently ignores other unparseable requirements.
• kedro build-docs. This crashes horribly when there's unparseable requirements in requirements.txt (KED-2346), which shouldn't be the case - a user should be able to build docs regardless of what's in their requirements file.

Problems

1. Inconsistent behaviour between kedro pipeline package and kedro package (and arguably even within kedro package).

2. kedro build-docs failing. The reason is this does pip install src[docs] to install the extra_requires (sphinx etc.), which will try to install install_requires also and then give an error because there's unparseable stuff in requirements.txt.

Solutions

1. Either make both commands silently ignore unparseable requirements or make both commands fail as soon as there's any unparseable requirement. I can see arguments both ways here but prefer the hard fail option - as @lorenabalan said, otherwise "it gives the wrong impression to the user that the resulting package can actually be run somewhere else, when in reality it can't".

2. The only options I see here are: silently ignore unparseable requirements in project setup.py (which conflicts with my preferred option for 1) or somehow get the kedro build-docs command to install the document requirements in a way that bypasses setup.py (but how?). I don't think there's any way of getting pip install src[docs] to ignore install_requires or parse requirements.txt differently from when you do kedro package (which also runs setup.py). So the behaviour of kedro package is currently tightly coupled to that of kedro build-docs thanks to pip install src[docs] - is there any way to change that?

[antonymilne]

(Comment copied over, originally written by @AntonyMilneQB)

Just chatted with @idanov; here's his opinion.

1. Slight preference for ignoring unparseable requirements but emitting a warning message saying that they're being ignored. But this isn't a strong opinion.

2. He has a clear solution here which is quite different from the current system. The docs requirements actually shouldn't go in extra_requires at all. Instead, they go in a new file requirements_dev.txt, and kedro build-docs will just pip install -r requirements_dev.txt. In fact, other requirements should go there too - looking at our project requirements, almost none of them are for kedro run - they're for kedro test, kedro lint,kedro package, etc. Arguably all of these are dev requirements (i.e. they're not needed to run a packaged kedro project) and could move to requirements_dev.txt. All the kedro X commands that require them would then need to call pip install -r requirements_dev.txt.

[antonymilne]

(Comment copied over, originally written by @idanov)

requirements_dev.txt or dev_requirements.txt or test_requirements.txt, whatever seems to be the standard in Python for build-time requirements.

[antonymilne]

(Comment copied over, originally written by @lorenabalan)

Probably related to Ivan's point - #844

[antonymilne]

Closed in favour of #1536 and #1293.