dev-dependencies that are also extras

[digitalresistor]

I've got a somewhat weird requirement:

Project A provides a bunch of pytest fixtures, and uses pytest itself for testing. I want to provide an extra on the resulting distributable that is called "testing".

So I created the following:

[tool.poetry.dependencies]
python = "*"
pyramid = "^1.9"
sqlalchemy = {version = "^1.2", extras = ["postgresql"]}
psycopg2 = {version = "^2.7", optional = true}
pytest = {version = "^3.5", optional = true}

[tool.poetry.dev-dependencies]
pytest = "^3.5"
pytest-cov = "^2.5"
coverage = "^4.5"

[tool.poetry.extras]
testing = ["pytest"]
postgresql = ["psycopg2"]

This will work correctly, however the dependency on pytest is specified twice. Once under tool.poetry.dependencies and another time under tool.poetry.dev-dependencies. What I would like to do is make sure that the same thing isn't listed twice, so I tried the following:

[tool.poetry.dependencies]
python = "*"
pyramid = "^1.9"
sqlalchemy = {version = "^1.2", extras = ["postgresql"]}
psycopg2 = {version = "^2.7", optional = true}

[tool.poetry.dev-dependencies]
pytest = "^3.5"
pytest-cov = "^2.5"
coverage = "^4.5"

[tool.poetry.extras]
testing = ["pytest"]
postgresql = ["psycopg2"]

Hoping that the testing extra would pick up pytest from dev-dependencies. However this is not the case.

So I moved pytest back up, and made it optional:

[tool.poetry.dependencies]
python = "*"
pyramid = "^1.9"
sqlalchemy = {version = "^1.2", extras = ["postgresql"]}
psycopg2 = {version = "^2.7", optional = true}
pytest = {version = "^3.5", optional = true}

[tool.poetry.dev-dependencies]
pytest-cov = "^2.5"
coverage = "^4.5"

[tool.poetry.extras]
testing = ["pytest"]
postgresql = ["psycopg2"]

Now it is correctly picked up, but there is no way to specify that when installing dev-dependencies that the extra testing should be automatically installed too, since that now contains the pytest dependency.

Trying:

mypackage = { version = "*", extras = ["testing"]}

Did not work since it could not find mypackage, and I am sure that if it could find it, it would likely download it from pypi which is not intended.

Is there a way to refer to self somehow? Or would it be possible to add a dependency in one place (define its version and extras/requirements) without redefining it in dev-dependencies?

The goal ultimately is that a secondary package would have the following pyproject.toml:

[tool.poetry]
name = "B-package"
version = "0.2.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "*"
mypackage = {version = "^0.1"}

[tool.poetry.dev-dependencies]
mypackage = {version = "*", extras = ["testing"]}

And only if you are developing would it install mypackage[testing] but for production it would install mypackage. Using the version constraint from tool.poetry.dependencies vs the constraint specified in tool.poetry.dev-dependencies.

This is related to secondary issue: #128

[dmfigol]

I have similar use-cases for ReadTheDocs automatic builds:
readthedocs/readthedocs.org#4912 (comment)

[sobolevn]

I also have an optional dev-dependency in wemake-django-template:

[tool.poetry.dev-dependencies]
ipython = {version = "*", optional = true }

[tool.poetry.extras]
shell = ["ipython"]

Currently it does not work. wemake-services/wemake-django-template#685

What happens? poetry creates an empty list [extras] inside the lock file.

[chrahunt]

@sobolevn this is by design as mentioned in #606 (comment).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cjw296]

This is still valid.

[thejohnfreeman]

I believe the best solution to this has been proposed in #1644.

[finswimmer]

As @thejohnfreeman wrote, this will be possible with dependency groups which are planned for poetry 1.2. I'll close this one in favor of #1644.

[sh-at-cs]

This hasn't really been solved by #1644, has it? It's still not possible to specify dependencies that belong to both an "extras" group and a Poetry group (like dev) without repeating them, version and all, is it? All that happened is that dev-dependencies became the "dev" group - how does that solve this?

Something like #4842 might be able to make some progress here.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.