Automated migration to groups during `poetry add --group dev`

[tomwatson1024]

Running poetry add --group dev (or poetry add --dev) with a dependency that already exists in the [tool.poetry.dev-dependencies] section adds a new entry in [tool.poetry.group.dev.dependencies that may conflict with the existing entry.

It's easily fixed (updating [tool.poetry.dev-dependencies] to [tool.poetry.group.dev.dependencies], and merging the sections if the extra entry has already been added) but it makes the Poetry 1.1 -> 1.2 upgrade painful if you've got lots of repos.

# syntax=docker/dockerfile:1.3-labs
FROM python:3.8
RUN pip install poetry==1.2.1
WORKDIR /src
COPY <<"EOF" pyproject.toml
[tool.poetry]
name = ""
version = "0.1.0"
description = ""
authors = [""]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]
tomli = "1.0.0"

EOF

# Using `poetry add --group dev` and `poetry add --dev` produce the same result.
RUN poetry add --group dev tomli@2.0.0
RUN cat pyproject.toml && poetry lock

produces

 > [6/6] RUN cat pyproject.toml && poetry lock:
#14 0.534 [tool.poetry]
#14 0.534 name = ""
#14 0.534 version = "0.1.0"
#14 0.534 description = ""
#14 0.534 authors = [""]
#14 0.534
#14 0.534 [tool.poetry.dependencies]
#14 0.534 python = "^3.8"
#14 0.534
#14 0.534 [tool.poetry.dev-dependencies]
#14 0.534 tomli = "1.0.0"
#14 0.534
#14 0.534 [[tool.poetry.source]]
#14 0.534 name = 'simple'
#14 0.534 url = 'https://pypi.org/simple'
#14 0.534 default = true
#14 0.534
#14 0.534
#14 0.534 [tool.poetry.group.dev.dependencies]
#14 0.534 tomli = "2.0.0"
#14 0.534
#14 1.122 Updating dependencies
#14 1.123 Resolving dependencies...
#14 1.133
#14 1.133 Because  depends on both tomli (2.0.0) and tomli (1.0.0), version solving failed.

edit: Removed unnecessary [tool.poetry.source] section in Dockerfile.

[neersighted]

This has nothing to do with lock files, but with project files. If you make use of groups in your project, naturally Poetry 1.1 will no longer be able to understand it, as it is a new feature. I have edited your issue to remove the link to the other issue as it is misleading.

This is currently a known drawback of the --dev switch -- one-time migration code that detects the old dev-depedencies and migrates them by rewriting the pyproject.toml to use groups before doing the 'add' would be sufficient.

However, such a feature was considered more trouble than it was worth as most users who migrate over will do so by hand-editing pyproject.toml, and what is happening is very obvious when you open the file.

I'm reclassifying this as a feature request -- it's welcome if anyone wants to write it, but was considered optional and not written by any maintainer as no one had much interest in a mild edge case like this.

[rzuckerm]

I noticed that the remove command seems to handle removing dependencies from dev-dependencies.

poetry/src/poetry/console/commands/remove.py

Line 73 in 844a720

elif group == "dev" and "dev-dependencies" in poetry_content:

I'm curious as to why the add command doesn't look in dev-dependencies when checking if a dependency is already present in pyproject.toml.

[neersighted]

However, such a feature was considered more trouble than it was worth as most users who migrate over will do so by hand-editing pyproject.toml, and what is happening is very obvious when you open the file.

I think I covered that -- both commands are ignorant of dev-dependencies and the only migration currently in the code is implicit conversion to a group.

Contributions are welcome, including if you think this deserves some mention in the docs or you want to introduce a migration in code. No one involved at the time thought it was worth the effort to implement and test, is all.

[rzuckerm]

I'm wondering if dev-dependencies should continue to be used if it exists. Maybe there could be a separate command to migrate from dev-dependencies to group.dev.dependencies?

[neersighted]

dev-dependencies is implicitly used/combined right now -- I don't think we can walk back from that behavior immediately -- warning and removing it in a future version is better, I think.

[rzuckerm]

So, how should add work when there are dev-dependencies under the following conditions given that foo = "^1.0" is in dev-dependencies?

1. poetry add foo
2. poetry add foo^1.1
3. poetry add bar

[neersighted]

I assume that's add -G dev? I think that's up to us to decide -- my immediate thoughts are we could do a wholesale migration the first time -G dev is used. That is to say, migrate everything from dev-dependencies to groups.dev, write pyproject.toml, and then do the add as normal.

Yes, this forces migration for users who are using -G dev on the command line, but we already effectively do that -- this just removes the split brain situation.