Wrong line updated when package listed twice in python manifest with environment / version markers

[jasonrudolph]

Some python packages have manifests (requirments.txt, requirement.in etc) that specify different versions of a single dependency based on the version of Python being used. For example, we can see this in the manifest for SAP/cf-python-logging-support:

django==1.11; python_version == '2.7'
django==2.2.13; python_version >= '3.5'

When Dependabot encounters such a manifest, it currently creates a pull request that changes the wrong line in the manifest. In the SAP/cf-python-logging-support#51 shown below, we see that the title claims to be upgrading django 1.11 to 1.11.29, but the diff shows that it's actually downgrading django from 2.2.13 to 1.11.29:

Package manager/ecosystem: pip

What you expected to see, versus what you actually saw:

I expect the diff to be this:

- django==1.11; python_version == '2.7'
+ django==1.11.29; python_version == '2.7'
django==2.2.13; python_version >= '3.5'

But it was actually this:

django==1.11; python_version == '2.7'
- django==2.2.13; python_version >= '3.5'
+ django==1.11.29; python_version >= '3.5'

For a minimal repository that reproduces this issue, please see https://github.com/jasonrudolph/dependabot-vs-repeated-python-deps.

[jasonrudolph]

@xlgmokha and I researched this yesterday, and captured a few early observations:

What if the versions are specified in the manifest in a different order?

In the bug report above, the manifest lists the dependencies like so:

django==1.11; python_version == '2.7'
django==2.2.13; python_version >= '3.5'

With that ordering, Dependabot chose the first django entry in the list and looked for an newer version (hence the PR title, "Bump django from 1.11 to 1.11.29"). But when Dependabot created the diff, it updated the last django entry in the list.

We wondered how it would behave if the manifest had listed the dependencies in a different order. So, we created a repository with a manifest like so:

django==2.2.13; python_version >= '3.5'
django==1.11; python_version == '2.7'

In this situation, Dependabot once again chose the first django entry in the list and looked for an newer version: It created a PR titled "Bump django from 2.2.13 to 3.1.7"). And once again, when Dependabot created the diff, it updated the last entry in the list like so:

  django==2.2.13; python_version >= '3.5'
- django==1.11; python_version == '2.7'
+ django==3.1.7; python_version == '2.7'

What are our options?

I think we can agree that the current behavior is wrong, but we suspect that there are various potential improvements that require significantly different amounts of effort:

1. Don't try to update these dependencies at all: If Dependabot sees that a dependency is listed multiple times in a manifest, don't try to update it at all. This isn't great, but doing nothing would arguably be an improvement over the current behavior of opening misleading/incorrect pull requests.

2. Update one instance of the dependency: If Dependabot sees that a dependency is listed multiple times in a manifest, pick one (and only one) instance of it and update the version for that instance. For example, given a manifest like this...

   django==1.11; python_version == '2.7'
   django==2.2.13; python_version >= '3.5'
   

   ... Dependabot might choose to only perform updates for the first instance of django (django==1.11; python_version == '2.7'). In that case, it would open a pull request titled "Bump django from 1.11 to 1.11.29", and the diff would look like this:

   - django==1.11; python_version == '2.7'
   + django==1.11.29; python_version == '2.7'
     django==2.2.13; python_version >= '3.5'

   People might be surprised that they're not getting Dependabot updates for the second instance of django (django==2.2.13; python_version >= '3.5'), but they'd still be getting value from the PRs that Dependabot is opening.

3. Update all instances of the dependency: Given a manifest like this...

   django==1.11; python_version == '2.7'
   django==2.2.13; python_version >= '3.5'
   

   ... Dependabot would open two PRs:

   PR 1 Title: "Bump django from 1.11 to 1.11.29"

   PR 1 Diff:

   - django==1.11; python_version == '2.7'
   + django==1.11.29; python_version == '2.7'
     django==2.2.13; python_version >= '3.5'

   PR 2 Title: "Bump django from 2.2.13 to 3.1.7"

   PR 2 Diff:

     django==1.11; python_version == '2.7'
   - django==2.2.13; python_version >= '3.5'
   + django==3.1.7; python_version >= '3.5'

We're not starting on any of these right away, but we wanted to capture this here to help kick-start this work when we're ready to pick it up.

[jeffwidman]

Another open source repo demonstrating this was reported by @adamjstewart in:

• Non-deterministic behavior for environment markers in requirements.txt #5300
• Bump numpy from 1.22.4 to 1.23.0 in /requirements microsoft/torchgeo#622