Dependency resolution fails for common dependency of app and library, even though a mutually acceptable version is available

[joseph-wakeling-frequenz]

System information

• dub version: 1.15.0
• OS Platform and distribution: Ubuntu 18.04
• compiler version irrelevant, but LDC 1.14.0 FWIW

Bug Description

An application project has a dub dependency which in turn has a dependency on unit-threaded:

"unit-threaded:assertions": ">=0.8.0 <0.9.0"

I tried adding a similar dependency to the main app, but with a wider supported range of versions:

"unit-threaded:assertions": ">=0.8.0 <0.10.0"

When trying to run dub test I get an error message:

Unresolvable dependencies to package unit-threaded:
  mylib >=0.0.0 @/home/me/projects/myapp/submodules/mylib depends on unit-threaded:assertions >=0.8.0 <0.9.0
  myapp ... depends on unit-threaded:assertions >=0.8.0 <0.10.0

... despite the fact that the dependencies are eminently resolvable (just use v0.8.0, which is compatible with both constraints). This can be established straightforwardly: if I use dub.selections.json to specify using v0.8.0, dub is perfectly happy with this.

How to reproduce?

A minimal example: dub build on the project at https://github.com/joseph-wakeling-frequenz/vibe-dependency-failure

Expected Behavior

DUB should recognize that v0.8.0 is a perfectly acceptable resolution of the two version constraints and choose that.

[joseph-wakeling-frequenz]

I just tried to create a VERY minimal example as follows:

(1) create a project in a directory subm with the following contents:

dub.json:

{
    "name": "subm",
    "description": "A minimal test case for DUB version resolution: the submodule",
    "dependencies": {
        "unit-threaded:assertions": ">=0.8.0 <0.9.0"
    }
}

source/subm.d:

module subm;

unittest
{
    import unit_threaded.assertions;

    (1 + 1).should == 2;
}

(2) create another project in another dir, vres, with the following contents:

dub.json:

{
    "name": "vres",
    "description": "A minimal test case for DUB version resolution",
    "dependencies": {
        "unit-threaded:assertions": ">=0.8.0 <0.10.0",
        "subm": {"path": "submodules/subm" }
    }
}

source/app.d:

import std.stdio;
import subm;

void main()
{
    writeln("Edit source/app.d to start your project.");
}

... and then mkdir submodules and git submodule add --name subm path/to/subm submodules/subm.

However, in this case, DUB flawlessly recognizes that unit-threaded:assertions v0.8.0 is a valid resolution of the dependency constraints. So it's really not clear what might be causing the failure in the other case.

[joseph-wakeling-frequenz]

Interestingly enough, in the original project, I get a version resolution failure even if the app specifies * as the unit-threaded:assertions version dependency.

[joseph-wakeling-frequenz]

I have another similar issue for clashes over the taggedalgebraic version, for the same project:

Unresolvable dependencies to package taggedalgebraic:
  eventcore 0.8.43 depends on taggedalgebraic >=0.10.12 <0.12.0-0
  std_data_json 0.18.3 depends on taggedalgebraic ~>0.10.1

... where again one would expect this to resolve straightforwardly to 0.10.13 (and where DUB accepts this with no problem if there is already a dub.selections.json in place).

What's weird is that this showed up after adding an extra submodule dependency to the affected project, even though that submodule has nothing to do with eventcore, std_data_json, or taggedalgebraic. If I remove that submodule, and the corresponding path-based DUB dependency, the above clash does not happen.

So, I'm wondering if having path-based dependencies in dub.json somehow interferes with dependency resolution in other ways.

[John-Colvin]

This bears a lot of resemblance to problems we've had at Symmetry. Thanks @joseph-wakeling-frequenz for doing the legwork to produce a useful bug report.

[joseph-wakeling-frequenz]

The inconsistency of it is weird, which makes me wonder (without having looked at any internals) if it's in some way related to how dependency info is stored and iterated over inside DUB -- so that adding 1 more dependency perhaps changes the order of decisions in a way that impacts outcomes.

I'm particularly suspicious because the taggedalgebraic issue started showing up again in a different context -- this time after adding an extra dependency on a public DUB project (pyd, which has nothing to do with any of the other packages).

@John-Colvin I'm putting together a little document on use-cases for build and dependency management that's intended to cover the situations that only crop up with in-house work. It's going to cover the basic needs (divorced from any particular technical solution) and then go over potential mechanisms to use with DUB, together with problems encountered. Would you be up for a chat about that some time to compare notes?

[joseph-wakeling-frequenz]

The inconsistency of it is weird

I should say: it's not a flaky Heisenbug, when it shows up it's consistent between builds, but it is strange when it shows up.

[Laeeth]

Do you think libsolve could be helpful?
https://github.com/openSUSE/libsolv

[Laeeth]

Obviously not directly if it's a data structure problem or representing the raw data problem.

[Laeeth]

Crazy thought. Wrap dub as a library for SIL so you can play with it from Jupyter and see what crazy things it is doing.

[s-ludwig]

My first suspicion would be that the error message that gets generated in ResolveException.this() skips some of the dependencies to the problematic package, thus hiding the real culprit (whether part of the algorithm or part of the dependency graph).

[joseph-wakeling-frequenz]

@s-ludwig yes, this has also been my suspicion, given that solving one alleged clash has often led to another being reported instead.

[joseph-wakeling-frequenz]

I've put together a minimal example of the problem here: https://github.com/joseph-wakeling-frequenz/vibe-dependency-failure

[joseph-wakeling-frequenz]

An interesting note: if I delete the dep2 dependency from that project (the second path-based dependency, which has no dependencies of its own), then the dependency clash vanishes.