Support different versions of the same dependency within a workspace

[alshdavid]

Problem

My Cargo workspace has multiple entry points that compile to executable/bin or dylib targets. Currently, when placed within a workspace, Cargo will attempt to combine their dependencies where they share the same "compatible" versions of a dependency.

The problem is that these project consume third party dependencies that have conflicting versions - either because library maintainers don't adhere to semver correctly or because library maintainers choose to specify exact versions of a dependency (e.g. =0.0.40).

My projects can be compiled on their own without issue however, if they share a workspace, cargo build will fail because Cargo tries/fails to resolve these conflicting dependencies.

Example Case

I have a workspace that has three packages

/crates-main
  /project-bin
    Cargo.toml
  /project-types
    Cargo.toml
/crates-plugins
  /plugin-dynamic-lib
    Cargo.toml
Cargo.toml

crates-main/project-bin
This is the main entrypoint for my application which compiles to an executable. This program is able to consume dynamic libraries using the libloading crate.

crates-main/project-types
This is a shared library compiled as a lib that has no external dependencies, only exporting types to be used by both the "plugins" and the main executable (a.k.a. the "contract").

crates-plugins/plugin-dynamic-lib
This is a library that is compiled to a dylib. It depends on project-typesfor the types required to initialize a plugin and will be consumed by the executable produced byproject-bin`

Note that this package could also be an executable, I am using a dylib in my example because that's my current use case

In summary we have 2 packages that compile to binaries (./project-bin and ./plugin-dynamic-lib.so) and one shared library that is statically linked within those two crates.

Problem

As an example, assume project-bin consumes log = "=0.4.20" and plugin-dynamic-lib consumes log = 0.4.21 (indirectly via a third party dependency external to the workspace).

In a combined workspace, cargo build will error saying that it cannot resolve a compatible version between the two specified.

In reality, these packages will compile to separate binaries so version conflicts of dependencies would not result in a material conflict at runtime.

Current Solution

To get around this today, I simply avoid using a Cargo workspace and compile the projects independently of each other from a build script where shared dependencies are referenced via my_pkg = { path = "../path/to/pkg" }.

The issue with this approach is that rust-analyzer is unable to provide suggestions for the packages when the top level folder is open in the editor - resulting in a less than ideal development experience

Proposed Solution

A few possible solutions to this:

Option 1

Allow for multiple incompatible dependencies to coexist within a workspace if their versions cannot be combined and their consumers are of crate-type bin or dylib

e.g. a package in crates.io with the latest version of 0.4.21

Consumer A

[bin]

[dependencies]
dependency = `^0.4.0`

Consumer B

[bin]

[dependencies]
dependency = `=0.4.20`

Consumer A would get 0.4.21
Consumer B would get 0.4.20

Option 2
Devise a way for rust-analyzer to work with multiple nested projects within a parent directory

Option 3
Perhaps some kind of support for workspaces, isolating dependencies between the workspaces

Notes

No response

[epage]

The problem is that these project consume third party dependencies that have conflicting versions - either because library maintainers don't adhere to semver correctly or because library maintainers choose to specify exact versions of a dependency (e.g. =0.0.40

One of the reasons Cargo is opinionated is to put pressure on libraries to "play nice" by

• Following Cargo community expectations around semver (which is different than a single person's interpretation)
• Discouraging the use of = operators (which we explicitly call out)

[epage]

As for the request, I feel like we should have an existing issue but my search skills are failing me.

• Specify in root that certain edges in the dependency graph should be "private" #7276 proposed the idea of private dependency graphs.
• In theory, RFC: Superseding public/private dependencies rfcs#3516 could help towards something like this.

I do wonder if we'll come to the point where dependency sub-graphs are treated as locked together, of the scale of gitoxide or bevy, making them act more like a cohesive single library. iiuc build-std has a specialized version of this.

[alshdavid]

put pressure on libraries to "play nice" by

I seriously wish this worked out in practice, it's been such a nightmare to work around. (you can skip the next part, I'm just ranting)

I'm writing a JavaScript bundler and want to embed Deno into it to function as a JavaScript runtime for dynamic plugins.

Deno depends on = versions of SWC because SWC doesn't play nice with semver.

I also depend on SWC directly however the conflict between Deno and my internal version of SWC prevents Cargo from resolving dependencies. Further, Deno has a fixed version of Tokio, log and other libraries which makes it difficult to embed.

To try to get around this, I created a new dylib crate in my workspace that bottles up my Deno integration and exposes behaviours via an FFI that I consume from my main application using libloading.

/deno_integration
/my_project

/target
  /debug
    my_project
    deno_integration.so

This is okay enough however my deno_integration dylib crate cannot be part of my workspace Cargo.toml because of those conflicting dependencies (even though it will compile to a discrete target).

So I add deno_integration to my workspace [workspace.exclude] and build it separately to the packages in my workspace.

cargo build # for the workspace
cd deno_integration && cargo build

If I have my workspace open in my editor, I get no rust-analyzer coverage on the deno_integration folder unless I open that folder separately in a new editor window.

It's such an annoying workflow - but it works and unblocks me for now.

[weihanglo]

I believe you've tried this. Just asking in case we forgot: Have you used the [patch] table to patch deno's Cargo.toml?

The other way can help upstream is helping them integrate cargo-semver-checks. The tool is quite handy and easy to use. Cargo the project has used it in every pull request

[epage]

My other question is if swc and deno are meant to be pulled in as libraries like this. These look to be fairly large, complex applications and sometimes the "libraries" for them are more meant for internal purposes. I know I maintain several bins whose lib portion is an implementation detail and I very intentionally do not offer semver compatibility guarantees for those APIs.

[alshdavid]

In the case of swc, it's definitely intended to be used as a library as that's its primary use case (it's an AST parser, etc) - and they have a reputation for poorly adhering to semver. I tried to politely bring it up with the maintainers but they were not receptive.

As for Deno, their crates and docs seem to indicate that this is a use case they support. I don't think this use case is a huge focus for them though as the dependency composition of their crates seem to indicate that.

There are other projects (like Supabase - an OSS "aws lambda"-like engine) who integrate Deno. Supabase's solution was to use some of the deno crates from crates.io and vendor the parts that conflicted

[jozanza]

fwiw, I have the same issue in my workspace, but it's with much smaller lib: zeroize.

This lib happens to be a dependency of a few dependencies across crates within my workspace. I never directly added it. However, I just added a new crate with a dependency on k256. This unfortunately introduced a version incompatibility. Previously, everything had worked well enough since cargo found the range of acceptable zeroize version to be >=1, <1.4. However k256 wants ^1.7.

In practice, since I don't have control of the zeroize dependency version directly in any workspace crates, I can only think of three less-than-ideal options to get around this while still keeping this new crate in the workspace:

1. Attempt to upgrade all of the deps of my workspace crates that depend on zeroize until 1.7 is the upper bounds.
2. Attempt to downgrade k256 until 1.4 is in the range it wants.
3. Fork zeroize on github and apply a patch section to Cargo.toml.

In this scenario, the packages are entirely separate and do not use each other as dependencies, so if zeroize happened to be a different versions in both packages, this wouldn't cause any problems for these packages. But cargo won't allow this for packages in the same workspace. I even attempted to remove the package from the workspace and import it by path, but that results in the same version conflict.

Overall, it's a pretty painful issue that seems increasingly likely to happen as a workspace grows and it's entirely dependency on how package authors specify their own version requirements, which you have no control over. I've run afoul of this issue numerous times over the past few months and it always a huge productivity loss resulting in many hours of twiddling with version numbers, opening issues and PRs in maintainer repos, and/or forking+patching. At this point, it's become the biggest issue I have maintaining a large codebase in rust.

[epage]

everything had worked well enough since cargo found the range of acceptable zeroize version to be >=1, <1.4

imo this is a bug and should be reported. Upper bounds should (almost) always be semver boundaries. Cargo's workflows are designed around that.

Overall, it's a pretty painful issue that seems increasingly likely to happen as a workspace grows and it's entirely dependency on how package authors specify their own version requirements,

Right now, there the design of Cargo puts pressure on people to be "conforming". If we loosen this, people will no longer be "conforming" and a lot of the uniqueness / value-proposition (of semver working) is lost. Cargo is intentionally opinionated. You mentioned running into it a lot. I rarely run into it. That doesn't invalidate your experience but to highlight that it isn't everywhere and that is the risk we introduce by loosening things up on this.

[jozanza]

Sure I take no issue with semver and putting pressure on authors to "conform" to it. Semver's cool and authors should adhere to it.

The issue here I'm raising (and what I assume is the topic at hand based on the issue title) is that there's no sane way to have two different versions of the same dependency in a workspace.

The semver version ranges are not problematic for these crates in isolation. However because cargo just wants them to use the same version in the workspace, it causes a lot of issues: poor dev experience, huge loss in productivity. It would be less bad if it was something that could be resolved unilaterally, but it often involves opening issues, forking, etc.

So in an ideal world, we could just opt out of unifying the dependency versions when needed. This is essentially what happens when we fork and patch, but its painstaking.

Regardless of how often developers run into this (we have very different experiences here), there should be a reasonable solution that doesn't require forking and opening github issues or upgrading and downgrading dependencies and compiling over and over to see what happens. The potential solutions right now all have heavy trade-offs and are quite time-consuming.

[epage]

Sure I take no issue with semver and putting pressure on authors to "conform" to it. Semver's cool and authors should adhere to it.

The issue here I'm raising (and what I assume is the topic at hand based on the issue title) is that there's no sane way to have two different versions of the same dependency in a workspace.

@jozanza the problem in your situation is that people are inherently not following semver because they are saying newer, compatible versions are not compatible. If they were, you wouldn't have a need to allow multiple versions.