fix references to git depencies in workspaces with inherited configuration #393
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It has a fairly narrow range of libgit2 versions it will accept from the system environment, so instead of manually providing a recent-enough version from nixpkgs-unstable, or downgrading the rust crate until the available version matches, we allow it to build its preferred version.
When copying out workspace member crates from a larger workspace, cargo will not be able to process the separated manifest if it still refers to e.g. the workspace license or workspace dependencies, because it can't find the workspace manifest. We can't help it find the workspace manifest, because crates in a cargo vendor directory aren't discovered recursively, and cargo doesn't resolve symlinks before checking the ancestors to find the workspace manifest.
This requires bumping the nodejs version used to build docs because nodejs_21 does not exist in the latest nixpkgs.
Open
|
I'm trying out the main branch of your fork and I'm getting the error below, any chance you know why thats happening? Full log |
|
Yeah, let's see if we can get that figured out. I don't see immediately what the problem is. But I do see that the error is coming from the inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
crate2nix = {
url = "github:hallettj/crate2nix";
inputs.nixpkgs.follows = "nixpkgs";
};
};Crate2nix relies on |
|
I tried overriding cargo with a newer version by adding the commit from this PR to my branch: #390. I then get the error below. I think cargo isn't accepting the vendored source and tries to download it again. I get the same error when I override the flake input as you suggested. Log |
|
@Pacman99 Oh damn - that might be due to Cargo changing its internal package ID format as I discussed in #381. Crate2nix uses package IDs, but currently assumes the old format. I suppose the fix is to update crate2nix to a recent version of cargo, and change any crate2nix code that generates package IDs to use the new format. In the meantime if you're using IFD, maybe you can get things working by switching to code generation mode? By that I mean generating a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a follow-up to the great groundwork already established by @tilpner in #298. I've merged the branch from that PR with the latest crate2nix code, and made some more fixes.
This PR depends on #382 - the number of changes here should go down after that PR is merged.
The problem comes up with dependency crates that are in workspace repos, and that have manifests that "inherit" configuration from the workspace manifest. For example a crate manifest might look something like this,
This is all fine for dependencies in registries (such as crates.io) because part of the process of publishing a crate to a registry is to rewrite the crate's
Cargo.tomlto inline all inherited configuration. But when crate2nix references git dependencies it doesn't apply that process. Instead it copies the crate subdirectory as-is into its own derivation. When that derivation builds, and a manifest parser attempts to look up workspace values it fails because the workspace manifest is not present in the derivation source. So the dependency fails to build.This PR reproduces some of the crate publication processing. Most of the new logic is in
normalize_manifest.rs. During the process of creating each dependency derivation theCargo.tomlfile is parsed, workspace configuration is inlined, and a newCargo.tomlfile is written. (The original file is preserved in the derivation source - it is renamed toCargo.toml.orig.)I chose to use a library called
cargo_tomlto do this processing because it is narrowly focused on this type of task, and as a result it is a small dependency. Butcargo_tomlis not perfect, and it doesn't inline every possible inherited configuration. In particular I found that it lets inherited lint settings go through unchanged. To prevent build failures in such situations there is a fixup step after normalization that deletes any workspace references that weren't inlined. This assumes that there isn't much reason to reference lint configuration in dependency derivations, and I think there is not.An alternative would be to use
cargo(the library) for normalization. It's the library that the cargo program uses so its behavior will exactly match what is generally expected.cargocould also replace other functionality that crate2nix requires - in particular it could replace use ofcargo_metadata. I'll note that cargo2nix usescargo. I didn't usecargoin this PR because it's a very big dependency, and incorporating it would be a larger change.If you have been waiting for this fix, and you don't want to wait on this PR to be merged, I have included this fix and a number of other fixes in the
mainbranch of my fork, https://github.com/hallettj/crate2nix