Failed to write registry URLS and feature map

[ericmcbride]

I made a commit in the last release to make sure CARGO_BAZEL_ISOLATED=false to be set when running bazel for privately hosted crates repositories and everything worked great locally when i ran it.

However with the 0.58.0 release im seeing the following

➜  my-repo git:(master) ✗ CARGO_DEBUG_TRACE=true CARGO_BAZEL_ISOLATED=false bazel build //my_crate
ERROR: /private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/external/rules_rust+/crate_universe/private/common_utils.bzl:55:13: Traceback (most recent call last):
	File "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/external/rules_rust+/crate_universe/extensions.bzl", line 998, column 37, in _crate_impl
		_generate_hub_and_spokes(
	File "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/external/rules_rust+/crate_universe/extensions.bzl", line 633, column 51, in _generate_hub_and_spokes
		splice_outputs = splice_workspace_manifest(
	File "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/external/rules_rust+/crate_universe/private/splicing_utils.bzl", line 174, column 19, in splice_workspace_manifest
		cargo_bazel_fn(
	File "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/external/rules_rust+/crate_universe/private/common_utils.bzl", line 88, column 23, in _execute
		return execute(
	File "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/external/rules_rust+/crate_universe/private/common_utils.bzl", line 55, column 13, in execute
		fail(_EXECUTE_ERROR_MESSAGE.format(
Error in fail: Command ["/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/modextwd/rules_rust++crate/cargo-bazel", "splice", "--output-dir", "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/modextwd/rules_rust++crate/crates_index/splicing-output", "--splicing-manifest", "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/modextwd/rules_rust++crate/crates_index/splicing_manifest.json", "--config", "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/modextwd/rules_rust++crate/crates_index/config.json", "--cargo-lockfile", "/Users/eric.mcbride/Programming/my-repo/Cargo.lock", "--cargo", "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/external/rules_rust++rust_host_tools+rust_host_tools/bin/cargo", "--rustc", "/private/var/tmp/_bazel_eric.mcbride/79b277eea1629c6c0e66b29d7b082943/external/rules_rust++rust_host_tools+rust_host_tools/bin/rustc"] failed with exit code 1.
STDOUT ------------------------------------------------------------------------

STDERR ------------------------------------------------------------------------
Error: Failed to write registry URLs and feature map

Caused by:
    0: Failed to get crate index config
    1: No such file or directory (os error 2)

[UebelAndre]

I suspect this is related to #3251

The No such file or directory is probably due to crate_universe (by way of the crates-index crate) looking for a local crate index in the wrong spot due to an incorrect or outdated hash implementation that determines what the directory would be named. These would look like

~/.cargo/registry/index
├── github.com-1ecc6299db9ec823
│   └── *
├── index.crates.io-1949cf8c6b5b557f
│   └── config.json
└── index.crates.io-6f17d22bba15001f
    └── config.json

Where one of index.crates.io is before Rust 1.85.0 and the other is after. My immediate suspect is the hashing I implemented in frewsxcv/rust-crates-index#184 is not quite correct (though I definitely tried!). Are you using a custom index url? Can you double check the hash from crates-index matches what Cargo produces on your machine? The tests here are probably the best way to check that:
https://github.com/frewsxcv/rust-crates-index/blob/857c063bf6a20a614d18947df4420e221d0c3c00/src/dirs.rs#L253-L322

[ericmcbride]

A followup to this @UebelAndre is if i set CARGO_BAZEL_ISOLATED=true it gets past that part no issue but then it cant pull the private crates repository (Other issue I had opened)

[UebelAndre]

Is there some config file in CARGO_HOME that’s used for auth? Are there credentials in the config.toml file or elsewhere?

[ericmcbride]

By default rust uses a credentials.toml file in the .cargo file path of a project to use then will fall back to ~/.cargo/credentials.toml. This is why CARGO_BAZEL_ISOLATED=true fails. Ive suggested adding that to the from_cargo method similar to the cargo_config option.

[UebelAndre]

I think it would be fine to allow credentials to be copied into the isolated CARGO_HOME. The reason I added isolation was to avoid a user’s config.toml from generating things no one else could. If you were interested in making that change 😅

[UebelAndre]

But I still suspect this is a mismatch in the hashing logic

[ericmcbride]

I can confirm downgrading crates-index dependency to 3.5 resolves my issue.

[UebelAndre]

I can confirm downgrading crates-index dependency to 3.5 resolves my issue.

That means the implementations is wrong for some kind of index 😞

[ericmcbride]

Looks like SparseRegistry. All the Registry kinds are working. Im cleansing my log file to remove proprietary information so give me a second and ill show you whats failing

[ericmcbride]

Splice 2025-03-05T17:02:56.214605Z  INFO write_registry_urls_and_feature_map: cargo_bazel::splicing::crate_index_lookup: ==== Cargo Package is Package {
...
}

Splice 2025-03-05T17:02:56.214605Z  INFO write_registry_urls_and_feature_map: cargo_bazel::splicing::crate_index_lookup: ==== Cargo Package is Package {
    name: Name("my_package"),
    version: Version {
        major: 1,
        minor: 3,
        patch: 366,
    },
    source: Some(SourceId {
        url: Url {
            scheme: "https",
            cannot_be_a_base: false,
            username: "",
            password: None,
            host: Some(Domain("my-artifactory")),
            port: None,
            path: "/artifactory/api/cargo/my-indexl/index/",
            query: None,
            fragment: None,
        },
        kind: SparseRegistry,
        precise: Some("locked"),
        name: None,
    }),
    checksum: Some(Sha256(9694cfb75c85b771f99d33bca3f2795aa88e36852dc86b371cf5016b1015b34b)),
    dependencies: [
        Dependency {
            name: Name("capnp"),
            version: Version {
                major: 0,
                minor: 17,
                patch: 2,
            },
            source: Some(SourceId {
                url: Url {
                    scheme: "https",
                    cannot_be_a_base: false,
                    username: "",
                    password: None,
                    host: Some(Domain("github.com")),
                    port: None,
                    path: "/rust-lang/crates.io-index",
                    query: None,
                    fragment: None,
                },
                kind: Registry,
                precise: Some("locked"),
                name: None,
            }),
        },
        Dependency {
            name: Name("capnp-rpc"),
            version: Version {
                major: 0,
                minor: 17,
                patch: 0,
            },
            source: Some(SourceId {
                url: Url {
                    scheme: "https",
                    cannot_be_a_base: false,
                    username: "",
                    password: None,
                    host: Some(Domain("github.com")),
                    port: None,
                    path: "/rust-lang/crates.io-index",
                    query: None,
                    fragment: None,
                },
                kind: Registry,
                precise: Some("locked"),
                name: None,
            }),
        },
        Dependency {
            name: Name("futures"),
            version: Version {
                major: 0,
                minor: 3,
                patch: 31,
            },
            source: Some(SourceId {
                url: Url {
                    scheme: "https",
                    cannot_be_a_base: false,
                    username: "",
                    password: None,
                    host: Some(Domain("github.com")),
                    port: None,
                    path: "/rust-lang/crates.io-index",
                    query: None,
                    fragment: None,
                },
                kind: Registry,
                precise: Some("locked"),
                name: None,
            }),
        },
        Dependency {
            name: Name("once_cell"),
            version: Version {
                major: 1,
                minor: 20,
                patch: 2,
            },
            source: Some(SourceId {
                url: Url {
                    scheme: "https",
                    cannot_be_a_base: false,
                    username: "",
                    password: None,
                    host: Some(Domain("github.com")),
                    port: None,
                    path: "/rust-lang/crates.io-index",
                    query: None,
                    fragment: None,
                },
                kind: Registry,
                precise: Some("locked"),
                name: None,
            }),
        },
        Dependency {
            name: Name("tokio"),
            version: Version {
                major: 1,
                minor: 42,
                patch: 0,
            },
            source: Some(SourceId {
                url: Url {
                    scheme: "https",
                    cannot_be_a_base: false,
                    username: "",
                    password: None,
                    host: Some(Domain("github.com")),
                    port: None,
                    path: "/rust-lang/crates.io-index",
                    query: None,
                    fragment: None,
                },
                kind: Registry,
                precise: Some("locked"),
                name: None,
            }),
        },
        Dependency {
            name: Name("tokio-stream"),
            version: Version {
                major: 0,
                minor: 1,
                patch: 17,
            },
            source: Some(SourceId {
                url: Url {
                    scheme: "https",
                    cannot_be_a_base: false,
                    username: "",
                    password: None,
                    host: Some(Domain("github.com")),
                    port: None,
                    path: "/rust-lang/crates.io-index",
                    query: None,
                    fragment: None,
                },
                kind: Registry,
                precise: Some("locked"),
                name: None,
            }),
        },
        Dependency {
            name: Name("tokio-util"),
            version: Version {
                major: 0,
                minor: 7,
                patch: 13,
            },
            source: Some(SourceId {
                url: Url {
                    scheme: "https",
                    cannot_be_a_base: false,
                    username: "",
                    password: None,
                    host: Some(Domain("github.com")),
                    port: None,
                    path: "/rust-lang/crates.io-index",
                    query: None,
                    fragment: None,
                },
                kind: Registry,
                precise: Some("locked"),
                name: None,
            }),
        },
        Dependency {
            name: Name("tracing"),
            version: Version {
                major: 0,
                minor: 1,
                patch: 41,
            },
            source: Some(SourceId {
                url: Url {
                    scheme: "https",
                    cannot_be_a_base: false,
                    username: "",
                    password: None,
                    host: Some(Domain("github.com")),
                    port: None,
                    path: "/rust-lang/crates.io-index",
                    query: None,
                    fragment: None,
                },
                kind: Registry,
                precise: Some("locked"),
                name: None,
            }),
        },
    ],
    replace: None,
}

[ericmcbride]

I did a cargo clean, ran cargo check (rust 1.78) and got the top my-artifactory dir in ~/.cargo/registry
I ran CARGO_DEBUG_TRACE=true CARGO_BAZEL_ISOLATED=false RUST_LOG=trace bazel build //my_crate and it generated the second if this helps

my-artifactory-5620b01d2f62b397 <- cargo 1.78
my-artifactory-2ad9854f03835c2e <- rules_rust 0.58.0 / 1.78

So maybe its running the new hash algorithm even on old versions of rust?

[UebelAndre]

I don't think rust 1.78 is actually being used. #3286 by default all crate_universe modules will use the rust_host_tools repository defined within rules_rust which is gonna be Rust 1.85.0

rules_rust/rust/extensions.bzl

Lines 14 to 17 in 13c596e

	_RUST_TOOLCHAIN_VERSIONS = [
	rust_common.default_version,
	DEFAULT_NIGHTLY_VERSION,
	]

rules_rust/rust/private/common.bzl

Line 34 in 13c596e

DEFAULT_RUST_VERSION = "1.85.0"

[ericmcbride]

I just updated to 1.85 and blew out the ~/.cargo/registries and ran cargo and then rules_rust / bazel, and the good thing is it didnt create another my-artifactory directory. Bad thing is i still get the same error making me think the hash is correct and maybe something else is going awry.