Improve cargo splicing and lock file generation in large projects

[hofbi]

In #3467, I made some experiments about when cargo splicing is called and which lock file can help to reduce this.

New Problem

Following the recommended approach of using a MODULE.bazel.lock on a large scale rust project shows a new issue. The MODULE.bazel.lock file stores the shasum of the Cargo.lock file because it is used in the following setup:

crate.from_cargo(
    cargo_lockfile = "//:Cargo.lock",
    manifests = [
        "//:Cargo.toml",
        # many more other Cargo.toml files
    ],
    # ...
)

This makes the MODULE.bazel.lock file practically unusable due to a high number of merge conflicts.
Any time a developer changes a dependency between internal Cargo.toml files, the Cargo.lock file is changed causing the shasum in the MODULE.bazel.lock file to change.

The proposed solution

Based on a discussion with @illicitonion, we identified that rules_rust does not really care about the internal cargo dependencies, but only about the external/third party dependencies.
For this, we wanted to introduce an intermediate lockfile that only contains the external dependencies.

crate.from_cargo(
    cargo_lockfile = "//:Cargo.bazel.lock",  # no internal deps anymore, just third party
    manifests = [
        "//:Cargo.toml",
        # many more other Cargo.toml files
    ],
    # ...
)

The Cargo.bazel.lock file is generated from the Cargo.lock file by removing all internal dependencies and only keeping the external dependencies.
Having only the external dependencies in the Cargo.bazel.lock file allows to avoid merge conflicts when changing internal dependencies.
The shasum of the Cargo.bazel.lock file in the MODULE.bazel.lock will only change if the external dependencies change.
This happens usually on a much smaller frequency than the internal dependencies.

In essence, this can be done with

cargo_lock_file = parse(Path("Cargo.lock").read_text())
cargo_lock_file["package"] = [
    package for package in cargo_lock_file["package"] if "source" in package
]
Path("Cargo.bazel.lock").write_text(dumps(cargo_lock_file))

Observation from the new solution

Trying this new solution did not work as expected.
After passing the Cargo.bazel.lock file to crate.from_cargo, splicing runs as expected.
However, rules_rust fully restores the content of the Cargo.lock file into the Cargo.bazel.lock file.

[illicitonion]

I can see a couple of reasonable things for us to do here...

The simplest would be to add an option for crate_universe to simply not overwrite the lockfile (effectively a "I manage my lockfile separately, trust me" option). This is a bit weird, as we wouldn't really have a way to error if rewriting was important, but "I only ever update via cargo not via bazel" feels like a reasonable workflow to support.

Possibly a more near or more messy solution could be that we could allow you to pass a Cargo.lock preprocessor to crate_universe, so that it would pass the Cargo.lock through it before writing it back to the repo. This would need to be something not built with Bazel (because repository rules run before actions), which may be a limitation that makes this impractical - it could be a downloaded file, a checked in file, or something on $PATH, but couldn't require building.

All of this is a little odd, but this feels like something worth experimenting with - working out how we can make lockfiles (both in rules_rust, and in Bazel proper) more large-repo friendly will be useful. We may want to avoid signing up to long-term API support guarantees for whatever attribute(s) we add while we try this out, but getting real-world experience with the experiments would be useful!

WDYT @hofbi?

[hofbi]

Maybe the best option to start is to add a "I manage my lockfile separately, trust me" flag.