Rewrite the build infrastructure

[aaronmondal]

Our previous build layout was fairly custom and hard to maintain.
Migrate to a more standards-conform layout that is compatible with good
Cargo and Bazel practices and feels natural to develop in.

This commit swaps the previous "one-crate-per-file" layout to a layout
where we have a few medium sized crates that are just fine-grained
enough to allow almost the same amount of parallelization as the
previous fine-grained layout.

The new layout enables dependency auto-updaters, standard cargo tooling
and doc autogeneration. This should improve DX quite a bit and gives us
some much needed compatibility with nix/cargo interop and should make
most standard cargo extensions compatible with the repository.

Some notable changes to the file layout:

• The previous error crate is no longer part of utils. It is now a
  standalone module. This was necessary to keep macro exports compatible
  with the Cargo build.
• The old scheduler.rs file is now split into action_scheduler.rs
  and worker_scheduler.rs since we don't want imports like
  native_link_scheduler::scheduler.
• Some tests now use a slightly different import mechanism for test-only
  dependencies.
• The store traits are now in native_link_utils, as it was used
  virtually everywhere. Moving this up in the dependency graph allows
  for higher parallelization of the other crates.

Added documentation targets for every crate (except the cas binary
itself, as this will require a dedicated commit). These doc targets can
be viewed by pointing a browser to
bazel-bin/thetarget.rustdoc/thetarget/index.html.

Added rustfmt configuration to standardize imports. We now use crate::
syntax which plays well with rusfmts's import formatting. The new layout
makes it easier to spot crate/directory interdependencies so the we can
remove those in future commits.

---

This change is 

[allada]

Could you update this PR and lets get it landed?

Reviewable status: 0 of 202 files reviewed, all discussions resolved (waiting on @blakehatch and @MarcusSorealheis)

[allada]

Reviewed 98 of 202 files at r1, 13 of 108 files at r2, all commit messages.
Reviewable status: 111 of 213 files reviewed, 9 unresolved discussions (waiting on @aaronmondal, @blakehatch, and @MarcusSorealheis)

---

BUILD.bazel line 11 at r2 (raw file):

exports_files(
    glob(["src/**", "tests/**"]) + [

nit: Lets list every file.

---

BUILD.bazel line 22 at r2 (raw file):

    name = "native_link",
    srcs = glob(
        include = ["src/**/*.rs"],

nit: Lets list every file.

---

BUILD.bazel line 100 at r2 (raw file):

rust_test_suite(
    name = "native_link_tests",
    srcs = glob(["tests/*.rs"]),

nit: Lets list every file.

---

Cargo.toml line 7 at r2 (raw file):

[workspace]
resolver = "2"

Do we need this now?

---

src/config.rs line 1 at r2 (raw file):

pub mod cas_server;

nit: Needs copyright.

---

src/grpc_service.rs line 1 at r2 (raw file):

pub mod ac_server;

nit: Needs copyright.

---

src/lib.rs line 1 at r2 (raw file):

pub mod config;

nit: Needs copyright.

---

src/scheduler.rs line 1 at r2 (raw file):

pub mod action_messages;

nit: Needs copyright.

---

src/store.rs line 15 at r2 (raw file):

// limitations under the License.

pub mod ac_utils;

yuck, can we somehow use crate:: instead?

[allada]

Reviewable status: 111 of 213 files reviewed, 10 unresolved discussions (waiting on @aaronmondal, @blakehatch, and @MarcusSorealheis)

---

BUILD.bazel line 74 at r2 (raw file):

rust_binary(
    name = "nlink",

Lets not rename this yet, we will do it in a follow up PR. Because there's things like terraform that will likely break.

[aaronmondal]

Reviewable status: 104 of 214 files reviewed, 3 unresolved discussions (waiting on @allada, @blakehatch, and @MarcusSorealheis)

---

BUILD.bazel line 74 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

Lets not rename this yet, we will do it in a follow up PR. Because there's things like terraform that will likely break.

Done.

---

Cargo.toml line 7 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

Do we need this now?

Unfortunately yes. Both the Bazel resolver and nix cargo wrappers need this workspace for some reason. The resolver = "2" line is technically redundant and we could remove that. I think there was some issue I can't remember where it was recommended to make the resover version explicit.

---

src/store.rs line 15 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

yuck, can we somehow use crate:: instead?

I removed the rest of the actual code from here and put it in a store_manager subcrate. The amount of files here is a bit much, but I'd say this is more of a "not hiding anything" type of thing. Arguably quite ugly, but this gives us very explicit import paths that (in conjunction with the now ordered imports in every file) make it much easier to see where we could reduce interdependencies and split this into sensible smaller crates.

There are a few alternatives to the thing.rs/thing directory approach, but this seems to be the current "recommended" way of doing things and it's at least better than the mod.rs approach where literally every file in the repo would just be called mod.rs 🤣

As a side note, a benefit of this explicit pub mod approach is that it makes it quite easy to turn features on and off by using markers like #[cfg(feature = "s3")].

[aaronmondal]

Reviewable status: 104 of 214 files reviewed, 3 unresolved discussions (waiting on @allada, @blakehatch, and @MarcusSorealheis)

---

src/store.rs line 15 at r2 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

I removed the rest of the actual code from here and put it in a store_manager subcrate. The amount of files here is a bit much, but I'd say this is more of a "not hiding anything" type of thing. Arguably quite ugly, but this gives us very explicit import paths that (in conjunction with the now ordered imports in every file) make it much easier to see where we could reduce interdependencies and split this into sensible smaller crates.

There are a few alternatives to the thing.rs/thing directory approach, but this seems to be the current "recommended" way of doing things and it's at least better than the mod.rs approach where literally every file in the repo would just be called mod.rs 🤣

As a side note, a benefit of this explicit pub mod approach is that it makes it quite easy to turn features on and off by using markers like #[cfg(feature = "s3")].

Actually, when I think about it, this pattern might be better than I initially thought. With the right code layout,thing.rs can act as the public API interface (think exposed header in C++ land) while all the files in the thing directory act as implementation details. In that case we'd not have the pub mod lines anymore and instead have a seemingly very clean interface where thing.rs contains the entire public API for a crate and everything else can be ignored by users. This would also make it easy to spot API changes since one only needs to look at history of the thing.rs file.

[aaronmondal]

FYI @allada In case you're wondering why the new targets are native-link-something and not native_link_somethin as is usually done in Bazel, this is because the dashes make it easier to distinguish build targets and crate names when grepping and it allows shell completion for targets like bazel build native-<TAB>

Also note that all the test targets are just called integration because the rust_test_suite automatically appends the _tests string and it looks nicer in the terminal this way.

[allada]

All the details we can fight another day.

Reviewed 4 of 202 files at r1, 2 of 108 files at r2, 1 of 1 files at r4, 119 of 119 files at r5, 12 of 12 files at r6, all commit messages.
Reviewable status: all files reviewed, 22 unresolved discussions (waiting on @aaronmondal, @blakehatch, and @MarcusSorealheis)

---

native-link-service/tests/ac_server_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-service/tests/bytestream_server_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-service/tests/cas_server_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/ac_utils_test.rs line 1 at r6 (raw file):

// Copyright 2023 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/compression_store_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/dedup_store_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/fast_slow_store_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/filesystem_store_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/memory_store_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/ref_store_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/shard_store_test.rs line 1 at r6 (raw file):

// Copyright 2023 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/size_partitioning_store_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-store/tests/verify_store_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-util/tests/async_fixed_buffer_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-util/tests/buf_channel_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-util/tests/evicting_map_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-util/tests/fastcdc_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-util/tests/fs_test.rs line 1 at r6 (raw file):

// Copyright 2023 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-util/tests/resource_info_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-util/tests/retry_test.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-worker/tests/utils/local_worker_test_utils.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

---

native-link-worker/tests/utils/mock_running_actions_manager.rs line 1 at r6 (raw file):

// Copyright 2022 The Turbo Cache Authors. All rights reserved.

nit: Wrong name.

[aaronmondal]

🥳

Reviewable status: 161 of 239 files reviewed, all discussions resolved (waiting on @allada, @blakehatch, and @MarcusSorealheis)