deps unexpectedly rebuilt after running `cargo nextest run` differently

[davepacheco]

I noticed all this working on branch "dap/nexus-inventory-2". Commit 376a37b is sync'd up with "main" from this morning. From there, I ran a full test suite run that failed one test:

$ ptime -m cargo nextest run --no-fail-fast
info: experimental features enabled: setup-scripts
    Finished test [unoptimized + debuginfo] target(s) in 1.08s
    Starting 892 tests across 99 binaries (2 skipped)
       SETUP [      1/1] crdb-seed: cargo run -p crdb-seed
             [ 00:00:00] [                                                                                            ]   0/894:          Finished dev [unoptimized + debuginfo] target(s) in 0.53s
     Running `target/debug/crdb-seed`
Oct 30 20:25:03.262 INFO Using existing CRDB seed tarball: `/dangerzone/omicron_tmp/crdb-base-dap/3af8e1b0d6d99d1eb564ed4f736ee088e760afd006b09b3fb54a50ef98d608b1.tar`
  SETUP PASS [      1/1] crdb-seed: cargo run -p crdb-seed
        PASS [   0.083s] authz-macros::proc-macro/authz-macros test_authz_dump
...
------------
     Summary [ 403.723s] 892 tests run: 891 passed (3 slow, 1 leaky), 1 failed, 2 skipped
        FAIL [  18.704s] omicron-nexus::test_all integration_tests::schema::dbinit_equals_sum_of_all_up
error: test run failed

real     6:47.092637560
user  1:19:44.454769533
sys     14:08.574488939
trap       41.302586601
tflt        0.500335124
dflt        3.788205567
kflt        0.048879750
lock 365:06:39.566636121
slp  16:25:43.533571797
lat   1:03:17.653685989
stop    26:13.939275810

I then tried to fix this test, which changed these files: nexus/db-model/src/schema.rs, schema/crdb/dbinit.sql, schema/crdb/9.0.0/ (a whole tree).

Then I tried to rerun just the failing test. But it looks like a whole lot of stuff got rebuilt, including a bunch of deps from outside the workspace:

$ ptime -m cargo nextest run -p omicron-nexus integration_tests::schema::dbinit_equals_sum_of_all_up
info: experimental features enabled: setup-scripts
   Compiling serde_json v1.0.107
   Compiling schemars v0.8.13
   Compiling openapiv3 v1.0.3
   Compiling reqwest v0.11.20
   Compiling usdt-impl v0.3.5
   Compiling postgres-types v0.2.6
   Compiling slog-json v2.6.1
   Compiling olpc-cjson v0.1.3
   Compiling tinytemplate v1.2.1
   Compiling httptest v0.15.4
   Compiling slog-bunyan v2.4.0
   Compiling criterion v0.5.1
   Compiling tokio-postgres v0.7.10
   Compiling ipnetwork v0.20.0
   Compiling typify-impl v0.0.13 (https://github.com/oxidecomputer/typify#de16c423)
   Compiling omicron-passwords v0.1.0 (https://github.com/oxidecomputer/omicron?branch=main#3dcc8d2e)
   Compiling crucible-client-types v0.1.0 (https://github.com/oxidecomputer/crucible?rev=da534e73380f3cc53ca0de073e1ea862ae32109b#da534e73)
   Compiling usdt-attr-macro v0.3.5
   Compiling usdt-macro v0.3.5
   Compiling diesel v2.1.3
   Compiling progenitor-client v0.3.0 (https://github.com/oxidecomputer/progenitor?branch=main#5c941c0b)
   Compiling usdt v0.3.5
   Compiling omicron-zone-package v0.8.3
   Compiling dropshot v0.9.1-dev (https://github.com/oxidecomputer/dropshot?branch=main#fa728d07)
   Compiling slog-dtrace v0.2.3
   Compiling opte-api v0.1.0 (https://github.com/oxidecomputer/opte?rev=258a8b59902dd36fc7ee5425e6b1fb5fc80d4649#258a8b59)
   Compiling crucible-common v0.0.1 (https://github.com/oxidecomputer/crucible?rev=da534e73380f3cc53ca0de073e1ea862ae32109b#da534e73)
   Compiling steno v0.4.0
   Compiling propolis_types v0.0.0 (https://github.com/oxidecomputer/propolis?rev=4019eb10fc2f4ba9bf210d0461dc6292b68309c2#4019eb10)
   Compiling opte v0.1.0 (https://github.com/oxidecomputer/opte?rev=258a8b59902dd36fc7ee5425e6b1fb5fc80d4649#258a8b59)
   Compiling crucible-protocol v0.0.0 (https://github.com/oxidecomputer/crucible?rev=da534e73380f3cc53ca0de073e1ea862ae32109b#da534e73)
   Compiling oxide-vpc v0.1.0 (https://github.com/oxidecomputer/opte?rev=258a8b59902dd36fc7ee5425e6b1fb5fc80d4649#258a8b59)
   Compiling gateway-sp-comms v0.1.1 (https://github.com/oxidecomputer/management-gateway-service?rev=2739c18e80697aa6bc235c935176d14b4d757ee9#2739c18e)
   Compiling opte-ioctl v0.1.0 (https://github.com/oxidecomputer/opte?rev=258a8b59902dd36fc7ee5425e6b1fb5fc80d4649#258a8b59)
   Compiling tough v0.12.5
   Compiling openapi-lint v0.4.0 (https://github.com/oxidecomputer/openapi-lint?branch=main#bb69a3a4)
   Compiling typify-macro v0.0.13 (https://github.com/oxidecomputer/typify#de16c423)
   Compiling typify v0.0.13 (https://github.com/oxidecomputer/typify#de16c423)
   Compiling progenitor-impl v0.3.0 (https://github.com/oxidecomputer/progenitor?branch=main#5c941c0b)
   Compiling progenitor-macro v0.3.0 (https://github.com/oxidecomputer/progenitor?branch=main#5c941c0b)
   Compiling progenitor v0.3.0 (https://github.com/oxidecomputer/progenitor?branch=main#5c941c0b)
   Compiling omicron-common v0.1.0 (https://github.com/oxidecomputer/omicron?branch=main#3dcc8d2e)
   Compiling dns-service-client v0.1.0 (https://github.com/oxidecomputer/omicron?branch=main#3dcc8d2e)
   Compiling propolis-client v0.1.0 (https://github.com/oxidecomputer/propolis?rev=4019eb10fc2f4ba9bf210d0461dc6292b68309c2#4019eb10)
   Compiling crucible-agent-client v0.0.1 (https://github.com/oxidecomputer/crucible?rev=da534e73380f3cc53ca0de073e1ea862ae32109b#da534e73)
   Compiling crucible-pantry-client v0.0.1 (https://github.com/oxidecomputer/crucible?rev=da534e73380f3cc53ca0de073e1ea862ae32109b#da534e73)
   Compiling dpd-client v0.1.0 (/home/dap/omicron-inventory/clients/dpd-client)
   Compiling mg-admin-client v0.1.0 (/home/dap/omicron-inventory/clients/mg-admin-client)
   Compiling ddm-admin-client v0.1.0 (/home/dap/omicron-inventory/clients/ddm-admin-client)
   Compiling async-bb8-diesel v0.1.0 (https://github.com/oxidecomputer/async-bb8-diesel?rev=1446f7e0c1f05f33a0581abd51fa873c7652ab61#1446f7e0)
   Compiling diesel-dtrace v0.2.0 (https://github.com/oxidecomputer/diesel-dtrace?branch=main#c1252df7)
   Compiling nexus-client v0.1.0 (https://github.com/oxidecomputer/omicron?branch=main#3dcc8d2e)
   Compiling oximeter v0.1.0 (https://github.com/oxidecomputer/omicron?branch=main#3dcc8d2e)
   Compiling internal-dns v0.1.0 (https://github.com/oxidecomputer/omicron?branch=main#3dcc8d2e)
   Compiling omicron-workspace-hack v0.1.0 (/home/dap/omicron-inventory/workspace-hack)
   Compiling omicron-rpaths v0.1.0 (/home/dap/omicron-inventory/rpaths)
   Compiling omicron-passwords v0.1.0 (/home/dap/omicron-inventory/passwords)
   Compiling dns-service-client v0.1.0 (/home/dap/omicron-inventory/clients/dns-service-client)
   Compiling gateway-client v0.1.0 (/home/dap/omicron-inventory/clients/gateway-client)
   Compiling oxide-client v0.1.0 (/home/dap/omicron-inventory/clients/oxide-client)
   Compiling api_identity v0.1.0 (/home/dap/omicron-inventory/api_identity)
   Compiling oximeter-macro-impl v0.1.0 (/home/dap/omicron-inventory/oximeter/oximeter-macro-impl)
   Compiling db-macros v0.1.0 (/home/dap/omicron-inventory/nexus/db-macros)
   Compiling authz-macros v0.1.0 (/home/dap/omicron-inventory/nexus/authz-macros)
   Compiling nexus-test-utils-macros v0.1.0 (/home/dap/omicron-inventory/nexus/test-utils-macros)
   Compiling nexus-db-model v0.1.0 (/home/dap/omicron-inventory/nexus/db-model)
   Compiling nexus-db-queries v0.1.0 (/home/dap/omicron-inventory/nexus/db-queries)
   Compiling omicron-nexus v0.1.0 (/home/dap/omicron-inventory/nexus)
   Compiling bootstore v0.1.0 (/home/dap/omicron-inventory/bootstore)
   Compiling omicron-common v0.1.0 (/home/dap/omicron-inventory/common)
   Compiling dns-server v0.1.0 (/home/dap/omicron-inventory/dns-server)
   Compiling oximeter-producer v0.1.0 (https://github.com/oxidecomputer/omicron?branch=main#3dcc8d2e)
   Compiling crucible v0.0.1 (https://github.com/oxidecomputer/crucible?rev=da534e73380f3cc53ca0de073e1ea862ae32109b#da534e73)
   Compiling illumos-utils v0.1.0 (/home/dap/omicron-inventory/illumos-utils)
   Compiling nexus-client v0.1.0 (/home/dap/omicron-inventory/clients/nexus-client)
   Compiling key-manager v0.1.0 (/home/dap/omicron-inventory/key-manager)
   Compiling oximeter v0.1.0 (/home/dap/omicron-inventory/oximeter/oximeter)
   Compiling nexus-types v0.1.0 (/home/dap/omicron-inventory/nexus/types)
   Compiling sled-agent-client v0.1.0 (/home/dap/omicron-inventory/clients/sled-agent-client)
   Compiling internal-dns v0.1.0 (/home/dap/omicron-inventory/internal-dns)
   Compiling ipcc-key-value v0.1.0 (/home/dap/omicron-inventory/ipcc-key-value)
   Compiling nexus-defaults v0.1.0 (/home/dap/omicron-inventory/nexus/defaults)
   Compiling omicron-certificates v0.1.0 (/home/dap/omicron-inventory/certificates)
   Compiling oximeter-client v0.1.0 (/home/dap/omicron-inventory/clients/oximeter-client)
   Compiling propolis v0.1.0 (https://github.com/oxidecomputer/propolis?rev=4019eb10fc2f4ba9bf210d0461dc6292b68309c2#4019eb10)
   Compiling omicron-gateway v0.1.0 (/home/dap/omicron-inventory/gateway)
   Compiling oximeter-db v0.1.0 (/home/dap/omicron-inventory/oximeter/db)
   Compiling omicron-test-utils v0.1.0 (/home/dap/omicron-inventory/test-utils)
   Compiling oximeter-instruments v0.1.0 (/home/dap/omicron-inventory/oximeter/instruments)
   Compiling sled-hardware v0.1.0 (/home/dap/omicron-inventory/sled-hardware)
   Compiling oximeter-producer v0.1.0 (/home/dap/omicron-inventory/oximeter/producer)
   Compiling cpuid_profile_config v0.0.0 (https://github.com/oxidecomputer/propolis?rev=4019eb10fc2f4ba9bf210d0461dc6292b68309c2#4019eb10)
   Compiling propolis-server-config v0.0.0 (https://github.com/oxidecomputer/propolis?rev=4019eb10fc2f4ba9bf210d0461dc6292b68309c2#4019eb10)
   Compiling propolis-server v0.1.0 (https://github.com/oxidecomputer/propolis?rev=4019eb10fc2f4ba9bf210d0461dc6292b68309c2#4019eb10)
   Compiling bootstrap-agent-client v0.1.0 (/home/dap/omicron-inventory/clients/bootstrap-agent-client)
   Compiling sp-sim v0.1.0 (/home/dap/omicron-inventory/sp-sim)
   Compiling nexus-test-interface v0.1.0 (/home/dap/omicron-inventory/nexus/test-interface)
   Compiling oximeter-collector v0.1.0 (/home/dap/omicron-inventory/oximeter/collector)
   Compiling nexus-inventory v0.1.0 (/home/dap/omicron-inventory/nexus/inventory)
   Compiling gateway-test-utils v0.1.0 (/home/dap/omicron-inventory/gateway-test-utils)
   Compiling omicron-sled-agent v0.1.0 (/home/dap/omicron-inventory/sled-agent)
   Compiling nexus-test-utils v0.1.0 (/home/dap/omicron-inventory/nexus/test-utils)
    Finished test [unoptimized + debuginfo] target(s) in 12m 22s
    Starting 1 test across 4 binaries (340 skipped)
       SETUP [      1/1] crdb-seed: cargo run -p crdb-seed
             [ 00:00:00] [                                                                                            ]   0/341:         Compiling omicron-test-utils v0.1.0 (/home/dap/omicron-inventory/test-utils)
   Compiling crdb-seed v0.1.0 (/home/dap/omicron-inventory/dev-tools/crdb-seed)
    Finished dev [unoptimized + debuginfo] target(s) in 6.27s
     Running `target/debug/crdb-seed`
Oct 30 20:54:49.331 INFO cockroach temporary directory: /dangerzone/omicron_tmp/.tmpmWMw1W
Oct 30 20:54:49.331 INFO cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/dangerzone/omicron_tmp/crdb-base-dap/.tmp9slTAd,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /dangerzone/omicron_tmp/.tmpmWMw1W/listen-url
Oct 30 20:54:49.332 INFO cockroach environment: BASH_SILENCE_DEPRECATION_WARNING=1 BAT_THEME=ansi BUNYAN_NO_PAGER=1 CARGO=/home/dap/.rustup/toolchains/1.72.1-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/dap/.cargo CARGO_MANIFEST_DIR=/home/dap/omicron-inventory/dev-tools/crdb-seed CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=crdb-seed CARGO_PKG_README=README.md CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CSCOPEOPTIONS=-r -p8 EDITOR=vim GOTRACEBACK=crash HOME=/home/dap HOST=ivanova LANG=en_US.UTF-8 LD_LIBRARY_PATH=/home/dap/omicron-inventory/target/debug/build/bzip2-sys-73f44f7f2703292f/out/lib:/home/dap/omicron-inventory/target/debug/build/ring-5495af41434671b7/out:/home/dap/omicron-inventory/target/debug/build/ring-d5da8041c384a28b/out:/home/dap/omicron-inventory/target/debug/deps:/home/dap/omicron-inventory/target/debug:/home/dap/.rustup/toolchains/1.72.1-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/dap/omicron-inventory/target/debug/build/bzip2-sys-4e554edf952aff8e/out/lib:/home/dap/omicron-inventory/target/debug/build/libgit2-sys-620730d494c5e2b4/out/build:/home/dap/omicron-inventory/target/debug/build/ring-9a5207eba4153586/out:/home/dap/omicron-inventory/target/debug/build/ring-c211d5d00c4c9cb1/out:/home/dap/omicron-inventory/target/debug/build/tofino-07050b6a5db31350/out:/home/dap/omicron-inventory/target/debug/deps:/home/dap/omicron-inventory/target/debug:/home/dap/.rustup/toolchains/1.72.1-x86_64-unknown-illumos/lib LESS=-P ?f%f .line %lb/%L .byte %bB?s/%s. ?e(END):?pB%pB\%..%t LOGNAME=dap MACH=i386 MACHINE_THAT_GOES_PING=1 NEXTEST=1 NEXTEST_ENV=/dangerzone/omicron_tmp/nextest-envD7MBUv NEXTEST_LD_LIBRARY_PATH=/home/dap/omicron-inventory/target/debug/build/bzip2-sys-4e554edf952aff8e/out/lib:/home/dap/omicron-inventory/target/debug/build/libgit2-sys-620730d494c5e2b4/out/build:/home/dap/omicron-inventory/target/debug/build/ring-9a5207eba4153586/out:/home/dap/omicron-inventory/target/debug/build/ring-c211d5d00c4c9cb1/out:/home/dap/omicron-inventory/target/debug/build/tofino-07050b6a5db31350/out:/home/dap/omicron-inventory/target/debug/deps:/home/dap/omicron-inventory/target/debug:/home/dap/.rustup/toolchains/1.72.1-x86_64-unknown-illumos/lib NEXTEST_RUN_ID=8df2ac74-7d2e-4630-b2cf-66cc60fb291f OLDPWD=/home/dap PAGER=less -X PATH=/home/dap/omicron-inventory/out/mgd/root/opt/oxide/mgd/bin:/home/dap/omicron-inventory/out/dendrite-stub/bin:/home/dap/omicron-inventory/out/clickhouse:/home/dap/omicron-inventory/out/cockroachdb/bin:/home/dap/omicron-inventory/out/dendrite-stub/bin:/home/dap/omicron-inventory/out/clickhouse:/home/dap/omicron-inventory/out/cockroachdb/bin:/home/dap/.cargo/bin::/home/dap/bin:/home/dap/install/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/gnu/bin:/opt/ooce/bin:/home/dap/tools/cockroachdb/bin:/home/dap/tools/clickhouse PS1=\[\033]0;\w (\h)\007\]\u@ivanova \[\]\W\[\] $  PWD=/home/dap/omicron-inventory RAWPS1=\u@ivanova \[\]\W\[\] $  RUSTUP_HOME=/home/dap/.rustup RUSTUP_TOOLCHAIN=1.72.1-x86_64-unknown-illumos RUST_RECURSION_COUNT=2 SHELL=/bin/bash SHLVL=1 SSH_AUTH_SOCK=/tmp/ssh-XXXXV2aaFT/agent.23288 SSH_CLIENT=172.20.17.110 54042 22 SSH_CONNECTION=172.20.17.110 54042 172.20.2.70 22 SSH_TTY=/dev/pts/4 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TERM=xterm-256color TMPDIR=/dangerzone/omicron_tmp TZ=US/Pacific USER=dap VISUAL=vim _=/bin/ptime
Oct 30 20:54:49.685 INFO cockroach pid: 3866
Oct 30 20:54:49.685 INFO cockroach listen URL: postgresql://root@[::1]:56655/omicron?sslmode=disable
Oct 30 20:54:49.685 INFO cockroach: populating
Oct 30 20:54:50.620 INFO cockroach: populated
Oct 30 20:54:55.210 INFO Created CRDB seed tarball: `/dangerzone/omicron_tmp/crdb-base-dap/839a4a3e0f3eefa8978da64abad5c4e4c8b92841438e933548807cb809f5f2ad.tar`
  SETUP PASS [      1/1] crdb-seed: cargo run -p crdb-seed
        PASS [  15.303s] omicron-nexus::test_all integration_tests::schema::dbinit_equals_sum_of_all_up
------------
     Summary [  27.603s] 1 test run: 1 passed, 340 skipped

real    12:51.083406867
user    49:17.068366771
sys      5:07.239031904
trap        1.238737045
tflt        0.322437996
dflt        5.063060379
kflt        0.026255013
lock  2:06:34.034705988
slp   2:31:44.564452485
lat        30.308367863
stop       43.578906150

This was unexpected -- I thought hakari was supposed to prevent that. I wondered if mine was out of date. But I think not?

$ cargo hakari verify
info: omicron-workspace-hack works correctly
$ echo $?
0

[davepacheco]

One problem is that I hadn't run cargo hakari manage-deps after adding a new crate as part of Nexus:

dap@ivanova omicron-inventory $ cargo hakari manage-deps 
info: operations to perform:

* add or update dependency omicron-workspace-hack v0.1.0 (at path workspace-hack) to packages:
   - nexus-inventory (at path nexus/inventory)

✔ proceed? · yes
    Updating crates.io index
dap@ivanova omicron-inventory $ git diff
diff --git a/Cargo.lock b/Cargo.lock
index fba96d19e..eb56e84cf 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -4466,6 +4466,7 @@ dependencies = [
  "gateway-messages",
  "gateway-test-utils",
  "nexus-types",
+ "omicron-workspace-hack",
  "slog",
  "strum",
  "tokio",
diff --git a/nexus/inventory/Cargo.toml b/nexus/inventory/Cargo.toml
index 7fc360259..3ed7e8b2d 100644
--- a/nexus/inventory/Cargo.toml
+++ b/nexus/inventory/Cargo.toml
@@ -14,6 +14,7 @@ nexus-types.workspace = true
 slog.workspace = true
 strum.workspace = true
 uuid.workspace = true
+omicron-workspace-hack.workspace = true
 
 [dev-dependencies]
 expectorate.workspace = true

I don't know if that's enough to cause this.

[sunshowers]

I don't know if that's enough to cause this.

Doubt that to be honest -- I don't think it's enough to cause this.

[davepacheco]

I noticed a similar thing today, just in one cargo nextest run:

$ EXPECTORATE=overwrite cargo nextest run --no-fail-fast -E 'not (package(/nexus/))'
info: experimental features enabled: setup-scripts
   Compiling omicron-nexus v0.1.0 (/home/dap/omicron/nexus)
   Compiling nexus-db-queries v0.1.0 (/home/dap/omicron/nexus/db-queries)
   Compiling omicron-dev v0.1.0 (/home/dap/omicron/dev-tools/omicron-dev)
   Compiling omicron-omdb v0.1.0 (/home/dap/omicron/dev-tools/omdb)
    Finished test [unoptimized + debuginfo] target(s) in 6m 57s
    Starting 377 tests across 100 binaries (2 skipped)
       SETUP [      1/1] crdb-seed: cargo run -p crdb-seed
             [ 00:00:00] [                                                                                              ]   0/379:         Compiling serde_json v1.0.108
   Compiling headers-core v0.2.0
   Compiling atomicwrites v0.4.2
   Compiling rcgen v0.11.3
   Compiling headers v0.3.9
   Compiling schemars v0.8.13
   Compiling usdt-impl v0.3.5
   Compiling openapiv3 v1.0.3
   Compiling postgres-types v0.2.6
   Compiling reqwest v0.11.20
   Compiling slog-json v2.6.1
   Compiling slog-bunyan v2.4.0
   Compiling tokio-postgres v0.7.10
   Compiling usdt-attr-macro v0.3.5
   Compiling usdt-macro v0.3.5
   Compiling typify-impl v0.0.13 (https://github.com/oxidecomputer/typify#de16c423)
   Compiling ipnetwork v0.20.0
   Compiling diesel v2.1.3
   Compiling usdt v0.3.5
   Compiling progenitor-client v0.3.0 (https://github.com/oxidecomputer/progenitor?branch=main#5c941c0b)
   Compiling dropshot v0.9.1-dev (https://github.com/oxidecomputer/dropshot?branch=main#fa728d07)
   Compiling typify-macro v0.0.13 (https://github.com/oxidecomputer/typify#de16c423)
   Compiling typify v0.0.13 (https://github.com/oxidecomputer/typify#de16c423)
   Compiling progenitor-impl v0.3.0 (https://github.com/oxidecomputer/progenitor?branch=main#5c941c0b)
   Compiling progenitor-macro v0.3.0 (https://github.com/oxidecomputer/progenitor?branch=main#5c941c0b)
   Compiling progenitor v0.3.0 (https://github.com/oxidecomputer/progenitor?branch=main#5c941c0b)
   Compiling omicron-workspace-hack v0.1.0 (/home/dap/omicron/workspace-hack)
   Compiling api_identity v0.1.0 (/home/dap/omicron/api_identity)
   Compiling omicron-common v0.1.0 (/home/dap/omicron/common)
   Compiling omicron-test-utils v0.1.0 (/home/dap/omicron/test-utils)
   Compiling crdb-seed v0.1.0 (/home/dap/omicron/dev-tools/crdb-seed)
    Finished dev [unoptimized + debuginfo] target(s) in 38.39s
     Running `target/debug/crdb-seed`

I would have expected hakari to make it so that once we'd built enough for cargo nextest to get going, we wouldn't then spend 38s building more dependencies to run crdb-seed.

[sunshowers]

Investigation results

OK, I spent some time looking at this and there are two big things that pop out:

1. Excluding xtask from hakari

We exclude the xtask package from hakari but include it in workspace.default-members. This causes serde_json in particular to be built with a different set of features, which has ripple effects up the build graph. Fix forthcoming (will exclude xtask from default-members as well).

2. Host and target deps built with separate options

One of the promises hakari makes is to be able to unify builds across host (build) and target (normal/dev) deps. This sadly doesn't work for us at all. That's because we build target (normal/dev) deps and host (build) deps with separate sets of options:

1. Target deps are built with debug = "line-tables-only", and host deps are built with debug = "false". This can be unified relatively easily.
2. Target deps are built with panic = "abort", while host deps are built with panic = "unwind". This can be addressed in one of two ways:
   1. Build host deps with panic = "abort". I don't think this is advisable -- it is useful to get backtraces and such from host deps.
   2. For the dev profile, build target deps with panic = "unwind" (we can stay with abort for the release profile). This I think makes more sense but would cause behavior differences between dev and release builds.

Either of the two options above would make a difference for compile times. The number of units (things to build, what shows up in the progress bar in cargo build) decreases massively. For nexus, with diff:

Expand for diff

diff --git a/Cargo.toml b/Cargo.toml
index b18b20aec..291a1a56d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -392,12 +392,16 @@ zip = { version = "0.6.6", default-features = false, features = ["deflate","bzip
 zone = { version = "0.3", default-features = false, features = ["async"] }
 
 [profile.dev]
-panic = "abort"
 # See https://github.com/oxidecomputer/omicron/issues/4009 for some background context here.
 # By reducing the debug level (though keeping enough to have meaningful
 # backtraces), we reduce incremental build time and binary size significantly.
 debug = "line-tables-only"
 
+# Setting build-override with line-tables-only means that we don't build two
+# copies of dependencies shared at build and runtime
+[profile.dev.build-override]
+debug = "line-tables-only"
+
 # `bindgen` is used by `samael`'s build script; building it with optimizations
 # makes that build script run ~5x faster, more than offsetting the additional
 # build time added to `bindgen` itself.

With the command cat tmp/nexus-unit-graph.json | jq -r '.units | length' and running cargo test --no-run -p omicron-nexus, I got:

Before: 1230 units (2m 25s fresh build)
After: 838 (2m 15s fresh build)

This is a pretty decent difference.

As a third option, we could just accept that we're going to have different builds on the host and target platforms forever, and tell hakari to not try and unify dependencies across the platforms.

How I investigated

I used the --unit-graph option:

export RUSTC_BOOTSTRAP=1
cargo test --no-run -Z unstable-options --unit-graph | jq > tmp/cargo-test-unit-graph.json
cargo test --no-run -p omicron-nexus -Z unstable-options --unit-graph | jq > tmp/nexus-unit-graph.json

Then, I investigated the two outputs, starting from serde_json since that showed up in a few example outputs. The differences were clearly apparent, and from there I started hammering down the details and iterating.

[sunshowers]

OK, so the crdb-seed issue is also interesting: what's happening there is that tests are always built with panic = unwind whether or not profile.dev.panic is abort (which makes sense for tests), but cargo run uses profile.dev.panic. This means that it would be best for dependency reuse that we use unwind for dev builds.

[rmustacc]

2. Build host deps with panic = "abort". I don't think this is advisable -- it is useful to get backtraces and such from host deps.

Is the assumption here that the reason we don't get the backtrace is because the generation of a core dump which would have such a backtrace is disabled?

[sunshowers]

Is the assumption here that the reason we don't get the backtrace is because the generation of a core dump which would have such a backtrace is disabled?

Ah I was thinking in terms of backtraces printed by Rust itself, e.g. by running RUST_BACKTRACE=1 which is pretty common while writing tests. We'd separately get a backtrace from core dumps if they're enabled.

[sunshowers]

All of the cases above should now be handled with #4535. Thanks again for all your patience.