cockroach starter could better report early termination

[davepacheco]

While debugging #1144, I found it would have been helpful if the test suite had reported exactly how cockroach had exited. This change adds more detail.

[davepacheco]

For reference, I applied this patch to cause cockroach to exit early and see how it would improve the output:

dap@ivanova omicron-fixes $ git diff
diff --git a/test-utils/src/dev/db.rs b/test-utils/src/dev/db.rs
index 2702d56b..d396d143 100644
--- a/test-utils/src/dev/db.rs
+++ b/test-utils/src/dev/db.rs
@@ -173,6 +173,9 @@ impl CockroachStarterBuilder {
         // directory rather than a random one.  That way, we can warn the user
         // if they start up two of them, and we can also clean up after unclean
         // shutdowns.
+        if self.store_dir.is_none() {
+            self.arg("bogus");
+        }
         let temp_dir =
             tempdir().with_context(|| "creating temporary directory")?;
         let store_dir = self

and I got this error:

test integration_tests::disks::test_disk_create_disk_that_already_exists_fails ... FAILED

failures:

---- integration_tests::disks::test_disk_create_disk_that_already_exists_fails stdout ----
log file: "/dangerzone/omicron_tmp/test_all-b48593a0eb0ea9a0-test_disk_create_disk_that_already_exists_fails.4953.0.log"
note: configured to log to "/dangerzone/omicron_tmp/test_all-b48593a0eb0ea9a0-test_disk_create_disk_that_already_exists_fails.4953.0.log"
thread 'integration_tests::disks::test_disk_create_disk_that_already_exists_fails' panicked at 'called `Result::unwrap()` on an `Err` value: Exited { exit_code: 1 }', /home/dap/omicron-fixes/test-utils/src/dev/mod.rs:142:42
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

which is much better than what we saw in #1144 because it explicitly says that we exited with status 1.

[jgallagher]

LGTM! 👍

[jgallagher]

It looks like tokio ultimately calls try_wait from std, which calls waitpid with WNOHANG. Per the manpage of waitpid (at least on illumos), it doesn't seem possible for any of the error cases to trigger (absent some weird bug where the process doesn't have the right pid or something like that), at least assuming we can't get EINTR if we passed WNOHANG (which I'm not sure about?).

[davepacheco]

Yeah, I don't remember the details (this comment didn't actually change here, it's pretty old) but I came to a similar conclusion and left this vague note.

[davepacheco]

Thanks for the review!