Bazel 9's no-op builds now take 3.5s

[rejuvenile]

Description of the bug:

Previously, on Bazel 8.5.1, my no-op builds (as executed repeatedly and locally) would take about 500 milliseconds. Now, under Bazel 9, they take 3.5 seconds. Eyeballing the UI, most of the time is taken "checking cached actions", but when looking at the profile, I see a mix of delays, which vary per build. Sometimes it's doExtendLeases / findMissingDigests, other times it's BazelWorkspaceStatusAction stable-status.txt / Action execute / actuallyCompleteAction / outputService.finalizeAction. The common denomentator is slowness. My builds use remote execution and caching, but appears to repro. without those features (which makes sense, as everything should be cached locally).

[fmeum]

When you say no-op, do you mean back-to-back Bazel builds of the same targets? Can you share a Starlark profile?

[rejuvenile]

Yes, back to back builds of the same targets.

Removing --experimental_remote_cache_ttl and --experimental_remote_cache_lease_extension makes the profiles clearer: one issue is a long wait in skyframeExecutor.evaluateBuildDriverKeys when executors aren't really doing anything. This lead me to reduce --jobs=5000 to --jobs=100 which removes most of the delay; the no-op is now only 800ms. From this I surmise that Bazel 9 isn't very good at scheduling large numbers of workers, at least when they don't have much to do n

With this fix, the remaining 400ms of additional delay comes from fsvc.getDirtyKeys, which goes from 40 ms on Bazel 8 to 400 ms on Bazel 9.

I'm sorry that I can't share the actual profile. I still suspect that --experimental_remote_cache_lease_extension is also much slower under Bazel 9, but as disabling it helps and I don't need it enabled, I haven't investigated further.

[rejuvenile]

I can make fsvc.getDirtyKeys fast again with --experimental_check_external_other_files=false, but it's not clear to me whether that strictly undoes the Bazel 9 regression, or simply makes Bazel check a lot fewer files.

[fmeum]

--experimental_check_external_other_files=false making your build faster with Bazel 9 is expected if you previously used --experimental_check_external_repository_files=false. Is that the case? The reason is that the repo contents cache, which is enabled by default in Bazel 9, changed the classification of most repo's files to be controlled by this other flag.

Could you share the flags that you are or were using? Feel free to redact broadly, string-valued flags rarely matter.

[fmeum]

cc @coeuvre for build driver keys/lease extension

@Wyverald @meisterT I am wondering whether checking all external repo files for modifications on every build is the right tradeoff. Modifications that happen while the Bazel server isn't running can't be caught anyway and as far as I understand there is no --watchfs support for this, resulting in a lot of stats with lots of external deps.

[fmeum]

@bazel-io flag

[rejuvenile]

--experimental_check_external_other_files=false making your build faster with Bazel 9 is expected if you previously used --experimental_check_external_repository_files=false. Is that the case?

I did not previously use it. I only discovered it when researching this regression.