feature: async digest handling

[rgrinberg]

Move the computation of digests to a background thread. For now, this is guarded behind a configuration option that requires this to be enabled.

Signed-off-by: Rudi Grinberg me@rgrinberg.com

[Alizter]

Then bench shows a noticeable slowdown:

Compare main:

1 | 0.9362449645996094
2 | 0.9103691577911377
3 | 0.9048709869384766
4 | 0.9169681072235107
5 | 0.925853967666626

with this pr:

1 | 0.9660398960113525
2 | 0.9446120262145996
3 | 0.9524080753326416
4 | 0.9530608654022217
5 | 0.9539759159088135

Also the clear build times make this clearer:

Main: 107.39912676811218
PR:   111.15323400497437

[rgrinberg]

Thanks for the initial numbers. This PR is waiting for a change in the way we handle digests to be more multi threading friendly.

[rgrinberg]

@Alizter and @jchavarri this is worth benchmarking now

[Alizter]

New bench results:

	main	PR	%diff
Clean	101.73068690299988	99.33992314338684	-2.37803%
Null	0.9008660316467285	0.9387650489807129
	0.9166581630706787	0.8799121379852295
	0.905055046081543	0.8913090229034424
	0.9088449478149414	0.9316670894622803
	0.9060719013214111	0.9277148246765137
Avg.	0.90749921798706	0.91387362480164	+0.699956%

And some benches on building Dune:

Main

Benchmark 1: ./dune.exe build @install --cache disabled
  Time (mean ± σ):      9.826 s ±  0.280 s    [User: 90.914 s, System: 28.879 s]
  Range (min … max):    9.290 s … 10.279 s    10 runs

PR

Benchmark 1: ./dune.exe build @install --cache disabled
  Time (mean ± σ):     10.206 s ±  0.342 s    [User: 91.557 s, System: 29.759 s]
  Range (min … max):    9.916 s … 11.012 s    10 runs

The first bench appears to be no slowdown, however the second result appears to tell me that there still is some slowdown on a clean build. This could be down to the fact that the second results use 20 jobs rather than 1.

[rgrinberg]

@Alizter digests aren't computed for null builds.

[Alizter]

OK that makes sense. If I get a chance I will try to test some chunkier repos to see if the slight slowdown is constant or also scales.

[rgrinberg]

@snowleopard this option is now under config.

[snowleopard]

Looks good, thanks!

[snowleopard]

The PR description needs to be updated though (it claims too much?).