reproducible builds: filesystem readdir order

[bmwiedemann]

While working on reproducible builds for openSUSE, I found that our ocaml-dune package varies through non-determinism from filesystem readdir order.

Expected Behavior

dune build should be reproducible

Actual Behavior

--- strings RPMS.1/usr/bin/dune
+++ strings RPMS.2/usr/bin/dune
[...]
 padding.0
-sha1_do_chunk
 sha512_do_chunk
+sha1_do_chunk
 scanmult
 shutdown_happened

Maybe related: ocaml-dune-3.12.2/_boot/source_files produced by ./.duneboot.exe --verbose also has order variations.

Reproduction

build on two different filesystems or use disorderfs or randomize readdir order

Specifications

• Version of dune (output of dune --version): ocaml-dune-3.12.2 with boot: remove single-command bootstrap #9735
• Version of ocaml (output of ocamlc --version) ocaml-4.14.1
• Operating system (distribution and version): openSUSE Tumbleweed

[emillon]

Hi,
Thanks for the report.
I'll have a look at this, but to clarify, can you tell me how you are creating /usr/bin/dune?

[bmwiedemann]

It is part of our rpm build - see
https://code.opensuse.org/package/ocaml-dune/blob/master/f/ocaml-dune.spec#_80 or
https://src.opensuse.org/rpm/ocaml-dune/src/commit/64cd15fe818c3d79c39cedbbf01e7ee662b726ee/ocaml-dune.spec#L80

[emillon]

I had a look at the build instructions. This part does not look right:

ocaml boot/bootstrap.ml --verbose ${jobs}
./_boot/dune.exe build \
	dune.install \
	--release \
	--profile dune-bootstrap \
	--verbose \
	${jobs} \
	%nil
mkdir .bin
ln -sv ../_boot/dune.exe .bin/dune

The first command will create _boot/dune.exe. This binary is then used to build a new binary (in _build/install/default/bin/dune).

If I understand correctly, you're installing _boot/dune instead of installing _build/install/default/bin/dune. They both should act similarly, but only the latter has been built by dune and is expected to be reproducible. So that's the one you should be installing in the rpm package. _boot/dune.exe is just and intermediate build artifact that's not expected to be reproducible.

(in the light of that, #9507 didn't really have to be fixed in terms of reproducibility, but it's still valuable to remove this code path)

[bmwiedemann]

I saw

/home/abuild/rpmbuild/BUILDROOT/ocaml-dune-3.13.0-0.x86_64/usr/bin/dune from /home/abuild/rpmbuild/BUILDROOT/ocaml-dune-3.13.0-0.x86_64/usr/bin/.#dune.dune-temp written per rename
    by exec="/home/abuild/rpmbuild/BUILD/ocaml-dune-3.13.0/.bin/dune", ["dune", "install", "--verbose", "--for-release-of-packages=dune", "-j16", "--prefix=/usr", "--libdir=/usr/lib64/ocaml", "--destdir=/home/abuild/rpmbuild/BUILDROOT/ocaml-dune-3.13.0-0.x86_64", "dune"] - started

so the final binary gets written there by dune install, but don't know where dune install reads the installed binary.

I also tried

-ln -sv ../_boot/dune.exe .bin/dune
+ln -sv ../_build/default/bin/dune.exe .bin/dune

and it did not help with the original issue.
Plus these dune.exe files had the same content after the build.

md5sum ./_build/default/bin/dune.exe ./_build/default/_boot/dune.exe _boot/dune.exe 
be020387f3b524d2c1105a4a26d7281c  ./_build/default/bin/dune.exe
be020387f3b524d2c1105a4a26d7281c  ./_build/default/_boot/dune.exe
be020387f3b524d2c1105a4a26d7281c  _boot/dune.exe

[emillon]

Sorry, I was confused about our bootstrap process. It's indeed intended that _boot/dune.exe is installed. I created #9861 to solve the issue, and tested it with disorderfs.