Non-deterministic source builds

[carlocab]

We struggled with updating deno at Homebrew for a while (cf. Homebrew/homebrew-core#66920) and it seems that part of the reason is that the build is non-deterministic. The exact same build configuration can succeed now and fail (edit: with the same version!) later and vice-versa.

This is partly summarized in the table at Homebrew/homebrew-core#66920 (comment). These are all attempts at building deno 1.7.0.

In particular, note that the build configuration is the same between Homebrew/homebrew-core@54c027e and Homebrew/homebrew-core@77e4ad8 (the first two commits in the table), but the two commits have starkly different build outcomes.

The last three commits in that table are also the same, with differing results.

[bnoordhuis]

https://github.com/Homebrew/homebrew-core/blob/300a7e0dff54e010c553ff539d883629c0c82e6c/Formula/deno.rb is the formula, correct? Some observations:

• you don't need xcode or sscache to build
• building V8 from source using your own llvm/clang: very much uncharted territory, V8 has a knack for stretching compilers to their limit

[carlocab]

you don't need xcode or sscache to build

Agreed re sccache, I just put it there to try to speed up the build. (It did not, but I didn't want to re-run CI another dozen times to try to get a successful build without it.)

I did try dropping the Xcode dependency, but all the builds failed when I did. Is it not needed to build rusty_V8? That was mentioned somewhere in Homebrew/homebrew-core#66920.

Of course, the randomness of the build process makes it hard to ascertain what dropping the Xcode dependency really did.

[avindra]

👍 I've been seeing signs of nondeterministic builds in deno builds at open build service. Packages are rebuilt when dependencies change, so deno is rebuilt fairly frequently. And when that happens, I see the binary diff warning each time.

Check the log I linked in the build service for /usr/bin/deno differs in ELF section .data to see an example of this.

EDIT: For context: I'm not using a cache.

[carlocab]

That's really intriguing, @avindra. Since when have you noticed this? I think this started happening at around version 1.6.0, but it could be earlier.

[avindra]

I wish I kept track of it and unfortunately OBS doesn't keep around old logs yet.

I started packaging around 1.5.4 and only noticed the binary diff warnings some releases after that. 1.6.0 sounds about right. I think your testing matrix might have better clues about when it regressed.

[carlocab]

That table is just three different build configurations for 1.7.0, so it doesn't really tell us anything about different deno versions, unfortunately. Though 1.6.0 is the release after 1.5.4, so perhaps it is around 1.6.0.

[avindra]

Did some sleuthing... rusty_v8 @ 0.16.0 builds deterministically, as well as jstime.

So we can infer in-determinism stems from denoland/deno. After comparing v1.5.4...v1.6.0, I think the root cause may be this line from the LSP mvp (#8515) :

deno/cli/lsp/sources.rs

Line 246 in 1f8b83b

return if let Ok(n) = modified.duration_since(SystemTime::UNIX_EPOCH)

It would appear that script_version may be reporting different numbers based on when the code was compiled.

[bnoordhuis]

cc @kitsonk

[kitsonk]

It would appear that script_version may be reporting different numbers based on when the code was compiled.

That is for the lsp to determine a "version" of a cached file. I can't conceive of a way that would possibly impact the binary, as this code is only invoked during the execution of Deno.

[avindra]

Thanks for looking at this @kitsonk. I debugged locally with some println!() around the version values and I didn't see any strings containing the version on startup when I ran cargo run lsp. The metadata field on the Sources struct is empty when the server starts up, which leads me to agree that there shouldn't be any static version data getting compiled in.

Interestingly, I just kicked the build off manually for jstime and it is clear that the jstime build is non-deterministic as well. This leads me to think there may be a bug in one of the upstreams (rust/llvm/clang,etc).

To be sure, I kicked off another build for rusty_v8 to see if I can get any build inconsistencies there (none seen so far).

EDIT: rusty_v8 rebuild is clean. So for some reason, only the jstime and deno builds are nondeterministic.

[avindra]

On the subject of build consistency.... here's a v8 related error that shows up in a number of recent builds:

[  330s]   [230/1261] LINK ./bytecode_builtins_list_generator
[  330s]   [231/1261] ACTION //v8:generate_bytecode_builtins_list(//build/toolchain/linux:clang_x64)
[  330s]   [232/1261] STAMP obj/v8/generate_bytecode_builtins_list.stamp
[  330s]   [233/1261] STAMP obj/v8/torque_base.stamp
[  330s]   [234/1261] LINK ./torque
[  330s]   FAILED: torque
[  330s]   python "../../../vendor/rusty_v8/build/toolchain/gcc_link_wrapper.py" --output="./torque" -- ../../../../../../../../usr/bin/clang++ -pie -Wl,--fatal-warnings -Wl,--build-id -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -fuse-ld=lld -Wl,--icf=all -Wl,--color-diagnostics -Wl,--no-call-graph-profile-sort -m64 -rdynamic -nostdlib++ -pie -Wl,--disable-new-dtags -Wl,-O2 -Wl,--gc-sections -o "./torque" -Wl,--start-group @"./torque.rsp"  -Wl,--end-group  -ldl -lpthread -lrt
[  330s]   clang-11.0: error: no such file or directory: '@./torque.rsp'
[  330s]   [235/1261] CXX obj/v8/src/inspector/inspector/remote-object-id.o
[  330s]   [236/1261] CXX obj/v8/src/inspector/inspector/string-16.o
[  330s]   [237/1261] CXX obj/v8/src/inspector/inspector/string-util.o
[  330s]   [238/1261] CXX obj/v8/src/inspector/inspector/test-interface.o
[  330s]   [239/1261] CXX obj/v8/src/inspector/inspector/Console.o
[  330s]   [240/1261] CXX obj/v8/src/inspector/inspector/search-util.o
[  330s]   [241/1261] CXX obj/v8/src/inspector/inspector/Protocol.o
[  330s]   [242/1261] CXX obj/v8/src/inspector/inspector/value-mirror.o
[  330s]   [243/1261] CXX obj/v8/src/inspector/inspector/Debugger.o
[  330s]   ninja: build stopped: subcommand failed.
[  330s]

The core error embedded there is:

[  330s]   clang-11.0: error: no such file or directory: '@./torque.rsp'

Even more mysterious than the error about a missing torque.rsp file is that triggering enough rebuilds will bypass this condition and allow the build to continue. 🤷‍♂️

[carlocab]

That actually looks like some of the errors I've encountered. Let me try to dig some of them up.

My earlier guess was that a parallel build isn't always building things in the correct order. However, that does not fully explain why, conditional on failure, the failure always seems to occur at roughly the same point.

[bnoordhuis]

Maybe a ninja bug? torque.rsp is written by ninja before it executes the command, then deleted afterwards.

[carlocab]

Actually, I take this back:

However, that does not fully explain why, conditional on failure, the failure always seems to occur at roughly the same point.

I get different kinds of failures for different build attempts. I even have this one, which I find somewhat amusing:

[rusty_v8 0.16.0] ninja: error: unknown target 'rusty_v8'

It may well be a bug in ninja. Which version of ninja do you use in your build system?

[avindra]

It may well be a bug in ninja. Which version of ninja do you use in your build system?

The ninja_gn_binaries repo is at 1.8.2. Although, I don't know if this binary is also used in the release/build process.

❯ wget https://github.com/denoland/ninja_gn_binaries/raw/master/linux64/ninja
❯ chmod +x ./ninja
❯ ./ninja --version
1.8.2

The latest ninja is 1.10.2, which is what openSUSE is using to build.

EDIT: fix version #

[carlocab]

The latest ninja is 1.12, which is what openSUSE is using to build.

Do you mean 1.10.2? That's what we use too, though I only switched to the latest ninja release instead of the vendored one to try to address build failures.

[carlocab]

Not sure where exactly the bug is (cargo, maybe?), but passing -j1 to cargo during the build seems to resolve our build problems: Homebrew/homebrew-core#72365

It seems that, without this flag, cargo triggers two simultaneous builds of rusty_v8. Not really sure why it's doing that.

[bnoordhuis]

Is that with V8_FROM_SOURCE= set in the environment? The non-source builds (static library download) had a concurrency issue that should have been resolved with denoland/rusty_v8#543.

[carlocab]

Is that with V8_FROM_SOURCE= set in the environment?

We set V8_FROM_SOURCE=1, which presumably does the same thing.

[bnoordhuis]

Yes, it does. Do you remember when you first starting seeing this?

rusty_v8@0.18.0 introduced denoland/rusty_v8@3d26ffb and I can see how some of the path mangling and environment hacking could confuse cargo...

...on the other hand, pretty much exact the same code ran when it was still its own cargo_gn crate.

[carlocab]

My guess is around 1.6.0, but I'll need to double-check the build logs.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[lucacasonato]

This is still an issue.

[06kellyjac]

Here's a diffoscope output I ran against the nixpkgs 1.11.2 build of deno (see https://r13y.com for details on reviewing reproducibility in nixpkgs)
https://gist.github.com/06kellyjac/43de7f28038a929eecf41dabf568a2d6

Nothing obvious is popping out to me but I also don't have much of a clue what to look for when using diffoscope yet

As a note 1.5.4 also didn't result in a reproducible build for me either. As another note we're using the released copy of librusty_v8.a for our builds atm, I haven't tried a 1.5.4 build with a v8 built also from source

[avindra]

EDIT: still seeing diffs in the elf (tested Jun 25)

I can build deterministically now.

[aslilac]

@lucacasonato has anyone been running into this still?

[avindra]

still seeing diffs in the binaries. unclear what the cause may be.

/usr/bin/deno differs in ELF sections
--- old /usr/bin/deno (objdump)
+++ new /usr/bin/deno (objdump)
 @@ -1295,27 +1295,27 @@
   005380 08000000 00000000 b9fcc901 00000000  ................
   005390 28235904 00000000 08000000 00000000  (#Y.............
   0053a0 ad5acd01 00000000 40235904 00000000  .Z......@#Y.....
 - 0053b0 08000000 00000000 28b6e503 00000000  ........(.......
 + 0053b0 08000000 00000000 a8b4e503 00000000  ................
   0053c0 50235904 00000000 08000000 00000000  P#Y.............
   0053d0 4bfeee03 00000000 60235904 00000000  K.......`#Y.....
   0053e0 08000000 00000000 e8feee03 00000000  ................
   0053f0 70235904 00000000 08000000 00000000  p#Y.............
   005400 d51bec03 00000000 80235904 00000000  .........#Y.....
 - 005410 08000000 00000000 d1fee503 00000000  ................
 + 005410 08000000 00000000 51fde503 00000000  ........Q.......
   005420 88235904 00000000 08000000 00000000  .#Y.............
 - 005430 a9e4e503 00000000 90235904 00000000  .........#Y.....
 - 005440 08000000 00000000 4012e603 00000000  ........@.......
 + 005430 29e3e503 00000000 90235904 00000000  )........#Y.....
 + 005440 08000000 00000000 c010e603 00000000  ................
   005450 98235904 00000000 08000000 00000000  .#Y.............
 - 005460 e6e5e503 00000000 a0235904 00000000  .........#Y.....
 - 005470 08000000 00000000 8c1be603 00000000  ................
 + 005460 66e4e503 00000000 a0235904 00000000  f........#Y.....
 + 005470 08000000 00000000 0c1ae603 00000000  ................
   005480 c0235904 00000000 08000000 00000000  .#Y.............
 - 005490 a2e4e503 00000000 c8235904 00000000  .........#Y.....
 - 0054a0 08000000 00000000 d1fee503 00000000  ................
 + 005490 22e3e503 00000000 c8235904 00000000  "........#Y.....
 + 0054a0 08000000 00000000 51fde503 00000000  ........Q.......
   0054b0 d0235904 00000000 08000000 00000000  .#Y.............
   0054c0 56ddfe03 00000000 d8235904 00000000  V........#Y.....
 - 0054d0 08000000 00000000 a9e4e503 00000000  ................
 + 0054d0 08000000 00000000 29e3e503 00000000  ........).......
   0054e0 e0235904 00000000 08000000 00000000  .#Y.............
 - 0054f0 08dee503 00000000 08245904 00000000  .........$Y.....
 + 0054f0 88dce503 00000000 08245904 00000000  .........$Y.....
   005500 08000000 00000000 110dd301 00000000  ................
   005510 10245904 00000000 08000000 00000000  .$Y.............
   005520 0c0dd301 00000000 18245904 00000000  .........$Y.....
 @@ -1369,20 +1369,20 @@
   005820 879cce01 00000000 78255904 00000000  ........x%Y.....
   005830 08000000 00000000 76bec901 00000000  ........v.......
   005840 80255904 00000000 08000000 00000000  .%Y.............
 - 005850 4712e603 00000000 88255904 00000000  G........%Y.....
 - 005860 08000000 00000000 7c12e603 00000000  ........|.......
 + 005850 c710e603 00000000 88255904 00000000  .........%Y.....
 + 005860 08000000 00000000 fc10e603 00000000  ................
   005870 90255904 00000000 08000000 00000000  .%Y.............
 - 005880 a912e603 00000000 98255904 00000000  .........%Y.....
 - 005890 08000000 00000000 dd12e603 00000000  ................
 + 005880 2911e603 00000000 98255904 00000000  )........%Y.....
 + 005890 08000000 00000000 5d11e603 00000000  ........].......
   0058a0 a0255904 00000000 08000000 00000000  .%Y.............
 - 0058b0 1313e603 00000000 a8255904 00000000  .........%Y.....
 - 0058c0 08000000 00000000 4013e603 00000000  ........@.......
 + 0058b0 9311e603 00000000 a8255904 00000000  .........%Y.....
 + 0058c0 08000000 00000000 c011e603 00000000  ................
   0058d0 b0255904 00000000 08000000 00000000  .%Y.............
 - 0058e0 7a13e603 00000000 b8255904 00000000  z........%Y.....
 - 0058f0 08000000 00000000 b113e603 00000000  ................
 + 0058e0 fa11e603 00000000 b8255904 00000000  .........%Y.....
 + 0058f0 08000000 00000000 3112e603 00000000  ........1.......
   005900 c0255904 00000000 08000000 00000000  .%Y.............
 - 005910 e013e603 00000000 c8255904 00000000  .........%Y.....
 - 005920 08000000 00000000 eb13e603 00000000  ................
 + 005910 6012e603 00000000 c8255904 00000000  `........%Y.....
+ 005920 08000000 00000000 6b12e603 00000000  ........k.......
  005930 f0255904 00000000 08000000 00000000  .%Y.............
  005940 0ecac901 00000000 f8255904 00000000  .........%Y.....
  005950 08000000 00000000 fac9c901 00000000  ................
@@ -1400,10 +1400,10 @@
  005a10 08000000 00000000 c10bca01 00000000  ................
  005a20 40265904 00000000 08000000 00000000  @&Y.............
  005a30 9a0cca01 00000000 48265904 00000000  ........H&Y.....
- 005a40 08000000 00000000 44b7e503 00000000  ........D.......
+ 005a40 08000000 00000000 c4b5e503 00000000  ................
  005a50 60265904 00000000 08000000 00000000  `&Y.............
  005a60 61deef03 00000000 68265904 00000000  a.......h&Y.....
- 005a70 08000000 00000000 cfb2e503 00000000  ................
+ 005a70 08000000 00000000 4fb1e503 00000000  ........O.......
  005a80 70265904 00000000 08000000 00000000  p&Y.............
  005a90 0205eb03 00000000 78265904 00000000  ........x&Y.....
  005aa0 08000000 00000000 22cbeb03 00000000  ........".......
@@ -1420,94 +1420,94 @@
  005b50 3e6fc901 00000000 d0265904 00000000  >o.......&Y.....
  005b60 08000000 00000000 c88eca01 00000000  ................
  005b70 e0265904 00000000 08000000 00000000  .&Y.............
- 005b80 e902e603 00000000 e8265904 00000000  .........&Y.....
- 005b90 08000000 00000000 e613e603 00000000  ................
+ 005b80 6901e603 00000000 e8265904 00000000  i........&Y.....
+ 005b90 08000000 00000000 6612e603 00000000  ........f.......
  005ba0 f0265904 00000000 08000000 00000000  .&Y.............
- 005bb0 aea2e503 00000000 f8265904 00000000  .........&Y.....
- 005bc0 08000000 00000000 2cfee503 00000000  ........,.......
+ 005bb0 2ea1e503 00000000 f8265904 00000000  .........&Y.....
+ 005bc0 08000000 00000000 acfce503 00000000  ................
  005bd0 00275904 00000000 08000000 00000000  .'Y.............
- 005be0 2a81e503 00000000 20275904 00000000  *....... 'Y.....
- 005bf0 08000000 00000000 be16e603 00000000  ................
+ 005be0 aa7fe503 00000000 20275904 00000000  ........ 'Y.....
+ 005bf0 08000000 00000000 3e15e603 00000000  ........>.......
  005c00 28275904 00000000 08000000 00000000  ('Y.............
- 005c10 08ede503 00000000 30275904 00000000  ........0'Y.....
- 005c20 08000000 00000000 c716e603 00000000  ................
+ 005c10 88ebe503 00000000 30275904 00000000  ........0'Y.....
+ 005c20 08000000 00000000 4715e603 00000000  ........G.......
  005c30 38275904 00000000 08000000 00000000  8'Y.............
- 005c40 cd16e603 00000000 40275904 00000000  ........@'Y.....
- 005c50 08000000 00000000 d916e603 00000000  ................
+ 005c40 4d15e603 00000000 40275904 00000000  M.......@'Y.....
+ 005c50 08000000 00000000 5915e603 00000000  ........Y.......
  005c60 48275904 00000000 08000000 00000000  H'Y.............
- 005c70 e316e603 00000000 50275904 00000000  ........P'Y.....
- 005c80 08000000 00000000 ee16e603 00000000  ................
+ 005c70 6315e603 00000000 50275904 00000000  c.......P'Y.....
+ 005c80 08000000 00000000 6e15e603 00000000  ........n.......
  005c90 58275904 00000000 08000000 00000000  X'Y.............
- 005ca0 fa16e603 00000000 60275904 00000000  ........`'Y.....
- 005cb0 08000000 00000000 0517e603 00000000  ................
+ 005ca0 7a15e603 00000000 60275904 00000000  z.......`'Y.....
+ 005cb0 08000000 00000000 8515e603 00000000  ................
  005cc0 68275904 00000000 08000000 00000000  h'Y.............
- 005cd0 3018e603 00000000 70275904 00000000  0.......p'Y.....
+ 005cd0 b016e603 00000000 70275904 00000000  ........p'Y.....
  005ce0 08000000 00000000 9861ff03 00000000  .........a......
  005cf0 78275904 00000000 08000000 00000000  x'Y.............
- 005d00 0a17e603 00000000 80275904 00000000  .........'Y.....
- 005d10 08000000 00000000 2818e603 00000000  ........(.......
+ 005d00 8a15e603 00000000 80275904 00000000  .........'Y.....
+ 005d10 08000000 00000000 a816e603 00000000  ................
  005d20 88275904 00000000 08000000 00000000  .'Y.............
- 005d30 1617e603 00000000 90275904 00000000  .........'Y.....
- 005d40 08000000 00000000 1e17e603 00000000  ................
+ 005d30 9615e603 00000000 90275904 00000000  .........'Y.....
+ 005d40 08000000 00000000 9e15e603 00000000  ................
  005d50 98275904 00000000 08000000 00000000  .'Y.............
  005d60 02f90204 00000000 a0275904 00000000  .........'Y.....
- 005d70 08000000 00000000 2617e603 00000000  ........&.......
+ 005d70 08000000 00000000 a615e603 00000000  ................
  005d80 a8275904 00000000 08000000 00000000  .'Y.............
- 005d90 2c17e603 00000000 b0275904 00000000  ,........'Y.....
+ 005d90 ac15e603 00000000 b0275904 00000000  .........'Y.....
  005da0 08000000 00000000 4a5aff03 00000000  ........JZ......
  005db0 b8275904 00000000 08000000 00000000  .'Y.............
- 005dc0 3a17e603 00000000 c0275904 00000000  :........'Y.....
+ 005dc0 ba15e603 00000000 c0275904 00000000  .........'Y.....
  005dd0 08000000 00000000 c2f6e903 00000000  ................
  005de0 c8275904 00000000 08000000 00000000  .'Y.............
- 005df0 4417e603 00000000 d0275904 00000000  D........'Y.....
+ 005df0 c415e603 00000000 d0275904 00000000  .........'Y.....
  005e00 08000000 00000000 c39df603 00000000  ................
  005e10 d8275904 00000000 08000000 00000000  .'Y.............
  005e20 b7990204 00000000 e0275904 00000000  .........'Y.....
- 005e30 08000000 00000000 4917e603 00000000  ........I.......
+ 005e30 08000000 00000000 c915e603 00000000  ................
  005e40 e8275904 00000000 08000000 00000000  .'Y.............
- 005e50 4f17e603 00000000 f0275904 00000000  O........'Y.....
- 005e60 08000000 00000000 5c17e603 00000000  ........\.......
+ 005e50 cf15e603 00000000 f0275904 00000000  .........'Y.....
+ 005e60 08000000 00000000 dc15e603 00000000  ................
  005e70 f8275904 00000000 08000000 00000000  .'Y.............
- 005e80 6617e603 00000000 00285904 00000000  f........(Y.....
- 005e90 08000000 00000000 6d17e603 00000000  ........m.......
+ 005e80 e615e603 00000000 00285904 00000000  .........(Y.....
+ 005e90 08000000 00000000 ed15e603 00000000  ................
  005ea0 08285904 00000000 08000000 00000000  .(Y.............
- 005eb0 7417e603 00000000 10285904 00000000  t........(Y.....
- 005ec0 08000000 00000000 7b17e603 00000000  ........{.......
+ 005eb0 f415e603 00000000 10285904 00000000  .........(Y.....
+ 005ec0 08000000 00000000 fb15e603 00000000  ................
  005ed0 18285904 00000000 08000000 00000000  .(Y.............
- 005ee0 8217e603 00000000 20285904 00000000  ........ (Y.....
- 005ef0 08000000 00000000 8c17e603 00000000  ................
+ 005ee0 0216e603 00000000 20285904 00000000  ........ (Y.....
+ 005ef0 08000000 00000000 0c16e603 00000000  ................
  005f00 28285904 00000000 08000000 00000000  ((Y.............
- 005f10 9517e603 00000000 30285904 00000000  ........0(Y.....
+ 005f10 1516e603 00000000 30285904 00000000  ........0(Y.....
  005f20 08000000 00000000 8bc7ee03 00000000  ................
  005f30 38285904 00000000 08000000 00000000  8(Y.............
  005f40 8ec7ee03 00000000 40285904 00000000  ........@(Y.....
- 005f50 08000000 00000000 a017e603 00000000  ................
+ 005f50 08000000 00000000 2016e603 00000000  ........ .......
  005f60 48285904 00000000 08000000 00000000  H(Y.............
- 005f70 aa17e603 00000000 50285904 00000000  ........P(Y.....
+ 005f70 2a16e603 00000000 50285904 00000000  *.......P(Y.....
  005f80 08000000 00000000 5b07ea03 00000000  ........[.......
  005f90 58285904 00000000 08000000 00000000  X(Y.............
- 005fa0 b417e603 00000000 60285904 00000000  ........`(Y.....
- 005fb0 08000000 00000000 be17e603 00000000  ................
+ 005fa0 3416e603 00000000 60285904 00000000  4.......`(Y.....
+ 005fb0 08000000 00000000 3e16e603 00000000  ........>.......
  005fc0 68285904 00000000 08000000 00000000  h(Y.............
  005fd0 6008ea03 00000000 70285904 00000000  `.......p(Y.....
  005fe0 08000000 00000000 68090104 00000000  ........h.......
  005ff0 78285904 00000000 08000000 00000000  x(Y.............
- 006000 c917e603 00000000 80285904 00000000  .........(Y.....
- 006010 08000000 00000000 cf17e603 00000000  ................
+ 006000 4916e603 00000000 80285904 00000000  I........(Y.....

[i-api]

dealing with this issue now, for a long time my binaries turned out the same, then one day they just started differing

[i-api]

not sure why

[BlackAsLight]

I don't know if it's relevant but until recently, rust-lang/cargo#14107, cargo was including host specific metadata that would change the hash of the output binary when built on other machines but with the same target.