zig prereq races as of 0.11.0-dev.1782+b52be973d

[motiejus]

Dear maintainers,

We still observe some very-hard-to-reproduce races when building libc/rt prerequisites.

Some anecdotal evidence:

error: FileNotFound

When executed on a fresh installation, zig build-exe toolchain/launcher.zig (exact command) sometimes fails with:

error: FileNotFound

This happens only on a fresh $ZIG_GLOBAL_CACHE_DIR (which we keep in /tmp/bazel-zig-cc). We have seen this happen on Darwin x86_64 and Darwin M1. We may saw it on Linux, but I no longer have the logs to verify. My memory is poor.

I tried to reproduce this on my MacOS machine overnight, without success. But we do receive a couple of complaints a week consistently over the last few weeks. Note that the sample size is quite large.

libcompiler_rt.a: No such file or directory

This happened on our CI yesterday:

/tmp/bazel-zig-cc/o/4421bb1adcf01feee7185ccb98640027/libcompiler_rt.a: No such file or directory

Unfortunately, I can no longer access the build host nor access it's global cache dir. It may be related.

Summary

I understand this is very little information to troubleshoot effectively. Here are the steps I am trying to do:

1. Capture the first error (FileNotFound) on any Linux machine and instruct the engineer to re-run the command under strace, to see which file they are missing. However, this was not reported on Linux for the last week or so: either it did not happen, or people learned to remove the cache directory and move on. Since this happens more on OSX, it would make sense to debug it here. However, our engineers cannot run dtruss for compliance reasons.
2. I will try to repro this on my MacOS machine again, but slightly differently.

Food for thought: is it time to reconsider how error context is propagated during the build phase, so errors could be augmented with additional context?

[motiejus]

I do have $ZIG_GLOBAL_CACHE_DIR archive from MacOS x86_64 right after when this happened. However, I wasn't able to plug in the archive to my machine and reproduce the issue.

If that helps, I could send over the archive privately.

[motiejus]

I have CI a machine in production where I can reproduce this.

<...>:~/.cache/bazel/_bazel_jenkins/d216a97b1920f6547de128d379fd31b3/external/zig_sdk/tools$ env ZIG_GLOBAL_CACHE_DIR=/tmp/bazel-zig-cc ZIG_LOCAL_CACHE_DIR=/tmp/bazel-zig-cc ../zig 'build-exe' '-OReleaseSafe' 'launcher.zig' '-s
tatic'
error: ld.lld: cannot open /tmp/bazel-zig-cc/o/8867000f2fcde52678126003825526ac/libc.a: No such file or directory
error: ld.lld: cannot open /tmp/bazel-zig-cc/o/116b7d9400e39e5a8d3fc65084a9ce7f/libcompiler_rt.a: No such file or directory
<...>:~/.cache/bazel/_bazel_jenkins/d216a97b1920f6547de128d379fd31b3/external/zig_sdk/tools$ 

I will do my best for it to keep it's state until tomorrow. If there is a command I can run there, I am all ears.

[andrewrk]

@motiejus and I have been discussing this issue on IRC, and he shared with me a tarball of the zig-cache directory from the affected system. We have made the following observations:

• the issue has occurred on both macOS and on Linux.
• in the zig-cache directory on the Linux computer, the h/ directory is very small, but the o/ directory has 3726 empty directories in it. The h/ directory is reproduced here:

h/0bb411f073f43ac89ac94f4702ec0974.txt
h/b97a8a3a3a84ebf83ea6cd9d984e1fd0.txt
h/timestamp

The only artifacts found in o/ apart from 3726 empty directories are:

o/8e8f0d3efc02827d94d9e35c6989e340/builtin.zig
o/a07523923b2b200807e54b2be152a486/builtin.zig
o/369045f44d4564feae17c4e32f314866/builtin.zig

What I find curious about this is that zig only creates a directory inside o/ if it is about to write a build artifact there. I can't fathom why there would be 3726 empty directories here, apart from a third party going in and deleting files.

As for a plan of attack here, I'd like to finish my work on #14647. I'm hoping to land this branch within 1 week. There is a high chance, that if there is a race condition in zig's caching system which is causing this issue, that it will be reproduced by either my local computer, or the CI, when testing this branch. If it does not get reproduced, that suggests for us to look into a different possible cause. Perhaps something more straightforward, hiding under our very noses.

Meanwhile, we have seen the merge of #14821 yesterday, addressing what could be a related issue to this one. This issue was indeed being triggered reliably by my work in the branch as mentioned above. Another avenue of attack that I will pursue is to examine the caching system and look for an equivalent bug to this one that was recently solved.

[andrewrk]

I have landed #14647, and as hoped, I have seen some sporadic issues that look related to this one. For example, in https://github.com/ziglang/zig/actions/runs/4444638053/jobs/7803016844:

/Users/runner/work/zig/zig/test/behavior/translate_c_macros.zig:7:11: error: C import failed: FileNotFound

Likewise, in https://github.com/ziglang/zig/actions/runs/4444638053/jobs/7803016681:

zig build-exe test ReleaseFast native: error: error: unable to open 'zig-local-cache/o/6b2e769a5b7088fc2f30d8d9c03c4e44': FileNotFound

This is likely a related issue, and has a completely straightforward solution: #14978

So, I will continue by addressing the tail end of issues that my build-parallel branch has exposed. I think it is very likely that, in a few weeks time, as these follow-up issues are resolved, it will also solve this issue (the one I am commenting on).

[motiejus]

To capture/troubleshoot such cases I am considering to run a ReleaseSafe, unstripped zig in production. Will run some benchmarks tomorrow on my end.

Also, #14923 would have been resolved much quicker with ReleaseSafe, since zld had an assertion that was skipped due to a ReleaseFast build.

In this specific case we would see the stack trace immediately where the error: FileNotFound is coming from.

Specific questions to maintainers (sorry for previous pings):

• thoughts on doing this on CI? just realized this is happening already.
• should we consider publishing unstripped ReleaseSafe nightlies in ziglang.org/download on top of the existing ReleaseSafe/strip ones?

Use case for publishing ReleaseSafe/unstripped builds:

• whoever wants to run it in production
• whoever is investigating an issue with a compiler will have one readily available (e.g. I still haven't learned how to build zig with system's llvm on my Mac).
• integration tests of 3rd party systems. Long story short, I will, once I fix the remaining Go unit tests with CC="zig cc", will propose an official CI runner to the Go team which does just that. For this use case we want a ReleaseSafe+debug build, so the Zig's stack traces that Go devs will be looking at will be actionable much quicker. No guarantees it will be accepted, but there is a chance given Uber's relationship with the Go team and our long-term interest to keep those two working.