Parallel workers running install scripts can interfere

[janv]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

We have a fairly large package.json. Since switching to yarn, our CI builds would randomly fail around 7% of the time when compiling binary modules.

What was super weird is that the C compiler complained about files that were apparently truncated. Those files were node.js header files, downloaded by node-gyp before the compilation process. Since gyp verifies the integrity of these files, they can't be corrupted at the source. The corruptions occurred in different files and on different lines every time.

After a lot of trial and error, I was unable to reproduce the error when I only installed the single offending module alone. This leads me to believe that the parallelization of work during yarn install somehow leads to two parallel downloads of the node headers that overwrite each other.

I have no hard evidence and trouble reproducing this. I was just wondering if this is really a possibility or if I'm barking up the wrong tree. I searched in a multitude of other directions, all without result. The randomness of the occurrence indicates a concurrency issue.

Please advise

If the current behavior is a bug, please provide the steps to reproduce.

This is extremely hard to reproduce, see above description. If desired I can attach our package.json and yarn.lock

What is the expected behavior?

Since external scripts cannot be guaranteed to be concurrent, don't execute them in parallel.

[yunxing]

Interesting. Each worker should only work within their own sandbox (the directory of the downloaded source files). That being said, nothing can stop them writing files to each other's directory in the "install" fields.

Maybe try to config the number of concurrent workers to 1?

[janv]

In our particular case, the workers were running the install scripts, which in turn used node-gyp to compile binaries. Node-Gyp downloads headers into ~/.node-gyp/6.7.0/include/node/

[janv]

We are solving this now by performing a npm install -g node-gyp && node-gyp install before running CI

[janv]

I have a full log of the installation process now: https://gist.github.com/janv/f80454d2c631bfeac2d1966d69294d3b#file-parallel-node-gyp-L1182 (I stripped the progressTick lines)

Shows clearly that there are two parallel downloads of the node headers happening (for the fibers and fsevents modules)

[janv]

And a failing example: https://gist.github.com/janv/15c14873bb4d9e2909b66e204c5d2615#file-failing-parallel-install-L39-L43

This is the exact same build, restarted a bunch of times until the error showed up.

In line 17 the fibers worker finishes downloading the headers.
In line 39 the fsevents worker starts downloading the headers.
In line 40, now the fibers worker comes back in and starts unpacking the archive that is concurrently being overwritten by the fsevents worker's download

Later we see the compiler barfing all over files it can't parse

[janv]

Even though the switch to yarn surfaced this issue, I think the problem lies more with node-gyp.
I opened an issue there: nodejs/node-gyp#1054

[Fishrock123]

node-gyp has messaged that it is probably an issue over here, at least in the short term

[bestander]

What would be a solution for this on Yarn side?
Considering that node-gyp scripts execution is outside of Yarn control.

[S-YOU]

may be add an command line option to turn off parallel compilation step can mitigate this issue.

[bestander]

@S-YOU, build parallelism is configured with a constant CHILD_CONCURRENCY https://github.com/yarnpkg/yarn/blob/master/src/constants.js#L30, feel free to send a PR to override it via CLI args or ENV var.

I am a bit worried about introducing this hack for the one package that does not support idempotent execution.
I would appeal to the maintainer to introduce a fix that allows concurrent installs, it is not only about yarn support, yarn just reveals this flaw with higher probability