[wasm] Fix a race condition in adding payloads for helix #65995

radical · 2022-03-01T03:55:28Z

The issue shows up only in Wasm.Build.Tests for the
TestUsingWorkloads!=true case, as clang --version failing with:

/datadisks/disk1/work/A0100914/p/build/emsdk/upstream/bin/clang: error while loading shared libraries: libLLVM-13git.so: cannot open shared object file: No such file or directory (TaskId:231)

For this non-workloads testing case, we copy emscripten from the
system copy to the git checkout. And then use that directory for the
payload.

The build logs show that the missing file (libLLVM-13git.so) does get
copied as part of copying /usr/local/emscripten/emsdk to
$(RepoRoot)/src/mono/wasm/emsdk. But the file, and a few others seem
to be missing in the final helix payload.

We add emsdk to the payload for target path build/emsdk.

But a recent change also added node for this case with a target path
build/emsdk/node, with an overlapping path with build/emsdk. I
believe this is causing an issue where these directories are being
processed in parallel, and cause some files get missed.

This commit:

Add node only when needed (skip WBT for example);
Use a non-overlapping path for node, build/emsdk-node.

Fixes #65956

ghost · 2022-03-01T03:55:33Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2022-03-01T03:55:43Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

The issue shows up only in Wasm.Build.Tests for the
TestUsingWorkloads!=true case, as clang --version failing with:

/datadisks/disk1/work/A0100914/p/build/emsdk/upstream/bin/clang: error while loading shared libraries: libLLVM-13git.so: cannot open shared object file: No such file or directory (TaskId:231)

For this non-workloads testing case, we copy emscripten from the
system copy to the git checkout. And then use that directory for the
payload.

The build logs show that the missing file (libLLVM-13git.so) does get
copied as part of copying /usr/local/emscripten/emsdk to
$(RepoRoot)/src/mono/wasm/emsdk. But the file, and a few others seem
to be missing in the final helix payload.

We add emsdk to the payload for target path build/emsdk.

But a recent change also added node for this case with a target path
build/emsdk/node, with an overlapping path with build/emsdk. I
believe this is causing an issue where these directories are being
processed in parallel, and cause some files get missed.

This commit:

Add node only when needed (skip WBT for example);
Use a non-overlapping path for node, build/emsdk-node.

Fixes #65956

Author:	radical
Assignees:	radical
Labels:	`arch-wasm`, `area-Build-mono`
Milestone:	-

radical · 2022-03-01T03:55:49Z

/azp run runtime-wasm

azure-pipelines · 2022-03-01T03:56:03Z

Azure Pipelines successfully started running 1 pipeline(s).

The issue shows up only in Wasm.Build.Tests for the `TestUsingWorkloads!=true` case, as `clang --version` failing with: ``` /datadisks/disk1/work/A0100914/p/build/emsdk/upstream/bin/clang: error while loading shared libraries: libLLVM-13git.so: cannot open shared object file: No such file or directory (TaskId:231) ``` For this non-workloads testing case, we copy `emscripten` from the system copy to the git checkout. And then use that directory for the payload. The build logs show that the missing file (`libLLVM-13git.so`) does get copied as part of copying `/usr/local/emscripten/emsdk` to `$(RepoRoot)/src/mono/wasm/emsdk`. But the file, and a few others seem to be missing in the final helix payload. We add `emsdk` to the payload for target path `build/emsdk`. But a recent change also added `node` for this case with a target path `build/emsdk/node`, with an overlapping path with `build/emsdk`. I believe this is causing an issue where these directories are being processed in parallel, and cause some files get missed. This commit: 1. Add `node` only when needed (skip WBT for example); 2. Use a non-overlapping path for `node`, `build/emsdk-node`. Fixes dotnet#65956

radical · 2022-03-01T06:28:13Z

/azp run runtime-wasm

azure-pipelines · 2022-03-01T06:28:26Z

Azure Pipelines successfully started running 1 pipeline(s).

radekdoulik

Looks like WasmBuildTests are now using wrong node, resulting in [wasm test] [07:28:52] fail: Error: NodeJS at '/usr/local/nodejs/node-v10.17.0-linux-x64/bin/node' has too low version '10.17.0'

radekdoulik · 2022-03-01T11:35:19Z

_HelixLocalNodePath is set to wrong location?

+ ls /datadisks/disk1/work/BC460A00/p/build/emsdk-node
ls: cannot access '/datadisks/disk1/work/BC460A00/p/build/emsdk-node': No such file or directory
+ export _HELIX_NODEJS_VERSION=
+ export _HELIX_NODEJS_PATH=/datadisks/disk1/work/BC460A00/p/build/emsdk-node/

radekdoulik · 2022-03-01T12:49:53Z

/azp run runtime-wasm

azure-pipelines · 2022-03-01T12:50:07Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2022-03-01T16:36:03Z

The windows test failures are #65978 .
And the AOT failures in Regex tests are OOM, unrelated to this PR.

radical · 2022-03-10T20:48:39Z

/backport to release/7.0-preview2

github-actions · 2022-03-10T20:48:53Z

Started backporting to release/7.0-preview2: https://github.com/dotnet/runtime/actions/runs/1965439035

github-actions · 2022-03-10T20:50:25Z

@radical an error occurred while backporting to release/7.0-preview2, please check the run log for details!

Error: @radical is not a repo collaborator, backporting is not allowed. If you're a collaborator please make sure your Microsoft team membership visibility is set to Public on https://github.com/orgs/microsoft/people?query=radical

* [wasm] Fix a race condition in adding payloads for helix The issue shows up only in Wasm.Build.Tests for the `TestUsingWorkloads!=true` case, as `clang --version` failing with: ``` /datadisks/disk1/work/A0100914/p/build/emsdk/upstream/bin/clang: error while loading shared libraries: libLLVM-13git.so: cannot open shared object file: No such file or directory (TaskId:231) ``` For this non-workloads testing case, we copy `emscripten` from the system copy to the git checkout. And then use that directory for the payload. The build logs show that the missing file (`libLLVM-13git.so`) does get copied as part of copying `/usr/local/emscripten/emsdk` to `$(RepoRoot)/src/mono/wasm/emsdk`. But the file, and a few others seem to be missing in the final helix payload. We add `emsdk` to the payload for target path `build/emsdk`. But a recent change also added `node` for this case with a target path `build/emsdk/node`, with an overlapping path with `build/emsdk`. I believe this is causing an issue where these directories are being processed in parallel, and cause some files get missed. This commit: 1. Add `node` only when needed (skip WBT for example); 2. Use a non-overlapping path for `node`, `build/emsdk-node`. Fixes dotnet#65956 * Change `_HelixLocalNodePath` evaluation order Co-authored-by: Radek Doulik <radekdoulik@gmail.com> (cherry picked from commit 1665aca)

ghost assigned radical Mar 1, 2022

radical added arch-wasm WebAssembly architecture area-Build-mono labels Mar 1, 2022

radical force-pushed the fix-node-error branch from bfa0cd0 to 1711d6d Compare March 1, 2022 06:27

radical marked this pull request as ready for review March 1, 2022 06:27

radical requested review from lewing, maraf and radekdoulik March 1, 2022 06:28

radekdoulik reviewed Mar 1, 2022

View reviewed changes

Change _HelixLocalNodePath evaluation order

637feb7

radekdoulik approved these changes Mar 1, 2022

View reviewed changes

radical merged commit 1665aca into dotnet:main Mar 1, 2022

radical deleted the fix-node-error branch March 1, 2022 16:37

ghost locked as resolved and limited conversation to collaborators Apr 10, 2022

[wasm] Fix a race condition in adding payloads for helix #65995

[wasm] Fix a race condition in adding payloads for helix #65995

Uh oh!

Conversation

radical commented Mar 1, 2022

Uh oh!

ghost commented Mar 1, 2022

Uh oh!

ghost commented Mar 1, 2022

Uh oh!

radical commented Mar 1, 2022

Uh oh!

azure-pipelines bot commented Mar 1, 2022

Uh oh!

radical commented Mar 1, 2022

Uh oh!

azure-pipelines bot commented Mar 1, 2022

Uh oh!

radekdoulik left a comment

Choose a reason for hiding this comment

Uh oh!

radekdoulik commented Mar 1, 2022

Uh oh!

radekdoulik commented Mar 1, 2022

Uh oh!

azure-pipelines bot commented Mar 1, 2022

Uh oh!

radical commented Mar 1, 2022

Uh oh!

radical commented Mar 10, 2022

Uh oh!

github-actions bot commented Mar 10, 2022

Uh oh!

github-actions bot commented Mar 10, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants