x64 V8 builds run out of memory

[targos]

Example: https://ci.nodejs.org/job/node-test-commit-v8-linux/6426/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/console

Here's what I see after some time if I run the build manually:

[targos]

It happens while running:

PATH=/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin ninja -C out.gn/x64.release/  d8 cctest inspector-test

[richardlau]

According to https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/ it looks like this started on 13 Feb 2025.

https://ci.nodejs.org/job/node-test-commit-v8-linux/6409/ (Feb 12) passed on nodejs/node@9ce1fff.
https://ci.nodejs.org/job/node-test-commit-v8-linux/6410/ (Feb 13) failed on nodejs/node@a7f648c.

I don't think any of nodejs/node@9ce1fff...a7f648c would explain the breakage though.

[targos]

Oh, I really thought it was the V8 update. I didn't see it also failed with main.

[targos]

apt logs around that date:

Start-Date: 2025-02-11  06:44:41
Commandline: /usr/bin/unattended-upgrade
Upgrade: vim:amd64 (2:8.2.3995-1ubuntu2.22, 2:8.2.3995-1ubuntu2.23), vim-common:amd64 (2:8.2.3995-1ubuntu2.22, 2:8.2.3995-1ubuntu2.23), vim-tiny:amd64 (2:8.2.3995-1ubuntu2.22, 2:8.2.3995-1ubuntu2.23), vim-runtime:amd64 (2:8.2.3995-1ubuntu2.22, 2:8.2.3995-1ubuntu2.23)
End-Date: 2025-02-11  06:44:44

Start-Date: 2025-02-11  06:44:45
Commandline: /usr/bin/unattended-upgrade
Upgrade: xxd:amd64 (2:8.2.3995-1ubuntu2.22, 2:8.2.3995-1ubuntu2.23)
End-Date: 2025-02-11  06:44:46

Start-Date: 2025-02-13  06:01:41
Commandline: /usr/bin/unattended-upgrade
Upgrade: tzdata:amd64 (2024b-0ubuntu0.22.04, 2024b-0ubuntu0.22.04.1)
End-Date: 2025-02-13  06:01:42

Start-Date: 2025-02-18  06:22:15
Commandline: /usr/bin/unattended-upgrade
Upgrade: intel-microcode:amd64 (3.20241112.0ubuntu0.22.04.1, 3.20250211.0ubuntu0.22.04.1)
End-Date: 2025-02-18  06:22:24

[targos]

I can only guess that something changed in Chromium's tooling (depot_tools et al).

[richardlau]

I saw https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/6438/ was running so ssh'ed into the machine to see what it was doing while it was running.

The build appears to be spawning a lot of python3 processes:

root@test-hetzner-ubuntu2204-x64-2 ~ # ps -ef | grep python3
...
iojs     2611881 2611878  0 10:41 ?        00:00:00 python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test
iojs     2611884 2611881  0 10:41 ?        00:00:00 python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test
iojs     2611887 2611884  0 10:41 ?        00:00:00 python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test
iojs     2611890 2611887  0 10:41 ?        00:00:00 python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test
iojs     2611893 2611890  0 10:41 ?        00:00:00 python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test
iojs     2611896 2611893  0 10:41 ?        00:00:00 python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test
iojs     2611899 2611896  0 10:41 ?        00:00:00 python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test
iojs     2611902 2611899  0 10:41 ?        00:00:00 python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test
root@test-hetzner-ubuntu2204-x64-2 ~ # ps -ef | grep python3 | wc -l
13160
root@test-hetzner-ubuntu2204-x64-2 ~ #

all of them with the command line:

python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools/ninja.py -C out.gn/x64.release/ -j 20 d8 cctest inspector-test

[targos]

Maybe this ninja.py proxy is the new thing?

[richardlau]

Maybe related to https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6259139? It landed on the day our CI started failing.

[richardlau]

(I'm in a meeting right now, but my suspicion is the proxy is spawning itself instead of the actual ninja binary.)

[cclauss]

Would lowering -j 20 give us more visibility or use less memory?