Repository navigation
OSX tests are stuck #3887
Description
Activity
Alpine might be facing a similar problem. It's stuck on CITGM: https://ci.nodejs.org/job/citgm-smoker/3472/
Hey folks,
I'm trying to finish the CI for nodejs/node#54560 (target is today) and osx test is taking too long (it seems stuck) >12h~
It's not stuck -- https://ci.nodejs.org/job/node-test-commit-osx/60982/ hasn't started yet (it's been waiting for a osx11-x64 machine for 12 hours).
https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx11-x64/ is incredibly backlogged -- looks like we only have one machine serving jobs.
Running the cleanup script on test-orka-macos11-x64-2.
Reacted by Ulises Gascón and Bohdantest-orka-macos11-x64-2 is back online and processing jobs. Backlog is currently 16 queued jobs (+1 citgm job) for osx11-x64.
Reacted by Ulises Gascón and BohdanGetting
java.io.IOException: No space left on deviceon test-orka-macos11-x64-1e.g.: https://ci.nodejs.org/job/node-test-commit-osx/61013/nodes=osx11-x64/console
Reacted by Matteo CollinaRan the cleanup script on test-orka-macos11-x64-1.
Disk space is too low again on test-orka-macos11-x64-2. What's happening?
Now both nodes are offline. We can't keep cleaning them manually every day.
Have the macOS Node.js builds got considerably larger?
18 GB (#3878 (comment)) sounds really high -- builds on Linux are only ~3 GB.
According to my local folders:
18G canary/out/Release 18G node/out/Release 17G v20.x/out/Release 18G v22.x/out/ReleaseI don't have a v18.x build.
Reacted by Richard LauI've run the cleanup script on test-orka-macos11-x64-2 and rebooted the machine. This is now reporting ~21GB of free space -- a node-test-commit-osx run has started and it is currently consuming 1.8 GB of that (and would be expected to grow to 18 GB).
test-orka-macos11-x64-2:~ iojs$ du -hs build/workspace/node-test-commit-osx/ 1.8G build/workspace/node-test-commit-osx/ test-orka-macos11-x64-2:~ iojs$ df -h Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk2s5s1 90Gi 14Gi 21Gi 41% 553788 941116212 0% / devfs 188Ki 188Ki 0Bi 100% 650 0 100% /dev /dev/disk2s4 90Gi 1.0Mi 21Gi 1% 1 941669999 0% /System/Volumes/VM /dev/disk2s2 90Gi 305Mi 21Gi 2% 1038 941668962 0% /System/Volumes/Preboot /dev/disk2s6 90Gi 592Ki 21Gi 1% 17 941669983 0% /System/Volumes/Update /dev/disk2s1 90Gi 54Gi 21Gi 73% 529196 941140804 0% /System/Volumes/Data map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /System/Volumes/Data/home test-orka-macos11-x64-2:~ iojs$I suppose one other question -- where is the tmp dir on the macOS machines?
/tmp/looks surprisingly empty when we know that the tests are leaving behindnode-coverage-*directories (#3864 -- I assume behaviour on macOS would be the same and these directories are being written somewhere).echo $TMPDIRshould print the location. I think the actual value is random and different on each macOS installation.13 remaining items
- added a commit that references this issue
on Sep 11, 2024 - added 3 commits that reference this issue
on Sep 12, 2024 - added 2 commits that reference this issue
on Sep 22, 2024 This was a symptom of having long lived OSX runners, which will soon be fixed by our transition to ephemeral Orka macos runners.
In the meantime, the disk was filling because macos spotlight indexing was indexing the builds , and creating new, unique uuid to filename mappings, which was filling up /private/var/db/uuidtext
I've disabled spotlight, and removed the spotlight databases.
We've now got 31GB free with a workspace on
test-orka-macos11-x64-1
And 52 GB free on
test-orka-macos11-x64-2
both of the orka-macos10.15-x64 machines have over 50GB available as well.
We should be able to re-enable these now, and they should last until they are replaced, shortly.
Reacted by Richard Lau and Ulises Gascón- added 2 commits that reference this issue
on Oct 2, 2024 All of the legacy OSX machines have been running successfully now for about 36 hours. The build results are very flappy between green and yellow status, and almost every yellow status is one particular test that keeps flagging as flaky:
Im not sure if that test is specifically flaky on OSX or if thats a problematic test in general, but its the vast majority of flaky results for the current OSX builds.
Can we close it?
Reacted by Antoine du Hamel and Ulises GascónCan we close it?
But part of what was being discussed here is that nodejs is using too much disk space to build. This still affects others, even if it does not affect your CI anymore. What has been done to fix that? Or is it being tracked elsewhere?
Hey folks,
I'm trying to finish the CI for nodejs/node#54560 (target is today) and osx test is taking too long (it seems stuck) >12h~