No space left on macOS host?

[targos]

See https://ci.nodejs.org/job/node-test-commit-osx/44401/nodes=osx1015/#showFailuresLink

[richardlau]

🤷
There have been three node-test-commit-osx builds on the same machine, https://ci.nodejs.org/computer/test-nearform-macos10.15-x64-3/builds, since then which all passed:

And

test-nearform-macos10:~ iojs$ df -h /Users/iojs/build/workspace/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   25Gi   15Gi    63%  309782 536799578    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$

[richardlau]

hmm. https://ci.nodejs.org/job/node-test-commit-osx/44429/nodes=osx1015/console also ran out of space on the same machine, test-nearform-macos10.15-x64-3.

13:54:44 fatal error: /Library/Developer/CommandLineTools/usr/bin/libtool: can't write to output file (No space left on device)
13:55:20 make[2]: *** [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/out/Release/libv8_initializers.a] Error 1

Again we have around 13G of free space.

test-nearform-macos10:~ iojs$ df -h
Filesystem      Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s5    51Gi   10Gi   13Gi    44%  487478 536621882    0%   /
devfs          186Ki  186Ki    0Bi   100%     642         0  100%   /dev
/dev/disk1s1    51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
/dev/disk1s4    51Gi  1.0Gi   13Gi     8%       2 537109358    0%   /private/var/vm
map auto_home    0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
test-nearform-macos10:~ iojs$ df -h /Users/iojs/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$ df -h /Users/iojs/build/workspace/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$ df -h /tmp
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$

The test-commit job does a git clean at the end. So somehow the job is chewing through 13G while running? (!)

[richardlau]

@AshCripps do you know if it's normal for /dev and /System/Volumes/Data/home to be 100% used on macs?

[AshCripps]

im going to say no, /System/Volumes/Data is the map of the user directory IIRC so its likely is a full disk. Catalina changed this reporting into a whole weird mess.

[tniessen]

So far, all three CI runs on nodejs/node#42628 failed with ENOSPC on osx.

[richardlau]

I'm just more and more confused:

[sxa]

FYI the builds really are using about 14GB while in flight:
During build (while running tests):

/dev/disk1s1    53710936 39596968   1474908    97%  284892 536824468    0%   /System/Volumes/Data

After build (Workspace cleared up):

Filesystem    1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s5     53710936 10873180  14855648    43%  487478 536621882    0%   /

Some of the files in out/Release are pretty big - these two account for 5GB alone;

3004068	libv8_base_without_compiler.a
2001752	libv8_initializers.a

and (probably obviously from the above) obj.target is quite large at over 6Gb

So we need to either get these expanded or find something else to delete. The disks are showing as around 51Gb at the moment. /var/db is quite large at about 8.7Gb but I don't know whether anything can be done about that. ccache for the iojs user is using 4.5Gb

[sxa]

Note - the -1 machine has a slightly larger disk than -2 and -3:

[sxa@sainz .ssh]$ for A in 1 2 3; do ssh test-nearform-macos10.15-x64-$A df -k .; done
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    68359372 26111720  29608124    47%  293179 683300541    0%   /System/Volumes/Data
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    53710936 24257204  16802816    60%  296611 536812749    0%   /System/Volumes/Data
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    53710936 39309828   1762048    96%  284926 536824434    0%   /System/Volumes/Data
[sxa@sainz .ssh]$ 

[AshCripps]

Note these machines cant really be any bigger as the host they are on started hitting issues when the VMs were a bigger size. (This was the roughly the largest size that was stable)

[sxa]

issues when the VMs were a bigger size.

Is that disk size specifically? Any idea what the issues were? I could understand if RAM/CPU allocation was overdone it would lead to issues but disk seems a little odd.

[AshCripps]

I think it was due to apples autoupdate feature meaning the host needed enough headroom other it crashes - im sure there was an issue on this in the build repo.

[richardlau]

Jenkins has automatically taken test-nearform-macos10.15-x64-3 offline due to low disk space.

We're now also seeing disk space issues on test-orka-macos11-x64-2,
e.g. https://ci.nodejs.org/job/node-test-commit-osx/44542/nodes=osx11-x64/console

10:55:10 not ok 3236 pummel/test-fs-readfile-tostring-fail
10:55:10   ---
10:55:10   duration_ms: 0.203
10:55:10   severity: fail
10:55:10   exitcode: 7
10:55:10   stack: |-
10:55:10     /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/pummel/test-fs-readfile-tostring-fail.js:69
10:55:10       throw err;
10:55:10       ^
10:55:10     
10:55:10     [Error: ENOSPC: no space left on device, write] {
10:55:10       errno: -28,
10:55:10       code: 'ENOSPC',
10:55:10       syscall: 'write'
10:55:10     }
10:55:10     
10:55:10     Node.js v18.0.0-pre
10:55:10   ...

[richardlau]

For test-orka-macos11-x64-2 I've removed /Users/iojs/build/workspace/node-test-commit-osx-as-shared-mdawson which was 1.5GB.