[Bug]: The CI has a high failure rate

[dkovba]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

1. Trigger PR checks.

Current behavior

There is a high chance of getting this:

2025-06-13T23:58:46+0000 error com.apple.containerization : [containerization_integration] ❌ test multiple concurrent processes with output stress failed: internal error (13): startProcess: failed to start process: Error Domain=NSPOSIXErrorDomain Code=9 "Bad file descriptor"
2025-06-13T23:58:46+0000 info com.apple.containerization : [containerization_integration] 
Integration suite completed in 19.997790932655334s with 8/10 passed and 1/10 skipped!
2025-06-13T23:58:46+0000 error com.apple.containerization : [containerization_integration] ❌

Expected Behavior

The tests pass.

Environment

Reproducible in CI only.

Code of Conduct

• I agree to follow this project's Code of Conduct

[dkovba]

Please also see apple/container#199.

[adityaramani]

Can you please share a link to a failed job?

[dkovba]

Can you please share a link to a failed job?

https://github.com/apple/containerization/actions/runs/15645903676/job/44083226358

[adityaramani]

Was able to repro this locally - adding some failure logs from one run. I noticed that the failure reason changes b/w runs. Maybe there are multiple cases where the guest agent fails?

2025-06-14T17:01:29+0000 debug vminitd : [vminitd] in supervisor lock to start process
Assertion failed: getStrongExtraRefCount() >= dec && "releasing reference whose refcount is already zero" (/home/build-user/source/swift-project/swift/stdlib/public/SwiftShims/swift/shims/RefCount.h: decrementStrongExtraRefCount: 578)
2025-06-14T17:01:29+0000 debug vminitd : id=exec-65 [vminitd] starting managed process
[    2.108787] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[    2.110375] CPU: 3 UID: 0 PID: 1 Comm: vminitd Not tainted 6.12.28 #1
[    2.110488] Call trace:
[    2.110629]  dump_backtrace+0x9c/0xd0
[    2.110713]  show_stack+0x14/0x1c
[    2.110788]  dump_stack_lvl+0xc4/0xe8
[    2.110837]  dump_stack+0x14/0x1c
[    2.110866]  panic+0x378/0x38c
[    2.110905]  do_exit+0x6e8/0x898
[    2.110940]  do_group_exit+0x30/0x90
[    2.110963]  pid_child_should_wake+0x0/0x60
[    2.111008]  invoke_syscall.constprop.0+0x58/0xf4
[    2.111170]  do_el0_svc+0x44/0xc8
[    2.111268]  el0_svc+0x50/0x1b8
[    2.111328]  el0t_64_sync_handler+0x134/0x150
[    2.111419]  el0t_64_sync+0x160/0x164
[    2.111449] SMP: stopping secondary CPUs
[    2.111588] Kernel Offset: 0x211accc00000 from 0xffff800080000000
[    2.111782] PHYS_OFFSET: 0xffffdb1480000000
[    2.111815] CPU features: 0x00,00000013,00222908,4203f403
[    2.111882] Memory Limit: none
[    2.111917] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 ]---

[elijah-wright]

I've looked at this for a few hours and I think the problem is because the test is configured to cause a condition where there's still unread data after the write file handles are closed. the kernel will deliver EPOLLIN | EPOLLHUP, which skips the readyToRead check in StandardIO.relay() and hits the .eof branch. that deallocates the buffer by calling cleanupRelay. the kernel then sends EPOLLHUP, which goes through the early guard and calls cleanupRelay. UnsafeMutableBufferPointer doesn't throw in cleanupRelay. I think the solution here is to only run cleanupRelay once. I can't test this because GitHub Actions doesn't have a runner for macOS Tahoe and I don't have a Mac but I could write a fix

[adityaramani]

Thanks! I was going down a different path, I think the writeFd in the relay method were being closed when the StandardIO object was deinited before the IO was fully piped, and the cleanupRelay was trying to call close on it again.

I can try your theory out as well, it maybe the case that both these things are happening - else you can create a PR and we can run the changes in the runners we have here.

[elijah-wright]

I only say that because I saw getStrongExtraRefCount() and deallocate() in your logs, which is a double free, not just an EBADF from close(writeFd)

I just created #153

[dkovba]

A link to a recent failed build: https://github.com/apple/containerization/actions/runs/15860447603/job/44716096085