Repository navigation
CircleCI build on Darwin (arm64) frequently segfaults and sometimes deadlocks #3556
Description
There seems to be a data race which only triggers on darwin_arm64. This is particularly bad, because sometimes Please doesn't detect it and instead gets deadlocked, and then spends 3 hours burning CircleCI credits to do nothing
WARNING: DATA RACE
Write at 0x00c000229b30 by goroutine 55:
runtime.recvDirect()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/chan.go:405 +0x7c
github.com/thought-machine/please/src/output.(*buildingTargets).handleOutput.(*BuildState).Stop.func1()
/Users/distiller/project/src/core/state.go:419 +0x40
sync.(*Once).doSlow()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/sync/once.go:78 +0x94
sync.(*Once).Do()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/sync/once.go:69 +0x40
github.com/thought-machine/please/src/core.(*BuildState).Stop()
/Users/distiller/project/src/core/state.go:418 +0x2e4
github.com/thought-machine/please/src/output.(*buildingTargets).handleOutput()
/Users/distiller/project/src/output/targets.go:107 +0x288
github.com/thought-machine/please/src/output.(*buildingTargets).ProcessResult.deferwrap1()
/Users/distiller/project/src/output/targets.go:84 +0x34
runtime.deferreturn()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/panic.go:668 +0x5c
github.com/thought-machine/please/src/output.MonitorState()
/Users/distiller/project/src/output/shell_output.go:57 +0x7c0
main.runPlease.func1()
/Users/distiller/project/src/please.go:1230 +0x94
Previous read at 0x00c000229b30 by goroutine 17861:
runtime.chansend1()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/chan.go:162 +0x2c
github.com/thought-machine/please/src/core.(*BuildState).addPendingParse.func1()
/Users/distiller/project/src/core/state.go:355 +0xec
Goroutine 55 (running) created at:
main.runPlease()
/Users/distiller/project/src/please.go:1229 +0x78c
main.Please()
/Users/distiller/project/src/please.go:1197 +0xe68
main.runBuild()
/Users/distiller/project/src/please.go:1341 +0x200
main.init.func1()
/Users/distiller/project/src/please.go:478 +0x54
main.execute()
/Users/distiller/project/src/please.go:1623 +0x6a0
main.main()
/Users/distiller/project/src/please.go:1627 +0x44
Goroutine 17861 (finished) created at:
github.com/thought-machine/please/src/core.(*BuildState).addPendingParse()
/Users/distiller/project/src/core/state.go:351 +0x190
github.com/thought-machine/please/src/core.(*BuildState).queueTarget()
/Users/distiller/project/src/core/state.go:1100 +0x4bc
github.com/thought-machine/please/src/core.(*BuildState).queueTargetAsync()
/Users/distiller/project/src/core/state.go:1181 +0x154
github.com/thought-machine/please/src/core.(*BuildState).queueResolvedTarget.func1.gowrap1()
/Users/distiller/project/src/core/state.go:1161 +0x58
==================
==================
WARNING: DATA RACE
Write at 0x00c000229c10 by goroutine 55:
runtime.recvDirect()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/chan.go:405 +0x7c
github.com/thought-machine/please/src/output.(*buildingTargets).handleOutput.(*BuildState).Stop.func1()
/Users/distiller/project/src/core/state.go:420 +0x58
sync.(*Once).doSlow()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/sync/once.go:78 +0x94
sync.(*Once).Do()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/sync/once.go:69 +0x40
github.com/thought-machine/please/src/core.(*BuildState).Stop()
/Users/distiller/project/src/core/state.go:418 +0x2e4
github.com/thought-machine/please/src/output.(*buildingTargets).handleOutput()
/Users/distiller/project/src/output/targets.go:107 +0x288
github.com/thought-machine/please/src/output.(*buildingTargets).ProcessResult.deferwrap1()
/Users/distiller/project/src/output/targets.go:84 +0x34
runtime.deferreturn()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/panic.go:668 +0x5c
github.com/thought-machine/please/src/output.MonitorState()
/Users/distiller/project/src/output/shell_output.go:57 +0x7c0
main.runPlease.func1()
/Users/distiller/project/src/please.go:1230 +0x94
Previous read at 0x00c000229c10 by goroutine 17897:
runtime.chansend1()
/Users/distiller/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.1.darwin-arm64/src/runtime/chan.go:162 +0x2c
github.com/thought-machine/please/src/core.(*BuildState).addPendingBuild.func1()
/Users/distiller/project/src/core/state.go:366 +0x6c
Goroutine 55 (running) created at:
main.runPlease()
/Users/distiller/project/src/please.go:1229 +0x78c
main.Please()
/Users/distiller/project/src/please.go:1197 +0xe68
main.runBuild()
/Users/distiller/project/src/please.go:1341 +0x200
main.init.func1()
/Users/distiller/project/src/please.go:478 +0x54
main.execute()
/Users/distiller/project/src/please.go:1623 +0x6a0
main.main()
/Users/distiller/project/src/please.go:1627 +0x44
Goroutine 17897 (finished) created at:
github.com/thought-machine/please/src/core.(*BuildState).addPendingBuild()
/Users/distiller/project/src/core/state.go:362 +0xe4
github.com/thought-machine/please/src/core.(*BuildState).queueTargetAsync()
/Users/distiller/project/src/core/state.go:1216 +0x50c
github.com/thought-machine/please/src/core.(*BuildState).queueResolvedTarget.func1.gowrap1()
/Users/distiller/project/src/core/state.go:1161 +0x58
Activity
Make the build-darwin CircleCI job time out after 15 minutes (#3557)
peterebden commentedon Jul 24, 2026
Strange that this triggers only on darwin_arm64. Also strange how it reports the data race - the 'write' line is closing a channel, the 'read' side is writing an item to that channel? Maybe it is trying to warn us about racing the two (which is legit, although we somewhat messily handle the potential panic).
It might be quite hard to fix that issue though without some major surgery on those core channels. I had some ideas about this some time back but never finished it (not sure if it is on a branch on my fork or lost to the mists of history now).
sean- commentedon Jul 24, 2026
Maybe related? #3512 I chased a few of these a while back but all of my PRs are still sitting there so I haven't made any noise about them recently.
peterebden commentedon Aug 29, 2026
Yeah, sorry about that. I'm trying to catch up now I'm (somewhat) back working on Please again.
Currently I have #3569 as a big-bang (hopefully) fix for a lot of internal parallelism stuff. Still need to get that landed & confirm everything works as expected, but I'm hoping that either solves some of these or makes them easier to reason about. (I think I found one potential existing deadlock along the way...).