Repository navigation
CI: fix ASan shard getting killed - #9615
Open
daywalker90 wants to merge 3 commits into
Open
daywalker90 wants to merge 3 commits into
daywalker90 wants to merge 3 commits into
Conversation
daywalker90
force-pushed
the
stream-mem-usage
branch
from
October 9, 2026 11:12
c6e0e47 to
66b369b
Compare
daywalker90
marked this pull request as draft
October 9, 2026 14:53
daywalker90
force-pushed
the
stream-mem-usage
branch
from
October 12, 2026 15:46
66b369b to
74e1858
Compare
`if: failure()` does not trigger when the VM gets killed, neither would `if: always()`. Instead we print used/available memory and the top five processes by total RSS every 15 seconds during the test step, so the readings before a kill survive in the job log. Per-process RSS alone was misleading since many lightningd processes each show small RSS. Changelog-None
test_parallel_opens starts about 600 channeld processes at full size. Under ASan that peaks near 33 GB, more than a 16 GB CI runner has, so the sanitizer shard is killed. Use 2 pairs x 10 opens when ASAN=1. The code path is unchanged and the full size still runs everywhere else. test_funding_cancel_race peaks near 15 GB PSS under ASan with 20 nodes, close to the 16 GB limit. Use 10 nodes when ASAN=1 (peak about 8 GB). The race is still exercised against each node.
Under ASan a handful of tests each need several GB and some reach
15-20 GB. When they land in the same shard and run together, the
shard is killed by the 16 GB runner. Mark them xdist_group('heavy')
and run the ASan/UBSan job with --dist loadgroup so they share one
worker and never overlap. Other jobs do not use loadgroup, so the
marker has no effect there.
daywalker90
changed the title
daywalker90
force-pushed
the
stream-mem-usage
branch
from
October 12, 2026 17:10
74e1858 to
7554971
Compare
daywalker90
marked this pull request as ready for review
October 12, 2026 17:12
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
add logging of memory usage instead of the dmesg step which would never trigger on a VM OOM kill. Also identified all tests using alot of memory during ASan and scaled them down. Also put them in the same xdist group so they run in serial.
Changelog-None