Repository navigation
fix(ci): break the aggregator starvation loop — 240 min deadline + Docker digest batching - #1127
Merged
Merged
Conversation
…cker digest batching No PR merged on this repository between 2026-06-29 and 2026-08-30. master was green on all 26 required checks the whole time; the blocker was a self-reinforcing loop in the merge gate. Required Checks Aggregator polls until every sibling workflow on the SHA reaches a terminal state, and treats a sibling still queued/in_progress at the deadline as a failure. The deadline was 90 minutes — itself a remediation after a 30-minute deadline "failed every PR" on 2026-05-09. A two-month Renovate backlog then opened ~44 concurrent dependency PRs, the queue again outran the deadline, and aggregators expired with no failing check (job 89636653774 ran 90m14s and died on "Build — Ubuntu gcc (CPU) + DNN: in_progress; CodeQL: queued; ShellCheck + shfmt: queued"). Because the aggregator is required, the expiry blocks the merge, which keeps the queue deep, which causes the next expiry. Two coordinated changes, one per half of the loop: - Deadline 90 -> 240 minutes, timeout-minutes 100 -> 250. The job only polls the checks API, so a longer ceiling costs a runner slot rather than compute; the hosted-runner hard cap is 360 minutes. - Batch Docker digest/pin refreshes into one Renovate PR. This was the only dependency class without a grouping rule, so ~11 base images each opened a separate PR and a separate full CI matrix run. ADR-1123. Note the allocator returned 1121, which collides with contributor PR #1081 — scripts/adr/next-free.sh queries origin only and cannot see ADR files on contributor forks. 1121 was released and 1123 claimed by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
11 tasks done
lusoris
enabled auto-merge (squash)
August 30, 2026 09:54
This was referenced Aug 30, 2026
This was referenced Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Breaks the loop that kept every PR unmergeable between 2026-06-29 and
2026-08-30 — including six
[SECURITY]dependency updates.Why
masterwas green on all 26 required checks the entire time. The blocker wasthe merge gate eating itself.
Required Checks Aggregatorpolls until every sibling workflow on the SHAreaches a terminal state, and treats a sibling still
queued/in_progressatthe deadline as a failure. The deadline was 90 minutes — itself already a
remediation, after a 30-minute deadline "failed every PR with
queuedfor twohours straight" on 2026-05-09.
A two-month Renovate backlog then opened ~44 concurrent dependency PRs. The
queue outran 90 minutes and aggregators expired with no failing check:
The aggregator is a required check, so the expiry blocks the merge → the PR
stays open → it re-runs CI on every master push → the queue stays deep → the next
aggregator expires. It does not drain on its own.
The fix — one change per half of the loop
1. Deadline 90 → 240 min (
timeout-minutes100 → 250). Makes the gate survivea deep queue. This job only polls the checks API — it runs no build — so a longer
ceiling costs a runner slot, not compute. Hosted-runner hard cap is 360 min.
2. Batch Docker digest/pin refreshes into one Renovate PR. Docker was the
only dependency class with no grouping rule (Actions, Go, Cargo and
Python-patch are all already grouped), so ~11 base images — debian, fedora,
ubuntu, golang, archlinux, python, both distroless variants,
docker/dockerfile,otel-collector, nvidia/cuda — each opened its own PR and its own full matrix run.
Makes the queue shallower at the source.
Deep-dive deliverables
ADR-1123§Alternatives considered (5 options, incl. two rejected outright for punching a correctness hole in the gate).changelog.d/changed/ci-throughput-aggregator-renovate.mdReproducer / smoke-test command
Both clean on this branch. The renovate.json edit is a surgical 7-line insert —
no reformatting of the surrounding file.
Bug status hygiene
no state delta: CI-throughput and dependency-batching configuration; no tracked
bug opened, closed, or reclassified.
Note for the reviewer
scripts/adr/next-free.sh --claimreturned 1121, which collides withcontributor PR #1081 (
ADR-1121, SYCL QSV zero-copy). The allocator queriesgit ls-remote --heads originand therefore cannot see ADR files oncontributor forks. 1121 was released and 1123 claimed by hand (1122 is
taken by contributor PR #1082). Worth a follow-up to teach the allocator about
open fork PRs.