Skip to content

fix(ci): break the aggregator starvation loop — 240 min deadline + Docker digest batching - #1127

Merged
lusoris merged 1 commit into
masterfrom
chore/ci-throughput-aggregator-renovate
Aug 30, 2026
Merged

lusoris merged 1 commit into
masterfrom
chore/ci-throughput-aggregator-renovate

Conversation

@lusoris

@lusoris lusoris commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

What

Breaks the loop that kept every PR unmergeable between 2026-06-29 and
2026-08-30 — including six [SECURITY] dependency updates.

Why

master was green on all 26 required checks the entire time. The blocker was
the merge gate eating itself.

Required Checks Aggregator polls until every sibling workflow on the SHA
reaches a terminal state, and treats a sibling still queued/in_progress at
the deadline as a failure. The deadline was 90 minutes — itself already a
remediation, after a 30-minute deadline "failed every PR with queued for two
hours straight"
on 2026-05-09.

A two-month Renovate backlog then opened ~44 concurrent dependency PRs. The
queue outran 90 minutes and aggregators expired with no failing check:

job 89636653774  03:24:39 → 04:54:53Z  (90m14s)
expired on: Build — Ubuntu gcc (CPU) + DNN: in_progress
            CodeQL (Actions): queued
            ShellCheck + shfmt: queued

The aggregator is a required check, so the expiry blocks the merge → the PR
stays open → it re-runs CI on every master push → the queue stays deep → the next
aggregator expires. It does not drain on its own.

The fix — one change per half of the loop

1. Deadline 90 → 240 min (timeout-minutes 100 → 250). Makes the gate survive
a deep queue. This job only polls the checks API — it runs no build — so a longer
ceiling costs a runner slot, not compute. Hosted-runner hard cap is 360 min.

2. Batch Docker digest/pin refreshes into one Renovate PR. Docker was the
only dependency class with no grouping rule (Actions, Go, Cargo and
Python-patch are all already grouped), so ~11 base images — debian, fedora,
ubuntu, golang, archlinux, python, both distroless variants, docker/dockerfile,
otel-collector, nvidia/cuda — each opened its own PR and its own full matrix run.
Makes the queue shallower at the source.

Deep-dive deliverables

  • Research digest — no digest needed: root cause read directly off the expired aggregator job logs and the workflow's own deadline comment.
  • Decision matrix — ADR-1123 §Alternatives considered (5 options, incl. two rejected outright for punching a correctness hole in the gate).
  • AGENTS.md invariant note — no rebase-sensitive invariants: fork-local CI workflow and Renovate config, no upstream counterpart.
  • Reproducer / smoke-test command — see below.
  • CHANGELOG fragment — changelog.d/changed/ci-throughput-aggregator-renovate.md
  • Rebase note — no rebase impact: fork-local CI configuration.

Reproducer / smoke-test command

# the two edits
grep -n 'timeout-minutes: 250'   .github/workflows/required-aggregator.yml   # line 32
grep -n '240 \* 60 \* 1000'      .github/workflows/required-aggregator.yml   # deadline

# config still parses, and the group exists exactly once
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/required-aggregator.yml')); print('YAML ok')"
python3 -c "import json; d=json.load(open('renovate.json')); \
  print('JSON ok,', len(d['packageRules']), 'rules;', \
  sum(r.get('groupName')=='Docker digests' for r in d['packageRules']), 'Docker-digest rule')"

Both clean on this branch. The renovate.json edit is a surgical 7-line insert —
no reformatting of the surrounding file.

Bug status hygiene

no state delta: CI-throughput and dependency-batching configuration; no tracked
bug opened, closed, or reclassified.

Note for the reviewer

scripts/adr/next-free.sh --claim returned 1121, which collides with
contributor PR #1081 (ADR-1121, SYCL QSV zero-copy). The allocator queries
git ls-remote --heads origin and therefore cannot see ADR files on
contributor forks
. 1121 was released and 1123 claimed by hand (1122 is
taken by contributor PR #1082). Worth a follow-up to teach the allocator about
open fork PRs.

…cker digest batching

No PR merged on this repository between 2026-06-29 and 2026-08-30. master was
green on all 26 required checks the whole time; the blocker was a
self-reinforcing loop in the merge gate.

Required Checks Aggregator polls until every sibling workflow on the SHA reaches
a terminal state, and treats a sibling still queued/in_progress at the deadline
as a failure. The deadline was 90 minutes — itself a remediation after a
30-minute deadline "failed every PR" on 2026-05-09. A two-month Renovate backlog
then opened ~44 concurrent dependency PRs, the queue again outran the deadline,
and aggregators expired with no failing check (job 89636653774 ran 90m14s and
died on "Build — Ubuntu gcc (CPU) + DNN: in_progress; CodeQL: queued;
ShellCheck + shfmt: queued"). Because the aggregator is required, the expiry
blocks the merge, which keeps the queue deep, which causes the next expiry.

Two coordinated changes, one per half of the loop:

- Deadline 90 -> 240 minutes, timeout-minutes 100 -> 250. The job only polls the
  checks API, so a longer ceiling costs a runner slot rather than compute; the
  hosted-runner hard cap is 360 minutes.
- Batch Docker digest/pin refreshes into one Renovate PR. This was the only
  dependency class without a grouping rule, so ~11 base images each opened a
  separate PR and a separate full CI matrix run.

ADR-1123. Note the allocator returned 1121, which collides with contributor
PR #1081 — scripts/adr/next-free.sh queries origin only and cannot see ADR files
on contributor forks. 1121 was released and 1123 claimed by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lusoris
lusoris enabled auto-merge (squash) August 30, 2026 09:54
This was referenced Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant