Skip to content

fix(dev-lead): multi-label issue creation can cancel the dev-lead dispatch (race) #443

Description

@don-petry

Summary

When an issue is created (or updated) with the dev-lead label plus one or more other labels in the same API call, the dev-lead agent can silently fail to be dispatched. GitHub fires a separate issues:labeled event per label; the dispatch job only acts when the triggering label is dev-lead, while a per-issue cancel-in-progress concurrency group lets a later (non-dev-lead) labeled event cancel the very run that would have acted. Net effect: the workflow reports success, but the agent never runs and no PR is opened.

Observed behaviour (concrete evidence)

While working issue #438 (created via gh issue create --label "dev-lead,bug,ci,automation"):

  • Four issues:labeled events fired at ~21:10, all sharing concurrency group dev-lead-issue-438.
  • Runs 27306552438 and 27306552128 completed success but the dispatch job classified intent as:
    INTENT_TYPE: skip
    INTENT_REASON: label-not-dev-lead
    
    i.e. they were triggered by the bug/ci/automation label events and correctly skipped.
  • The other two runs were cancelled by cancel-in-progress — and the run triggered by the dev-lead label was among them, so it never dispatched the agent.
  • Result after 25 min: issue OPEN, 0 comments, no in-progress label, no branch, no PR. A "successful" workflow that did nothing.
  • Recovery: cycling the label alone (remove + re-add dev-lead, no competing events) fired run 27308171856
    INTENT_TYPE: issue
    INTENT_REASON: issue-labeled-dev-lead
    
    agent ran (Pre-flight → Install engine CLIs → Run issue, all success) → PR feat: implement issue #438 — fix(compliance): remove dev-lead workflow manipulation from re-trigger sweep (Step 1 is out of scope) #442 opened.

This also means such issues are not self-healing: the daily compliance-retrigger sweep only re-triggers issues carrying the compliance-audit label, so a plain dev-lead issue stuck this way stays stuck indefinitely.

Root Cause

Per standards/ci-standards.md:421-428, the issue-intent job is gated and serialised as:

if: >-
  github.event_name == 'issues' && github.event.action == 'labeled' &&
    github.event.label.name == 'dev-lead'
concurrency:
  group: dev-lead-issue-${{ github.event.issue.number }}
  cancel-in-progress: true

Two facts combine into the race:

  1. Gate keys on the triggering label (github.event.label.name), not on whether the issue currently has the dev-lead label. So only the single event for the dev-lead label can act.
  2. cancel-in-progress collapses all same-issue events, and the cancellation happens at the workflow/concurrency layer before the if: decides a run is a no-op. So a non-acting run (wrong label) can cancel the one acting run (dev-lead label) purely based on arrival order.

When dev-lead is not the last label applied, the acting run loses the race.

Reproduction

  1. gh issue create --repo <org>/<repo> --label "dev-lead,bug" --title t --body b
  2. Observe the dev-lead runs: the dev-lead-triggered run is cancelled; survivors skip with label-not-dev-lead.
  3. Issue receives no agent dispatch (no plan comment, no PR).
  4. Confirm recovery by cycling dev-lead alone.

Recommended Fix

Gate on the issue's current label set rather than the triggering label, and let per-issue concurrency collapse the duplicate events onto a single run that still acts:

if: >-
  github.event_name == 'issues' && github.event.action == 'labeled' &&
    contains(github.event.issue.labels.*.name, 'dev-lead')
concurrency:
  group: dev-lead-issue-${{ github.event.issue.number }}
  cancel-in-progress: true

With this, every one of the N labeled events passes the gate; cancel-in-progress collapses them to the last-arriving run, which now acts (because the gate no longer depends on which label triggered it). Pair with an idempotency guard so re-entrant events can't double-dispatch (skip if a dev-lead/issue-<n>-* PR or fresh in-progress label already exists — the same check dl_dev_lead_active already encodes).

Apply the same fix to the org template (standards/workflows/dev-lead.yml / the reusable) and roll it to the fleet.

Acceptance Criteria

  • Creating an issue in a single API call with dev-lead plus ≥1 other label dispatches the agent exactly once (not zero).
  • Dispatch does not depend on dev-lead being the last label applied.
  • Adding dev-lead to an issue that already carries other labels (single add) still dispatches.
  • At most one concurrent agent run per issue; simultaneous events never produce duplicate PRs.
  • Label cycling / re-trigger recovery path (relied on by compliance-retrigger.sh) still works.
  • Regression test simulates multiple near-simultaneous issues:labeled events where dev-lead is not last, and asserts exactly one agent dispatch.
  • Standard (standards/ci-standards.md) and template updated to match.

Context

Discovered while monitoring #438 (whose fix, PR #442, is unrelated to this race). Part of the Compliance program / dev-lead reliability work. Sibling to #431/#432 (lost-labeled-event recovery).

Note: when assigning this to dev-lead, add the dev-lead label on its own (or cycle it last) to avoid triggering the very race described here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    automationAutomation improvements and gapsbugBug reportsciCI/CD pipeline issuesdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions