Skip to content

Retry upload-asset pushes after concurrent branch updates - #51893

Merged
pelikhan merged 8 commits into
mainfrom
copilot/upload-asset-fix-concurrent-race
Aug 11, 2026
Merged

Retry upload-asset pushes after concurrent branch updates#51893
pelikhan merged 8 commits into
mainfrom
copilot/upload-asset-fix-concurrent-race

Conversation

Copilot AI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Concurrent workflow runs can publish assets from the same branch tip, causing one push to fail with a non-fast-forward rejection and leaving its asset unavailable.

  • Conflict recovery
    • Detect fetch first and non-fast-forward push rejections.
    • Fetch the exact remote assets branch, rebase, and retry up to three total attempts.
  • Failure handling
    • Fail immediately for permanent errors such as permission or branch-protection failures.
    • Preserve Git diagnostics when retries are exhausted.
  • Coverage
    • Cover successful conflict recovery, bounded retry exhaustion, and non-retryable failures.

Run URL: https://github.com/github/gh-aw/actions/runs/31450173394> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 24.1 AIC · ⌖ 5.21 AIC · ⊞ 6.1K ·

Comment /souschef to run again


@copilot quick triage: your follow-up already landed after the prior sous-chef nudge, and checks now look complete. Please do one final pr-finisher pass focused on the remaining maintainer confidence items from the Matt review response, confirm the branch is clean after the rebase-abort safeguard test, and hand back a short maintainer-facing summary.

Branch refresh was requested. Run: https://github.com/github/gh-aw/actions/runs/31454952282> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 15.9 AIC · ⌖ 5.21 AIC · ⊞ 8.5K ·

Comment /souschef to run again


Run: https://github.com/github/gh-aw/actions/runs/31454952282> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 15.9 AIC · ⌖ 5.21 AIC · ⊞ 8.5K ·

Comment /souschef to run again

Copilot AI and others added 2 commits August 11, 2026 01:23
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix concurrent runs race on shared assets branch Retry upload-asset pushes after concurrent branch updates Aug 11, 2026
Copilot AI requested a review from pelikhan August 11, 2026 01:26
@pelikhan
pelikhan marked this pull request as ready for review August 11, 2026 01:30
Copilot AI balanced review requested due to automatic review settings August 11, 2026 01:30
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #51893 does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold: 100).

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

PR Code Quality Reviewer completed the code quality review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.individual.githubcopilot.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.individual.githubcopilot.com"

See Network Configuration for more information.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Ponytail Reviewer completed successfully!

Lean already. Ship.

Generated by Ponytail Reviewer for #51893

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds bounded recovery for concurrent asset-branch updates.

Changes:

  • Retries non-fast-forward pushes after fetching and rebasing.
  • Preserves diagnostics and avoids retrying permanent failures.
  • Tests recovery, retry exhaustion, and non-retryable errors.
Show a summary per file
File Description
actions/setup/js/upload_assets.cjs Implements conflict-aware push retries.
actions/setup/js/upload_assets.test.cjs Covers retry and failure scenarios.

Review details

Tip

Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Balanced

@github-actions github-actions Bot mentioned this pull request Aug 11, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry logic for concurrent branch updates looks correct and well-tested. Implementation uses --porcelain with ignoreReturnCode, joins stderr+stdout for error detection, uses a regex covering both non-fast-forward and fetch first patterns, retries with fetch+rebase, and throws immediately for non-concurrent failures. Tests cover all three code paths. No blocking issues found.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · sonnet46 · 19.2 AIC · ⌖ 9.05 AIC · ⊞ 5.4K

@github-actions

Copy link
Copy Markdown
Contributor

🧪 Test Quality Sentinel Report

Test Quality Score: 100/100 — Excellent

Analyzed 3 test(s): 3 design, 0 implementation, 0 violation(s).

📊 Metrics (3 tests)
Metric Value
Analyzed 3 (Go: 0, JS: 3)
✅ Design 3 (100%)
⚠️ Implementation 0 (0%)
Edge/error coverage 3 (100%)
Duplicate clusters 0
Inflation No (2:1 ratio at threshold)
🚨 Violations 0
Test File Classification Notes
should fetch, rebase, and retry after a concurrent push upload_assets.test.cjs:372 behavioral_contract Refactored to use realistic git error output; excellent pattern matching
should stop after three failed push attempts upload_assets.test.cjs:384 behavioral_contract Boundary test for 3-attempt limit; properly validates retry exhaustion
should not retry a non-concurrent push failure upload_assets.test.cjs:396 behavioral_contract NEW: Critical edge case—tests that permission errors skip retry logic

Key Strengths:

  • High design coverage: All 3 tests verify user-visible behavior (retry logic)
  • Comprehensive edge-case testing: New test isolates concurrent vs. non-concurrent failure modes—essential safety signal
  • Proper mocking refactor: Shift from exception-throwing to realistic getExecOutput mocks reduces false confidence in the retry loop
  • Boundary testing: 3-attempt limit is properly validated; error discrimination logic is tested

Test Inflation Analysis:

  • Production: 11 lines added (new concurrent-race retry logic)
  • Tests: 22 lines added (setup mock, 3 comprehensive tests)
  • Ratio: 2:1 (at threshold)—justified for a critical retry/recovery feature

Verdict

Passed. 0% implementation tests (threshold: 30%). All tests verify design invariants—retry behavior, attempt limits, and failure-mode discrimination. New edge-case test is a high-value safety signal for concurrent push handling.

🧪 Test quality analysis by Test Quality Sentinel · haiku45 · 20.6 AIC · ⌖ 3.76 AIC · ⊞ 7.6K ·
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 100/100. 0% implementation tests (threshold: 30%).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd — requesting changes on two correctness issues before merging.

📋 Key Themes & Highlights

Key Issues

  • Missing git rebase --abort — if rebase encounters a conflict the working tree is left in REBASE_HEAD state, corrupting subsequent Git operations in the same runner (high risk for CI environments).
  • stderr/stdout ordering for --porcelain — porcelain rejection lines go to stdout; current join order puts stderr first, so the regex and error message may miss them on some git versions.
  • Minor: retry-exhaustion test could be more explicit about the surfaced error value.

Positive Highlights

  • ✅ Retry cap (3 attempts) and fast-fail for permanent errors (permissions, branch protection) are well-designed.
  • ✅ Three-scenario test suite (recovery, exhaustion, non-retryable) matches the three code paths exactly — great TDD coverage.
  • ✅ Warning log on each retry provides actionable diagnostics for operators.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 · 29.6 AIC · ⌖ 6.71 AIC · ⊞ 7.1K
Comment /matt to run again

const remoteBranch = `refs/remotes/origin/${normalizedBranchName}`;
await exec.exec("git", ["fetch", "--no-tags", "origin", `+refs/heads/${normalizedBranchName}:${remoteBranch}`]);
await exec.exec("git", ["rebase", remoteBranch]);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] If git rebase fails (e.g. merge conflict), the process throws without aborting the in-progress rebase, leaving the working tree in a broken REBASE_HEAD state for any subsequent Git operations in the same runner.

💡 Suggested fix: abort rebase on failure
try {
  await exec.exec('git', ['rebase', remoteBranch]);
} catch (rebaseErr) {
  await exec.exec('git', ['rebase', '--abort'], { ignoreReturnCode: true });
  throw rebaseErr;
}

Without --abort, a rebase conflict corrupts the workspace for any further Git operations.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 56402d0. Added try/catch around git rebase that calls git rebase --abort with ignoreReturnCode: true before rethrowing, so the working tree is never left in a broken REBASE_HEAD state. A test for this path was also added.

}
const pushError = [pushResult.stderr, pushResult.stdout].filter(Boolean).join("\n").trim() || `git push exited with code ${pushResult.exitCode}`;
const isNonFastForward = /non-fast-forward|fetch first/i.test(pushError);
if (!isNonFastForward || attempt === maxPushAttempts) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] The error message is built by joining stderr then stdout, but --porcelain output (including rejection lines like ! [rejected] ...) goes to stdout, not stderr. On a rejection, stderr is often empty and stdout carries the diagnostic — so the order should be stdout first, or both should be shown clearly.

💡 Suggested fix
const pushError = [pushResult.stdout, pushResult.stderr].filter(Boolean).join("\n").trim()
  || `git push exited with code ${pushResult.exitCode}`;

This ensures the porcelain rejection line (! [rejected]) is the leading text in the error message and in the regex test for isNonFastForward.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5e1da83. Swapped the join order to [pushResult.stdout, pushResult.stderr] so the porcelain rejection line (! [rejected]) leads the error message and is the primary input to the isNonFastForward regex. Test mocks updated to place the rejection text in stdout to match actual --porcelain behaviour.

expect(mockExec.exec).toHaveBeenCalledWith("git", ["fetch", "--no-tags", "origin", "+refs/heads/assets/test-workflow:refs/remotes/origin/assets/test-workflow"]);
expect(mockExec.exec).toHaveBeenCalledWith("git", ["rebase", "refs/remotes/origin/assets/test-workflow"]);
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The retry-exhaustion test verifies getExecOutput is called 3 times, but doesn't assert the final error message thrown (via mockCore.setFailed) contains the right content. If the error were swallowed or reformatted, the test would still pass.

💡 Suggested assertion
expect(mockCore.setFailed).toHaveBeenCalledWith(
  expect.stringContaining('non-fast-forward')
);

This is partially present but the test name says "stop after three attempts" — adding an explicit assertion about what error is surfaced makes the specification complete.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("non-fast-forward")) was already present in the test as of d67cbba. The test mock was also updated in 5e1da83 to put the rejection text in stdout (matching --porcelain output), so the assertion now exercises the correct code path end-to-end.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot quick triage: this PR has a fresh blocking Matt Pocock review requesting changes. Please address the missing git rebase --abort cleanup on retry failure and verify the porcelain output handling, then run the pr-finisher skill. Run: https://github.com/github/gh-aw/actions/runs/31450173394

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 24.1 AIC · ⌖ 5.21 AIC · ⊞ 6.1K ·
Comment /souschef to run again

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot quick triage: this PR has a fresh blocking Matt Pocock review requesting changes. Please address the missing git rebase --abort cleanup on retry failure and verify the porcelain output hand...

Addressed in commit $(git rev-parse --short HEAD). Added try/catch around git rebase that calls git rebase --abort (with ignoreReturnCode: true) before rethrowing, so the working tree is always left in a clean state when rebase fails. Added a test that mocks a conflicting rebase and verifies --abort is called and the error surfaces via setFailed. The porcelain output handling was already correct — both stderr and stdout are combined (in that order) so rejection messages from either stream are captured.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot quick triage: your follow-up already landed after the prior sous-chef nudge, and checks now look complete. Please do one final pr-finisher pass focused on the remaining maintainer confidence items from the Matt review response, confirm the branch is clean after the rebase-abort safeguard test, and hand back a short maintainer-facing summary.

Branch refresh was requested. Run: https://github.com/github/gh-aw/actions/runs/31454952282

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 15.9 AIC · ⌖ 5.21 AIC · ⊞ 8.5K ·
Comment /souschef to run again

…t push --porcelain output

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@pelikhan
pelikhan merged commit fb35db2 into main Aug 11, 2026
9 checks passed
@pelikhan
pelikhan deleted the copilot/upload-asset-fix-concurrent-race branch August 11, 2026 03:40
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.86.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

upload-asset: concurrent runs race on the shared assets branch (single non-retried push)

4 participants