Skip to content

fix(ci): emit a copyable audit-doc skeleton when the dependency gate fails - #3747

Merged
MohammadHaroonAbuomar merged 4 commits into
microsoft:mainfrom
sylvesterkaczmarek:fix/3722-dependabot-audit-template
Sep 13, 2026
Merged

MohammadHaroonAbuomar merged 4 commits into
microsoft:mainfrom
sylvesterkaczmarek:fix/3722-dependabot-audit-template

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek Sylvester Kaczmarek (sylvesterkaczmarek) commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Make the dependency-audit gate actionable when a semver-major Dependabot PR cannot supply the required audit document itself.

Problem

Major Dependabot updates are intentionally not exempt from vendored-patch-audit.sh, but Dependabot cannot author the required docs/dependency-audits/... file.

The check therefore fails with only a generic instruction, leaving a maintainer to reconstruct the required document shape manually.

Changes

  • keep the existing major-update audit requirement unchanged
  • emit a ready-to-copy dependency-audit document skeleton when the gate fails
  • include the required audit headings and changed dependency-file context
  • write the same guidance to $GITHUB_STEP_SUMMARY when running in GitHub Actions
  • add regression tests for terminal output and job-summary output

Testing

Added focused regression coverage in:

scripts/tests/test_vendored_patch_audit.py

The tests cover the missing-audit failure path and GitHub Actions summary generation.

The branch is based on upstream main at 7d0cef5d.

Addresses #3722.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added tests size/M Medium PR (< 200 lines) labels Aug 16, 2026
@github-actions

Copy link
Copy Markdown

PR Review Summary

Check Status Details
🔍 Code Review ⚠️ Missing No current-run comment
🛡️ Security Scan ⚠️ Missing No current-run comment
🔄 Breaking Changes ⚠️ Missing No current-run comment
📝 Docs Sync ⚠️ Missing No current-run comment
🧪 Test Coverage ⚠️ Missing No current-run comment

Verdict: ⚠️ AI review incomplete; ready for human review

AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims.

@github-actions

Copy link
Copy Markdown

🟡 Contributor Check: MEDIUM

Check Result
Profile MEDIUM
Credential LOW
Overall MEDIUM

Automated check by AGT Contributor Check.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor check flagged MEDIUM risk label Aug 16, 2026
@imran-siddique Imran Siddique (imran-siddique) changed the title Fix/3722 dependabot audit template fix(ci): emit a copyable audit-doc skeleton when the dependency gate fails Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed #3722. This is the right shape of fix for it, and I checked the template against the repository's own convention rather than reading it for plausibility.

First, housekeeping: I retitled this PR. It was Fix/3722 dependabot audit template, and Validate PR title had been failing on it since 2026-08-16 with No release type found in pull request title. Seven days red for a reason that had nothing to do with the code. The gate wants a conventional-commits prefix and a subject that does not start with a capital. New title is fix(ci): emit a copyable audit-doc skeleton when the dependency gate fails. Nothing else changed, and the gate is green on the current run.

The scope is right and the restraint is the good part. The major-update audit requirement is untouched. Both rules in #3722 were individually correct: the exemption should not cover majors, and a doc that is not in the diff is not an audit of that diff. The gap was that nothing told a maintainer what to write. That is what this closes, and it closes it without weakening either rule. Widening the exemption would have been the tempting fix and the wrong one.

The template matches the actual convention, which I verified three ways.

  • docs/dependency-audits/README.md lists required sections as "Which dependencies changed and why", "Security advisory relevance (CVE numbers if applicable)", "Breaking change risk assessment". The three headings emitted correspond one to one.
  • 2026-08-07-cryptography-50-cloud-board.md carries exactly the title / last_reviewed / owner frontmatter this emits, so the skeleton is not inventing a requirement the gate does not have; it is reproducing what real audit docs look like.
  • The filename pattern in the template is the one vendored-patch-audit.sh greps for, including the date prefix.

That same audit doc is direct evidence the deadlock is real rather than theoretical. Its opening line reads: supersedes dependabot #3588, same bump, the audit-trail gate requires this doc in the bumping PR, which dependabot cannot author for majors. Someone already paid this cost by hand, and this PR is what would have saved them the reconstruction.

Writing to $GITHUB_STEP_SUMMARY as well as stdout is worth calling out. The failing job log is where nobody looks; the summary is on the PR page. That is the difference between a message that exists and a message that gets read.

Non-blocking: the template headings are Title Case ("Which Dependencies Changed And Why") while the README and the existing audit docs use sentence case. Cosmetic, and worth matching if you touch this again.

Approving. I have push but not merge on this repository, so it needs a maintainer.

@imran-siddique

Copy link
Copy Markdown
Collaborator

Sylvester Kaczmarek (@sylvesterkaczmarek) heads up that I edited the PR title directly rather than asking you to. It was failing Validate PR title since 16 August purely on format: the gate needs a conventional-commits prefix and a lowercase subject start, and Fix/3722 dependabot audit template has neither. The code was never the problem. Title is now fix(ci): emit a copyable audit-doc skeleton when the dependency gate fails and that check is green. Full review above.

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I’ve applied the non-blocking suggestion and changed the three generated audit headings to sentence case, with the regression assertions updated accordingly.

The new push automatically dismissed the previous approval, so a re-review would be appreciated when convenient. The fresh Actions are currently awaiting maintainer approval to run.

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Imran Siddique (@imran-siddique) The sentence-case headings and their regression assertions are updated. That follow-up dismissed your earlier approval. Please re-review the current revision.

@MohammadHaroonAbuomar MohammadHaroonAbuomar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Commits 26e2c4a and e90d9ac have no Signed-off-by trailer. The DCO check has not run yet because the fork workflows are gated, but it will fail once a maintainer approves them. Please git rebase --signoff and force-push; everything else in this PR is already verified and this is the only remaining item.

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

MohammadHaroonAbuomar DCO sign-off is now fixed on all commits at 688359b7. All PR commits include Signed-off-by and are GitHub-verified; code content is unchanged. Please re-review when convenient.

Signed-off-by: Sylvester Kaczmarek <assistant@SylvesterKaczmarek.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <assistant@SylvesterKaczmarek.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

MohammadHaroonAbuomar The DCO item is fixed on the current head: all PR commits now include a Signed-off-by trailer matching the commit author identity. No substantive code change. Could you please re-review when convenient?

@MohammadHaroonAbuomar
MohammadHaroonAbuomar dismissed their stale review September 13, 2026 19:02

Re-verified in the group review; asks addressed.

@MohammadHaroonAbuomar MohammadHaroonAbuomar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after group integration review (tests, gates and adversarial pass on the combined change).

@MohammadHaroonAbuomar
MohammadHaroonAbuomar merged commit dc2f6f3 into microsoft:main Sep 13, 2026
125 checks passed
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Thanks for the review and merge.

Karim Mehalebi (karimad) pushed a commit to karimad/agent-governance-toolkit that referenced this pull request Sep 14, 2026
…fails (microsoft#3747)

* fix(ci): emit dependency audit template on gate failure

Signed-off-by: Sylvester Kaczmarek <assistant@SylvesterKaczmarek.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

* test(ci): cover dependency audit failure guidance

Signed-off-by: Sylvester Kaczmarek <assistant@SylvesterKaczmarek.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

* style(ci): use sentence-case audit headings

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

* test(ci): align audit heading expectations

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

---------

Signed-off-by: Sylvester Kaczmarek <assistant@SylvesterKaczmarek.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Yuvraj Singh (yuvrajsingh2428) pushed a commit to yuvrajsingh2428/agent-governance-toolkit that referenced this pull request Oct 1, 2026
…fails (microsoft#3747)

* fix(ci): emit dependency audit template on gate failure

Signed-off-by: Sylvester Kaczmarek <assistant@SylvesterKaczmarek.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

* test(ci): cover dependency audit failure guidance

Signed-off-by: Sylvester Kaczmarek <assistant@SylvesterKaczmarek.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

* style(ci): use sentence-case audit headings

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

* test(ci): align audit heading expectations

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>

---------

Signed-off-by: Sylvester Kaczmarek <assistant@SylvesterKaczmarek.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: yuvrajsingh2428 <offcyuvi2428@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:MEDIUM Contributor check flagged MEDIUM risk size/M Medium PR (< 200 lines) tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants