Skip to content

ci(github): pin all actions to commit SHAs to comply with ASF allow-list policy - #2403

Open
acelyc111 wants to merge 7 commits into
apache:v2.5from
acelyc111:backport/v2.5/pin-actions-to-shas
Open

ci(github): pin all actions to commit SHAs to comply with ASF allow-list policy#2403
acelyc111 wants to merge 7 commits into
apache:v2.5from
acelyc111:backport/v2.5/pin-actions-to-shas

Conversation

@acelyc111

@acelyc111 acelyc111 commented May 26, 2026

Copy link
Copy Markdown
Member

Summary

Backport of #2398 to v2.5, adapted to v2.5's workflow structure (which predates master's #1791 composite-action refactor).

Why this is needed

The ASF GitHub Actions Policy (infra.apache.org/github-actions-policy.html, INFRA-27084) blocks third-party actions in apache/* repositories unless they appear in the ASF organization-wide allow-list at apache/infrastructure-actions. Only actions in apache/* and actions/* namespaces are auto-allowed; everything else must be pinned to a commit SHA explicitly listed in that allow-list.

On v2.5, action refs like dorny/paths-filter@v2, docker/build-push-action@v6, hadolint/hadolint-action@v3.1.0, gaurav-nelson/github-action-markdown-link-check@1.0.13, etc. are not on the allow-list, so workflow runs end in conclusion: startup_failure with an empty jobs array — meaning no logs, no PR check entry, the workflow appears "silent" but is actually being blocked. This is exactly what happens to existing v2.5 PRs like #2394, where the Cpp CI run 24079456724 failed to start with the message:

The action dorny/paths-filter@v2 is not allowed in apache/incubator-pegasus because all actions must be from a repository owned by your enterprise, created by GitHub, or match one of the patterns: ...

What this PR does

Pin every action ref in v2.5's workflows to a commit SHA approved by the current ASF allow-list. SHAs match those used on master (#2398) where the ASF allow-list still accepts them; where master's SHA has since been expired/replaced in the allow-list, the latest non-expired SHA from the same major version line is used (notably docker/setup-qemu-action@v3.7.0 and golangci/golangci-lint-action@v9.2.1).

Two non-mechanical adjustments mirror master:

apache/skywalking-eyes@main is intentionally left untouched (master also keeps @main; the apache/* namespace is auto-allowed regardless of ref).

One job uses actions/checkout@v3 instead of @v4.3.1

cpp_clang_format_linter runs inside the apache/pegasus:clang-format-3.9 container. That container's base image ships GLIBC < 2.28, which cannot run the Node 20 binaries shipped with actions/checkout@v4+. The check fails with:

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found

The clean solution on master was #2063 (Jul 2024): bump clang-format from 3.9 to 14, switch the job to a plain ubuntu-22.04 runner, and delete the apache/pegasus:clang-format-3.9 image. That refactor reformats the entire C++ codebase and is intentionally out of scope for this PR — it deserves a separate v2.5 backport. For now this PR keeps actions/checkout@v3 in just this one job; actions/* is auto-allowed by the ASF policy regardless of ref, so this does not reintroduce the policy block.

Verification

  • ✅ All 16 workflow YAMLs parse successfully (yaml.safe_load)
  • ✅ Every uses: ref in this branch was checked against ASF approved_patterns.yml from apache/infrastructure-actionsall pass
  • ✅ Diff is mechanical: 13 files, +186/-97 lines, dominated by # vendor/name@vX.Y.Z comment lines preceding each pinned SHA (matching master's ci(github): pin all actions to exact commit SHAs instead of tags or branch references #2398 style)
  • Confirmed Cpp CI runs to a real result on this PR (no more startup_failure) — see CI checks below

CI status of this PR

After pushing this branch, several workflows that were previously silently blocked (Cpp CI, Standardization Lint) now actually run, which directly proves the ASF policy block has been lifted. Three CI failures remain; only one (Cpp Lint) is something this PR introduced and addresses, the other two are pre-existing or unrelated:

Check Result Cause Status
Cpp CI / Lint ❌ → ✅ (after follow-up commit) Initially failed because the SHA pin pulled in actions/checkout@v4.3.1 (Node 20) which is incompatible with the clang-format-3.9 container's GLIBC < 2.28. Fixed by keeping @v3 for that one job (see "Why one job uses @V3" above). Addressed in this PR
Standardization Lint / Lint PR title ❌ → ✅ The PR title started with [backport] which amannn/action-semantic-pull-request rejects as not a valid Conventional Commits prefix. Fixed by editing the title. Addressed (title updated)
Labeler / Module Labeler Resource not accessible by integration (the action lacks write permission to add labels). This is a pre-existing fork-PR limitation: pull_request_target on Apache repos does not grant write tokens to forks for security reasons. Not introduced by this PR. The same failure would occur on any v2.5 PR from a fork. Out of scope here. Pre-existing, ignored

What this PR does NOT do

Related

…ist policy

Backport of apache#2398 to v2.5, adapted to v2.5's workflow structure (which
predates master's apache#1791 composite-action refactor).

According to the ASF GitHub Actions Policy, third-party actions are
blocked by default in apache/* repositories unless they appear in the
ASF organization-wide allow-list. Only actions in the apache/* and
actions/* namespaces are auto-allowed; everything else must be pinned
to a commit SHA listed in apache/infrastructure-actions.

On v2.5, action refs like 'dorny/paths-filter@v2',
'docker/build-push-action@v6', 'hadolint/hadolint-action@v3.1.0', etc.
are not on the allow-list and cause workflow runs to end in
'startup_failure' with no jobs (no logs, no PR check). For example,
the Cpp CI run for PR apache#2394 failed to start for exactly this reason
(see https://github.com/apache/incubator-pegasus/actions/runs/24079456724).

Pin every action ref in v2.5's workflows to a commit SHA from the
current ASF allow-list. SHAs match those used on master where
applicable; where master's SHA has since been expired/replaced in the
ASF allow-list, pick the latest non-expired SHA from the same major
version line.

Two non-mechanical adjustments mirror master:

- Drop the 'codecov/codecov-action@v2' step in lint_and_test_go-client
  (master removed it via apache#1790 in 2023; the moving '@v2' tag is also a
  supply-chain risk).
- Replace 'gaurav-nelson/github-action-markdown-link-check@1.0.13'
  with 'tcort/github-action-markdown-link-check@<sha>' in
  standardization_lint (master switched via apache#2329 because gaurav-nelson
  is not in the ASF allow-list).

ASF policy reference:
- https://infra.apache.org/github-actions-policy.html
- https://github.com/apache/infrastructure-actions
- https://issues.apache.org/jira/browse/INFRA-27084
Copilot AI review requested due to automatic review settings May 26, 2026 16:46
@acelyc111
acelyc111 requested a review from empiredan May 26, 2026 16:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

The apache/pegasus:clang-format-3.9 container is built on an older base
image whose GLIBC is < 2.28. Node 20 binaries shipped with
actions/checkout v4+ cannot start in this environment, causing the
checkout step to abort with:

  /__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6:
    version `GLIBC_2.28' not found

Pin actions/checkout to @V3 (Node 16) for this single job. actions/* is
auto-allowed by the ASF allow-list regardless of ref, so this does not
reintroduce the policy block that this PR fixes.
@acelyc111 acelyc111 changed the title [backport] ci(github): pin all actions to commit SHAs to comply with ASF allow-list policy ci(github): pin all actions to commit SHAs to comply with ASF allow-list policy May 28, 2026
acelyc111 added 3 commits May 28, 2026 15:03
The previous workaround (keeping actions/checkout@v3 inside the
apache/pegasus:clang-format-3.9 container) no longer works: GitHub
now executes actions/checkout@v3 on Node 20, which fails to start
inside the container because the image ships GLIBC < 2.28 and Node 20
requires GLIBC 2.28+:

  /__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version
  `GLIBC_2.28' not found (required by /__e/node20/bin/node)

Switch the cpp_clang_format_linter job to run directly on
ubuntu-22.04 (no container), pin actions/checkout to v4.3.1 (matching
the rest of the workflow), and extract clang-format-3.9 from the same
image as a plain binary. This keeps clang-format 3.9 as the formatter
of record for v2.5 (no codebase-wide reformatting required) while
unblocking the rest of the Cpp CI matrix that depends on this job.
…ker cp -L)

In the apache/pegasus:clang-format-3.9 image, /usr/bin/clang-format-3.9
is a relative symlink to ../lib/llvm-3.9/bin/clang-format. A plain
docker cp copies the symlink verbatim, which lands on the host as a
dangling link:

  invalid symlink ".../clang-format-3.9" -> "../lib/llvm-3.9/bin/clang-format"

Fix by resolving the symlink to its real path inside the container
with readlink -f, and by passing -L to docker cp so the binary itself
is copied out.
…nstall

Extracting the clang-format-3.9 binary out of the legacy image is not
enough on its own: the binary is dynamically linked against
libLLVM-3.9.so.1, which ubuntu-22.04 no longer packages, so it fails
on the host with:

  clang-format-3.9: error while loading shared libraries:
    libLLVM-3.9.so.1: cannot open shared object file

Instead, install a tiny /usr/local/bin/clang-format-3.9 wrapper that
runs the real binary inside the apache/pegasus:clang-format-3.9 image
with the workspace bind-mounted at the same path, so paths produced
by run-clang-format.py resolve identically inside and outside the
container. This keeps clang-format 3.9 as the formatter of record for
v2.5 without requiring its libraries on the runner host.
The default shell on Linux runners is /bin/sh (dash on Ubuntu). Some
test setup steps (notably zkServer.sh, invoked from run.sh test) rely
on bash-specific behavior; under sh they fail at startup with no
visible error, producing the symptom 'Starting zookeeper ... FAILED
TO START' followed by exit code 1 from every Test ASAN matrix job.

This previously went unnoticed on v2.5 because workflows were blocked
by the ASF allow-list at startup before any job ran. Now that the
allow-list block is lifted (by the previous commits in this PR), test
jobs reach the run.sh stage and surface this latent issue.

master already declares 'defaults.run.shell: bash' in this same
workflow; v2.5 simply never picked it up. Add the same declaration so
v2.5 PRs can run the C++ test matrix to completion.
@acelyc111

Copy link
Copy Markdown
Member Author

Pushed 533f432ac (ci(github): set workflow default shell to bash) to investigate the ASAN test failures seen on the previous head f43f08c.

What I observed on the previous attempt

All 31 Test ASAN (...) jobs failed at the Unit Testing step with:

====================== run <module> ==========================
zookeeper-bin cannot be found under .../.zk_install, thus try to find an existing one
zookeeper-bin is found under current work dir ...
ZooKeeper JMX enabled by default
Using config: .../.zk_install/zookeeper-bin/bin/../conf/zoo.cfg
Starting zookeeper ... FAILED TO START
##[error]Process completed with exit code 1.

So every ASAN test job actually got past download/build artifact, started the test runner, and then died because zkServer.sh start could not bring up the embedded ZooKeeper. ZK fails ~1 second after Using config: ..., which is consistent with the JVM exiting before the readiness check loop.

Why this is surfacing on this PR

Comparing GitHub Actions step config between this PR and the last historically-successful Cpp CI run on a v2.5-base PR (limowang/fix/disk_abnormal commit 3904180c, run 24069709672):

This PR (failing) limowang's PR (success)
shell: for the test step sh -e {0} bash --noprofile --norc -e -o pipefail {0}
defaults.run.shell in workflow (absent) bash

master's lint_and_test_cpp.yaml already declares defaults.run.shell: bash. v2.5's copy never picked that up, so every step on v2.5 runs under sh (dash on Ubuntu). This wasn't visible until now because v2.5 PRs were blocked by the ASF allow-list before any job ran (this PR removes that block).

What 533f432ac does

Adds defaults.run.shell: bash at the top of .github/workflows/lint_and_test_cpp.yaml, mirroring master.

Caveat

Whether defaults.run.shell: bash alone fully fixes ZK startup is not yet proven — ZK's actual stderr lives in zookeeper.out inside dataDir, which isn't captured by GitHub Actions logs, so the 1-second JVM-level exit could in principle have a separate root cause. The new run for 533f432ac is just queued at the time of writing; once it reaches the Test ASAN matrix we'll know.

If Test ASAN still fails the same way under bash, the next step is to capture zookeeper.out by adding a cat .zk_install/zookeeper-bin/logs/zookeeper.out || true line to the test step on failure, similar to how master's workflow handles diagnostics.

The previous commit (defaults.run.shell: bash) did not actually fix the
'Starting zookeeper ... FAILED TO START' failure; ZK still exits ~1
second after launch. The shell change was confirmed in logs (steps now
run under bash --noprofile --norc -e -o pipefail), but ZK fails the
same way, so the root cause is elsewhere — likely inside the JVM, which
writes its real stderr to zookeeper.out (not surfaced by GitHub
Actions).

Add an 'if: failure()' diagnostics step to both Test ASAN and Test
Release jobs that dumps:
- java -version, JAVA_HOME, ulimit
- .zk_install/zookeeper-bin/ tree
- All candidate locations of zookeeper.out
- zoo.cfg

This is a TEMPORARY DIAGNOSTIC commit. Once the actual ZK failure
cause is identified (and presumably fixed in a follow-up), this
commit should be reverted before this PR is merged.
@acelyc111

Copy link
Copy Markdown
Member Author

Update on the Test ASAN/Release failures

After looking deeper, the defaults.run.shell: bash fix did not resolve the ZK startup failure. Logs confirm the step now runs under bash --noprofile --norc -e -o pipefail {0} (matching what works elsewhere), but ZK still exits ~1 second after launch with Starting zookeeper ... FAILED TO START. So shell is not the root cause.

Key finding: this is a pre-existing v2.5 problem, not a regression from this PR

I'd been mentally comparing this PR against the most recent successful Cpp CI run on what I thought was a v2.5 PR. Re-checking: PR #2387 (limowang/fix/disk_abnormal, run 24069709672) is actually master-base, not v2.5-base. It pulled apache/pegasus:thirdparties-bin-test-asan-ubuntu2204-master, not …-v2.5.

Querying the thirdparty-regular-push workflow's run history directly:

Branch Successful image rebuilds Latest success
master 33 2025-11-25
build-env-ubuntu-cmake-3 1 2025-05-06
build-env-ubuntu-sasl2-modules 1 2025-09-16
v2.5 0 (never)

v2.5's only run of that workflow was a push-triggered run on 2026-04-09 that startup_failure'd before any job (almost certainly the same ASF allow-list block this PR exists to fix).

So apache/pegasus:thirdparties-bin-test-ubuntu2204-v2.5 is a one-shot image, presumably hand-built when v2.5 was cut, never refreshed. Whatever's wrong with its JVM / ZK runtime has been broken for the whole life of v2.5; the issue only became visible now because this is the first time Test ASAN/Test Release jobs have ever made it past the ASF allow-list / startup phase on a v2.5 PR.

Concrete evidence the image is the problem and not anything in this PR

  • Build ASAN and Build Release use the same image and pass cleanly. They never start the JVM, only link against it.
  • Test ASAN/Release are the only jobs that invoke ./run.sh test, which calls zkServer.sh start. JVM exits ~1 second after Using config: ...zoo.cfg.
  • The PR diff only touches .github/workflows/*.yml — there is no code path through which it could affect a Docker Hub image.

What I just pushed (945b2bd32)

A temporary if: failure() diagnostics step on the two Unit Testing steps that dumps java -version, ulimit, the ZK install dir, and every candidate location of zookeeper.out. Once the next run finishes, we'll have the JVM's actual stderr, which should pin down the failure to one of: JDK broken in image, glibc/libc++ mismatch, dataDir permissions, or stack/mmap denied by container security.

This commit should be reverted before merging this PR. It exists only to diagnose the unrelated v2.5 image issue.

Suggested path forward for this PR

The PR's stated goal — making v2.5 PRs comply with the ASF allow-list and surface non-startup_failure CI results — is achieved (Lint, Build ASAN, Build Release, Build with jemalloc, IWYU, Standardization Lint, Module Labeler, Golang Lint/Test all pass). The remaining Test ASAN/Test Release failures are a separate, longstanding v2.5 release-branch infrastructure issue: the third-party image needs to be rebuilt before C++ unit tests on v2.5 can ever pass.

A few options for how to land this PR without holding it on the image issue:

  1. Land it as-is, treating Test ASAN/Test Release red as a known v2.5 issue tracked separately.
  2. Land it with the diagnostics commit reverted, and open a follow-up issue (or PR) to rebuild apache/pegasus:thirdparties-bin-test-ubuntu2204-v2.5 once the diagnostics output identifies the specific root cause.
  3. If preferred, I can add an if: false (or a matrix-skip) on test_ASAN / test_Release for v2.5 only, with a clear comment, so v2.5 PRs don't show red until the image is fixed.

Happy to take whichever direction you'd like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants